We present high time-resolution ULTRACAM optical and NOTCam infrared observations of the edge-on black hole X-ray transient Swift J1357.2-0933. Our data taken in 2012 and 2013 show the system to be at its pre-outburst magnitude and so the system is in quiescence. In contrast to other X-ray transients, the quiescent light curves of Swift J1357.2-0933 do not show the secondary star's ellipsoidal modulation. The optical light curve is dominated by variability with an optical fractional rms of ~35 per cent, a factor of >3 larger than what is observed in other systems at similar time-resolution. Optical flare events lasting 2-10min with amplitudes of up to ~1.5 mag are seen as well as numerous rapid ~0.8 mag dip events which are similar to the optical dips seen in outburst. Similarly the infrared J-band light curve is dominated by variability with a fractional rms of ~21 per cent and flare events lasting 10--30 min with amplitudes of up to ~1.5 mag are observed. The quiescent optical to mid-infrared spectral energy distribution in quiescence is dominated by a non-thermal component with a power--law index of -1.4, (the broad-band rms SED has a similar index) which arises from optically thin synchrotron emission most likely originating in a weak jet; the lack of a peak in the spectral energy distribution rules out advection-dominated models. Using the outburst amplitude--period relation for X-ray transients we estimate the quiescent magnitude of the secondary star to lie in the range V_min=22.7 to 25.6, which when combined with the absolute magnitude of the expected M4.5 V secondary star allows us to constrain to the distance to lie in the range 0.5 to 6.3 kpc. (Abridged)