Achieving a high signal-to-noise ratio is challenging in electron scattering experiments that require low average probe current or low total electron dose, e.g., time resolved hard-matter or radiation-sensitive soft matter experiments. A promising method for improving the signal-to-noise ratio when electron counts are low is to structure the electron wavefunction with optical fields and then retrieve the image via reconstruction algorithms in a single pixel imaging approach. When the electron-optical interaction is inelastic, such a scheme requires an electron energy filter. Here, we present numerical simulations of a time-of-flight energy filtering scheme for use in ultrafast electron microscopy, where a radiofrequency deflector cavity placed at the bottom of an electron microscope column provides a time-dependent momentum kick, dispersing the energy bands of the beam on a downstream detector. We estimate the filtering performance for electron single pixel imaging with an electron beam wavefunction shaped by a high intensity, highly coherent ultrafast light pulse and discuss future practical aspects.