Cavity-based X-ray Free Electron Lasers (FELs) such as the X-ray regenerative amplifier FEL (XRAFEL) [1] and the X-ray FEL oscillator [2] have drawn great interest as a means of producing high-brightness, fully coherent and stable hard x-ray pulses for high-repetition rate FELs [3]. However, high efficiency outcoupling of the stored cavity x-ray radiation remains challenging. Here we present a novel XRAFEL design to achieve efficient cavity outcoupling or Q-switching by introducing energy chirp in the electron beam while leaving the high-quality X-ray optics intact. During the FEL interaction, electron beam with an linear energy chirp can be slightly compressed or decompressed by the undulator, which leads to a gradual shift of radiation frequency outside the bandwidth of the Bragg crystal for efficient outcoupling. Our simulation results show that substantial power can be outcoupled from the X-ray cavity driven by chirped electron beams at 100 kHz repetition rate. We also discuss parameter tradeoff in such an XRAFEL scheme and a practical way to achieve the desired fast chirp control by a small, normal-conducting RF station in the LCLS-II [4]. [1] Z. Huang and R. D. Ruth. PRL96, 144801 (2006). [2] K.-J. Kim, Y. Shvyd'ko, S. Reiche, PRL100 244802 (2008). [3] G. Marcus, et al., PRL125, 254801 (2020). [4] M. Nasr, et al., in proceedings of IPAC'16 (Busan, Korea,2016).