A high temporal stability of produced photon pulses is a key parameter for some classes of experiments, e.g., those using a pump-probe scheme. A longitudinal intra bunch-train feedback system, that reduces the intra bunch-train and the train-to-train arrival time fluctuations down to the sub-10 fs level was implemented at the European X-ray free electron laser (EuXFEL). The low arrival time jitter of the electron beam is preserved in the generated photon pulses. However, over long measurement periods, additional environmental factors acting on different time scales have to be considered. These factors include the temperature, relative humidity and in case of the European XFEL ground motions due to ocean activities. Mitigation of the residual timing drifts between pump laser and FEL pulses requires additional measures to disentangle the overlaid effects. The latest results and future challenges for the long-term arrival time stabilization will be presented.