JACoW is a publisher in Geneva, Switzerland that publishes the proceedings of accelerator conferences held around the world by an international collaboration of editors.
TY - UNPB AU - Nilsson, J.M.C. AU - Harper, J.R. AU - Jones, M.D. AU - Richter, T.S. ED - Furukawa, Kazuro ED - Yan, Yingbing ED - Leng, Yongbin ED - Chen, Zhichu ED - Schaa, Volker R.W. TI - Managing high-performance data flows and file structures J2 - Proc. of ICALEPCS2021, Shanghai, China, 14-22 October 2021 CY - Shanghai, China T2 - International Conference on Accelerator and Large Experimental Physics Control Systems T3 - 18 LA - english AB - The beam intensity at the European Spallation Source will necessitate a high performance acquisition and recording system for the data from the user experiments. In addition to high neutron counts rates the expected large number of dynamic measurements per day calls for a flexible system that supports a high variability in sample set-ups. Apache Kafka has been chosen as the central data switchboard to handle all the event driven data sources from detectors as well as from the EPICS controls system. The file writing system centres around a facility wide pool of HDF5 file-writers that uses Apache Kafka also for command and control. File-writing jobs are posted to a topic on Apache Kafka and picked up by individual workers. This centralises and optimises resources, as I/O load can be balanced between different neutron instruments. Command messages embed a NeXus compliant structure to capture the raw data in a community agreed format. To simplify correctly defining the file structure, physical device locations can the visualised. Data inspection can be applied to find available data sources and easily allocate them locations in the file. PB - JACoW Publishing CP - Geneva, Switzerland ER -