JACoW logo

Journals of Accelerator Conferences Website (JACoW)

JACoW is a publisher in Geneva, Switzerland that publishes the proceedings of accelerator conferences held around the world by an international collaboration of editors.

RIS citation export for WEBL03: Managing high-performance data flows and file structures

AU  - Nilsson, J.M.C.
AU  - Harper, J.R.
AU  - Jones, M.D.
AU  - Richter, T.S.
ED  - Furukawa, Kazuro
ED  - Yan, Yingbing
ED  - Leng, Yongbin
ED  - Chen, Zhichu
ED  - Schaa, Volker R.W.
TI  - Managing high-performance data flows and file structures
J2  - Proc. of ICALEPCS2021, Shanghai, China, 14-22 October 2021
CY  - Shanghai, China
T2  - International Conference on Accelerator and Large Experimental Physics Control Systems
T3  - 18
LA  - english
AB  - The beam intensity at the European Spallation Source will necessitate a high performance acquisition and recording system for the data from the user experiments. In addition to high neutron counts rates the expected large number of dynamic measurements per day calls for a flexible system that supports a high variability in sample set-ups. Apache Kafka has been chosen as the central data switchboard to handle all the event driven data sources from detectors as well as from the EPICS controls system. The file writing system centres around a facility wide pool of HDF5 file-writers that uses Apache Kafka also for command and control. File-writing jobs are posted to a topic on Apache Kafka and picked up by individual workers. This centralises and optimises resources, as I/O load can be balanced between different neutron instruments. Command messages embed a NeXus compliant structure to capture the raw data in a community agreed format. To simplify correctly defining the file structure, physical device locations can the visualised. Data inspection can be applied to find available data sources and easily allocate them locations in the file.
PB  - JACoW Publishing
CP  - Geneva, Switzerland
ER  -