WEBL03
ESS, Copenhagen, Denmark
STFC/RAL/ISIS, Chilton, Didcot, Oxon, United Kingdom
Tessella, Abingdon, United Kingdom
| Paper | Title | Page |
|---|---|---|
WEBL03 |
Managing high-performance data flows and file structures | |
|
||
| The beam intensity at the European Spallation Source will necessitate a high performance acquisition and recording system for the data from the user experiments. In addition to high neutron counts rates the expected large number of dynamic measurements per day calls for a flexible system that supports a high variability in sample set-ups. Apache Kafka has been chosen as the central data switchboard to handle all the event driven data sources from detectors as well as from the EPICS controls system. The file writing system centres around a facility wide pool of HDF5 file-writers that uses Apache Kafka also for command and control. File-writing jobs are posted to a topic on Apache Kafka and picked up by individual workers. This centralises and optimises resources, as I/O load can be balanced between different neutron instruments. Command messages embed a NeXus compliant structure to capture the raw data in a community agreed format. To simplify correctly defining the file structure, physical device locations can the visualised. Data inspection can be applied to find available data sources and easily allocate them locations in the file. | ||
|
|
Slides WEBL03 [1.851 MB] | |
| Cite • | reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml) | |