JACoW logo

Journals of Accelerator Conferences Website (JACoW)

JACoW is a publisher in Geneva, Switzerland that publishes the proceedings of accelerator conferences held around the world by an international collaboration of editors.


BiBTeX citation export for WEBL03: Managing high-performance data flows and file structures

@unpublished{nilsson:icalepcs2021-webl03,
  author       = {J.M.C. Nilsson and J.R. Harper and M.D. Jones and T.S. Richter},
  title        = {{Managing high-performance data flows and file structures}},
  booktitle    = {Proc. ICALEPCS'21},
  language     = {english},
  intype       = {presented at the},
  series       = {International Conference on Accelerator and Large Experimental Physics Control Systems},
  number       = {18},
  venue        = {Shanghai, China},
  publisher    = {JACoW Publishing, Geneva, Switzerland},
  month        = {03},
  year         = {2022},
  note         = {presented at ICALEPCS'21 in Shanghai, China, unpublished},
  abstract     = {{The beam intensity at the European Spallation Source will necessitate a high performance acquisition and recording system for the data from the user experiments. In addition to high neutron counts rates the expected large number of dynamic measurements per day calls for a flexible system that supports a high variability in sample set-ups. Apache Kafka has been chosen as the central data switchboard to handle all the event driven data sources from detectors as well as from the EPICS controls system. The file writing system centres around a facility wide pool of HDF5 file-writers that uses Apache Kafka also for command and control. File-writing jobs are posted to a topic on Apache Kafka and picked up by individual workers. This centralises and optimises resources, as I/O load can be balanced between different neutron instruments. Command messages embed a NeXus compliant structure to capture the raw data in a community agreed format. To simplify correctly defining the file structure, physical device locations can the visualised. Data inspection can be applied to find available data sources and easily allocate them locations in the file.}},
}