| Title |
Modular and Scalable Archiving for EPICS and Other Time Series Using ScyllaDB and Rust |
| Authors |
- D. Werder, T. Humar
PSI, Villigen PSI, Switzerland
|
| Abstract |
At PSI we currently run too many different products with the common goal of archiving timestamped data. This includes EPICS Channel Archiver as well as Archiver Appliance for EPICS IOC’s, a buffer storage for beam-synchronous data at SwissFEL, and more. This number of monolithic solutions is too large to maintain and overlaps in functionality. Each solution brings their own storage engine, file format and centralized design which is hard to scale. In this talk I report on how we factored the system into modular components with clean interfaces. At the core, the different storage engines and file formats have been replaced by ScyllaDB, which is an open source product with enterprise support and remarkable adoption in the industry. We gain from its distributed, fault-tolerant and scalable design. The ingest of data into ScyllaDB is factored into components according to the different type of protocols of the sources, e.g. Channel Access. Here we build upon the Rust language and achieve robust, maintainable and performant services. One interface to access and process the recorded data is the HTTP retrieval service. This service offers e.g. search among the channels by various criteria, full event data as well as aggregated and binned data in either json or binary formats. This service can also run user-defined data transformations and act as a source for Grafana for a first view into recorded channel data. Our setup for SwissFEL ingests the ~370k EPICS updates/s from ~220k PVs (scalar and waveform), having rates between 0.1 and 100 Hz.
|
| Paper |
download WE3BCO01.PDF [0.270 MB / 4 pages] |
| Slides |
download WE3BCO01_TALK.PDF [1.174 MB] |
| Cite |
download ※ BibTeX
※ LaTeX
※ Text/Word
※ RIS
※ EndNote |
| Conference |
ICALEPCS2023 |
| Series |
International Conference on Accelerator and Large Experimental Physics Control Systems (19th) |
| Location |
Cape Town, South Africa |
| Date |
09-13 October 2023 |
| Publisher |
JACoW Publishing, Geneva, Switzerland |
| Editorial Board |
Volker RW Schaa (GSI, Darmstadt, Germany); Andy Götz (ESRF, Grenoble, France); Johan Venter (SARAO, Cape Town, South Africa); Karen White (SNS, Oak Ridge, TN, USA); Marie Robichon (ESRF, Grenoble, France); Vivienne Rowland (SARAO, Cape Town, South Africa) |
| Online ISBN |
978-3-95450-238-7 |
| Online ISSN |
2226-0358 |
| Received |
04 October 2023 |
| Revised |
09 November 2023 |
| Accepted |
14 December 2023 |
| Issued/td>
| 14 December 2023 |
| DOI |
doi:10.18429/JACoW-ICALEPCS2023-WE3BCO01 |
| Pages |
1008-1011 |
| Copyright |
 Published by JACoW Publishing under the terms of the Creative Commons Attribution 4.0 International license. Any further distribution of this work must maintain attribution to the author(s), the published article's title, publisher, and DOI. |
|
