| Title |
Data Management Infrastructure for European XFEL |
| Authors |
- J. Malka, S. Aplin, D. Boukhelef, K. Filippakopoulos, L.G. Maia, T. Piszczek, Mr. Previtali, J. Szuba, K. Wrona
EuXFEL, Schenefeld, Germany
- S. Dietrich, MA. Gasthuber, J. Hannappel, M. Karimi, Y. Kemp, R. Lueken, T. Mkrtchyan, K. Ohrenberg, F. Schlünzen, P. Suchowski, C. Voss
DESY, Hamburg, Germany
|
| Abstract |
Effective data management is crucial to ensure research data is easily accessible and usable. We will present design and implementation of the European XFEL data management infrastructure supporting high level data management services. The system architecture comprises four layers of storage systems, each designed to address specific challenges. The first layer, referred to as online, is designed as a fast cache to accommodate extreme high rates (up to 15GB/s) of data generated during experiment at single scientific instrument. The second layer, called high-performance storage, provides necessary capabilities for data processing both during and after experiments. The layers are incorporated into a single infiniband fabric and connected through a 4km long 1Tb/s link. This allows fast data transfer from the European XFEL experiment hall to the DESY computing center. The third layer, mass-storage, extends the capacity of data storage system to allow mid-term data access for detailed analysis. Finally, the tape archive, provides data safety and long-term archive (5-10years). The high performance and mass storage systems are connected to computing clusters. This allows users to perform near-online and offline data analysis or alternatively export data outside of the European XFEL facility. The data management infrastructure at the European XFEL has the capacity to accept and process up to 2PB of data per day, which demonstrates the remarkable capabilities of all the sub-services involved in this process.
|
| Paper |
download WE1BCO02.PDF [1.023 MB / 6 pages] |
| Cite |
download ※ BibTeX
※ LaTeX
※ Text/Word
※ RIS
※ EndNote |
| Conference |
ICALEPCS2023 |
| Series |
International Conference on Accelerator and Large Experimental Physics Control Systems (19th) |
| Location |
Cape Town, South Africa |
| Date |
09-13 October 2023 |
| Publisher |
JACoW Publishing, Geneva, Switzerland |
| Editorial Board |
Volker RW Schaa (GSI, Darmstadt, Germany); Andy Götz (ESRF, Grenoble, France); Johan Venter (SARAO, Cape Town, South Africa); Karen White (SNS, Oak Ridge, TN, USA); Marie Robichon (ESRF, Grenoble, France); Vivienne Rowland (SARAO, Cape Town, South Africa) |
| Online ISBN |
978-3-95450-238-7 |
| Online ISSN |
2226-0358 |
| Received |
06 October 2023 |
| Revised |
23 October 2023 |
| Accepted |
08 December 2023 |
| Issued/td>
| 12 December 2023 |
| DOI |
doi:10.18429/JACoW-ICALEPCS2023-WE1BCO02 |
| Pages |
952-957 |
| Copyright |
 Published by JACoW Publishing under the terms of the Creative Commons Attribution 4.0 International license. Any further distribution of this work must maintain attribution to the author(s), the published article's title, publisher, and DOI. |
|
