ICALEPCS2019 - List of Authors (Wrona, K.)

Paper	Title	Page
TUCPR02	Data Exploration and Analysis with Jupyter Notebooks	799
	H. Fangohr, M. Beg, M. Bergemann, V. Bondar, S. Brockhauser, C. Carinan, R. Costa, F. Dall’Antonia, C. Danilevski, J.C. E, W. Ehsan, S.G. Esenov, R. Fabbri, S. Fangohr, G. Flucke, C. Fortmann-Grote, D. Fulla Marsa, G. Giovanetti, D. Goeries, S. Hauf, D.G. Hickin, T. Jarosiewicz, E. Kamil, M. Karnevskiy, Y. Kirienko, A. Klimovskaia, T.A. Kluyver, M. Kuster, L. Le Guyader, A. Madsen, L.G. Maia, D. Mamchyk, L. Mercadier, T. Michelat, J. Möller, I. Mohacsi, A. Parenti, M. Reiser, R. Rosca, D.B. Rück, T. Rüter, H. Santos, R. Schaffer, A. Scherz, M. Scholz, A. Silenzi, M. Spirzewski, J. Sztuk, J. Szuba, S. Trojanowski, K. Wrona, A.A. Yaroslavtsev, J. Zhu EuXFEL, Schenefeld, Germany S. Brockhauser BRC, Szeged, Hungary A. Campbell, A. Götz, J. Kieffer ESRF, Grenoble, France H. Fangohr University of Southampton, Southampton, United Kingdom E. Fernandez-del-Castillo, G. Sipos The EGI Foundation, Amsterdam, The Netherlands J. Hall, E. Pellegrini, J.F. Perrin ILL, Grenoble, France T. Holm Rod, J.R. Selknaes, J.W. Taylor ESS, Copenhagen, Denmark J. Reppin, F. Schlünzen, M. Schuh DESY, Hamburg, Germany
	Funding: With support from EU’s H{2}020 grants 823852 (PaNOSC) and #676541 (OpenDreamKit), the Gordon and Betty Moore Foundation GBMF #4856, the EPSRC’s CDT (EP/L015382/1) and program grant (EP/N032128/1). Jupyter notebooks are executable documents that are displayed in a web browser. The notebook elements consist of human-authored contextual elements and computer code, and computer-generated output from executing the computer code. Such outputs can include tables and plots. The notebook elements can be executed interactively, and the whole notebook can be saved, re-loaded and re-executed, or converted to read-only formats such as HTML, LaTeX and PDF. Exploiting these characteristics, Jupyter notebooks can be used to improve the effectiveness of computational and data exploration, documentation, communication, reproducibility and re-usability of scientific research results. They also serve as building blocks of remote data access and analysis as is required for facilities hosting large data sets and initiatives such as the European Open Science Cloud (EOSC). In this contribution we report from our experience of using Jupyter notebooks for data analysis at research facilities, and outline opportunities and future plans.
	Slides TUCPR02 [15.943 MB]
DOI •	reference for this paper ※ https://doi.org/10.18429/JACoW-ICALEPCS2019-TUCPR02
About •	paper received ※ 24 September 2019 paper accepted ※ 20 October 2019 issue date ※ 30 August 2020
Export •	reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)

WECPR03	Status of the Karabo Control and Data Processing Framework	936
	G. Flucke, N. Al-Qudami, M. Beg, M. Bergemann, V. Bondar, D. Boukhelef, S. Brockhauser, C. Carinan, R. Costa, F. Dall’Antonia, C. Danilevski, W. Ehsan, S.G. Esenov, R. Fabbri, H. Fangohr, D. Fulla Marsa, G. Giovanetti, D. Goeries, S. Hauf, D.G. Hickin, E. Kamil, Y. Kirienko, A. Klimovskaia, T.A. Kluyver, D. Mamchyk, T. Michelat, I. Mohacsi, A. Muennich, A. Parenti, R. Rosca, D.B. Rück, H. Santos, R. Schaffer, A. Silenzi, K. Wrona, C. Youngman, J. Zhu EuXFEL, Schenefeld, Germany S. Brockhauser BRC, Szeged, Hungary H. Fangohr University of Southampton, Southampton, United Kingdom
	To achieve a tight integration of instrument control and (online) data analysis, the European XFEL decided in 2011 to develop Karabo, a custom control and data processing system. Karabo provides control via event-driven communication. Signal/slot and request/reply patterns are implemented via a central message broker. Data pipelines for e.g. scientific workflows or detector calibration are implemented as direct TCP/IP connections. The core entities of Karabo are self-describing devices written in C++ or Python. They represent hardware, orchestrate other devices, or provide system services like data logging and configuration storage. To operate Karabo, a Python command line interface and a generic GUI written in PyQt are provided. Control and data widgets compose Karabo scenes that are provided by devices or are manually customized and stored together with device configurations in a central database. Since 2016, Karabo is used to commission and operate the currently three photon beam lines and six scientific instruments at the European XFEL. This contribution summarizes the status of Karabo, highlights achievements and lessons learned, and gives an outlook for future directions. Heisen, B., et al. (2013) In 14th International Conference on Accelerator and Large Experimental Physics Control Systems, ICALEPCS 2013. San Francisco, CA.
	Slides WECPR03 [2.660 MB]
DOI •	reference for this paper ※ https://doi.org/10.18429/JACoW-ICALEPCS2019-WECPR03
About •	paper received ※ 27 September 2019 paper accepted ※ 09 October 2019 issue date ※ 30 August 2020
Export •	reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)

Paper

Title

Page

Data Exploration and Analysis with Jupyter Notebooks

799

H. Fangohr, M. Beg, M. Bergemann, V. Bondar, S. Brockhauser, C. Carinan, R. Costa, F. Dall’Antonia, C. Danilevski, J.C. E, W. Ehsan, S.G. Esenov, R. Fabbri, S. Fangohr, G. Flucke, C. Fortmann-Grote, D. Fulla Marsa, G. Giovanetti, D. Goeries, S. Hauf, D.G. Hickin, T. Jarosiewicz, E. Kamil, M. Karnevskiy, Y. Kirienko, A. Klimovskaia, T.A. Kluyver, M. Kuster, L. Le Guyader, A. Madsen, L.G. Maia, D. Mamchyk, L. Mercadier, T. Michelat, J. Möller, I. Mohacsi, A. Parenti, M. Reiser, R. Rosca, D.B. Rück, T. Rüter, H. Santos, R. Schaffer, A. Scherz, M. Scholz, A. Silenzi, M. Spirzewski, J. Sztuk, J. Szuba, S. Trojanowski, K. Wrona, A.A. Yaroslavtsev, J. Zhu
EuXFEL, Schenefeld, Germany
S. Brockhauser
BRC, Szeged, Hungary
A. Campbell, A. Götz, J. Kieffer
ESRF, Grenoble, France
H. Fangohr
University of Southampton, Southampton, United Kingdom
E. Fernandez-del-Castillo, G. Sipos
The EGI Foundation, Amsterdam, The Netherlands
J. Hall, E. Pellegrini, J.F. Perrin
ILL, Grenoble, France
T. Holm Rod, J.R. Selknaes, J.W. Taylor
ESS, Copenhagen, Denmark
J. Reppin, F. Schlünzen, M. Schuh
DESY, Hamburg, Germany

Funding: With support from EU’s H{2}020 grants 823852 (PaNOSC) and #676541 (OpenDreamKit), the Gordon and Betty Moore Foundation GBMF #4856, the EPSRC’s CDT (EP/L015382/1) and program grant (EP/N032128/1).
Jupyter notebooks are executable documents that are displayed in a web browser. The notebook elements consist of human-authored contextual elements and computer code, and computer-generated output from executing the computer code. Such outputs can include tables and plots. The notebook elements can be executed interactively, and the whole notebook can be saved, re-loaded and re-executed, or converted to read-only formats such as HTML, LaTeX and PDF. Exploiting these characteristics, Jupyter notebooks can be used to improve the effectiveness of computational and data exploration, documentation, communication, reproducibility and re-usability of scientific research results. They also serve as building blocks of remote data access and analysis as is required for facilities hosting large data sets and initiatives such as the European Open Science Cloud (EOSC). In this contribution we report from our experience of using Jupyter notebooks for data analysis at research facilities, and outline opportunities and future plans.

Slides TUCPR02 [15.943 MB]

DOI •

reference for this paper ※ https://doi.org/10.18429/JACoW-ICALEPCS2019-TUCPR02

About •

paper received ※ 24 September 2019 paper accepted ※ 20 October 2019 issue date ※ 30 August 2020

Export •

reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)

WECPR03

Status of the Karabo Control and Data Processing Framework

936

G. Flucke, N. Al-Qudami, M. Beg, M. Bergemann, V. Bondar, D. Boukhelef, S. Brockhauser, C. Carinan, R. Costa, F. Dall’Antonia, C. Danilevski, W. Ehsan, S.G. Esenov, R. Fabbri, H. Fangohr, D. Fulla Marsa, G. Giovanetti, D. Goeries, S. Hauf, D.G. Hickin, E. Kamil, Y. Kirienko, A. Klimovskaia, T.A. Kluyver, D. Mamchyk, T. Michelat, I. Mohacsi, A. Muennich, A. Parenti, R. Rosca, D.B. Rück, H. Santos, R. Schaffer, A. Silenzi, K. Wrona, C. Youngman, J. Zhu
EuXFEL, Schenefeld, Germany
S. Brockhauser
BRC, Szeged, Hungary
H. Fangohr
University of Southampton, Southampton, United Kingdom

To achieve a tight integration of instrument control and (online) data analysis, the European XFEL decided in 2011 to develop Karabo*, a custom control and data processing system. Karabo provides control via event-driven communication. Signal/slot and request/reply patterns are implemented via a central message broker. Data pipelines for e.g. scientific workflows or detector calibration are implemented as direct TCP/IP connections. The core entities of Karabo are self-describing devices written in C++ or Python. They represent hardware, orchestrate other devices, or provide system services like data logging and configuration storage. To operate Karabo, a Python command line interface and a generic GUI written in PyQt are provided. Control and data widgets compose Karabo scenes that are provided by devices or are manually customized and stored together with device configurations in a central database. Since 2016, Karabo is used to commission and operate the currently three photon beam lines and six scientific instruments at the European XFEL. This contribution summarizes the status of Karabo, highlights achievements and lessons learned, and gives an outlook for future directions.
* Heisen, B., et al. (2013) In 14th International Conference on Accelerator and Large Experimental Physics Control Systems, ICALEPCS 2013. San Francisco, CA.

Slides WECPR03 [2.660 MB]

DOI •

reference for this paper ※ https://doi.org/10.18429/JACoW-ICALEPCS2019-WECPR03

About •

paper received ※ 27 September 2019 paper accepted ※ 09 October 2019 issue date ※ 30 August 2020

Export •

reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)