Distributed Caching at Cloud Scale with Apache Ignite for the C2MON Framework
307
T. Marques Oliveira, M. Bräger, B. Copy, S.E. Halastra, D. Martin Anido, A. Papageorgiou Koufidis
CERN, Geneva, Switzerland
The CERN Control and Monitoring platform (C2MON) is an open-source platform for industrial controls data acquisition, monitoring, control and data publishing. Its high availability, fault tolerance and redundancy make it a perfect fit to handle the complex and critical systems present at CERN. C2MON must cope with the ever-increasing flows of data produced by the CERN technical infrastructure, such as cooling and ventilation or electrical distribution alarms, while maintaining integrity and availability. Distributed caching is a common technique to dramatically increase the availability and fault tolerance of redundant systems. For C2MON we have replaced the existing legacy Terracotta caching framework with Apache Ignite. Ignite is an enterprise grade, distributed caching platform, with advanced cloud-native capabilities. It enables C2MON to handle high volumes of data with full transaction support and makes C2MON ready to run in the cloud. This article first explains the challenges we met when integrating Apache Ignite into the C2MON framework, and then demonstrates how Ignite enhances the capabilities of a monitor and control system in an industrial controls environment.
Implementing an Event Tracing Solution with Consistently Formatted Logs for the SKA Telescope Control System
311
S.N. Twum, W.A. Bode, A.F. Joubert, K. Madisa, P.S. Swart, A.J. Venter
SARAO, Cape Town, South Africa
A. Bridger
ROE, UTAC, Edinburgh, United Kingdom
A. Bridger
SKAO, Macclesfield, United Kingdom
Funding:South African Radio Astronomy Observatory The SKA telescope control system comprises several devices working on different hierarchies on different sites to provide a running observatory. The importance of logs, whether in its simplest form or correlated, in this system as well as any other distributed system is critical to fault finding and bug tracing. The SKA logging system will collect logs produced by numerous networked kubernetes deployments of devices and processes running a combination off-the-shelf, derived and bespoke software. The many moving parts of this complex system are delivered and maintained by different agile teams on multiple SKA Agile Release Trains. To facilitate an orderly and correlated generation of events in the running telescope, we implement a logging architecture which enforces consistently formatted logs with event tracing capability. We discuss the details of the architecture design and implementation, ending off with the limitations of the tracing solution in the context of a multiprocessing environment.
P.P. Goryl, M. Liszcz
S2Innovation, Kraków, Poland
S. Blanch-Torné
ALBA-CELLS Synchrotron, Cerdanyola del Vallès, Spain
R. Bourtembourg, A. Götz
ESRF, Grenoble, France
V. Hardion
MAX IV Laboratory, Lund University, Lund, Sweden
L. Pivetta
Elettra-Sincrotrone Trieste S.C.p.A., Basovizza, Italy
In 2019, the Tango Controls Collaboration decided to write down a formal specification of the existing Tango Controls protocol as Requests For Comments (RFC). The work resulted in a Markdown-formatted specification rendered in HTML and PDF on Readthedocs.io. The specification is already used as a reference during Tango Controls source code maintenance and for prototyping a new implementation. All collaborating institutes and several companies were involved in the work. In addition to providing the reference, the effort brought the Community more value: review and clarification of concepts and their implementation in the core libraries in C++, Java and Python. This paper summarizes the results, provides technical and organizational details about writing the RFCs for the existing protocol and presents the impact and benefits on future maintenance and development of Tango Controls.
The Square Kilometre Array (SKA) is an international effort to build two radio interferometers in South Africa and Australia forming one Observatory monitored and controlled from global headquarters (GHQ) based in the United Kingdom at Jodrell Bank. SKA is highly focused on adopting CI/CD practices for its software development. CI/CD stands for Continuous Integration \& Delivery and/or Deployment. Continuous Integration is the practice of merging all developers’ local copies into the mainline frequently. Continuous Delivery is the approach of developing software in short cycles ensuring it can be released anytime, and Continuous Deployment is the approach of delivering the software into operational use frequently and automatically. This paper analyses the decisions taken by the Systems Team (a specialized agile team devoted to developing and maintaining the tools that allow continuous practices) to promote the CI/CD practices with the TANGO-controls framework.
Pysmlib: A Python Finite State Machine Library for EPICS
330
D. Marcato, G. Arena, D. Bortolato, F. Gelain, G. Lilli, V. Martinelli, E. Munaron, M. Roetta, G. Savarese
INFN/LNL, Legnaro (PD), Italy
M.A. Bellato
INFN- Sez. di Padova, Padova, Italy
In the field of Experimental Physics and Industrial Control Systems (EPICS)*, the traditional tool to implement high level procedures is the Sequencer*. While this is a mature, fast, and well-proven software, it comes with some drawbacks. For example, it’s based on a custom C-like programming language which may be unfamiliar to new users and it often results in complex, hard to read code. This paper presents pysmlib, a free and open source Python library developed as a simpler alternative to the EPICS Sequencer. The library exposes a simple interface to develop event-driven Finite State Machines (FSM), where the inputs are connected to Channel Access Process Variables (PV) thanks to the PyEpics** integration. Other features include parallel FSM with multi-threading support and input sharing, timers, and an integrated watchdog logic. The library offers a lower barrier to enter and greater extensibility thanks to the large ecosystem of scientific and engineering python libraries, making it a perfect fit for modern control system requirements. Pysmlib has been deployed in multiple projects at INFN Legnaro National Laboratories (LNL), proving its robustness and flexibility. * L. R. Dalesio, M. R. Kraimer, and A. J. Kozubal. "EPICS architecture." ICALEPCS. Vol. 91. 1991. ** M. Newville, et al., pyepics/pyepics Zenodo. http://doi.org/10.5281/zenodo.592027
