ICALEPCS2011 - Table of Session: FRBHMU (Distributed Computing)

Paper

Title

Page

The Design of the Alba Control System: A Cost-Effective Distributed Hardware and Software Architecture.

1318

D.F.C. Fernández-Carreiras, D.B. Beltrán, T.M. Coutinho, G. Cuní, J. Klora, O. Matilla, R. Montaño, C. Pascual-Izarra, S. Pusó, R. Ranz, A. Rubio, S. Rubio-Manrique
CELLS-ALBA Synchrotron, Cerdanyola del Vallès, Spain

The control system of Alba is highly distributed from both hardware and software points of view. The hardware infrastructure for the control system includes in the order of 350 racks, 20000 cables and 6200 equipments. More than 150 diskless industrial computers, distributed in the service area and 30 multicore servers in the data center, manage several thousands of process variables. The software is, of course, as distributed as the hardware. It is also a success story of the Tango Collaboration where a complete software infrastructure is available "off the shelf". In addition Tango has been productively complemented with the powerful Sardana framework, a great effort in terms of development, which nowadays, several institutes benefit from. The whole installation has been coordinated from the beginning with a complete cabling and equipment database, where all the equipment, cables, connectors are described and inventoried. The so called "cabling database" is core of the installation. The equipments and cables are defined there. The basic configurations of the hardware like MAC and IP addresses, DNS names, etc. are also gathered in this database, allowing the network communication files and declaration of variables in the PLCs to be created automatically. This paper explains the design and the architecture of the control system, describes the tools and justifies the choices made. Furthermore, it presents and analyzes the figures regarding cost and performances.

Slides FRBHMUST01 [4.616 MB]

FRBHMUST02

Towards High Performance Processing in Modern Java Based Control Systems

1322

M. Misiowiec, W. Buczak, M. Buttner
CERN, Geneva, Switzerland

CERN controls software is often developed on Java foundation. Some systems carry out a combination of data, network and processor intensive tasks within strict time limits. Hence, there is a demand for high performing, quasi real time solutions. Extensive prototyping of the new CERN monitoring and alarm software required us to address such expectations. The system must handle dozens of thousands of data samples every second, along its three tiers, applying complex computations throughout. To accomplish the goal, a deep understanding of multithreading, memory management and interprocess communication was required. There are unexpected traps hidden behind an excessive use of 64 bit memory or severe impact on the processing flow of modern garbage collectors, including the state of the art Oracle GarbageFirst. Tuning JVM configuration significantly affects the execution of the code. Even more important is the amount of threads and the data structures used between them. Accurately dividing work into independent tasks might boost system performance. Thorough profiling with dedicated tools helped understand the bottlenecks and choose algorithmically optimal solutions. Different virtual machines were tested, in a variety of setups and garbage collection options. The overall work provided for discovering actual hard limits of the whole setup. We present this process of architecting a challenging system in view of the characteristics and limitations of the contemporary Java runtime environment.
http://cern.ch/marekm/icalepcs.html

Slides FRBHMUST02 [4.514 MB]

FRBHMUST03

Thirty Meter Telescope Observatory Software Architecture

1326

K.K. Gillies, C. Boyer
TMT, Pasadena, California, USA

The Thirty Meter Telescope (TMT) will be a ground-based, 30-m optical-IR telescope with a highly segmented primary mirror located on the summit of Mauna Kea in Hawaii. The TMT Observatory Software (OSW) system will deliver the software applications and infrastructure necessary to integrate all TMT software into a single system and implement a minimal end-to-end science operations system. At the telescope, OSW is focused on the task of integrating and efficiently controlling and coordinating the telescope, adaptive optics, science instruments, and their subsystems during observation execution. From the software architecture viewpoint, the software system is viewed as a set of software components distributed across many machines that are integrated using a shared software base and a set of services that provide communications and other needed functionality. This paper describes the current state of the TMT Observatory Software focusing on its unique requirements, architecture, and the use of middleware technologies and solutions that enable the OSW design.

Slides FRBHMUST03 [3.788 MB]

FRBHMULT04

Towards a State Based Control Architecture for Large Telescopes: Laying a Foundation at the VLT

1330

R. Karban, N. Kornweibel
ESO, Garching bei Muenchen, Germany
D.L. Dvorak, M.D. Ingham, D.A. Wagner
JPL, Pasadena, California, USA

Large telescopes are characterized by a high level of distribution of control-related tasks and will feature diverse data flow patterns and large ranges of sampling frequencies; there will often be no single, fixed server-client relationship between the control tasks. The architecture is also challenged by the task of integrating heterogeneous subsystems which will be delivered by multiple different contractors. Due to the high number of distributed components, the control system needs to effectively detect errors and faults, impede their propagation, and accurately mitigate them in the shortest time possible, enabling the service to be restored. The presented Data-Driven Architecture is based on a decentralized approach with an end-to-end integration of disparate independently-developed software components, using a high-performance standards-based communication middle-ware infrastructure, based on the Data Distribution Service. A set of rules and principles, based on JPL's State Analysis method and architecture, are established to avoid undisciplined component-to-component interactions, where the Control System and System Under Control are clearly separated. State Analysis provides a model-based process for capturing system and software requirements and design, helping reduce the gap between the requirements on software specified by systems engineers and the implementation by software engineers. The method and architecture has been field tested at the Very Large Telescope, where it has been integrated into an operational system with minimal downtime.

Slides FRBHMULT04 [3.504 MB]

FRBHMULT05

Middleware Trends and Market Leaders 2011

1334

A. Dworak, P. Charrue, F. Ehm, W. Sliwinski, M. Sobczak
CERN, Geneva, Switzerland

The Controls Middleware (CMW) project was launched over ten years ago. Its main goal was to unify middleware solutions used to operate CERN accelerators. An important part of the project, the equipment access library RDA, was based on CORBA, an unquestionable standard at the time. RDA became an operational and critical part of the infrastructure, yet the demanding run-time environment revealed some shortcomings of the system. Accumulation of fixes and workarounds led to unnecessary complexity. RDA became difficult to maintain and to extend. CORBA proved to be rather a cumbersome product than a panacea. Fortunately, many new transport frameworks appeared since then. They boasted a better design, and supported concepts that made them easy to use. Willing to profit from the new libraries, the CMW team updated user requirements, and in their terms investigated eventual CORBA substitutes. The process consisted of several phases: a review of middleware solutions belonging to different categories (e.g. data-centric, object-, and message-oriented) and their applicability to a communication model in RDA; evaluation of several market recognized products and promising start-ups; prototyping of typical communication scenarios; testing the libraries against exceptional situations and errors; verifying that mandatory performance constraints were met. Thanks to the performed investigation the team have selected a few libraries that suit their needs better than CORBA. Further prototyping will select the best candidate.

Slides FRBHMULT05 [8.508 MB]

FRBHMULT06

EPICS V4 Expands Support to Physics Application, Data Acsuisition, and Data Analysis

1338

L.R. Dalesio, G. Carcassi, M.A. Davidsaver, M.R. Kraimer, R. Lange, N. Malitsky, G. Shen
BNL, Upton, Long Island, New York, USA
T. Korhonen
Paul Scherrer Institut, 5232 Villigen PSI, Switzerland
J. Rowland
Diamond, Oxfordshire, United Kingdom
M. Sekoranja
Cosylab, Ljubljana, Slovenia
G.R. White
SLAC, Menlo Park, California, USA

Funding: Work supported under auspices of the U.S. Department of Energy under Contract No. DE-AC02-98CH10886 with Brookhaven Science Associates, LLC, and in part by the DOE Contract DE-AC02-76SF00515
EPICS version 4 extends the functionality of version 3 by providing the ability to define, transport, and introspect composite data types. Version 3 provided a set of process variables and a data protocol that adequately defined scalar data along with an atomic set of attributes. While remaining backward compatible, Version 4 is able to easily expand this set with a data protocol capable of exchanging complex data types and parameterized data requests. Additionally, a group of engineers defined reference types for some applications in this environment. The goal of this work is to define a narrow interface with the minimal set of data types needed to support a distributed architecture for physics applications, data acquisition, and data analysis.

Slides FRBHMULT06 [0.188 MB]

Paper	Title	Page
FRBHMUST01	The Design of the Alba Control System: A Cost-Effective Distributed Hardware and Software Architecture.	1318
	D.F.C. Fernández-Carreiras, D.B. Beltrán, T.M. Coutinho, G. Cuní, J. Klora, O. Matilla, R. Montaño, C. Pascual-Izarra, S. Pusó, R. Ranz, A. Rubio, S. Rubio-Manrique CELLS-ALBA Synchrotron, Cerdanyola del Vallès, Spain
	The control system of Alba is highly distributed from both hardware and software points of view. The hardware infrastructure for the control system includes in the order of 350 racks, 20000 cables and 6200 equipments. More than 150 diskless industrial computers, distributed in the service area and 30 multicore servers in the data center, manage several thousands of process variables. The software is, of course, as distributed as the hardware. It is also a success story of the Tango Collaboration where a complete software infrastructure is available "off the shelf". In addition Tango has been productively complemented with the powerful Sardana framework, a great effort in terms of development, which nowadays, several institutes benefit from. The whole installation has been coordinated from the beginning with a complete cabling and equipment database, where all the equipment, cables, connectors are described and inventoried. The so called "cabling database" is core of the installation. The equipments and cables are defined there. The basic configurations of the hardware like MAC and IP addresses, DNS names, etc. are also gathered in this database, allowing the network communication files and declaration of variables in the PLCs to be created automatically. This paper explains the design and the architecture of the control system, describes the tools and justifies the choices made. Furthermore, it presents and analyzes the figures regarding cost and performances.
	Slides FRBHMUST01 [4.616 MB]

FRBHMUST02	Towards High Performance Processing in Modern Java Based Control Systems	1322
	M. Misiowiec, W. Buczak, M. Buttner CERN, Geneva, Switzerland
	CERN controls software is often developed on Java foundation. Some systems carry out a combination of data, network and processor intensive tasks within strict time limits. Hence, there is a demand for high performing, quasi real time solutions. Extensive prototyping of the new CERN monitoring and alarm software required us to address such expectations. The system must handle dozens of thousands of data samples every second, along its three tiers, applying complex computations throughout. To accomplish the goal, a deep understanding of multithreading, memory management and interprocess communication was required. There are unexpected traps hidden behind an excessive use of 64 bit memory or severe impact on the processing flow of modern garbage collectors, including the state of the art Oracle GarbageFirst. Tuning JVM configuration significantly affects the execution of the code. Even more important is the amount of threads and the data structures used between them. Accurately dividing work into independent tasks might boost system performance. Thorough profiling with dedicated tools helped understand the bottlenecks and choose algorithmically optimal solutions. Different virtual machines were tested, in a variety of setups and garbage collection options. The overall work provided for discovering actual hard limits of the whole setup. We present this process of architecting a challenging system in view of the characteristics and limitations of the contemporary Java runtime environment. http://cern.ch/marekm/icalepcs.html
	Slides FRBHMUST02 [4.514 MB]

FRBHMUST03	Thirty Meter Telescope Observatory Software Architecture	1326
	K.K. Gillies, C. Boyer TMT, Pasadena, California, USA
	The Thirty Meter Telescope (TMT) will be a ground-based, 30-m optical-IR telescope with a highly segmented primary mirror located on the summit of Mauna Kea in Hawaii. The TMT Observatory Software (OSW) system will deliver the software applications and infrastructure necessary to integrate all TMT software into a single system and implement a minimal end-to-end science operations system. At the telescope, OSW is focused on the task of integrating and efficiently controlling and coordinating the telescope, adaptive optics, science instruments, and their subsystems during observation execution. From the software architecture viewpoint, the software system is viewed as a set of software components distributed across many machines that are integrated using a shared software base and a set of services that provide communications and other needed functionality. This paper describes the current state of the TMT Observatory Software focusing on its unique requirements, architecture, and the use of middleware technologies and solutions that enable the OSW design.
	Slides FRBHMUST03 [3.788 MB]

FRBHMULT04	Towards a State Based Control Architecture for Large Telescopes: Laying a Foundation at the VLT	1330
	R. Karban, N. Kornweibel ESO, Garching bei Muenchen, Germany D.L. Dvorak, M.D. Ingham, D.A. Wagner JPL, Pasadena, California, USA
	Large telescopes are characterized by a high level of distribution of control-related tasks and will feature diverse data flow patterns and large ranges of sampling frequencies; there will often be no single, fixed server-client relationship between the control tasks. The architecture is also challenged by the task of integrating heterogeneous subsystems which will be delivered by multiple different contractors. Due to the high number of distributed components, the control system needs to effectively detect errors and faults, impede their propagation, and accurately mitigate them in the shortest time possible, enabling the service to be restored. The presented Data-Driven Architecture is based on a decentralized approach with an end-to-end integration of disparate independently-developed software components, using a high-performance standards-based communication middle-ware infrastructure, based on the Data Distribution Service. A set of rules and principles, based on JPL's State Analysis method and architecture, are established to avoid undisciplined component-to-component interactions, where the Control System and System Under Control are clearly separated. State Analysis provides a model-based process for capturing system and software requirements and design, helping reduce the gap between the requirements on software specified by systems engineers and the implementation by software engineers. The method and architecture has been field tested at the Very Large Telescope, where it has been integrated into an operational system with minimal downtime.
	Slides FRBHMULT04 [3.504 MB]

FRBHMULT05	Middleware Trends and Market Leaders 2011	1334
	A. Dworak, P. Charrue, F. Ehm, W. Sliwinski, M. Sobczak CERN, Geneva, Switzerland
	The Controls Middleware (CMW) project was launched over ten years ago. Its main goal was to unify middleware solutions used to operate CERN accelerators. An important part of the project, the equipment access library RDA, was based on CORBA, an unquestionable standard at the time. RDA became an operational and critical part of the infrastructure, yet the demanding run-time environment revealed some shortcomings of the system. Accumulation of fixes and workarounds led to unnecessary complexity. RDA became difficult to maintain and to extend. CORBA proved to be rather a cumbersome product than a panacea. Fortunately, many new transport frameworks appeared since then. They boasted a better design, and supported concepts that made them easy to use. Willing to profit from the new libraries, the CMW team updated user requirements, and in their terms investigated eventual CORBA substitutes. The process consisted of several phases: a review of middleware solutions belonging to different categories (e.g. data-centric, object-, and message-oriented) and their applicability to a communication model in RDA; evaluation of several market recognized products and promising start-ups; prototyping of typical communication scenarios; testing the libraries against exceptional situations and errors; verifying that mandatory performance constraints were met. Thanks to the performed investigation the team have selected a few libraries that suit their needs better than CORBA. Further prototyping will select the best candidate.
	Slides FRBHMULT05 [8.508 MB]

FRBHMULT06	EPICS V4 Expands Support to Physics Application, Data Acsuisition, and Data Analysis	1338
	L.R. Dalesio, G. Carcassi, M.A. Davidsaver, M.R. Kraimer, R. Lange, N. Malitsky, G. Shen BNL, Upton, Long Island, New York, USA T. Korhonen Paul Scherrer Institut, 5232 Villigen PSI, Switzerland J. Rowland Diamond, Oxfordshire, United Kingdom M. Sekoranja Cosylab, Ljubljana, Slovenia G.R. White SLAC, Menlo Park, California, USA
	Funding: Work supported under auspices of the U.S. Department of Energy under Contract No. DE-AC02-98CH10886 with Brookhaven Science Associates, LLC, and in part by the DOE Contract DE-AC02-76SF00515 EPICS version 4 extends the functionality of version 3 by providing the ability to define, transport, and introspect composite data types. Version 3 provided a set of process variables and a data protocol that adequately defined scalar data along with an atomic set of attributes. While remaining backward compatible, Version 4 is able to easily expand this set with a data protocol capable of exchanging complex data types and parameterized data requests. Additionally, a group of engineers defined reference types for some applications in this environment. The goal of this work is to define a narrow interface with the minimal set of data types needed to support a distributed architecture for physics applications, data acquisition, and data analysis.
	Slides FRBHMULT06 [0.188 MB]