PCaPAC2014 - List of Authors (Kopmann, A.)

Paper

Title

Page

Computing Infrastructure for Online Monitoring and Control of High-throughput DAQ Electronics

10

S.A. Chilingaryan, C.M. Caselle, T. Dritschler, T. Faragó, A. Kopmann, U. Stevanovic, M. Vogelgesang
KIT, Eggenstein-Leopoldshafen, Germany

New imaging stations with high-resolution pixel detectors and other synchrotron instrumentation have ever increasing sampling rates and put strong demands on the complete signal processing chain. Key to successful systems is high-throughput computing platform consisting of DAQ electronics, PC hardware components, communication layer and system and data processing software components. Based on our experience building a high-throughput platform for real-time control of X-ray imaging experiments, we have designed a generalized architecture enabling rapid deployment of data acquisition system. We have evaluated various technologies and come up with solution which can be easily scaled up to several gigabytes-per-second of aggregated bandwidth while utilizing reasonably priced mass-market products. The core components of our system are an FPGA platform for ultra-fast data acquisition, Infiniband interconnects and GPU computing units. The presentation will give an overview on the hardware, interconnects, and the system level software serving as foundation for this high-throughput DAQ platform. This infrastructure is already successfully used at KIT's synchrotron ANKA.

Slides WCO201 [2.948 MB]

WCO202

Data Management at the Synchrotron Radiation Facility ANKA

13

D. Ressmann, A. Kopmann, V. Mauch, W. Mexner, A. Vondrous
KIT, Eggenstein-Leopoldshafen, Germany

The complete chain from submitting a proposal, collecting meta data, performing an experiment, towards analysis of these data and finally long term archive will be described. During this process a few obstacles have to be tackled. The workflow should be transparent to the user as well as to the beamline scientists. The final data will be stored in NeXus compatible HDF5 container format. Because the transfer of one large file is more efficient than transferring many small files, container formats enable a faster transfer of experiment data. At the same time HDF5 supports to store meta data together with the experiment data. For large data sets another implication is the performance to download the files. Furthermore the analysis software might not be available at each home institution; as a result it should be an option to access the experiment data on site. The meta data allows to find, analyse, preserve and curate the data in a long term archive, which will become a requirement fairly soon.

Slides WCO202 [2.380 MB]

FPO001

InfiniBand interconnects for high-throughput data acquisition in a TANGO environment

164

FPI01

use link to see paper's listing under its alternate paper code

T. Dritschler, S.A. Chilingaryan, T. Faragó, A. Kopmann, M. Vogelgesang
KIT, Eggenstein-Leopoldshafen, Germany

Advances in computational performance allow for fast image-based control. To realize efficient control loops in a distributed experiment setup, large amounts of data need to be transferred, requiring high-throughput networks with low latencies. In the European synchrotron community, TANGO has become one of the prevalent tools to remotely control hardware and processes. In order to improve the data bandwidth and latency in a TANGO network, we realized a secondary data channel based on native InfiniBand communication. This data channel is implemented as part of a TANGO device and by itself is independent of the main TANGO network communication. TANGO mechanisms are used for configuration, thus the data channel can be used by any TANGO-based software that implements the corresponding interfaces. First results show that we can achieve a maximum bandwidth of 30 Gb/s which is close to the theoretical maximum of 32 Gb/s, possible with our 4xQDR InfiniBand test network, with average latencies as low as 6 μs. This means that we are able to surpass the limitations of standard TCP/IP networks while retaining the TANGO control schemes, enabling high data throughput in a TANGO environment.

Slides FPO001 [0.511 MB]

Poster FPO001 [3.767 MB]

FPO002

Picosecond Sampling Electronics for Terahertz Synchrotron Radiation

167

FPI02

use link to see paper's listing under its alternate paper code

C.M. Caselle, B.M. Balzer, M. Brosi, S.A. Chilingaryan, T. Dritschler, V. Judin, A. Kopmann, A.-S. Müller, L. Petzold, J. Raasch, L. Rota, M. Siegel, N.J. Smale, J.L. Steinmann, M. Vogelgesang, M. Weber, S. Wuensch
KIT, Karlsruhe, Germany

To study the synchrotron terahertz emission superconducting (YBCO) film detectors are used with the intrinsic response time in the order of a few picoseconds. For fast, continuous sampling of the individual THz ultra-short pulses a novel digitizer system has been developed. The system consists of detector, wideband low-noise amplifier, fast pulse digitizer board, back-end readout board. High-end graphic processing units (GPUs) perform real-time data analysis. Four samples with 12 bit are recorded in parallel for each fast pulse with programmable sampling times in the range of 3 to 100 ps. A new bus master DMA engine connected to PCI express endpoint has been developed to ensure a continuous high data throughput of up to 4 GByte/s. This heterogeneous real-time system architecture based on FPGA and GPU has successful been used for on-line pulse reconstruction and evaluations and calculates the peak amplitude of each pulse and the time between consecutive bunches with a picosecond time resolution at ANKA. A Fast Fourier Transform (FFT) is performed on-line for the frequency analysis of the CSR undulations.

Slides FPO002 [1.153 MB]

FPO026

ADEI and Tango Archiving System – A Convenient Way to Archive and Represent Data

213

D. Haas, S.A. Chilingaryan, A. Kopmann, W. Mexner, D. Ressmann
KIT, Eggenstein-Leopoldshafen, Germany

Tango offers an efficient and powerful archiving mechanism of Tango attributes in a MySQL database. The tool Mambo allows an easy configuration of all to be archived data. This approved archiving concept was successfully introduced to ANKA (Angströmquelle Karlsruhe). To provide an efficient and intuitive web-based interface instead of complex database queries, the TANGO Archiving System was integrated into the “Advanced Data Extraction Infrastructure ADEI”. ADEI is intended to manage data of distributed heterogeneous devices in large-scale physics experiments. ADEI contains internal pre-processing, data quality checks and an intuitive web interface, that guarantees fast access and visualization of huge a data sets stored in the attached data sources like MySQL databases or data files. ADEI and the Tango archiving system have been successfully tested at ANKA's imaging beamlines. It is intended to deploy both at all ANKA beamlines.

Poster FPO026 [0.938 MB]

FCO202

OpenGL-Based Data Analysis in Virtualized Self-Service Environments

237

V. Mauch, M. Bonn, S.A. Chilingaryan, A. Kopmann, W. Mexner, D. Ressmann
KIT, Eggenstein-Leopoldshafen, Germany

Funding: Federal Ministry of Education and Research, Germany
Modern data analysis applications for 2D/3D data samples apply complex visual output features which are often based on OpenGL, a multi-platform API for rendering vector graphics. They demand special computing workstations with a corresponding CPU/GPU power, enough main memory and fast network interconnects for a performant remote data access. For this reason, users depend heavily on available free workstations, both temporally and locally. The provision of virtual machines (VMs) accessible via a remote connection could avoid this inflexibility. However, the automatic deployment, operation and remote access of OpenGL-capable VMs with professional visualization applications is a non-trivial task. In this paper, we discuss a concept for a flexible analysis infrastructure that will be part in the project ASTOR, which is the abbreviation for “Arthropod Structure revealed by ultra-fast Tomography and Online Reconstruction”. We present an Analysis-as-a-Service (AaaS) approach based on the on-demand allocation of VMs with dedicated GPU cores and a corresponding analysis environment to provide a cloud-like analysis service for scientific users.

Slides FCO202 [1.126 MB]

Paper	Title	Page
WCO201	Computing Infrastructure for Online Monitoring and Control of High-throughput DAQ Electronics	10
	S.A. Chilingaryan, C.M. Caselle, T. Dritschler, T. Faragó, A. Kopmann, U. Stevanovic, M. Vogelgesang KIT, Eggenstein-Leopoldshafen, Germany
	New imaging stations with high-resolution pixel detectors and other synchrotron instrumentation have ever increasing sampling rates and put strong demands on the complete signal processing chain. Key to successful systems is high-throughput computing platform consisting of DAQ electronics, PC hardware components, communication layer and system and data processing software components. Based on our experience building a high-throughput platform for real-time control of X-ray imaging experiments, we have designed a generalized architecture enabling rapid deployment of data acquisition system. We have evaluated various technologies and come up with solution which can be easily scaled up to several gigabytes-per-second of aggregated bandwidth while utilizing reasonably priced mass-market products. The core components of our system are an FPGA platform for ultra-fast data acquisition, Infiniband interconnects and GPU computing units. The presentation will give an overview on the hardware, interconnects, and the system level software serving as foundation for this high-throughput DAQ platform. This infrastructure is already successfully used at KIT's synchrotron ANKA.
	Slides WCO201 [2.948 MB]

WCO202	Data Management at the Synchrotron Radiation Facility ANKA	13
	D. Ressmann, A. Kopmann, V. Mauch, W. Mexner, A. Vondrous KIT, Eggenstein-Leopoldshafen, Germany
	The complete chain from submitting a proposal, collecting meta data, performing an experiment, towards analysis of these data and finally long term archive will be described. During this process a few obstacles have to be tackled. The workflow should be transparent to the user as well as to the beamline scientists. The final data will be stored in NeXus compatible HDF5 container format. Because the transfer of one large file is more efficient than transferring many small files, container formats enable a faster transfer of experiment data. At the same time HDF5 supports to store meta data together with the experiment data. For large data sets another implication is the performance to download the files. Furthermore the analysis software might not be available at each home institution; as a result it should be an option to access the experiment data on site. The meta data allows to find, analyse, preserve and curate the data in a long term archive, which will become a requirement fairly soon.
	Slides WCO202 [2.380 MB]

FPO001	InfiniBand interconnects for high-throughput data acquisition in a TANGO environment	164
FPI01	use link to see paper's listing under its alternate paper code
	T. Dritschler, S.A. Chilingaryan, T. Faragó, A. Kopmann, M. Vogelgesang KIT, Eggenstein-Leopoldshafen, Germany
	Advances in computational performance allow for fast image-based control. To realize efficient control loops in a distributed experiment setup, large amounts of data need to be transferred, requiring high-throughput networks with low latencies. In the European synchrotron community, TANGO has become one of the prevalent tools to remotely control hardware and processes. In order to improve the data bandwidth and latency in a TANGO network, we realized a secondary data channel based on native InfiniBand communication. This data channel is implemented as part of a TANGO device and by itself is independent of the main TANGO network communication. TANGO mechanisms are used for configuration, thus the data channel can be used by any TANGO-based software that implements the corresponding interfaces. First results show that we can achieve a maximum bandwidth of 30 Gb/s which is close to the theoretical maximum of 32 Gb/s, possible with our 4xQDR InfiniBand test network, with average latencies as low as 6 μs. This means that we are able to surpass the limitations of standard TCP/IP networks while retaining the TANGO control schemes, enabling high data throughput in a TANGO environment.
	Slides FPO001 [0.511 MB]
	Poster FPO001 [3.767 MB]

FPO002	Picosecond Sampling Electronics for Terahertz Synchrotron Radiation	167
FPI02	use link to see paper's listing under its alternate paper code
	C.M. Caselle, B.M. Balzer, M. Brosi, S.A. Chilingaryan, T. Dritschler, V. Judin, A. Kopmann, A.-S. Müller, L. Petzold, J. Raasch, L. Rota, M. Siegel, N.J. Smale, J.L. Steinmann, M. Vogelgesang, M. Weber, S. Wuensch KIT, Karlsruhe, Germany
	To study the synchrotron terahertz emission superconducting (YBCO) film detectors are used with the intrinsic response time in the order of a few picoseconds. For fast, continuous sampling of the individual THz ultra-short pulses a novel digitizer system has been developed. The system consists of detector, wideband low-noise amplifier, fast pulse digitizer board, back-end readout board. High-end graphic processing units (GPUs) perform real-time data analysis. Four samples with 12 bit are recorded in parallel for each fast pulse with programmable sampling times in the range of 3 to 100 ps. A new bus master DMA engine connected to PCI express endpoint has been developed to ensure a continuous high data throughput of up to 4 GByte/s. This heterogeneous real-time system architecture based on FPGA and GPU has successful been used for on-line pulse reconstruction and evaluations and calculates the peak amplitude of each pulse and the time between consecutive bunches with a picosecond time resolution at ANKA. A Fast Fourier Transform (FFT) is performed on-line for the frequency analysis of the CSR undulations.
	Slides FPO002 [1.153 MB]

FPO026	ADEI and Tango Archiving System – A Convenient Way to Archive and Represent Data	213
	D. Haas, S.A. Chilingaryan, A. Kopmann, W. Mexner, D. Ressmann KIT, Eggenstein-Leopoldshafen, Germany
	Tango offers an efficient and powerful archiving mechanism of Tango attributes in a MySQL database. The tool Mambo allows an easy configuration of all to be archived data. This approved archiving concept was successfully introduced to ANKA (Angströmquelle Karlsruhe). To provide an efficient and intuitive web-based interface instead of complex database queries, the TANGO Archiving System was integrated into the “Advanced Data Extraction Infrastructure ADEI”. ADEI is intended to manage data of distributed heterogeneous devices in large-scale physics experiments. ADEI contains internal pre-processing, data quality checks and an intuitive web interface, that guarantees fast access and visualization of huge a data sets stored in the attached data sources like MySQL databases or data files. ADEI and the Tango archiving system have been successfully tested at ANKA's imaging beamlines. It is intended to deploy both at all ANKA beamlines.
	Poster FPO026 [0.938 MB]

FCO202	OpenGL-Based Data Analysis in Virtualized Self-Service Environments	237
	V. Mauch, M. Bonn, S.A. Chilingaryan, A. Kopmann, W. Mexner, D. Ressmann KIT, Eggenstein-Leopoldshafen, Germany
	Funding: Federal Ministry of Education and Research, Germany Modern data analysis applications for 2D/3D data samples apply complex visual output features which are often based on OpenGL, a multi-platform API for rendering vector graphics. They demand special computing workstations with a corresponding CPU/GPU power, enough main memory and fast network interconnects for a performant remote data access. For this reason, users depend heavily on available free workstations, both temporally and locally. The provision of virtual machines (VMs) accessible via a remote connection could avoid this inflexibility. However, the automatic deployment, operation and remote access of OpenGL-capable VMs with professional visualization applications is a non-trivial task. In this paper, we discuss a concept for a flexible analysis infrastructure that will be part in the project ASTOR, which is the abbreviation for “Arthropod Structure revealed by ultra-fast Tomography and Online Reconstruction”. We present an Analysis-as-a-Service (AaaS) approach based on the on-demand allocation of VMs with dedicated GPU cores and a corresponding analysis environment to provide a cloud-like analysis service for scientific users.
	Slides FCO202 [1.126 MB]