PCaPAC2014 - List of Authors (Vogelgesang, M.)

Paper

Title

Page

Computing Infrastructure for Online Monitoring and Control of High-throughput DAQ Electronics

10

S.A. Chilingaryan, C.M. Caselle, T. Dritschler, T. Faragó, A. Kopmann, U. Stevanovic, M. Vogelgesang
KIT, Eggenstein-Leopoldshafen, Germany

New imaging stations with high-resolution pixel detectors and other synchrotron instrumentation have ever increasing sampling rates and put strong demands on the complete signal processing chain. Key to successful systems is high-throughput computing platform consisting of DAQ electronics, PC hardware components, communication layer and system and data processing software components. Based on our experience building a high-throughput platform for real-time control of X-ray imaging experiments, we have designed a generalized architecture enabling rapid deployment of data acquisition system. We have evaluated various technologies and come up with solution which can be easily scaled up to several gigabytes-per-second of aggregated bandwidth while utilizing reasonably priced mass-market products. The core components of our system are an FPGA platform for ultra-fast data acquisition, Infiniband interconnects and GPU computing units. The presentation will give an overview on the hardware, interconnects, and the system level software serving as foundation for this high-throughput DAQ platform. This infrastructure is already successfully used at KIT's synchrotron ANKA.

Slides WCO201 [2.948 MB]

FPO001

InfiniBand interconnects for high-throughput data acquisition in a TANGO environment

164

FPI01

use link to see paper's listing under its alternate paper code

T. Dritschler, S.A. Chilingaryan, T. Faragó, A. Kopmann, M. Vogelgesang
KIT, Eggenstein-Leopoldshafen, Germany

Advances in computational performance allow for fast image-based control. To realize efficient control loops in a distributed experiment setup, large amounts of data need to be transferred, requiring high-throughput networks with low latencies. In the European synchrotron community, TANGO has become one of the prevalent tools to remotely control hardware and processes. In order to improve the data bandwidth and latency in a TANGO network, we realized a secondary data channel based on native InfiniBand communication. This data channel is implemented as part of a TANGO device and by itself is independent of the main TANGO network communication. TANGO mechanisms are used for configuration, thus the data channel can be used by any TANGO-based software that implements the corresponding interfaces. First results show that we can achieve a maximum bandwidth of 30 Gb/s which is close to the theoretical maximum of 32 Gb/s, possible with our 4xQDR InfiniBand test network, with average latencies as low as 6 μs. This means that we are able to surpass the limitations of standard TCP/IP networks while retaining the TANGO control schemes, enabling high data throughput in a TANGO environment.

Slides FPO001 [0.511 MB]

Poster FPO001 [3.767 MB]

FPO002

Picosecond Sampling Electronics for Terahertz Synchrotron Radiation

167

FPI02

use link to see paper's listing under its alternate paper code

C.M. Caselle, B.M. Balzer, M. Brosi, S.A. Chilingaryan, T. Dritschler, V. Judin, A. Kopmann, A.-S. Müller, L. Petzold, J. Raasch, L. Rota, M. Siegel, N.J. Smale, J.L. Steinmann, M. Vogelgesang, M. Weber, S. Wuensch
KIT, Karlsruhe, Germany

To study the synchrotron terahertz emission superconducting (YBCO) film detectors are used with the intrinsic response time in the order of a few picoseconds. For fast, continuous sampling of the individual THz ultra-short pulses a novel digitizer system has been developed. The system consists of detector, wideband low-noise amplifier, fast pulse digitizer board, back-end readout board. High-end graphic processing units (GPUs) perform real-time data analysis. Four samples with 12 bit are recorded in parallel for each fast pulse with programmable sampling times in the range of 3 to 100 ps. A new bus master DMA engine connected to PCI express endpoint has been developed to ensure a continuous high data throughput of up to 4 GByte/s. This heterogeneous real-time system architecture based on FPGA and GPU has successful been used for on-line pulse reconstruction and evaluations and calculates the peak amplitude of each pulse and the time between consecutive bunches with a picosecond time resolution at ANKA. A Fast Fourier Transform (FFT) is performed on-line for the frequency analysis of the CSR undulations.

Slides FPO002 [1.153 MB]

Paper	Title	Page
WCO201	Computing Infrastructure for Online Monitoring and Control of High-throughput DAQ Electronics	10
	S.A. Chilingaryan, C.M. Caselle, T. Dritschler, T. Faragó, A. Kopmann, U. Stevanovic, M. Vogelgesang KIT, Eggenstein-Leopoldshafen, Germany
	New imaging stations with high-resolution pixel detectors and other synchrotron instrumentation have ever increasing sampling rates and put strong demands on the complete signal processing chain. Key to successful systems is high-throughput computing platform consisting of DAQ electronics, PC hardware components, communication layer and system and data processing software components. Based on our experience building a high-throughput platform for real-time control of X-ray imaging experiments, we have designed a generalized architecture enabling rapid deployment of data acquisition system. We have evaluated various technologies and come up with solution which can be easily scaled up to several gigabytes-per-second of aggregated bandwidth while utilizing reasonably priced mass-market products. The core components of our system are an FPGA platform for ultra-fast data acquisition, Infiniband interconnects and GPU computing units. The presentation will give an overview on the hardware, interconnects, and the system level software serving as foundation for this high-throughput DAQ platform. This infrastructure is already successfully used at KIT's synchrotron ANKA.
	Slides WCO201 [2.948 MB]

FPO001	InfiniBand interconnects for high-throughput data acquisition in a TANGO environment	164
FPI01	use link to see paper's listing under its alternate paper code
	T. Dritschler, S.A. Chilingaryan, T. Faragó, A. Kopmann, M. Vogelgesang KIT, Eggenstein-Leopoldshafen, Germany
	Advances in computational performance allow for fast image-based control. To realize efficient control loops in a distributed experiment setup, large amounts of data need to be transferred, requiring high-throughput networks with low latencies. In the European synchrotron community, TANGO has become one of the prevalent tools to remotely control hardware and processes. In order to improve the data bandwidth and latency in a TANGO network, we realized a secondary data channel based on native InfiniBand communication. This data channel is implemented as part of a TANGO device and by itself is independent of the main TANGO network communication. TANGO mechanisms are used for configuration, thus the data channel can be used by any TANGO-based software that implements the corresponding interfaces. First results show that we can achieve a maximum bandwidth of 30 Gb/s which is close to the theoretical maximum of 32 Gb/s, possible with our 4xQDR InfiniBand test network, with average latencies as low as 6 μs. This means that we are able to surpass the limitations of standard TCP/IP networks while retaining the TANGO control schemes, enabling high data throughput in a TANGO environment.
	Slides FPO001 [0.511 MB]
	Poster FPO001 [3.767 MB]

FPO002	Picosecond Sampling Electronics for Terahertz Synchrotron Radiation	167
FPI02	use link to see paper's listing under its alternate paper code
	C.M. Caselle, B.M. Balzer, M. Brosi, S.A. Chilingaryan, T. Dritschler, V. Judin, A. Kopmann, A.-S. Müller, L. Petzold, J. Raasch, L. Rota, M. Siegel, N.J. Smale, J.L. Steinmann, M. Vogelgesang, M. Weber, S. Wuensch KIT, Karlsruhe, Germany
	To study the synchrotron terahertz emission superconducting (YBCO) film detectors are used with the intrinsic response time in the order of a few picoseconds. For fast, continuous sampling of the individual THz ultra-short pulses a novel digitizer system has been developed. The system consists of detector, wideband low-noise amplifier, fast pulse digitizer board, back-end readout board. High-end graphic processing units (GPUs) perform real-time data analysis. Four samples with 12 bit are recorded in parallel for each fast pulse with programmable sampling times in the range of 3 to 100 ps. A new bus master DMA engine connected to PCI express endpoint has been developed to ensure a continuous high data throughput of up to 4 GByte/s. This heterogeneous real-time system architecture based on FPGA and GPU has successful been used for on-line pulse reconstruction and evaluations and calculates the peak amplitude of each pulse and the time between consecutive bunches with a picosecond time resolution at ANKA. A Fast Fourier Transform (FFT) is performed on-line for the frequency analysis of the CSR undulations.
	Slides FPO002 [1.153 MB]