The Joint Accelerator Conferences Website (JACoW) is an international collaboration that publishes the proceedings of accelerator conferences held around the world.
TY - CONF AU - Joubert, A.F. AU - Merry, B. ED - Schaa, Volker RW ED - Costa, Isidre ED - Fernández, David ED - Matilla, Ãscar TI - Orchestrating MeerKAT's Distributed Science Data Processing Pipelines J2 - Proc. of ICALEPCS2017, Barcelona, Spain, 8-13 October 2017 C1 - Barcelona, Spain T2 - International Conference on Accelerator and Large Experimental Control Systems T3 - 16 LA - english AB - The 64-antenna MeerKAT radio telescope is a precursor to the Square Kilometre Array. The telescope's correlator beamformer streams data at 600 Gb/s to the science data processing pipeline that must consume it in real time. This requires significant compute resources, which are provided by a cluster of heterogeneous hardware nodes. Effective utilisation of the available resources is a critical design goal, made more challenging by requiring multiple, highly configurable pipelines. We initially used a static allocation of processes to hardware nodes, but this approach is insufficient as the project scales up. We describe recent improvements to our distributed container deployment, using Apache Mesos for orchestration. We also discuss how issues like non-uniform memory access (NUMA), network partitions, and fractional allocation of graphical processing units (GPUs) are addressed using a custom scheduler for Mesos. PB - JACoW CP - Geneva, Switzerland SP - 1152 EP - 1157 KW - ion KW - controls KW - framework KW - GPU KW - network DA - 2018/01 PY - 2018 SN - 978-3-95450-193-9 DO - 10.18429/JACoW-ICALEPCS2017-THBPA04 UR - http://jacow.org/icalepcs2017/papers/thbpa04.pdf ER -