SKA South Africa, National Research Foundation of South Africa, Cape Town, South Africa
Data indexed by time is continuously collected from instruments, environment and users. Samples are recorded from sensors or software components at specific times, starting as simple numbers and increasing in complexity as associated values accrue e.g. status and acquisition times. A sample is more than a triple and evolves into a document. Besides variance, volume and veracity also increase and the time series database (TSDB) has to process hundreds of GB/day. Also, users performing analyses have ever increasing demands e.g. in <10s plot all target coordinates over 24h of 64 radio telescope dishes, recorded at 1Hz. Besides the many short-term queries, trend analyses over long periods and in-depth enquiries by specialists around past events e.g. critical hardware failure or scientific discovery, are performed. This paper discusses the solution used for the MeerKAT radio telescope under construction by SKA-SA in South Africa. System architecture and performance characteristics of the developed TSDB are explained. We demonstrate how we broke the mould of using general-purpose database technologies to build a TSDB by rather utilising technologies employed in distributed file storage.
SKA South Africa, National Research Foundation of South Africa, Cape Town, South Africa
The MeerKAT radio telescope control system generates monitoring and scheduling data that internal and external systems require to operate. Distributing this data in near real-time, requires a scalable messaging strategy to ensure optimal performance regardless of the number of systems connected. Internal systems include the MeerKAT Graphical User Interfaces, the MeerKAT Science Data Processing subsystem and the MeerKAT Correlator Beamformer subsystem. External systems include Pulsar Timing User Supplied Equipment, MeerLICHT and the Search for Extraterrestrial Intelligence (SETI). Many more external systems are expected to join MeerKAT in the future. This paper describes the strategy adopted by the Control and Monitoring team to distribute near real-time monitoring and scheduling data at scale. This strategy is implemented using standard web technologies and the publish/subscribe design pattern.
