Author: Gaspar, C.
Paper Title Page
MOBAUST06 The LHCb Experiment Control System: on the Path to Full Automation 20
 
  • C. Gaspar, F. Alessio, L.G. Cardoso, M. Frank, J.C. Garnier, R. Jacobsson, B. Jost, N. Neufeld, R. Schwemmer, E. van Herwijnen
    CERN, Geneva, Switzerland
  • O. Callot
    LAL, Orsay, France
  • B. Franek
    STFC/RAL, Chilton, Didcot, Oxon, United Kingdom
  
  LHCb is a large ex­per­i­ment at the LHC ac­cel­er­a­tor. The ex­per­i­ment con­trol sys­tem is in charge of the con­fig­u­ra­tion, con­trol and mon­i­tor­ing of the dif­fer­ent sub-de­tec­tors and of all areas of the on­line sys­tem: the De­tec­tor Con­trol Sys­tem (DCS), sub-de­tec­tor's volt­ages, cool­ing, tem­per­a­tures, etc.; the Data Ac­qui­si­tion Sys­tem (DAQ), and the Run-Con­trol; the High Level Trig­ger (HLT), a farm of around 1500 PCs run­ning trig­ger al­go­rithms; etc. The build­ing blocks of the con­trol sys­tem are based on the PVSS SCADA Sys­tem com­ple­ment­ed by a con­trol Frame­work de­vel­oped in com­mon for the 4 LHC ex­per­i­ments. This frame­work in­cludes an "ex­pert sys­tem" like tool called SMI++ which we use for the sys­tem au­toma­tion. The full con­trol sys­tem runs dis­tribut­ed over around 160 PCs and is log­i­cal­ly or­gan­ised in a hi­er­ar­chi­cal struc­ture, each level being ca­pa­ble of su­per­vis­ing and syn­chro­niz­ing the ob­jects below. The ex­per­i­ment's op­er­a­tions are now al­most com­plete­ly au­to­mat­ed driv­en by a top-lev­el ob­ject called Big-Broth­er which pi­lots all the ex­per­i­ment's stan­dard pro­ce­dures and the most com­mon er­ror-re­cov­ery pro­ce­dures. Some ex­am­ples of au­to­mat­ed pro­ce­dures are: pow­er­ing the de­tec­tor, act­ing on the Run-Con­trol (Start/Stop Run, etc.) and mov­ing the ver­tex de­tec­tor in/out of the beam, all driv­en by the state of the ac­cel­er­a­tor or re­cov­er­ing from er­rors in the HLT farm. The ar­chi­tec­ture, tools and mech­a­nisms used for the im­ple­men­ta­tion as well as some op­er­a­tional ex­am­ples will be shown.  
slides icon Slides MOBAUST06 [1.451 MB]  
 
MOPMN019 Controling and Monitoring the Data Flow of the LHCb Read-out and DAQ Network 281
 
  • R. Schwemmer, C. Gaspar, N. Neufeld, D. Svantesson
    CERN, Geneva, Switzerland
  
  The LHCb read­out uses a set of 320 FPGA based boards as in­ter­face be­tween the on-de­tec­tor hard­ware and the GBE DAQ net­work. The boards are the log­i­cal Level 1 (L1) read-out elec­tron­ics and ag­gre­gate the ex­per­i­ment's raw data into event frag­ments that are sent to the DAQ net­work. To con­trol the many pa­ram­e­ters of the read-out boards, an em­bed­ded PC is in­clud­ed on each board, con­nect­ing to the boards ICs and FPGAs. The data from the L1 boards is sent through an ag­gre­ga­tion net­work into the High Level Trig­ger farm. The farm com­pris­es ap­prox­i­mate­ly 1500 PCs which at first as­sem­ble the frag­ments from the L1 boards and then do a par­tial re­con­struc­tion and se­lec­tion of the events. In total there are ap­prox­i­mate­ly 3500 net­work con­nec­tions. Data is pushed through the net­work and there is no mech­a­nism for re­send­ing pack­ets. Loss of data on a small scale is ac­cept­able but care has to be taken to avoid data loss if pos­si­ble. To mon­i­tor and debug loss­es, dif­fer­ent probes are in­sert­ed through­out the en­tire read-out chain to count frag­ments, pack­ets and their rates at dif­fer­ent po­si­tions. To keep uni­for­mi­ty through­out the ex­per­i­ment, all con­trol soft­ware was de­vel­oped using the com­mon SCADA soft­ware, PVSS, with the JCOP frame­work as base. The pre­sen­ta­tion will focus on the low level con­trols in­ter­face de­vel­oped for the L1 boards and the net­work­ing probes, as well as the in­te­gra­tion of the high level user in­ter­faces into PVSS. We will show the way in which the users and de­vel­op­ers in­ter­act with the soft­ware, con­fig­ure the hard­ware and fol­low the flow of data through the DAQ net­work.  
 
MOPMN028 Automated Voltage Control in LHCb 304
 
  • L.G. Cardoso, C. Gaspar, R. Jacobsson
    CERN, Geneva, Switzerland
  
  LHCb is one of the 4 LHC ex­per­i­ments. In order to en­sure the safe­ty of the de­tec­tor and to max­i­mize ef­fi­cien­cy, LHCb needs to co­or­di­nate its own op­er­a­tions, in par­tic­u­lar the volt­age con­fig­u­ra­tion of the dif­fer­ent sub-de­tec­tors, ac­cord­ing to the ac­cel­er­a­tor sta­tus. A con­trol soft­ware has been de­vel­oped for this pur­pose, based on the Fi­nite State Ma­chine toolk­it and the SCADA sys­tem used for con­trol through­out LHCb (and the other LHC ex­per­i­ments). This soft­ware per­mits to ef­fi­cient­ly drive both the Low Volt­age (LV) and High Volt­age (HV) sys­tems of the 10 dif­fer­ent sub-de­tec­tors that con­sti­tute LHCb, set­ting each sub-sys­tem to the re­quired volt­age (eas­i­ly con­fig­urable at run-time) based on the ac­cel­er­a­tor state. The con­trol soft­ware is also re­spon­si­ble for mon­i­tor­ing the state of the Sub-de­tec­tor volt­ages and adding it to the event data in the form of sta­tus-bits. Safe and yet flex­i­ble op­er­a­tion of the LHCb de­tec­tor has been ob­tained and au­to­mat­ic ac­tions, trig­gered by the state changes of the ac­cel­er­a­tor, have been im­ple­ment­ed. This paper will de­tail the im­ple­men­ta­tion of the volt­age con­trol soft­ware, its flex­i­ble run-time con­fig­u­ra­tion and its usage in the LHCb ex­per­i­ment.  
poster icon Poster MOPMN028 [0.479 MB]  
 
WEBHAUST01 LHCb Online Infrastructure Monitoring Tools 618
 
  • L.G. Cardoso, C. Gaspar, C. Haen, N. Neufeld, F. Varela
    CERN, Geneva, Switzerland
  • D. Galli
    INFN-Bologna, Bologna, Italy
  
  The On­line Sys­tem of the LHCb ex­per­i­ment at CERN is com­posed of a very large num­ber of PCs: around 1500 in a CPU farm for per­form­ing the High Level Trig­ger; around 170 for the con­trol sys­tem, run­ning the SCADA sys­tem - PVSS; and sev­er­al oth­ers for per­form­ing data mon­i­tor­ing, re­con­struc­tion, stor­age, and in­fras­truc­ture tasks, like databas­es, etc. Some PCs run Linux, some run Win­dows but all of them need to be re­mote­ly con­trolled and mon­i­tored to make sure they are cor­rect­ly run­ning and to be able, for ex­am­ple, to re­boot them when­ev­er nec­es­sary. A set of tools was de­vel­oped in order to cen­tral­ly mon­i­tor the sta­tus of all PCs and PVSS Pro­jects need­ed to run the ex­per­i­ment: a Farm Mon­i­tor­ing and Con­trol (FMC) tool, which pro­vides the lower level ac­cess to the PCs, and a Sys­tem Overview Tool (de­vel­oped with­in the Joint Con­trols Pro­ject – JCOP), which pro­vides a cen­tral­ized in­ter­face to the FMC tool and adds PVSS pro­ject mon­i­tor­ing and con­trol. The im­ple­men­ta­tion of these tools has pro­vid­ed a re­li­able and ef­fi­cient way to man­age the sys­tem, both dur­ing nor­mal op­er­a­tions but also dur­ing shut­downs, up­grades or main­te­nance op­er­a­tions. This paper will pre­sent the par­tic­u­lar im­ple­men­ta­tion of this tool in the LHCb ex­per­i­ment and the ben­e­fits of its usage in a large scale het­ero­ge­neous sys­tem.  
slides icon Slides WEBHAUST01 [3.211 MB]