Please use this identifier to cite or link to this item: http://earchive.tpu.ru/handle/11683/52921
Title: Building analytical platform with Big Data solutions for log files of PanDA infrastructure
Authors: Alekseev, Aleksandr Aleksandrovich
Barreiro Megino, F. G.
Klimentov, A. A.
Korchuganova, T. A.
Maendo, T.
Padolski, S. V.
Keywords: платформы; Big Data; лог-файлы; инфраструктура; высокопроизводительные системы; программное обеспечение; данные; обработка; хранение
Issue Date: 2018
Publisher: IOP Publishing
Citation: Building analytical platform with Big Data solutions for log files of PanDA infrastructure / A. A. Alekseev [et al.] // Journal of Physics: Conference Series. — Bristol : IOP Publishing, 2018. — Vol. 1015 : Information Technologies in Business and Industry (ITBI2018) : International Conference, January 17-20, 2018, Tomsk, Russian Federation : [proceedings]. — [032003, 6 p.].
Abstract: The paper describes the implementation of a high-performance system for the processing and analysis of log files for the PanDA infrastructure of the ATLAS experiment at the Large Hadron Collider (LHC), responsible for the workload management of order of 2M daily jobs across the Worldwide LHC Computing Grid. The solution is based on the ELK technology stack, which includes several components: Filebeat, Logstash, ElasticSearch (ES), and Kibana. Filebeat is used to collect data from logs. Logstash processes data and export to Elasticsearch. ES are responsible for centralized data storage. Accumulated data in ES can be viewed using a special software Kibana. These components were integrated with the PanDA infrastructure and replaced previous log processing systems for increased scalability and usability. The authors will describe all the components and their configuration tuning for the current tasks, the scale of the actual system and give several real-life examples of how this centralized log processing and storage service is used to showcase the advantages for daily operations.
URI: http://earchive.tpu.ru/handle/11683/52921
Appears in Collections:Материалы конференций

Files in This Item:
File SizeFormat 
dx.doi.org-10.1088-1742-6596-1015-3-032003.pdf499,59 kBAdobe PDFView/Open


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.