Technologies for Logging via the Network

Microservices can support central logging by sending log data directly via the network. Most log libraries support such an approach. Special protocols like GELF (Graylog Extended Log Format)1 can be used for this or long-established protocols like syslog, which is the basis for logging in UNIX systems. Tools like the logstash- forwarder,[1] [2] Beaver,[3] or Woodchuck[4] are meant to send local files via the network to a central log server. They are sensible in cases where the log data is supposed to be also locally stored in files.

ELK for Centralized Logging

Logstash, Elasticsearch, and Kibana can serve as tools for the collection and processing of logs on a central server (see Figure 11.1). These tools form the ELK stack (Elasticsearch, Logstash, Kibana).

  • • With the aid of Logstash[5] log files can be parsed and collected by servers in the network. Logstash is a very powerful tool. It can read data from a source, modify or filter data, and finally write it into a sink. Apart from importing logs from the network and storage in Elasticsearch, Logstash supports many other data sources and data sinks. For example, data can be read from message queues or databases or written into them. Finally, Logstash can also parse data and supplement it—for example, time stamps can be added to each log entry, or individual fields can be cut out and further processed.
  • • Elasticsearch[6] stores log data and makes it available for analyses. Elasticsearch cannot only search the data with full text search, but it can also search in individual fields of structured data and permanently store the data like a database. Finally, Elasticsearch offers statistical functions and can use those to analyze
ELK Infrastructure for Log Analysis

Figure 11.1 ELK Infrastructure for Log Analysis

data. As a search engine Elasticsearch is optimized for fast response times so that the data can be analyzed quasi-interactively.

• Kibana[7] is a web user interface that enables analysis of data from Elastic- search. In addition to simple queries, statistical evaluations, visualizations and diagrams can be created.

All three tools are open source projects and are available under the Apache 2.0 license.

  • [1]
  • [2]
  • [3]
  • [4]
  • [5]
  • [6]
  • [7]
< Prev   CONTENTS   Source   Next >