Satisnet Ltd, Basepoint Innovation Centre, 110 Butterfield Great Marlings, Luton, Bedfordshire, LU2 8DL enquiry@satisnet.co.uk
+44 (0) 1582 434320

An Introduction to the ELKeBMWSC(Tn) Stack

An Introduction to the ELKeBMWSC(Tn) Stack

An Introduction to the ELKeBMWSC(Tn) Stack

Introduction

ELK Stack is currently the preferred stack of software for DIY logging. It has been compared to the likes of Splunk, offering the incentive of being open-source. The ELK Stack is initially composed of three software packs:

  • Elasticsearch
  • Logstash
  • And Kibana

For an initial deployment these three components can cope, however the truth is that an ELK Stack requires more than the three mentioned technologies for a successful production deployment.

What is the ELK Stack?

ELK Stack is also known as Elastic Stack; it starts with a combination of three individual technologies that have been tied to work with each other. Each individual technology in the ELK Stack handled one part of the general logging equation:

  • Elasticsearch - Data storage and searching
  • Logstash - Gathering and formatting
  • Kibana - Reporting and analysis

The three components are good, but do not cover the full services required for a healthy, well assembled, all-round, production ready logging stack. Luckily enough, there are additional technologies that can be added to the initial stack that can maintain the health and security, as well as tools to collect and distribute data.

The ELKeBMWS Stack

Production environments are required to be reliable and fault tolerant. For the initial ELK Stack to be production ready, it will need to be supported by other tools. Along with this, health monitoring and redundancy is a major consideration, like all software used in production. The following is a list of additional tools that can be integrated into the ELK Stack, which will assist it to a state of production-worthy.

Beats

In the initial ELK Stack, Logstash would need to be installed on every machine, which in many situations is not ideal. In an enterprise network it would usually be preferred to have a central point of access to process and filter log data. It is possible to install Logstash on a single server and have data shipped to it via the assistance of Beats.

Beats can be installed on every machine being logged, allowing Logstash to be a centralised receiving server waiting for events to be sent from Beats. Beats can define and control the process of sending data from different log types to Logstash. A few variations of Beats that support different event types are:

  • Packetbeat
  • Filebeat
  • Winlogbeat
Marvel

Production services need to be monitored based on health and condition. This area of responsibility can be accomplished via an integration with Marvel. Marvel is aimed to monitor and report on well-being of all the Elastic Stack components. As the stack scales, the requirement for component health monitoring only grows. The importance of Marvel being an essential piece within the production stack is crucial.

Watcher

One of the most important aspects of any logging security tool is to alert administrators when critical events have been flagged. The Elastic Stack integrates with a useful tool called Watcher to provide this essential role. Watcher examines incoming log entries and sends notifications when certain events occur. Kibana can report on the event and Logstash can disperse it; for internal Security Operation staff to be notified immediately of problems before they grow uncontrollably. Watcher supports email-based notifications and other services based on configuration.

Shield

A large consideration to take when installing services is the security layer that surrounds it. Shield was created to meet security needs for the Elastic Stack and to create a way of centrally manage security amongst all components within the stack. It is recommended to use this product over non-Elastic security methods.

Another important security consideration is the use of HTTP communication by default. Typically this should be changed when moving into a production state. An update from HTTP to HTTPS can be done by using Shield with a little bit of additional configuration and the required certificates.

Cluster/Tribe Nodes

Last of all there are scaling concerns that need to be strategically planned. Elasticsearch is built using clusters to help handle and distribute Elasticsearch queries around the network. Clusters include master and data nodes, and potentially clients. Clusters understandably fill up with data over time, and when this happens it will be necessary to scale.

A careful strategic plan to deploy additional nodes within your network topology is required as the Elasticsearch index grows. Eventually, even clusters won’t be sufficient to store all the data Elasticsearch encompasses. All components of Elasticsearch (master nodes, data nodes, and clients) stores information about an Elasticsearch cluster for proper routing called the cluster state.

Eventually, with the cluster state ever increasing, it will be large enough to slow down performance speeds.

If and when this occurs, it will be time to present the current stack with an additional tool called Tribe Nodes. Tribe Nodes allow searching across Elasticsearch clusters thus spreading the load over many machines instead of bulk loading a single machine.

Installing a strong and scalable production Elastic Stack build is more than the initial Elasticsearch, Logstash, and Kibana. A full list of minimum production requirements are as follows:

  • Elasticsearch
  • Logstash
  • Kibana (with an Elasticsearch client)
  • Beats (per server and data-type being logged)
  • Marvel
  • Watcher
  • Shield
  • Clusters
  • Tribe nodes (not initially, but eventually over time)

A well planned, secured, scalable and healthy Elastic Stack build will require each one of the above mentioned tools. Until then it will not come close to a fully functioning production log security tool.

To summarise, the ELK Stack should really be called the ELKeBMWSC(Tn) Stack.