ELK Stack is one of the popular Log Analytics Solution in the current IT Industry. It collects logs from all services, applications, servers, networks and tools to a centralized location for analyzing and processing. This all helps to troubleshoot problems, monitor services and reduce time to solve the operational issues. It is also used in Business Intelligence to identify the users and their behaviour.
Why Log Analysis becoming important?
We usually face slow performance issues, high cpu utilizations and bugs in our application code. In order to rectify them we need find the root cause of such problems which may be time taking and will need more effort to debug them. This tool helps to view the performance metrics combined with all the events generated by apps, operating systems and network devices.
We are currently moving to the public clouds for our requirement and in that environment, performance isolation is becoming difficult to reach. The performance of these virtual machines can fluctuate based on loads, infrastructure servers, environments and number of active users.
Log Management Platform can monitor all such issues and can process any type of log which can later be visualized.
In order to use ELK Stack to monitor the perfomance of our system, we need some tools and integration to be done.
Open Source Products in ELK Stack
If we consider any monitoring solution monitored by ELK then the following process takes place – Metric Beat collects the server metrics and ship that data to Elastic Search, where we can search and visualize using Kibana. After we get the basic setup working, Logstash is used for additional parsing.
Flow of this Stack
Beats (Data Collection) => Logstash (Data Aggregation/Processing) => Elastic Search (Indexing and Storing) => Kibana (Analysis and Visualization)
So ELK Stack is the collection of four open source products as we mentioned in the above lines
1 . Elastic Search
Elastic Search is a real time, distributed storage, search and analytics engine. It can be used for many purposes but, it excels in index streaming of semi structured data like in logs and decoded network packets.
It can be installed on our hardware to run or we can use hosted Elastic Search Service on Elastic Cloud.
Kibana is an open source analytics and visualization platform designed to work with Elastic Search. It is used to search, view and analyise data stored in Elastic Search Indices. The data can be visualized in form of tables, charts and graphs.
Kibana can be installed on the same server where Elastic Search is installed. If it is installed on different server, then the change of URL:Port (Eastic Search Server) is required in the config file kibana.yml
Beats are open source that we installed as agents on our server to ship the data to Elastic Search directly or via Logstash for further process.
Beat is installed as an independent product. We have different types of beats to capture different type of Data
- AuditBeat : Capturing Audit Data
- FileBeat : Capturing Log Files
- MetricBeat : Capturing Metrics
- HeartBeat : Availability Monitoring
- PacketBeat : Network Traffic
- WinLogBeat : Windows Event Logs
Logstash is an open source powerful tool that integrates with other deployments. It offers a large number of plugins to help us parse, enrich, transform and buffer data for variety of sources. It injects data from multiple sources, transforms it and send it to Elastic Search.
Why ELK Stack is popular?
ELK Stack is popular because it fulfils the need of log analytical solution. Splunk has been in the market since long but it is too expensive for the smaller companies such as start ups and SAAS Businesses. ELK is a simple but robust log analysis platform that costs a fraction of the price.