SIEM and Log Management with ELK Stack

Introduction

A SIEM (Security Information and Event Management) system collects, correlates, and analyzes log data from across your infrastructure to detect security incidents. The ELK Stack (Elasticsearch, Logstash, Kibana) combined with Beats agents is a popular open-source SIEM platform.

ELK Stack Components

Elasticsearch: Stores and indexes log data, provides fast search
Logstash: Log processing pipeline (parse, filter, enrich)
Kibana: Web UI for dashboards and visualization
Filebeat/Metricbeat: Lightweight agents that ship logs from servers
Elastic SIEM: Built-in security detection in Kibana

Quick Install with Docker Compose

YAML

# docker-compose.yml
version: '3'
services:
  elasticsearch:
    image: elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
    volumes:
      - esdata:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"

  kibana:
    image: kibana:8.11.0
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      - elasticsearch

  logstash:
    image: logstash:8.11.0
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
    ports:
      - "5044:5044"
    depends_on:
      - elasticsearch

volumes:
  esdata:

Logstash Pipeline Configuration

RUBY

# /logstash/pipeline/syslog.conf
input {
  beats {
    port => 5044
  }
}

filter {
  # Parse syslog format
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }

  # Enrich with GeoIP for external IPs
  if [src_ip] and [src_ip] !~ /^(10.|192.168.|172.1[6-9].|172.2[0-9].|172.3[01].)/ {
    geoip {
      source => "src_ip"
      target => "geoip"
    }
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
}

Installing Filebeat on Servers

BASH

# Install Filebeat on each server you want to monitor
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.11.0-amd64.deb
dpkg -i filebeat-8.11.0-amd64.deb

# Configure /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/syslog
    - /var/log/auth.log
    - /var/log/nginx/*.log
  fields:
    type: syslog
    server: web01

output.logstash:
  hosts: ["logstash-server:5044"]

# Enable modules for common services
filebeat modules enable system nginx

# Start Filebeat
systemctl start filebeat
systemctl enable filebeat

Creating Kibana Dashboards

Go to Kibana → Stack Management → Index Patterns
Create pattern: logs-*
Go to Discover → search and filter logs
Go to Dashboard → Create → Add visualizations

Useful Kibana Queries (KQL)

TEXT

# Failed SSH logins
event.type: authentication_failure

# Connections from a specific country
geoip.country_name: "Unknown"

# High frequency events from one IP
source.ip: 1.2.3.4

# Nginx 5xx errors
http.response.status_code >= 500

Security Detection Rules

Kibana SIEM has built-in detection rules. Enable them:

TEXT

Kibana → Security → Rules → Load Elastic prebuilt rules

Common rules to enable:

Linux: Unusual Process Execution
Linux: SSH Brute Force
Network: DNS Activity to Unusual TLD
Windows: Credential Dumping

Custom Alert with ElastAlert

YAML

# /etc/elastalert/rules/ssh_brute_force.yaml
name: SSH Brute Force
type: frequency
index: logs-*
num_events: 10
timeframe:
  minutes: 5
filter:
  - term:
      program: sshd
  - term:
      message: "Failed password"
alert:
  - email
email:
  - "security@company.com"
smtp_host: mail.company.com

Summary

ELK Stack is a powerful open-source SIEM platform
Deploy Filebeat agents on all servers to centralize logs
Use Logstash pipelines to parse and enrich log data
Create Kibana dashboards for real-time visibility
Enable Elastic Security rules to detect common attack patterns