Grafana Loki: Log Aggregation for Infrastructure

Introduction

Grafana Loki is a horizontally scalable, highly available log aggregation system inspired by Prometheus. Unlike Elasticsearch which indexes the full content of logs, Loki only indexes metadata (labels) and stores log content compressed. This makes Loki much more cost-effective while still enabling powerful queries via LogQL. This guide teaches you to build a complete log aggregation stack: Promtail (collector) → Loki (storage) → Grafana (visualization).

The PLG Stack Architecture

TEXT

┌─────────────────────────────────────────────────────────┐
│                    Your Infrastructure                   │
│                                                          │
│  [App Server]      [Web Server]      [Database]          │
│  /var/log/*.log    /var/log/nginx/   /var/log/postgres/  │
│       │                 │                  │             │
│  [Promtail]        [Promtail]         [Promtail]         │
└────────────────────┼──────────────────────┘             │
                     │ HTTP/gRPC push                      │
                     ▼                                     │
              ┌─────────────┐                              │
              │    Loki      │  ← Stores compressed logs   │
              └──────┬──────┘                              │
                     │ Query (LogQL)                       │
                     ▼                                     │
              ┌─────────────┐                              │
              │   Grafana    │  ← Dashboard & Alerting     │
              └─────────────┘                              │

Deploying the PLG Stack with Docker Compose

YAML

# docker-compose.yml
version: '3.8'

services:
  loki:
    image: grafana/loki:2.9.4
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml
      - loki-data:/loki
    networks:
      - monitoring

  promtail:
    image: grafana/promtail:2.9.4
    volumes:
      - /var/log:/var/log:ro           # Host log files (read-only)
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail-config.yaml:/etc/promtail/config.yaml
    command: -config.file=/etc/promtail/config.yaml
    networks:
      - monitoring
    depends_on:
      - loki

  grafana:
    image: grafana/grafana:10.2.3
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=your-secure-password
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana-datasource.yaml:/etc/grafana/provisioning/datasources/loki.yaml
    networks:
      - monitoring
    depends_on:
      - loki

volumes:
  loki-data:
  grafana-data:

networks:
  monitoring:
    driver: bridge

Loki Configuration

YAML

# loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://alertmanager:9093

# Retention: keep logs for 30 days
limits_config:
  retention_period: 720h  # 30 days
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32

Promtail Configuration

YAML

# promtail-config.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml   # Tracks file read position

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  # Collect nginx access logs
  - job_name: nginx
    static_configs:
      - targets:
          - localhost
        labels:
          job: nginx
          host: web-01
          __path__: /var/log/nginx/*.log
    pipeline_stages:
      - regex:
          expression: '^(?P<remote_addr>S+) - (?P<remote_user>S+) [(?P<time_local>[^]]+)] "(?P<method>S+) (?P<request>S+) (?P<protocol>[^"]+)" (?P<status>d+) (?P<body_bytes_sent>d+)'
      - labels:
          method:
          status:
      - drop:
          expression: ".*GET /health.*"  # Drop health check noise

  # Collect systemd journal logs
  - job_name: systemd-journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
        host: web-01
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'

  # Collect Docker container logs
  - job_name: docker-containers
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/?(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: 'service'

Grafana Data Source Configuration

YAML

# grafana-datasource.yaml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    jsonData:
      maxLines: 1000
      derivedFields:
        # Auto-link trace IDs to Jaeger
        - datasourceUid: jaeger
          matcherRegex: "traceID=(w+)"
          name: TraceID
          url: '$${__value.raw}'

LogQL Query Language

LogQL is Loki's query language, similar to PromQL but for logs:

LOGQL

# Basic log stream selector
{job="nginx", status="500"}

# Filter by regex
{job="nginx"} |= "error" != "health"

# Parse and filter structured JSON logs
{job="app-api"} | json | level="error" | duration > 200ms

# Count errors per minute (metric query)
sum by (status) (
  rate({job="nginx"}[1m])
)

# Top 5 slowest API endpoints
topk(5,
  sum by (request) (
    rate({job="nginx"} | logfmt | duration > 0 [5m])
  )
)

# Find all 5xx errors in production
{job=~"nginx|apache", environment="production"} |= "5" | regexp "HTTP/[0-9.]+ [5][0-9][0-9]"

# Calculate error rate
sum(rate({job="api", level="error"}[5m]))
/
sum(rate({job="api"}[5m]))

Creating Grafana Dashboards for Logs

JSON

// Example Grafana panel configuration (JSON)
{
  "type": "logs",
  "title": "Application Error Logs",
  "datasource": "Loki",
  "targets": [
    {
      "expr": "{job="app-api", level="error"} | json",
      "legendFormat": "",
      "refId": "A"
    }
  ],
  "options": {
    "dedupStrategy": "signature",
    "showLabels": false,
    "showTime": true,
    "sortOrder": "Descending",
    "wrapLogMessage": true
  }
}

Setting Up Loki Alert Rules

YAML

# loki-rules.yaml
groups:
  - name: application-alerts
    rules:
      # Alert when error rate exceeds 5%
      - alert: HighErrorRate
        expr: |
          sum(rate({job="api", level="error"}[5m]))
          /
          sum(rate({job="api"}[5m])) > 0.05
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High error rate in API"
          description: "Error rate is {{ $value | humanizePercentage }} over last 5 minutes"

      # Alert on authentication failures
      - alert: AuthenticationFailures
        expr: |
          sum(rate({job="app"} |= "authentication failed" [5m])) > 10
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Multiple authentication failures detected"

Scaling Loki for Production

YAML

# loki-config.yaml for distributed mode
# Use S3/GCS instead of filesystem for storage
common:
  storage:
    s3:
      endpoint: s3.amazonaws.com
      region: us-east-1
      bucketnames: my-loki-bucket
      access_key_id: ${S3_ACCESS_KEY}
      secret_access_key: ${S3_SECRET_KEY}
      s3forcepathstyle: false

# Separate components for horizontal scaling
# Run as: loki -target=ingester / loki -target=querier / loki -target=distributor

Loki's label-based approach keeps storage costs low while maintaining query speed. The key insight is that Loki shines when you already know what you're looking for — use Grafana dashboards for operational visibility and LogQL for ad-hoc investigation during incidents.