Unleashing Loki: Taming Logs in Kubernetes

Greetings, code warriors! Brace yourselves for a tech adventure. Today, we're exploring Loki, the log-aggregation maestro crafted to untangle your app's log mayhem. Fasten your seatbelts as we walk through setting up Loki in a K3s self-hosted cluster. It's the next chapter in my kube-prometheus-stack saga. Ready? Let's roll.

Unlike other logging systems, Grafana Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels). Log data itself is then compressed and stored in chunks in object stores such as S3 or GCS, or even locally on the filesystem. A small index and highly compressed chunks simplifies the operation and significantly lowers the cost of Loki.

Prepare for Loki's conquest in your Kubernetes realm. Verify a running Kubernetes cluster, wield Helm >=3 (the Kubernetes package manager), and have your trusty terminal by your side. Don't forget the coffee—tech's spartan gods insist on it.

Loki Deployment Modes

Simple Scalable Mode

Image by Grafana Labs

Loki's simple scalable deployment mode separates execution paths into read, write, and backend targets. These targets can be scaled independently, letting you customize your Loki deployment to meet your business needs for log ingestion and log query so that your infrastructure costs better match how you use Loki. The simple scalable deployment mode can scale up to a few TBs of logs per day

Monolithic Mode

Image by Grafana Labs

The simplest mode of operation is the monolithic deployment mode. This mode runs all of Loki’s microservice components inside a single process as a single binary or Docker image. It is good for upto 100GB of logs per day.

Microservices Mode

Image by Grafana Labs

The microservices deployment mode runs components of Loki as distinct processes, allowing limitless scale.

Summoning the Log Sleuth

Steering clear of unnecessary complexities, I'm opting for the monolithic Loki setup, fine-tuning it precisely for my needs. Rolling with Loki version 2.8 or higher, I'll stick to the recommended TSDB and local file storage, giving object storage the subtle nod of avoidance. Because, really, who needs excess baggage? Keep it lean, keep it efficient.

Helm Up!

Add the Grafana chart repository:

helm repo add grafana https://grafana.github.io/helm-charts

Loki Configuration

Channel your inner Norse god and tweak Loki's config to your liking.

The configurations were hand-picked from Source 1, Source 2 and Source 3

deploymentMode: SingleBinary
loki:
    auth_enabled: false
    commonConfig:
        replication_factor: 1
    storage:
        type: 'filesystem'
    storage_config:
        tsdb_shipper:
            active_index_directory: '/var/loki/tsdb-index'
            cache_location: '/var/loki/tsdb-cache'
            cache_ttl: 24h
    schemaConfig:
        configs:
            - from: 2024-01-01
            store: tsdb
            object_store: filesystem
            schema: v13
            index:
                prefix: loki_index_
                period: 24h
    # tsdb needs this configuration
    query_scheduler:
        max_outstanding_requests_per_tenant: 32768
    # 15 days retention, enforced by compactor
    limits_config:
        retention_period: 360h
    compactor:
        working_directory: '/var/loki/retention-compactor'
        compaction_interval: 10m
        retention_enabled: true
        retention_delete_delay: 2h
        retention_delete_worker_count: 150
        delete_request_store: 'filesystem'
        compactor_ring:
            kvstore:
                store: inmemory

    rulerConfig:
        # File path to store temporary rule files.
        rule_path: '/tmp/loki/rules'
        wal:
            dir: '/var/loki/ruler-wal'
        # Configures backend rule storage for a local file system directory.
        storage:
            type: 'local'
            local:
                directory: '/var/loki/rules'
        ring:
            kvstore:
                store: inmemory
        enable_api: true
        enable_alertmanager_v2: true
        alertmanager_url: http://alertmanager.monitoring:9093

singleBinary:
    replicas: 1
    resources:
        limits:
            cpu: 1
            memory: 6Gi
        requests:
            cpu: 0.5
            memory: 2Gi
    nodeSelector:
        region: 'australia'
        kubernetes.io/os: 'linux'
    persistence:
        enabled: true
        enableStatefulSetAutoDeletePVC: true
        size: 10Gi
        storageClass: 'local-path'
        accessModes:
            - ReadWriteOnce

read:
    replicas: 0
backend:
    replicas: 0
write:
    replicas: 0

ingress:
    enabled: false
gateway:
    enabled: false

test:
    enabled: false
lokiCanary:
    enabled: false

# memcache auto deployed, to disable =>
chunksCache:
    enabled: false

resultsCache:
    enabled: false

No need to showcase Loki to the world, so no Ingress. Since it's a monolith, no internal request load balancing either - hence, no gateway. No expert moves here; just lots of doc reading, multiple trial and error rounds, and a scenic tour through GitHub issues. This setup is my victory dance. For a deeper dive into configurations, the official chart doc is the treasure map.

Unleash the Sleuth

Deploy Loki with Helm , Specify the absolute path when creating the values file, because precision matters.

helm install loki grafana/loki --namespace monitoring --values path/to/loki-monolith.yml

Let the Logs Flow

Deploy Promtail on your kubernetes nodes—Loki's trusty sidekick. It discovers targets, scrapes logs from your apps, attaches labels and sends them to the Loki instance. Config is a breeze, with a default Daemonset. The URL resolves locally since both run in the same namespace, monitoring.

promtail.yml

resources:
    limits:
        cpu: 100m
        memory: 512Mi

config:
    clients:
        - url: http://loki:3100/loki/api/v1/push

helm install promtail grafana/promtail --namespace monitoring --values path/to/promtail.yml

Grafana is Your Watson

Loki integrates seamlessly with Grafana, giving you a powerful dashboard to visualize and analyze your logs. Write LogQL queries (think Sherlock's deductions) to uncover hidden patterns and troubleshoot issues.

In a prior post, I covered setting up Kube-Prometheus-Stack. I'll reuse those values but integrate Loki as a datasource for Grafana.

# previous configs..
grafana:
    # other configs..
    # .........
    additionalDataSources:
        - name: Loki
          type: loki
          access: proxy
          url: http://loki:3100/
          isDefault: false
          manageAlerts: true

That's a wrap! Now, with Grafana, I can seamlessly query for logs, using the built in visual query builder, allowing me to query Namespaces, Containers or basically any labels. Thanks for joining me on this journey!

querying logs of loki instance, saved by loki

Final Words

With Loki in your arsenal, your kubernetes cluster's logs become a treasure trove of insights, not a confusing mess. Remember, with great log power comes great responsibility (don't unleash Ragnarok on your storage!). Now go forth, tame those logs, and uncover the secrets they hold!