Centralized Logging with Graylog

Home Centralized Logging with Graylog

Centralized Logging with Graylog

3rd Jun 2023

Ever feel annoying that you need to SSH in to server just for checking container logs? On few containers it may be fine, but when the deployment get high in my count i had total ~70 containers in total. Then when something happeng to container i need to SSH-ing to my server and check container logs just to look what happens. One of the tool that can help to centralized the container logs is Graylog, when this graylog doesn't work alone, mean it need mongodb and opensearch/elastic to work, but it pretty lightweight to use.

Now lets dig in to deployment

Compose, you can modify as you wish to

version: "3.8"
services:
mongodb:
image: "mongo:5.0"
volumes:
  - "mongodb_data:/data/db"
restart: "always"

opensearch:
image: "opensearchproject/opensearch:2.4.0"
environment:
  - "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g"
  - "bootstrap.memory_lock=true"
  - "discovery.type=single-node"
  - "action.auto_create_index=false"
  - "plugins.security.ssl.http.enabled=false"
  - "plugins.security.disabled=true"
ulimits:
  memlock:
    hard: -1
    soft: -1
  nofile:
    soft: 65536
    hard: 65536
volumes:
  - "os_data:/usr/share/opensearch/data"
restart: "always"

graylog:
hostname: "server"
image: "${GRAYLOG_IMAGE:-graylog/graylog:5.1}"
depends_on:
  opensearch:
    condition: "service_started"
  mongodb:
    condition: "service_started"
entrypoint: "/usr/bin/tini -- wait-for-it opensearch:9200 --  /docker-entrypoint.sh"
environment:
  GRAYLOG_NODE_ID_FILE: "/usr/share/graylog/data/config/node-id"
  GRAYLOG_PASSWORD_SECRET: "${GRAYLOG_PASSWORD_SECRET:?Please configure GRAYLOG_PASSWORD_SECRET in the .env file}"
  GRAYLOG_ROOT_PASSWORD_SHA2: "${GRAYLOG_ROOT_PASSWORD_SHA2:?Please configure GRAYLOG_ROOT_PASSWORD_SHA2 in the .env file}"
  GRAYLOG_HTTP_BIND_ADDRESS: "0.0.0.0:9000"
  GRAYLOG_HTTP_EXTERNAL_URI: "http://localhost:9000/"
  GRAYLOG_ELASTICSEARCH_HOSTS: "http://opensearch:9200"
  GRAYLOG_MONGODB_URI: "mongodb://mongodb:27017/graylog"
  GRAYLOG_ROOT_TIMEZONE: "Asia/Jakarta"
ports:
- "5044:5044/tcp"   # Beats
- "5140:5140/udp"   # Syslog
- "5140:5140/tcp"   # Syslog
- "5555:5555/tcp"   # RAW TCP
- "5555:5555/udp"   # RAW TCP
- "13000:9000/tcp"   # Server API
- "12201:12201/tcp" # GELF TCP
- "12201:12201/udp" # GELF UDP
- "13301:13301/tcp" # Forwarder data
- "13302:13302/tcp" # Forwarder config
volumes:
  - "graylog_data:/usr/share/graylog/data/data"
  - "graylog_journal:/usr/share/graylog/data/journal"
restart: "always"
volumes:
  mongodb_data:
  os_data:
  graylog_data:
  graylog_journal:

Setup .env file

# Generate one by using for example: pwgen -N 1 -s 96
GRAYLOG_PASSWORD_SECRET="somegeneratedrandomnumber
# Create one by using for example: echo -n yourpassword | shasum -a 256
GRAYLOG_ROOT_PASSWORD_SHA2="somegeneratedshasum"

When generated password you may see this somerandomnumber - there is additional dash, ignore that ONLY the random alphnumeric matter.`

Bring up services Graylog logs

2023-06-02 17:37:47,327 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [0.0.0.0:9000]
2023-06-02 17:37:47,328 INFO : org.graylog2.shared.initializers.JerseyService - Started REST API at <0.0.0.0:9000>
2023-06-02 17:37:47,329 INFO : org.graylog2.shared.initializers.ServiceManagerListener - Services are healthy
2023-06-02 17:37:47,329 INFO : org.graylog2.bootstrap.ServerBootstrap - Services started, startup times in ms: {LocalKafkaMessageQueueWriter [RUNNING]=0, InputSetupService [RUNNING]=0, LocalKafkaMessageQueueReader [RUNNING]=0, FailureHandlingService [RUNNING]=0, GracefulShutdownService [RUNNING]=1, ConfigurationEtagService [RUNNING]=1, BufferSynchronizerService [RUNNING]=2, UserSessionTerminationService [RUNNING]=4, PrometheusExporter [RUNNING]=4, OutputSetupService [RUNNING]=4, EtagService [RUNNING]=5, UrlWhitelistService [RUNNING]=5, GeoIpDbFileChangeMonitorService [RUNNING]=5, JobSchedulerService [RUNNING]=5, StreamCacheService [RUNNING]=10, MongoDBProcessingStatusRecorderService [RUNNING]=18, LocalKafkaJournal [RUNNING]=18, LookupTableService [RUNNING]=19, PeriodicalsService [RUNNING]=80, JerseyService [RUNNING]=1336}
2023-06-02 17:37:47,329 INFO : org.graylog2.shared.initializers.InputSetupService - Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2023-06-02 17:37:47,334 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.

Login to web, access http://server-ip:13000

user: admin password: yourpassword that used for generate
Create GELF Listener. What exactly this listener? tldr; an udp port that listen to docker-daemon throwing logs. On graylog web, go to System -> Inputs -> Choose GELF upd dropdown -> Launch new input -> A new popup Check global -> Set title docker-daemon -> Launch input at bottom. Congrats you already listening.

Before you set docker-daemon to globally direct log to Graylog you can test out by modify one of deployment using gelf logging output. Place this on your one deployment compose and recreate container.
```
logging:
  driver: "gelf"
  options:
    gelf-address: "udp://server-ip:12201"
```
Soon you will see the logs coming to Graylog dashboard, don't forget set autoupdate. If the logs coming then you ready to set docker-daemon push through Graylog

Set docker-daemon log to Graylog, edit /etc/docker/daemon.json, add this part or modify if you already had one.

"log-driver": "gelf",
"log-opts": {
  "gelf-address": "udp://server-ip:12201",
  "gelf-compression-type": "none"
},

Edit: I love how live stream log graylog doing, so smooth and realtime.

sources:

Previous Post Next Post