We already talk about centralized logging with graylog but the caveat of it, is that the memory resource hog that about 4GBs in total, one for graylog itself and the other one is for opensearch/elasticsearch. As my server got limited amount of only 32GBs, im looking on more lightweight solution. Then here comes fluentd + openobserve, when the concept is pretty alike the graylog, this stack combo is far simpler. Lets jump in to installation!

1. The compose

   services:
   app:
     image: public.ecr.aws/zinclabs/openobserve:latest
     environment:
       - ZO_DATA_DIR=/data
       - [email protected]
       - ZO_ROOT_USER_PASSWORD=openobservepass
     volumes:
       - data:/data
     ports:
       - 5070:5080
     restart: unless-stopped
   
   volumes:
   data:

2. Then need to deploy our listener, different with graylog that listener created from the graylog itself, in this deployment we deploy separate listener, that is fluentd

   services:
   logger:
     image: fluent/fluentd:edge
     ports:
       - 24224:24224
     volumes:
       - ./fluentd.conf:/fluentd/etc/docker.conf
     environment:
       - FLUENTD_CONF=docker.conf
     restart: unless-stopped

   Now, about the fluentd conf, we can get this config from openobserve. Log in to openobserve -> Click on Ingestion sidebar -> Select the fluentd -> Copy and Paste to fluentd.conf. This fluentd.conf will be mounted to our fluentd container. Basically fluentd will redirect the logs to openobserve.

   

   Then deploy the fluentd.

3. Like graylog with the gelf modification on docker daemon.json we also need update the docker daemon to use fluentd output.

   "log-driver": "fluentd",
   "log-opts": {
    "fluentd-address": "hostip:24224",
    "fluentd-async": "true"
   }

4. Restart docker service, and the logs will splurge into fluentd and fluentd will relay to openobserve. This is my openobserve logs.

   

Resource comparison: Graylog stack

Openobserve stack