How to Setup Grafana Loki for Free Log Management
Setup log management using an open-source solution.
Why Do We Need Log Management?
Let’s say you're running your application on a dedicated server. Then a bug occurred on your application, you need to debug it, the first thing you reach out to is most probably will be the log file, you will be looking for the application stack trace. If your application does not have much traffic then writing the application to the log file and using cat, tail, and grep is more than enough to find the specific log regarding that error.
But what happened when the logs are spread in several files? Let’s say you have set up a log rotation that will rotate the log every few hours or a certain file size has been reached. The same stack trace can be split into several files, finding the stack trace in several log files is a time-consuming process. What’s more nowadays microservices pattern is trending, each service will write log to its own log file. Find an error across the services log files is such a hassle.
Log management gives us an interface to look for the same trace in several log files as in the log is only written on a single file.
Why Grafana Loki?
There are several log management solutions that exist. Most of them have free to pay schema but only to a certain limit, from 100 MB to 100 GB. Grafana Loki is an open-source solution. Hence, it’s free without any limitation.
Another popular open-source solution is ELK (Elasticsearch, Logstash, and Kibana) stack. In order to run it, we need JVM which inevitably requires a lot of resources to run. On the Elastic documentation, it says a machine with less than 8 GB memory tends to be counterproductive.
On the other hand, Grafana Loki can be run smoothly on a relatively small server. I have been running Grafana Loki on my hobby machine which only has 2 core and 2 GB memory without any hiccup for over 2 years now.
Prerequisite
An Ubuntu server running Linux version ≥18.04.
Steps
1. Download the configuration files.
$ sudo mkdir /opt/loki
$ cd /opt/loki$ wget https://raw.githubusercontent.com/grafana/loki/master/cmd/loki/loki-local-config.yaml$ wget https://raw.githubusercontent.com/grafana/loki/main/clients/cmd/promtail/promtail-local-config.yaml
Remember to modify promtail-local-config.yaml log file path to point at your application log.
If you cannot download the configuration files, create them manually using the configuration below.
promtail-local-config.yaml
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log # modify to point at your application log fileloki-local-config.yaml
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
ingester:
wal:
enabled: true
dir: /tmp/wal
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 1h # Any chunk not receiving new logs in this time will be flushed
max_chunk_age: 1h # All chunks will be flushed when they hit this age, default is 1h
chunk_target_size: 1048576 # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is
reached first
chunk_retain_period: 30s # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
max_transfer_retries: 0 # Chunk transfers disabled
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /tmp/loki/boltdb-shipper-active
cache_location: /tmp/loki/boltdb-shipper-cache
cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
shared_store: filesystem
filesystem:
directory: /tmp/loki/chunks
compactor:
working_directory: /tmp/loki/boltdb-shipper-compactor
shared_store: filesystem
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: false
retention_period: 0s
ruler:
storage:
type: local
local:
directory: /tmp/loki/rules
rule_path: /tmp/loki/rules-temp
alertmanager_url: http://localhost:9093
ring:
kvstore:
store: inmemory
enable_api: true3. Download Grafana Loki and Promtail.
$ wget https://github.com/grafana/loki/releases/download/v2.2.1/loki-linux-amd64.zip$ sudo unzip loki-linux-amd64.zip$ wget https://github.com/grafana/loki/releases/download/v2.2.1/promtail-linux-amd64.zip$ sudo unzip promtail-linux-amd64.zip
4. Ensure binaries and configurations exists.
$ lsOutput:
root@arwego:/opt/loki# ls -l
total 153964
-rwxr-xr-x 1 root root 53501952 Apr 6 07:55 loki-linux-amd64
-rw-r--r-- 1 root root 17277632 Apr 6 07:58 loki-linux-amd64.zip
-rw-r--r-- 1 root root 1832 May 13 13:18 loki-local-config.yaml
-rwxr-xr-x 1 root root 67333584 Apr 6 07:57 promtail-linux-amd64
-rw-r--r-- 1 root root 19522160 Apr 6 07:58 promtail-linux-amd64.zip
-rw-r--r-- 1 root root 303 May 13 13:18 promtail-local-config.yaml
4. Setup systemd for Promtail.
$ sudo cat <<EOF >/etc/systemd/system/promtail.service
[Unit]
Description=Promtail service
After=network.target
[Service]
Type=simple
#User=promtail
ExecStart=/opt/loki/promtail-linux-amd64 -config.file /opt/loki/promtail-local-config.yaml
Restart=always
[Install]
WantedBy=multi-user.target
EOF$ sudo systemctl start promtail.service
$ sudo systemctl enable promtail.service
5. Setup systemd for Loki.
$ sudo cat <<EOF >/etc/systemd/system/loki.service
[Unit]
Description=Loki service
After=network.target
[Service]
Type=simple
#User=loki
ExecStart=/opt/loki/loki-linux-amd64 -config.file /opt/loki/loki-local-config.yaml
Restart=always
[Install]
WantedBy=multi-user.target
EOF$ sudo systemctl start loki.service
$ sudo systemctl enable loki.service
6. Ensure both services is up and running.
$ sudo systemctl status promtailOutput:
● promtail.service - Promtail service
Loaded: loaded (/etc/systemd/system/promtail.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2021-05-13 13:34:38 WIB; 1min 10s ago
Main PID: 11234 (promtail-linux-)
Tasks: 11 (limit: 19006)
Memory: 36.3M
CGroup: /system.slice/promtail.service
└─11234 /opt/loki/promtail-linux-amd64 -config.file /opt/loki/promtail-local-config.yaml$ sudo systemctl status lokiOutput:
● loki.service - Loki service
Loaded: loaded (/etc/systemd/system/loki.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2021-05-13 13:34:15 WIB; 2min 43s ago
Main PID: 11194 (loki-linux-amd6)
Tasks: 9 (limit: 19006)
Memory: 66.0M
CGroup: /system.slice/loki.service
└─11194 /opt/loki/loki-linux-amd64 -config.file /opt/loki/loki-local-config.yaml$ curl http://localhost:3100/metricsOutput:
# HELP cortex_cache_corrupt_chunks_total Total count of corrupt chunks found in cache.
# TYPE cortex_cache_corrupt_chunks_total counter
cortex_cache_corrupt_chunks_total 0
# HELP cortex_chunk_store_chunks_per_query Distribution of #chunks per query.
# TYPE cortex_chunk_store_chunks_per_query histogram
cortex_chunk_store_chunks_per_query_bucket{le="10"} 0
cortex_chunk_store_chunks_per_query_bucket{le="80"} 0
...
7. Install Grafana.
$ sudo apt-get install -y apt-transport-https
$ sudo apt-get install -y software-properties-common wget$ wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -$ echo "deb https://packages.grafana.com/enterprise/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list$ sudo apt-get update
$ sudo apt-get install grafana$ sudo systemctl daemon-reload
$ sudo systemctl start grafana-server
$ sudo systemctl enable grafana-server
8. Ensure Grafana is up and running.
$ sudo systemctl status grafana-serverOutput:
● grafana-server.service - Grafana instance
Loaded: loaded (/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2021-05-13 13:58:26 WIB; 1min 27s ago
Docs: http://docs.grafana.org
Main PID: 16311 (grafana-server)
Tasks: 10 (limit: 19006)
Memory: 17.0M
CGroup: /system.slice/grafana-server.service$ curl http://localhost:3000Output:
<a href="/login">Found</a>.
8. Go to your browser and open http://localhost:3000/login.
9. Login using the default credential:
username: admin
password: admin10. Create a new password.
11. Go to http://localhost:3000/explore and add a new data source.
12. Add Loki data source.
13. Input http://localhost:3100 for the HTTP URL field.
14. Click Save & Test. You should a green pop up saying that data sources connected and labels found.
15. Go to http://localhost:3000/explore again. Your Loki data source should show up now.
16. Enter {job=”varlogs”} on the query field and run it by typing CTRL + Enter or the blue button on the right top corner. You should see some logs are shown.
Voilà, and we are done. We have successfully set up Grafana Loki for or log management.
Troubleshooting
Conflicting Port
The most common problem is conflicting port. Grafana is running on port 3000, Loki is running on port 3100, and Promtail is running on 9080. Ensure these ports are not occupied by other services. If it’s occupied, you will need to change the configuration to an available port and restart the services.
Missing Log
The most common problem is you forget to update the promtail-local-config.yaml file path to point at your application log. Another common problem, your application actually does not write any log to the log file.
Understanding Query Syntax
It’s using Grafana LogQL. You can learn more about it here.
References
- Grafana Loki: https://grafana.com/docs/loki/latest/
- Grafana LogQL: https://grafana.com/docs/loki/latest/logql/
- Hardware specification for ELK: https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html
