Promtail Config : Getting Started with Promtail

There are many logging solutions available for dealing with log data. Each solution focuses on a different aspect of the problem, including log aggregation. These tools and software are both open-source and proprietary and can be integrated into cloud providers platforms. They also offer a range of capabilities that will meet your needs. Grafana Loki, a new industry solution.

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. A Loki-based logging stack consists of 3 components: promtail is the agent, responsible for gathering logs and sending them to Loki, loki is the main server and Grafana for querying and displaying the logs. In this article, I will talk about the 1^st component, that is Promtail.

Promtail Introduction

Promtail is an agent that ships local logs to a Grafana Loki instance, or Grafana Cloud. It is typically deployed to any machine that requires monitoring. Below are the primary functions of Promtail:

• Discovers targets
• Log streams can be attached using labels
• Logs are pushed to the Loki instance

Promtail currently can tail logs from two sources. These are the local log files and the systemd journal (on AMD64 machines). Promtail must first find information about its environment before it can send any data from log files directly to Loki. This includes locating applications that emit log lines to files that require monitoring. Prometheus’ service discovery mechanism is borrowed by Promtail, but it only currently supports static and Kubernetes service discovery. Promtail is deployed to each local machine as a daemon and does not learn label from other machines. While kubernetes service Discovery fetches the Kubernetes API Server required labels, static covers all other uses.

Configuration for Promtail Web Server

Prometheus’s promtail configuration is done using a scrape_configs section. relabel_configs allows you to control what you ingest and what you drop and the final metadata to attach to the log line. You can configure the web server that Promtail exposes in the Promtail.yaml configuration file:

server:
http_listen_address: 127.0.0.1
http_listen_port: 9080

Loki Push API

Promtail can be configured to receive logs via another Promtail client or any Loki client. This is done by exposing the Loki Push API using the loki_push_api Scrape configuration. This might prove to be useful in a few situations:

• Complex network infrastructures that allow many machines to egress are not ideal.
• You are using Docker Logging Driver to create complex pipelines or extract metrics from logs.
• serverless setups where many ephemeral log sources want to send to Loki, sending to a Promtail instance with use_incoming_timestamp == false can avoid out-of-order errors and avoid having to use high cardinality labels.

Promtail Shipping

Once Promtail has set of targets (i.e. things to read from like files), and all labels have been correctly set, it will begin tailing (continuously reading the logs from targets). After enough data has been read into memory, or after a timeout, it flushes the logs to Loki as one batch.

Promtail will keep track of the offset it last read in a position file as it reads data from sources (files, systemd journal, if configurable). By default, the positions file is stored at /var/log/positions.yaml. Promtail can continue reading from the same location it left in case the Promtail instance is restarted.

Promtail Installation

1. Get Promtail binary zip at the release page. Download Promtail binary zip from the release page

curl -s https://api.github.com/repos/grafana/loki/releases/latest | grep browser_download_url |  cut -d '"' -f 4 | grep promtail-linux-amd64.zip | wget -i –

After the file has been downloaded, extract it to /usr/local/bin

unzip promtail-linux-amd64.zip
sudo mv promtail-linux-amd64 /usr/local/bin/promtail

Check version:

$ promtail --version
promtail, version 2.6.0 (branch: HEAD, revision: 525040a32)
build user:       root@5d9e7a4c92e6
build date:       2022-01-12T16:48:53Z
go version:       go1.16.2
platform:         linux/amd64

2. In the /usr/local/bin directory, create a YAML configuration for Promtail:

sudo vim /etc/promtail-local-config.yaml

3. Add the following content to the file:

server:
http_listen_port: 9080
grpc_listen_port: 0

positions:
filename: /data/loki/positions.yaml

clients:
- url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: system

static_configs:
- targets:
- localhost

labels:
job: varlogs

__path__: /var/log/*log

4. Make a service for Promtail. We will now configure Promtail to be a service, so it can continue running in the background.

sudo vim /etc/systemd/system/promtail.service

Add this script:

[Unit]
Description=Promtail service
After=network.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/promtail -config.file /etc/promtail-local-config.yaml

[Install]
WantedBy=multi-user.target
EOF

5. Reload and start Promtail service

sudo systemctl daemon-reload
sudo systemctl start promtail.service

Confirm if service is in running state:

$ systemctl status promtail.service

promtail.service – Promtail service

Loaded: loaded (/etc/systemd/system/promtail.service; disabled; vendor preset: enabled)

Active: active (running) since Thu 2022-07-07 10:22:16 UTC; 5s ago

Main PID: 13667 (promtail)

Tasks: 6 (limit: 1267)

Memory: 8.4M

CGroup: /system.slice/promtail.service

└─15381 /usr/local/bin/promtail -config.file /etc/promtail-local-config.yaml

Jul 07 10:22:16 ubuntu systemd[1]: Started Promtail service.

Jul 07 10:22:16 ubuntu promtail[13667]: level=info ts=2022-07-07T10:22:16.812189099Z caller=server.go:225 http=[::]:9080 grpc=[::]:35499 msg=”server listening on>

Jul 07 10:22:16 ubuntu promtail[13667]: level=info ts=2020-07-07T11

This example uses Promtail for reading the systemd-journal. The promtail user will not yet have the permissions to access it. Add the user promtail into the systemd-journal group

usermod -a -G systemd-journal promtail

You can stop the Promtail service at any time by typing

sudo service promtail stop
sudo service promtail status

Configure Firewall

Remote access may be possible if your Promtail server has been running. If localhost is not required to connect to your server, type

iptables -A INPUT -p tcp -s localhost — dport 9080 -j ACCEPT
iptables -A INPUT -p tcp — dport 9080 -j DROP
iptables -L

Final Thoughts

So that is all the fundamentals of Promtail you needed to know. Promtail is a logs collector built specifically for Loki. It uses the same service discovery as Prometheus and includes analogous features for labelling, transforming, and filtering logs before ingestion into Loki. Go ahead, setup Promtail and ship logs to Loki instance or Grafana Cloud.