SpringBoot integration of lightweight logging system loki - 1

The project is formalized, the logging system is indispensable. The majority of logging platforms recommended to build based on ELK, but ELK is relatively heavy, the architecture is too large, small and medium-sized projects are not very good to hold, I hope to find a simple, if you really can not find then use ELK. before the reserve some technical candidates library, looked through the fruit of a logging system called Loki, but a search is mostly this guy in the movie. I couldn’t help but think, “Is this reliable?

Loki

In fact, the logging system is developed by a reliable team, Grafana Labs, a big name in monitoring, known for open source data visualization tools Grafana, monitoring system Prometheus, etc.

Loki is their open source project inspired by Prometheus, a scalable, highly available, multi-tenant log aggregation system. Designed with the idea of making log aggregation easier, it is designed to be very cost effective and easy to operate. It does not index the contents of logs , but rather a set of tags for each log stream . It consists of three main parts.

• Promtail is the log collector that collects the application’s logs and sends them to Loki.
• Loki is used for log storage and parsing, and provides a query API for downstream presentation.
• Grafana is responsible for visualizing Loki’s logs.

Looks pretty good, I’m going to give it a try.

simply try it

I ran a small DEMO first when I encountered something new, and to be honest, there is no detailed hands-on DEMO for this currently introduced in China, especially the part that interfaces with Spring Boot. I followed the original documentation, and spent a little more than half a day to finally run successfully.

Loki installation

The following is my modified Docker Compose script, according to your own needs to change to start Loki with one click.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:2.2.1
    container_name: loki-service
    volumes:
#    将loki的配置文件挂载到本地 c:/docker/loki 目录
      - c:/docker/loki:/etc/loki/
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/loki.yml
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.2.1
    container_name: promtail-service
    volumes:  
    #  为了读取本地的日志目录，这个是个默认配置目的就是为跑起来，生产肯定不是这样的。   
      - c:/docker/log:/var/log/
    #  promtail  的配置文件也挂载到本地 c:/docker/promtail目录  
      - c:/docker/promtail:/etc/promtail/
    command: -config.file=/etc/promtail/promtail.yml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    container_name: grafana-service
    ports:
      - "3000:3000"
    networks:
      - loki

The above mount directories c:/docker/loki and c:/docker/promtail you adjust the location to suit yourself.

Loki configuration

The -config.file=/etc/loki/loki.yml in the above file is the configuration file for Loki, we need to put the configuration file loki.yml under c:/docker/loki in advance, I use the default configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h       # Any chunk not receiving new logs in this time will be flushed
  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
  chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  max_transfer_retries: 0     # Chunk transfers disabled

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

compactor:
  working_directory: /loki/boltdb-shipper-compactor
  shared_store: filesystem

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

ruler:
  storage:
    type: local
    local:
      directory: /loki/rules
  rule_path: /loki/rules-temp
  alertmanager_url: http://localhost:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true

Don’t get hung up on what these configuration items do, run it first and check the documentation when you need it.

Promtail configuration

Similar to Loki, Promtail should also be configured in the c:/docker/promtail directory of the local mount promtail.yml, and the default configuration is used here.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          # 这个跟挂载的位置有点关系，你可以猜猜
          __path__: /var/log/*log

I guess /var/log/*log is where the logs are read, so I mounted it locally at c:/docker/log and will get some logs into this local directory and see if I can read them.

Start Loki

After configuring, execute docker-compose -f <docker-compose.yml path> up command, it will download the image and start three Docker containers first. After success, open http://localhost:3000/ and login to Grafana with the default account password admin/admin. Then add the data source as Loki in the sidebar.

Then configure Loki’s URL as http://loki:3100, then click OK and test, a green prompt means success.

We are using Docker Compose, so hostname is the service name loki.

Then click on the compass-shaped icon Explore in the sidebar to enter the logging UI, which is nothing at this point.

You have to create some logs, get a Spring Boot application, then configure the logging options in application.yml and start the application to generate some logs.

1
2
3
4
5
6

logging:
  file:
  # 弄到疑似Promtail的日志读取路径试试
    path: c:/docker/log
  level:
    org: debug

Then I entered a Loki query expression from the documentation {filename="/var/log/spring.log"} , and the file name went to c:/docker/log and there it was!

Summary

Today I demoed Spring Boot from scratch against the newest logger Loki, and it looks really good. Learn something new, be clear about its scenarios, be clear about your goals at each step, run up the DEMO, then go into customization, and finally the underlying principles. The more you learn the more skilled you are, the more you learn the more you know your shortcomings, in order to have goals and directions, do not learn blindly, and do not pursue the underlying principles too much. Well, later I will continue to research and explore how Loki in the production environment of the actual use.

Reference https://felord.cn/loki.html