Kubernetes + Spring Cloud Integration Link Tracking SkyWalking

1. Overview

1. What is SkyWalking?

Application performance monitoring tool for distributed systems, designed for microservices, cloud-native architectures and container-based (Docker, K8s, Mesos) architectures. Provides distributed tracing, service grid telemetry analysis, metric aggregation and visualization all-in-one solution.

Official website address: http://skywalking.apache.org/

2. SkyWalking features

Multiple monitoring tools, language probes and Service Mesh
Multilingual automated probes, Java, .NET Core and Node.JS
Lightweight and efficient, no need for big data
Modularity, with multiple mechanisms for UI, storage, and cluster management available
Alarm support
Excellent visualization scheme

3. Overall structure

The entire architecture, divided into four parts: top, bottom, left and right.

In order to make the description simpler, we discard the Metric metrics related and focus on the Tracing link related functions.

Top part Agent: responsible for collecting link information from the application and sending it to SkyWalking OAP server. Currently, it supports Tracing data information provided by SkyWalking, Zikpin, Jaeger and so on. And what we currently use is that SkyWalking Agent collects SkyWalking Tracing data and passes it to the server.
The lower part SkyWalking OAP: responsible for receiving the Tracing data information sent by Agent, then analyzing (Analysis Core), storing to the external storage (Storage), and finally providing the query (Query) function.
Right part Storage : Tracing data storage. Currently, ES, MySQL, Sharding Sphere, TiDB and H2 are supported. The main consideration is that the SkyWalking development team’s own production environment uses ES as the main storage.
Left part SkyWalking UI: responsible for providing console, viewing links, etc.

A simple overview of the principle is shown in the following figure.

2. Build skywalking

1. Environment preparation

Mkubernetes version: 1.18.5
Nginx Ingress version: 2.2.8
Helm version: 3.2.4
Persistent storage driver: NFS

2、Deployment with charts

This article is about how to use Helm Charts to deploy SkyWalking to a Kubernetes cluster, the related documentation can be found in skywalking-kubernetes

Four approaches are currently recommended.

Start a local helm repo using the helm serve provided by helm 3
deploy with a local chart file
Use the repo feature provided by harbor
Deploy directly from the official repo (not available yet)

Note: The chart of skywalking has not been submitted to the official repository yet, please refer to the first three ways to deploy first

2.1. Download the chart file

You can deploy skywalking directly using local files, follow the steps above to download the skywalking chart and then deploy it directly using the following command.

git clone https://github.com/apache/skywalking-kubernetes
cd skywalking-kubernetes/chart
helm repo add elastic https://helm.elastic.co
helm dep up skywalking
export SKYWALKING_RELEASE_NAME=skywalking  # 定义自己的名称
export SKYWALKING_RELEASE_NAMESPACE=default  # 定义自己的命名空间

2.2. Define the existing es parameter file

Modify values-my-es.yaml:

oap:
  image:
    tag: 8.1.0-es7      # Set the right tag according to the existing Elasticsearch version
  storageType: elasticsearch7

ui:
  image:
    tag: 8.1.0

elasticsearch:
  enabled: false
  config:               # For users of an existing elasticsearch cluster,takes effect when `elasticsearch.enabled` is false
    host: elasticsearch-client
    port:
      http: 9200
    user: "elastic"         # [optional]
    password: "admin@123"     # [optional]

2.3, helm installation

1
2

helm install "${SKYWALKING_RELEASE_NAME}" skywalking -n "${SKYWALKING_RELEASE_NAMESPACE}" \
  -f ./skywalking/values-my-es.yaml

After the installation was completed, we verified the following installation.

$ kubectl get deployment -n skywalking
NAME                READY   UP-TO-DATE   AVAILABLE   AGE
my-skywalking-oap   2/2     2            2           9m
my-skywalking-ui    1/1     1            1           9m

3. Using Skywalking Agent

There are three ways to use agent in Java

Use the official base image provided
Build the agent package into an existing base image
mount the agent in sidecar mode (recommended)

1. Use the official base image

Check the official docker hub provided base image, just need to build your service image is From this image, directly integrated into Jenkins can be more convenient

2. Build the agent package into an existing base image

The reason for this approach is: the official image is a compact image, and openjdk, many commands may not have, need to install their own secondary, here omitted.

3. sidecar mode mount agent

Since the service is deployed in Kubernetes, the advantage of using Skywalking Agent in this way is that you don’t need to modify the original base image or rebuild a new service image, but mount the agent’s required files to the existing service image in sidecar mode by sharing the volume.

1. build skywalking agent image

Build it yourself, refer to: https://hub.docker.com/r/prophet/skywalking-agent

Build with the following dockerfile.

FROM alpine:3.8

LABEL maintainer="zuozewei@hotmail.com"

ENV SKYWALKING_VERSION=8.1.0

ADD http://mirrors.tuna.tsinghua.edu.cn/apache/skywalking/${SKYWALKING_VERSION}/apache-skywalking-apm-${SKYWALKING_VERSION}.tar.gz /

RUN tar -zxvf /apache-skywalking-apm-${SKYWALKING_VERSION}.tar.gz && \
    mv apache-skywalking-apm-bin skywalking && \
    mv /skywalking/agent/optional-plugins/apm-trace-ignore-plugin* /skywalking/agent/plugins/ && \
    echo -e "\n# Ignore Path" >> /skywalking/agent/config/agent.config && \
    echo "# see https://github.com/apache/skywalking/blob/v8.1.0/docs/en/setup/service-agent/java-agent/agent-optional-plugins/trace-ignore-plugin.md" >> /skywalking/agent/config/agent.config && \
    echo 'trace.ignore_path=${SW_IGNORE_PATH:/health}' >> /skywalking/agent/config/agent.config

`1`	`docker build -t 172.16.106.237/monitor/skywalking-agent:8.1.0 .`

After the docker build is finished, just push it to the repository.

2. Using sidecar mount

The sample configuration file is as follows.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-skywalking
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-skywalking
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: demo-skywalking
    spec:
      initContainers:
        - name: init-skywalking-agent
          image: 172.16.106.237/monitor/skywalking-agent:8.1.0
          command:
            - 'sh'
            - '-c'
            - 'set -ex;mkdir -p /vmskywalking/agent;cp -r /skywalking/agent/* /vmskywalking/agent;'
          volumeMounts:
            - mountPath: /vmskywalking/agent
              name: skywalking-agent
      containers:
        - image: nginx:1.7.9
          imagePullPolicy: Always
          name: nginx
          ports:
            - containerPort: 80
              protocol: TCP
          volumeMounts:
            - mountPath: /opt/skywalking/agent
              name: skywalking-agent
      volumes:
        - name: skywalking-agent
          emptyDir: {}

The above is the deployment.yaml file for mounting the sidecar, taking nginx as an example, the main way to mount the agent is by sharing the volume, first initContainers mounts the sw-agent-sidecar /vmskywalking/agent and cp the agent directory in the image built above to the /vmskywalking/agent directory, after that nginx also mounts the skywalking-agent volume when it starts and mounts it to the container’s /opt/skywalking /agent directory of the container, which completes the sharing process.

4. Transforming Spring Cloud Applications

1. docker package and push to the repository

Modify the dockerfile configuration to integrate the skywalking agent:

FROM insideo/centos7-java8-build
VOLUME /tmp
ADD mall-admin.jar app.jar
RUN bash -c 'touch /app.jar'
RUN ln -snf /usr/share/zoneinfo/Asia/Shanghai  /etc/localtime && echo Asia/Shanghai > /etc/timezone
ENTRYPOINT ["java","-Dapp.id=svc-mall-admin","-javaagent:/opt/skywalking/agent/skywalking-agent.jar","-Dskywalking.agent.service_name=svc-mall-admin","-Dskywalking.collector.backend_service=my-skywalking-oap.skywalking.svc.cluster.local:11800","-jar","-Dspring.profiles.active=prod","-Djava.security.egd=file:/dev/./urandom","/app.jar"]

Once you’ve changed it, run maven package directly to package the project as an image.

Note that.

When k8s creates a Service, it creates the appropriate DNS entry. The format of this entry is <service-name>. <namespace-name>.svc.cluster.local, which means that if the container only uses <service-name>, it will resolve as a local service to the namespace. For cross-namespace access, a fully qualified domain name is required.

2. Write the deployment script for the yaml version of k8s

Here I will use one of the services as an example.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: svc-mall-admin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: svc-mall-admin
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: svc-mall-admin
    spec:
      initContainers:
        - name: init-skywalking-agent
          image: 172.16.106.237/monitor/skywalking-agent:8.1.0
          command:
            - 'sh'
            - '-c'
            - 'set -ex;mkdir -p /vmskywalking/agent;cp -r /skywalking/agent/* /vmskywalking/agent;'
          volumeMounts:
            - mountPath: /vmskywalking/agent
              name: skywalking-agent
      containers:
        - image: 172.16.106.237/mall_repo/mall-admin:1.0
          imagePullPolicy: Always
          name: mall-admin
          ports:
            - containerPort: 8180
              protocol: TCP
          volumeMounts:
            - mountPath: /opt/skywalking/agent
              name: skywalking-agent
      volumes:
        - name: skywalking-agent
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: svc-mall-admin
spec:
  ports:
    - name: http
      port: 8180
      protocol: TCP
      targetPort: 8180
  selector:
    app: svc-mall-admin

Then you can run it directly, and it will be able to run the project all up.

5. Test Verification

When finished, you can go to SkyWalking UI to check whether the link collection is successful.

1. Test the application API

First, request the API provided by the Spring Cloud application, because we want to trace the link.

2. View SkyWalking UI interface

Here, we will see three very important concepts in SkyWalking.

Service (Service): represents a series or group of workloads that provide the same behavior to a request. When using the Agent or SDK, you can define the name of the service. If you don’t, SkyWalking will use the name you define on the platform (say Istio). Here, we can see that the service for the Spring Cloud application is svc-mall-admin, which is what we defined in the agent environment variable service_name.
Service Instance: Each workload in the above set of workloads is called an instance. Like pods in Kubernetes, a service instance is not necessarily a process on the operating system. But when you’re using an Agent, a service instance is actually a real process on the OS. Here, we can see that the Spring Cloud application’s service is UUUID@hostname, which is automatically generated by the Agent.
Endpoint: The path of the request received for a particular service, such as the URI path for HTTP and the class name + method signature for the gRPC service.

Here, we can see an endpoint of the Spring Cloud application for the API interface /mall-admin/admin/login.

For more information about agent parameters, please refer to https://github.com/apache/skywalking/blob/v8.1.0/docs/en/setup/service-agent/java-agent/README.md

Click the “Topology Map” menu to enter the interface to view the topology map.

Click on the “Trace” menu to access the screen for viewing link data at

6. Summary

This article details how to use Kubernetes + Spring Cloud integration SkyWalking, by the way, call chain monitoring is an essential component in the current microservice system, distributed tracking, service grid telemetry analysis, metric aggregation and visualization is still quite useful, here we chose Skywalking, the specific reasons and details The specific reasons and details of how to play are not detailed here.

Source code for this article.

https://github.com/zuozewei/blog-example/tree/master/Kubernetes/k8s-skywalking

Reference https://xie.infoq.cn/article/9e5026e2237cc0d57fafa2d57

Table of Contents