分布式追踪之opentelemetry

Opentelemetry介绍

OpenTelemetry（OpenTelemetry官方文档）统一数据采集和标准规范，目前推荐使用Prometheus + Grafana做Metrics存储、展示，使用Jaeger（Jaeger官方文档）做分布式跟踪的存储和展示

选自他人博客：Opentelemetry Collector的配置和使用

操作环境：k8s v1.16.15

kubectl部署

官方博客

阿里云帮助中心-安装OpenTelemetry探针

部署opentelemetry-operator

https://github.com/open-telemetry/opentelemetry-operator

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yaml

wget https://github.com/open-telemetry/opentelemetry-operator/releases/download/v0.43.0/opentelemetry-operator.yaml

# 我这里k8s版本为v1.16.15，所以admissionReviewVersions,修改v1 => v1beta1

kubectl apply -f opentelemetry-operator.yaml

docker pull kubesphere/kube-rbac-proxy:v0.8.0
docker tag kubesphere/kube-rbac-proxy:v0.8.0 gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0

部署opentelemetry-collector

官方文档-Collector配置

contrib配置

vim collector.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: opentelemetry
---
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: opentelemetry
  namespace: opentelemetry
spec:
  image: otel/opentelemetry-collector-contrib:latest
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      # batch:

    exporters:
      logging:
      jaeger:
        endpoint: jaeger-collector.observability:14250
        tls:
          insecure: true
      prometheus:
        endpoint: "localhost:8889"
        namespace: "default"

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [jaeger]
        logs:
          receivers: [otlp]
          exporters: [logging]
        metrics:
          receivers: [otlp]
          exporters: [prometheus]

kubectl apply -f collector.yaml

部署Auto-Instrumentation

OpenTelemetry Java Automatic Instrumentation

Agent配置（Manual Instrumentation参考SDK配置）

支持的组件库

vim instrumentation.yaml

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://otel-collector.observability:4317
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: 1
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest

kubectl apply -f instrumentation.yaml

Instrumentation注入

Java

my-instrumentation和需要注入的服务需要在统一命名空间，然后使用以下注解进行准入控制

instrumentation.opentelemetry.io/inject-java: 'true'

• “true” - Instrumentation-从命名空间注入和资源。
• “my-instrumentation”- InstrumentationCR 实例的名称。
• “false”- 不要注入

kubectl patch deployment xxx -p ‘{“spec”: {“template”: {“metadata”: {“annotations”: {“instrumentation.opentelemetry.io/inject-java”: “true”}}}}}’

Python

Github-Docs

instrumentation.opentelemetry.io/inject-python: 'true'

https://opentelemetry-python.readthedocs.io/en/latest/index.html

https://opentelemetry-python-contrib.readthedocs.io/en/latest/index.html

部署jaeger-operator

https://github.com/jaegertracing/jaeger-operator

kubectl create namespace observability
kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.31.0/jaeger-operator.yaml -n observability

部署jaeger

https://www.jaegertracing.io/docs/1.18/opentelemetry/

https://github.com/jaegertracing/jaeger-operator/tree/main/examples

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
  namespace: observability
spec:
  ingress:
#    enabled: false
    enabled: true
    hosts:
      - jaeger.xxx.com
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch-master.istio-system:9200
  query:
#    serviceType: NodePort
    options:
      max-clock-skew-adjustment: '0s'
  sampling:
    options:
      default_strategy:
        type: probabilistic
        param: 0.5

Helm部署Grafana loki

[1]利用Opentelemetry+Loki+Temp+Granafa构建端到端的可观测平台：

https://juejin.cn/post/7050134410229710884

[2]我的使用版本：0.45.0

https://github.com/grafana/helm-charts/releases/download/loki-distributed-0.45.0/loki-distributed-0.45.0.tgz

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install -n loki loki grafana/loki-stack --set grafana.enabled=false,prometheus.enabled=false,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false,loki.persistence.enabled=true,loki.persistence.storageClassName=loki-db,loki.persistence.size=10Gi

https://grafana.com/docs/tempo/latest/getting-started/

Helm部署Tempo（可选）

• single binary
• microservices

https://github.com/grafana/tempo/tree/main/example/helm

https://www.cnblogs.com/centos-python/articles/14743546.html?ivk_sa=1024320u

• datasource
• loki
• Derived fields

TraceID
(?:trace_id)=(\w+)
${__value.raw}

Helm 安装

opentelemety-operator

admissionWebhooks安装有问题，待研究

https://github.com/open-telemetry/opentelemetry-helm-charts

value.yaml参考：

https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-operator

参考网上教程生成自签名证书
域名：
opentelemetry-operator-webhook-service.istio-system.svc

kubectl create secret tls opentelemetry-operator-controller-manager-service-cert \
    --cert=opentelemetry-operator-webhook-service.istio-system.svc/opentelemetry-operator-webhook-service.istio-system.svc.crt \
    --key=opentelemetry-operator-webhook-service.istio-system.svc/opentelemetry-operator-webhook-service.istio-system.svc.key \
    -n istio-system

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

git clone https://github.com/open-telemetry/opentelemetry-helm-charts
# 我这里k8s版本为v1.16.15，所以admissionReviewVersions,修改v1 => v1beta1

helm install opentelemetry-operator opentelemetry-helm-charts/charts/opentelemetry-operator -n istio-system --set manager.image.tag=v0.43.0 --set kubeRBACProxy.image.repository=kubesphere/kube-rbac-proxy --set admissionWebhooks.certManager.enabled=false 

kubectl get MutatingWebhookConfiguration opentelemetry-operator-mutating-webhook-configuration

opentelemety-collector

https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector

kubectl create ns observability

helm install opentelemetry-collector open-telemetry/opentelemetry-collector -n observability -f my-values.yaml --set image.tag=0.46.0

jaeger-operator

https://github.com/jaegertracing/helm-charts

value.yaml参考：

https://github.com/jaegertracing/helm-charts/tree/main/charts/jaeger-operator

k8s v1.16 使用jaeger-operator-2.22.0

helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm repo update

helm install jaeger-operator jaegertracing/jaeger-operator -n istio-system --set image.tag=1.22.0

jaeger

https://github.com/jaegertracing/helm-charts/tree/main/charts/jaeger

helm install jaeger jaegertracing/jaeger -n observability --set provisionDataStore.cassandra=false --set storage.type=elasticsearch  --set storage.elasticsearch.host=elasticsearch-master.istio-system --set storage.elasticsearch.port=9200 --values my-values.yaml