containerd/docs/tracing.md

Tracing

containerd supports OpenTelemetry tracing since v1.6.0. Tracing currently targets only gRPC calls.

Sending traces from containerd daemon

By configuring io.containerd.tracing.processor.v1.otlp plugin. containerd daemon can send traces to the specified OpenTelemetry endpoint.

version = 2

[plugins."io.containerd.tracing.processor.v1.otlp"]
    endpoint = "http://localhost:4318"

The following options are supported.

• endpoint: The address of a server that receives OpenTelemetry Protocol.
• protocol: OpenTelemetry supports multiple protocols. The default value is "http/protobuf". "grpc" is also supported.
• insecure: Disable transport security when the protocol is "grpc". The default is false. "http/protobuf" always uses the schema provided by the endpoint and the value of this setting being ignored.

The sampling ratio and the service name on the traces could be configured by io.containerd.internal.v1.tracing plugin.

version = 2

[plugins."io.containerd.internal.v1.tracing"]
    sampling_ratio = 1.0
    service_name = "containerd"

Sending traces from containerd client

By configuring its underlying gRPC client, containerd's Go client can send traces to an OpenTelemetry endpoint.

Note that the Go client's methods and gRPC calls are not 1:1. Single method call would issue multiple gRPC calls.

func clientWithTrace() error {
	exp, err := otlptracehttp.New(ctx,
		otlptracehttp.WithEndpoint("localhost:4318"),
		otlptracehttp.WithInsecure(),
	)
	if err != nil {
		return err
	}

	res, err := resource.New(ctx, resource.WithAttributes(
		semconv.ServiceNameKey.String("CLIENT NAME"),
	))
	if err != nil {
		return err
	}

	provider := trace.NewTracerProvider(
		trace.WithSampler(trace.AlwaysSample()),
		trace.WithSpanProcessor(trace.NewSimpleSpanProcessor(exp)),
		trace.WithResource(res),
	)
	otel.SetTracerProvider(provider)
	otel.SetTextMapPropagator(propagation.TraceContext{})

    ...

    dialOpts := []grpc.DialOption{
        grpc.WithTransportCredentials(insecure.NewCredentials()),
        grpc.WithStreamInterceptor(otelgrpc.StreamClientInterceptor()),
        grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()),
    }
    client, ctx, cancel, err := commands.NewClient(context, containerd.WithDialOpts(dialOpts))
    if err != nil {
        return err
    }
    defer cancel()

    ctx, span := tracing.StartSpan(ctx, "OPERATION NAME")
    defer span.End()
    ...
}