Update tracing documentation to add details about manual instrumentation

Signed-off-by: Swagat Bora <sbora@amazon.com>

docs/tracing.md

@@ -85,3 +85,102 @@ func clientWithTrace() error {
     defer span.End()
     ...
 }
+```
+## Manual instrumentation
+
+OpenTelemetry provides language specific [API](https://pkg.go.dev/go.opentelemetry.io/otel) libraries to instrument parts of your application that are not covered by automatic instrumentation.
+
+In Containerd, a thin wrapper library defined in `tracing/tracing.go` provides additional functionality and makes it easier to use the OpenTelemetry API for manual instrumentation.
+
+### Creating a new span
+
+To create a new span, use the `tracing.StartSpan()` method. You should already have a global TracerProvider set by configuring `io.containerd.tracing.processor.v1.otlp` plugin, else this will only create an instance of a NoopSpan{}.
+
+```
+func CreateContainer(ctx context.Context, r *runtime.CreateContainerRequest) error {
+    ctx, span := tracing.StartSpan(ctx,
+        tracing.Name(criSpanPrefix, "CreateContainer") // name of the span
+        tracing.WithAttribute("sandbox.id",r.GetPodSandboxId(), //attributes to be added to the span
+        )
+	defer span.End() // end the span once the function returns
+    ...
+}
+```
+Mark the span complete at the end of workflow by calling `Span.End()`. In the above example, we use 'defer' to ensure that the span is properly closed and its duration is recorded.
+
+### Adding Attributes to a span
+
+You can add additional attributes to the span using `Span.SetAttributes()`. Attributes can be added during span creation (by passing `tracing.WithAttribute()` to tracing.StartSpan()) or at any other time during the lifecycle of a span before it has completed.
+
+```
+func CreateContainer(ctx context.Context, r *runtime.CreateContainerRequest) error {
+    ctx, span := tracing.StartSpan(ctx,
+        tracing.Name(criSpanPrefix, "CreateContainer")
+        tracing.WithAttribute("sandbox.id",r.GetPodSandboxId(),
+        )
+	defer span.End()
+    ...
+    containerId := util.GenerateID()
+    containerName := makeContainerName(metadata, sandboxConfig.GetMetadata())
+
+    //Add new attributes to the existing span
+    span.SetAttributes(
+		tracing.Attribute("container.id", containerId),
+		tracing.Attribute("container.name", containerName),
+	)
+    ...
+}
+```
+### Adding an Event to a span
+Use `Span.AddEvent()` to add an event to an existing span. A [span event](https://opentelemetry.io/docs/instrumentation/go/manual/#events) is a specific occurrence within a span, such as the completion of an operation or the occurrence of an error. Span events can be used to provide additional information about the operation represented by the span, and can be used for debugging or performance analysis.
+
+The below example shows how we can add an event to the span to mark the execution of an NRI hook.
+```
+func CreateContainer(ctx context.Context, r *runtime.CreateContainerRequest) error {
+    span := tracing.SpanFromContext(ctx) // get the current span from context
+    ...
+    ...
+    if c.nri.isEnabled() {
+        // Add an event to mark start of an NRI api call
+        span.AddEvent("start NRI postCreateContainer request")
+
+        err = c.nri.postCreateContainer(ctx, &sandbox, &container)
+        if err != nil {
+            span.RecordError("NRI postCreateContainer request failed") //record error
+			log.G(ctx).WithError(err).Errorf("NRI post-create notification failed")
+		}
+
+        // Add an event to mark completion of the request
+        span.AddEvent("finished NRI postCreateContainer request")
+	}
+    ...
+    // You can also add additional attributes to an event
+    span.AddEvent("container created",
+		tracing.Attribute("container.create.duration", time.Since(start).String()),
+	)
+
+	return &runtime.CreateContainerResponse{ContainerId: id}, nil
+}
+```
+## Naming Convention
+
+OpenTelemetry maintains a set of recommended [semantic conventions](https://opentelemetry.io/docs/reference/specification/overview/#semantic-conventions) for different types of telemetry data, such as traces and metrics, to help users of the OpenTelemetry libraries and tools to collect and use telemetry data in a consistent and interoperable way.
+
+Manually instrumented spans in Containerd follow the conventions defined for [Spans](https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/) and [Attributes](https://opentelemetry.io/docs/reference/specification/common/attribute-naming/)
+
+### Span Names
+* Dot-separated notation.
+* Span Names may include relative path to the package.
+* Span Names should include a name that represents the specific component or service performing the operation.
+* For example: "pkg.cri.sbserver.CreateContainer"
+   * "pkg.cri.sbserver" - relative path to the package
+   * "CreateContainer" - describes the operation that is traced
+
+### Attribute Names
+* Lower-case.
+* Dot-separated notation.
+* Use a namespace based representation.
+* For example: "http.method.get" , http.method.post".
+   * "http" - represents the general category of the attribute.
+   * "method" - specific aspect or property of the attribute.
+   * "get" - additional detail or context.