Integrate with manual OpenTelemetry

Kubit accepts OpenTelemetry traces from the stock OTLP/HTTP exporter, so you can send spans without the Kubit SDK. This page covers the connection details, a Python setup, a Node setup and a curl check. It takes about 15 minutes. Use it when you already run an OpenTelemetry pipeline, can’t add a dependency, or work in a language Kubit has no SDK for. The Kubit SDK for Python and Kubit SDK for Node do the same with less code, and filter out spans that aren’t LLM-related before they leave your process.

Before you start

A Kubit Agent workspace and its API key in KUBIT_API_KEY
For Python: Python 3.9 or later
For Node: Node.js 18 or later and OpenTelemetry JS SDK v2

Kubit authenticates spans with an ingestion API key, read from the KUBIT_API_KEY environment variable. Each key belongs to one workspace.

Way to get a key	How
Coding agent	Run `/kubit-integrate` in Claude Code or Cursor. It creates or selects a workspace, mints a key and writes `KUBIT_API_KEY` to `.env.local` or `.env`. See Agent plugin.
Kubit app	Go to Settings > Workspace > API Keys and click Generate API Key. Add an optional Note (up to 50 characters) and click Generate.

The app shows a new key only once: This is the only time we'll show you this key. Copy it now. Copy it, or click Download for a .env file, then click I’ve Saved It. The downloaded file names the key KUBIT_OTEL_API_KEY, so rename it to KUBIT_API_KEY. Availability: API Keys shows only to the Owner and Admin roles, and only in Agent and User workspaces.

Keep the key out of source control. Put it in .env or your secrets manager, and check that .env is in .gitignore.

Prompt for your coding agent

Add Kubit tracing to this repo with the stock OpenTelemetry SDK. Follow https://learn.kubit.ai/agent-observability/integrate/manual-otel.md exactly.
If the repo already creates a TracerProvider, add a BatchSpanProcessor with the Kubit exporter to it. Don't register a second provider.
Read KUBIT_API_KEY from .env. Show me the diff before writing any file.

Connection details

Setting	Value
Endpoint	`https://otel.kubit.ai/v1/traces`, method `POST`. Use the full path, including `/v1/traces`.
Auth header	`x-api-key: <your-key>`. Kubit doesn’t read the `Authorization` header.
Protocol	OTLP/HTTP. Send `Content-Type: application/x-protobuf` from real clients. `application/json` is accepted too. gRPC isn’t supported.
Compression	gzip accepted
Max request size	10 MB
Max size of one span	1 MB. Larger spans are dropped and counted in the response’s `rejected_spans`.
Rate limit	2000 requests per second per API key, with bursts up to 4000

Set service.name, service.version and deployment.environment on the resource. Kubit maps service.version to Client: Version and deployment.environment to Client: Environment.

Python

Install the OpenTelemetry SDK and the OTLP/HTTP exporter:

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

You have no OpenTelemetry setup

Create kubit_tracing.py and import it first in your entrypoint, before any LLM client is created:

kubit_tracing.py

import os

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

exporter = OTLPSpanExporter(
    endpoint="https://otel.kubit.ai/v1/traces",
    headers={"x-api-key": os.environ["KUBIT_API_KEY"]},
)

provider = TracerProvider(
    resource=Resource.create(
        {
            "service.name": "support-agent",
            "service.version": "1.0.0",
            "deployment.environment": "production",
        }
    )
)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

main.py

import kubit_tracing  # first import, before any LLM client is created

You already have a TracerProvider

Add a second BatchSpanProcessor with the Kubit exporter to the provider you already register. Don’t create a second provider. OpenTelemetry keeps the first global provider and ignores the second.

tracing.py

import os

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor

kubit_exporter = OTLPSpanExporter(
    endpoint="https://otel.kubit.ai/v1/traces",
    headers={"x-api-key": os.environ["KUBIT_API_KEY"]},
)
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(kubit_exporter))

Run this after your app calls trace.set_tracer_provider().

Trace LLM calls

Add the OpenTelemetry GenAI instrumentor for your LLM client, and turn on message content so Polaris has text to label:

pip install opentelemetry-instrumentation-openai-v2
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

kubit_tracing.py

from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor

OpenAIInstrumentor().instrument()

Add these lines at the end of kubit_tracing.py, after trace.set_tracer_provider(provider).

Add user and session IDs

Set user.id and session.id on the root span of each request, so Kubit can group traces into sessions and join them to users:

handler.py

from opentelemetry import trace

tracer = trace.get_tracer("support-agent")

def handle_message(user_id: str, conversation_id: str, text: str) -> str:
    with tracer.start_as_current_span("invoke_agent support-agent") as span:
        span.set_attribute("gen_ai.operation.name", "invoke_agent")
        span.set_attribute("user.id", user_id)
        span.set_attribute("session.id", conversation_id)
        return run_agent(text)  # your existing agent code

Send a test span

KUBIT_API_KEY=<your-key> python -c "
import kubit_tracing
from opentelemetry import trace
with trace.get_tracer('support-agent').start_as_current_span('chat.completion') as span:
    span.set_attribute('gen_ai.operation.name', 'chat')
    span.set_attribute('gen_ai.request.model', 'gpt-4o')
trace.get_tracer_provider().force_flush()
"

force_flush() sends the span before the process exits. Without it, the span can wait in the batch and be lost.

Node

Install the OpenTelemetry JS SDK v2 packages and the OTLP/HTTP protobuf exporter:

npm install @opentelemetry/api \
  @opentelemetry/exporter-trace-otlp-proto \
  @opentelemetry/resources@^2 \
  @opentelemetry/sdk-trace-base@^2 \
  @opentelemetry/sdk-trace-node@^2

You have no OpenTelemetry setup

Create kubit-tracing.ts and import it first in your entrypoint, before any LLM client is created:

kubit-tracing.ts

import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";

const apiKey = process.env.KUBIT_API_KEY;
if (!apiKey) throw new Error("KUBIT_API_KEY is not set");

const exporter = new OTLPTraceExporter({
  url: "https://otel.kubit.ai/v1/traces",
  headers: { "x-api-key": apiKey },
});

export const tracerProvider = new NodeTracerProvider({
  resource: resourceFromAttributes({
    "service.name": "support-agent",
    "service.version": "1.0.0",
    "deployment.environment": "production",
  }),
  spanProcessors: [new BatchSpanProcessor(exporter)],
});
tracerProvider.register();

index.ts

import "./kubit-tracing"; // first import, before any LLM client is created

You already have a tracer provider

OpenTelemetry JS SDK v2 takes span processors only when the provider is created. Add the Kubit processor to the spanProcessors list of the provider or NodeSDK you already create:

instrumentation.ts

import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { NodeSDK } from "@opentelemetry/sdk-node";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";

const apiKey = process.env.KUBIT_API_KEY;
if (!apiKey) throw new Error("KUBIT_API_KEY is not set");

const kubitProcessor = new BatchSpanProcessor(
  new OTLPTraceExporter({
    url: "https://otel.kubit.ai/v1/traces",
    headers: { "x-api-key": apiKey },
  }),
);

export const sdk = new NodeSDK({
  serviceName: "support-agent",
  spanProcessors: [
    // your existing processors stay here
    kubitProcessor,
  ],
});
sdk.start();

Add user and session IDs

handler.ts

import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("support-agent");

export async function handleMessage(userId: string, conversationId: string, text: string): Promise<string> {
  return tracer.startActiveSpan("invoke_agent support-agent", async (span) => {
    span.setAttribute("gen_ai.operation.name", "invoke_agent");
    span.setAttribute("user.id", userId);
    span.setAttribute("session.id", conversationId);
    try {
      return await runAgent(text); // your existing agent code
    } finally {
      span.end();
    }
  });
}

Send a test span

send-test-span.ts

import { trace } from "@opentelemetry/api";
import { tracerProvider } from "./kubit-tracing";

trace.getTracer("support-agent").startActiveSpan("chat.completion", (span) => {
  span.setAttribute("gen_ai.operation.name", "chat");
  span.setAttribute("gen_ai.request.model", "gpt-4o");
  span.end();
});

await tracerProvider.forceFlush();

KUBIT_API_KEY=<your-key> npx tsx send-test-span.ts

Choose which spans to send

The stock exporter sends every span your provider records, including HTTP, database and framework spans from other auto-instrumentation. The Kubit SDK drops those before they leave your process. With a manual setup, limit what you send in one of these ways:

Way	How
Instrument only LLM libraries	Install GenAI instrumentors, and no HTTP or database auto-instrumentation, on the provider that sends to Kubit
Use the workspace filter	Add drop and keep rules in Data > Collection > Filtering. Sampling and masking rules are on the Sampling and Masking tabs.
Switch to the Kubit SDK	`KubitSpanProcessor` keeps only spans with a `gen_ai.*` attribute or from a known LLM library. See Kubit SDK for Python or Kubit SDK for Node.

Set the endpoint on the exporter

Pass the endpoint and header to the exporter in code, as in the examples above. The OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS environment variables apply to every OpenTelemetry exporter in the process. If another backend shares the process, setting them for Kubit sends that backend’s spans to Kubit too.

Check the connection with curl

This request sends one span as OTLP/JSON. It works from any machine, so it separates a key or network problem from a problem in your app:

NOW=$(date +%s)
curl -sS -X POST https://otel.kubit.ai/v1/traces \
  -H "Content-Type: application/json" \
  -H "x-api-key: $KUBIT_API_KEY" \
  -w "\nHTTP %{http_code}\n" \
  --data-binary @- <<EOF
{
  "resourceSpans": [{
    "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "curl-smoke-test"}}]},
    "scopeSpans": [{
      "scope": {"name": "curl"},
      "spans": [{
        "traceId": "5b8efff798038103d269b633813fc60c",
        "spanId": "eee19b7ec3c1b174",
        "name": "chat.completion",
        "kind": 2,
        "startTimeUnixNano": "${NOW}000000000",
        "endTimeUnixNano": "$((NOW + 1))000000000",
        "attributes": [
          {"key": "gen_ai.operation.name", "value": {"stringValue": "chat"}},
          {"key": "gen_ai.request.model", "value": {"stringValue": "gpt-4o"}}
        ]
      }]
    }]
  }]
}
EOF

The last line of the output is HTTP 200 when Kubit accepted the span. Run it again with a different traceId to send a second test trace.

Verify

Open Agent > Activity in the Kubit app, set Time Range to Last 1 day and click Execute. The chat.completion trace from your test span is in the list. Validate in Activity shows what to check in the trace.

If it fails

Symptom	Cause	Fix
HTTP `401` with `{"detail":"Missing x-api-key header"}`	The exporter sends no `x-api-key` header, or sends the key as `Authorization`	Pass `headers={"x-api-key": ...}` (Python) or `headers: { "x-api-key": ... }` (Node) to the exporter
HTTP `401` with another message	The key is wrong, revoked or from another environment	Generate a new key in Settings > Workspace > API Keys
HTTP `415` with `{"detail":"Only application/x-protobuf and application/json are supported"}`	The exporter sends another content type, such as gRPC	Use the OTLP/HTTP exporter: `opentelemetry-exporter-otlp-proto-http` or `@opentelemetry/exporter-trace-otlp-proto`
HTTP `404`	The endpoint is missing `/v1/traces`	Use `https://otel.kubit.ai/v1/traces` in full
HTTP `413`	The request body is over 10 MB	Lower the batch size, for example `max_export_batch_size` on `BatchSpanProcessor`
HTTP `429`	Over 2000 requests per second for this key	Raise the batch size or the delay between batches
`curl` works, your app’s spans don’t arrive	The process exits before the batch is exported, or the provider was registered twice	Flush before exit, and add the Kubit processor to your existing provider

More checks are in Is my data flowing?.

Next steps

Quickstart: check the extraction mapping and validate in Activity.
Kubit SDK for Python: the same setup with span filtering and masking built in.

Start here

Agent Observability

Product Analytics

Bring your own warehouse

Integrate with manual OpenTelemetry

Before you start

Prompt for your coding agent

Connection details

Python

You have no OpenTelemetry setup

You already have a TracerProvider

Trace LLM calls

Add user and session IDs

Send a test span

Node

You have no OpenTelemetry setup

You already have a tracer provider

Add user and session IDs

Send a test span

Choose which spans to send

Set the endpoint on the exporter

Check the connection with curl

Verify

If it fails

Next steps

​Before you start

​Prompt for your coding agent

​Connection details

​Python

​You have no OpenTelemetry setup

​You already have a TracerProvider

​Trace LLM calls

​Add user and session IDs

​Send a test span

​Node

​You have no OpenTelemetry setup

​You already have a tracer provider

​Add user and session IDs

​Send a test span

​Choose which spans to send

​Set the endpoint on the exporter

​Check the connection with curl

​Verify

​If it fails

​Next steps

Before you start

Prompt for your coding agent

Connection details

Python

You have no OpenTelemetry setup

You already have a TracerProvider

Trace LLM calls

Add user and session IDs

Send a test span

Node

You have no OpenTelemetry setup

You already have a tracer provider

Add user and session IDs

Send a test span

Choose which spans to send

Set the endpoint on the exporter

Check the connection with curl

Verify

If it fails

Next steps