> ## Documentation Index
> Fetch the complete documentation index at: https://learn.kubit.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Integrate with manual OpenTelemetry

> Send spans to Kubit with the stock OpenTelemetry SDK and OTLP/HTTP exporter in Python or Node, without installing a Kubit package.

Kubit accepts OpenTelemetry traces from the stock OTLP/HTTP exporter, so you can send spans without the Kubit SDK. This page covers the connection details, a Python setup, a Node setup and a `curl` check. It takes about 15 minutes.

Use it when you already run an OpenTelemetry pipeline, can't add a dependency, or work in a language Kubit has no SDK for. The [Kubit SDK for Python](https://learn.kubit.ai/agent-observability/integrate/sdk-python.md) and [Kubit SDK for Node](https://learn.kubit.ai/agent-observability/integrate/sdk-node.md) do the same with less code, and filter out spans that aren't LLM-related before they leave your process.

## Before you start

* [ ] A Kubit **Agent** workspace and its API key in `KUBIT_API_KEY`
* [ ] For Python: Python 3.9 or later
* [ ] For Node: Node.js 18 or later and OpenTelemetry JS SDK v2

Kubit authenticates spans with an ingestion API key, read from the `KUBIT_API_KEY` environment variable. Each key belongs to one workspace.

| Way to get a key | How |
| - | - |
| **Coding agent** | Run `/kubit-integrate` in Claude Code or Cursor. It creates or selects a workspace, mints a key and writes `KUBIT_API_KEY` to `.env.local` or `.env`. See [Agent plugin](https://learn.kubit.ai/agent-observability/integrate/agent-plugin.md). |
| **Kubit app** | Go to **Settings** > **Workspace** > **API Keys** and click **Generate API Key**. Add an optional **Note** (up to 50 characters) and click **Generate**. |

The app shows a new key only once: `This is the only time we'll show you this key. Copy it now.` Copy it, or click **Download** for a `.env` file, then click **I've Saved It**. The downloaded file names the key `KUBIT_OTEL_API_KEY`, so rename it to `KUBIT_API_KEY`.

**Availability:** **API Keys** shows only to the Owner and Admin roles, and only in **Agent** and **User** workspaces.

> [!WARNING]
>
> Keep the key out of source control. Put it in `.env` or your secrets manager, and check that `.env` is in `.gitignore`.

## Prompt for your coding agent

```text theme={null}
Add Kubit tracing to this repo with the stock OpenTelemetry SDK. Follow https://learn.kubit.ai/agent-observability/integrate/manual-otel.md exactly.
If the repo already creates a TracerProvider, add a BatchSpanProcessor with the Kubit exporter to it. Don't register a second provider.
Read KUBIT_API_KEY from .env. Show me the diff before writing any file.
```

## Connection details

| Setting | Value |
| - | - |
| **Endpoint** | `https://otel.kubit.ai/v1/traces`, method `POST`. Use the full path, including `/v1/traces`. |
| **Auth header** | `x-api-key: <your-key>`. Kubit doesn't read the `Authorization` header. |
| **Protocol** | OTLP/HTTP. Send `Content-Type: application/x-protobuf` from real clients. `application/json` is accepted too. gRPC isn't supported. |
| **Compression** | gzip accepted |
| **Max request size** | **10 MB** |
| **Max size of one span** | **1 MB.** Larger spans are dropped and counted in the response's `rejected_spans`. |
| **Rate limit** | **2000 requests per second** per API key, with bursts up to 4000 |

Set `service.name`, `service.version` and `deployment.environment` on the resource. Kubit maps `service.version` to **Client: Version** and `deployment.environment` to **Client: Environment**.

## Python

Install the OpenTelemetry SDK and the OTLP/HTTP exporter:

```bash theme={null}
pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
```

### You have no OpenTelemetry setup

Create `kubit_tracing.py` and import it first in your entrypoint, before any LLM client is created:

```python kubit_tracing.py theme={null}
import os

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

exporter = OTLPSpanExporter(
    endpoint="https://otel.kubit.ai/v1/traces",
    headers={"x-api-key": os.environ["KUBIT_API_KEY"]},
)

provider = TracerProvider(
    resource=Resource.create(
        {
            "service.name": "support-agent",
            "service.version": "1.0.0",
            "deployment.environment": "production",
        }
    )
)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
```

```python main.py theme={null}
import kubit_tracing  # first import, before any LLM client is created
```

### You already have a TracerProvider

Add a second `BatchSpanProcessor` with the Kubit exporter to the provider you already register. Don't create a second provider. OpenTelemetry keeps the first global provider and ignores the second.

```python tracing.py theme={null}
import os

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor

kubit_exporter = OTLPSpanExporter(
    endpoint="https://otel.kubit.ai/v1/traces",
    headers={"x-api-key": os.environ["KUBIT_API_KEY"]},
)
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(kubit_exporter))
```

Run this after your app calls `trace.set_tracer_provider()`.

### Trace LLM calls

Add the OpenTelemetry GenAI instrumentor for your LLM client, and turn on message content so Polaris has text to label:

```bash theme={null}
pip install opentelemetry-instrumentation-openai-v2
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
```

```python kubit_tracing.py theme={null}
from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor

OpenAIInstrumentor().instrument()
```

Add these lines at the end of `kubit_tracing.py`, after `trace.set_tracer_provider(provider)`.

### Add user and session IDs

Set `user.id` and `session.id` on the root span of each request, so Kubit can group traces into sessions and join them to users:

```python handler.py theme={null}
from opentelemetry import trace

tracer = trace.get_tracer("support-agent")

def handle_message(user_id: str, conversation_id: str, text: str) -> str:
    with tracer.start_as_current_span("invoke_agent support-agent") as span:
        span.set_attribute("gen_ai.operation.name", "invoke_agent")
        span.set_attribute("user.id", user_id)
        span.set_attribute("session.id", conversation_id)
        return run_agent(text)  # your existing agent code
```

### Send a test span

```bash theme={null}
KUBIT_API_KEY=<your-key> python -c "
import kubit_tracing
from opentelemetry import trace
with trace.get_tracer('support-agent').start_as_current_span('chat.completion') as span:
    span.set_attribute('gen_ai.operation.name', 'chat')
    span.set_attribute('gen_ai.request.model', 'gpt-4o')
trace.get_tracer_provider().force_flush()
"
```

`force_flush()` sends the span before the process exits. Without it, the span can wait in the batch and be lost.

## Node

Install the OpenTelemetry JS SDK v2 packages and the OTLP/HTTP protobuf exporter:

```bash theme={null}
npm install @opentelemetry/api \
  @opentelemetry/exporter-trace-otlp-proto \
  @opentelemetry/resources@^2 \
  @opentelemetry/sdk-trace-base@^2 \
  @opentelemetry/sdk-trace-node@^2
```

### You have no OpenTelemetry setup

Create `kubit-tracing.ts` and import it first in your entrypoint, before any LLM client is created:

```typescript kubit-tracing.ts theme={null}
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";

const apiKey = process.env.KUBIT_API_KEY;
if (!apiKey) throw new Error("KUBIT_API_KEY is not set");

const exporter = new OTLPTraceExporter({
  url: "https://otel.kubit.ai/v1/traces",
  headers: { "x-api-key": apiKey },
});

export const tracerProvider = new NodeTracerProvider({
  resource: resourceFromAttributes({
    "service.name": "support-agent",
    "service.version": "1.0.0",
    "deployment.environment": "production",
  }),
  spanProcessors: [new BatchSpanProcessor(exporter)],
});
tracerProvider.register();
```

```typescript index.ts theme={null}
import "./kubit-tracing"; // first import, before any LLM client is created
```

### You already have a tracer provider

OpenTelemetry JS SDK v2 takes span processors only when the provider is created. Add the Kubit processor to the `spanProcessors` list of the provider or `NodeSDK` you already create:

```typescript instrumentation.ts theme={null}
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { NodeSDK } from "@opentelemetry/sdk-node";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";

const apiKey = process.env.KUBIT_API_KEY;
if (!apiKey) throw new Error("KUBIT_API_KEY is not set");

const kubitProcessor = new BatchSpanProcessor(
  new OTLPTraceExporter({
    url: "https://otel.kubit.ai/v1/traces",
    headers: { "x-api-key": apiKey },
  }),
);

export const sdk = new NodeSDK({
  serviceName: "support-agent",
  spanProcessors: [
    // your existing processors stay here
    kubitProcessor,
  ],
});
sdk.start();
```

### Add user and session IDs

```typescript handler.ts theme={null}
import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("support-agent");

export async function handleMessage(userId: string, conversationId: string, text: string): Promise<string> {
  return tracer.startActiveSpan("invoke_agent support-agent", async (span) => {
    span.setAttribute("gen_ai.operation.name", "invoke_agent");
    span.setAttribute("user.id", userId);
    span.setAttribute("session.id", conversationId);
    try {
      return await runAgent(text); // your existing agent code
    } finally {
      span.end();
    }
  });
}
```

### Send a test span

```typescript send-test-span.ts theme={null}
import { trace } from "@opentelemetry/api";
import { tracerProvider } from "./kubit-tracing";

trace.getTracer("support-agent").startActiveSpan("chat.completion", (span) => {
  span.setAttribute("gen_ai.operation.name", "chat");
  span.setAttribute("gen_ai.request.model", "gpt-4o");
  span.end();
});

await tracerProvider.forceFlush();
```

```bash theme={null}
KUBIT_API_KEY=<your-key> npx tsx send-test-span.ts
```

## Choose which spans to send

The stock exporter sends every span your provider records, including HTTP, database and framework spans from other auto-instrumentation. The Kubit SDK drops those before they leave your process. With a manual setup, limit what you send in one of these ways:

| Way | How |
| - | - |
| Instrument only LLM libraries | Install GenAI instrumentors, and no HTTP or database auto-instrumentation, on the provider that sends to Kubit |
| Use the workspace filter | Add drop and keep rules in **Data** > **Collection** > **Filtering**. Sampling and masking rules are on the **Sampling** and **Masking** tabs. |
| Switch to the Kubit SDK | `KubitSpanProcessor` keeps only spans with a `gen_ai.*` attribute or from a known LLM library. See [Kubit SDK for Python](https://learn.kubit.ai/agent-observability/integrate/sdk-python.md) or [Kubit SDK for Node](https://learn.kubit.ai/agent-observability/integrate/sdk-node.md). |

## Set the endpoint on the exporter

Pass the endpoint and header to the exporter in code, as in the examples above. The `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` and `OTEL_EXPORTER_OTLP_HEADERS` environment variables apply to every OpenTelemetry exporter in the process. If another backend shares the process, setting them for Kubit sends that backend's spans to Kubit too.

## Check the connection with curl

This request sends one span as OTLP/JSON. It works from any machine, so it separates a key or network problem from a problem in your app:

```bash theme={null}
NOW=$(date +%s)
curl -sS -X POST https://otel.kubit.ai/v1/traces \
  -H "Content-Type: application/json" \
  -H "x-api-key: $KUBIT_API_KEY" \
  -w "\nHTTP %{http_code}\n" \
  --data-binary @- <<EOF
{
  "resourceSpans": [{
    "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "curl-smoke-test"}}]},
    "scopeSpans": [{
      "scope": {"name": "curl"},
      "spans": [{
        "traceId": "5b8efff798038103d269b633813fc60c",
        "spanId": "eee19b7ec3c1b174",
        "name": "chat.completion",
        "kind": 2,
        "startTimeUnixNano": "${NOW}000000000",
        "endTimeUnixNano": "$((NOW + 1))000000000",
        "attributes": [
          {"key": "gen_ai.operation.name", "value": {"stringValue": "chat"}},
          {"key": "gen_ai.request.model", "value": {"stringValue": "gpt-4o"}}
        ]
      }]
    }]
  }]
}
EOF
```

The last line of the output is `HTTP 200` when Kubit accepted the span. Run it again with a different `traceId` to send a second test trace.

## Verify

Open **Agent** > **Activity** in the Kubit app, set **Time Range** to **Last 1 day** and click **Execute**. The `chat.completion` trace from your test span is in the list. [Validate in Activity](https://learn.kubit.ai/agent-observability/get-started/quickstart.md#validate-in-activity) shows what to check in the trace.&#x20;

## If it fails

| Symptom | Cause | Fix |
| - | - | - |
| HTTP `401` with `{"detail":"Missing x-api-key header"}` | The exporter sends no `x-api-key` header, or sends the key as `Authorization` | Pass `headers={"x-api-key": ...}` (Python) or `headers: { "x-api-key": ... }` (Node) to the exporter |
| HTTP `401` with another message | The key is wrong, revoked or from another environment | Generate a new key in **Settings** > **Workspace** > **API Keys** |
| HTTP `415` with `{"detail":"Only application/x-protobuf and application/json are supported"}` | The exporter sends another content type, such as gRPC | Use the OTLP/HTTP exporter: `opentelemetry-exporter-otlp-proto-http` or `@opentelemetry/exporter-trace-otlp-proto` |
| HTTP `404` | The endpoint is missing `/v1/traces` | Use `https://otel.kubit.ai/v1/traces` in full |
| HTTP `413` | The request body is over 10 MB | Lower the batch size, for example `max_export_batch_size` on `BatchSpanProcessor` |
| HTTP `429` | Over 2000 requests per second for this key | Raise the batch size or the delay between batches |
| `curl` works, your app's spans don't arrive | The process exits before the batch is exported, or the provider was registered twice | Flush before exit, and add the Kubit processor to your existing provider |

More checks are in [Is my data flowing?](https://learn.kubit.ai/agent-observability/get-started/data-flowing.md).

## Next steps

* [Quickstart](https://learn.kubit.ai/agent-observability/get-started/quickstart.md#check-the-extraction-mapping): check the extraction mapping and validate in Activity.
* [Kubit SDK for Python](https://learn.kubit.ai/agent-observability/integrate/sdk-python.md): the same setup with span filtering and masking built in.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.