Kubit SDK for Python

kubit-otel sends OpenTelemetry spans from a Python app to Kubit. It has two entry points: configure() when Kubit is your only exporter, and attach() when your app already creates a TracerProvider. This page covers both, plus the settings to check before production.

Before you start

Python 3.9 or later
OpenTelemetry Python SDK 1.20 or later. kubit-otel installs opentelemetry-sdk>=1.20.0 and opentelemetry-exporter-otlp-proto-http>=1.20.0,<2 if they’re missing.
A Kubit API key in KUBIT_API_KEY

Kubit authenticates spans with an ingestion API key, read from the KUBIT_API_KEY environment variable. Each key belongs to one workspace.

Way to get a key	How
Coding agent	Run `/kubit-integrate` in Claude Code or Cursor. It creates or selects a workspace, mints a key and writes `KUBIT_API_KEY` to `.env.local` or `.env`. See Agent plugin.
Kubit app	Go to Settings > Workspace > API Keys and click Generate API Key. Add an optional Note (up to 50 characters) and click Generate.

The app shows a new key only once: This is the only time we'll show you this key. Copy it now. Copy it, or click Download for a .env file, then click I’ve Saved It. The downloaded file names the key KUBIT_OTEL_API_KEY, so rename it to KUBIT_API_KEY. Availability: API Keys shows only to the Owner and Admin roles, and only in Agent and User workspaces.

Keep the key out of source control. Put it in .env or your secrets manager, and check that .env is in .gitignore.

Prompt for your coding agent

Add Kubit tracing to this Python repo. Follow https://learn.kubit.ai/agent-observability/integrate/sdk-python.md exactly.
If the repo already creates a TracerProvider, use attach(). Otherwise use configure().
Read KUBIT_API_KEY from .env. Show me the diff before writing any file.

Install

pip install kubit-otel

uv add kubit-otel

poetry add kubit-otel

Choose configure() or attach()

Your app today	Use	Why
No OpenTelemetry setup	`configure()`	Creates a `TracerProvider` with the Kubit exporter and registers it as the global provider
Creates its own `TracerProvider` and calls `trace.set_tracer_provider()`	`attach()`	Adds `KubitSpanProcessor` to your provider. No second provider, and your existing exporters keep working.
Sends traces to Langfuse or Braintrust	`attach()`	Same as above. The Agent plugin handles the sink-specific wiring.

If your app sets its own global TracerProvider, call configure() only after trace.set_tracer_provider(), or use attach(). Called after, configure() adds KubitSpanProcessor to your provider and merges its resource attributes into the provider’s resource. Kubit’s values win on a key collision, so pass service_name, or your provider’s service.name becomes default. Called before, configure() registers its own provider first. OpenTelemetry then ignores your app’s set_tracer_provider() call, and your existing exporters stop receiving spans.

You have no OpenTelemetry setup

Create kubit_instrumentation.py and import it first in your entrypoint.

kubit_instrumentation.py

import os
from kubit_otel import configure

configure(
    api_key=os.environ["KUBIT_API_KEY"],
    service_name="support-agent",
    service_version="1.0.0",
    resource_attributes={"deployment.environment": "production"},
)

main.py

import kubit_instrumentation  # first import, before any LLM client is created

configure() parameters

Parameter	Required?	Default	Notes
api_key	Yes		The workspace ingestion key. Read it from `KUBIT_API_KEY`.
service_name	Optional	`default`	Becomes the OpenTelemetry `service.name`. Set it. Kubit uses it to tell agents apart.
service_version	Optional	Not set	Becomes `service.version`. Set it to your release or git SHA so you can compare versions.
resource_attributes	Optional	Not set	A dict merged into the OpenTelemetry resource, e.g. `{"deployment.environment": "production"}`
endpoint	Optional	`https://otel.kubit.ai/v1/traces`	Overrides the trace endpoint. The `KUBIT_OTEL_ENDPOINT` environment variable does the same. The SDK doesn’t read `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`.
should_export_span	Optional	LLM spans only	A function that takes a span and returns `True` to send it to Kubit
mask	Optional	Not set	A function that redacts a span before export. See Mask sensitive content. Needs `kubit-otel` 0.7.3 or later.

configure() has no batch or sampling parameters. It sends spans through KubitSpanProcessor with its default batch settings: a queue of 2048 spans, a batch every 5000 ms, up to 512 spans per export and a 30000 ms export timeout.

You already have a TracerProvider

Call attach() after trace.set_tracer_provider(), in the module that creates the provider:

tracing.py

import os
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from kubit_otel import attach

provider = TracerProvider(resource=Resource.create({"service.name": "support-agent"}))
# your existing span processors and exporters stay here
trace.set_tracer_provider(provider)

attach(api_key=os.environ["KUBIT_API_KEY"])

attach() finds the global provider and adds KubitSpanProcessor to it. Your provider’s resource (service.name, service.version) applies to the spans Kubit receives.

attach() parameters

Parameter	Required?	Notes
api_key	Yes	The workspace ingestion key
endpoint	Optional	Same as in `configure()`
should_export_span	Optional	Same as in `configure()`
mask	Optional	Same as in `configure()`. Needs `kubit-otel` 0.7.3 or later.

Availability: attach() must run after trace.set_tracer_provider(). Called earlier, it raises RuntimeError and adds nothing to any provider.

Trace LLM and tool calls

KubitSpanProcessor sends only LLM spans by default: spans from the kubit-sdk tracer, spans with any gen_ai.* attribute and spans from a known LLM instrumentation scope. To get one span per LLM call, add the OpenTelemetry GenAI instrumentor for your client after configure() or attach():

kubit_instrumentation.py

import os
from kubit_otel import configure
from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor

configure(api_key=os.environ["KUBIT_API_KEY"], service_name="support-agent")
OpenAIInstrumentor().instrument()

pip install opentelemetry-instrumentation-openai-v2
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true puts prompt and response text on the spans. Polaris labels spans from that text. For tool calls, wrap each tool in its own span so it shows up as a step in the execution path:

tools.py

from opentelemetry import trace

tracer = trace.get_tracer("support-agent")

def search_orders(customer_id: str, query: str) -> list[dict]:
    with tracer.start_as_current_span("execute_tool search_orders") as span:
        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", "search_orders")
        results = run_order_search(customer_id, query)  # your existing tool code
        span.set_attribute("search.result_count", len(results))
        return results

Add user and session IDs

Kubit groups traces into sessions and joins them to users from IDs on your spans. Set user.id and session.id on the root span of each request. The gen_ai.operation.name attribute keeps your own span past the default filter, which drops spans with no gen_ai.* attribute:

handler.py

from opentelemetry import trace

tracer = trace.get_tracer("support-agent")

def handle_message(user_id: str, conversation_id: str, text: str) -> str:
    with tracer.start_as_current_span("invoke_agent support-agent") as span:
        span.set_attribute("gen_ai.operation.name", "invoke_agent")
        span.set_attribute("user.id", user_id)
        span.set_attribute("session.id", conversation_id)
        return run_agent(text)  # your existing agent code

Use the same user ID your product analytics events use, so agent runs join to user behavior. The default extraction mapping reads enduser.id or user.id for the user, and session.id or gen_ai.conversation.id for the session. To read other keys, edit the mapping as described in Check the extraction mapping.

Mask sensitive content

The mask option redacts span content before the span leaves your process. KubitSpanProcessor runs your function on each span that passes the filter, before the span is queued for export. mask needs kubit-otel 0.7.3 or later and works with configure(), attach() and KubitSpanProcessor.

kubit_instrumentation.py

import os
import re
from kubit_otel import configure
from kubit_otel.mask import mask_events, set_attr

CARD_RE = re.compile(r"\b(?:\d[ -]*?){13,19}\b")

def mask(span):
    prompt = (span.attributes or {}).get("gen_ai.prompt")
    if isinstance(prompt, str):
        set_attr(span, "gen_ai.prompt", CARD_RE.sub("[REDACTED CC]", prompt))
    mask_events(span, lambda e: None if e.name == "gen_ai.user.message" else e)
    return span

configure(api_key=os.environ["KUBIT_API_KEY"], service_name="support-agent", mask=mask)

This mask replaces card numbers in gen_ai.prompt and drops gen_ai.user.message events, where newer GenAI instrumentors record the prompt. The helpers in kubit_otel.mask are the supported way to change a span inside mask:

Helper	What it does
`set_attr(target, key, value)`	Adds or overwrites an attribute on a span or an event
`delete_attr(target, key)`	Removes an attribute from a span or an event. Does nothing if the key is absent.
`mask_events(span, fn)`	Calls `fn` on each event. Keeps the event `fn` returns and drops the event when `fn` returns `None`.

Rule	Detail
Synchronous	The processor doesn’t await coroutines
No I/O or network calls	The function runs on the thread that ends the span
Returns a span	Usually the same span, changed with the helpers
Runs on every span that passes the filter	Child tool spans, retries and LangGraph or Mastra spans included. Handle every span shape, or limit your logic by `span.name` or `span.instrumentation_scope.name`.
Changes spans, doesn’t drop them	To drop a span, use `should_export_span`

Constraint: When mask raises or returns None, the SDK sends a tombstone in place of the span. The tombstone keeps the trace and span IDs, parent, timing and kind. Its attributes and events are removed, its status is ERROR with the description kubit-otel mask failed, and its kubit.sdk.mask_error attribute holds the exception class name or returned_none. Unmasked data is never sent. The SDK logs one of these lines, with the traceback for a raised exception:

Output

[kubit-otel error] mask raised; tombstoning span  span_name=<name> span_id=<id> err=<message>
[kubit-otel error] mask returned None; tombstoning span (use should_export_span to filter)  span_name=<name> span_id=<id>

Constraint: A KubitExporter that you wrap in your own span processor applies neither mask nor the default filter. Apply both in your processor.

Before production

Check	Why
`service_version` set to your release	Reports can group by Client: Version to compare releases
`deployment.environment` set	Keeps dev and production traces apart
Short-lived jobs call `trace.get_tracer_provider().force_flush()` before exit	Spans still in the batch are lost when the process exits
User and session IDs on the root span	Without them Kubit can’t group traces into sessions or join them to users. See Add user and session IDs.
Sensitive fields masked before export	Message content goes to Kubit for labeling. See Mask sensitive content.

Verify

Send a test span with debug logging on:

KUBIT_API_KEY=<your-key> KUBIT_OTEL_LOG_LEVEL=debug python -c "
import kubit_instrumentation
from opentelemetry import trace
trace.get_tracer('kubit-sdk').start_span('hello-kubit').end()
trace.get_tracer_provider().force_flush()
"

force_flush() sends the span now. Without it, the span waits up to 5000 ms in the batch. With the configure() file from You have no OpenTelemetry setup, the output on stderr includes these lines, among other debug lines:

Output

[kubit-otel info] kubit_otel configured  mode=registered service_name=support-agent service_version=1.0.0 endpoint=<from OTEL_EXPORTER_OTLP_*>
[kubit-otel debug] Exported batch to kubit  span_count=1

endpoint=<from OTEL_EXPORTER_OTLP_*> is the label kubit-otel prints when you don’t pass endpoint. The SDK still sends to KUBIT_OTEL_ENDPOINT or https://otel.kubit.ai/v1/traces. With attach(), the first line is [kubit-otel info] kubit_otel attached endpoint=<from OTEL_EXPORTER_OTLP_*>. KUBIT_OTEL_LOG_LEVEL takes debug, info, warn or error, and the default is info. Then open Agent > Activity, set Time Range to Last 1 day and click Execute. The hello-kubit trace is in the list. Validate in Activity shows what to check in the trace.

If it fails

Symptom	Cause	Fix
`KeyError: 'KUBIT_API_KEY'`	Env var not set, or `.env` not loaded yet	Export it, or call `load_dotenv()` before `import kubit_instrumentation`
`RuntimeError: kubit_otel.attach() requires a TracerProvider already registered as the global OTel provider. Call kubit_otel.configure() instead, or install a provider first (e.g. via Langfuse or opentelemetry-sdk).`	`attach()` ran before `set_tracer_provider()`	Move `attach()` after `trace.set_tracer_provider(provider)`
Spans reach Kubit but not your old exporter, and OpenTelemetry logs `Overriding of current TracerProvider is not allowed`	`configure()` ran before your app’s `trace.set_tracer_provider()`, so your app’s provider was never registered	Move `configure()` after `trace.set_tracer_provider(provider)`, or replace it with `attach()` there
LLM spans have no prompt or response text	Content capture is off	`export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true`
No spans from a cron job or script	Process exits before the batch flushes	Call `trace.get_tracer_provider().force_flush()` at the end
`[kubit-otel warn] KubitExporter export failed span_count=...`	The export request failed. A bad or revoked API key and a network failure log the same line, because it doesn’t include the HTTP status. `kubit-otel` 0.7.2 and later log it.	Check that `KUBIT_API_KEY` holds a current key for this workspace, then work through Is my data flowing?
`[kubit-otel error] mask raised; tombstoning span span_name=...`	Your `mask` function raised an exception	Fix the function using the traceback in the log. See Mask sensitive content.

Next steps

Quickstart: check the extraction mapping and validate in Activity.
Is my data flowing?: step-by-step checks.

Start here

Agent Observability

Product Analytics

Bring your own warehouse

Kubit SDK for Python

Before you start

Prompt for your coding agent

Install

Choose configure() or attach()

You have no OpenTelemetry setup

configure() parameters

You already have a TracerProvider

attach() parameters

Trace LLM and tool calls

Add user and session IDs

Mask sensitive content

Before production

Verify

If it fails

Next steps

​Before you start

​Prompt for your coding agent

​Install

​Choose configure() or attach()

​You have no OpenTelemetry setup

​configure() parameters

​You already have a TracerProvider

​attach() parameters

​Trace LLM and tool calls

​Add user and session IDs

​Mask sensitive content

​Before production

​Verify

​If it fails

​Next steps

Before you start

Prompt for your coding agent

Install

Choose configure() or attach()

You have no OpenTelemetry setup

configure() parameters

You already have a TracerProvider

attach() parameters

Trace LLM and tool calls

Add user and session IDs

Mask sensitive content

Before production

Verify

If it fails

Next steps