> ## Documentation Index > Fetch the complete documentation index at: https://learn.kubit.ai/llms.txt > Use this file to discover all available pages before exploring further. # Kubit SDK for Python > Send OpenTelemetry spans from a Python agent to Kubit with the kubit-otel SDK, as the only exporter or next to an existing TracerProvider. `kubit-otel` sends OpenTelemetry spans from a Python app to Kubit. It has two entry points: `configure()` when Kubit is your only exporter, and `attach()` when your app already creates a `TracerProvider`. This page covers both, plus the settings to check before production. ## Before you start * [ ] Python 3.9 or later * [ ] OpenTelemetry Python SDK 1.20 or later. `kubit-otel` installs `opentelemetry-sdk>=1.20.0` and `opentelemetry-exporter-otlp-proto-http>=1.20.0,<2` if they're missing. * [ ] A Kubit API key in `KUBIT_API_KEY` Kubit authenticates spans with an ingestion API key, read from the `KUBIT_API_KEY` environment variable. Each key belongs to one workspace. | Way to get a key | How | | - | - | | **Coding agent** | Run `/kubit-integrate` in Claude Code or Cursor. It creates or selects a workspace, mints a key and writes `KUBIT_API_KEY` to `.env.local` or `.env`. See [Agent plugin](https://learn.kubit.ai/agent-observability/integrate/agent-plugin.md). | | **Kubit app** | Go to **Settings** > **Workspace** > **API Keys** and click **Generate API Key**. Add an optional **Note** (up to 50 characters) and click **Generate**. | The app shows a new key only once: `This is the only time we'll show you this key. Copy it now.` Copy it, or click **Download** for a `.env` file, then click **I've Saved It**. The downloaded file names the key `KUBIT_OTEL_API_KEY`, so rename it to `KUBIT_API_KEY`. **Availability:** **API Keys** shows only to the Owner and Admin roles, and only in **Agent** and **User** workspaces. > [!WARNING] > > Keep the key out of source control. Put it in `.env` or your secrets manager, and check that `.env` is in `.gitignore`. ## Prompt for your coding agent ```text theme={null} Add Kubit tracing to this Python repo. Follow https://learn.kubit.ai/agent-observability/integrate/sdk-python.md exactly. If the repo already creates a TracerProvider, use attach(). Otherwise use configure(). Read KUBIT_API_KEY from .env. Show me the diff before writing any file. ``` ## Install ```bash pip theme={null} pip install kubit-otel ``` ```bash uv theme={null} uv add kubit-otel ``` ```bash poetry theme={null} poetry add kubit-otel ``` ## Choose configure() or attach() | Your app today | Use | Why | | - | - | - | | No OpenTelemetry setup | `configure()` | Creates a `TracerProvider` with the Kubit exporter and registers it as the global provider | | Creates its own `TracerProvider` and calls `trace.set_tracer_provider()` | `attach()` | Adds `KubitSpanProcessor` to your provider. No second provider, and your existing exporters keep working. | | Sends traces to Langfuse or Braintrust | `attach()` | Same as above. The [Agent plugin](https://learn.kubit.ai/agent-observability/integrate/agent-plugin.md) handles the sink-specific wiring. | > [!WARNING] > > If your app sets its own global `TracerProvider`, call `configure()` only after `trace.set_tracer_provider()`, or use `attach()`. Called after, `configure()` adds `KubitSpanProcessor` to your provider and merges its resource attributes into the provider's resource. Kubit's values win on a key collision, so pass `service_name`, or your provider's `service.name` becomes `default`. Called before, `configure()` registers its own provider first. OpenTelemetry then ignores your app's `set_tracer_provider()` call, and your existing exporters stop receiving spans. ## You have no OpenTelemetry setup Create `kubit_instrumentation.py` and import it first in your entrypoint. ```python kubit_instrumentation.py theme={null} import os from kubit_otel import configure configure( api_key=os.environ["KUBIT_API_KEY"], service_name="support-agent", service_version="1.0.0", resource_attributes={"deployment.environment": "production"}, ) ``` ```python main.py theme={null} import kubit_instrumentation # first import, before any LLM client is created ``` ### configure() parameters | Parameter | Required? | Default | Notes | | - | - | - | - | | **api\_key** | **Yes** | | The workspace ingestion key. Read it from `KUBIT_API_KEY`. | | **service\_name** | Optional | `default` | Becomes the OpenTelemetry `service.name`. Set it. Kubit uses it to tell agents apart. | | **service\_version** | Optional | Not set | Becomes `service.version`. Set it to your release or git SHA so you can compare versions. | | **resource\_attributes** | Optional | Not set | A dict merged into the OpenTelemetry resource, e.g. `{"deployment.environment": "production"}` | | **endpoint** | Optional | `https://otel.kubit.ai/v1/traces` | Overrides the trace endpoint. The `KUBIT_OTEL_ENDPOINT` environment variable does the same. The SDK doesn't read `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`. | | **should\_export\_span** | Optional | LLM spans only | A function that takes a span and returns `True` to send it to Kubit | | **mask** | Optional | Not set | A function that redacts a span before export. See [Mask sensitive content](#mask-sensitive-content). Needs `kubit-otel` 0.7.3 or later. | `configure()` has no batch or sampling parameters. It sends spans through `KubitSpanProcessor` with its default batch settings: a queue of 2048 spans, a batch every 5000 ms, up to 512 spans per export and a 30000 ms export timeout. ## You already have a TracerProvider Call `attach()` after `trace.set_tracer_provider()`, in the module that creates the provider: ```python tracing.py theme={null} import os from opentelemetry import trace from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from kubit_otel import attach provider = TracerProvider(resource=Resource.create({"service.name": "support-agent"})) # your existing span processors and exporters stay here trace.set_tracer_provider(provider) attach(api_key=os.environ["KUBIT_API_KEY"]) ``` `attach()` finds the global provider and adds `KubitSpanProcessor` to it. Your provider's resource (`service.name`, `service.version`) applies to the spans Kubit receives. ### attach() parameters | Parameter | Required? | Notes | | - | - | - | | **api\_key** | **Yes** | The workspace ingestion key | | **endpoint** | Optional | Same as in `configure()` | | **should\_export\_span** | Optional | Same as in `configure()` | | **mask** | Optional | Same as in `configure()`. Needs `kubit-otel` 0.7.3 or later. | **Availability:** `attach()` must run after `trace.set_tracer_provider()`. Called earlier, it raises `RuntimeError` and adds nothing to any provider. ## Trace LLM and tool calls `KubitSpanProcessor` sends only LLM spans by default: spans from the `kubit-sdk` tracer, spans with any `gen_ai.*` attribute and spans from a known LLM instrumentation scope. To get one span per LLM call, add the OpenTelemetry GenAI instrumentor for your client after `configure()` or `attach()`: ```python kubit_instrumentation.py theme={null} import os from kubit_otel import configure from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor configure(api_key=os.environ["KUBIT_API_KEY"], service_name="support-agent") OpenAIInstrumentor().instrument() ``` ```bash theme={null} pip install opentelemetry-instrumentation-openai-v2 export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true ``` `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true` puts prompt and response text on the spans. Polaris labels spans from that text. For tool calls, wrap each tool in its own span so it shows up as a step in the execution path: ```python tools.py theme={null} from opentelemetry import trace tracer = trace.get_tracer("support-agent") def search_orders(customer_id: str, query: str) -> list[dict]: with tracer.start_as_current_span("execute_tool search_orders") as span: span.set_attribute("gen_ai.operation.name", "execute_tool") span.set_attribute("gen_ai.tool.name", "search_orders") results = run_order_search(customer_id, query) # your existing tool code span.set_attribute("search.result_count", len(results)) return results ``` ## Add user and session IDs Kubit groups traces into sessions and joins them to users from IDs on your spans. Set `user.id` and `session.id` on the root span of each request. The `gen_ai.operation.name` attribute keeps your own span past the default filter, which drops spans with no `gen_ai.*` attribute: ```python handler.py theme={null} from opentelemetry import trace tracer = trace.get_tracer("support-agent") def handle_message(user_id: str, conversation_id: str, text: str) -> str: with tracer.start_as_current_span("invoke_agent support-agent") as span: span.set_attribute("gen_ai.operation.name", "invoke_agent") span.set_attribute("user.id", user_id) span.set_attribute("session.id", conversation_id) return run_agent(text) # your existing agent code ``` Use the same user ID your product analytics events use, so agent runs join to user behavior. The default extraction mapping reads `enduser.id` or `user.id` for the user, and `session.id` or `gen_ai.conversation.id` for the session. To read other keys, edit the mapping as described in [Check the extraction mapping](https://learn.kubit.ai/agent-observability/get-started/quickstart.md#check-the-extraction-mapping). ## Mask sensitive content The `mask` option redacts span content before the span leaves your process. `KubitSpanProcessor` runs your function on each span that passes the filter, before the span is queued for export. `mask` needs `kubit-otel` 0.7.3 or later and works with `configure()`, `attach()` and `KubitSpanProcessor`. ```python kubit_instrumentation.py theme={null} import os import re from kubit_otel import configure from kubit_otel.mask import mask_events, set_attr CARD_RE = re.compile(r"\b(?:\d[ -]*?){13,19}\b") def mask(span): prompt = (span.attributes or {}).get("gen_ai.prompt") if isinstance(prompt, str): set_attr(span, "gen_ai.prompt", CARD_RE.sub("[REDACTED CC]", prompt)) mask_events(span, lambda e: None if e.name == "gen_ai.user.message" else e) return span configure(api_key=os.environ["KUBIT_API_KEY"], service_name="support-agent", mask=mask) ``` This mask replaces card numbers in `gen_ai.prompt` and drops `gen_ai.user.message` events, where newer GenAI instrumentors record the prompt. The helpers in `kubit_otel.mask` are the supported way to change a span inside `mask`: | Helper | What it does | | - | - | | `set_attr(target, key, value)` | Adds or overwrites an attribute on a span or an event | | `delete_attr(target, key)` | Removes an attribute from a span or an event. Does nothing if the key is absent. | | `mask_events(span, fn)` | Calls `fn` on each event. Keeps the event `fn` returns and drops the event when `fn` returns `None`. | | Rule | Detail | | - | - | | Synchronous | The processor doesn't await coroutines | | No I/O or network calls | The function runs on the thread that ends the span | | Returns a span | Usually the same span, changed with the helpers | | Runs on every span that passes the filter | Child tool spans, retries and LangGraph or Mastra spans included. Handle every span shape, or limit your logic by `span.name` or `span.instrumentation_scope.name`. | | Changes spans, doesn't drop them | To drop a span, use `should_export_span` | **Constraint:** When `mask` raises or returns `None`, the SDK sends a tombstone in place of the span. The tombstone keeps the trace and span IDs, parent, timing and kind. Its attributes and events are removed, its status is `ERROR` with the description `kubit-otel mask failed`, and its `kubit.sdk.mask_error` attribute holds the exception class name or `returned_none`. Unmasked data is never sent. The SDK logs one of these lines, with the traceback for a raised exception: ```text Output theme={null} [kubit-otel error] mask raised; tombstoning span span_name= span_id= err= [kubit-otel error] mask returned None; tombstoning span (use should_export_span to filter) span_name= span_id= ``` **Constraint:** A `KubitExporter` that you wrap in your own span processor applies neither `mask` nor the default filter. Apply both in your processor. ## Before production | Check | Why | | - | - | | `service_version` set to your release | Reports can group by **Client: Version** to compare releases | | `deployment.environment` set | Keeps dev and production traces apart | | Short-lived jobs call `trace.get_tracer_provider().force_flush()` before exit | Spans still in the batch are lost when the process exits | | User and session IDs on the root span | Without them Kubit can't group traces into sessions or join them to users. See [Add user and session IDs](#add-user-and-session-ids). | | Sensitive fields masked before export | Message content goes to Kubit for labeling. See [Mask sensitive content](#mask-sensitive-content). | ## Verify Send a test span with debug logging on: ```bash theme={null} KUBIT_API_KEY= KUBIT_OTEL_LOG_LEVEL=debug python -c " import kubit_instrumentation from opentelemetry import trace trace.get_tracer('kubit-sdk').start_span('hello-kubit').end() trace.get_tracer_provider().force_flush() " ``` `force_flush()` sends the span now. Without it, the span waits up to 5000 ms in the batch. With the `configure()` file from [You have no OpenTelemetry setup](#you-have-no-opentelemetry-setup), the output on stderr includes these lines, among other debug lines: ```text Output theme={null} [kubit-otel info] kubit_otel configured mode=registered service_name=support-agent service_version=1.0.0 endpoint= [kubit-otel debug] Exported batch to kubit span_count=1 ``` `endpoint=` is the label `kubit-otel` prints when you don't pass `endpoint`. The SDK still sends to `KUBIT_OTEL_ENDPOINT` or `https://otel.kubit.ai/v1/traces`. With `attach()`, the first line is `[kubit-otel info] kubit_otel attached endpoint=`. `KUBIT_OTEL_LOG_LEVEL` takes `debug`, `info`, `warn` or `error`, and the default is `info`. Then open **Agent** > **Activity**, set **Time Range** to **Last 1 day** and click **Execute**. The `hello-kubit` trace is in the list. [Validate in Activity](https://learn.kubit.ai/agent-observability/get-started/quickstart.md#validate-in-activity) shows what to check in the trace. ## If it fails | Symptom | Cause | Fix | | - | - | - | | `KeyError: 'KUBIT_API_KEY'` | Env var not set, or `.env` not loaded yet | Export it, or call `load_dotenv()` before `import kubit_instrumentation` | | `RuntimeError: kubit_otel.attach() requires a TracerProvider already registered as the global OTel provider. Call kubit_otel.configure() instead, or install a provider first (e.g. via Langfuse or opentelemetry-sdk).` | `attach()` ran before `set_tracer_provider()` | Move `attach()` after `trace.set_tracer_provider(provider)` | | Spans reach Kubit but not your old exporter, and OpenTelemetry logs `Overriding of current TracerProvider is not allowed` | `configure()` ran before your app's `trace.set_tracer_provider()`, so your app's provider was never registered | Move `configure()` after `trace.set_tracer_provider(provider)`, or replace it with `attach()` there | | LLM spans have no prompt or response text | Content capture is off | `export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true` | | No spans from a cron job or script | Process exits before the batch flushes | Call `trace.get_tracer_provider().force_flush()` at the end | | `[kubit-otel warn] KubitExporter export failed span_count=...` | The export request failed. A bad or revoked API key and a network failure log the same line, because it doesn't include the HTTP status. `kubit-otel` 0.7.2 and later log it. | Check that `KUBIT_API_KEY` holds a current key for this workspace, then work through [Is my data flowing?](https://learn.kubit.ai/agent-observability/get-started/data-flowing.md#check-2-key-and-connection) | | `[kubit-otel error] mask raised; tombstoning span span_name=...` | Your `mask` function raised an exception | Fix the function using the traceback in the log. See [Mask sensitive content](#mask-sensitive-content). | ## Next steps * [Quickstart](https://learn.kubit.ai/agent-observability/get-started/quickstart.md#check-the-extraction-mapping): check the extraction mapping and validate in Activity. * [Is my data flowing?](https://learn.kubit.ai/agent-observability/get-started/data-flowing.md): step-by-step checks. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.