> ## Documentation Index
> Fetch the complete documentation index at: https://learn.kubit.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Kubit SDK for Node

> Send OpenTelemetry spans from a Node or TypeScript agent to Kubit with the @kubit-ai/otel SDK, on its own or next to another OpenTelemetry SDK.

`@kubit-ai/otel` sends OpenTelemetry spans from a Node or TypeScript app to Kubit. It has two entry points: `configure()` when Kubit is your only OpenTelemetry setup, and `KubitSpanProcessor` when your app already creates a tracer provider or a `NodeSDK`. This page covers both, the span filter, and the settings to check before production.

## Before you start

* [ ] Node.js 18 or later
* [ ] OpenTelemetry JS SDK v2. Version 1 isn't supported.
* [ ] A Kubit API key in `KUBIT_API_KEY`

Kubit authenticates spans with an ingestion API key, read from the `KUBIT_API_KEY` environment variable. Each key belongs to one workspace.

| Way to get a key | How |
| - | - |
| **Coding agent** | Run `/kubit-integrate` in Claude Code or Cursor. It creates or selects a workspace, mints a key and writes `KUBIT_API_KEY` to `.env.local` or `.env`. See [Agent plugin](https://learn.kubit.ai/agent-observability/integrate/agent-plugin.md). |
| **Kubit app** | Go to **Settings** > **Workspace** > **API Keys** and click **Generate API Key**. Add an optional **Note** (up to 50 characters) and click **Generate**. |

The app shows a new key only once: `This is the only time we'll show you this key. Copy it now.` Copy it, or click **Download** for a `.env` file, then click **I've Saved It**. The downloaded file names the key `KUBIT_OTEL_API_KEY`, so rename it to `KUBIT_API_KEY`.

**Availability:** **API Keys** shows only to the Owner and Admin roles, and only in **Agent** and **User** workspaces.

> [!WARNING]
>
> Keep the key out of source control. Put it in `.env` or your secrets manager, and check that `.env` is in `.gitignore`.

## Prompt for your coding agent

```text theme={null}
Add Kubit tracing to this Node repo. Follow https://learn.kubit.ai/agent-observability/integrate/sdk-node.md exactly.
If the repo already creates a NodeTracerProvider or NodeSDK, add KubitSpanProcessor to its spanProcessors. Otherwise use configure().
Read KUBIT_API_KEY from .env. Show me the diff before writing any file.
```

## Install

Install the package and its OpenTelemetry peer dependencies:

```bash theme={null}
npm install @kubit-ai/otel \
  @opentelemetry/api \
  @opentelemetry/exporter-trace-otlp-proto \
  @opentelemetry/resources@^2 \
  @opentelemetry/sdk-trace-base@^2 \
  @opentelemetry/sdk-trace-node@^2
```

| Peer dependency | Minimum version |
| - | - |
| `@opentelemetry/api` | `1.7.0` |
| `@opentelemetry/exporter-trace-otlp-proto` | `0.200.0` |
| `@opentelemetry/resources` | `2.0.0` |
| `@opentelemetry/sdk-trace-base` | `2.0.0` |
| `@opentelemetry/sdk-trace-node` | `2.0.0` |

## Choose configure() or KubitSpanProcessor

| Your app today | Use | Why |
| - | - | - |
| No OpenTelemetry setup | `configure()` | Creates and registers a tracer provider with Kubit's processor in one call |
| Creates its own `NodeTracerProvider` or `NodeSDK` | `KubitSpanProcessor` | Adds Kubit next to your existing processors on the one provider |
| Sends traces to Langfuse, Braintrust or another OpenTelemetry-based SDK | `KubitSpanProcessor` | Same as above. The [Agent plugin](https://learn.kubit.ai/agent-observability/integrate/agent-plugin.md) handles the sink-specific wiring. |

`@kubit-ai/otel` has no `attach()`, unlike the Python SDK. OpenTelemetry JS SDK v2 takes span processors only when the provider is created, so there's no way to add one to a provider that's already running.

> [!WARNING]
>
> Don't call `configure()` if your app already sets up a tracer provider or `NodeSDK`. `configure()` registers a second global provider and replaces the first, so your existing traces stop reaching their current destination.

## You have no OpenTelemetry setup

Create `kubit-instrumentation.ts` and import it first in your entrypoint, before any LLM client is created:

```typescript kubit-instrumentation.ts theme={null}
import { configure } from "@kubit-ai/otel";

const apiKey = process.env.KUBIT_API_KEY;
if (!apiKey) throw new Error("KUBIT_API_KEY is not set");

export const tracerProvider = configure({
  apiKey,
  serviceName: "support-agent",
  serviceVersion: "1.4.0",
  resourceAttributes: { "deployment.environment": "production" },
});
```

```typescript index.ts theme={null}
import "./kubit-instrumentation"; // first import, before any LLM client is created
```

Change `serviceName`, `serviceVersion` and `deployment.environment` to match your app. Kubit uses them to tell agents and releases apart.

### configure() options

| Option | Required | Default | Notes |
| - | - | - | - |
| **apiKey** | **Yes** | | Your Kubit API key. The SDK doesn't read it from the environment, so pass it in. |
| **serviceName** | Optional | `default` | Becomes the OpenTelemetry `service.name`. Set it. Kubit uses it to tell agents apart. |
| **serviceVersion** | Optional | Not set | Becomes `service.version` |
| **resourceAttributes** | Optional | `{}` | Extra resource attributes, such as `deployment.environment` |
| **endpoint** | Optional | `https://otel.kubit.ai/v1/traces` | Overrides the trace endpoint. The `KUBIT_OTEL_ENDPOINT` environment variable does the same. |
| **shouldExportSpan** | Optional | LLM spans only | Decides which spans go to Kubit. See [Span filtering](#span-filtering). |
| **mask** | Optional | Not set | A function that redacts a span before export. See [Mask sensitive content](#mask-sensitive-content). Needs `@kubit-ai/otel` 0.7.3 or later. |

`configure()` returns the tracer provider it registered.

## You already have a tracer provider

Pass `KubitSpanProcessor` in `spanProcessors` next to your other processors, when you create the provider.

### NodeTracerProvider

```typescript tracing.ts theme={null}
import { KubitSpanProcessor } from "@kubit-ai/otel";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";

const apiKey = process.env.KUBIT_API_KEY;
if (!apiKey) throw new Error("KUBIT_API_KEY is not set");

const provider = new NodeTracerProvider({
  resource: resourceFromAttributes({ "service.name": "support-agent" }),
  spanProcessors: [
    // your existing processors stay here
    new KubitSpanProcessor({ apiKey }),
  ],
});
provider.register();
```

### NodeSDK

```typescript instrumentation.ts theme={null}
import { KubitSpanProcessor } from "@kubit-ai/otel";
import { NodeSDK } from "@opentelemetry/sdk-node";

const apiKey = process.env.KUBIT_API_KEY;
if (!apiKey) throw new Error("KUBIT_API_KEY is not set");

export const sdk = new NodeSDK({
  serviceName: "support-agent",
  spanProcessors: [
    // your existing processors stay here
    new KubitSpanProcessor({ apiKey }),
  ],
});
sdk.start();
```

### KubitSpanProcessor options

`KubitSpanProcessor` is an OpenTelemetry `BatchSpanProcessor` that drops non-LLM spans before they're queued.

| Option | Required | Default | Notes |
| - | - | - | - |
| **apiKey** | **Yes** | | Your Kubit API key, sent in the `x-api-key` header |
| **endpoint** | Optional | `https://otel.kubit.ai/v1/traces` | Same resolution as `configure()` |
| **shouldExportSpan** | Optional | LLM spans only | See [Span filtering](#span-filtering) |
| **mask** | Optional | Not set | Same as in `configure()`. Needs `@kubit-ai/otel` 0.7.3 or later. |
| **maxQueueSize** | Optional | `2048` | Spans held before new ones are dropped |
| **scheduledDelayMillis** | Optional | `5000` | Time between export batches, in milliseconds |
| **maxExportBatchSize** | Optional | `512` | Spans per export request |
| **exportTimeoutMillis** | Optional | `30000` | Timeout for each export, in milliseconds |

For full control, `KubitExporter` is the plain exporter without filtering, masking or batching. Wrap it in your own processor.

## Span filtering

By default, `KubitSpanProcessor` sends only LLM-related spans to Kubit. HTTP, database and framework spans from other auto-instrumentation stay out of your workspace.

A span is sent when any of these is true:

| Rule | Example |
| - | - |
| It was created by the Kubit SDK tracer, `kubit-sdk` | Spans from Kubit's own helpers |
| It has a `gen_ai.*` attribute | `gen_ai.request.model`, `gen_ai.tool.name` |
| Its instrumentation scope starts with a known LLM prefix | `ai` (Vercel AI SDK), `@arizeai/openinference`, `langfuse-sdk`, `langsmith`, `braintrust` |

`KNOWN_LLM_INSTRUMENTATION_SCOPE_PREFIXES` exports the full prefix list.

The default filter drops these spans even when they match a rule above:

| Dropped span | Why |
| - | - |
| LangGraph spans named `ChannelWrite...`, `__start__` or `__end__` | Graph coordination with no LLM data |
| Mastra spans with `mastra.span.type` set to `model_chunk` | One span per streamed token chunk |

### Keep spans from your own framework

```typescript tracing.ts theme={null}
import { KubitSpanProcessor, getInstrumentationScopeName, isDefaultExportSpan } from "@kubit-ai/otel";

const apiKey = process.env.KUBIT_API_KEY;
if (!apiKey) throw new Error("KUBIT_API_KEY is not set");

export const kubitProcessor = new KubitSpanProcessor({
  apiKey,
  shouldExportSpan: ({ otelSpan }) =>
    isDefaultExportSpan(otelSpan) ||
    (getInstrumentationScopeName(otelSpan)?.startsWith("my-framework") ?? false),
});
```

### Send every span

```typescript tracing.ts theme={null}
import { KubitSpanProcessor } from "@kubit-ai/otel";

const apiKey = process.env.KUBIT_API_KEY;
if (!apiKey) throw new Error("KUBIT_API_KEY is not set");

export const kubitProcessor = new KubitSpanProcessor({ apiKey, shouldExportSpan: () => true });
```

If `shouldExportSpan` throws, the span is dropped.

The workspace applies its own filter after the SDK's. A new workspace starts with the same keep and drop rules, so spans you keep with `shouldExportSpan` can still be dropped there. Change it in **Data** > **Collection** > **Filtering**. Sampling and masking rules are on the **Sampling** and **Masking** tabs there.

## Trace LLM and tool calls

Kubit receives whatever spans your app produces. Spans from an LLM instrumentation library pass the default filter. With the Vercel AI SDK, turn on telemetry per call:

```typescript chat.ts theme={null}
import "./kubit-instrumentation";
import { openai } from "@ai-sdk/openai";
import { generateText } from "ai";

const { text } = await generateText({
  model: openai("gpt-4o"),
  prompt: "Where is my order?",
  experimental_telemetry: { isEnabled: true },
});
console.log(text);
```

For tool calls, wrap each tool in its own span so it shows up as a step in the execution path:

```typescript tools.ts theme={null}
import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("support-agent");

export async function searchOrders(customerId: string, query: string): Promise<unknown[]> {
  return tracer.startActiveSpan("execute_tool search_orders", async (span) => {
    span.setAttribute("gen_ai.operation.name", "execute_tool");
    span.setAttribute("gen_ai.tool.name", "search_orders");
    const results = await runOrderSearch(customerId, query); // your existing tool code
    span.setAttribute("search.result_count", results.length);
    span.end();
    return results;
  });
}
```

## Add user and session IDs

Kubit groups traces into sessions and joins them to users from IDs on your spans. Set `user.id` and `session.id` on the root span of each request. The `gen_ai.operation.name` attribute keeps your own span past the default filter, which drops spans with no `gen_ai.*` attribute:

```typescript handler.ts theme={null}
import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("support-agent");

export async function handleMessage(userId: string, conversationId: string, text: string): Promise<string> {
  return tracer.startActiveSpan("invoke_agent support-agent", async (span) => {
    span.setAttribute("gen_ai.operation.name", "invoke_agent");
    span.setAttribute("user.id", userId);
    span.setAttribute("session.id", conversationId);
    try {
      return await runAgent(text); // your existing agent code
    } finally {
      span.end();
    }
  });
}
```

Use the same user ID your product analytics events use, so agent runs join to user behavior. The default extraction mapping reads `enduser.id` or `user.id` for the user, and `session.id` or `gen_ai.conversation.id` for the session. To read other keys, edit the mapping as described in [Check the extraction mapping](https://learn.kubit.ai/agent-observability/get-started/quickstart.md#check-the-extraction-mapping).

## Mask sensitive content

The `mask` option redacts span content before the span leaves your process. `KubitSpanProcessor` runs your function on each span that passes the filter, before the span is queued for export. `mask` needs `@kubit-ai/otel` 0.7.3 or later and works with `configure()` and `KubitSpanProcessor`.

```typescript kubit-instrumentation.ts theme={null}
import { configure } from "@kubit-ai/otel";
import { maskEvents, setAttr } from "@kubit-ai/otel/mask";

const apiKey = process.env.KUBIT_API_KEY;
if (!apiKey) throw new Error("KUBIT_API_KEY is not set");

const CARD_RE = /\b(?:\d[ -]*?){13,19}\b/g;

export const tracerProvider = configure({
  apiKey,
  serviceName: "support-agent",
  mask: (span) => {
    const prompt = (span.attributes as Record<string, unknown>)["gen_ai.prompt"];
    if (typeof prompt === "string") {
      setAttr(span, "gen_ai.prompt", prompt.replace(CARD_RE, "[REDACTED CC]"));
    }
    maskEvents(span, (e) => (e.name === "gen_ai.user.message" ? null : e));
    return span;
  },
});
```

This mask replaces card numbers in `gen_ai.prompt` and drops `gen_ai.user.message` events, where newer GenAI instrumentors record the prompt.

The helpers in `@kubit-ai/otel/mask` are the supported way to change a span inside `mask`:

| Helper | What it does |
| - | - |
| `setAttr(target, key, value)` | Adds or overwrites an attribute on a span or an event |
| `deleteAttr(target, key)` | Removes an attribute from a span or an event. Does nothing if the key is absent. |
| `maskEvents(span, fn)` | Calls `fn` on each event. Keeps the event `fn` returns and drops the event when `fn` returns `null` or `undefined`. |

| Rule | Detail |
| - | - |
| Synchronous | The processor doesn't await promises |
| No I/O or network calls | The function runs when the span ends, inside `onEnd` |
| Returns a span | Usually the same span, changed with the helpers |
| Runs on every span that passes the filter | Child tool spans, retries and LangGraph, Mastra or Vercel AI SDK spans included. Handle every span shape, or limit your logic by `span.name` or `span.instrumentationScope.name`. |
| Changes spans, doesn't drop them | To drop a span, use `shouldExportSpan` |

**Constraint:** When `mask` throws or returns `null` or `undefined`, the SDK sends a tombstone in place of the span. The tombstone keeps the trace and span IDs, parent, timing and kind. Its attributes and events are removed, its status is `ERROR` with the message `kubit-otel mask failed`, and its `kubit.sdk.mask_error` attribute holds the error's `name` or `returned_null`. Unmasked data is never sent. The SDK logs one of these lines:

```text Output theme={null}
[kubit-otel error] mask raised; tombstoning span  span_name=<name> span_id=<id> err=<message> stack=<stack>
[kubit-otel error] mask returned null; tombstoning span (use shouldExportSpan to filter)  span_name=<name> span_id=<id>
```

## Before production

| Check | Why |
| - | - |
| `serviceVersion` set to your release | Reports can group by **Client: Version** to compare releases |
| `deployment.environment` set | Keeps dev and production traces apart |
| Short-lived scripts and serverless handlers flush before exit | Spans wait up to 5 seconds in the batch. They're lost if the process exits first. Call `forceFlush()` on the provider, or `shutdown()` on `NodeSDK`. |
| `KUBIT_OTEL_ENDPOINT` set only on purpose | It redirects every Kubit export from the process |
| User and session IDs on the root span | Without them Kubit can't group traces into sessions or join them to users. See [Add user and session IDs](#add-user-and-session-ids). |
| Sensitive fields masked before export | Message content goes to Kubit for labeling. See [Mask sensitive content](#mask-sensitive-content). |

`@kubit-ai/otel` doesn't read the standard `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` variables. They apply to the whole process and would redirect Kubit's traces if another OpenTelemetry SDK set them. Use `KUBIT_OTEL_ENDPOINT` instead.

## Verify

Send a test span with debug logging on:

```typescript send-test-span.ts theme={null}
import { configure } from "@kubit-ai/otel";
import { trace } from "@opentelemetry/api";
import type { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";

const apiKey = process.env.KUBIT_API_KEY;
if (!apiKey) throw new Error("KUBIT_API_KEY is not set");

const provider = configure({ apiKey, serviceName: "support-agent" }) as NodeTracerProvider;

trace.getTracer("support-agent").startActiveSpan("chat.completion", (span) => {
  span.setAttribute("gen_ai.request.model", "gpt-4o");
  span.setAttribute("gen_ai.usage.input_tokens", 10);
  span.setAttribute("gen_ai.usage.output_tokens", 5);
  span.end();
});

await provider.forceFlush();
```

```bash theme={null}
KUBIT_OTEL_LOG_LEVEL=debug npx tsx send-test-span.ts
```

The output includes these lines, among other debug lines:

```text Output theme={null}
[kubit-otel info] kubit_otel configured  service_name=support-agent service_version=- endpoint=<redacted>
[kubit-otel debug] KubitSpanProcessor forceFlush
[kubit-otel debug] Exported batch to kubit  span_count=1
```

`endpoint=<redacted>` is what `@kubit-ai/otel` prints when you don't pass `endpoint`. The SDK still sends to `KUBIT_OTEL_ENDPOINT` or `https://otel.kubit.ai/v1/traces`. `Exported batch to kubit` means the collector accepted the batch. No `Dropped span due to shouldExportSpan filter` line for `chat.completion` means the span passed the filter.

Then open **Agent** > **Activity**, set **Time Range** to **Last 1 day** and click **Execute**. The `chat.completion` trace is in the list. [Validate in Activity](https://learn.kubit.ai/agent-observability/get-started/quickstart.md#validate-in-activity) shows what to check in the trace.&#x20;

## If it fails

| Symptom | Cause | Fix |
| - | - | - |
| `Error: KUBIT_API_KEY is not set` | The env var isn't set in the shell that runs Node, or `.env` isn't loaded yet | Export it, or load `.env` (for example with `import "dotenv/config"`) before `kubit-instrumentation` |
| `[kubit-otel debug] Dropped span due to shouldExportSpan filter  span_name=...` | The span has no `gen_ai.*` attribute and isn't from a known LLM scope | Add `gen_ai.*` attributes, or extend `shouldExportSpan` as in [Span filtering](#span-filtering) |
| Your existing traces stop arriving after you add Kubit | `configure()` registered a second provider | Remove `configure()` and add `KubitSpanProcessor` to your existing provider |
| Spans from a short script never show up | The process exited before the 5-second batch was exported | Call `forceFlush()` on the provider before exit |
| The app is on OpenTelemetry JS SDK v1 | `configure()` uses v2-only APIs (`resourceFromAttributes()` and `spanProcessors` at construction) | Upgrade the OpenTelemetry packages to v2. The peer dependencies require `@opentelemetry/resources`, `@opentelemetry/sdk-trace-base` and `@opentelemetry/sdk-trace-node` 2.0.0 or later. `@kubit-ai/otel` has no runtime version check. |
| `[kubit-otel warn] KubitExporter export failed  span_count=...` | The export request failed. A bad or revoked API key and a network failure log the same line, because it doesn't include the HTTP status. Versions 0.7.2 and later log it. | Check that `KUBIT_API_KEY` holds a current key for this workspace, then work through [Is my data flowing?](https://learn.kubit.ai/agent-observability/get-started/data-flowing.md#check-2-key-and-connection). |
| `[kubit-otel error] mask raised; tombstoning span  span_name=...` | Your `mask` function threw | Fix the function using the stack in the log. See [Mask sensitive content](#mask-sensitive-content). |

## Next steps

* [Quickstart](https://learn.kubit.ai/agent-observability/get-started/quickstart.md#check-the-extraction-mapping): check the extraction mapping and validate in Activity.
* [Agent plugin](https://learn.kubit.ai/agent-observability/integrate/agent-plugin.md): keep Langfuse or Braintrust as your sink and add Kubit next to it.
* [Is my data flowing?](https://learn.kubit.ai/agent-observability/get-started/data-flowing.md): step-by-step checks.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.