> ## Documentation Index
> Fetch the complete documentation index at: https://learn.kubit.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Add your own eval questions

> Score every agent conversation against your own yes or no, choice or score questions with Polaris, at no extra LLM cost.

Polaris scores every conversation against the questions on the **Evaluator** page: four system questions and any you add. Evals run on Polaris and are included in Agent Observability at no extra cost, so you score every run instead of a sample. This page shows how to add a question and get it answered. It takes about 5 minutes, plus the time the run takes.

## Before you start

* [ ] A workspace with agent traces. See the [Quickstart](https://learn.kubit.ai/agent-observability/get-started/quickstart.md).
* [ ] A question you want answered for every conversation, written so the answer is yes or no, a choice from a list, or a score.

## Steps

### 1. Open the Evaluator

Open **Evaluator**. **Questions** lists the system questions: **Answer relevant**, **Task resolved**, **Right tools** and **Grounded in tool results**.

![The Questions section of the Evaluator, with an Add question button above the system questions Answer relevant, Task resolved, Right tools and Grounded in tool results. Each shows its question text and Edit, Pause and Remove buttons.](https://mintcdn.com/kubit/iy-nIyF9Xtsr91WV/images/agent-observability/evaluator-questions.png?fit=max\&auto=format\&n=iy-nIyF9Xtsr91WV\&q=85\&s=3bf2d6794392c439029c5640d5197b82)

### 2. Add a custom question

Under **Custom questions**, click **Add custom question**. Pick a template under **Start from a template**, or write your own. Pick its **Answer type**: **Yes / no**, **Choice** or **Score**.

![The New custom question page. Under Start from a template are the templates Refund promised, Policy followed and Asked for a person (Yes / no), Contact reason and Reply language (Choice) and Helpfulness (Score). Below them, Answer type offers Yes / no, Choice and Score, above an Instructions box.](https://mintcdn.com/kubit/iy-nIyF9Xtsr91WV/images/agent-observability/evaluator-new-custom-question.png?fit=max\&auto=format\&n=iy-nIyF9Xtsr91WV\&q=85\&s=19a7c6112353bb21501d7f3d22cd3bba)

| **Field** | **Required** | **What it sets** |
| - | - | - |
| **Answer type** | Required | **Yes / no**, **Choice** or **Score**. Can't be changed after you save. |
| **Instructions** | Required | The question or rule, in your own words. Up to 1,000 characters. |
| **Short name (optional)** | Optional | The label's name, such as `refund_promised`. Lowercase letters, digits and underscores, starting with a letter, up to 64 characters. Left empty, it's made from the first words of the instructions. Can't be changed after you save. |
| **Looks at** | Required, defaults to **The whole conversation** | **The whole conversation**, **The final reply**, **Each tool call**, **Each model call**, **Each retrieval** or **Each agent step**. A step type is judged on up to the latest 20 of those steps. |
| **Yes means** and **No means** | Required for **Yes / no** | What each answer means. Up to 300 characters each. |
| Options | Required for **Choice** | 2 to 20 options, each with a lowercase name and a meaning. Click **Add an option**. |
| Scores | Required for **Score** | 2 to 10 whole numbers from 0 to 100, lowest first, each with a meaning. Click **Add a score**. |
| **How sure it must be (%)** | Optional, under **More options** | 1 to 99. Defaults to 50 for **Yes / no**. |
| **Ask about** | Optional, under **More options** | **Every conversation** or **A Published Filter**. Only Agent workspaces can pick a Published Filter. |
| **Label type** | Optional, under **More options** | **Action**, **Exception**, **Misstep**, **Warning** or **Behavior** (the default). |

Click **Add question**. A workspace can have 10 custom questions turned on. An 11th is saved paused, and the button reads **Add paused**.

### 3. Get the question asked

When your new question is asked depends on the workspace:

| **Workspace** | **What to do** |
| - | - |
| **Sandbox** | Click **Relabel** on an upload. Relabelling asks every question again and replaces the upload's labels. The Evaluator labels one upload at a time. |
| Warehouse (**Warehouse**, **MLflow Tracing**, **Cortex Agents**, **Google ADK**, **Langfuse**) | On the **Evaluator** page, pick a **Window** of days and click **Label conversations**. Check **Relabel conversations that already have labels** to ask conversations labelled earlier. Nothing is scheduled. |
| **Agent** | Nothing. Every hour, the Evaluator answers it for each new conversation. It isn't asked of earlier conversations. |

### 4. Read the answers

When the run ends, open **Activity** and click **Refresh**. The **Labels** column shows each trace's labels. A **Yes / no** question adds a label named after its short name, such as `refund_promised`. A **Choice** or **Score** question adds the short name plus the answer, such as `contact_reason_billing`.

![The Activity traces grid with a Timeline of trace counts in 30-minute bars and errors in red, above the columns Timestamp, Name, Spans, Latency (Ms), Tokens, Model, Labels, Trace ID and User ID. The Labels column shows labels such as completed, session\_not\_completed, answered and low\_judge\_confidence.](https://mintcdn.com/kubit/iy-nIyF9Xtsr91WV/images/agent-observability/activity-labels.png?fit=max\&auto=format\&n=iy-nIyF9Xtsr91WV\&q=85\&s=363d5e7136da16ebb01235a5b51ed41a)

Click a trace to see every answer under **Labels and feedback** > **Custom questions**. Until a question is answered on an Agent workspace, it reads `Not answered yet. New answers come every hour.`

## Verify

Open **Activity** after the run and confirm that labeled traces carry an answer to your question.

## If it fails

| **Symptom** | **Cause** | **Fix** |
| - | - | - |
| **Relabel** is disabled | A run is in progress | Wait until the upload no longer shows `Running` |
| **Labels** reads `Not analysed yet` | The Evaluator hasn't labeled that trace | Wait for the run to finish, then click **Refresh** |
| `10 custom questions are on already, so this one is saved paused.` | 10 custom questions are already turned on | Pause another question, then turn this one on |
| `The Evaluator doesn't support this warehouse, so custom questions would never be answered here.` | The workspace's warehouse isn't Snowflake, Databricks, BigQuery or ClickHouse | Use a workspace on a supported warehouse |

## Next Steps

* [Polaris](https://learn.kubit.ai/agent-observability/use/polaris.md#evaluator-questions): the system questions and the constraints.
* [Find Missteps that return OK](https://learn.kubit.ai/agent-observability/use/find-missteps.md): use labels to find silent failures.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.