Skip to main content
Polaris scores every conversation against the questions on the Evaluator page: four system questions and any you add. Evals run on Polaris and are included in Agent Observability at no extra cost, so you score every run instead of a sample. This page shows how to add a question and get it answered. It takes about 5 minutes, plus the time the run takes.

Before you start

  • A workspace with agent traces. See the Quickstart.
  • A question you want answered for every conversation, written so the answer is yes or no, a choice from a list, or a score.

Steps

1

Open the Evaluator

Open Evaluator. Questions lists the system questions: Answer relevant, Task resolved, Right tools and Grounded in tool results.
The Questions section of the Evaluator, with an Add question button above the system questions Answer relevant, Task resolved, Right tools and Grounded in tool results. Each shows its question text and Edit, Pause and Remove buttons.
2

Add a custom question

Under Custom questions, click Add custom question. Pick a template under Start from a template, or write your own. Pick its Answer type: Yes / no, Choice or Score.
The New custom question page. Under Start from a template are the templates Refund promised, Policy followed and Asked for a person (Yes / no), Contact reason and Reply language (Choice) and Helpfulness (Score). Below them, Answer type offers Yes / no, Choice and Score, above an Instructions box.
Click Add question. A workspace can have 10 custom questions turned on. An 11th is saved paused, and the button reads Add paused.
3

Get the question asked

When your new question is asked depends on the workspace:
4

Read the answers

When the run ends, open Activity and click Refresh. The Labels column shows each trace’s labels. A Yes / no question adds a label named after its short name, such as refund_promised. A Choice or Score question adds the short name plus the answer, such as contact_reason_billing.
The Activity traces grid with a Timeline of trace counts in 30-minute bars and errors in red, above the columns Timestamp, Name, Spans, Latency (Ms), Tokens, Model, Labels, Trace ID and User ID. The Labels column shows labels such as completed, session_not_completed, answered and low_judge_confidence.
Click a trace to see every answer under Labels and feedback > Custom questions. Until a question is answered on an Agent workspace, it reads Not answered yet. New answers come every hour.

Verify

Open Activity after the run and confirm that labeled traces carry an answer to your question.

If it fails

Next Steps