When to use
Pick the criteria mode by what you compare:
Comparison criteria
The comparison criteria set what you compare: one control group and one or two variants.Breakdown mode
Breakdown mode compares values of one field. Select a dimension field, then choose which value is the control and which values are the variants. Example: Field =Model Name, Control = gpt-4, Variant A = claude-sonnet.
Segment mode
Segment mode compares saved cohorts. Select one cohort as the control and one or two other cohorts as variants. Example: Control = the “Free tier users” cohort, Variant A = the “Premium users” cohort.Experiment mode
Experiment mode compares named variants of a formal experiment. Select an experiment ID, then choose which variant is the control and which are the treatment arms.Constraints
Metrics
The metric is what you measure to compare the control and the variants.
For a Funnel metric, you choose which step’s conversion to test. You can also add time to convert as a second metric.
Test settings
The test settings set the statistical parameters of the hypothesis test.Test type
Use two-tailed when you don’t expect a particular direction. Use one-tailed when you expect an improvement.
Significance level
The significance level is the highest p-value at which a result counts as statistically significant. A p-value of 0.05 means a 5% chance that the observed difference comes from random variation. A lower threshold is stricter.Confidence level
The confidence level is the probability that the true effect falls inside the reported confidence interval.Subject
The subject sets how sample size is counted: by User, Trace, Session or Span. These four subjects are fixed in Agent, Cortex Agents, MLflow Tracing, Google ADK and Langfuse workspaces. In User and Warehouse workspaces, the subjects come from your dataset.Constraints
Read the results
The results show which statistical test ran, a summary table, per-variant detail and a chart of each variant’s lift.Statistical method
Kubit picks the test by metric type.Results summary
The results table has one row per metric. A star marks the leading (higher-performing) variant.Variant detail
Hover over a variant to see its full statistics.Performance chart
The performance chart shows each variant’s lift as a point with error bars for its confidence interval. The horizontal axis is lift in percent. When a variant’s confidence interval doesn’t cross zero, its result is statistically significant.MCP prompt examples
A Significance prompt for the Kubit MCP server names the groups you compare, with the control marked (“control” or “baseline”), the metric and the date range. Add the criteria mode when it isn’t Breakdown, and the test settings when they aren’t the defaults (two-tailed, p = 0.05, 95% confidence). Say “one-tailed” for a directional test, or “at p=0.01” or “with 99% confidence” for a stricter threshold. A test can compare up to 2 variants against 1 control.“Is the difference in average latency between gpt-4 and claude-sonnet statistically significant?”
“Run a significance test comparing model gpt-4 (control) vs claude-sonnet on average latency over the last 30 days at 99% confidence”
“Test whether premium users have significantly different error rates than free users, using the ‘Premium’ and ‘Free Tier’ cohorts”
“Run a one-tailed significance test on step 3 conversion of the checkout funnel, comparing Android vs iOS users, testing whether Android is better”
Related
- Cohorts: build the segments that Segment mode compares.
- Funnel report: build the funnel a Funnel metric tests.
- Retention report: build the retention report a Retention metric tests.