> ## Documentation Index > Fetch the complete documentation index at: https://learn.kubit.ai/llms.txt > Use this file to discover all available pages before exploring further. # Significance report > Test whether a difference between a control group and up to 2 variants is statistically significant with the Kubit Significance report. A Significance report tests whether a difference between two groups is statistically real or random variation. This page covers the Significance report's settings: comparison criteria, metrics and test settings, and how to read the results. ## When to use | Use case | Example | | - | - | | A/B testing | Does a new prompt template convert significantly better than the control template? | | Segment comparison | Do power users have a different error rate than casual users? | | Model evaluation | Does switching LLM models change latency or cost significantly? | | Feature rollout | Does a new tool call implementation improve response quality over the previous version? | | Experiment analysis | Analyze a formal experiment with named control and treatment arms. | Pick the criteria mode by what you compare: | You compare | Mode | | - | - | | Values of one field, such as model A and model B | **Breakdown** | | Saved cohorts, such as power users and casual users | **Segment** | | Named variants of a formal experiment | **Experiment** | ## Comparison criteria The comparison criteria set what you compare: one control group and one or two variants. ### Breakdown mode Breakdown mode compares values of one field. Select a dimension field, then choose which value is the **control** and which values are the **variants**. **Example:** Field = `Model Name`, Control = `gpt-4`, Variant A = `claude-sonnet`. ### Segment mode Segment mode compares saved [cohorts](https://learn.kubit.ai/product-analytics/reports/cohorts.md). Select one cohort as the **control** and one or two other cohorts as **variants**. **Example:** Control = the "Free tier users" cohort, Variant A = the "Premium users" cohort. ### Experiment mode Experiment mode compares named variants of a formal experiment. Select an experiment ID, then choose which variant is the **control** and which are the **treatment arms**. ### Constraints | Constraint | Value | | - | - | | Control group | Exactly **1**, required | | Variants | **1** or **2**. Variant A is required, Variant B is optional. | ## Metrics The metric is what you measure to compare the control and the variants. | Type | Configuration | Use it for | | - | - | - | | **Measure** | Pick a measure function and a field | A quick comparison without a saved report | | **Query** | Pick a saved [Query report](https://learn.kubit.ai/product-analytics/reports/query.md) and one of its measures | Reusing a metric you already defined | | **Funnel** | Pick a saved [Funnel report](https://learn.kubit.ai/product-analytics/reports/funnel.md) and the step to measure | Testing conversion at one funnel step | | **Retention** | Pick a saved [Retention report](https://learn.kubit.ai/product-analytics/reports/retention.md) | Testing a retention difference | For a Funnel metric, you choose which step's conversion to test. You can also add time to convert as a second metric. ## Test settings The test settings set the statistical parameters of the hypothesis test. ### Test type | Type | Tests whether the variant is | Default | | - | - | - | | **Two-Tailed** | Significantly different from the control, higher or lower | **Yes** | | **One-Tailed** | Significantly better than the control, in one direction only | No | Use two-tailed when you don't expect a particular direction. Use one-tailed when you expect an improvement. ### Significance level The significance level is the highest p-value at which a result counts as statistically significant. A p-value of 0.05 means a 5% chance that the observed difference comes from random variation. A lower threshold is stricter. | Setting | Value | | - | - | | Range | **0.01** to **0.10** | | Default | **0.05** | ### Confidence level The confidence level is the probability that the true effect falls inside the reported confidence interval. | Setting | Value | | - | - | | Range | **90%** to **99%** | | Default | **95%** | ### Subject The subject sets how sample size is counted: by **User**, **Trace**, **Session** or **Span**. These four subjects are fixed in Agent, Cortex Agents, MLflow Tracing, Google ADK and Langfuse workspaces. In User and Warehouse workspaces, the subjects come from your dataset. ### Constraints | Constraint | Value | | - | - | | Time granularity | **All Time only.** A Significance test aggregates across the full date range. | | Sampling | **Not supported.** The test uses all data. | ## Read the results The results show which statistical test ran, a summary table, per-variant detail and a chart of each variant's lift. ### Statistical method Kubit picks the test by metric type. | Metric type | Test | What it does | | - | - | - | | Measure, Query, Retention | Welch's t-test | Compares the means of two groups without assuming equal variances | | Funnel | Chi-square test | Tests independence in a converted or not converted contingency table | ### Results summary The results table has one row per metric. A star marks the leading (higher-performing) variant. | Column | Shows | | - | - | | **Metric** | The metric tested | | **Control** | Mean, standard deviation and sample size of the control group | | **Variant A**, **Variant B** | Lift, confidence interval, statistical significance and sample size of each variant | ### Variant detail Hover over a variant to see its full statistics. | Field | Meaning | | - | - | | **Lift** | Change relative to the control: (variant mean − control mean) / control mean × 100% | | **Lift Confidence Interval** | The range the true lift likely falls in, from lower % to upper % | | **Stat Sig** | True or False, with the p-value. True means the difference is unlikely to come from chance. | | **Power** | Statistical power of the test, from 0 to 1. Higher power means a more reliable result. | | **Test Score** | The t-statistic (Welch's t-test) or the χ² statistic (chi-square test) | | **Delta** | Absolute difference in means: variant mean − control mean | | **Mean** | Average metric value of the variant | | **Std Deviation** | Spread of the metric values | | **Sample Size** | Number of subjects in the variant | | **Std Error** | Standard error of the difference. Not shown for Funnel metrics. | ### Performance chart The performance chart shows each variant's lift as a point with error bars for its confidence interval. The horizontal axis is lift in percent. When a variant's confidence interval doesn't cross zero, its result is statistically significant. ## MCP prompt examples A Significance prompt for the Kubit MCP server names the groups you compare, with the control marked ("control" or "baseline"), the metric and the date range. Add the criteria mode when it isn't Breakdown, and the test settings when they aren't the defaults (two-tailed, p = 0.05, 95% confidence). Say "one-tailed" for a directional test, or "at p=0.01" or "with 99% confidence" for a stricter threshold. A test can compare up to 2 variants against 1 control. > "Is the difference in average latency between gpt-4 and claude-sonnet statistically significant?" > "Run a significance test comparing model gpt-4 (control) vs claude-sonnet on average latency over the last 30 days at 99% confidence" > "Test whether premium users have significantly different error rates than free users, using the 'Premium' and 'Free Tier' cohorts" > "Run a one-tailed significance test on step 3 conversion of the checkout funnel, comparing Android vs iOS users, testing whether Android is better" ## Related * [Cohorts](https://learn.kubit.ai/product-analytics/reports/cohorts.md): build the segments that Segment mode compares. * [Funnel report](https://learn.kubit.ai/product-analytics/reports/funnel.md): build the funnel a Funnel metric tests. * [Retention report](https://learn.kubit.ai/product-analytics/reports/retention.md): build the retention report a Retention metric tests. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.