Measuring Confidence

Measuring confidence in your LTV reports

Confidence measures the percentage a random sample of your test audience will show as, or greater difference than the mean of the baseline.

When you set a baseline in your report, AdLibertas will calculate the statistical significance (p-value) of your comparison audiences. The confidence (1 – p) percentage returned is the likelihood a random sample will show as, or greater difference than the mean.

How to include confidence in AdLibertas reporting

When creating a report with more than one audience, you’ll be able to select a baseline. That baseline is used as the calculation for your comparison. All other audiences or variants will be compared to the baseline.

How confidence is calculated.

Statistical significance is determined by calculating the p-value of a one-tailed test. Confidence is calculated as (1—p-value).

Understanding the outcome.

Confidence returns the likelihood a randomly selected sample of users, one user from A and one from B, will have an outcome as, or greater, than the displayed mean.

Best practices for using confidence in your reporting

Most scientists and statisticians often strive for 95%+ confidence levels in experiments but you’ll find a range that works for your purposes. An AB test with <50% confidence isn’t necessarily a failure, it is just less than half of the users in the experiment will fall closer together than the mean value.

Also – like mean values– a small number of users will distort your confidence levels, so be sure to keep in mind the number of users at the end of the experiment.

Looking for more tips, read our article on setting up an effective framework for AB tests.