Measuring Confidence
Measuring confidence in your LTV reports
Confidence measures the percentage a random sample of your test audience will show as, or greater difference than the mean of the baseline.
When you set a baseline in your report, AdLibertas will calculate the statistical significance (p-value) of your comparison audiences. The confidence (1 – p) percentage returned is the likelihood a random sample will show as, or greater difference than the mean.

How confidence is calculated.
Statistical significance is determined by calculating the p-value of a one-tailed test. Confidence is calculated as (1—p-value).
Confidence returns the likelihood a randomly selected sample of users, one user from A and one from B, will have an outcome as, or greater, than the displayed mean.

Most scientists and statisticians often strive for 95%+ confidence levels in experiments but you’ll find a range that works for your purposes. An AB test with <50% confidence isn’t necessarily a failure, it is just less than half of the users in the experiment will fall closer together than the mean value.
Also – like mean values– a small number of users will distort your confidence levels, so be sure to keep in mind the number of users at the end of the experiment.
Last modified 5mo ago