AB Test Calculator

AB Test Calculator

The AB test Calculator calculates the statistical significance of two sample spaces to find out whether the samples are statistically significant or not.

What is AB Test?

Statistical significance test in which two populations are related to checking if they differ significantly on a characteristic. it is basically a kind of Z-test. The objective of the AB test is to show that two proportions are not the same.

What is a Z-score?

A Z-score expresses a value's relationship to a group of values from the same distribution. The Z-score is calculated in the number of standard deviations from the group's mean. A higher Z-score means the value is more away from the distribution's mean.

Method of calculation:

1. Determine their sample sizes (n₁ and n₂)
2. Determine the number of positive results in each group (t₁ and t₂)
3. Calculate the population's proportions using the formulas

                             p₁ = t₁ / n₁ and p₂ = t₂ / n₂

4. Calculate the overall sample proportion

                             p = t₁ + t₂ / n₁ + n₂

5. Calculate the Z-score

6. Compare the absolute values of the results. If the absolute value of the Z-score is greater than or equal to the absolute value of the alpha level’s Z-score then the samples are statistically significant.

Example section:

In this section, we have given the step-by-step calculation of the AB test.

Example 1:

Compare the following groups and find out whether these are statistically significant or not,

if the confidence interval is 90%

Group 1

Sample size = 40

No. of positive results = 2

Conversion rate = 5%

Group 2

Sample size = 12

No. of positive results = 5

Conversion rate = 41.7%

Solution:

Step 1: Extract the data

Sample size = n₁ = 40

No. of positive results = t₁ = 2

The conversion rate in percentage (%) = 5

Group 2

Sample size = n₂ =12

No. of positive results = t₂ = 5

The conversion rate in percentage (%) = 41.7

Step 2: Apply the formula

p₁ = t₁ / n₁

p₁ = 2 / 40

p₁ = 0.05

p₂ = t₂ / n₂

p₂ = 5 / 12

p₂ = 0.417

Step 3: Calculate “p”

p = t₁+t₂ / n₁+n₂

p= 2+5 / 40 + 12

p = 0.135

Step 4: Calculate the Z-score

Z-score = p₁ - p₂ / √{p * (1 - p) * (1/n₁ + 1/n₂)}

Z-score = 0.05 - 0.417 / √{0.135 * (1 – 0.135) * (1 / 40 + 1 / 12)}

Z-score = -3.263

Alpha level’s z-score = 1.645

Step 5: Comparison

For the comparison, compare the absolute value of both the Z-score and Alpha level’s Z-score.

The absolute value of Z-score = 3.263

Absolute value Alpha level’s z-score = 1.645

As the absolute value of “Z-score” is greater than the absolute value of “Alpha level’s z-score” the samples are “statistically significant”.

 References:

A/B testing statistics: An intuitive guide for non-mathematicians | Conversion Sciences.

A/B testing - A complete guide to statistical testing. | Towards Data Science.