A/B Test

Analyzes A/B test result data based on Bayesian algorithm.

Input Data

Input data should be an A/B test result data. Each row should represent a subject or subjects, such as a visitor with the result of either converted or not. It should have following columns.

• Target Variable - A column that indicates that the subject(s) converted or not.

• Explanatory Variable - A column that indicates the subject(s) belongss to group A or B. The data type can be either logical, numeric or text. It should have only 2 unique values.

• Value - Optional. A column with number of subject(s) each row represents.

• Expected Average of CR - Mean of prior probability distribution (assumed to be a beta distribution) of conversion rate. If not specified, uniform distribution is used as non-informative prior.

• Expected SD of CR - Standard deviation of prior probability distribution (assumed to be a beta distribution) of conversion rate. If not specified, uniform distribution is used as non-informative prior.

Properties

• Revert A/B - If set to TRUE, which group to considered as A or B is reverted.

How to Use This Feature

1. Click Analytics View tab.

2. If necessary, click "+" button on the left of existing Analytics tabs, to create a new Analytics.

3. Select "A/B Test - Bayesian" for Analytics Type.

4. Select Conversion Rate column with "Conversion Rate (CR)" column selector.

5. Select Total Population column with "Total Population" column selector.

6. (Optional) Enter an expected average of CR value from empirical estimation or historical data.

7. (Optional) Enter an expected Standard Deviation of CR value from empirical estimation or historical data.

8. (Optional) Select a column to group subjects with "Repeat By" column selector. For each group, a separate small chart will be displayed.

9. Click Run button to run the analytics.

10. Select view type (explained below) by clicking view type link to see each type of generated visualization.

"Summary" View

"Summary" View displays the summary of the Bayesian A/B test analysis result in a table format.

This example is with Repeat By.

Output Data

Following is the list of columns in the raw data displayed in the "Summary" View.

• Group Columns - If Repeat By is specified, the columns appears in the output data frame.

• Group - Group name. Either A or B.

• AB Identifier - The actual A/B Identifier value.

• Total Population - Total Population value.

• Converted - Number of population converted.

• Conversion Rate - A number column that conatins the ratio of the converted numbers against the total numbers.

• Chance of Being Better - Probability of being better. Between 0 and 1.

• Expected Improvement Rate - Expected improvement rate of conversion rate.

• Credible Interval Low - Lower bound of credible interval of the expected improvement rate.

• Credible Interval High - Upper bound of Credible interval of the expected improvement rate.

• Expected Loss Rate - Average loss rate of conversion rate by choosing A, assuming A was actually worse.

"Improvement Rate" View

"Improvement Rate" View displays the estimated distribution of the performance improvement rate.

This example is with Repeat By.

"Posteriors" View

"Posteriors" View displays posteriors of the conversion rate.

This example is with Repeat By.

"Prior" View

"Prior" View displays prior beta distribution derived from Expected Average of CR and Expected SD of CR.

This example is with Repeat By.

R Package

The "A/B Test - Bayesian" Analysis uses bayesAB R Package under the hood.

Exploratory R Package

For details about bayesAB usage in Exploratory R Package, please refer to the github repository