How Froomle supports AB testing

This page describes how our platform supports AB testing.

Concepts

We support the following concepts:

  • User_group: This field describes which group a user is in and thus which party provides recommendations for this user.

    • This value is set by Froomle in the recommendations and should be maintained in following integration events (click_on_recommendation, impression, …​).

    • For Froomle recommendations, the value froomle is used for this field.

    • Froomle provides a fallback mechanism in case our personalized recommendation service is down or responding too slowly. In this case , such recommendations will contain the value fallback.

    • Alternatively you can specify the user_group in your recommendation request and agree with us how to handle the different versions

  • Version: This field describes the version of the recommendations served to the Froomle user group.

    • By default Froomle defines the field and uses different versions of its recommendation platform to find the optimal results through AB testing (e.g. Froomle_Baseline, Froomle_Experiment1, Froomle_Experiment2, …​).

    • When defining AB tests together with the customer, we define the number of versions and the version naming together.

    • Like the user_group, the version field should be maintained in integration events following the recommendations.

    • Optionally you can manage your own user to version assignments and in the recommendation request specify the version you want us to use in our recommendations.

AB Test scenarios

Compare different versions of Froomle configurations

  1. Single user group Froomle, set by Froomle

  2. Different versions of Froomle algorithms, set by Froomle

Compare Froomle configuration(s) with non-Froomle recommendations

These can be in-house 'baseline' recommendations, NO recommendations or third party recommendations.

  1. Froomle receives requests for ALL user groups (also for the groups for which we don’t need to provide recommendations!).

    1. Either the user groups are split by the Customer, either by Froomle.

    2. Froomle configures Froomle versions and the baseline recommendations in its own platform for the froomle user_group, and uses the agreed recommendations configuration for each user group.

    3. Froomle returns empty recommendations for the non-froomle user_groups.

  2. Customer can use the user_group variable returned by Froomle to show/hide the right recommendations.

  3. Integration events should be sent to Froomle for ALL user groups to enable us to measure & visualize (relative) results. Ideally this entails what is recommended (recommendation), what is shown (impression) and what is interacted/clicked (click_on_recommendation)

The below figure illustrates our capabilities:

Overview of AB test capabilities with or without external recommendation provider.