AB Testing and Benchmarking

To ensure reliable performance comparisons, we strongly suggest letting Froomle handle the traffic splitting and user segmentation.

Benchmarking through the Froomle platform ensures a "level playing field" where user assignment is consistent, and metrics for all groups—whether powered by Froomle or your own logic—are collected and processed in the exact same way.

Scenario 1: Froomle-Managed Split (Recommended)

In this setup, Froomle automatically assigns users to groups based on a configured percentage. There are two common ways to run this:

Option A: Full A/B Test (Model vs. Model)

Froomle compares two different internal strategies (e.g., "Trending" vs. "Personalized"). * Response: You receive items for both groups. * Action: Simply render the items returned. You do not need to know which model produced them.

Option B: Froomle vs. Internal Baseline

Froomle compares its recommendations (Group A) against your existing solution (Group B/Control). * Response: * Froomle Group: API returns personalized items. * Control Group: API returns an empty items array. * Action: Check if items were returned. If yes, render them. If no, render your fallback/baseline.

Group assignment is strictly based on the device_id. This ensures that anonymous users who later log in (or users without a user_id) remain in the same test group throughout their session on that device.

How it works (for both options):

  1. Request: Make a standard request without any group information.

  2. Tracking: Send standard impression/click events with the request_id. Froomle automatically links the event to the correct group on the backend.

// 1. Standard Request (No user_group needed)
POST /<environment>/recommendations/requests
{
  "user_id": "user-123",
  "page_type": "home",
  "lists": [
    {
      "list_name": "home_page_trending",
      "list_size": 10
    }
  ]
}

// 2. Response Example (Option B - Control Group)
// Froomle decides this user is in control, returns empty items.
{
  "request_id": "req-abc-123",
  "user_group": "control",
  "lists": [
    {
      "list_name": "home_page_trending",
      "items": []
    }
  ]
}

Scenario 2: Customer-Managed Split

Use this only if you operate your own A/B testing platform and must strictly control user assignment yourself.

How it works: 1. Request: You determine the group and must explicitly pass the user_group parameter in the request body. 2. Response: Froomle respects your decision. * If you send user_group: "froomle", it returns items. * If you send user_group: "customer", it returns empty items. 3. Tracking: You must pass the same user_group back in all tracking events (impressions, clicks) to ensure attribution.

The amount and names of the user groups are to be agreed upon with your Froomle account manager.

// 1. Explicit Request
POST /<environment>/recommendations/requests
{
  "user_id": "user-123",
  "page_type": "home",
  "user_group": "customer", // You decided this
  "lists": [
    {
      "list_name": "home_page_trending",
      "list_size": 10
    }
  ]
}

User Group Overview

Feature Froomle-Managed Split (Recommended) Customer-Managed Split

Who assigns the group?

Froomle

Customer (You)

Content in response?

Option A (Model vs Model): Items for all groups
Option B (Vs Baseline): Items (Froomle) or Empty (Control)

Items (Froomle group)
Empty (Customer group)

Pass user_group in Request?

No

Yes (Mandatory)

Pass user_group in Events?

No

Yes (Mandatory)

Event Requirements

For the benchmark to be valid, you must send tracking events for all groups, even if Froomle returned no items and you rendered a fallback.

  • Impressions: Send an impression event for the modules you displayed.

  • Clicks: Track clicks on the items.

If you strictly follow this pattern, the Froomle dashboard will automatically show side-by-side performance metrics for both strategies.

Related Topics