AB Testing and Benchmarking

To ensure reliable performance comparisons, we strongly suggest letting Froomle handle the traffic splitting and user segmentation.

Benchmarking through the Froomle platform ensures a "level playing field" where user assignment is consistent, and metrics for all groups—whether powered by Froomle or your own logic—are collected and processed in the exact same way.

For frontend SDK integrations, split ownership and rendering ownership are two separate decisions.

First decide who assigns the split:

  • Froomle-managed split

  • customer-managed split

Then, when you implement a benchmarked recommendation placement, decide how customer/control content is provided:

  • application-managed branch rendering: use the response user_group in your own UI code; render customer/control content for the customer/control group and returned Froomle items for the treatment/Froomle group

  • SDK-managed control-aware placement: keep the control content inside the SDK-managed placement so the SDK can preserve or replace it and auto-track both branches, including when Froomle-managed assignment resolves the customer/control group and returns an empty items array

Choose that rendering shape during recommendation implementation, not during the earlier page/business event rollout. For the SDK-specific ownership model and rendering shapes, see Benchmarking and A/B Testing with the SDK.

Scenario 1: Froomle-Managed Split (Recommended)

In this setup, Froomle automatically assigns users to groups based on a configured percentage. There are two common ways to run this:

Option A: Full A/B Test (Model vs. Model)

Froomle compares two different internal strategies (e.g., "Trending" vs. "Personalized").

  1. Response: You receive items for both groups.

  2. Action: Simply render the items returned. You do not need to know which model produced them.

Option B: Froomle vs. Internal Baseline

Froomle compares its recommendations (Group A) against your existing solution (Group B/Control).

  1. Response:

    • Treatment/Froomle group: API returns personalized items.

    • Customer/control group: API returns an empty items array.

  2. Action: Use the response user_group as the branch authority. If Froomle resolves the treatment/Froomle group, render returned Froomle items, or empty if the response is empty. If Froomle resolves the customer/control group, render the customer baseline/control content, regardless of whether the response contains returned items. In SDK-managed control-aware placement, provide identifiable control content to the SDK; when Froomle resolves the customer/control group, the SDK keeps the provided control content and can still auto-track it.

Group assignment is strictly based on the device_id. This ensures that anonymous users who later log in (or users without a user_id) remain in the same test group throughout their session on that device.

How it works (for both options):

  1. Request: Make a standard request without any group information.

  2. Tracking: Send standard impression/click events with the request_id. Froomle automatically links the event to the correct group on the backend.

// 1. Standard Request (No user_group needed)
POST /<environment>/recommendations/requests
{
  "user_id": "user-123",
  "page_type": "home",
  "lists": [
    {
      "list_name": "home_page_trending",
      "list_size": 10
    }
  ]
}

// 2. Response Example (Option B - Customer/control group)
// Froomle decides this user is in the customer/control group, returns empty items.
{
  "request_id": "req-abc-123",
  "user_group": "customer",
  "lists": [
    {
      "list_name": "home_page_trending",
      "items": []
    }
  ]
}

Frontend rendering must follow the resolved group, not merely the presence of returned items:

Resolved branch Returned items Expected rendering

Customer/control

Non-empty

Render customer/control content. Ignore returned Froomle items by default.

Customer/control

Empty

Render customer/control content.

treatment/Froomle

Non-empty

Render returned Froomle items.

treatment/Froomle

Empty

Render empty.

Scenario 2: Customer-Managed Split

Use this only if you operate your own A/B testing platform and must strictly control user assignment yourself.

What you need to provide:

  1. The benchmark group for the current visitor, using the group names agreed with your Froomle account team.

  2. The correct request shape for that group. Customer/control requests pass user_group explicitly, for example user_group: "customer". Treatment/Froomle requests omit user_group so Froomle can resolve the treatment branch through the normal version configuration.

  3. A frontend ownership choice for each placement. Use SDK-owned/control-aware placement when the customer/control content lives inside the SDK-managed placement and exposes stable item identity. Use application-owned/manual placement when you render the customer/control content yourself.

  4. Recommendation tracking for both branches, either through SDK-owned placement metadata or through manual impression/click events.

How it works:

  1. Request: You determine the group. Pass user_group only for the customer/control branch. Omit user_group for the treatment/Froomle branch.

  2. Response: Froomle respects your decision.

    • If the visitor is in the treatment/Froomle group, render returned Froomle items, or empty if the response is empty.

    • If you send user_group: "customer", render your customer/control content. Returned Froomle items must not be rendered for this branch by default.

  3. Tracking: Tracking events (impressions, clicks) must be attributable to the same branch. Customer/control events carry user_group. Treatment/Froomle events omit user_group and are attributed through the request_id. In SDK-owned placements the SDK handles this; in custom/manual flows you must follow the same final payload contract yourself.

The request/response split and the frontend rendering/tracking shape are separate concerns. Sending user_group: "customer" only tells Froomle which benchmark branch to use. It does not, by itself, make your existing customer-rendered cards automatically attributable by the browser SDK.

For browser SDK integrations, choose one of these tracking ownership models for the customer/control branch:

Frontend ownership model What you render Tracking requirement

SDK-managed control-aware placement

Your control item lives inside the SDK-managed recommendation placement.

Provide a stable item identity for the control item. In declarative DOM/script-tag mode this means data-froomle-id on each control card. In React this means passing raw control items through benchmark.controlItem or benchmark.controlItems together with getId(…​) and, when needed, getItemType(…​). The raw item itself must contain the fields your React card renderer needs. The SDK can then stamp the returned request_id and user_group and auto-track impressions/clicks.

Application-managed customer/control rendering

Your application renders or keeps control content outside the SDK-owned placement contract.

You own recommendation tracking for that branch. Send impression/click events manually with action_item, action_item_type, list_name, request_id, and user_group.

For declarative DOM/script-tag integrations, data-froomle-reco identifies the recommendation list/placement only. It is not the item identity.

If the customer/control branch returns empty items and the visible customer cards have no data-froomle-id, the browser SDK can keep the cards visible, but it cannot know which item was shown and cannot automatically send recommendation impressions/clicks for those cards.

For frontend browser SDK integrations, there is one important consent interaction:

  • Under consent level 0 or 1, customer-managed benchmark recommendation requests stay anonymous (for example device_id: "no-consent", version: "no-consent", no histories.pageviews). Customer/control requests still send the configured control user_group; treatment/Froomle requests omit request-root user_group. Request-local histories.exclude entries may still be sent for response deduplication.

  • Under consent level 0, the browser SDK does not send tracking events.

  • Under consent level 1, browser SDK events stay anonymous. Customer/control recommendation events keep the benchmark user_group; treatment/Froomle recommendation events omit user_group and rely on request_id attribution.

The amount and names of the user groups are to be agreed upon with your Froomle account team.

If you are unsure which benchmark setup fits your use case, contact your Froomle account team before implementing the placement. They can help define the benchmark contract, ownership model, and rollout plan.

 
// 1. Explicit customer/control request
POST /<environment>/recommendations/requests
{
  "user_id": "user-123",
  "page_type": "home",
  "user_group": "customer", // You decided this
  "lists": [
    {
      "list_name": "home_page_trending",
      "list_size": 10
    }
  ]
}

User Group Overview

Feature Froomle-Managed Split (Recommended) Customer-Managed Split

Who assigns the group?

Froomle

Customer (You)

Content in response?

Option A (Model vs Model): Items for all groups
Option B (Vs Baseline): Typically items for treatment/Froomle and empty for customer/control; frontend still follows user_group.

Typically items for user_group: "froomle" and empty for user_group: "customer"; frontend still follows user_group.

Pass user_group in Request?

No

Only for customer/control. Omit it for treatment/Froomle.

user_group in Events?

Not required when request_id links the event to the response; SDK-owned placements may include it when available.

Required for customer/control events. Omit it for treatment/Froomle events and rely on request_id attribution. SDK-owned placements handle this automatically.

Event Requirements

For the benchmark to be valid, you must send tracking events for all groups, including customer/control branches where you render your own content.

  • Impressions: Send an impression event for the modules you displayed.

  • Clicks: Track clicks on the items.

  • Attribution: The final event payload must contain enough recommendation metadata to attribute the event to the displayed item and benchmark branch.

For recommendation-related benchmark events, the most important implementation question is what the integration must provide so the SDK can build a complete final event payload:

Attribution field What you need to provide What the SDK can provide automatically

action_item

For customer/control cards, expose stable item identity to the SDK, for example data-froomle-id in declarative DOM/script-tag mode or benchmark.controlItem.getId(…​) / benchmark.controlItems.getId(…​) in React. React SDK-managed control items are raw render items, so they also need the fields your card renderer reads. If you do not use SDK-owned rendering, pass the shown item id manually.

For SDK-rendered Froomle items, the SDK uses the returned recommendation item id.

action_item_type

Provide the type when it is not obvious or not article, for example data-froomle-item-type in DOM or getItemType(…​) in React control adapters. Manual flows should pass it explicitly.

The SDK can reuse the type from SDK-owned item metadata when available.

list_name

Provide the placement/list name through the SDK request or DOM attribute, for example data-froomle-reco="article_similar". Manual flows should pass it explicitly.

In SDK-owned recommendation placements, the SDK copies the list name from the placement or request.

request_id

Do not invent or hard-code this value. In manual flows, store the response request_id and pass it with the impression/click events.

In SDK-owned placements, the SDK copies it from the Froomle recommendation response.

user_group

Configure the benchmark group correctly, for example setBenchmark({ mode: "customer-managed", currentGroup: "customer" }) or the equivalent script attributes. Manual customer/control flows should pass the resolved control group explicitly. Manual treatment/Froomle flows should omit user_group.

SDK-owned placements carry the resolved group into customer/control recommendation events and omit it for treatment/Froomle recommendation events.

When the browser SDK owns the rendered placement and the control item has stable identity, the SDK can populate these fields automatically. When the rendered control content is outside that SDK-owned contract, pass the fields yourself.

If you strictly follow this pattern, the Froomle dashboard will automatically show side-by-side performance metrics for both strategies.

Related Topics