# A/B Testing and Benchmarking

> Current Froomle documentation · [View canonical page](https://docs.froomle.com/froomle-doc-main/v2/guide/topics/ab-testing-benchmarking.html)

Benchmarking compares recommendation strategies while keeping assignment and attribution consistent. Use Froomle-managed assignment when Froomle owns the split, or customer-managed assignment when the customer’s experimentation system owns it. Both use the same measurement contract.

> **IMPORTANT:**
> 
> Assignment ownership and rendering ownership are separate decisions. First decide who assigns the visitor to a branch, then decide which application component renders and tracks the content for that branch.

> **TIP:**
> 
> Agree the number and exact names of benchmark groups with your Froomle account team before implementation. Use those names consistently in configuration, response-driven rendering, and reporting. Whether an outbound request or event contains `user_group` depends on assignment ownership as defined below.
> 
> This guide uses **control** and **treatment** as the branch names. Unless different group names are agreed, the SDK configuration maps the control branch to `customer` and the treatment branch to `froomle`.

<a id="_choose_the_benchmark_shape_and_assignment_owner"></a>

## Choose the Benchmark Shape and Assignment Owner

First decide whether the benchmark compares Froomle models or uses existing control content. For control-vs-treatment, choose the assignment owner:

1.  **Froomle-managed model vs. model.** Froomle assigns the visitor and every branch renders returned recommendations. This is the simplest setup and uses the normal recommendation integration; control content and frontend control-item identity are not required.
    
2.  **Froomle-managed control vs. treatment.** Use this when existing editorial or internal content is the control and Froomle assigns the visitor. The frontend must explicitly enable `froomle-managed` benchmark behavior and give every control item a stable item ID and the correct type so both branches can be measured.
    
3.  **Customer-managed control vs. treatment.** Use this when the customer’s experimentation system assigns the visitor. The frontend must explicitly enable `customer-managed` behavior, supply the externally resolved current group, and give every control item a stable item ID and the correct type.
    

> **IMPORTANT:**
> 
> For a control-vs-treatment benchmark implemented with the frontend SDK, benchmark configuration and control-item identity are two separate requirements. You need both:
> 
> -   **Benchmark mode:** configure `froomle-managed` or `customer-managed`. For customer-managed assignment, also provide the current group resolved by the external experimentation system.
>     
> -   **Stable control identity:** provide an ID and item type for every displayed control item. Declarative DOM integrations should prefer direct `data-froomle-id` / `data-froomle-item-type`, and may use the SDK’s selector fallback when existing CMS markup cannot be changed. React integrations provide `getId(…​)` and, when needed, `getItemType(…​)` adapters.
>     
> 
> Setting only an ID selector does not enable A/B testing. Setting only benchmark mode does not make unidentified control content measurable.

<a id="_control_identity_preference"></a>

### Control Identity Preference

For SDK-managed declarative DOM placements, use identity sources in this order:

1.  **Direct DOM identity (preferred):** put `data-froomle-id` and, when needed, `data-froomle-item-type` on the control item itself.
    
2.  **Selector fallback:** when existing CMS markup cannot be changed, configure a selector and source attribute from which the SDK copies the control identity.
    

If neither approach can identify the rendered control item, the application must own its rendering and recommendation-event tracking instead.

<a id="_scenario_1_froomle_managed_split"></a>

## Scenario 1: Froomle-Managed Split

Froomle assigns visitors to the configured groups. The assignment is stable for the device, including anonymous visitors who later log in.

<a id="_option_a_full_ab_test_model_vs_model"></a>

### Option A: Full A/B Test (Model vs. Model)

Froomle compares two or more recommendation strategies. The response contains the items selected for the visitor’s assigned strategy, and the application renders the returned result.

<a id="_option_b_control_vs_treatment"></a>

### Option B: Control vs. Treatment

Existing editorial or internal content is the control; Froomle recommendations are the treatment. The response `user_group` is the branch authority.

| Resolved branch | Returned items | Expected rendering |
| --- | --- | --- |
| Control | Non-empty or empty | Render control content. Do not select a branch from item count alone. |
| Treatment | Non-empty | Render the returned Froomle items. |
| Treatment | Empty | Render the agreed empty or fallback state for the Froomle branch. |

For Froomle-managed assignment:

-   Do not pass `user_group` to select the request branch.
    
-   Preserve the returned `request_id` and resolved `user_group` through rendering.
    
-   Attribute impressions and clicks to the response that produced the rendered branch, but do not echo the returned `user_group` in those events.
    
-   Froomle assigns the version and group server-side.
    

<a id="_scenario_2_customer_managed_split"></a>

## Scenario 2: Customer-Managed Split

Use customer-managed assignment when the customer’s experimentation platform should decide the visitor’s branch.

For each request:

1.  Resolve the visitor’s branch in the customer experimentation system.
    
2.  Pass the agreed `user_group` for the control branch.
    
3.  Omit `user_group` for the treatment branch so normal Froomle version resolution applies.
    
4.  Render content according to the resolved branch, not according to whether returned items happen to be empty.
    
5.  Track impressions and clicks for the rendered branch with the same attribution decision.
    

Control recommendation events carry the agreed control `user_group`. Treatment events omit `user_group` and rely on the returned `request_id` for attribution.

<a id="_user_group_overview"></a>

## User Group Overview

| Concern | Froomle-managed | Customer-managed |
| --- | --- | --- |
| Who assigns the branch? | Froomle | Customer experimentation system |
| Pass `user_group` in the request? | No | Only for control; omit it for treatment |
| Rendering authority | Returned `user_group` | Customer decision, represented by the agreed request contract |
| Control events | Omit `user_group`; preserve request attribution and let the Events API assign it | Include the agreed control `user_group` |
| Treatment events | Omit `user_group`; attribute through `request_id` | Omit `user_group` and attribute through `request_id` |

> **NOTE:**
> 
> This matrix is implementation guidance for manual/custom API calls and bare programmatic SDK rendering. For SDK-owned declarative DOM and React placements, the SDK applies it automatically: keep rendering through the supported SDK primitives and do not duplicate branch-attribution logic in application code.

<a id="_event_requirements"></a>

## Event Requirements

A benchmark is valid only when shown content is measured for every branch.

-   Send an `impression` for every displayed recommendation or control item included in the benchmark.
    
-   Send a `click_on_recommendation` when a displayed recommendation or control item is clicked.
    
-   Keep item, placement, request, and branch attribution aligned with what the visitor actually saw.
    

| Attribution field | Shared contract |
| --- | --- |
| `action_item` | Stable ID of the item that was shown or clicked. |
| `action_item_type` | Item type used by the synchronized catalog. |
| `list_name` | Placement/list being benchmarked. |
| `request_id` | ID returned by the recommendation response. Never invent or hard-code it. |
| `user_group` | Include only the externally assigned control group in a customer-managed split. Omit it for customer-managed treatment and for every Froomle-managed branch. |

When both branches follow this attribution contract, Froomle dashboards can report their performance side by side using the same measurement pipeline.

<a id="_frontend_sdk_control_vs_treatment_checklist"></a>

### Frontend SDK Control-vs-Treatment Checklist

Before enabling traffic for a control-vs-treatment benchmark, verify all of the following:

-   The placement explicitly configures `froomle-managed` or `customer-managed` benchmark mode. An ID selector alone is not benchmark configuration.
    
-   Customer-managed assignment supplies the externally resolved current group; it is not hard-coded to one group for every visitor.
    
-   Every control item has one stable ID and the correct item type. For declarative DOM, prefer direct `data-froomle-id` and `data-froomle-item-type`; use selector fallback only when existing CMS markup cannot be changed.
    
-   The control branch keeps control content and the treatment branch renders returned Froomle content according to the resolved assignment.
    
-   Both branches produce recommendation impressions and clicks with the same `list_name` and returned `request_id` contract.
    
-   `window.FroomleFrontendSdkRuntime.diagnostics.benchmark` shows the expected mode and resolved branch. For selector fallback, `diagnostics.benchmark.controlIdentity.lastResults` confirms whether each placement used an explicit ID, a selector-derived ID, or failed as missing or ambiguous.
    

<a id="_consent"></a>

## Consent

Benchmark assignment must not override consent rules. Anonymous requests remain anonymous, and tracking events are sent only at a consent level that permits them. Keep only the routing information needed to resolve the agreed benchmark branch.

<a id="_implement_the_benchmark"></a>

## Implement the benchmark

| Method | Implementation guide |
| --- | --- |
| Frontend SDK | [Benchmarking and A/B Testing with the SDK](https://docs.froomle.com/froomle-doc-main/v2/js_lib/jslib.html#BenchmarkingSdk) |
| Manual / custom integration | [Manual recommendation benchmarking](https://docs.froomle.com/froomle-doc-main/v2/guide/manual-integration/recommendations.html#ManualRecommendationBenchmarking) and [Manual event integration](https://docs.froomle.com/froomle-doc-main/v2/guide/manual-integration/events.html) |

<a id="_related_topics"></a>

## Related Topics

-   [Performance & Metrics](https://docs.froomle.com/froomle-doc-main/v2/guide/topics/performance-metrics.html)
    
-   [Event model and tracking contract](https://docs.froomle.com/froomle-doc-main/v2/guide/topics/tracking-events.html)
    
-   [Recommendation request model](https://docs.froomle.com/froomle-doc-main/v2/guide/topics/recommendation-requests.html)