Synchronizing Item Metadata

Froomle needs information on your item catalog (articles/videos/blogposts / …), including metadata, for several reasons:

  • to know which items to recommend.

  • to share this metadata back to you in our recommendation response. This could save you some time fetching this information from your own back-end.

  • to allow for filters/constraints on the recommended items, e.g. we can limit the age of articles to recommend

  • to train our algorithms to give your users the best recommendations

At the start of the integration process, we expect to receive at least the current catalog. If possible, also older items can be sent to us (e.g. up to a few months old).

After this first data upload, we expect to receive updates on new or updated items quickly. For fast-changing content like articles or media, updates should be (near-)real-time in order to provide the most relevant recommendations at all times.

It is very important that we receive ALL of your items, items which are missing from our database will negatively impact our recommendation quality.

Item Identifiers & Item Types

Item types can be used to distinguish items of substantially different kinds, such as news articles from videos, or physical products from services. However, smaller differences such as different kinds of household appliances do not typically warrant separate item types. Usually our customers have no more than a few item types at a time.

Fundamentally different kinds of items generally require different kinds of metadata. Therefore, each item type can have its own metadata structure.

If there is no such distinction between items in your system, then there will be just one item type, and you don’t have to worry about item types at all.

As we mentioned before, each item should have a string identifier. Such an identifier alone doesn’t fully identify an item, however. Items are fully identified by the pair (item_type, item_id). As such, two items of a different item type with the same item ID are considered unrelated.

We prefer you communicate these item types explicitly wherever possible. If you are unable to communicate item types to Froomle, contact us to discuss if they can be determined from context.

Preparing item metadata

Before you send your metadata to Froomle we ask you to format it according to one of our Item Metadata Templates so we can properly ingest it. If necessary we are flexible to perform translations from your structure to our standard, provided you clearly describe which mapping needs to be done. Before sending us your item metadata it is important that it is prepared correctly, so we can parse and ingest it without any issues. To simplify the preparation of your metadata we have a limited number of data templates that you can adhere to. Each template corresponds to a specific use case of the Froomle platform. While not all fields within a template are required, we highly encourage you to provide as many fields as possible within the chosen template, as it will improve your personalization performance.

Some fields in our templates are nested which doesn’t map directly to some of our accepted formats (e.g. CSV). For these formats we ask you to include these nested fields as their serialized JSON representation.

Item Integration Connectors

API Integration

We summarize the most important actions for managing your metadata using the Items API, for more information see API Reference.

You can manage your metadata using our API, with the following methods:

  • PUT : to add a new item or update an existing item.

  • DISABLE: to ensure that an item will no longer be recommended. The item will remain in our database and can be still used in our models.

  • DELETE : to delete items from our database. Only use this option if the items are actually deleted from the website.

  • GET: retrieve the metadata from an existing item. Useful to check if all data is persisted like expected in our database.

SFTP Upload

Our example on data upload describes how to upload item metadata using SFTP.

Feed Read (beta)

If your item catalog is available on an endpoint we can access, Froomle can fetch the data. We support API or an RSS or XML file. If you want to use this option you need to specify out the location, format & mapping rules of the data to our standard Based on this we can modify our (python) script to fetch & ingest the data.

⚠️ The Feed read by Froomle integration option is in beta, therefore it involves custom development and timeline. If you are interested in this feature, please contact the Froomle support team.

Item Metadata templates

For our most popular use cases we have created more specialized templates which extend the base template with extra information tailored to each use case. If your use case maps on one of these specialized templates you can leverage the extra level of detail they provide and improve your personalization experience even more.

  • News Template

  • E-commerce Template

  • Social Network Template

Field

Description

environment*

string

Required. The environment to which the item belongs.

item_id*

string

Required. Available for filtering. The unique identifier of the item.

item_type*

string

Required. Available for filtering. Items is the collection of all different entities on your platform. Item type allows you to further specify which type of entity this item represents. E.g. article, video, profile, category, …​ In the recommendation request a filter can be specified to only return items of a certain type.

language

string

Recommended. Available for filtering. The metadata’s main language, if an item’s metadata is delivered in multiple languages specify the most dominant language here. The provided value should be a ISO 639-1 standard language code like en, fr, nl, en-us, nl-be, …​ In the recommendation request a filter can be specified to only return items of a certain language.

title

string

Recommended. The title of the item. If your item is available in multiple languages use the title_map field instead. Metadata like this allows Froomle to further optimize the recommendations and provide demo functionality in our portal.

title_map

map<string, string>

Optional. A mapping of language_code → title. Use this field when your title has multiple translations. Keys in this map should be ISO 639-1 standard language codes like en, fr, nl, en-us, nl-be, …​

description

string

Recommended. The description of the item. If your item contains translations use the description_map field instead. Metadata like this allows Froomle to further optimize the recommendations.

description_map

map<string, string>

Optional. A mapping of language_code → description. Use this field when your description has multiple translations. Keys in this map should be ISO 639-1 standard language codes like en, fr, nl, en-us, nl-be, …​

uri

string

Recommended. The uri of the item. If your item has a resource identifier unique to each language use the uri_map field instead. Recommended to provide demo functionality in our portal.

uri_map

map<string, string>

Recommended. A mapping of language_code → uri. Use this field when your item has resource identifiers unique to each language. Keys in this map should be ISO 639-1 standard language codes like en, fr, nl, en-us, nl-be, …​

categories

list<list<string>>

Recommended. Available for filtering. As items often belong to a hierarchical category structure you can specify each path within the structure that can be used to reach the current item. Categories within a path should be listed from most general to most specific. In the recommendation request a filter can be specified to only return items from the given categories. Recommended to provide demo functionality in our portal.

tags

list<string>

Recommended. Available for filtering. Tags are often associated with items to provide extra context. While this field is optional it should be clear that tags carry valuable information for personalization algorithms. In the recommendation request a filter can be specified to only return items that carry a certain tag. Recommended to provide demo functionality in our portal.

images

list<string>

Recommended. If your item contains one or more images you can optionally provide their URLs using this field. Recommended to provide demo functionality in our portal.

parent

string

Optional. When your items are structured in a hierarchy you can supply the parent identifier using this field.

ancestry_level

string

Optional. When your items are structured in a hierarchy you can specify the level of the provided item in the hierarchy. Valid values are (from highest to lowest in the hierarchy): group, item and model.

item_attributes

map<string, string>

Optional. A mapping of any item attributes to values. We can return these attributes in our recommendation response and use them for analysis purposes.

publication_datetime*

string

Required. ISO-8601 formatted timestamp string of the publication datetime of the news item.

end_datetime

string

Optional. ISO-8601 formatted timestamp string of the end of publication datetime of the news item.

access_type

string

Optional. When access to an article is restricted for certain users, use this field to specify the restriction. Valid values are: FREE, METERED and PAID.

Field

Description

environment*

string

Required. The environment to which the item belongs.

item_id*

string

Required. Available for filtering. The unique identifier of the item.

item_type*

string

Required. Available for filtering. Items is the collection of all different entities on your platform. Item type allows you to further specify which type of entity this item represents. E.g. article, video, profile, category, …​ In the recommendation request a filter can be specified to only return items of a certain type.

language

string

Recommended. Available for filtering. The metadata’s main language, if an item’s metadata is delivered in multiple languages specify the most dominant language here. The provided value should be a ISO 639-1 standard language code like en, fr, nl, en-us, nl-be, …​ In the recommendation request a filter can be specified to only return items of a certain language.

title

string

Recommended. The title of the item. If your item is available in multiple languages use the title_map field instead. Metadata like this allows Froomle to further optimize the recommendations and provide demo functionality in our portal.

title_map

map<string, string>

Optional. A mapping of language_code → title. Use this field when your title has multiple translations. Keys in this map should be ISO 639-1 standard language codes like en, fr, nl, en-us, nl-be, …​

description

string

Recommended. The description of the item. If your item contains translations use the description_map field instead. Metadata like this allows Froomle to further optimize the recommendations.

description_map

map<string, string>

Optional. A mapping of language_code → description. Use this field when your description has multiple translations. Keys in this map should be ISO 639-1 standard language codes like en, fr, nl, en-us, nl-be, …​

uri

string

Recommended. The uri of the item. If your item has a resource identifier unique to each language use the uri_map field instead. Recommended to provide demo functionality in our portal.

uri_map

map<string, string>

Recommended. A mapping of language_code → uri. Use this field when your item has resource identifiers unique to each language. Keys in this map should be ISO 639-1 standard language codes like en, fr, nl, en-us, nl-be, …​

categories

list<list<string>>

Recommended. Available for filtering. As items often belong to a hierarchical category structure you can specify each path within the structure that can be used to reach the current item. Categories within a path should be listed from most general to most specific. In the recommendation request a filter can be specified to only return items from the given categories. Recommended to provide demo functionality in our portal.

tags

list<string>

Recommended. Available for filtering. Tags are often associated with items to provide extra context. While this field is optional it should be clear that tags carry valuable information for personalization algorithms. In the recommendation request a filter can be specified to only return items that carry a certain tag. Recommended to provide demo functionality in our portal.

images

list<string>

Recommended. If your item contains one or more images you can optionally provide their URLs using this field. Recommended to provide demo functionality in our portal.

parent

string

Optional. When your items are structured in a hierarchy you can supply the parent identifier using this field.

ancestry_level

string

Optional. When your items are structured in a hierarchy you can specify the level of the provided item in the hierarchy. Valid values are (from highest to lowest in the hierarchy): group, item and model.

item_attributes

map<string, string>

Optional. A mapping of any item attributes to values. We can return these attributes in our recommendation response and use them for analysis purposes.

price*

double

Required. The retail/listing price of the item.

currency_code*

string

Required. ISO-4217 compliant triple-character currency code. E.g. EUR, USD, GBP, …​

costs

map<string, double>

Optional. Extra cost-related monetary amounts like: VAT, profit, purchase price, …​

stock_state

string

Optional. The current state of the product’s stock. Valid values are: IN_STOCK, OUT_OF_STOCK, PREORDER and BACKORDER.

available_stock

integer

Optional. Current number of articles in stock for this item.

original_price

double

Optional. The original price of a discounted product.

discount_percentage

double

Optional. The percentage of discount for discounted products. When both price and original_price are specified we can automatically compute this field if you don’t specify it. When you also provide this field along with the original_price and price fields, the provided discount percentage value will also be validated using both price values.

Field

Description

environment*

string

Required. The environment to which the item belongs.

item_id*

string

Required. Available for filtering. The unique identifier of the item.

item_type*

string

Required. Available for filtering. Items is the collection of all different entities on your platform. Item type allows you to further specify which type of entity this item represents. E.g. article, video, profile, category, …​ In the recommendation request a filter can be specified to only return items of a certain type.

language

string

Recommended. Available for filtering. The metadata’s main language, if an item’s metadata is delivered in multiple languages specify the most dominant language here. The provided value should be a ISO 639-1 standard language code like en, fr, nl, en-us, nl-be, …​ In the recommendation request a filter can be specified to only return items of a certain language.

title

string

Recommended. The title of the item. If your item is available in multiple languages use the title_map field instead. Metadata like this allows Froomle to further optimize the recommendations and provide demo functionality in our portal.

title_map

map<string, string>

Optional. A mapping of language_code → title. Use this field when your title has multiple translations. Keys in this map should be ISO 639-1 standard language codes like en, fr, nl, en-us, nl-be, …​

description

string

Recommended. The description of the item. If your item contains translations use the description_map field instead. Metadata like this allows Froomle to further optimize the recommendations.

description_map

map<string, string>

Optional. A mapping of language_code → description. Use this field when your description has multiple translations. Keys in this map should be ISO 639-1 standard language codes like en, fr, nl, en-us, nl-be, …​

uri

string

Recommended. The uri of the item. If your item has a resource identifier unique to each language use the uri_map field instead. Recommended to provide demo functionality in our portal.

uri_map

map<string, string>

Recommended. A mapping of language_code → uri. Use this field when your item has resource identifiers unique to each language. Keys in this map should be ISO 639-1 standard language codes like en, fr, nl, en-us, nl-be, …​

categories

list<list<string>>

Recommended. Available for filtering. As items often belong to a hierarchical category structure you can specify each path within the structure that can be used to reach the current item. Categories within a path should be listed from most general to most specific. In the recommendation request a filter can be specified to only return items from the given categories. Recommended to provide demo functionality in our portal.

tags

list<string>

Recommended. Available for filtering. Tags are often associated with items to provide extra context. While this field is optional it should be clear that tags carry valuable information for personalization algorithms. In the recommendation request a filter can be specified to only return items that carry a certain tag. Recommended to provide demo functionality in our portal.

images

list<string>

Recommended. If your item contains one or more images you can optionally provide their URLs using this field. Recommended to provide demo functionality in our portal.

parent

string

Optional. When your items are structured in a hierarchy you can supply the parent identifier using this field.

ancestry_level

string

Optional. When your items are structured in a hierarchy you can specify the level of the provided item in the hierarchy. Valid values are (from highest to lowest in the hierarchy): group, item and model.

item_attributes

map<string, string>

Optional. A mapping of any item attributes to values. We can return these attributes in our recommendation response and use them for analysis purposes.

publication_datetime*

string

Required. ISO-8601 formatted timestamp string of the publication datetime of the content item.

end_datetime

string

Optional. ISO-8601 formatted timestamp string of the end of publication datetime of the content item.

owner

string

Optional. User identifier of the owner of the content.

privacy

string

Optional. Privacy setting applied to the content piece which controls the visibility of the item. Valid values are: me, friends, friends_of_friends and public.

News Metadata Example

Below you can find an example of how an article should be structured when synchronizing it to the Froomle platform.

  • Simple Example

  • Complete Example

{
    "environment": "{environment}",
    "item_id": "client_item_id_001",
    "item_type": "article",
    "title": "Froomle title",
    "description": "Lorem ipsum ... Article/product full text",
    "uri": "https://www.froomle.com/client_item_id_001",
    "categories": [
        "/Sports/Football",
        "/News/National"
    ],
    "tags": [
        "Red Devils",
        "Martinez",
        "Heizel",
        "Qatar2022"
    ],
    "images": [
        "https://www.froomle.com/content/client_item_id_001.jpg"
    ],
    "publication_datetime": "2021-07-29T13:02:03+02:00",
}
{
    "environment": "{environment}",
    "item_id": "client_item_id_001",
    "item_type": "article",
    "language": "en-us",
    "title": "Froomle title",
    "description": "Lorem ipsum ... Article/product full text",
    "uri": "https://www.froomle.com/client_item_id_001",
    "categories": [
        "/Sports/Football",
        "/News/National"
    ],
    "tags": [
        "Red Devils",
        "Martinez",
        "Heizel",
        "Qatar2022"
    ],
    "images": [
        "https://www.froomle.com/content/client_item_id_001.jpg",
        "https://www.froomle.com/content/client_item_id_001_2.jpg"
    ],
    "publication_datetime": "2021-07-29T13:02:03+02:00",
    "end_datetime" : "2021-08-15T13:02:03+02:00",
    "access_type": "FREE", // or METERED/PAID
    "item_attributes": {
        "is_dpa": 1,
        "url_test": "https://www.froomle.com/client_item_id_004",
        "img_test": "https://www.froomle.com/content/client_item_id_003.jpg",
    }
}

Additional information

Applying filters to recommendation requests

Using the Configurations API (which can be managed by you or by Froomle) constraints can be configured and saved in advance to apply them during a recommendation request in the future. A multitude of configurations can be stored at the same time allowing you to dynamically change the look and feel of your requested recommendations depending on the context of the recommendations.

Examples of such filters are:

  • Only recommend items within the same category as the current item.

  • Do not recommend products with a price lower than €20.

  • Only recommend products that are currently discounted.

  • Do not recommend news articles with a publication time older than 24 hours.

Content based recommendations

Contrary to collaborative filtering (use browsing behavior of other users for recommendations), any content based recommendation techniques need metadata to work. If you want to test these algorithms or use specific modules that require these algorithms, we need metadata. This can be labels/categories of any items but also mood, complexity, length of articles.

Timestamps

It can be useful to include timestamps as part of the item metadata. Commonly, time restrictions on the recommendability of an item apply. For instance, we may not be allowed to recommend news articles that are too old, or a soon to be released product may not be allowed to be shown before a certain time.

When sharing timestamps between different systems – such as between yours and ours – the time offset is often a cause for confusion. A timestamp without this information (either by context or explicitly given) does not uniquely define a point in time.

If you provide us with timestamps, please either:

  • Provide a time offset with it, such that it is a fully qualified timestamp. With the ISO-8601 notation, the preferable format is:

    yyyy-mm-ddThh:mm:ss+|-hh:mm

    For example:

    2019-09-06T12:00:00+02:00

    This way no assumptions need to be made.

  • Provide us timestamps in UTC+00. For any timestamp without any time zone or offset information, we assume UTC+00.

Improving analysis and reporting

Numbers only tell us so much. Interesting relations and correlations can be uncovered more easily when there is metadata that you can link to each item. We extensively use metadata like title, tags, categories, …​ during our own analysis to gain new insights and plan our next move in your personalization story. When reporting back to you we incorporate as much information as possible to provide clear reporting that also explains why and isn’t just limited to a single improved metric.