Resilience and Fallbacks

Froomle is a managed service hosted on the internet, subject to the same availability constraints as any online system. Your website must remain functional and performant even when Froomle is unavailable, degraded, or slow to respond.

Why resilience matters

Froomle provides personalized recommendations, which enhance user experience but are not critical to core functionality. A recommendation outage should never cascade into a broken website. Users expect your pages to load and function regardless of third-party service status.

Common failure modes:

Network timeouts — Connection to Froomle succeeds but response takes >5 seconds (user abandons page)
HTTP errors — 5xx server errors, circuit breaker trips, rate limits
DNS failures — Transient or prolonged; common during infrastructure migrations
Cascading failures — Your queue backs up, connections pool exhausts, application becomes unresponsive
Regional outages — Froomle is hosted on Google Cloud Platform. Any outages there will affect our service’s availability.

Design principles

Fail open, not closed. If Froomle is unavailable, show your page without recommendations rather than a 500 error.

Always set timeouts. Froomle can impose time constraints on our recommendation generation, returning either an error or an empty response if timeliness cannot be met. However, this timeout cannot account for network latency, nor is it fail-safe in case of erratic behavior in the Froomle backend. Therefore, we strongly suggest you implement timeouts on your end when requesting Froomle recommendations (500ms to 2s depending on your SLA). Requests without timeouts can hang indefinitely, blocking page loads.

Degrade gracefully. Fallback strategies, in order of preference: 1. Show editorial or curated recommendations 2. Show no recommendations, but show the page 3. Serve cached recommendations (from Redis or local cache)

Monitor failures. Track timeouts, error rates, and latency percentiles. Alert when anomalies appear.

Implementation patterns

Client-side timeout (JavaScript)

Always set a timeout when calling Froomle’s API from your frontend.

const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(), 2000); // 2 second timeout

try {
  const response = await fetch('https://froomle.com/api/recommendations', {
    method: 'POST',
    signal: controller.signal,
    body: JSON.stringify(request)
  });
  clearTimeout(timeoutId);

  if (!response.ok) {
    console.warn('Froomle returned:', response.status);
    // Show cached or fallback recommendations
    showFallbackRecommendations();
    return;
  }

  const recommendations = await response.json();
  renderRecommendations(recommendations);
} catch (error) {
  clearTimeout(timeoutId);
  if (error.name === 'AbortError') {
    console.warn('Froomle request timed out');
  } else {
    console.warn('Froomle request failed:', error);
  }
  showFallbackRecommendations();
}

Server-side timeout (backend API call)

Set a timeout on your backend HTTP client when fetching recommendations server-side.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# Retry logic for transient failures
session = requests.Session()
retry_strategy = Retry(
  total=2,
  status_forcelist=[429, 500, 502, 503, 504],
  allowed_methods=["GET", "POST"],
  backoff_factor=1
)
adapter = HTTPAdapter(max_retries=retry_strategy)
session.mount("http://", adapter)
session.mount("https://", adapter)

try:
  response = session.post(
    'https://froomle.com/api/recommendations',
    json=request_payload,
    timeout=2.0,  # 2 second timeout
    headers={'Authorization': f'Bearer {token}'}
  )
  response.raise_for_status()
  return response.json()
except requests.exceptions.Timeout:
  logger.warning('Froomle request timed out')
  return get_cached_recommendations()
except requests.exceptions.RequestException as e:
  logger.warning(f'Froomle request failed: {e}')
  return get_cached_recommendations()

Caching layer

Caching is a delicate and challenging affair in the context of dynamic, personalized content.

Your cache is also fallible. Cache backends (Redis, Memcached) can crash, evict data, or lose consistency. Cache-based fallbacks are not guaranteed—have a secondary fallback (editorial recommendations, empty state) if cache also fails.
Cache is not personalized. By definition, you’re serving the same recommendations to different users if you fall back to cache. This is acceptable for transient failures but should not become your long-term strategy. Notify your team if fallback rates spike; it may indicate a Froomle issue.
Feedback on cached recommendations can confuse Froomle’s models. If you serve cached (stale) recommendations to a user and they click, that feedback signal goes back to Froomle. Over time, Froomle’s algorithms may learn to recommend content based on stale data or past context, degrading quality. This is especially problematic if you cache recommendations across different users or sessions.

To prevent stale cache from polluting Froomle’s training data, we recommend periodically fetching recommendations for an anonymous user (e.g., device_id: "cache") for your cache. When a user clicks on a recommendation from this cache, tag it with user_group: "cache" in your feedback payload so Froomle’s team can identify and filter these signals if needed.

def refresh_cache():
  """Fetch fresh recommendations for anonymous user to serve as fallback."""
  try:
    recs = fetch_recommendations_for_user(
      user_id="anonymous",
      device_id="cache",
      # Optional: other params like user_group for context
    )
    cache.set("fallback_recommendations", recs, ttl=3600)
  except Exception as e:
    logger.error(f'Failed to refresh fallback cache: {e}')

def handle_click_on_recommendation(item_id, from_cache=False):
  """Send feedback to Froomle."""
  payload = {
    "item_id": item_id,
    "device_id": "cache" if from_cache else get_current_user_device_id(),
    "action": "click"
  }
  # If the recommendation came from cache, tag it
  if from_cache:
    payload["user_group"] = "cache"

  send_feedback_to_froomle(payload)

Note: Caching is not yet built into the Froomle SDK. You’ll need to implement cache management and feedback tagging manually on your side. Contact Froomle support if you want caching semantics formalized in the SDK.

Circuit breaker pattern

Stop calling Froomle if it’s clearly broken; restore calls gradually when it recovers.

from pybreaker import CircuitBreaker

froomle_breaker = CircuitBreaker(
  fail_max=10,  # Open after 10 failures
  reset_timeout=60,  # Try again after 60 seconds
  listeners=[  # Add logging
    lambda cb, *args: logger.info(f'Circuit breaker: {cb.name} state changed')
  ]
)

def get_recommendations(user_id):
  try:
    return froomle_breaker.call(fetch_from_froomle, user_id)
  except CircuitBreaker.CircuitBreakerListener:
    logger.warning('Froomle circuit breaker is open')
    return get_cached_recommendations()

Monitoring and alerting

Track these metrics to catch failures early:

Request latency (p50, p95, p99) — Alert if p99 exceeds your timeout
Error rate — Alert if >1% of requests fail
Timeout rate — Alert if >0.1% timeout
Cache hit rate — Should be high if fallback is working
Circuit breaker state — Alert if circuit opens

Example Prometheus queries:

# Request latency (95th percentile)
histogram_quantile(0.95, froomle_request_duration_seconds_bucket)

# Error rate
rate(froomle_request_errors_total[5m])

# Timeout rate
rate(froomle_request_timeouts_total[5m])

SLA expectations

Froomle targets high availability, but like any system, maintenance windows, incidents, and regional outages occur. Plan for:

99.5-99.9% availability — Expect 3-5 hours of downtime per year
Transient errors (1-2%) — Normal; handled by retries
Latency spikes — During traffic peaks or deploys
Regional degradation — Rare but possible

Your fallback strategy should handle these gracefully. Never assume Froomle is always available.

Fallback recommendation strategies

Editorial picks

Curate a set of evergreen recommendations shown when personalization fails.

{
  "items": [
    {"item_id": "top_article_2024", "score": 1.0},
    {"item_id": "trending_product_jan", "score": 0.9},
    {"item_id": "editor_pick_sports", "score": 0.85}
  ]
}

Random sampling

Show a random subset of popular or recent items.

Empty state

Show no recommendations but keep the page functional. Many users won’t notice.

Testing resilience

Inject failures during testing to verify fallbacks work:

Mock Froomle endpoint to return 500, 503, or timeout
Add network latency (tc, toxiproxy, or browser dev tools)
Kill Froomle connection mid-response
Test cache miss + Froomle down scenario

Verify that your website loads, renders, and functions without errors.