Embedding-based Retrieval at Scribd

Categories: Data Analysis, Business & Sales, Productivity | Pricing: Free | Official Website ↗

Scribd's system uses embeddings and Elasticsearch to provide personalized content recommendations with low latency and faceted search capabilities.

This article details Scribd's approach to building an embedding-based retrieval system for content recommendations. The system addresses the challenge of personalizing recommendations for millions of users across a vast library of ebooks and audiobooks, moving beyond collaborative filtering to Transformer-based sequential recommendations. The core problem involves efficiently utilizing millions of user and item embeddings to serve recommendations online, reliably, and with low latency. The system is designed to support faceted search, allowing pre-filtering of results based on user-specific business rules like language or geographic availability, alongside similarity search in the embedding space. After evaluating several options, Elasticsearch was chosen for its ability to meet requirements such as pre-filtering, support for distance metrics, data updates without downtime, high availability, and monitoring capabilities. The implementation involves using Elasticsearch indices to store user and item embeddings and metadata, leveraging its script_score and _msearch APIs for efficient querying and multi-search operations.

Key Features

Embedding-based content retrieval
Faceted search capabilities
Personalized recommendations
Low-latency query responses
Support for exact distance metrics (dot product, euclidean distance)
Data updates without downtime
High availability
Monitoring and alerting integration

Pros

Enables highly personalized content recommendations
Supports complex faceted search requirements
Achieves low-latency query responses (p95 < 100ms)
Allows data updates without service interruption
Leverages a widely used tool (Elasticsearch) with strong community support and documentation

Cons

Elasticsearch lacks true in-memory partial updates, requiring batch updates
Deploying the cluster across multiple availability zones increases query latency
Requires careful shard configuration for optimal performance
Vespa, an alternative, had deployment and documentation challenges
Open Distro for Elasticsearch lacked prefiltering support

Use Cases

Generating personalized content recommendations on a homepage
Filtering search results based on multiple user-specific criteria
Serving millions of embeddings for real-time content discovery

Best For

Large content platforms
Companies needing personalized recommendation systems
Developers building scalable search and retrieval systems

Integrations: Elasticsearch, AWS EC2, Kibana, Datadog

Platforms: Web

Watch demo on YouTube ↗

View full Embedding-based Retrieval at Scribd profile on Tools-Radar | Browse Data Analysis tools | Alternatives to Embedding-based Retrieval at Scribd

Tools-Radar is a free directory of 10,000+ AI tools — discover, compare, and choose the right AI software for your needs. Visit tools-radar.com