Open Source - Recommendation System

X For You Feed Algorithm

github.com/xai-org/x-algorithm

* 19.5k Stars | 3.4k Forks | 227 Watchers
recommendation-system machine-learning transformer ranking Rust Python

Core Components

4
Main Components
7
Pipeline Stages
15
Engagement Predictions
12
Filters

"We have eliminated every single hand-engineered feature and most heuristics from the system. The Grok-based transformer does all the heavy lifting."

* Home Mixer

Orchestration layer that assembles the For You feed using CandidatePipeline framework. Coordinates all stages from hydration to selection.

Rust - Orchestration

* Thunder

In-memory post store for in-network posts. Sub-millisecond lookups from accounts you follow. Consumes Kafka events, auto-trims old posts.

Rust - In-Network

* Phoenix

ML engine for out-of-network content. Two-tower retrieval + Grok-based transformer ranking with candidate isolation.

Rust - ML Engine

* Grox

Content understanding pipeline with classifiers, embedders, and task execution for spam detection, categorization, PTOS policy.

Rust - Content Analysis

Pipeline Stages

1

Query Hydration

Fetch user context: engagement history, following list, preferences, starter packs, impression bloom filters

2

Candidate Sourcing

Thunder (in-network, followed accounts) + Phoenix Retrieval (out-of-network, ML-discovered)

3

Candidate Hydration

Enrich with post data, author info, media entities, video duration, subscription status

4

Pre-Scoring Filters

Remove duplicates, old posts, self-posts, blocked authors, muted keywords, seen content

5

Scoring

Phoenix ML predictions + weighted combination + author diversity attenuation

6

Selection

Sort by final score, select top K candidates

7

Post-Selection

VFFilter (deleted/spam/violence/gore) + conversation dedup

Scoring Algorithm

Phoenix predicts 15 engagement probabilities. The weighted scorer combines them into a final score:

Final Score = SUM (weight_i x P(action_i))

Positive actions (like, repost, share) have positive weights. Negative actions (block, mute, report) have negative weights, pushing down undesirable content.

Engagement Predictions

P(favorite)
P(reply)
P(repost)
P(quote)
P(click)
P(profile_click)
P(video_view)
P(photo_expand)
P(share)
P(dwell)
P(follow_author)
P(not_interested)
P(block_author)
P(mute_author)
P(report)

FAQ

What is the X For You Feed Algorithm?
Open source recommendation system powering the For You feed. Combines in-network posts (via Thunder) and out-of-network posts (via Phoenix), ranked using a Grok-based transformer.
What programming languages?
Rust (57.4%) for performance-critical components like Home Mixer, Thunder, Phoenix, Grox. Python (42.6%) for ML tooling and inference scripts.
Relationship to Grok-1?
Phoenix transformer is ported from Grok-1 open source release by xAI, adapted specifically for recommendation use cases to predict engagement probabilities.
How does scoring work?
Phoenix predicts 15 engagement probabilities. Weighted Scorer computes: Final Score = SUM (weight_i x P(action_i)). Positive actions have positive weights, negative actions have negative weights.
What is candidate isolation?
During transformer inference, candidates cannot attend to each other - only to user context. This ensures scores don't depend on which posts are in the batch, making them consistent and cacheable.
What are the two candidate sources?
1. Thunder (In-Network): Posts from followed accounts, sub-millisecond in-memory lookups. 2. Phoenix Retrieval (Out-of-Network): ML-discovered posts via two-tower similarity search.
What filtering stages?
Pre-scoring: 10 filters (duplicates, age, self-posts, muted keywords, blocked authors, etc.). Post-selection: VFFilter (spam/violence) and DedupConversationFilter.
Key design decisions?
No hand-engineered features, candidate isolation for cacheable scores, hash-based embeddings, multi-action prediction (15 types), composable pipeline architecture.

Glossary

Recommendation System

ML system retrieving, ranking, filtering content for personalized user feeds.

Transformer Model

Grok-based neural network architecture for predicting engagement probabilities.

Two-Tower Model

Neural retrieval with separate user and candidate towers for similarity search.

Candidate Isolation

Attention masking ensuring candidates only attend to user context for cacheable scores.

In-Network Posts

Content from accounts the user follows, retrieved via Thunder.

Out-of-Network Posts

ML-discovered content via Phoenix similarity search across global corpus.

How to Use This Repository

Understand the Architecture

Review README.md for overall architecture. Four main components: Home Mixer (orchestration), Thunder (in-network), Phoenix (ML), Grox (content understanding). Read component READMEs for details.

Run End-to-End Inference

Execute phoenix/run_pipeline.py as single entry point: retrieval -> ranking from exported checkpoints. Pre-trained mini Phoenix model (~3GB Git LFS) included for out-of-box inference.

Extend the Pipeline

Use candidate-pipeline framework to add new sources, hydrations, filters, scorers. Implements traits: Source, Hydrator, Filter, Scorer, Selector, SideEffect with parallel execution.

Implement Custom Scoring

Add new scorers by implementing the Scorer trait. Receive candidates and user context, return scores for weighted combination and author diversity adjustment.

Configure Ads Blending

Use home-mixer/ads module for ad injection and positioning. Includes brand-safety tracking respecting sensitive content boundaries.

Knowledge Graph Explorer

Interactive visualization of entities and relationships from the repository. Click nodes to open in URIBurner resolver.

Explore Knowledge Graph using SPARQL

Graph: DAV/demos/daas/x-algorithm-minimax_m2.5free-1.ttl