CloudAI-X X Algorithm Pipeline
The X recommendation algorithm processes posts through an 8-stage pipeline to generate the "For You" feed. Each stage transforms, filters, or scores the candidate posts.
Pipeline Overview
┌─────────────────────────────────────────────────────────────────────────────┐
│ X RECOMMENDATION PIPELINE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ User Request │
│ │ │
│ ▼ │
│ ┌─────────────┐ │
│ │ 1. Query │ Hydrate user features, action history, socialgraph │
│ │ Hydration │ │
│ └──────┬──────┘ │
│ ▼ │
│ ┌─────────────┐ Thunder (in-network) + Phoenix (out-of-network) │
│ │ 2. Sources │ In-network: Posts from followed accounts │
│ │ │ Out-of-network: ML retrieval from all posts │
│ └──────┬──────┘ │
│ ▼ │
│ ┌─────────────┐ │
│ │ 3. Candidate│ Fetch tweet text, author data, visibility status │
│ │ Hydration │ │
│ └──────┬──────┘ │
│ ▼ │
│ ┌─────────────┐ │
│ │ 4. Pre-Score│ Age, duplicates, safety, blocked authors │
│ │ Filtering │ │
│ └──────┬──────┘ │
│ ▼ │
│ ┌─────────────┐ Phoenix ML → WeightedScorer → AuthorDiversity → OON │
│ │ 5. Scoring │ Each scorer adds/adjusts candidate.score │
│ │ │ │
│ └──────┬──────┘ │
│ ▼ │
│ ┌─────────────┐ │
│ │ 6. Selection│ TopKScoreSelector: Keep top N by final score │
│ │ │ │
│ └──────┬──────┘ │
│ ▼ │
│ ┌─────────────┐ │
│ │ 7. Post- │ Conversation dedup, previously seen, keywords │
│ │ Filtering │ │
│ └──────┬──────┘ │
│ ▼ │
│ ┌─────────────┐ │
│ │ 8. Side │ Logging, analytics, impression tracking │
│ │ Effects │ │
│ └──────┬──────┘ │
│ ▼ │
│ Feed Response │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Stage Details
1. Query Hydration
Enriches the request with user context:
- User features (followed users, blocked users, muted users)
- User action sequence (engagement history for ML)
- Muted keywords
- Subscription status
- Bloom filters for seen posts
2. Sources
Two candidate sources provide posts:
Thunder Source (In-Network)
// home-mixer/sources/thunder_source.rs
// Posts from accounts the user follows
served_type: Some(pb::ServedType::ForYouInNetwork)
- Queries Thunder service with user's following list
- Returns recent posts from followed accounts
- Includes conversation context (ancestors, reply chains)
Phoenix Source (Out-of-Network)
// home-mixer/sources/phoenix_source.rs
fn enable(&self, query: &ScoredPostsQuery) -> bool {
!query.in_network_only // Disabled for "Following" tab
}
served_type: Some(pb::ServedType::ForYouPhoenixRetrieval)
- ML-based retrieval using user embedding
- Finds relevant posts from the entire corpus
- Enabled for "For You", disabled for "Following"
3. Candidate Hydration
Fetches full post data:
- Tweet text content
- Author information
- Media metadata (video duration)
- Visibility filtering results
- Subscription requirements
4. Pre-Score Filtering
Removes ineligible candidates before expensive ML scoring:
AgeFilter- Too oldDropDuplicatesFilter- Duplicate IDsVFFilter- Safety violationsAuthorSocialgraphFilter- Blocked/muted authorsCoreDataHydrationFilter- Missing dataIneligibleSubscriptionFilter- Subscription required
5. Scoring (4 Stages)
a) PhoenixScorer
// home-mixer/scorers/phoenix_scorer.rs
// Calls Phoenix ML to predict engagement probabilities
Produces phoenix_scores with 18 action probabilities.
b) WeightedScorer
// home-mixer/scorers/weighted_scorer.rs
// Combines probabilities into single score
weighted_score = Σ(weight × P(action))
Produces weighted_score from action predictions.
c) AuthorDiversityScorer
// home-mixer/scorers/author_diversity_scorer.rs
// Penalizes multiple posts from same author
multiplier = (1 - floor) × decay^position + floor
Adjusts scores to promote variety.
d) OONScorer
// home-mixer/scorers/oon_scorer.rs
// Adjusts out-of-network post scores
if !in_network: score *= OON_WEIGHT_FACTOR
Balances in-network vs out-of-network content.
6. Selection
// home-mixer/selectors/top_k_score_selector.rs
pub struct TopKScoreSelector;
impl Selector<ScoredPostsQuery, PostCandidate> for TopKScoreSelector {
fn score(&self, candidate: &PostCandidate) -> f64 {
candidate.score.unwrap_or(f64::NEG_INFINITY)
}
fn size(&self) -> Option<usize> {
Some(params::TOP_K_CANDIDATES_TO_SELECT)
}
}
Keeps top K posts by final score.
7. Post-Score Filtering
Fine-grained filtering after selection:
DedupConversationFilter- One post per conversationRetweetDeduplicationFilter- One version per underlying postPreviouslySeenPostsFilter- Remove seen postsPreviouslyServedPostsFilter- Remove from current sessionMutedKeywordFilter- User keyword mutesSelfTweetFilter- Remove own posts
8. Side Effects
Non-blocking operations after response:
- Impression logging
- Analytics events
- Cache updates
Data Flow Summary
Candidates start with:
├── tweet_id, author_id (from Sources)
├── tweet_text, metadata (from Hydration)
├── phoenix_scores (from PhoenixScorer)
├── weighted_score (from WeightedScorer)
├── score (from AuthorDiversity + OON)
└── Final ranking by score
PostCandidate Structure
pub struct PostCandidate {
pub tweet_id: i64,
pub author_id: u64,
pub tweet_text: String,
pub in_reply_to_tweet_id: Option<u64>,
pub retweeted_tweet_id: Option<u64>,
pub retweeted_user_id: Option<u64>,
pub phoenix_scores: PhoenixScores, // ML predictions
pub weighted_score: Option<f64>, // After WeightedScorer
pub score: Option<f64>, // Final score
pub served_type: Option<ServedType>, // Source type
pub in_network: Option<bool>, // Following or not
pub ancestors: Vec<u64>, // Conversation context
pub video_duration_ms: Option<i32>, // For VQV eligibility
pub visibility_reason: Option<FilteredReason>,
pub subscription_author_id: Option<u64>,
// ...
}
Source Configuration
| Tab | Thunder (In-Network) | Phoenix (Out-of-Network) |
|---|---|---|
| For You | Enabled | Enabled |
| Following | Enabled | Disabled |
Related Skills
/x-algo-scoring- Detailed scoring formula/x-algo-filters- All filter implementations/x-algo-engagement- Action types and signals/x-algo-ml- Phoenix ML model architecture