How Cerebras Built Its Enterprise Knowledge Base

Source X post: https://x.com/cerebras/status/2077822555159945507?s=46
Canonical source: https://www.cerebras.ai/blog/how-we-built-our-knowledge-base
Captured: 2026-07-17 08:37 KST
Fetch method: X preview via ultimate-fetcher --json; canonical blog via Jina fallback because direct site fetch returned 500/timeout.
Existing related entity: 20260528-cerebras-systems

Summary

Cerebras describes how it built an internal knowledge base that answers more than 15,000 employee questions per day within three months of launch. The system is designed around a practical constraint: company knowledge does not live in one clean system. It is spread across Slack, GitHub, documents, Jira, custom databases, and team-specific tools.

Instead of forcing everyone into a single source of truth, Cerebras built a unified collection and query layer over the places where knowledge already lives. The core architecture uses a Postgres table that stores embeddings, summaries, and metadata from many sources, with connectors that normalize each source into a common queryable interface.

Slack is treated as a first-class source because it contains current engineering discussion. Cerebras found that raw vector search was insufficient for Slack because messages vary in density, length, and context dependence. The production system combines full-text search, embeddings, inverse document frequency, age decay, and reranking so that exact error strings, paraphrases, rare tokens, and recency each contribute to retrieval quality.

The system also includes real-time Slack ingestion through Socket Mode, thread/message handling, bursting for large ingestion jobs, code repository indexing via CocoIndex, custom data sources, planning and tool fan-out, reranking, MCP, web UI, organization/project scoping, and onboarding defaults.

Key ideas

• Meet data where it lives: the system avoids the fantasy of a single source of truth and indexes existing work surfaces directly.
• Hybrid retrieval is mandatory for enterprise Slack: exact tokens, embeddings, rare-token weighting, and age decay solve different failure modes.
• One normalized query substrate: Slack, code repositories, documents, and custom databases become queryable through a shared embedding/metadata interface.
• Reranking and planning matter: no single scorer is trusted; the system fuses candidate views and uses planning/tool fan-out for broader questions.
• Auth and scoping are part of retrieval: enterprise search must enforce permissions and tailor search to teams/projects.
• Onboarding is a product surface: default project scoping gives new employees high-signal answers before they know the relevant channels or repositories.

Excerpted source structure

• Meeting data where it lives
• Anatomy of a knowledge base
• Slack
• How we process unstructured Slack conversations
• Socket Mode
• Threads and messages
• Bursting
• Code repositories
• Using CocoIndex to maintain code embeddings
• Custom data sources
• Planning and tool fan-out
• Reranking
• MCP
• Web UI
• Organization
• Projects and scoped search
• Onboarding and defaults
• Final Thoughts

References in source

• Malkov and Yashunin, Efficient and Robust Approximate Nearest Neighbor Search Using HNSW
• Anthropic, Introducing Contextual Retrieval
• Cormack, Clarke, and Büttcher, Reciprocal Rank Fusion
• Search-o1
• Anthropic, Code Execution with MCP
• Lost in the Middle
• Anthropic, Use XML Tags
• Slack Engineering, How Slack AI Processes Billions of Messages
• Improving Agents, Best Nested Data Format
• Cursor, Improving Agent with Semantic Search

Source note

This clipping preserves the public technical article framing rather than the short X article card. The X post was used as provenance; the canonical blog was used as content source.