Smart Context Compression

How it works

LOD (Levels of Detail) uses SourcePrep's trace graph to understand your code's structure — functions, classes, imports, docstrings — and extracts at variable fidelity based on relevance score. No model inference, no GPU, no dependencies.

• Score-aware — high-relevance code stays full, low-relevance compresses more
• Tier-adaptive — Claude/Gemini (50K budget) gets more full-source files; local models (20K) get tighter compression
• Code-aware — understands functions, classes, imports (not just token probabilities)
• Instant — <10ms per file (no model inference)
• No extra license — LOD compression is part of the core engine on every plan, including the $0 self-hosted build

Levels of Detail

Ratios measured on real SourcePrep source files (7K–18K chars). Files with more code inside functions achieve higher ratios; files with heavy module-level constants compress less at LOD 2.

Tier-Adaptive Compression

SourcePrep detects your AI tool from the MCP handshake and adapts compression to match the context window. Larger windows get more full-source files; smaller windows get tighter compression so the same structural information fits.

Tier detection is automatic — no configuration needed. The first prep call in each session also gets a 50% orientation boost for richer initial context.

Compression flow

The extractor uses SourcePrep's pre-computed trace graph (symbol spans, class hierarchy, import edges) to know exactly where functions start and end — no re-parsing needed at query time.

Usage

Via MCP (automatic)

LOD compression is always active for MCP tool calls. When you call prep_search, results are automatically LOD-compressed based on your client tier. No configuration needed.

Via Dashboard

LOD compression is always applied. In the Context Assembler panel, click “Assemble” — each source citation automatically shows an LOD{n} badge and compression ratio.

Via API

Response metadata

When LOD compression is active, the response includes a compression object and per-chunk lod / compression_ratio fields:

{
  "context": "[src/auth.py | lod=2]\ndef login(...):\n    ...",
  "chunks": [
    {
      "source_path": "src/auth.py",
      "score": 0.47,
      "lod": 2,
      "compression_ratio": 2.6
    }
  ],
  "compression": {
    "enabled": true,
    "mode": "lod",
    "input_chars": 8200,
    "output_chars": 1840,
    "lod_distribution": { "0": 1, "2": 3, "4": 2 }
  }
}

What the assembled output looks like

The Context Output panel in the dashboard renders the same payload your AI agent receives — including the per-chunk LOD badges and compression-ratio annotations.

--- Source: src/prep/core/indexer.py:45-78 ---
def build_index(project_path: str, config: IndexConfig) -> Index:
    """Build a semantic index for the given project."""
    scanner = FileScanner(project_path, config)
    files = scanner.scan()
    chunks = chunker.chunk_files(files)
    embeddings = embed_chunks(chunks, config.model)
    return Index(chunks, embeddings)

--- Source: src/prep/api/routes.py:120-145 ---
@app.post("/projects/{project_id}/search")
async def search_project(project_id: str, request: SearchRequest):
    """Semantic search in project."""
    project = registry.get(project_id)
    results = project.index.search(request.query, k=request.k)
    return {"success": True, "data": {"results": results}}

Supported languages

LOD extraction supports signature detection and import recognition for:

For unsupported languages, files fall back to LOD 0 (full source) gracefully.

Fallback behaviour

Compression is best-effort. If the trace graph doesn't have symbol data for a file (e.g., the file was added after the last build), SourcePrep falls back to LOD 0 and returns the full source. The fallback field in the chunk metadata will be true.

Roadmap: Language Compression for Documentation

We evaluated language-aware compression for markdown and documentation files (an LLMLingua-2-based approach) and removed it — quality testing showed unacceptable term retention on technical documentation. Structural LOD compression for code remains the shipped engine; documentation compression may be revisited later.