Fly Labs
Ask FlyBot

Context Gateway – Compress agent context before it hits the LLM

We built an open-source proxy that sits between coding agents (Claude Code, OpenClaw, etc.) and the LLM, compressing tool outputs before they enter the context window.Demo: https://www.youtube.com/watch?v=-vFZ6MPrwjw#t=9s.Motivation: Agents are terrible at managing context. A single file read or grep can dump thousands of tokens into the window, most of it noise. This isn't just expensive — it actively degrades quality. Long-context benchmarks consistently show steep accuracy drops as context grows (OpenAI's GPT-5.4 eval goes from 97.2% at 32k to 36.6% at 1M https://openai.com/index/introducing-gpt-5-4/).Our solution uses small language models (SLMs): we look at model internals and train classifiers to detect which parts of the context carry the most signal. When a tool returns output, we compress it conditioned on the intent of the tool call—so if the agent called grep looking for error handling patterns, the SLM keeps the relevant matches and strips the rest.If the model later needs something we removed, it calls expand() to fetch the original output. We also do background compaction at 85% window capacity and lazy-load tool descriptions so the model only sees tools relevant to the current step.The proxy also gives you spending caps, a dashboard for tracking running and past sessions, and Slack pings when an agent is sitting there waiting on you.Repo is here: https://github.com/Compresr-ai/Context-Gateway. You can try it with: curl -fsSL https://compresr.ai/api/install | sh Happy to go deep on any of it: the compression model, how the lazy tool loading works, or anything else about the gateway. Try it out and let us know how you like it!

Hacker NewsToolAISource
0Sign in to voteCopy link

FL score

78

out of 100

Verdict

BUILD

high confidence

Competition

7

competitors found, emerging market, funded players

Trend

5 community mentions

An open-source proxy that intelligently compresses AI agent context to reduce LLM costs and improve performance, targeting developers of coding agents.

The pain

AI coding agents struggle with 'context rot,' dumping thousands of noisy tokens into LLM context windows, leading to unexpectedly high costs and degraded performance/accuracy (e.g., OpenAI's 36.6% accuracy at 1M tokens vs. 97.2% at 32k).

The gap

While some existing tools manage context, none offer a dedicated open-source proxy that uses SLMs for intent-conditioned compression of tool outputs with an `expand()` mechanism, specifically addressing the high cost and quality degradation for coding agents.

Build angle

Focus on optimizing the SLM's compression accuracy and reliability for common coding agent tasks, providing clear metrics on token savings and performance improvements, and offering a robust, easy-to-integrate proxy API.

Strengths

  • Addresses a real, specific, and severe pain point (cost and quality degradation) for AI agent users.
  • Existing open-source MVP and demo provide a strong foundation for validation and iteration.
  • The technical approach (SLM-driven, intent-conditioned compression) offers a novel and defensible angle.
  • Market is emerging and growing, with strong investor interest in LLM efficiency solutions.
  • Clear payment signals due to unexpected cost spikes and performance issues.

Risks

  • The complexity of training, maintaining, and continuously improving the SLM-based compression model for a solo builder is high.
  • Funded competitors are active in the broader AI agent/code completion space, potentially building similar features.
  • Open-source monetization strategy requires careful planning beyond the technical solution.
  • The 'expand()' mechanism's performance and reliability are crucial for user trust.
  • Convincing users to adopt an additional proxy layer, even for benefits, can face integration friction.

Next steps

  1. Gather more specific user feedback from the existing open-source project about compression efficacy, ease of integration, and actual cost savings.
  2. Identify the 'narrowest wedge' of use cases (e.g., specific coding agent tasks) where the compression offers the most immediate and undeniable value.
  3. Explore potential premium features or a managed service model that could provide a clear monetization path for an open-source tool.
  4. Interview potential users about their current workarounds and specific metrics for successful context compression (e.g., 'reduce costs by X% or performance improves by Y%').
  5. Benchmark against LangChain's existing context compression features and other large context window solutions to highlight clear advantages.

Questions about this idea?

FlyBot reads the scoring and gives you a second opinion on “Context Gateway – Compress agent context before it hits the LLM”.

Open FlyBot