Fly Labs
Ask FlyBot

Frugon – Find which LLM calls a cheaper model could handle (local, MIT)

I started leaning in on AI heavily this year, as I wanted to get more done autonomously, but then my token usage climbed dramatically to the point where my weekly quota would run out before the end of the week, sometimes a couple of days into the week.I realised I had to do something about it else I'd have to double my spend. So I decided to start tracking my cost per task type. This revealed that a lot of my spend went to searches/scans or simple things like scouting tasks.I then decided to turn this into a simple CLI tool that can be used to read your OpenAI-style logs locally, and analyze the cost and compare this spend to other models, then show you how much you could potentially save by switching those calls to a cheaper model.When you run analyze you get an offline estimate priced against LiteLLM and gated by LMArena tiers. The general savings bands come from the research published by RouteLLM; but you can confirm this yourself using 2 commands --measure (shows the prompt-response output side by side) and --judge (a model chosen to do the comparisons). These send a sample of the prompts from the logs to the candidate models - either the default choice or set by you. This call goes directly to the model provider (never through me) as any normal LLM call would, and the response is shown and judged to either be better or worse or a tie.It's deliberately small, because I tend to over complicate/think things sometimes: analyze + capture + a few commands, doing three jobs. Cost, quality visibility, routing recommendation.Nothing is hosted. capture is an optional local proxy on your own machine, and there's no endpoint in the path of your data. You can confirm this by checking the source.I included a demo so you can check out the output. It has a synthetic 56k call log (a month's worth) showing how costs can drop from $549.46 to $343.91 a month. A 37.4% saving.Try it: uvx frugon analyze --demo or uv tool install frugon Then point it

Hacker NewsToolAISource
0Sign in to voteCopy link

FL score

62

out of 100

Verdict

VALIDATE

medium confidence

Competition

No competitor data yet

Trend

No signal yet

A CLI tool that analyzes LLM API logs to identify which tasks could run on cheaper models without quality loss.

The pain

Teams using Claude/GPT-4 heavily hit token limits or budget caps mid-week because they're routing simple tasks (searches, scans, summaries) to expensive models. They lack visibility into which tasks actually need the expensive model versus which could run on cheaper alternatives.

The gap

No existing tool connects three things: (1) your actual cost breakdown by task type, (2) quality benchmarks showing which cheaper models handle specific tasks adequately, and (3) a way to test this locally before committing to routing changes. Most solutions are either dashboards without recommendations or routing frameworks that require infrastructure changes.

Build angle

Start by validating that users actually have the problem (token overspend + task-level cost visibility gap). The demo shows 37% savings, but that's synthetic data. Real validation requires 10-15 users running this against their actual logs and confirming: (a) the savings estimate matches reality after they implement routing changes, and (b) they actually implement it (adoption is the real test). Then decide if this stays a CLI tool or becomes a hosted service with persistent log analysis.

Strengths

  • Solves a real, quantifiable problem for power users of LLM APIs, token budget overruns are common and painful
  • No data leaves user's machine (local-first design removes trust barrier for cost-sensitive teams)
  • Deliberately minimal scope (analyze + capture + judge) reduces complexity and shipping time
  • Includes measurement mechanism (--measure and --judge) to validate recommendations before switching, not just theoretical savings
  • Built by someone who experienced the problem firsthand, not speculated

Risks

  • Market size unclear: how many teams actually hit token limits and care enough to optimize? Might be a niche within a niche (power users of LLM APIs who also track costs)
  • Adoption friction: even with savings estimates, routing changes require engineering work. Tool shows the opportunity but doesn't implement the routing, user has to do that manually
  • Competitive pressure: LiteLLM, Anthropic, and OpenAI could bake this analysis into their dashboards within 6 months, making a standalone tool obsolete
  • Quality judgment is subjective: --judge command relies on another LLM to compare outputs, which can be inconsistent or biased toward certain model families
  • Synthetic demo doesn't prove real-world savings: 37% savings on synthetic data doesn't guarantee users see that in production (task distribution, prompt complexity, and quality thresholds vary widely)
  • No clear monetization path: if it's open-source and local-only, how do you capture value? Hosting it removes the privacy advantage

Questions about this idea?

FlyBot reads the scoring and gives you a second opinion on “Frugon – Find which LLM calls a cheaper model could handle (local, MIT)”.

Open FlyBot