FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x: SKIP, 35/100 | Ideas Lab

A testing harness that exposes massive cost variance in API calls to the same model, letting users optimize which harness to use.

The pain

Engineers running inference on frontier models (Claude, GPT-4, etc.) don't know why identical requests cost 17x different amounts. They're overpaying without visibility into what's driving the variance.

The gap

No tool currently maps cost-per-pass variance to specific harness configurations. Users manually test or guess. The relationship between harness choice and actual billing remains opaque.

Build angle

Start by documenting the 9 harnesses and their cost profiles empirically. Build a simple comparison matrix showing cost per inference for identical prompts. Then add a cost optimizer that recommends which harness to use based on latency/cost tradeoffs. Charge per API call analyzed or as a SaaS dashboard.

Strengths

Solves a real, quantifiable problem (17x cost variance is massive)
Targets high-spend users who have budget to optimize
Low barrier to initial validation, just run tests and publish results
Potential to become a standard reference for model cost comparison

Risks

Unclear if the 17x variance is real or a measurement artifact; needs verification
API providers may change pricing or harness behavior, making data stale quickly
Market may be too small, only power users care about harness-level optimization
Difficult to build a defensible moat; competitors can replicate the testing methodology
Requires ongoing maintenance as models and APIs evolve

Fly Labs Method

Is the pain real, is there a gap, is it the right time, can one person build it.

35

Value Equation

Dream outcome and how likely it feels, against the time and effort it costs.

28

One-Person Business

Curiosity pull, identity fit, and a path from free value to paid for a solo creator.

42

Viral Frameworks

Hook strength, shareability, and how cheaply it can be tested.

38

Builder Lens

Evidence the problem exists, timing, defensibility, and a model that fits on a napkin.

31

Five lenses, one composite. How scoring works

The angle

Start by documenting the 9 harnesses and their cost profiles empirically. Build a simple comparison matrix showing cost per inference for identical prompts. Then add a cost optimizer that recommends which harness to use based on latency/cost tradeoffs. Charge per API call analyzed or as a SaaS dashboard.

No market research recorded for this idea yet.

SaaS founders lack a platform for finding partners and organizing joint advertising campaigns with bloggers to reduce costs by 5 times while maintaining the same reach

Other

BUILD

85FL score

Founders waste hours daily on repetitive tasks that AI could automate instantly

Founders spend excessive time on routine tasks that AI could complete in minutes, but no simple, purpose-built tool exists to handle them automatically without complex setup.

Other

BUILD

82FL score

Owners of modern Macs can't launch old games due to complex instructions involving emulators and file conversion

Other

BUILD

79FL score

All Other ideas