Fly Labs
Ask FlyBot

Captain (YC W26) – Automated RAG for Files

Hi HN, we’re Lewis and Edgar, building Captain to simplify unstructured data search (https://runcaptain.com). Captain automates the building and maintenance of file-based RAG pipelines. It indexes cloud storage like S3 and GCS, plus SaaS sources like Google Drive. There’s a quick walkthrough at https://youtu.be/EIQkwAsIPmc.We also put up this demo site called “Ask PG’s Essays” which lets you ask/search the corpus of pg’s essays, to get a feel for how it works: https://pg.runcaptain.com. The RAG part of this took Captain about 3 minutes to set up.Here are some sample prompts to get a feel for the experience:“When do we do things that don't scale? When should we be more cautious?” https://pg.runcaptain.com/?q=When%20do%20we%20do%20things%20...“Give me some advice, I'm fundraising” https://pg.runcaptain.com/?q=Give%20me%20some%20advice%2C%20...“What are the biggest advantages of Lisp” https://pg.runcaptain.com/?q=what%20are%20the%20biggest%20ad...A good production RAG pipeline takes substantial effort to build, especially for file workloads. You have to handle ETL or text extraction, chunking, embedding, storage, search, re-ranking, inference, and often compliance and observability – all while optimizing for latency and reliability. It’s a lot to manage. grep works well in some cases, but for agents, semantic search provides significantly higher performance. Cursor uses both and reports 6.5%–23.5% accuracy gains from vector search over grep (https://cursor.com/blog/semsearch).We’ve spent the past four years scaling RAG pipelines for companies, and Edgar’s work at Purdue’s NLP lab directly informed our chunking techniques. In conversations with dozens of engineers, we repeatedly saw DIY pipelines produce inconsistent results, even after weeks of tuning. Many teams lacked clarity on which retrieval strategies best fit their data.We realized that a system t

Hacker NewsToolAISource
0Sign in to voteCopy link

FL score

71

out of 100

Verdict

VALIDATE

high confidence

Competition

7

competitors found, emerging market, funded players

Trend

8 community mentions

Automated RAG for Files simplifies building and maintaining production-ready LLM pipelines from unstructured data, targeting engineers frustrated with complex frameworks and partial solutions.

The pain

Building and maintaining production-grade Retrieval Augmented Generation (RAG) pipelines for unstructured files (from cloud storage or SaaS) is incredibly complex, time-consuming, and often leads to inconsistent results. Engineers struggle with ETL, chunking, embedding, storage, search, re-ranking, compliance, and observability, even with existing frameworks, which are clunky, hard to debug, or only solve part of the problem.

The gap

While many tools and frameworks exist (LangChain, LlamaIndex, Haystack, Unstructured.io, Vertex AI Search), they either offer only partial solutions (e.g., just extraction), are overly complex and hard to debug, or lack full automation across the entire RAG pipeline. There's a gap for a truly automated, end-to-end, 'done-for-you' RAG solution for files that delivers consistency and simplicity, particularly for specific use cases or user types.

Build angle

Develop an automated platform that ingests unstructured data from common cloud/SaaS sources (S3, GCS, Google Drive), handles the entire RAG pipeline (ETL, optimized chunking, embedding, storage, search, re-ranking) with minimal setup, and focuses on delivering consistent, high-accuracy results for specific file types or industry verticals. Start with a very narrow, demonstrable use case that showcases extreme simplicity and superior performance compared to existing options.

Strengths

  • Addresses a significant, well-documented, and growing pain point in the AI/LLM industry.
  • Clear and specific user frustrations with existing solutions provide distinct opportunities for improvement.
  • Founders possess deep expertise, which is crucial for tackling this complex problem.
  • The market has demonstrated willingness to pay for RAG-related solutions, evidenced by competitor pricing and funding.
  • RAG is becoming increasingly essential for practical, factual LLM applications.

Risks

  • Highly crowded market with many well-funded competitors and Big Tech presence (Google), making differentiation and gaining market share very difficult for a solo builder.
  • The technical complexity of building a truly 'automated, end-to-end' RAG pipeline for diverse unstructured data sources is immense, posing a significant buildability challenge for a solo founder.
  • Existing frameworks, while flawed, are deeply entrenched, making switching costs high for users already invested in them.
  • Requires significant ongoing development to keep up with evolving LLM technology, data sources, and RAG optimization techniques.
  • Difficulty in validating a 'narrowest wedge' that is both simple enough to build quickly and compelling enough to stand out in the noise.

Next steps

  1. Deeply interview specific target users (e.g., developers at mid-sized e-commerce companies struggling with product documentation RAG) to validate that current solutions' frustrations are truly severe enough to pay for a new, automated alternative.
  2. Identify the absolute narrowest, most painful, and easiest-to-serve vertical/use case (e.g., specific file types within a specific cloud storage for a particular LLM use case) that can be fully automated with a demonstrable, quantifiable improvement.
  3. Develop a robust proof-of-concept for this narrow use case, focusing on ease of setup and measurable performance gains over a 'DIY + LangChain/LlamaIndex' baseline.
  4. Explore partnerships or integrations with existing specialized tools (like Unstructured.io for parsing) to reduce build complexity, rather than reinventing every component of the pipeline.

Questions about this idea?

FlyBot reads the scoring and gives you a second opinion on “Captain (YC W26) – Automated RAG for Files”.

Open FlyBot