Fly Labs
Ask FlyBot

Robust LLM extractor for websites in TypeScript

We've been building data pipelines that scrape websites and extract structured data for a while now. If you've done this, you know the drill: you write CSS selectors, the site changes its layout, everything breaks at 2am, and you spend your morning rewriting parsers.LLMs seemed like the obvious fix — just throw the HTML at GPT and ask for JSON. Except in practice, it's more painful than that:- Raw HTML is full of nav bars, footers, and tracking junk that eats your token budget. A typical product page is 80% noise. - LLMs return malformed JSON more often than you'd expect, especially with nested arrays and complex schemas. One bad bracket and your pipeline crashes. - Relative URLs, markdown-escaped links, tracking parameters — the "small" URL issues compound fast when you're processing thousands of pages. - You end up writing the same boilerplate: HTML cleanup → markdown conversion → LLM call → JSON parsing → error recovery → schema validation. Over and over.We got tired of rebuilding this stack for every project, so we extracted it into a library.Lightfeed Extractor is a TypeScript library that handles the full pipeline from raw HTML to validated, structured data:- Converts HTML to LLM-ready markdown with main content extraction (strips nav, headers, footers), optional image inclusion, and URL cleaning - Works with any LangChain-compatible LLM (OpenAI, Gemini, Claude, Ollama, etc.) - Uses Zod schemas for type-safe extraction with real validation - Recovers partial data from malformed LLM output instead of failing entirely — if 19 out of 20 products parsed correctly, you get those 19 - Built-in browser automation via Playwright (local, serverless, or remote) with anti-bot patches - Pairs with our browser agent (@lightfeed/browser-agent) for AI-driven page navigation before extractionWe use this ourselves in production at Lightfeed, and it's been solid enough that we decided to open-source it.GitHub: https://githu

Hacker NewsToolAISource
0Sign in to voteCopy link

FL score

65

out of 100

Verdict

VALIDATE

high confidence

Competition

12

competitors found, emerging market, funded players

Trend

No signal yet

A TypeScript library for robust LLM-powered web data extraction, solving major pain points for data engineers and developers.

The pain

Developers building web data pipelines with LLMs constantly battle broken scrapers, token waste from messy HTML, malformed JSON outputs, and the need to re-write complex error handling and schema validation boilerplate for every project. This leads to pipeline crashes, late-night debugging, and significant time investment.

The gap

While many general web scraping tools and no-code solutions exist, there's a clear whitespace for a developer-centric, open-source TypeScript library that specifically tackles the *robustness challenges* of LLM-based extraction. Existing tools often lack the granular control, type safety (Zod), partial data recovery, and HTML preparation (stripping noise) required for production-grade LLM data pipelines.

Build angle

Build out the Lightfeed Extractor open-source library with continued focus on robustness, developer experience, and community contributions. Explore monetization strategies by offering premium features, hosted API access, or managed services built on top of the library, targeting high-volume users or enterprises that value reliability and support.

Strengths

  • Extremely specific, real, and severe pain point for a technical audience.
  • Strong founder-market fit; built by creators for their own production use.
  • Clear differentiation as a developer-focused, open-source TypeScript library.
  • Addresses multiple common user frustrations directly (e.g., broken scrapers, malformed JSON, boilerplate).
  • Operating in a growing market (LLM and web data extraction).

Risks

  • Open-source monetization challenge: converting a free library into a sustainable business for a solo builder.
  • Ongoing maintenance burden for the library in a rapidly changing web and LLM landscape.
  • Potential for LLMs to improve significantly, reducing the need for some current boilerplate.
  • Competition from funded players in the broader web scraping market who might integrate similar features.
  • The 'urgency' to pay for a library (vs. using free tools or building in-house) may be lower than for a full SaaS.

Next steps

  1. Gather external validation on specific monetization ideas: would developers pay for a hosted API, premium plugins, or support plans? What price points?
  2. Define a clear roadmap for commercial features that extend the core open-source library.
  3. Engage with the TypeScript and data engineering communities to understand their preferred method of consuming such a solution (library vs. API vs. managed service).
  4. Focus on building a strong community around the open-source project to create a defensible asset.

Questions about this idea?

FlyBot reads the scoring and gives you a second opinion on “Robust LLM extractor for websites in TypeScript”.

Open FlyBot