RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon
Hi HN, we're Sanchit and Shubham (YC W26). We built a fast inference engine for Apple Silicon. LLMs, speech-to-text, text-to-speech – MetalRT beats llama.cpp, Apple's MLX, Ollama, and sherpa-onnx on every modality we tested. Custom Metal shaders, no framework overhead.Also, we've open-sourced RCLI, the fastest end-to-end voice AI pipeline on Apple Silicon. Mic to spoken response, entirely on-device. No cloud, no API keys.To get started: brew tap RunanywhereAI/rcli https://github.com/RunanywhereAI/RCLI.git brew install rcli rcli setup # downloads ~1 GB of models rcli # interactive mode with push-to-talk Or: curl -fsSL https://raw.githubusercontent.com/RunanywhereAI/RCLI/main/install.sh | bash The numbers (M4 Max, 64 GB, reproducible via `rcli bench`):LLM decode – 1.67x faster than llama.cpp, 1.19x faster than Apple MLX (same model files): - Qwen3-0.6B: 658 tok/s (vs mlx-lm 552, llama.cpp 295) - Qwen3-4B: 186 tok/s (vs mlx-lm 170, llama.cpp 87) - LFM2.5-1.2B: 570 tok/s (vs mlx-lm 509, llama.cpp 372) - Time-to-first-token: 6.6 msSTT – 70 seconds of audio transcribed in *101 ms*. That's 714x real-time. 4.6x faster than mlx-whisper.TTS – 178 ms synthesis. 2.8x faster than mlx-audio and sherpa-onnx.We built this because demoing on-device AI is easy but shipping it is brutal. Voice is the hardest test: you're chaining STT, LLM, and TTS sequentially, and if any stage is slow, the user feels it. Most teams fall back to cloud APIs not because local models are bad, but because local inference infrastructure is.The thing that's hard to solve is latency compounding. In a voice pipeline, you're stacking three models in sequence. If each adds 200ms, you're at 600ms before the user hears a word, and that feels broken. You can't optimize one stage and call it done. Every stage needs to be fast, on one device, with no network round-trip to hide behind
FL score
out of 100
Verdict
high confidence
Competition
8
competitors found, emerging market
Trend
1 community mentions
A technically superior AI inference engine for Apple Silicon that delivers breakthrough performance for on-device, multi-modal AI, with clear market demand but complex monetization for a solo builder.
The pain
The gap
Build angle
Strengths
- Addresses a significant and growing market need for faster, private, on-device AI.
- Demonstrated technical superiority with impressive performance benchmarks.
- Clear, measurable value proposition and specific pain points addressed.
- Strong demand for local AI solutions to avoid cloud costs and privacy concerns.
- Open-source RCLI provides strong initial validation and community engagement.
Risks
- High technical complexity makes it challenging for a solo builder without deep low-level optimization expertise.
- Monetization strategy for a core open-source inference engine is unclear and needs definition.
- Highly competitive landscape with strong open-source incumbents (Apple MLX, llama.cpp, Ollama).
- Risk of incumbents improving their performance to narrow the gap over time.
- Requires continuous R&D to maintain performance advantage with new models and hardware.
Next steps
- Explicitly define a commercialization strategy beyond the open-source RCLI. How will the core engine be monetized for solo builders or small teams?
- Interview developers (especially those shipping on-device AI products) to validate willingness to pay for the performance gains, and for what type of packaging (e.g., licensed SDK, premium hosted service, managed models).
- Explore niche applications where the performance advantage is absolutely critical and where companies have budget (e.g., specialized voice AI in regulated industries, low-latency industrial automation).
- Consider bundling the engine with specific, high-value AI models as a 'pro' offering or a SaaS platform for easier deployment.
Fly Labs Method
Is the pain real, is there a gap, is it the right time, can one person build it.
- Problem clarity
- 26
- Solution gap
- 21
- Willingness to pay
- 20
- Buildability
- 15
This idea targets a critical performance gap in on-device AI inference for Apple Silicon, driven by a real need for faster, more private local models. The technical innovation provides a strong solution, and market signals suggest a willingness to pay for superior performance, despite high build complexity.
Value Equation
Dream outcome and how likely it feels, against the time and effort it costs.
Excellent market fit and compelling value proposition for a growing trend, but monetization strategy and build complexity for a solo founder need careful consideration beyond open source.
One-Person Business
Curiosity pull, identity fit, and a path from free value to paid for a solo creator.
A highly technical idea with clear problem clarity and strong leverage, but the solo builder fit, build simplicity, and clear monetization are significant challenges.
Viral Frameworks
Hook strength, shareability, and how cheaply it can be tested.
Strong value proposition and specific target audience, but the open-source nature presents distribution and business model challenges for micro-SaaS, requiring further validation for commercial viability.
Builder Lens
Evidence the problem exists, timing, defensibility, and a model that fits on a napkin.
Strong evidence of a real, specific problem with clear user demand and a promising narrow wedge, with good future fit, but long-term competitive landscape needs careful navigation.
Why this verdict
Five lenses, one composite. How scoring works
The angle
This weekend
- Explicitly define a commercialization strategy beyond the open-source RCLI. How will the core engine be monetized for solo builders or small teams?
- Interview developers (especially those shipping on-device AI products) to validate willingness to pay for the performance gains, and for what type of packaging (e.g., licensed SDK, premium hosted service, managed models).
- Explore niche applications where the performance advantage is absolutely critical and where companies have budget (e.g., specialized voice AI in regulated industries, low-latency industrial automation).
- Consider bundling the engine with specific, high-value AI models as a 'pro' offering or a SaaS platform for easier deployment.
Who is already there, emerging market
- OllamaVisit
An open-source platform that enables users to run large language models locally on their computers without requiring cloud connectivity.
Pricing: Free and open source. Offers a 'Pro' tier for cloud models and private model sharing.
- llama.cppVisit
A highly optimized inference engine designed to run LLMs efficiently on CPUs and GPUs, even on low-power or edge devices, with a focus on low-level performance and control.
Pricing: Free and open source.
- Apple MLXVisit
An open-source array framework developed by Apple for machine learning on Apple Silicon, offering efficient, flexible, and highly tuned performance.
Pricing: Free and open source.
- LM StudioVisit
A user-friendly desktop application with a GUI for downloading, managing, and chatting with local LLMs, often using llama.cpp under the hood.
Pricing: Not explicitly stated, but generally positioned as a free tool for local use.
- vLLMVisit
A fast and easy-to-use library for LLM inference and serving, designed for high-throughput, multi-user scenarios with optimizations like PagedAttention.
Pricing: Free and open source.
- LocalAIVisit
An open-source platform that operates locally and is available for free, intended to serve as a direct alternative to the OpenAI API, enabling on-premises inferencing for various AI functionalities.
Pricing: Free and open source.
- GPT4AllVisit
An Ollama alternative that balances user-friendliness with a commitment to privacy, designed to run entirely on your local machine, completely offline, and optimized for consumer-grade hardware.
Pricing: Free.
- Text-Generation-WebUI (oobabooga)Visit
A comprehensive web-based interface for running LLMs locally, built on top of frameworks like PyTorch and Hugging Face Transformers, offering chat, notebook-style prompting, and model management.
Pricing: Free and open source.
What they charge
What people say, 1 mentions
Just Launched: Document Processing SaaS With 99.9% OCR Accuracy + AI That Actually Understands What It's Reading
r/SaaS
Recent news
MetalRT Brings the First Unified AI Inference Engine to Apple Silicon
RunAnywhere Blog, March 18 2026
Tether's QVAC Launches World's First Cross-Platform BitNet LoRA Framework to Enable Billion-Parameter AI Training and Inference on Consumer GPUs and Smartphones
Tether.io, March 17 2026
Local LLMs Apple Silicon Mac 2026 | M1 M2 M3 Guide - SitePoint
SitePoint, March 13 2026
MLX is not faster. I benchmarked MLX vs llama.cpp on M1 Max across four real workloads. Effective tokens/s is quite an issue. What am I missing? Help me with benchmarks and M2 through M5 comparison. : r/LocalLLaMA
Reddit, March 12 2026
Sovereign LLM Inference on Apple Silicon - Medium
Medium, February 09 2026
Market signals
The market for AI inference on Apple Silicon is a growing and significant niche. Recent developments indicate a strong trend towards running powerful AI models locally on consumer devices, driven by desires for faster responses, greater privacy, and independence from cloud services. Apple's unified memory architecture is a key advantage, enabling Macs to run models that exceed typical consumer GPU VRAM limits. Several open-source projects are actively competing in this space, and recent funding rounds like those in the broader local AI / LLM space (e.g., for specialized tools or platforms integrating these inference engines) suggest continued investment and growth.
What frustrates people
- Ollama is significantly slower than cloud alternatives and can have terrible default settings that are hard to edit.
- Ollama updates more slowly and has less support for the newest models.
- MLX may not consistently outperform llama.cpp in certain real-world workloads, particularly for long input prompts and shorter outputs.
- MLX's current quantization options are mainly 4-bit, while GGUF (used by llama.cpp) offers a wider range.
- LM Studio is closed-source, making it less transparent than open-source alternatives.
Semble – Code search for agents that uses 98% fewer tokens than grep
Hey HN! We (Stephan and Thomas) recently open-sourced Semble. We kept running into the same problem while using Claude Code on large codebases: when the agent can't find something directly, it falls back to grep, reading full files or launching subagents. This uses a lot of tokens, and often still misses the relevant code. There are existing tools for this, but they were either too slow to index on demand, needed API keys, or had poor retrieval quality.Semble is our solution for this. It combines static Model2Vec embeddings (using our latest static model: potion-code-16M) with BM25, fused via RRF and reranked with code-aware signals. Everything runs on CPU since there's no transformers involved. On our benchmark of ~1250 query/document pairs across 63 repos and 19 languages, it uses 98% fewer tokens than grep+read and reaches 99% of the retrieval quality of a 137M-parameter code-trained transformer, while being ~200x faster.Main features:- Token-efficient: 98% fewer tokens than grep+read- Fast: ~250ms to index a typical repo on our benchmark, ~1.5ms per query on CPU (very large repos may take longer)- Accurate: 0.854 NDCG@10, 99% of the best transformer setup we tested- MCP server: drop-in for Claude Code, Cursor, Codex, OpenCode- Zero config: no API keys, no GPU, no external servicesInstall in Claude Code with: claude mcp add semble -s user -- uvx --from "semble[mcp]" sembleOr check our README for other installation instructions, benchmarks, and methodology:Semble: https://github.com/MinishLab/sembleBenchmarks: https://github.com/MinishLab/semble/tree/main/benchmarksModel: https://huggingface.co/minishlab/potion-code-16MLet us know if you have any feedback or questions!
AI
Postgres extension for BM25 relevance-ranked full-text search
Last summer we faced a conundrum at my company, Tiger Data, a Postgres cloud vendor whose main business is in timeseries data. We were trying to grow our business towards emerging AI-centric workloads and wanted to provide a state-of-the-art hybrid search stack in Postgres. We'd already built pgvectorscale in house with the goal of scaling semantic search beyond pgvector's main memory limitations. We just needed a scalable ranked keyword search solution too.The problem: core Postgres doesn't provide this; the leading Postgres BM25 extension, ParadeDB, is guarded behind AGPL; developing our own extension appeared daunting. We'd need a small team of sharp engineers and 6-12 months, I figured. And we'd probably still fall short of the performance of a mature system like Parade/Tantivy.Or would we? I'd be experimenting long enough with AI-boosted development at that point to realize that with the latest tools (Claude Code + Opus) and an experienced hand (I've been working in database systems internals for 25 years now), the old time estimates pretty much go out the window.I told our CTO I thought I could solo the project in one quarter. This raised some eyebrows.It did take a little more time than that (two quarters), and we got some real help from the community (amazing!) after open-sourcing the pre-release. But I'm thrilled/exhausted today to share that pg_textsearch v1.0 is freely available via open source (Postgres license), on Tiger Data cloud, and hopefully soon, a hyperscalar near you:https://github.com/timescale/pg_textsearchIn the blog post accompanying the release, I overview the architecture and present benchmark results using MS-MARCO. To my surprise, we were not only able to meet Parade/Tantivy's query performance, but exceed it substantially, measuring a 4.7x advantage on query throughput at scale:https://www.tigerdata.com/blog/pg-textsearch-bm25-fu
AI
Consus (YC F24) – LLM gateway for companies handling sensitive data
Hey Hacker News! I'm Eric, the founder of Consus. We route all AI requests to ITAR restricted and controlled unclassified information (CUI) compliant infrastructure. Consus itself satisfies ITAR restrictions, and we recently completed our FedRAMP equivalency audit against the 410 FedRAMP High controls. FedRAMP High is the U.S. government’s strictest security baseline for cloud services handling sensitive unclassified data. A breach of this data could cause severe or catastrophic harm. Achieving FedRAMP equivalency, or authorization, is required before processing CUI information.Demo: https://youtu.be/6MYxJe6P4oYAerospace and defense is experiencing rapid growth. Unfortunately, most companies are struggling to use AI because of necessary data protections & compliance risk. These companies can move even faster if they could use the AI tools that most other industries take for granted.I spent most of my career prior to Consus in aerospace & defense. I launched national security payloads at SpaceX, and was an early employee at Castelion. I’ve watched first hand how compliance red-tape makes it difficult for companies to adopt new technologies and trust vendors that touch their sensitive data. For example, I worked with a satellite manufacturer that doesn’t allow any AI tools, even for non-sensitive work.Even as AI has become extremely useful over the past 9-12 months, I was surprised by how many aerospace & defense companies refused to touch it. They could see how it could be helpful, but they were scared about model providers training on their data, violating an ITAR rule, or risk losing new business because of a failed cyber assessment. These are all rational fears.There were a lot of ways to solve some/all of these problems, but we focused on building a compliant AI inference layer that can "plug" into any AI tool or workflow. Just as OpenRouter integrates with most tools, Consus can connect to tools like ChatGPT or Clau
AI
Questions about this idea?
FlyBot reads the scoring and gives you a second opinion on “RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon”.