Understudy – Teach a desktop agent by demonstrating a task once
I built Understudy because a lot of real work still spans native desktop apps, browser tabs, terminals, and chat tools. Most current agents live in only one of those surfaces.Understudy is a local-first desktop agent runtime that can operate GUI apps, browsers, shell tools, files, and messaging in one session. The part I'm most interested in feedback on is teach-by-demonstration: you do a task once, the agent records screen video + semantic events, extracts the intent rather than coordinates, and turns it into a reusable skill.Demo video: https://www.youtube.com/watch?v=3d5cRGnlb_0In the demo I teach it: Google Image search -> download a photo -> remove background in Pixelmator Pro -> export -> send via Telegram. Then I ask it to do the same for Elon Musk. The replay isn't a brittle macro: the published skill stores intent steps, route options, and GUI hints only as a fallback. In this example it can also prefer faster routes when they are available instead of repeating every GUI step.Current state: macOS only. Layers 1-2 are working today; Layers 3-4 are partial and still early. npm install -g @understudy-ai/understudy understudy wizard GitHub: https://github.com/understudy-ai/understudyHappy to answer questions about the architecture, teach-by-demonstration, or the limits of the current implementation.
FL score
out of 100
Verdict
high confidence
Competition
24
competitors found, emerging market, big tech present, funded players
Trend
1 community mentions
A local-first, cross-application desktop agent that learns complex tasks by observing a single demonstration and extracting intent, aiming to solve the unreliability of current AI agents.
The pain
The gap
Build angle
Strengths
- Extremely high pain point for users with existing AI agent solutions, validating demand.
- Unique 'teach-by-demonstration' for intent extraction across all desktop surfaces, addressing a major gap.
- Local-first approach offers privacy advantages over cloud competitors.
- A functional demo already exists, proving technical feasibility for an MVP.
- The market for AI agents is rapidly growing, indicating strong future demand.
- Open-source model can foster community and attract contributors.
Risks
- Crowded market with many funded competitors and Big Tech players (Meta, Anthropic, Perplexity).
- Achieving robust, production-grade reliability across diverse and constantly updating desktop environments is a significant technical challenge for a solo builder.
- Monetization strategy needs clear definition beyond the open-source core to sustain a solo builder.
- High bar for 'intent extraction' to be genuinely non-brittle and handle edge cases, preventing it from just being another macro tool.
- Distribution and gaining market share against well-known, funded entities will be difficult.
Next steps
- Actively solicit feedback from early users (especially power users and developers) on the existing demo/installable version.
- Identify the single most painful, repetitive cross-application workflow that Understudy can reliably automate today, and focus initial marketing/validation efforts there (narrowest wedge).
- Explore potential business models (e.g., SaaS for advanced features, enterprise licensing, expert services) and validate willingness to pay for a reliable version of the solution.
- Focus on improving robustness and expanding OS compatibility beyond macOS as user base grows.
Fly Labs Method
Is the pain real, is there a gap, is it the right time, can one person build it.
- Problem clarity
- 27
- Solution gap
- 19
- Willingness to pay
- 19
- Buildability
- 13
Strong problem validated by user frustration with current agents, a clear gap for reliable, cross-surface, intent-based automation, and demonstrated buildability, but with significant competition.
Value Equation
Dream outcome and how likely it feels, against the time and effort it costs.
High potential due to severe market pain and strong value proposition, but significant build complexity and competitive landscape.
One-Person Business
Curiosity pull, identity fit, and a path from free value to paid for a solo creator.
Strong idea with clear problem, capable builder, and intuitive user experience, but monetization strategy needs refinement.
Viral Frameworks
Hook strength, shareability, and how cheaply it can be tested.
Strong value proposition and reachable audience, but significant risks in reliability and competitive landscape require careful validation.
Builder Lens
Evidence the problem exists, timing, defensibility, and a model that fits on a napkin.
High potential with a clear, painful problem, an elegant solution, and initial validation through a working demo.
Why this verdict
Five lenses, one composite. How scoring works
The angle
This weekend
- Actively solicit feedback from early users (especially power users and developers) on the existing demo/installable version.
- Identify the single most painful, repetitive cross-application workflow that Understudy can reliably automate today, and focus initial marketing/validation efforts there (narrowest wedge).
- Explore potential business models (e.g., SaaS for advanced features, enterprise licensing, expert services) and validate willingness to pay for a reliable version of the solution.
- Focus on improving robustness and expanding OS compatibility beyond macOS as user base grows.
Who is already there, emerging market
- OpenClaw
Open-source local-first desktop AI agent with access to files, apps, and terminal for task automation.
Pricing: free (open source, BYOK)
- Manus Desktop (My Computer)Funded
Meta-backed local AI agent app for macOS/Windows controlling local files, apps, CLI, and desktop automation.
Pricing: unknown
- Fazm.ai
Voice-first open-source macOS desktop agent using accessibility APIs for app control with local knowledge graph memory.
Pricing: free (open source MIT, BYOK)
- LobsterAI
Open-source desktop agent for executing tasks across apps, research, analysis, long-running automation.
Pricing: free (open source)
- TeachAnAgent
Browser extension to record interactions and teach AI agents by example/demonstration.
Pricing: free
- Perplexity Computer Agent (Comet)Funded
Local browser/desktop agent controlling real browser, logged-in accounts, tasks like posting/research.
Pricing: unknown
- Claude CoworkFunded
Anthropic's real-time collaborative desktop agent/canvas for pair working.
Pricing: subscription (Claude Pro)
- MiniMax Agent DesktopFunded
Chinese local AI workspace accessing files, email, calendar, agentic workflows.
Pricing: free
- Caesr AIVisit
An AI agent for automating anything on any device (web, desktop, mobile). It clicks, types, and navigates based on a prompt.
Pricing: Not explicitly stated, but user reviews mention using it for repetitive tasks, implying a potential subscription or usage-based model.
A desktop robotic process automation system that automates tedious tasks like financial data entry with a text prompt, performing clicks and keystrokes.
Pricing: Not explicitly stated, but focuses on businesses and enterprises.
- ComputerXVisit
A smart agent that handles computer work from automation to web research to creating deliverables, using natural language input. Available on Web and Desktop (Mac Apple Silicon).
Pricing: Offers free credits during beta and a 30% off subscription code 'LAUNCH30' through August 2025.
A desktop app that brings Manus out of the cloud and onto your computer, allowing an AI agent to work directly with local files, tools, and apps on macOS and Windows, using text prompts to execute tasks across files, tools, and apps.
Pricing: Not explicitly stated on Product Hunt, but previous Manus launches exist.
Gaps they leave open
- True one-shot demonstration recording/learning across surfaces
- Seamless native desktop/browser/terminal/chat integration
- Reliable long-term memory without resets
- Production reliability beyond demos (no crashes/context loss)
What people say, 1 mentions
Giving AI agents API keys is a nightmare, so I built a single gateway for them
r/SaaS
Recent news
Meta's Manus Launches Desktop App With AI Agent for Tasks Across Files, Apps
PCMag, March 18 2026
GTC Spotlights NVIDIA RTX PCs and DGX Sparks Running Latest Open Models and AI Agents Locally
NVIDIA, March 17 2026
The 10 Best AI Agents for Desktop Automation in 2026
Fazm Blog, March 14 2026
Understudy: Teach Desktop Agent Tasks with One Demonstration - AIToolly
AIToolly, March 12 2026
Show HN: Understudy – Teach a desktop agent by demonstrating a task once
James Routley (Hacker News), March 13 2026
Market signals
The market shows strong interest in AI desktop agents that learn from demonstration and natural language, with a growing number of YC-backed startups and recent product launches focusing on cross-application automation and intent-based learning rather than rigid scripting.
Semble – Code search for agents that uses 98% fewer tokens than grep
Hey HN! We (Stephan and Thomas) recently open-sourced Semble. We kept running into the same problem while using Claude Code on large codebases: when the agent can't find something directly, it falls back to grep, reading full files or launching subagents. This uses a lot of tokens, and often still misses the relevant code. There are existing tools for this, but they were either too slow to index on demand, needed API keys, or had poor retrieval quality.Semble is our solution for this. It combines static Model2Vec embeddings (using our latest static model: potion-code-16M) with BM25, fused via RRF and reranked with code-aware signals. Everything runs on CPU since there's no transformers involved. On our benchmark of ~1250 query/document pairs across 63 repos and 19 languages, it uses 98% fewer tokens than grep+read and reaches 99% of the retrieval quality of a 137M-parameter code-trained transformer, while being ~200x faster.Main features:- Token-efficient: 98% fewer tokens than grep+read- Fast: ~250ms to index a typical repo on our benchmark, ~1.5ms per query on CPU (very large repos may take longer)- Accurate: 0.854 NDCG@10, 99% of the best transformer setup we tested- MCP server: drop-in for Claude Code, Cursor, Codex, OpenCode- Zero config: no API keys, no GPU, no external servicesInstall in Claude Code with: claude mcp add semble -s user -- uvx --from "semble[mcp]" sembleOr check our README for other installation instructions, benchmarks, and methodology:Semble: https://github.com/MinishLab/sembleBenchmarks: https://github.com/MinishLab/semble/tree/main/benchmarksModel: https://huggingface.co/minishlab/potion-code-16MLet us know if you have any feedback or questions!
AI
Postgres extension for BM25 relevance-ranked full-text search
Last summer we faced a conundrum at my company, Tiger Data, a Postgres cloud vendor whose main business is in timeseries data. We were trying to grow our business towards emerging AI-centric workloads and wanted to provide a state-of-the-art hybrid search stack in Postgres. We'd already built pgvectorscale in house with the goal of scaling semantic search beyond pgvector's main memory limitations. We just needed a scalable ranked keyword search solution too.The problem: core Postgres doesn't provide this; the leading Postgres BM25 extension, ParadeDB, is guarded behind AGPL; developing our own extension appeared daunting. We'd need a small team of sharp engineers and 6-12 months, I figured. And we'd probably still fall short of the performance of a mature system like Parade/Tantivy.Or would we? I'd be experimenting long enough with AI-boosted development at that point to realize that with the latest tools (Claude Code + Opus) and an experienced hand (I've been working in database systems internals for 25 years now), the old time estimates pretty much go out the window.I told our CTO I thought I could solo the project in one quarter. This raised some eyebrows.It did take a little more time than that (two quarters), and we got some real help from the community (amazing!) after open-sourcing the pre-release. But I'm thrilled/exhausted today to share that pg_textsearch v1.0 is freely available via open source (Postgres license), on Tiger Data cloud, and hopefully soon, a hyperscalar near you:https://github.com/timescale/pg_textsearchIn the blog post accompanying the release, I overview the architecture and present benchmark results using MS-MARCO. To my surprise, we were not only able to meet Parade/Tantivy's query performance, but exceed it substantially, measuring a 4.7x advantage on query throughput at scale:https://www.tigerdata.com/blog/pg-textsearch-bm25-fu
AI
Consus (YC F24) – LLM gateway for companies handling sensitive data
Hey Hacker News! I'm Eric, the founder of Consus. We route all AI requests to ITAR restricted and controlled unclassified information (CUI) compliant infrastructure. Consus itself satisfies ITAR restrictions, and we recently completed our FedRAMP equivalency audit against the 410 FedRAMP High controls. FedRAMP High is the U.S. government’s strictest security baseline for cloud services handling sensitive unclassified data. A breach of this data could cause severe or catastrophic harm. Achieving FedRAMP equivalency, or authorization, is required before processing CUI information.Demo: https://youtu.be/6MYxJe6P4oYAerospace and defense is experiencing rapid growth. Unfortunately, most companies are struggling to use AI because of necessary data protections & compliance risk. These companies can move even faster if they could use the AI tools that most other industries take for granted.I spent most of my career prior to Consus in aerospace & defense. I launched national security payloads at SpaceX, and was an early employee at Castelion. I’ve watched first hand how compliance red-tape makes it difficult for companies to adopt new technologies and trust vendors that touch their sensitive data. For example, I worked with a satellite manufacturer that doesn’t allow any AI tools, even for non-sensitive work.Even as AI has become extremely useful over the past 9-12 months, I was surprised by how many aerospace & defense companies refused to touch it. They could see how it could be helpful, but they were scared about model providers training on their data, violating an ITAR rule, or risk losing new business because of a failed cyber assessment. These are all rational fears.There were a lot of ways to solve some/all of these problems, but we focused on building a compliant AI inference layer that can "plug" into any AI tool or workflow. Just as OpenRouter integrates with most tools, Consus can connect to tools like ChatGPT or Clau
AI
Questions about this idea?
FlyBot reads the scoring and gives you a second opinion on “Understudy – Teach a desktop agent by demonstrating a task once”.