Gemma 4 Multimodal Fine-Tuner for Apple Silicon

About six months ago, I started working on a project to fine-tune Whisper locally on my M2 Ultra Mac Studio with a limited compute budget. I got into it. The problem I had at the time was I had 15,000 hours of audio data in Google Cloud Storage, and there was no way I could fit all the audio onto my local machine, so I built a system to stream data from my GCS to my machine during training.Gemma 3n came out, so I added that. Kinda went nuts, tbh.Then I put it on the shelf.When Gemma 4 came out a few days ago, I dusted it off, cleaned it up, broke out the Gemma part from the Whisper fine-tuning and added support for Gemma 4.I'm presenting it for you here today to play with, fork and improve upon.One thing I have learned so far: It's very easy to OOM when you fine-tune on longer sequences! My local Mac Studio has 64GB RAM, so I run out of memory constantly.Anywho, given how much interest there is in Gemma 4, and frankly, the fact that you can't really do audio fine-tuning with MLX, that's really the reason this exists (in addition to my personal interest). I would have preferred to use MLX and not have had to make this, but here we are. Welcome to my little side quest.And so I made this. I hope you have as much fun using it as I had fun making it.-Matt

Hacker NewsApr 7ToolAISource

FL score

out of 100

Verdict

BUILD

high confidence

Competition

competitors found, emerging market, funded players

Trend

No signal yet

A local Gemma 4 multimodal audio fine-tuner for Apple Silicon, solving OOM issues and MLX limitations, built by a solo developer.

The pain

ML engineers and researchers on Apple Silicon Macs constantly run out of memory (OOM) when fine-tuning large multimodal LLMs like Gemma 4, especially on longer sequences. Existing open-source MLX tools lack robust audio fine-tuning capabilities, forcing users to expensive, less private cloud GPUs or preventing local fine-tuning altogether.

The gap

There's a specific 'tooling gap' for efficient Gemma 4 multimodal *audio* fine-tuning on Apple Silicon that current MLX-based solutions and other local LLM runners don't adequately address, particularly concerning OOM management and native audio support.

Build angle

Develop a specialized, memory-optimized fine-tuning framework for Gemma 4 multimodal (audio) models on Apple Silicon, initially as a CLI tool, potentially adding a user-friendly GUI or integrations as a paid offering.

Strengths

Addresses a very specific, high-severity pain point for a technical audience.
The builder has already created a working solution, demonstrating deep expertise and passion.
Leverages growing trends in local AI and multimodal models.
Clear technical gap identified where existing, funded solutions fall short (audio fine-tuning, OOM on longer sequences with MLX).
Cost savings and privacy are strong motivators for local fine-tuning.

Risks

Currently an open-source project; monetization strategy for a solo builder is unclear.
The target audience, while dedicated, is niche. Scaling could be challenging.
Ongoing development and maintenance for a complex, memory-intensive tool could be demanding for a solo builder.
Risk of Apple or other incumbents releasing native, more robust solutions that directly address this gap.
CLI-only interface might deter less technical users, requiring further development for broader adoption.

Next steps

Actively engage with the current GitHub users and broader developer communities (Reddit, Hacker News) to deeply understand their specific pain points, feature requests, and *what they would pay for*.
Explore potential paid offerings: a polished GUI wrapper, premium support, advanced memory optimization features, or a managed cloud service offering custom fine-tuning using this technology.
Validate pricing for potential commercial offerings by surveying the target audience and potentially launching a minimal paid beta program.
Prioritize features that directly address the most severe OOM issues and offer unique advantages over cloud services or other local tools.

Fly Labs Method

Is the pain real, is there a gap, is it the right time, can one person build it.

Problem clarity: 24
Solution gap: 19
Willingness to pay: 15
Buildability: 11

A strong technical solution to a specific pain point (Gemma 4 audio fine-tuning, OOM) on Apple Silicon, built by the developer. Monetization path for a solo builder is currently unclear but there is a clear market need for efficient local fine-tuning.

Value Equation

Dream outcome and how likely it feels, against the time and effort it costs.

This idea targets a high-pain, growing market with a clear value proposition, though monetizing it as a solo open-source project will be a key challenge.

One-Person Business

Curiosity pull, identity fit, and a path from free value to paid for a solo creator.

An excellent creator-problem fit with a clear, niche problem, but potential challenges in broad audience reach and defining a clear monetization path for a solo builder.

Viral Frameworks

Hook strength, shareability, and how cheaply it can be tested.

Strong target audience and clear value proposition for a validated technical problem. The main hurdle is transitioning from an open-source project to a micro-SaaS with a defined business model and distribution strategy.

Builder Lens

Evidence the problem exists, timing, defensibility, and a model that fits on a napkin.

Addresses a deeply felt pain for a specific, technical audience with a clear improvement over the status quo. The product exists as a narrow wedge, and future-fit is strong.

Why this verdict

The idea is technically sound and addresses a real, painful problem for a specific, engaged audience. The builder has already proven capability by creating the solution. However, as an open-source project, its viability as a solo business requires a clear and validated monetization path. The 'VALIDATE_FIRST' verdict is to ensure there's enough willingness to pay for a commercial version or service built upon this strong technical foundation, rather than just using the free open-source tool. The composite score of 70 reflects the strong problem/solution fit and market potential, tempered by the monetization uncertainty.

Five lenses, one composite. How scoring works

The angle

This weekend

Actively engage with the current GitHub users and broader developer communities (Reddit, Hacker News) to deeply understand their specific pain points, feature requests, and *what they would pay for*.
Explore potential paid offerings: a polished GUI wrapper, premium support, advanced memory optimization features, or a managed cloud service offering custom fine-tuning using this technology.
Validate pricing for potential commercial offerings by surveying the target audience and potentially launching a minimal paid beta program.
Prioritize features that directly address the most severe OOM issues and offer unique advantages over cloud services or other local tools.

Who is already there, emerging market

mlx-tuneVisit
An open-source library leveraging Apple's MLX framework for efficient local LLM fine-tuning on Apple Silicon, reducing reliance on cloud GPUs.
Pricing: Free (open-source)
M-CourtyardVisit
A desktop application with a GUI for local LLM fine-tuning on Apple Silicon, built with Tauri + React and leveraging MLX.
Pricing: Free (open-source)
InstructLabFundedVisit
An open-source project by IBM and Red Hat that provides an easy way to tune and run models, including local fine-tuning on Apple Silicon.
Pricing: Free (open-source)
Unsloth-MLXVisit
An MLX-powered library that brings the Unsloth fine-tuning experience to Apple Silicon, aiming for code portability between Mac and cloud GPUs.
Pricing: Free (open-source)
UnslothVisit
A framework that optimizes LLM fine-tuning, making it faster and reducing VRAM usage, with support for Gemma models.
Pricing: Offers free Colab notebooks for fine-tuning; likely commercial offerings for optimized use but specific pricing not found for self-hosted tooling.
OllamaFundedVisit
A tool for running large language models locally, including Gemma 4, with an OpenAI-compatible API.
Pricing: Free (open-source)
SuperWhisperVisit
A macOS speech-to-text app offering local-only models for privacy and fast input across applications.
Pricing: $249 lifetime plan or $84.99/year for Pro features; free version with limited access to small models.
VoibeVisit
A Mac dictation app optimized for Apple Silicon, offering offline privacy and using OpenAI's Whisper models for fast, accurate transcription.
Pricing: $99 lifetime plan or $44.10/year.
VoiceInkVisit
An open-source Mac-only voice-to-text app focused on speed, accuracy, and privacy, running fully offline with local AI models.
Pricing: Free by building from source or $39 one-time for pre-built convenience.
Wispr FlowVisit
A voice-to-text tool built for fast writing on Mac, Windows, and iPhone, with cloud-based processing for consistent performance.
Pricing: Specific pricing not found, but noted as offering a 'premium experience'.
Gemma Multimodal Fine-Tuner (mattmireles/gemma-tuner-multimodal)Visit
An open-source toolkit to fine-tune Gemma 4 and 3n with audio, images, and text on Apple Silicon, including streaming from cloud storage.
Pricing: Free (open-source)

Funded players

What they charge

The current landscape for local LLM fine-tuning on Apple Silicon is predominantly open-source and free, leveraging frameworks like Apple's MLX. Commercial tools focusing on specific applications like dictation (e.g., SuperWhisper, Voibe, VoiceInk) offer free tiers with paid upgrades for advanced features or lifetime access, ranging from $39 (one-time) to $249 (lifetime). Cloud-based fine-tuning services (e.g., Together AI, Vast.ai, Hyperstack) provide GPU rental by the hour (e.g., H100 SXM from $1.75/hr, A100 PCIe from $0.64/hr) or monthly subscriptions for dedicated resources.

Gemma 4 Multimodal Fine-Tuner for Apple Silicon

Semble – Code search for agents that uses 98% fewer tokens than grep

Postgres extension for BM25 relevance-ranked full-text search

Consus (YC F24) – LLM gateway for companies handling sensitive data