AnySpeechRecognizer and AnyDecisionModel are two new open-source Swift packages that provide familiar Swift APIs while letting developers choose different model engines for speech-to-text and decision tasks on Apple platforms.
What the packages do
-
AnySpeechRecognizer: preserves the design of Apple's SpeechAnalyzer API (introduced in macOS 26 and iOS 26) but allows switching the recognition engine. It supports Core ML–converted models such as NVIDIA Parakeet variants and Alibaba Qwen3-ASR, and it adds speaker diarization using Streaming Sortformer.
-
AnyDecisionModel: offers typed results and probabilities for yes/no, multiple-choice, and rating-scale questions. Instead of parsing generated text, it derives answers from the model's label probabilities.
Both packages can download open models from the Hugging Face Hub and run them on-device.
AnySpeechRecognizer details
AnySpeechRecognizer mirrors Apple types like SpeechAnalyzer, SpeechTranscriber, DictationTranscriber, SpeechDetector, and AssetInventory but makes the engine pluggable. If your code uses the shared API, switching from import Speech to import AnySpeechRecognizer is enough; Apple’s Speech framework remains the default engine.
Using a different engine (for example, Parakeet) typically only changes how you instantiate the transcriber; asset installation, running the analyzer, and reading async results remain the same.
Supported models and Core ML conversions
The package publishes Core ML conversions of several models on the Hugging Face Hub, with source checkpoints and file checksums pinned:
- Parakeet TDT 0.6B v3 — mattt/parakeet-tdt-0.6b-v3-coreml — source nvidia/parakeet-tdt-0.6b-v3 — CC BY 4.0
- Parakeet Ultra — mattt/parakeet-ultra-coreml — source moondream/parakeet-ultra — CC BY 4.0
- Parakeet Unified EN 0.6B — mattt/parakeet-unified-en-0.6b-coreml — source nvidia/parakeet-unified-en-0.6b — NVIDIA Open Model License
- Qwen3-ASR 0.6B — mattt/qwen3-asr-0.6b-coreml — source Qwen/Qwen3-ASR-0.6B-hf — Apache 2.0
- Qwen3-ASR 1.7B — mattt/qwen3-asr-1.7b-coreml — source Qwen/Qwen3-ASR-1.7B-hf — Apache 2.0
- Streaming Sortformer 4spk v2 and v2.1 — mattt/diar-streaming-sortformer-4spk--coreml — source nvidia/diar_streaming_sortformer_4spk- — CC BY 4.0 / NVIDIA Open Model License
The Parakeet repositories include an optional 8-second encoder to speed live input, which reduces live analysis time by roughly one third at the cost of a somewhat higher word error rate. Conversion workflows and checks mostly follow FluidInference’s mobius scripts.
Download sizes and caching
Models are downloaded on demand. Approximate sizes:
- About 600 MB per Parakeet model, plus ~590 MB for the live encoder
- Qwen3-ASR: ~0.9–2.4 GB depending on variant
- Sortformer diarization model: ~250 MB
Files go into the standard Hugging Face cache, so they can be shared across a Mac app, Python, and other Hugging Face tools.
Speaker diarization
AnySpeechRecognizer adds a SpeakerDiarizer module that runs NVIDIA’s Streaming Sortformer on Core ML alongside an Apple or Parakeet transcriber on the same timeline. The diarizer tracks up to four speakers, including overlapping speech, and constructs a SpeakerTimeline that attributes timed words to speakers. Sortformer v2 is the default; v2.1 (trained with more meeting data) is also available.
Measured accuracy and performance
Measurements reported in the project used the hear command-line tool on an M4 Max Mac Studio running macOS 27.0.1. Word error rates (WER) on LibriSpeech files of 3–8 minutes were:
- Parakeet v3, file: test-clean 1.55%, test-other 2.78%
- Parakeet v3, live: test-clean 4.73%, test-other 4.29%
- Parakeet Ultra, file: 1.66% / 2.55%
- Parakeet Ultra, live: 2.65% / 3.28%
- Parakeet Unified, file: 1.43% / 2.19%
- Parakeet Unified, live: 1.45% / 2.42%
- Apple’s transcriber, file: 2.12% / 3.97%
Additional performance notes:
- Parakeet transcribes an hour of audio from a file in roughly 20–40 seconds, or live in about 1.5–2.5 minutes, and uses roughly 100 MB of memory after Core ML compilation.
- Qwen3-ASR is notably slower and requires several gigabytes of memory.
- Meetings with overlapping speech are challenging: on AMI recordings all engines tested had WERs above 23%.
The authors emphasize these numbers come from a single machine and dataset set; they provide tools (hear evaluate and natural-speech-eval.py) so developers can reproduce evaluations on their own hardware and audio.
AnyDecisionModel details
AnyDecisionModel offers a Swift API to pose independent questions about a single state (text or JSON). The model-and-session pattern will be familiar to users of AnyLanguageModel: a DecisionSession holds the state, and each question (probability, choice, score) queries that state. Questions do not mutate the session state, and batch decisions are supported.
Example usage in the article shows:
- session.probability(of:) returns a Double
- session.choice(..., from: SomeEnum.self) returns a Choice<T> containing the selected case, per-option probabilities, and a confidence value
- session.score(..., levels: [...]) returns a Score with expected level, per-level probabilities, and confidence
Enums that conform to Choosable can be used exhaustively in switch statements.
Demo: Emoji Magnet
Emoji Magnet is a macOS demo app with 343 emoji. Given a phrase (for example, “things you wear”), the app asks the model a yes/no question for each emoji and shows matching emoji. With the hosted Jev backend, all 343 questions are answered in one request, producing a visual result in about 0.5 seconds per phrase. Locally, with MLX and Qwen3 4B, the backend uses prefix caching to evaluate the state once and then answers 16 questions per forward pass; on an M4 Max, 16 answers take ~0.45 seconds once the model is loaded. A full pass over 343 emoji is about 10 seconds, with group results appearing as they arrive.
Decision heads (Clef) vs. next-token probabilities
Reading next-token probabilities works with general-purpose language models, but models trained with a decision head can be more direct. Cloudflare’s Clef models include a trained decision head that maps state+schema to probabilities for all allowed options across questions in one forward pass. AnyDecisionModel can run such models (example: Clef-Flash) via MLXHeadDecisionModel, which reads decisions from the trained head rather than next-token scores.
The Clef 4-bit model is a ~6.2 GB download; its model card recommends a Mac with at least 16 GB of memory. Clef models can also read images and video, but the backend in AnyDecisionModel accepts only text and JSON states.
Calibration and diagnostics
Raw model probabilities are often overconfident. Local backends accept a Calibration that applies a temperature to answers and a bias to yes/no questions. Developers should use separate datasets to fit calibration and to evaluate accuracy. The next-token backends also support rotationDebiasing: rotate option orders and average probabilities, which costs an extra pass per rotation. Each local choice/score also returns a confidence value based on probability entropy; confidence measures concentration, not correctness. Diagnostics report how much probability the model assigned to allowed answers—low values indicate the model placed probability elsewhere.
Supported backends in AnyDecisionModel
AnyDecisionModel provides six backends:
- MLX: runs small language models on Apple silicon with mlx-swift-lm and downloads models from the Hub.
- MLX decision heads: runs models with trained decision heads (e.g., Clef) via MLX.
- Core AI: similar to MLX but uses Core AI models on macOS 27 / iOS 27.
- llama.cpp: local GGUF models via llama.swift.
- Jev: calls TypeSafe’s Jev over the System One API (or any compatible server such as Kev).
- Ollama: calls a System One decision model on a local Ollama server or Ollama cloud (requires Ollama 0.35.0+, or 0.40.5+ for cloud models).
Note: next-token backends label answers as yes/no, A–Z, or 0–9, so choices are limited to 26 options and scores to 10 levels; decision heads have no such limit. The package uses Swift package traits so that, without traits, it has no third-party dependencies. Enabling the MLX, CoreAI, or Llama trait adds local backends. Jev and Ollama backends are included and also build on Linux.
Requirements and getting started
- Both packages require Swift 6.2+ and macOS 26 or iOS 26+.
- Parakeet and Sortformer runtimes and the MLX backend require Apple silicon.
- AnyDecisionModel’s MLX build tasks use Xcode Swift 6.4+ and the Metal toolchain; Core AI requires Xcode 27 and an OS 27 runtime.
Example Package.swift dependencies from the article:
dependencies: [ .package(url: "https://github.com/mattt/AnySpeechRecognizer", from: "0.1.0"), .package(url: "https://github.com/mattt/AnyDecisionModel", from: "0.6.0", traits: ["MLX"]) ]
A compatibility note: AnyDecisionModel’s CoreAI trait currently cannot be combined with AnySpeechRecognizer due to coreai-kit and swift-huggingface version constraints.
Tools, examples and contribution
- AnySpeechRecognizer includes the hear CLI tool for transcription, speaker labeling, and evaluation. Each hear command requires --backend apple, --backend parakeet, or --backend qwen; run hear prepare with the same backend to install assets before analysis.
- AnyDecisionModel-Examples contains example apps including Emoji Magnet.
Both repositories welcome issues and pull requests on GitHub. The project acknowledges contributions and support from NVIDIA (Parakeet, Sortformer, NeMo), Moondream (Parakeet Ultra), FluidInference (conversion workflows), Cloudflare (Clef), and contributors to AnyLanguageModel whose patterns these packages reuse.
