On June 22, 2026, Onur Solmaz (with collaborators ben burtenshaw and shaun smith) published a post showing how local language models can be used to triage issues and pull requests (PRs) in the OpenClaw open‑source repository in near real time. The writeup argues that medium‑sized local models, when run on suitable on‑prem hardware, can filter and notify maintainers about relevant items quickly and cheaply — provided the system is secured and properly orchestrated. The author runs experiments on an NVIDIA GB10 machine with 128 GB unified memory (referred to as a small DGX Spark box).
Method — an agentic approach with a restricted shell
Rather than using a static classifier (e.g., a BERT-style model), the team built an agent‑driven pipeline: a local model such as gemma-4-26b-a4b or qwen3.6-35b-a3b runs inside a harness and returns structured outputs to assign labels. The harness used is pi-based and the specific agent is called localpager-agent. By default the agent receives the PR title, body and a truncated diff excerpt and may optionally perform read-only inspections on the local repository.
For safety the models are not given full shell access. Instead the authors use reposhell, a restricted, read-only shell that allows commands like ls, find, cat, grep, rg and a limited set of git queries. That lets the model check package metadata or file paths locally (for example, reading an extensions package.json) without being able to modify files or perform arbitrary network calls.
A concrete case described in the post: qwen3.6-35b-a3b initially considered the coding_agent_integrations label for openclaw/openclaw#84621 ("Fix Kimi tool-call rewriting stop reason handling") because of a changed path under extensions/kimi-coding. Using reposhell to inspect files (ls, cat) revealed the package is @openclaw/kimi-provider, so the model corrected its labels to inference_api and tool_calling and explicitly excluded coding_agent_integrations.
Architecture — flow from GitHub to notification
The pipeline uses openclaw/gitcrawl as a local mirror. New PRs and issues are normalized and written to localpager's SQLite database. For each new item, localpager enqueues a classification job. A worker builds a GitHub context object (title, body, labels, author, state, optional comments, changed files and diff excerpts) and renders that into a prompt passed to localpager-agent. The agent may use reposhell to fetch extra context but must output the final classification as JSON (final_json). Results are stored back in SQLite and routed to Discord according to deterministic notification rules configured by the user.
Labeling is agentic while notification is deterministic to reduce unnecessary inference and minimize error surface for the notification stage.
Models, hardware and measured performance
The team evaluated gemma-4-26b-a4b and qwen3.6-35b-a3b on the GB10 hardware. gemma was served with vLLM and optimizations including NVFP4 quantization, prefix caching, FP8 KV cache and other backend tweaks to improve throughput.
They measured performance on a curated 330-row evaluation set. Each item received multiple reference labels (five annotations per item: 3× GPT-5.5 and 2× Opus 4.8) and labels were accepted only when models agreed, producing a stable ground truth for comparison. No prompt fine-tuning was required to get useful results from gemma or qwen. Summary metrics reported in the post were:
-
gemma-4-26b-a4b (26B total, ~4B active)
- Precision: 0.716 ± 0.010
- Recall: 0.905 ± 0.004
- F1: 0.800 ± 0.008
- Exact match: 0.410 ± 0.014
- False positives: 227.0 ± 10.5
- False negatives: 60.0 ± 2.6
- Wall seconds / row: 1.41 ± 0.04
- Output tok/s per worker: 25
- Aggregate output tok/s: 402.6
- Concurrency: 16
-
qwen3.6-35b-a3b (35B total, ~3B active)
- Precision: 0.831 ± 0.007
- Recall: 0.818 ± 0.006
- F1: 0.824 ± 0.002
- Exact match: 0.540 ± 0.014
- False positives: 105.7 ± 6.4
- False negatives: 115.3 ± 4.0
- Wall seconds / row: 13.51 ± 0.79
- Output tok/s per worker: 50
- Aggregate output tok/s: 145.3
- Concurrency: 4
-
DeepSeek-V4-Flash (reference run, 284B total, ~13B active; run once)
- Precision: 0.938
- Recall: 0.714
- F1: 0.811
- Exact match: 0.509
- False positives: 30
- False negatives: 181
- Wall seconds / row: 144.14
- Output tok/s per worker: 13
- Aggregate output tok/s: 13
- Concurrency: 1
These figures show trade-offs: Gemma achieved higher recall and much lower wall‑clock time per row, Qwen offered higher precision and exact matches with fewer false positives, while DeepSeek had the fewest false positives but was too slow for real‑time use on the tested hardware. The authors note these throughput numbers reflect their chosen optimizations and are not absolute maxima; for example gemma exceeded 700 aggregate output tokens/s in a separate probe.
Continuous validation and audit loop
Instead of relying solely on local inference for every new item, the team runs a batched SOTA cloud job (e.g., GPT-5.5) every n hours (they use two‑hour runs as an example) to audit local labels. The OpenClaw cron job runs sandboxed, writes a machine‑readable file with labels assigned by the cloud model, and a script compares those labels against localpager's labels to compute false positives and negatives. Example outputs include identified false negatives and false positives for issues and PRs; the post notes this audit loop was performed for research and adds a cost estimate (roughly 40k GPT-5.5 tokens per 2‑hour run, amounting to a few cents per run under API pricing, or around $9/month at 12 runs per day) when used as a single batch audit.
Takeaways
The authors call the task a case of "high throughput triage" and argue that agentic classification with fast local models is a promising pattern for real-time filtering in narrowly scoped domains like open‑source contributions. gemma-4-26b-a4b and qwen3.6-35b-a3b are useful starting points because they zero‑shot reasonably well without fine‑tuning. The approach generalizes to other domains (news categorization, social media filtering, support‑ticket triage, moderation appeals, sales outreach filtering, research alerting).
They also stress operational considerations: limit model privileges (reposhell), reserve GPU bandwidth for tasks that need inference, and use deterministic rules for the simplest pipeline stages to reduce error and latency. Combining local models for fast triage with periodic cloud audits is a practical strategy during trials before a full transition to local-only inference.
Notes
- The post clarifies the term “free”: local inference is cheap if you already own the hardware, but electricity and hardware costs still apply.
- DeepSeek‑V4‑Flash was used as a reference labeler earlier, but it proved inconsistent across runs and too slow for the tested on‑prem hardware, so it was not chosen as the main local agent.
- Full topic lists, configuration and examples are part of the OpenClaw project resources referenced in the post.
