Model launches

AI-generated text

Running Prism's Ternary Bonsai 2 27B GGUF with Prism's llama.cpp fork

A user reports steps to run Prism’s compressed Ternary-Bonsai-2-27B GGUF model locally using Prism’s fork of llama.cpp and its bundled llama-server.

Running Prism's Ternary Bonsai 2 27B GGUF with Prism's llama.cpp fork

AI Newsroom · 17 September 2026 · 3 min read

A user comment outlines how to run Prism-ML’s compressed Ternary-Bonsai-2-27B GGUF model locally using Prism’s fork of llama.cpp and the included llama-server. The note shares the required download commands, the model filename and approximate size, recommended server flags and port 8331, observed throughput on an Apple M5 Pro, and a runtime warning about the Metal tensor API.

What and why

The model referenced is Ternary-Bonsai-2-27B-PTQ1_0.gguf, hosted by Prism-ML on Hugging Face. The commenter warns that these GGUF files require Prism’s modified llama.cpp build (PrismML-Eng/llama.cpp) to function.

The goal is to test a near-lossless compressed Bonsai 2 27B model locally, either via the bundled llama-server web UI or through its API.

Steps provided (commands from the comment)

  1. Work directory: /tmp
  2. Download and extract the Prism macOS runtime tarball (example: macOS ARM64 binary):

curl -fL https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-7dffb15/llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz -o bonsai-runtime.tar.gz tar -xzf bonsai-runtime.tar.gz

  1. Download the GGUF model (the comment indicates a size of about 5.95 GB):

curl -fL https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/resolve/main/Ternary-Bonsai-2-27B-PTQ1_0.gguf -o Ternary-Bonsai-2-27B-PTQ1_0.gguf

  1. Start the server — the commenter used port 8331 with this example command:

./llama-prism-b10685-7dffb15/llama-server
-m Ternary-Bonsai-2-27B-PTQ1_0.gguf
--port 8331 -ngl 99 -fa on -c 32768

These flags in the original note are intended to tune memory and performance (for example, "-c 32768" sets the context size).

  1. Access the built-in web UI at http://localhost:8331 or send a prompt via the API. The comment gave an example API call:

uvx llm openai endpoint http://127.0.0.1:8331/v1 --model bonsai-2-27b --responses hi

Observed behavior and performance

  • The GGUF model file is approximately 5.95 GB according to the comment.
  • The server was tested on an Apple M5 Pro. Measured throughput was roughly 20 tokens/second; after a server restart the commenter observed 44 tokens/second on one run.
  • On startup the server logged: "ggml_metal_device_init: - the tensor API is not supported in this environment - disabling". This indicates the Metal tensor API was not available in that environment, which may affect GPU acceleration.

Why this matters

The comment provides practical, hands-on instructions for anyone wanting to run Prism’s compressed Bonsai 2 27B model locally, particularly on macOS ARM hardware. It clarifies the need for Prism’s llama.cpp fork, shows realistic token throughput figures for an M5 Pro, and flags a Metal API warning that could impact performance.

Notes and caveats

  • The information comes from a single Hacker News comment; readers should consult PrismML-Eng/llama.cpp and the Hugging Face model page for official documentation and updates.
  • Performance depends on hardware and environment; the token/s numbers reflect the commenter’s personal measurements.

Related stories