Someone on Reddit asked for my actual configuration rather than another benchmark screenshot, so here it is in full: BIOS, kernel command line, compose file, systemd unit, the numbers, and the things that cost me time.

The goal was a boring appliance. Boot the box, the model is ready, agents connect to an OpenAI-compatible endpoint, nothing else runs on the machine.

A warning before the copy-paste: this is my configuration, not a tuning guide. Of the kernel settings below, upstream has A/B tested exactly one. The rest are inherited and unmeasured on my hardware.

The box

AMD Ryzen AI MAX+ 395 (“Strix Halo”), 16C/32T, Radeon 8060S iGPU (gfx1151), 128 GB LPDDR5X-8000, about 256 GB/s of memory bandwidth. Fedora 44, kernel 7.2.

The single most important thing about this hardware: it is capacity-rich and bandwidth-limited. A mixture-of-experts model with few active parameters flies; a dense model of the same size crawls. That one fact drives every choice below, including the model.

The model is Qwen3.8-Flash-Next: 125B total parameters, 6B active per token, 262k native context. The server is Halogen Flash Server, which is purpose-built for this chip and refuses to start on anything else. llama.cpp runs the model too, and was about 5x slower on prefill for me.

BIOS

Two settings, and the first one is the counterintuitive one.

  • UMA / VRAM carve: the minimum (512 MB). Do not pre-carve 64 or 96 GB. On an APU it is all the same LPDDR5X; GTT allocates dynamically at runtime. A big static carve only steals from the page cache, which this model needs badly (see the memory section).
  • Above 4G Decoding: on. UEFI boot.

OS

Fedora, because it ships a kernel new enough for gfx1151 without backports. The minimum is 6.18.4. Check that linux-firmware is not the broken 20251125 build: rpm -q linux-firmware.

Run it headless. A desktop session cost me about 7 GiB of the margin:

sudo systemctl set-default multi-user.target

Kernel parameters. These raise the GTT ceiling from about 92 to about 124 GiB; below roughly 103 GiB the engine cannot allocate at all:

sudo grubby --update-kernel=ALL --args="amdgpu.gttsize=126976 ttm.pages_limit=32505856 amdgpu.vm_update_mode=0 amdgpu.sg_display=0 amd_iommu=off"
sudo reboot

# verify
cat /proc/cmdline
cat /sys/class/drm/card*/device/mem_info_gtt_total   # want ~133 GB (124 GiB), not ~92

The two sizes are sizes, not constants. For 64 GB of RAM use amdgpu.gttsize=63488 ttm.pages_limit=16252928. Pasting the 128 GB row onto a 64 GB machine asks the driver for more GTT than the machine has.

Do not add amdgpu.noretry=0. I had it, and an earlier version of this writeup recommended it. Upstream’s docs now explain that it makes the GPU retry a lost page mapping instead of faulting, which converts a clean fault into a silent hang, and it buys no speed. I removed it. Leave it unset so the driver picks its own default.

amd_iommu=off is the one setting upstream actually measured: worth 13-16% of prefill. It also disables DMA translation machine-wide and takes the NPU with it. On a box that exists to serve one model that is the right trade; on a shared machine, think about it.

No ROCm on the host. The container image carries its own userspace; the host supplies the amdgpu kernel driver and /dev/kfd plus /dev/dri. So:

sudo dnf install -y docker-compose-plugin
sudo systemctl enable --now docker

Nothing else runs on this machine. Other containers compete for the file cache the model needs.

Layout

/srv/halogen-models/     # weights, mounted read-only (~124 GB)
  qwen38-flash-next-w4b.hgn            # 124 GB trunk
  qwen38-flash-next-w4b.overlay.hgn    # quality sidecar, auto-loaded, no env needed
  qwen38-flash-next-vision.hgn         # 0.86 GB vision tower, OFF unless pointed at
/srv/halogen-cache/      # prompt cache disk tier - real filesystem with direct I/O, NOT tmpfs
/srv/stacks/halogen/     # docker-compose.yml + .env

sudo mkdir -p /srv/halogen-models /srv/halogen-cache /srv/stacks/halogen
hf download peonist-ai/halogen-qwen3.8-flash-next --local-dir /srv/halogen-models

The cache directory has to be a real filesystem that supports direct I/O. A tmpfs or the container’s own overlay is refused, and the disk cache silently stays memory-only.

docker-compose.yml

services:
  halogen:
    image: ghcr.io/peonist-ai/halogen-flash-server:0.12.3   # pin exactly; never :latest
    container_name: halogen
    restart: unless-stopped
    stop_grace_period: 60s     # the disk cache writes each turn's rows behind the request
    devices:
      - /dev/kfd
      - /dev/dri
    group_add:
      # Numeric on purpose: docker resolves group NAMES in the CONTAINER's /etc/group,
      # which has neither. Check yours with: getent group video render
      - "39"     # video
      - "105"    # render
    ipc: host
    ulimits:
      memlock: -1              # locked weights; the mmap/lock fails without this
    volumes:
      - /srv/halogen-models:/models:ro
      - /srv/halogen-cache:/cache
    ports:
      # NEVER 0.0.0.0 - the API has no auth of its own, so the network is the access control
      - "${BIND_IP:?set BIND_IP in .env}:11435:8731"
    environment:
      HALOGEN_API_PORT: "8731"
      HALOGEN_KV_SLOTS: "3"               # conversations generating at once (~115 MB each)
      HALOGEN_REASONING_EFFORT: "low"     # unset == the template's xhigh; see gotcha 1
      HALOGEN_MAX_TOKENS_DEFAULT: "32768" # budget when a request sends none (hard cap 65536)
      HALOGEN_CACHE_DIR: /cache
      HALOGEN_CACHE_DISK_GIB: "64"
      HALOGEN_CACHE_PRUNE_OLD: "1"        # else every upgrade leaves a 30-60 GB subtree behind
      HALOGEN_VISION_TOWER: /models/qwen38-flash-next-vision.hgn

.env is one line: BIND_IP=<your LAN or VPN address>.

Everything else stays at its default deliberately: HALOGEN_CTX 262144, HALOGEN_KV_POOL_POSITIONS 524288 (~35 GB), HALOGEN_HOST_RESERVE_GIB 20, HALOGEN_MAX_TOK 32768.

Memory, which is the whole game

On a 128 GB box the split at runtime is roughly:

  • 68 GiB of weights locked in RAM
  • 14 GiB of KV pool
  • 21 GiB of working memory
  • 20 GiB reserved as page cache for the model’s 47.7 GB n-gram lookup table

That last one is not slack. The table is read from disk through the file cache as each request needs it. Starve it and long prefills collapse to single-digit tokens per second with the disk busy. This is also the reason for the 512 MB BIOS carve: every GB you hand the GPU statically is a GB the table loses.

Slots are cheap, about 115 MB each. The pool is what decides how many conversations stay resident, and the sizing rule is:

pool >= sum over live conversations of (prompt + max_tokens)

Note that a request reserves its whole reply budget up front, so a 32k default costs pool even when the reply is 200 tokens. If you need a bigger pool, HALOGEN_MAX_TOK=16384 returns about 9 GiB for roughly 8-9% of prefill speed. Lower the pool before you touch the 20 GiB reserve.

Boot behaviour

restart: unless-stopped did not bring the container back after a host reboot. At shutdown the engine exits rc=1 through its takedown path, and Docker treats a container that exited on its own as stopped rather than crashed, so it simply stayed down. I found this out the interesting way, with the box back up and nothing serving.

So, /etc/systemd/system/halogen-stack.service:

[Unit]
Description=Halogen Flash Server (compose stack)
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/srv/stacks/halogen
ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose stop
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now halogen-stack

Startup takes about 30 seconds with a warm page cache, a couple of minutes cold.

Watching it

curl -s http://$BIND_IP:11435/health | jq '.version, .status'
docker logs -f halogen

Lines worth recognising at startup:

  • GTT in use before this start: N GiB of M — tens of GiB in use with no process holding the GPU means a previous engine’s exit never gave it back. A reboot is the fix; removing containers will not do it.
  • chat template ... thinking control probe passed — the server checks that your tokenizer’s template can actually turn thinking off.
  • kv pool: NNNNNN positions and the cache-pruning summary.

And per request:

serve_api: mtp 117 tok in 2.93s = 39.89 t/s | prompt 10330, prefill 8.64s = 1196 t/s | pool 10496/524288 2%

Numbers

Cold prompts, engine-reported, server otherwise idle, version 0.12.1:

contextprefilldecode
10k1,250 tok/s43 tok/s
43k1,423 tok/s38 tok/s
147k1,408 tok/s~38 tok/s

Prefill is flat past 160k, which is the real selling point of this setup. And cold is the rare case: with the prompt cache on, a 75k conversation that is 99% cached prefills in about 1.2 seconds instead of 60. That is what makes a long agent session usable rather than painful.

Five things that cost me time

1. An unset reasoning_effort means xhigh, and the token budget covers thinking. My agent’s context-compaction call deliberately sends no budget and no effort. The result: it thought its way through the entire 32,768-token budget, 671 seconds of it, and returned truncated with finish_reason: length, so compaction failed and the conversation wedged just under its limit. Setting a server-side floor fixed it; the same call then completed in seconds. The signature in the log is a request that generated exactly the cap.

2. The disk prompt cache is keyed to the exact build. Engine version, weights, context, levers: change any of them and you get a fresh subtree, while the size budget bounds only the live one. I reached 187 GB of dead cache across five builds before turning on pruning. The trade-off is that a rollback then starts cold.

3. Benchmark from the engine’s own request line, not from client timing. I timed the first visible word of a streamed reply, which put the model’s thinking time inside “prefill” and excluded it from “decode” while still counting those tokens. On a 147k request that was 50 seconds in the wrong column, and it made me report a 25% decode gain that did not exist. Use the server’s line, use a unique prefix per request if you want cold numbers, and take at least three samples before believing a decode delta: decode varies ±6% run to run because speculative acceptance depends on the text generated. Prefill repeats within 1%.

4. Several releases fixed real bugs, not just features. Worth staying current. 0.11.5 fixed two alternating conversations evicting each other’s cache on every turn, which presents as slowness and never as an error. 0.11.9 stopped the watchdog killing an engine that was merely stalled inside the kernel, which had been leaving GPU memory held by nothing until a reboot. 0.12.1 fixed a ~5% cold-prefill regression that had stood since 0.11.3. 0.12.2 stopped a GPU fault writing a 52 GB core dump. 0.12.3 capped a watchdog deferral that let a genuinely wedged engine sit for 30 minutes.

5. A warm answer is not bitwise identical to a cold one under the default cache mode. Irrelevant for agent work; relevant if you care about reproducibility, in which case HALOGEN_PROMPT_CACHE=1 trades cache coverage for identity.

Clients

Anything that speaks the OpenAI API: point it at http://<BIND_IP>:11435/v1. One tip from gotcha 1: set the thinking level explicitly in the client rather than leaving it unset, or you get maximum effort by default and it spends the same budget as the answer.