RANMA.cpp

RAdeon Narrative llaMA.cpp — a Radeon-optimized llama.cpp fork for roleplay and narrative inference.

This is a hobby project maintained by a single developer. Issues are welcome. Pull requests are accepted on the terms of CONTRIBUTING.md. See Project policy.

RANMA.cpp is llama.cpp with a small, curated set of changes aimed at one workload: interactive roleplay and narrative generation on consumer AMD Radeon GPUs, running on Windows.

That workload looks different from the general-purpose serving that upstream optimizes for:

• One user (or a handful of slots), not dozens of concurrent requests.
• Long, growing multi-turn contexts where most of the prompt was already seen last turn.
• Decode throughput at long context matters more than batch prefill throughput.
• Sessions stay open for hours with idle gaps between turns.
• Large MoE models that do not fit in VRAM, or even in RAM.

Some of the optimizations that pay off here trade away multi-request throughput or generality, so they are not appropriate for upstream. Others are general improvements and may be submitted upstream. This fork is where both kinds live together in a tested state.

It can also serve one person's own development work in a single session, such as a coding assistant on the same machine.

It is not a general replacement for llama.cpp. The expert cache does not change its placement while a request is processed: it installs a new plan only when every slot is idle, so when several sessions send requests in turn it rarely gets the chance. If you serve many users, use upstream.

• Expert cache - for a MoE model whose routed experts do not fit in VRAM, the server profiles which experts the router selects and keeps the most valuable set in a VRAM budget (--expert-l1-mib); the other experts are read from host memory over PCIe, and with --expert-l2-mib from the GGUF file when they do not fit in RAM either. It turns host-direct on and loads with --load-mode none by itself. docs/ranma/expert-cache.md, docs/ranma/expert-cache-l2.md
  • Give the slot count with -np (-np 1 for one user). Without it the server runs four slots: the state kept per slot grows with them (for Qwen3.8-Flash-Next at a 256K context the recurrent state and the compute buffers grow by several GiB), which can overflow VRAM, and a finite host tier is refused with finite L2 needs positive batch/parallel bounds.
• Layer compute mode (--layer-mode on) - for the same models, a prompt step is one wide ubatch computed layer by layer, so every expert that is not in VRAM is read from host memory once per layer instead of once per ubatch (GLM-5.3, Qwen3.8 and DeepSeek V4 / V4.1 Flash in EXL3, with the expert cache). The width and the buffers are sized from the model and the free memory. Off by default. docs/ranma/layer-mode.md
• Smart MTP draft length (--spec-smart) - with --spec-type draft-mtp, llama-server chooses the draft length at every step from the measured verification time per width and a calibrated acceptance of the draft probabilities, without per-model thresholds. On by default with draft-mtp; --spec-smart-store PATH keeps the estimates across restarts. docs/ranma/spec-smart.md
• EXL3 weights - ExLlamaV3 EXL3 quantizations run from GGUF files, on the CPU and on HIP (GEMV for decode, WMMA GEMM on RDNA4 for prompts), with host-direct expert banks and the expert cache. They need GGUF files in this fork's format: an ordinary GGUF or an EXL3 safetensors checkpoint is not read directly. Take the converted files from Hugging Face (Qwen3.8-Flash-Next, DeepSeek-V4-Flash-0731, DeepSeek-V4.1-Flash, DeepSeek-V4.1-Flash-UNCENSORED). For another EXL3 model, convert its EXL3 checkpoint with convert_hf_to_gguf.py. Several architectures have been converted and run (checked models), but not every model is guaranteed to work yet. These files use tensor types of this fork and do not open in upstream llama.cpp, LM Studio, Ollama or other llama.cpp-based tools. docs/ranma/exl3.md
• DeepSeek V4.1 Flash - runs from the upstream pull request for this model, which is not merged yet, with EXL3 support and the DSpark drafter on top. If upstream merges a different implementation, this fork follows upstream, and GGUF files made for the current one, EXL3 GGUF files in particular, may stop loading and need to be converted again. docs/ranma/deepseek-v41.md
  • Both EXL3 files on Hugging Face need the engram tables, which are published once for every quantization in DeepSeek-V4.1-Flash-RANMA-Engram-GGUF.

Starting settings for the published DeepSeek V4.1 Flash files by hardware (context, expert cache budgets, draft model): docs/ranma/recipes.md.

Every other change, one line each with its page: docs/ranma/README.md.

The release ranma_20261018 (upstream 8f574b0ad) measured DeepSeek V4.1 Flash in EXL3 3.0 bpw with layer-major prefill and n-gram drafting on the Radeon AI PRO R9700: server scenarios (coding, Korean roleplay; warm, one slot, the model's recommended sampling) and prompts of 512 to 32768 tokens. The 128 and 192 GB systems are emulated on a 224 GB machine (DDR5-5600 in a flex-channel layout, effective bandwidth about that of DDR5-4900) by taking physical memory away (record, with all four decode statistics, the draft acceptance, where a long prompt spends its time and memory).

The latest numbers of the other models (Qwen3.8-Flash-Next, DeepSeek V4 Flash and GLM-5.3 Flash in EXL3 with MTP, ranma_20261013), the comparisons against upstream (ranma_20261009 and earlier) and every earlier record, each with the release it was measured at, are in docs/ranma/benchmarks; the conditions the records share are in the method. The benchmark of a release is the last commit of its series (docs/ranma/releases.md).

Planned, not done yet. Nothing here is promised or scheduled.

• DeepSeek V4 Flash: try further performance work. It may not pay off.
• Experimental RDNA3 and NVIDIA (CUDA) builds: open the paths that do not depend on RDNA4 instructions and check only that they compile. Whether they work is not guaranteed; reports and pull requests are welcome.

The maintainer develops and tests every change only on this environment. Some changes rely on Windows APIs or on gfx1201-specific behavior; on other platforms they may have no effect or may misbehave. Other GPUs, backends and operating systems may work, but nothing is guaranteed there: releases are made without testing them. Reports and pull requests are welcome (CONTRIBUTING.md).

✔ tested by the maintainer at the current release. ◇ reported by a user who tested it on their own machine at the release named; not tested by the maintainer. ○ the HIP device code compiles for gfx1100 (the EXL3 kernels also for gfx1101 and gfx1102); not run. ✕ not implemented for that GPU yet; it takes a slower general path. ? not confirmed. Optimizations that only make sense with RDNA4 instructions or RDNA4 tuning, such as the RDNA4 matmul selection and the shared expert unit, are not listed.

1. Follow upstream closely. The master branch is the unmodified upstream ggml-org/llama.cpp commit that the latest release is based on. The ranma branch is the latest release: master plus the curated patch set, rebuilt for every release on a recent upstream. That means the history of ranma is rewritten regularly: commit hashes change from release to release, and a commit whose reason has disappeared (upstream landed an equivalent, or the feature stopped paying for itself) is dropped rather than carried along. Do not build long-lived work on ranma hashes. Every release is a tag (ranma_YYYYMMDD) that never moves and keeps the exact commits that the docs/ranma/ pages cite. See docs/ranma/releases.md.
2. Curate, do not accumulate. Every change must be explainable in a few sentences, measurable on the target workload, and small enough to carry across upstream updates. Features that stop paying for themselves are removed.
3. Be honest about trade-offs. Each feature documents what it costs: concurrency, VRAM, generality, or complexity.
4. Upstream what belongs upstream. Changes that are general improvements may be submitted to llama.cpp rather than kept only here.

RANMA.cpp builds exactly like upstream. For the primary target, follow the HIP section of docs/build.md on Windows: install the ROCm SDK, open an x64 Native Tools Command Prompt for VS, then:

set PATH=%HIP_PATH%\bin;%PATH%
cmake -S . -B build -G Ninja -DGGML_HIP=ON -DGPU_TARGETS=gfx1201 ^
  -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_BUILD_TYPE=Release
cmake --build build

Prebuilt Windows binaries may be published on the releases page once the first curated features land.

• Issues: welcome. Bug reports on the target environment and reproducible performance regressions are most useful. Feature requests are read but not promised.
• Pull requests: accepted on conditions, against ranma_upstream. There is no promise of review, merge or response time; the pull request carries its own verification and test environment; an accepted change may be squashed, reordered, rewritten or dropped in a later release. See CONTRIBUTING.md.
• Sponsorship: GitHub Sponsors. It does not buy review, features, or support.
• Upstream contributions: selected features from this fork may be submitted to llama.cpp under the upstream contribution rules.

How a release is made and how far each ref is verified: docs/ranma/releases.md. What each release changed: docs/ranma/changelog.md.

Upstream is tracked as the git remote upstream. When the series is rebased, upstream changes to this README and other fork-owned files are reviewed by hand and applied or dropped as appropriate.

Everything not covered above is unchanged from llama.cpp:

• Build guide · Install · Docker
• Supported models · Multimodal
• llama-server · Function calling
• Speculative decoding · Multi-GPU

RANMA.cpp is distributed under the MIT License, the same license as llama.cpp.

All of the heavy lifting is the work of ggml-org/llama.cpp and its contributors. This fork exists only to carry a narrow set of workload-specific changes on top of it.

The EXL3 format support follows ExLlamaV3 (MIT License, turboderp); see NOTICE.