What llama-modchip v2.5.4 is
synchopate/llama-modchip v2.5.4 describes a CPU-first fork of the llama.cpp ecosystem aimed at mixture-of-experts checkpoints that stock builds cannot open. Its headline claim is a 284B-parameter DeepSeek-V4-Flash running inside a 64GB Linux machine with no GPU, at 56.93 GiB of weights in the MaxQ-Fractal build or 56.40 GiB in the VQ3 build.
Two custom block-64 formats carry the compression. Fractal-64 stores 30 bytes per 64 weights at 3.75 bits per weight. LUT-VQ3-64 stores 24 bytes per 64 weights at 3.00 bits per weight. Around the formats sit HARP-Cache KV management, /dev/shm Slot-0 prefix snapshots, and Stockfish-PVS v2 speculative decoding.
Every performance and quality figure below traces back to the maintainer's README unless marked otherwise.
GGML types 42 and 43 mean you run their runtime
Fractal-64 registers as GGML_TYPE_Q2_0 = 42 and reuses that upstream identifier in place, while LUT-VQ3-64 adds GGML_TYPE_VQ3_0 = 43. Vanilla llama.cpp, Ollama, vLLM and LM Studio cannot load the resulting GGUFs without the llama-modchip runtime. A daily.dev summary from 8 November 2026 describes the same failure from the error side, with unpatched builds rejecting the custom tensor type.
The repository ships patch files and prebuilt server binaries rather than pointing at a pinned upstream commit. The Linux release includes an avx512 asset and an avx2 asset; the avx2 build is compiled with -mno-avx512f and -DGGML_AVX_VNNI=OFF, targets the x86-64-v3 baseline, and the unsuffixed binary is an identical copy of it. Windows takes a separate route: a PowerShell script builds a native MinGW-w64 UCRT64 AVX-512 engine for either the DeepSeek or the Nemotron target, and the README reports native compilation on Windows 11 verified up to 1M context.
What this material does not give is the exact upstream llama.cpp commit or pull request the fork is based on. If you need that for a dependency lock, it has to come from the repository itself.
AVX-512 VBMI is the fast path, not the entry ticket
The LUT-VQ3-64 decode path is described as decoding 128 3D Lloyd-Max centroids in registers with AVX-512 VBMI VPERMI2B. The project's incremental VQ3 patch is listed as shipping AVX-512 VBMI, AVX2 and scalar code paths. The Fractal-64 decode path is listed as single-cycle dyadic VPSHUFB, which is not an AVX-512-only instruction.
A pre-AVX-512 x86-64-v3 machine is served by the avx2 binary, and Apple Silicon runs through Metal, with the README reporting 57.65 tok/s on an M1 Max, 78.60 tok/s on an M3 Max and 27.0 tok/s on an M5 Max for the 284B model.
What no source here answers is how the 284B VQ3 build performs on an AVX2-only host.
The 64GB claim is a margin claim
Numbers first. The MaxQ-Fractal build runs at 57.54–58.13 GiB anonymous-heap VmRSS at -c 32768 -ub 512, and the project states a 57.54–59.13 GiB range across builds. A 64GB Linux host offers roughly 62.5 GiB usable RAM.
That margin is why the README prescribes different settings for different hosts. Desktop use stays at -c 32768 -ub 512, with zram enabled or heavy browsers closed. The -c 262144 (256K token) configuration is documented for headless 64GB nodes running zram or swap, or for machines with 96GB or more. The reason is mechanical: --load-mode none --repack holds repacked weights in anonymous heap, which the kernel cannot page back from disk under desktop memory pressure.
Two secondary write-ups describe the 256K figure differently from the README's own guidance. Zima Store Online reported on 10 November 2026 a 56.56 GiB file and 57.61 GiB peak resident memory in a 256K configuration. The README pairs the 57.54–59.13 GiB range with the 32K desktop default and pushes 256K onto larger or headless hosts. Where the accounts disagree, the configuration flags you actually launch with describe your run better than any headline does.
Whose DeepSeek conversion is this?
The model is DeepSeek-V4-Flash-0731, listed at 284.33B total parameters with 13.17B active per token and unpruned Top-6 routing. The README describes a single-pass conversion across all 1,328 tensors from BF16, FP8 or MXFP4 weights, splitting into 174 dense bottlenecks, 317 Fractal-64 MLA and indexer tensors, and 129 routed-MoE tensors weighted by a 1.5M-token calibration file. Four variants exist: MaxQ-Fractal as the safety-aligned default, an opt-in Abliterated-Fractal, VQ3, and a cross-platform Fractal build.
The material here shows no official upstream GGUF at this size. The Hugging Face repositories named in one write-up are the author's own conversions, Synchopate/DeepSeek-V4-Flash-0731-Fractal-64GB-CPU-GGUF and a Nemotron counterpart. Project code is Apache-2.0. That license does not automatically extend to the weight shards, and each model repository's card governs redistribution.
Reading the throughput numbers
Four different measurements share one performance section, and they are not interchangeable.
- Cold prefill: the VQ3 build records 41.11 tok/s over a pp67–pp128 run with zero cached tokens, and 44.36 tok/s warm on the same row. Across the 284B builds the README quotes 36.1–41.1 cold and 44.4 warm.
- Raw single-stream decode: 8.66 tok/s for a tg64 run at N=1 with unpruned Top-6 routing, 8.61 across three runs. The cross-variant range is 8.66–9.06 tok/s.
- Agentic bursts: 19.4–25.3 tok/s on one build and roughly 11–21 tok/s on the others.
Two pieces of context keep those figures honest. The host is a 2-channel DDR5-5600 platform with roughly 75 GB/s sustained bandwidth, and the README states that higher sustained rates require multi-channel servers or a machine like the M5 Max with 286.5 GB/s of unified memory. The drafter is the project's Stockfish-PVS v2 scheme, which AI Weekly describes as borrowing the chess engine's principal-variation search with a 1.64 MB Zobrist hash table. Speculative decoding leaves the target model to verify every accepted token, and the extra memory stays modest because the proposer is much smaller than the target.
Reproducing the quality gate
The 284B MaxQ build scores 70.0 out of 80 on benchmarks/eval_quality_and_tools.py, broken down as 30/30 tools, 20/30 code and 20/20 needle. The VQ3 and cross-platform Fractal builds score 64.0/80 each. Note the denominator: the 284B gate is out of 80 functional points, while the 30B Nemotron build reports 100.0/100.0. Comparing the two headline scores directly mixes two scales.
Perplexity and divergence numbers, all from the README. On wiki.test.raw at 8,192 tokens: Fractal 11.5288 ± 0.55630, VQ3 12.2345 ± 0.59759, MaxQ 11.6139 ± 0.58453 (within 0.15 sigma of Fractal), Abliterated 11.3944 ± 0.56500. Direct signal fidelity on the uncompressed blk.1.ffn_up_exps.weight tensor is 17.75 dB SQNR and 99.15% cosine similarity for Fractal-64, 13.57 dB and 97.78% for LUT-VQ3-64.
Methodology matters for the speed numbers and not for the quality ones. The repo documents 16-core Zen 5 with --cpu-range 0-15 --cpu-strict 1 and Transparent HugePages enabled. Run those first when you want to know whether a build is behaving.
When the 30B MoE is the better agent choice
Nemotron-3.5-30B-A3B carries 3.0B active parameters and lands at 13.30 GiB in VQ3-64, with WikiText-2 perplexity of 7.4400 ± 0.2626 against a 6.9582 ± 0.2415 BF16 reference; the Fractal-64 build is 14.36 GiB at 7.3610. Resident memory is 14.6 GiB at -c 32768 and 16.04 GiB at -c 262144. Served gate decode runs 24.18–25.02 tok/s against 19.54 tok/s for Q4_0.
The choice follows the workload. A session that needs the larger model's reasoning has to fit inside a 3 to 5 GiB margin, which means short context, no browser, and zram on. Anything past that pushes the 284B onto 96GB-class hardware or a headless node.
What to log when you try it
Do these in order.
- Record the GGUF revision and the file size in GiB for each shard.
- Record which binary asset you launched, avx512 or avx2, and its SHA256.
- Record total and available RAM before the process starts.
- Launch with -c 32768 -ub 512.
- Read peak VmRSS after the model finishes loading and the server goes idle.
- Run the tool, code and needle gate and save the JSON output unchanged.
- Time cold prefill and warm prefill as separate measurements.
- Time N=1 decode and speculative bursts as separate measurements.
A run that reports one tok/s figure without saying which of the four it is has told you very little.
Questions agent operators ask
Does llama-modchip need a patched llama.cpp fork?
Yes. The GGUFs use GGML type 42 and type 43, which vanilla llama.cpp, Ollama, vLLM and LM Studio reject.
Does it run on a CPU without AVX-512?
The project ships an x86-64-v3 avx2 binary built with -mno-avx512f, aimed at Haswell through Raptor Lake and Zen 1 through Zen 3. AVX-512 VBMI is the fast path for LUT-VQ3-64 decoding specifically.
How much RAM do I actually need for the 284B model?
Weights occupy 56.40 to 56.93 GiB depending on the build, and resident memory lands at 57.54–59.13 GiB against roughly 62.5 GiB usable on a 64GB Linux host. That leaves 3 to 5 GiB of working room, which is why the desktop default is -c 32768 -ub 512 with zram. For 256K context the README points at headless 64GB nodes with swap or zram, or at 96GB or more.
Is there an official DeepSeek-V4-Flash GGUF at 284B?
The conversions described in the repository are the maintainer's own single-pass builds from BF16, FP8 or MXFP4 weights. The material reviewed contains no official upstream release at this size, and the Hugging Face repositories named in coverage are the author's.
What do 41.1 cold prefill and 9.06 N=1 decode actually measure?
Cold prefill is prompt processing with zero cached tokens over a pp67–pp128 run, and warm prefill on the same configuration reaches 44.36 tok/s. The 25.3 tok/s figure belongs to speculative bursts under the served gate, not to plain decode.
Has anyone independently reproduced the numbers?
Not in the material reviewed here. A daily.dev summary from 8 November 2026 recorded community skepticism about output quality at the claimed compression level and criticism of the README's readability.
Source notes
Figures come from the synchopate/llama-modchip README and benchmark documentation, version v2.5.4, undated in the material reviewed. Zima Store Online published its analysis on 10 November 2026. vLLM's CPU documentation supplied background on AVX-512 versus AVX2 build fallbacks in CPU runtimes, and the Jetson AI Lab speculative decoding tutorial supplied background on how drafting and verification work. Neither of those two describes llama-modchip.