ANE Training: Cracking Open Apple's Neural Engine

I. TL;DR + Verdict

Two artifacts are tangled in this tweet. The real breakthrough is maderix's (Manjeet Singh) reverse-engineering of Apple's Neural Engine private APIs to enable training via backpropagation¹ — the first known ANE training, built collaboratively with Claude Opus 4.6, one weekend at a time. Brian Roemmele's tweet wraps this in ZHC@Home hype about rivaling Colossus-scale data centers via distributed Mac compute⁶ — which is physically impossible due to network bandwidth constraints: 26 minutes to sync gradients for a 10B model on home internet vs 22ms on NVLink.¹¹

Donna reproduced ANE training on Eric's M4 Max MacBook Pro (128GB, macOS 15.3.1): 2,000 training steps, loss dropping from 0.500 to 0.001, with exec() restart and checkpoint/resume working correctly. The technique is real. The hype is not.

II. The Artifact

Brian Roemmele's tweet⁵ (7,564 likes, 1,111 RTs) announces that his "Zero-Human Company" is testing ANE training based on maderix's open-source breakthrough. A follow-up tweet claims "at 20% idle from just a few million opt-in M4 Macs, ZHC@Home could rival or exceed single massive data center clusters (e.g., 1-2x Colossus scale)"⁶ at 10-20 MW vs 2 GW centralized.

Two layers are tangled — one is real engineering, the other is engagement farming.

III. How It's Actually Done

ANE Architecture

The ANE is a graph execution engine — not a GPU. It takes a compiled neural network graph and executes the entire thing as one atomic operation.² The M4's ANE (codename H16G) has 16 cores, a queue depth of 127 evaluation requests, independent DVFS, and hard power gating that drops to exactly 0 milliwatts when idle.²

The Private API Path

CoreML is not the only way in. The _ANEClient class provides direct access to the compile → load → evaluate pipeline.² The full chain: _ANEClient → _ANEInMemoryModelDescriptor → MIL (Model Intermediate Language) → E5 compiled binary → IOSurface I/O. No CoreML, no Metal, no disk-based compilation needed.

MIL is a typed SSA representation. Linear layers are expressed as 1×1 convolutions because the ANE is fundamentally a convolution engine.⁴ Tensors use NCDHW format [1, Channels, 1, Spatial].

Training Architecture (6 kernels per step)

The Weight-Baking Problem

Apple designed the ANE for inference — weights are irrevocably baked at compile time.⁴ For training, this means recompiling every batch when weights change. Combined with a ~119 compile limit per process (the compiler leaks resources),¹ the workaround is: async compilation overlapped with evaluation, gradient accumulation across 10 steps, and exec() process restart with checkpoint/resume to reset the compiler.

Power Efficiency

IV. Donna's Reproduction Attempt

Hardware: M4 Max MacBook Pro, 128GB RAM, macOS 15.3.1. ANE framework confirmed at /System/Library/PrivateFrameworks/AppleNeuralEngine.framework/.

Step 1 — Clone & Build

git clone https://github.com/maderix/ANE.git — 7 seconds, clean. Built peak benchmark (inmem_peak.m) and training programs (train_large, tiny_train) with zero compilation errors.

Step 2 — Peak Benchmark: ANE Responds

Step 3 — Training: Loss Converges

Ran the 2-layer tiny_train program (x:[16,64] → W1:[128,64] → ReLU → W2:[64,128] → y:[16,64]). 2,000 steps, gradient accumulation every 10 steps, pipeline-parallel compilation.

step 0     loss=0.500921    ANE=1.95 GFLOPS
step 200   loss=0.499994    ANE=1.17 GFLOPS
step 400   loss=0.360518    ANE=1.04 GFLOPS   <-- learning kicks in
[exec() restart at step 480, 100 compiles, loss=0.121289]
step 780   loss=0.042563    ANE=1.24 GFLOPS
step 880   loss=0.003584    ANE=1.32 GFLOPS
[exec() restart at step 960, loss=0.001957]
step 1840  loss=0.001435    ANE=1.27 GFLOPS
step 1990  loss=0.001428    ANE=1.27 GFLOPS   <-- converged

Step 4 — 12-Layer Stories110M

Compiled train_large (the full 12-layer, 109M-param Llama2 architecture) without errors. Initialized successfully: "dim=768 hidden=2048 heads=12 seq=256 vocab=32000 layers=12, Params: 109.53M, Kernels: 72." Requires TinyStories dataset (~993MB) for actual training — not attempted for time, but the compilation and initialization prove the pipeline works on M4 Max.

Step 5 — What Failed

inmem_bench and sram_probe return FAIL(-1) on M4 Max — these benchmarks probe SRAM topology at specific channel configurations calibrated for M4 base. Not a training blocker.

V. Why Roemmele's ZHC@Home Claim Is Physically Impossible

The claim: "At 20% idle from just a few million opt-in M4 Macs, ZHC@Home could rival or exceed single massive data center clusters (e.g., 1-2x Colossus scale) in raw FP16-equivalent compute."⁶

1. The Bandwidth Wall. Home internet upload (0.026 GB/s) means 26 minutes to sync gradients for a 10B parameter model. NVLink does it in 22ms — a 70,000× gap.¹¹ Even DiLoCo-style algorithms that minimize inter-node communication can't close a gap this large.⁹ macOS Tahoe 26.2 added local Mac clustering via RDMA over Thunderbolt 5,¹⁵ but that's local networking — not distributed internet training.

2. Utilization Reality. maderix himself says ANE utilization is ~2-3% of peak.¹ The optimized single-layer case hits 11.2%. CPU operations dominate training time by 10×: classifier matmul 9.1ms, cross-entropy 14.4ms, vs ANE eval 9.6ms for the 12-layer model.⁴

3. Raw Compute Gap. At theoretical 19 TFLOPS peak per Mac, 5 million Macs = 95 exaFLOPS. Colossus (xAI, 200K H100s) ≈ 400 exaFLOPS. But sustained ANE training throughput is 1-2 TFLOPS, not 19. Real aggregate ≈ 5-10 exaFLOPS — 40-80× less than Colossus.³

4. Training vs Inference. The ANE was designed for inference. Weights baked at compile time. Recompile every batch. 119 compile limit per process.¹ These are fundamental hardware constraints, not software bugs.

5. "Wages" Are Fantasy. A reply thread calculates $1-2.80/hr per Mac Studio.⁶ No distributed training network has ever paid participants. Folding@home and BOINC are volunteer. DePIN networks (Render, Akash) pay for inference, not training.¹⁰ There is no demand side for distributed ANE training compute.

VI. Feasibility Verdict

Donna's Verdict

Eric's Verdict

VII. Critical Assessment

Is the output the skill, or is the tool the skill? The real skill is systems-level reverse engineering: Obj-C runtime introspection, binary format analysis, hardware topology inference via scaling experiments.² ANE training itself is a proof of concept, not a production capability. Utilization is 2-3% of peak with fundamental architectural constraints.

Opportunity cost. Learning deeper ANE internals would take weeks with zero payoff for Eric. The investment thesis insight (ANE efficiency → Mac inference value) can be absorbed in 5 minutes. Correct move: note the data, don't go deeper.