arm64 macOS machines

macOS 26 is the last release for Intel Macs. Every macOS after it runs on Apple silicon only, so a macOS machine that keeps getting updates has to be an arm64 one. This page covers qemu-vmapple, the QEMU that runs one: QEMU's vmapple machine under TCG with an emulated M1 CPU on any PC, or under KVM on an arm64 host, with a GPU from Reims on either.

warning

Apple's licence lets macOS run in a virtual machine only on Apple hardware. Booting it on a PC breaks that licence. Nothing in this repository or the image contains macOS or Apple's boot ROM, and CI never boots macOS. Whoever boots it supplies both, from their own Mac.

Where it runs

Host	How the guest runs	State
An Apple silicon Mac running Linux	qemu-vmapple under KVM, as Asahi Linux runs it; macOS needs two host-kernel patches	KVM checked on an emulated arm64 host, not yet on a Mac
Another arm64 machine with KVM and pointer authentication	qemu-vmapple under KVM	the same check, with EL0 sharing EL1's keys
Any PC, x86-64 or arm64	qemu-vmapple, every instruction translated by TCG	the emulator and its check are built, macOS not yet booted on it

vmapple is the virtual Mac that Apple's own hypervisor (Virtualization.framework) gives a macOS guest. It is not a model of a physical M1. Its devices are paravirtual, so QEMU has most of them already, and the guest kernel is XNU's VMAPPLE build. What a guest needs from the CPU is what an M1 shows it, plus a few hypercalls.

What qemu-vmapple is

nixos/pkgs/default.nix builds it from nixpkgs' QEMU with the patches in nixos/pkgs/patches/qemu-vmapple, for the aarch64 system emulator only.

• 0001 is the code of 27on86, steelbrain's fork, at 509a4ce. It lets vmapple start without HVF, sets the guest's 24 MHz timebase, and keeps guest time steady across several vCPUs. With it, macOS 13 boots to a shell over SSH on an x86-64 PC. It accepted every pointer signature, because nobody outside Apple has the cipher an M1 signs with.
• 0002 adds an apple-m1 CPU, an M1 performance core, and makes it vmapple's default under TCG.
• 0003 answers Apple's pointer authentication hypercalls and checks signatures again.
• 0004 runs vmapple under KVM (see "Under KVM").
• 0005 adds apv-gpu-proxy, a GPU served by another process (see "A GPU").

The apple-m1 CPU

QEMU's max CPU offers every feature QEMU emulates. That includes SVE, SME and MTE, which no M1 had. The VMAPPLE kernel probes for SME at run time and would turn it on. The apple-m1 CPU shows the guest an M1's feature set instead. The sources are macOS's own sysctl hw.optional.arm on an M1 and the VMAPPLE kernel's build configuration (pexpert/arm64/VMAPPLE.h):

MIDR_EL1	0x611f0231, implementer Apple, part 0x023 (t8103 Firestorm)
Present	AES, PMULL, SHA1, SHA256, SHA512, SHA3, CRC32, LSE, LSE2, RDM, FP16, FHM, DotProd, JSCVT, FCMA, LRCPC2, FlagM2, DPB2, SB, FRINTTS, SSBS2, DIT, CSV2, CSV3, ECV, PAuth with Apple's own algorithm
Absent	SVE, SME, MTE, BTI, BF16, I8MM, RNDR, SPECRES, PAuth2, FPAC
VMAPPLE.h asks for	TLBI by range, 16K pages, a GICv3
Pages	16K and 4K, no 64K
Addresses	36 bits, what Apple's hypervisor gives a guest on an M1

The PMU stays the architected PMUv3 that 27on86's guest booted with. An M1's own counters are Apple's and are not modelled.

Apple's pointer authentication

A guest of Apple's hypervisor never writes its pointer authentication keys. XNU's VMAPPLE kernel asks the hypervisor for them, through these hypercalls (osfmk/arm64/hv_hvc.h):

Function	Call	What it does
0xc1000000	SET_INITIAL_STATE	every key to its default
0xc1000001	GET_DEFAULT_KEYS	the default A, B, EL0 and G keys, in x1 to x4
0xc1000002	SET_A_KEYS	the A keys, which the kernel signs its heap with
0xc1000003	SET_B_KEYS	the B keys, each thread's return-address key
0xc1000004	SET_EL0_DIVERSIFIER	a process's own key for EL0
0xc1000005	SET_EL0_DIVERSIFIER_AT_EL1	the same, and whether EL1 uses it too
0xc1000006	SET_G_KEY	the key for PACGA
0xc10000f0	NOP	tells a current host from an old one

Without 0003 those calls returned "not supported". macOS 13 never looked. From macOS 26 on, XNU checks the result of SET_INITIAL_STATE and spins forever if it is not 0, so that guest would stop at its first one.

On an M1 the EL0 key lives in the KERNKEY register pair, and APCTL says which exception levels mix it into their keys. Asahi's register notes and m1n1's hypervisor name both. The KVM patches in aquarat/apple-macos-arm64-vm-guest showed which values Apple's hypervisor puts there. Under TCG the EL0 key is mixed into QEMU's own PAC algorithm instead of Apple's. The guest cannot tell the difference, because it only ever compares signatures this CPU made. What it does rely on now holds:

• A pointer EL0 signed fails at EL1, and the other way round.
• EL1 accepts EL0's signatures while the kernel borrows EL0's key.
• A new key breaks every pointer signed with the old one.

-machine vmapple,pac-auth=off brings back 27on86's behaviour, which accepts every signature. That is for comparing against a guest that booted before 0003.

Under KVM

On an arm64 Linux host, -machine vmapple,accel=kvm runs the guest on the host's own CPU. 0004 is aquarat's KVM support from qemu-reims-vgpu, which runs macOS 13 and 26 guests on Asahi Linux every day:

• KVM passes Apple's pointer authentication hypercalls up to QEMU through an SMCCC filter (Linux 6.4 or later), and QEMU writes the vCPU's key registers.
• PSCI, the GIC and the timers stay in the kernel. Every timer, the GIC maintenance interrupt and the PMU get their PPIs, and KVM's PMU is set up before the first run.

The EL0 key needs Apple's KERNKEY and APCTL registers, which only an Apple silicon CPU has. KVM shows them to QEMU only with aquarat's VM-key patch in the host kernel. On any other arm64 host, and on a Mac without the patch, QEMU says so once and EL0 shares EL1's keys. The guest still signs and checks its pointers consistently, but one process could forge another's.

A macOS guest under KVM also stores to its GIC with instructions that give KVM no syndrome. aquarat's second host-kernel patch emulates those. Both patches are in apple-macos-arm64-vm-guest (patches/linux-7.1.13-*). A Linux guest needs neither.

The guest sees the host's CPU, not apple-m1: KVM cannot change what the CPU it runs on does.

A GPU

macOS already has a driver for vmapple's paravirtual GPU, AppleParavirtGPU.kext. On a Mac, Apple's ParavirtualizedGraphics framework serves it. On Linux, Reims does: it decodes the driver's command stream and runs the guest's Metal on the host's Vulkan, translating Metal's compiled shaders to SPIR-V with metal2vulkan. Nothing is installed in the guest and the image is not patched.

Reims runs as its own process, reims-vgpu-backend (in the overlay), and 0005's apv-gpu-proxy device connects vmapple's GPU slot to it. Guest RAM must be shared memory so the backend can map it:

reims-vgpu-backend --socket /run/user/$UID/gpu.sock &
qemu-system-aarch64 \
  -machine vmapple,gfx-device=apv-gpu-proxy,aux-ram-share=on,memory-backend=ram \
  -object memory-backend-memfd,id=ram,size=8G,share=on \
  -global apv-gpu-proxy.socket=/run/user/$UID/gpu.sock \
  ...

The backend uses whatever Vulkan driver the host has (VK_DRIVER_FILES picks one). On Asahi, aquarat renders on Honeykrisp with patches to Mesa. On a PC it is the PC's own driver. If the backend dies, the guest stops with the reason, so a host driver crash does not take QEMU down with it. gfx-device=none keeps 27on86's inert windows for a headless guest.

Running a guest

You need three things from a Mac, none of which the image carries:

• AVPBooter.vmapple2.bin, the boot ROM, from /System/Library/Frameworks/Virtualization.framework/Versions/A/Resources/
• the guest's AUX and root disks, restored from Apple's IPSW on the Mac, for instance with macosvm
• the guest's ECID, from that restore

Then:

qemu-system-aarch64 \
  -machine vmapple,uuid=$ECID,accel=tcg \
  -smp 4 -m 8G -nographic \
  -bios AVPBooter.vmapple2.bin \
  -drive file=aux.img,if=pflash,format=raw \
  -drive file=root.img,if=pflash,format=raw

27on86's scripts and experiment records cover the rest: serial access, an SSH service, boot arguments and how it was validated.

How it is checked

The apple-m1 flake check (nixos/tests/apple-m1.nix) builds a small firmware from nixos/tests/apple-m1/ and boots it on vmapple under TCG. It reads the CPU's ID registers, makes every hypercall above, and signs and authenticates from EL0 and EL1. It then runs again with pac-auth=off. It needs no Apple software and no KVM, so it runs on both architectures' builders. The same firmware on the unpatched fork fails the feature checks and the first three hypercalls.

Built with -DGENERIC_KVM_HOST, the firmware is for KVM on an arm64 host that is not a Mac. It skips the CPU checks and the six that need separate EL0 keys. CI has no arm64 host with KVM, so this one is run by hand: on an emulated Neoverse V1 running Linux with KVM, all 28 checks pass.

What is not done

• Booting macOS on it. Nobody has booted a macOS guest on these patches yet. The next steps are 27on86's macOS 13 guest with authentication on, then macOS 26, then the same under KVM on a Mac.
• Installing macOS without a Mac. The restore needs Virtualization.framework.
• Metal from a guest. Reims and the device are built, and Reims' own Vulkan suites pass on lavapipe, but no macOS guest has drawn through them in this build. Under TCG on a PC the device and the backend connect and map guest RAM, but nothing has drawn through them yet.
• The host-kernel patches. LosOS does not boot on an Apple silicon Mac yet, so it carries no kernel for one.
• Speed. TCG translates every instruction, so expect a guest many times slower than on a Mac.
• A machine in derisk. Nothing creates or starts this guest from the desktop. Once it boots, macos-app-server inside it and app-windows on the host would show its apps as desktop windows, over TCP port 7180 with a shared token. Intel macOS on a PC takes the other route, dockur's container under rootless Podman.