Moving a running model’s layers between GPUs mid-reply (8 ms, no re-prefill)

I've been working on an inference engine called jitllm. The idea came from my background in operating systems and JIT compilers: generate kernels for whatever hardware is there, and treat layers a bit like processes...