PR speeds 300GB model loads from 4m54s to 1m38s; GGML_RPC_LOAD_THREADS optimization on 4060Ti platforms reduces load time to ~1.5 minutes
Read the original at old.reddit.com→- On b10173 - "state":"loading" 4min54sec. - With this PR and GGML_RPC_LOAD_THREADS 12 - "state":"loading" 1min38sec The PR is close to ready, will need a docs change if they want to keep the new...
Original headline: "I got tired of my 300GB model loads taking 5min on RPC. PR 26291 speeds it 300% to 1min30sec (4060ti+ddr4) + (4060ti+ddr5)"
Coverage timeline
- Aug 8, 03:37 UTC r/LocalLLaMA lead source I got tired of my 300GB model loads taking 5min on RPC. PR 26291 speeds it 300% to 1min30sec (4060ti+ddr4) + (4060ti+ddr5)