↓ Skip to main content

Qwen

Proxmox, vLLM and Hermes: the software on my local AI PC

·1549 words·8 mins
The perfect software, like the perfect hardware, doesn’t exist: it depends on what you want to achieve. I favour stability and reliability, and I always use open-source software or, in the case of LLMs, open-weight models. In AI everything changes at breakneck speed: chasing every new release is impossible and not very useful either, but you often find yourself having to try a new version of something or tweak a delicate parameter. It takes next to nothing to break everything, beyond repair. Without a solid safety net, the PC stays out of action until you find the time to fix it.