As an alternative to mounting full /dev/dri, there are symlinks under /dev/dri/by-path/ which you can use. IIUC they are linked by the used PCI slot, hence should be as stable as the physical order of the cards.
I use this pattern in a compose file:
...
devices:
- source: /dev/dri/by-path/pci-XXX-render
target: /dev/dri/renderD128
permissions: rwm
In case you are unaware, friends don’t let friends use ollama. The llama.cpp CLI has become pretty much equally easy to use, plus since you use Docker there’s no need to compile anything.
Edit: typos and missing words
Haha, that sounds like procrastination to me.
Welcome.
The service starts as it should, right? The errors appear only on request, right? Any other sections in the compose logs that seem relevant?
You use podman-compose with the Podman backend?
And there are no volumes mounted, just to be sure?
No expert but don’t the logs look like the requests are coming through?
I guess some people consider it cool. I’m not one of them.
Yeah right? At least a layer of VPN on top.
Hugging Face used the incident to promote running open-weights LLMs on-prem, which a large part of their business focusses on.
Pixelfed
Well compared to the strix, 400GB/s is not that bad, I think with fast system RAM and expert offloading you could squeeze quite something out of it when running stuff in the 100b-a10b regions.
Your bigger problem is going to be future software support.
In case you missed the Ornith 1.0 release (Qwen and Gemma RL finetunes for agentic / coding workloads), they look interesting to bridge the gap until we see larger 3.6 models or a 3.7 release. I didn’t test them yet but according to benchmarks, the 35b MoE seems to be more or less on par with Qwen3.6 27b dense, while ofc a lot faster.
You can control how much context should be fitted with --fit-ctx and how much space the algorithm should leave unallocated (even on a per-GPU basis) with --fit-target.
I currently run Qwen3.6-27b on llama.cpp and use it via openwebui. Mostly, I use it for web research via tavily, to a lesser extent for coding and interactively learning about things that are new to me but common in training data (such as basic math or ML concepts).
Given the 27b is a dense model, I think the numbers are quite ok. Curious about the quant tho.
The cool thing about the strix is its large unified memory, but it lacks memory bandwith for compute intensive workloads. Something like Qwen3.5-122b MoE with only like 12b active parameters might run at twice the speed if it fits the configuration.
Since implementation of the --fit parameter and its relatives, and --fit on becoming the default, llama.cpp intelligently decides what to offload. For me, it made --n-cpu-moe obsolete.
Your biggest issue with 2010 cards will be software (inference engine) support, I assume.
I used to run SFTPGO in a container as a NAS type service. Their default image is based on Debian, but they offer Alpine versions. Password authentication is enabled by default.