muppeth

Interesting. Is there a non-apple solution like this?

is the mac mini really that good? running 12-14b models on my radeon rx 7600xt is ok’ish but i do “feel it” while running 7-8b models sometimes just doesn’t feel enough. I wonder where does mac mini land in here.

Wow! very cool rack you got there. I too started using mini pcs for local test servers or general home servers. But unlike yours mine are just dumped behind the screen on my desk (3 in total). For LLM stuff atm I use 16GB radeon but thats connected to my desktop. In the future I would love to build a proper rack like yours and perhaps move the GPU to a dedicated minipc.

As for the upgrades, like what others stated already, I would just go for more pc’s rather then rpi.

This is great. I have couple of those HP machines which are awesome but was just stacking them on my desk. 10 inch rack will be great for them. Need to do some hunting.

Disroot founder here 👋 if you have any questions I would be glad to answer. Been doogfooding all services myself as primary communication means. We are small team of ethusiasts and we mainly reuse code writtem by awesome FLOSS contributors in form of services. Not reinventing wheel here. Next year we will celebrate 10 years of existance \o/