New server has been acquired. Debian 13 has been installed.
GS308EP switches have been acquired and installed.
Now, I’m working to migrate to the new machine. 3 1/2 years ago when I started futzing with Docker, I sorta followed guides and guessed, abused it trying to make it do things it wasn’t designed for, and flipped switches I likely shouldn’t have flipped, so the set up is more than a little shabby.
As a result, I’ll likely end more redeploying than migrating the containers.
So rather than go forward with Docker blindly, I want to reassess whether I shouldn’t look into Proxmox, LXC, or Podman instead of Docker, or maybe something else entirely?
Work is just about done dumping ESX for Nutanix, but both of those seem overkill for my needs.
Of course the forums for any of the solutions make their own out to be the best thing since sliced bread and the others useless, so I’m hoping to get a more nuanced answer here.
You must log in or register to comment.
There is no wrong answer but the right answer is Proxmox.
If you’ve got Debian already installed, I cannot resist advocating for Incus (stable branch from Zabbly repo with web ui https://blog.simos.info/how-to-install-and-setup-the-incus-web-ui/) in lieu of proxmox. Does the same thing but you don’t have to rip out the kernel Debian uses.
With Debian 13 you have access to podman quadlets, use that for any non-vm needs. The ease of docker compose files easily removes reason for programs in LXC containers, and podman removes reason for docker in an LXC. LXC is left only for programs that aren’t containerized. VMs for security DMZ. Podman for bulk of stuff you want.
Good luck!
I was in a similar spot not too long ago, setting up a firewall and general network box. I was going to go with Proxmox but a fellow Lemmy guy strongly advocated for Incus on top of vanilla Debian. I was intrigued and ended up going for it. Learned a lot about networking with systemd (bridging, IP assignment and so on) for things I could have gotten for free in Proxmox (literally a few clicks), and had to fight Incus to work with a FreeBSD VM for Opnsense, but I love the setup now. Pure debian with a few Incus VMs and Docker inside of those as needed. So clean!
Came in here to mention Incus if no one had.
I love it. I have three “home production” servers running Proxmox, but mostly because Proxmox is one of very few LTS/comercially-supported ways to run Linux in a supported way with root (and everything else on ZFS). And while its web UI is still a bit clunky in places, it comes in handy some times.
However, Incus automation is just… superior.
incus launch --vm images:debian/13 foo, wait a few seconds thenincus exec foo -- bashand I’m root on a console of a ready-to-go Debian VM. Without--vm, it’s a lightweight LXC container. And Ansible supports running commands throughincus exec, so you can provision stuff WITHOUT BOTHERING TO SET UP ANYTHING.AND, it works remotely without fuss, so I can set up an Incus remote on a beefy server and spawn VMs nearly transparently. +
incus file pull|pushto transfer files.I’m kinda pondering scripting removal of the Proxmox bits from a Proxmox install, so that I just keep their ZFS support and run Incus on top.
I use proxmox has the base OS to deploy different virtual machines/LXCs depending on what I need. I have a dedicated docker lxc among those. Say what you will about docker, and man do people have opinions, but proxmox is probably the best way to run a multi lxc/VM setup. Highly recommend regardless of what else you do.
I have a dedicated LXC container called docker where I install all my docker images.
@Vendetta9076 @JeanValjean I happened to be searching on running docker inside lxc today. There were quite a few suggestions on configs to make them not go kurplunk. I think i might follow them…
It’s not worth the headache IMO. Just run a docker VM and use lxc for the one-off systems that you want to experiment with.
I have a “production” docker VM and a “sandbox” docker VM and prod only ever runs compose files that I’ve vetted in sandbox. Super stable, basically bulletproof, and still has the flexibility to experiment and break stuff without affecting my core services.
I use systemd-nspawn to absolutely pissoff everyone :)
systemd-nspawn launched virt-qemu-run instance of proxmox running a windows hyper-v vm
Docket-compose is the easiest and most commonly used format imo
Proxmox is great. I run each service in its own lxc/vm. Many of the vms also run docker. Probably overkill, but backups and restores via Proxmox Backup Server are super easy when segregated this way
This. Unless you want to go through the hell of multiple redundant architecture to deal with updates breaking things and backups. If you are The only person who is maintaining the multiple services that you’re running the backup feature alone is the reason to go with Proxmox. Upgrade to a lcx container didn’t go like you thought it would, roll back the backup, 30 sec and done. Also making liberal use of alpine have made it smooth and simple.
where do you run PBS? in a vm on the main proxmox host?
Maybe you could try nixos?
Hahaha. I see you’re being down arrowed a bit… But I was thinking the same thing.
Reason I was thinking nixos was because op was talking about " flipping switches" they couldn’t flip back. Nixos has that sorted out.
Plus… It sounds like op isn’t afraid of trying new and possibly difficult things on a new server they’ve acquired
The truth is that proxmox is likely the best option here. But nixos could be a fun ( type2 fun) challenge if they’re into that kind of thing
Especially without any additional context or knowledge about their background, directed at someone clearly only starting out, this is incredibly bad advice.
Edit: typos (italic), sorry that was probably hard to read.
Oh I didnt see the downvotes😅
Hm, to explain my reasoning:
-
I saw them talking about stuff that seemed complex to me and seem to have some experience with servers (already had a server), so I’d say they know way more about me and I’m going to set up a nixos server in like a month (been using them on my laptop for ~9months, but not in any fancy way). If I can, they I guess they probably can too.
-
I thought about throwing an idea to consider. I’m too in the process of learning so I cant give a good plan yet:/
I see that in the end they ask for a nuanced answer, rip, I failed to do that
-
Proxmox can run lxc containers natively.
Personally I keep a Debian VM for docker, a holdover from before hypervisors supported containers natively. I use docker compose and it Just Works™.
Definitely Proxmox as base OS, Docker inside.
Run Docker inside a LXC or VM
and run Proxmox inside that VM
What?
they claim 3% overhead, I bet you could get pretty deep before noticing.
Proxmox with all of its warts appears to be the better hypervisor. If your host has the headroom like others suggested: proxmox with a VM hosting docker/podman.
I have to say running proxmox saved me from running to the basement or opening up IPMI a few times already when making suspect changes to a VM that otherwise would have taken down the network of my host.
ATM I’m running Proxmox with Nixos VMs running mixed docker/podman containers. It works out pretty well for my use case and with some opentofu fiddling I have most of my infrastructure defined in config files if that’s a rabbit hole you want to go down.
As for my experience with docker vs podman I have to say podman can pretty much do whatever docker does with the exception of docker swarm. You may have to do some digging to handle more advanced networking/gpu setups.
The only thing I have running docker atm is a gluetun container because container to container networking took more than 15m of research with podman so I fell back to the very well documented path using docker.
i just came into containerization recently and skipped Docker completely and went straight to podman. it’s been fantastic and integrates well with systemd. i would recommend anyone new to do the same.
since you’re not new, and are looking for something different, i would say do podman instead of docker.
I concur. Podman is superior in my opinion. It’s more secure by default (rootless containers) and can do pretty much everything docker can do naively (you can literally alias docker to podman in your shell and it will work)
It’s not as easy to find info on some of the systemd specific stuff (Quadlets), but once you figure that out, it’s pretty amazing.
I ended up making up my own scripts to allow me to create new system users, pre-loaded with aliases and shortcut functions to make my life easier ( automatic quadlet container file generation, pre-set network rules, etc), but it is not required.
All the info is there, but starting out it can be a bit overwhelming.
My containers are pretty much self sufficient now. I just intervene when something needs major updating or config changes
Both. Depends on what you want to achieve
docker in a vm in proxmox!
no but really. I don’t like that the vm host’s kernel is shared with all the containers.
then maybe have 2 or 3 VMs, one of which hosts essential services (not like your nextcloud, but services that are necessary to make the (internal or whole home) network work like DNS and DHCP, maybe monitoring, and another that hosts your convenience services (jellyfin, nextcloud). the first is easier to fix if it breaks, the second is not the end if the world if it does. maybe also have a third for services that would be painful to have offline (nextcloud), but try to limit it to few tasks because the goal is simplicity. this setup can help with distro upgrade difficulties, and vm custom configuration issues when you have set up something that seems to work but will prevent VM startup sometime in the future.podman could be a good idea too.
Proxmox and Docker don’t really do the same thing. They live in the same area, but the coverage is very different. You can always use docker when your host is running proxmox: either individually or in groups inside of an lxc, or all in w dedicated VM, or even natively on the same house if you prefer chaos. But you can’t do the opposite: Sometimes you just need a VM. Maybe you only need a couple of devices, and you know they run on or are even designed for docker, then that’s the better option. In all other cases, and when just getting started, proxmox is just the way more universal solution if you’re only planning on having a single host (for now).
The management tools in proxmox are great. The community scripts are a fantastic resource and only work with proxmox. I would suggest you set it up natively, not on top of Debian though, even if that’s already installed. Not the least of the reasons are to be able to use ZFS easily, including on on the boot partition (select that in the installer).
Finally, if you’re gonna stick with docker, like others said: consider podman. That really does the same thing docker does, but it’s fully open source. Arguably it’s better in some areas, but on the flip side might, in occasion, require fiddling with something intended specifically for docker and using advanced setups.
Also there really is no wrong answer, either. And you can always change whatever you choose.
Maybe use Docker with a UI like
lazydockerorPortainer.If you feel safe with Docker, perhaps experiment with Proxmox as a base. Then you can separate your environments into different VMs.
