> That is, a LXD container runs a full Linux system, exactly as it would be when run on metal or in a VM.
This is what makes LXD so nice. It's the power and flexibility of a VM with the speed, api, and runability of containers.
I always hated that docker forced you to treat a container as a single process and spoke about it as if it's a virtue. Let me decide how many or little processes I want running in a container. Even google runs ssh server alongside their applications.
With LXD you can easily port entire services over to containers without completely rewriting the way they behave.
Indeed, LXD would seem to be a faster, more efficient way of doing what you would do with traditional VMs. I don't see a lot of overlap with docker, despite the superficial similarity.
The major security issue is that containers share the same kernel as the host, with the whole kernel syscall interface as attack vector. VMs run separate kernels that only interface with the host via the hypervisor.
Well, there's also the issue that "Linux containers" are not a kernel primitive. So there's plenty of syscalls that aren't even namespaced (the keyring ones come to mind). Docker handles this by having a default seccomp profile that disables a bunch of syscalls that aren't namespaced.
However, the development of Zones and Jails (on SunOS/Solaris/illumos and FreeBSD) was much more security focused, with the default being "that's not allowed in a (Jail|Zone) until we can make sure it's safe". I really wish Linux had just ported Jails or inspired their security model on them.
I do a lot of work with the internals of Docker and runC at SUSE. Trust me, it's not pretty how you have to set up "Linux containers" and there's 1001 gotchas.
> I always hated that docker forced you to treat a container as a single process and spoke about it as if it's a virtue. Let me decide how many or little processes I want running in a container.
You can have as many processes as you like running inside a Docker container, you just a need a single process to "boot" them, even if it's a simple shell script.
There are quite a few issues with running a systemd-based distro in a container. The first being that /sys and /proc are basically not namespaced. So if you're not careful, systemd might start modifying your host. There's some work to improve this (the cgroup namespace -- yes, that's a thing now) but the point remains that allowing a root process to start touching /proc and /sys inside a container won't end well. I believe that LXD tries to get around this problem by mounting a FUSE filesystem on /proc and /sys that does a bunch of fakery. I don't like that, to be honest. It should be the kernel's job to namespace things properly.
Congratulations on 2.0! I tried out LXD a few months ago after hitting a few walls with Vagrant, and I've been happy since.
Mark Shuttleworth's normally responsive on Google+, and he helped point me in the right direction with initial issues that I encountered.
Stéphane and the team have also been helpful on Github with issues.
The first thing I struggled with in this containers business was "what's the difference between LXC/LXD and Docker et. al.?", and I am glad that he addressed it first in the post.
I've got north of 20 containers on our machines, and LXD allows me to create a container, play around with software config on it, and when I get it right; I just leave that container and update it regularly like the host it's on.
You can also use LXC as Vagrant backend to get away from the crashy and buggy VirtualBox kernel driver. Though ~all the Vagrant images you find online are tied to VirtualBox.
I build out major productions systems on top of LXC at Heroku, and to this day it is still working incredibly well.
So I am really looking forward to further iterations of the tools by Stéphane Graber and co at Canonical.
The main thing that jumps out at me is a REST API for this layer. A REST API with Golang bindings to run containers feels so much better than old-school systems programming with exec.
I'm currently all in on Docker in production for this very reason. Coordinating stuff with a golang client over HTTP is really really productive.
If I need to run Linux software, I can use an lx-branded zone inside of SmartOS, and run at the speed of bare metal while having ZFS, mdb, kdb, dbx and DTrace, CPU-bursting, and change kernel schedulers on the fly.
But then, as I am about to run Linux software inside of an lx-branded zone on SmartOS, I realize that SmartOS has 15,000 packages, with most the of the software I would have run on Linux, available natively on SmartOS:
You can think of it as competition. Other people can think of it as something that they can make people who are already familiar with systems that use the Linux kernel.
SmartOS is paranoid about correctness of operation and data integrity; Linux and LXC containers are a poor competition in that area, when the OS comes with memory overcommit turned on by default, and default filesystems do not include data write guarantees nor block checksums...
No sarcasm. This is a simple explanation of where LXD fits into the stack.
If you don't know what LXC is maybe this isn't clear. But LXC stands for LinuX Containers, and is extremely fundamental technology in the containerization space.
(You should possibly have replied to the post above me)
> LXC stands for LinuX Containers, and is extremely fundamental technology in the containerization space.
Not exactly.
LXC was amoung the first pieces of software to take advantage of namespaces when they were added to the kernel; but LXC is not often used. Most of the various containerisation software now available does not use LXC, and it not based on the same codebase.
I like almost everything about LXD, except the attitude towards networking. The project has rejected simple solutions such as port forwarding, saying that managing the network shouldn't be LXD's job. Instead, they'd like the user to manually configure their own bridges or routes or iptables chains.
I can kinda understand their point of view, there's no simple solution that will please everyone. But most developers or IT folks aren't networking experts, and LXD won't be an intuitive tool for them without a simpler mode of operation.
I have just finished a small LXC deployment, and the networking did take a little bit of time to figure out what direction to go.
I ended up choosing to use bridging to connect all the physical network adapters to the ones in the containers. This is nice because I set up each container with its own IP address, which travels with it when the container is moved to a new host.
> [...] most developers or IT folks aren't networking experts, and LXD won't be an intuitive tool for them without a simpler mode of operation.
Really, no expertise is required. Just basic understanding how the heck the
network works. If somebody can't grasp what bridge interface is or how NAT
operates, he apparently doesn't have the qualifications for writing software.
It simultaneously irks me and pleases me that the first result on DuckDuckGo for "lxd" is the Legion of Extraordinary Dancers.
(Google gets it right, presumably because of all my code searches.)
"[...] a 2010–2011 web series about two groups of rival dancers: The Alliance of the Dark who are the villains and The Legion of Extraordinary Dancers, the heroes, who discover they have superpowers referred to as 'the Ra' through their dance abilities."
I'm moving my home NAS back to Debian after a couple of painful months [1] [2] with FreeNAS 9. Would this give me jail-like functionality under Linux? The one thing I enjoyed about FreeNAS/FreeBSD was how easy it was to create and tear down jails for all kinds of uses.
LXC is very nice. It's very VM-like and why I prefer it for almost everything I deploy instead of using Docker.
I know how to manage machines. I don't have the same confidence for Docker-images.
LXD seems to be a construction on top of LXC to make managing it easier "at scale", so if all you want is to manually be able to construct "container VMs" on demand, I think LXC is closer to what you actually want: It let's you manage the containers directly, near the iron, without anything getting in your way.
Don't get me wrong: LXD seems to have a genuine value-proposal, but for this use-case it seems to be a slight step of indirection, and my guess is that this will cause you more troubles than it's worth.
I want to understand how Linux Container works, started to study the cgroup code in Linux kernel and LXD. Here are couple of notes on Tracing the code of CGroup in Linux kernel and LXD I took.
Does LXD provide any type of cluster management behavior? As in if I have two host machines and a dozen containers and host A goes down, is it possible to configure a live migration of the latest known state from host A to host B using LXD alone? Or do I need something built on top of it to do that?
LXD and any sort of cluster seem incompatible at the moment. LXD, for example, stores all the LXC config info in a local sqlite database. When you launch a container, it turns the sqlite stored info into an LXC config file, and spawns LXC.
You can control remote instances of the LXD daemon, but there's no syncing back and forth from Host1-LXD1 to Host1-LXD2...they operate separately. You can migrate a container from one host to the other, but it's pushing everything (config/image/etc) when you ask it to migrate.
So, it's not clustered itself, and it keeps the important info somewhere that's hard to cluster.
It would be interesting if they would replace the local sqlite store with something similar to etcd. That still leaves the issue of the disk images though.
Is anybody making an effort to put the LXD environment on Centos 7? I'd really like to try it but all I have provisioned right now is Centos 7 (and below).
This is what makes LXD so nice. It's the power and flexibility of a VM with the speed, api, and runability of containers.
I always hated that docker forced you to treat a container as a single process and spoke about it as if it's a virtue. Let me decide how many or little processes I want running in a container. Even google runs ssh server alongside their applications.
With LXD you can easily port entire services over to containers without completely rewriting the way they behave.