Hey everybody, my name is Eric Ernst, I'm excited to be here to talk about our open source
projects and some of the community work that we're doing around containers on macOS.
So last June, our team at Apple Open Source, the Apple Containerization Framework and
Container Julien projects, which enabled developers to be able to create, run, build, push,
and continue to image directly on their Mac in a way that focuses on security and privacy.
Really, what we did is we introduced two key projects, one of them, a command line tool called
Container, naming as hard, and it allows users, again, to quickly be able to get
up and running containers on top of a macOS operating system.
The second thing that we open source project wise is Containerization, and this is a framework
that allows us to build solutions like that command line tool container.
Users will typically interact with just a container that command line tool, but the core
functionality for the project is really behind simply the eyes in Containerization.
So I want to talk about what I'm going to focus on in this quick session, so I'm going
to talk about some of the design principles that guided the work that we're doing.
And then I want to briefly talk about how we designed it to be extensible in this example
is how developers can go ahead and do that, as well as showing some of the key APIs
in Containerization, again, motivation that is pretty easy to build on top of this.
We'll talk about how we wrote it in Swift, and what that experience has been.
It was a new process for the team, and then we're going to talk a bit more about just
overall resource community and what we're doing going forward.
So first off, let's talk about the actual design side of things.
So in order to run the Linux container on macOS, you need to virtualize a Linux environment.
A historical solution is to spawn a large virtual machine for all the different containers.
Resources are allocated to that one virtual machine.
The containers are added to it.
These resources are devied up per container.
When you need to share additional directories and files from your Mac, they're passed
to that virtual machine before then they are then passed to the corresponding container
with the needs it.
So one of the areas we focused on in the security, and for that, we want to have the same
level isolation, typical platforms have for their large virtual machines, and apply that
to every container that is launched.
We also want to reduce the need for core utilities or any dynamic libraries or anything
else inside of the guest, and this helps both reduce the attack surface and also the
burdened of maintenance costs for keeping these up to date.
We also focus on privacy.
Because of that, we want to limit access of what files are needed for that guest virtual
machine and do it on a per container basis instead.
So only the container requesting a specific directory at that point should have access
to those contents.
And we want to hit these goals, we'll still provide an efficient and perform an experience
for end users.
So for security or goal, we'll provide the same level isolation, used by the large virtual
machine, and apply that to each and every single container that is started.
Containerization does this by running each container in its own lightweight virtual machine,
we'll still provide in sub-second-start times.
This also provides the benefit that each container has its own dedicated IP address.
So if you've used different solutions, usually we end up having to do is do different port forwarding
magic in order to be able to access the different service.
So in this now, you can just access the direct IP of that container workload instead.
When sharing directories and files, only the container requesting that directory has access
to those contents, and resources like CPU and memory, if there's no containers running, no
resources will be allocated.
Now I've said lightweight virtual machine several times, so let me dig into what I mean by
that.
To be lightweight, we first focus on the actual machine of self, specifically what devices
associated with it that we need and what a re-verchalizing.
Using Apple's virtualization framework, we are using the minimal set of devices that we
need for the user experience and nothing else, so being in a pair of virtualized devices
not chipset or machines or anything else, just a virtual socket, virtual block device, network
device, and shader file system.
This is more aligned with a micro-VM type of model.
Also what we're doing is making sure that that machine itself is sized appropriately.
So we have a reasonable default, but then again, we're sizing the virtual machine
based on the requests of the actual workload itself, same exact thing for CPU.
So it's lightweight machine, the next is what do we do and turn that machine on.
In our case, we direct boot against Linux kernel.
There's no need for any kind of firmware, bootloader, or anything else.
This saves on boot time as well as it reduces our bill of materials of what we need.
Now for this direct booted kernel, we also want to make sure it's a minimal configuration.
We chose NARM 64 configuration that, again, addresses our needs, but minimizes otherwise
what we would have inside of there.
Reduce the tax surface, reduce the footprint of that actual kernel, as well as helps provide
a perform boot time.
Getting from reset to user space is very fast.
So speaking of user space, I want to talk about what the user space looks like inside of
our guest.
So historically, when using a large virtual machine, you have different dynamic libraries, utilities,
and everything else inside of it.
You're booting a full system.
The file system?
Yeah, I said that.
Okay, sorry.
If it's security, we want to reduce the attack surface.
So we don't have any core utilities.
There's no dynamic libraries.
There's no lib-c implementation or anything else.
We created our own init process that is purpose built just for running containers in this
constrained environment.
And we call it VMInitD, which is written in Swift, it's statically built because there's
no libraries.
And it's designed to manage a full cycle of processes associated with running the container
inside that guest environment.
Now running as initial process comes with a bunch of responsibilities before, as well as
during execution of containers.
So it's responsible for doing things like the actual interface IP is set up, mounting the
file system, such as any volumes or the actual RITFS itself.
It is responsible for launch and supervision of all the processes inside of it, namely
the container entry point.
And it has a year PCAPI that we have for it in order for us to be able to derive that from
the host side.
Looking at our three focus areas for security, privacy and performance with a minimal kernel,
we kind of hit both security as well as performance, I would say, for direct booting,
quicker boot as well.
Using that static purpose built init process helps, again, have a much faster boot time
for us as well as a reduced attack surface inside of the guest.
And each workload being inside its own virtual machine kind of helps across the board
I would say for all three of these.
So let's talk about extending then.
So I don't have time to really go into much demos and everything else, but it's a command
tool to container run container.
Any image you want, as you would expect.
One of the things we wanted to look at, make sure there, yeah.
So we wanted to make it so that people can extend it for their own use cases and actually
for us to easily extend it for our own internal use cases as well.
So what we did was we created a plug-in framework for it and this kind of behaves a little
bit like you would have forget subcommands.
So with this, it allows us to have other developers who want to contribute, but don't
want to make changes to the core code.
Maybe they're specific for the use case or custom workflow or anything else.
They can go ahead and easily just extend what we have functionality wise.
We can just add simple CLI subcommands or these could be different services.
XPC services, the maybe are tied to the lifecycle of the whole thing running or just a container
running or anything else.
So we actually use plugins today for a bunch of the components that we need in order
to be able to run the baseline tool itself.
And then later on the talk, I'll talk about a couple of plugins that we're kind of thinking
about for open.
Creating this easy, it's auto-discovery, you just drop it into no location with a little
bit of config made it data.
And after that, it's picked up automatically the next time you're on the CLI command, you'll
see it.
So that's extending container.
But containerization itself, if I were to talk to all the developers in the team, I would
say that the command link was fine, it has high utility for people, but really the most
proud of containerization itself and all the different APIs that are provided inside of
it.
So with that in mind, I want to talk about how you can easily build a client like ours on
top of containerization.
So as I mentioned before, containerization is where the core logic is for working with
Linux containers on macOS.
To this time, we created many useful modules, which could be useful in their own right.
You don't have to be making containers to want to use some of these or take them for anything
else.
We have containerization OS, interacting with low-level OS components like terminal and
different process management.
We have a minimal net link library for being able to do interface management inside of
a Linux guest with OCI so we can have different runtime primitives as well as different container
image types and everything.
Then we also create an EXT-401, which is I think pretty interesting, it makes it so that
way we can create and manipulate EXT-405 systems from macOS so you don't have to boot
Linux kernel and do all this, you can actually do it directly, then have that raw block
device and boot with it.
Throughout all of this, we made sure to make heavy use of protocols, which in Swift really
means that you can go ahead and replace this with a different implementation and make
it plugable and everything else.
In addition to protocols, we really did focus on the APIs themselves and we're not one
dot zero so they still move around a little bit but really looking at it and to make
sure that we have different layers for this.
So at the highest level, it makes it really easy and I'll show a quick example to just
be able to run a Linux environment on top of macOS.
Kind of in the mid-level, you have different virtual machine management type of APIs where
process management APIs in there and then you can go lower level where we get into more
file system and Linux systems management type of layers and they're kind of build on top
of each other so you can go as deep as you want depending on your application.
At this point, I'm going to risk trying to change the terminal and show that we can make
and macOS application use this Linux in very little code.
We did it.
Okay.
I hope that the people in the back can see it okay.
This is 38 lines in the file of which maybe we have about six function calls.
If I look at it, one of the top level types of we have is container manager where this
is taking a kernel and then knit FS reference.
That's that minimal kernel that every container is going to use for each individual copy
of it then.
Then knit FS is what we do for that VMNID process that I talked about is static built.
With that, we can now go ahead and create containers using that factory.
With the container create, what we need is the reference to the actual OCI image.
Again, standard OCI image that we're using building and consuming and then just whatever
a unique name to it, you can figure it for whatever resources you want, set terminal because
I'll just do it a little echo because we're here, I can type that and then this is just
setting up all the metadata after we execute that.
It's pulling down and unpacking and making a block device that we can use inside the gas
for that root of S which is the container image.
You create, we do start, this is at this point, boot of the VM, starts the process, and
then we wait.
It was a shell or an interactive or anything else, it would keep going in our case that
weight should come back pretty quick once the echo completes and then we'll stop and
on the defer will clean up.
So that was about six function calls and at that point, you can have a custom macOS application,
yes it's using a container image but you could use, that's just a payload.
You can run a Linux environment and about 10 lines and have that embedded in Swift on top
of your Mac.
So it built and then if we just run it, it'll just say hello world eventually, yes,
okay, that took a while, I didn't like that, I'll taste it on.
Okay.
So let me tell you about Swift a little bit, background of the team is a lot of folks
who contribute to container D, run C authors, different tools like this, so like very container
run time focused, very C focused and very, very go focused.
This is our first project using Swift, so I'm just going to give a few different data points
of what the experience was like.
First, C interoperability, you know, with terminal and all the different containerization
OS, all these different modules use a lot of ciscalls and things like this so we often
do need to call in the C. Compared to using Go, it was a lotty, it was very easy, the interoperability,
really, it couldn't be any easier.
The second area was around enums and tech unions, this helped, it really changed how
we're designing area APIs and containerization, we found them really quite nice and useful.
Memory safety, again, is pretty important given our focus on security, you know, using
optionals, designed to prevent psych faults, it was, they're pretty nice to use.
In addition, you know, it's kind of table stakes, we need a static SDK available for Linux,
and it, you know, meets our needs so that we can have that minimal guest image, and we're
often calling into different macOS frameworks and everything else, so being able to call
Swift into these frameworks was a pretty natural and easy thing to do.
Now, let's talk about the open source community aspects, tell you some of the goals we have
and some of the work that we're going to be doing going forward.
We open source the last year and we've been really excited, it's been a pretty positive response
so far. Now that it's in the open, our first party is encouraging contributions in trying
to develop in the open with a community. So we welcome all feedback, if it's issues, PRs, questions,
ideas, anything, it really, this brings us a ton of joy. So we'd love that. We're also looking
forward to seeing how different projects can build on top of containerization, since it's
standalone framework, again, developers can build directly on top of it and use the APIs
to build their own custom solutions. And finally, we want to expand the container ecosystem
through that plug-in architecture. So transitioning to kind of more of a roadmap type of focus,
we have several features and integrations which we're currently working through and thinking about.
One of them is it's trying to create a container plug-in that will allow for a more long
running Linux environments on top of macOS. So like, less the femoral compared to containers.
I would say that a lot of this is motivated like, hey, we like how WSL does this and kind of having
that type of UX where we start to blur the Unix environments a little bit back and forth.
So that's an interesting area. Another one that we're looking at is to be able to have a
femoral Kubernetes environment locally. So this is effectively, how do we work with things like
or mini-cube or any of these? We're doing some work in order to be able to see if we can
be like a provider with kind and upstream. They were having like, we probably won't be able to do that.
Container is pretty different from a lot of the other macOS based solutions that are kind or
supporting because we do have pretty nice resource management of a one-to-one mapping that
the container to a VM. So since our focus is on the other ones, essentially the feedback from
Ben was like, hey, you guys should just write your own it's easy. So we're going to look at that
for a femoral environment pretty shortly here. That'll probably be a container plug-in.
In addition to this, we're always looking to improve our performance. So we currently use build kit
and build kits great build kits in the facto. Every machine is built kit using it inside of a
guest across from Swift across the machine boundary for GRPC. It's the performance that's
suboptimal on our side and we want it to be better. The problem is fixing it, it's complex,
as far as being able to have the equivalent functionality. So this is kind of a much larger thing
that we've been talking about in the community and trying to figure out how we'd push that forward.
On top of that, we want more ecosystem integration. So this is more like if we do Kubernetes
of how can we build on top of containerization to have something that would fit underneath that
container runtime interface type of API. And in the grace of you could run a more typical
Kubernetes stack on this. One of the most common things is, hey, this is cool. Can I use compose
and we say not yet. And dev containers are kind of similar to these two. The API that they kind of
have is more like the Docker CLI. So it's challenging for us to decide how much should we make
it a drop in replacement versus being opinionated for a very easy entry point. So this is still
something we're looking at. They're different community PRs for adding compose, which is super fun,
but not all the way there yet. So we're still kind of working through that. Next step, final
wrapping up is, if you're on a Mac, you can try it out. Again, we would really love feedback
for curious developers who want to use their own use case. Check out the APIs and containerization,
build your own solution or extend it with plugins. If the intersection of virtualization containers
are interesting, are growing as well on the team. But thank you. I appreciate it. And we have stickers
in some things here as well.