Hello everyone, my name is Wilson and I work for Feature Fusion doing mostly continuous stuff
and Linux run on stuff around that together with Stefan Grabber.
Hello, so Stefan Grabber, I'm the city of Feature Fusion.
I do some other stuff but I'm indeed also the particular for all Linux containers, one of the
main trainers of Incas alongside Christian, Max, but for the folks and we're going to be talking
about resource management in containers.
So first of all, let's start from quickly looking at the ways to limit resources and Linux.
We have plenty of ways, first of all, people may configure the Linux kernel through the
using the kernel parameters, for example, let's CPUs or limit the number of CPUs that can
be used.
Obviously, CSFS can be used to configure stuff, some CSCOs that get a set affinity or obviously
Ulimits and Cgroups for containers, probably Cgroups is the main one and sometimes because
of these kernel at the same time doesn't provide any clear way to tell the user space
about, for example, how many CPUs I can use right now.
The user space application developers need to understand a lot about how stuff works inside
to fight an answer to questions like that.
And in system containers, we take care of that and we try to, in LexiPers, we try to make
any application to run inside the container and feel like it's the full host kind of access,
right?
And we obviously need to emulate a virtualize a lot of things, we need to make sure that
the user issues the common like uptime or free or H-top user can see the right stuff appears
in the numbers and it can be really, really complex, problem, sometimes even unsolvable
because we have no, not enough kernel API, in this case, we need to develop that APIs
for us and here are the list of things, it's not a full-of-course that we need to virtualize
in Lexi to make the system containers to work properly.
And how it works really quickly, so basically, application runs inside the container,
of course, it's a philosophical term because nobody knows what the container is, right?
But in case of Lexi, it's well defined, so we have all the name, we use all the name spaces,
we have C groups, and let's say process inside the container issues, wants to read some
file from ProCAP time.
So what we have right now, we usually over mount ProCAP time, file with file from specially
crafted, syntactical file system called LexiFS, which is a fused-based file system, so it
managed by the fused-demon, the tool that works in user space and gets a request from the
kernel, makes some actions on that request depending on which file it is, right, and
provides the answer to the request, right?
So basically, the key point here is that we basically over mounting every single file we
need or dietary that we need to virtualize.
And internally, on the fused-lexiFS fused-demon side, we need to solve, usually we need to follow
this pattern, we need to determine which container process belongs to, and it can be
challenging, because obviously, again, there is no such thing as container really in the
kernel, on the kernel standpoint, right?
So usually for us, it means that we need to determine the initial process of the container,
so we rely on, we assume that the name space is used, of course, so we use some trickery
to figure out who is the initial processing that container, so which means pin name space
in this case, and so we usually using this, this is using a unique circuit, and circuit
control message called SDM credentials to do that, so basically, we're creating a unique
circuit, we're creating, we're working a process, we're entering a pin name space of that
process, that issued the fused request, then we're sending, especially, crafted SCM message
with the PAD number one, and we kind of are using, we know that kernel, what kernel does,
is kernel does the proper PAD translation, so if you have a unique circuit, and one side
of that, if you have a unique circuit player, and one side lives in one pin name space,
and another side lives in another pin name space, then if you send the PAD through a circuit
control message, then we know that the kernel will translate PAD for us, so basically, we kind
of are using this thing to figure out which PAD the process has on the host, if that
process has a PAD number one inside the pin name space, so this is how we solve the first
thing, and second, and then we have the another very much more complex thing, because
it's not always, again, it's not always even possible to figure out which effectively
means process has, so we need to calculate that, sometimes it can be trivial, I can case
of uptime, obviously, what we do, which has to go to the PAD number one in that continuum
and figure out how for how long this process runs, and this is at that time of the
continuum, but for some cases, like memory, it can be really not trivial, because depending
on the C group version you rely on, there can be no direct mapping, because in C groups
we have software, it's hard-limits, other things, and it's really complex thing, and then
basically, once we figure out all the stuff, we can build the proper output, because all
the files we emulate, they are textual format files, and then we just give the reply back
to the kernel, which the user, I mean, the user request reply, and the kernel can answer
the user space and block the process and give the answer, which, but what I covered
is really basics, we have seen some difficulties, for a long time, for example, we
just start views, because it's also complex thing, because we have no device in space,
and it's hard to understand, it's hard to formally define which block devices I used
by container and which are not, because there is no such thing in the kernel, right?
And sometimes people struggle to see what they expect with regards to swap utilization
inside the system container, sometimes people complain about H-top, not showing the right
values, expected values, when people doing stress load, because for example, if you do a
stress load on the CPU, on the host, then inside the container H-top, you'll show you that
all the CPUs are busy, while inside the container you have nice thing, at the same time,
but it's not something we can really fix just in user space, we obviously need some kernel
support for that, and the problem is that previously in C-group 1, we had a special controller
called CPU Act, which allowed to account the CPU time spent, but in C-group 2 unfortunately,
we don't have any analog for that, probably because of performance reasons, because it's
kind of costly thing, another challenge is that not everything goes from pro-cafers, right?
So we have some C-scores that can allow user space to retrieve some information, like the
easiest example, probably season for C-score, it allows to also retrieve a bunch of information
like load, average, uptime, payment force, up and for and other things, and we cannot
really hijack that using LX CFS, and also there are some other sources of information,
like C-SFS, especially with regards to this information about CPU topology and a number
of CPUs and CPUs states, like in case if they're a hot-hot-logobal, right?
And I've seen just recently, maybe like a year ago, we have received a real world
use case is not a synthetic L1 when inside the VM I don't remember maybe it was open stack VM
but I'm not sure but anyways like it was a VM and the person was argue was not happy about seeing
really weird results in the monitoring tool because the monitoring tool I think it was something
is something that was not exported probably it was showing really weird values in terms of how many
CPUs this container have because container had just one CPU but this thing was showing
much more and after analyzing I figured out that actually the problem was that the VM the inside
which this container is deployed was having like 12 CPU slots while only 3 CPUs were present
and the interesting thing was that the kernel was handy this really in the final way in
on CSFS the number of CPUs user might see on CSFS was less the number of CPUs in a
C group you won CPU Act file so in CPU Act file there was like 12 CPUs while in the CSFS there was only
3 CPUs and I was like okay that's weird let's go to the kernel code and it was kind of obvious
because in the kernel code one thing which I mean CPU Act was using for each present CPU macro
is the kernel iterator over the CPUs while the CSFS was using CPU Act was using for each possible
CPU which larger set so again like usually if you just I don't know running stuff on your laptop
probably you have just everything equal you you won't probably want to face that but on on on in some
cases we see especially on the series we see you should like that that's it I will just give the
microphone to Stefan
hello it's me again all right so I just wanted to show the current state of things and then we can talk
a little bit about the future depending on how much time have so let me just get a quick terminal up
somewhere okay that's here
all right so if we're just looking at the basics and we get ourselves the the then
13 container and we give it two CPUs to the exam okay and then we want to go to the same thing
with Alpine okay we go in the in the WN1
3 and 4 are both happy they're both using plot files let's see if it's doing a job everyone's happy
let's go look at the other one oops there's no bash in Alpine um free yeah 32 gigs of RAM
and product yeah that one looks good um which one is wrong not a CPU some of the like effectively
we get some inconsistent information there and the reason I is that some of the tools in Alpine
are now using system for instead of using XFS well instead of using plot thankfully in Gus
how to work around for that so we can do security this goes intercept system for sure
let me start this thing
and we're back to normal but now it means we've got two completely different systems that need to
there with that we've got the system going toception code inside of Incus that needs to
catch every single code it's info and then go and get the data and return the right thing
as well as having LexiFS also do all of the podcast masking for all of that information
that's gonna duplicate data we can share some amount of code here and there but that's kind of
knowing it's also basically losing battle we know that you know there's more and more of
those kind of things that are gonna keep popping up and there is no really appetite in the Linux
internet external for having the probe vertices info view respect C groups or respect other
kind of limits that's just not something that anyone wants to have an interest for and so we need to
look at what can we do about this stuff and it's basically what I'll we're saying a continuous plan
and it's because not because it's gonna take us 10 years to implement it's gonna take 10 years to
have a big amount of adoption of this thing so basically what we need is to get to the point where
user space no longer passes product to figure out their resource limits or no longer really tries
to use this info to get their resource limits instead the the rough idea is to try and have a
probably a rest rate with then a CIPI library as well as a volume small service
that will provide the same information so if you're written in rust we can use the
create and you can use that to get a very reliable answer as to what your resource limits actually
look like as well as some other non-ambient system info whether it's like LSMs or that can
and if you're not interested but you're in some dynamically linked language running on the system
you can link to the library and get the same information either in both of those cases you can
either query about yourself or any other process because you're typically doing that query in
process so if there's any security checks or whatever they will care probably and then there's
option number three which is for what happens if you're dealing with cross compiles that
think binaries things that go binaries some drama stuff that kind of thing where it's not really practical
to depend on a shared library that gets pretty quickly so the idea is to effectively have
a circuit activated process running on the host system which you can quickly open a unique circuit
too and then make the same query over over that and then we get the SEM thread on the other side
and can figure out who the color process is where they are and then return the right response
in this scenario because you're effectively doing your queries through a third-party process
you wouldn't only be able to do it against yourself like you only get your own resource information
you can't use that to try and trick the system into getting information about processes that
you're not otherwise access allowed to access well that's kind of the rough idea
we've been talking about that before it's the second time we talk about it at first
them that's time it was called leave my campaign or something or not from Netflix which
didn't really go anywhere but roughly same concept we might try and do a bit more work on that
these days since future fusion we've recently got Alex on board and we've got a bit of time to do
some low-level work we might try and get it done Lex CFS would be an obvious thing to post over
to that as a way to make sure that all of those kind of APIs are all correctly supported and then
we probably start pushing for like specific run times probably like go in Java for like hey
instead of doing your own thing which is always wrong basically right now and like every time
you run Java in the container it's getting just slightly wrong data instead of doing
maps use this library or use this demon effectively if you get that data if it's available
and see if things get better and if it is then hopefully you can slowly get more and more
at least more more of the programming languages and run times to handle that and that will
leave us with all of the other tools like you know top h.ps whatever all of those guys will still
need to be bought it over which is why it's going to take a long long time like we need to find
what's the best way to get a bit of adoption and kind of try and get the ball rolling and hopefully
it's no balls and more and more folks not using this nobody likes passing truck it's really tricky
it's often wrong so I think people would actually welcome a clean library to this kind of stuff
another point and thankfully that is not in the room but otherwise you would have asked me
it like hey why what about using you know system discourse using the system the API to figure
that stuff out and the answer to that is like yeah that works great if you're on a system with
system the running system the things but if you are like a static binary running inside of an
OCI container will actually be on the binary on disk and you want to know about your limits
you're not going to have a system to talk to like you don't have it in its system you're just
a single binary running in a in a container so in that scenario being able to talk to that
unique sockets that would just be re-explosed by darker re-explosed by anything else will get
to the information you want or if you've written in a language that makes that easy you could do
the query yourself by using either the rest rate or the library so that's kind of the idea
if we're getting pretty close to out of time we can maybe try one question maybe
the question yeah if doesn't it
I do agree to give a read on any plans for dealing with LXF's crashes I'm going to come a lot closer
sorry hey it's on have you read them any plans about dealing with LXF's crashes
oh LXF's crashes yeah the main so yeah the occasionally LXF's crashes we are looking at the
crashes these days the main the main difference is that LXF used to work for canonical and have
maybe five minutes a month to look at those is now working for me and it's going to get a bit more
time to look at those but like and we've had some ideas to handle LXF's crashes better with things
like effectively being able to handle crashes and re-attach the fuel smart system I think you did
some work on that right effectively you're supporting like a some kind of resume of fused so that
you can if there's a crash you can still keep the mounts as they are and through start LXF's re-attach
it's all of the mounts and you don't need to restart all the containers at least so we've done
some work towards that which we would make things all there we're done