WEBVTT 00:00.000 --> 00:10.000 Next up, we've got Tino's going to be talking about imutable all the way down. 00:10.000 --> 00:15.000 They're going to go away. 00:15.000 --> 00:19.000 Hey, Fox. 00:19.000 --> 00:24.000 Thanks for joining the container, there are room for them. 00:25.000 --> 00:30.000 So I have a few live demos, and those demos are downloading a lot of stuff. 00:30.000 --> 00:35.000 I tried, and the previous talk back there, and it worked like 50 years per cent. 00:35.000 --> 00:41.000 Well, but you know, if one of the demos gets delayed, I can just go to the next one of the demos are automated. 00:41.000 --> 00:44.000 So you'll be able to download them from a repo. 00:44.000 --> 00:50.000 Also, if any one of you in the audience is like leaching stuff over Wi-Fi, please stop. 00:50.000 --> 00:52.000 It's going to break my demos. 00:52.000 --> 00:57.000 Hi, I'm Tino, I'm a maintainer of flat-cut container Linux. 00:57.000 --> 01:01.000 I'll give up presents, I'm a messer-down, or you can drop me an email. 01:01.000 --> 01:05.000 I usually talk about operating systems. 01:05.000 --> 01:12.000 It kind of a regular in the container, therefore, actually container optimised operating systems that don't get in the way of your workloads. 01:12.000 --> 01:19.000 But today, I'm going to climb one level up, and I'm going to talk about cluster plumbing. 01:19.000 --> 01:27.000 And that is how we kind of envision shipping Kubernetes, when you're a Kubernetes operator, as an immutable image, 01:27.000 --> 01:32.000 that includes basically all of the bits and pieces that you need, including configuration, 01:32.000 --> 01:34.000 basically all of the static stuff. 01:34.000 --> 01:40.000 This allows us to separate Kubernetes from the operating system, which has a number of benefits. 01:40.000 --> 01:44.000 It's easier to use, you have separation of concerns and Fs, slides on that later. 01:44.000 --> 01:51.000 But it also enables very easily straightforward in life and place updates. 01:51.000 --> 01:59.000 Before we start, my last demo will be cluster API, and that holy shit, this is small. 01:59.000 --> 02:07.000 That usually takes a little longer to initialize. 02:07.000 --> 02:15.000 So I'm going to deploy my cluster right now, and that looks good. 02:15.000 --> 02:23.000 So this is using the cluster API Azure provider, and that's because I worked there. 02:23.000 --> 02:25.000 And I get to use it for free. 02:25.000 --> 02:27.000 I'm not trying to get you to use Azure. 02:27.000 --> 02:32.000 This is the request for me, so that's why I'm using an Azure deployment here. 02:32.000 --> 02:35.000 All right, anyways, let's start. 02:35.000 --> 02:40.000 So I said in your shipping, this is a mutable images. 02:40.000 --> 02:44.000 And we're using a format there that's called extension images. 02:44.000 --> 02:47.000 It really has like a full one specification. 02:47.000 --> 02:50.000 You can go there, you can click there, basically read all of that. 02:50.000 --> 02:55.000 There's at least one implementation that's using this specification. 02:55.000 --> 03:02.000 And it basically allows you to wrap application binaries, application supplementary file, files, 03:02.000 --> 03:09.000 and static configuration, and an immutable file system image that only consists of the bits that you want to ship. 03:09.000 --> 03:15.000 This immutable image will be merged into your route operating system file system. 03:15.000 --> 03:19.000 Usually at boot, but it can happen at any point in time that you choose. 03:19.000 --> 03:23.000 There's a nice tie-in with an update mechanism called system pieces update. 03:23.000 --> 03:31.000 That update mechanism also works on single files, and if that file happens to be a system extension, 03:31.000 --> 03:35.000 then you'll basically get these in place updates. 03:35.000 --> 03:41.000 So inside of an extension image, as mentioned, you just spread your binaries. 03:41.000 --> 03:47.000 The way you want to find them in your route file system after merging. 03:47.000 --> 03:51.000 If you want to start services or use configuration files, 03:51.000 --> 03:55.000 that don't need much changing, you include it in a Sussex as well. 03:55.000 --> 03:58.000 Maybe some repascripts to get stuff started or bootstrapped. 03:58.000 --> 04:01.000 And you can include system units there. 04:01.000 --> 04:06.000 So services are being started at the point in time where stuff is being merged. 04:06.000 --> 04:10.000 Just to reiterate on how this looks on your node. 04:10.000 --> 04:16.000 So imagine your ship, Kubernetes, and an image, then it would basically contain this directory tree, 04:16.000 --> 04:21.000 a little shortened here, with the Kubernetes binaries, some startup files. 04:21.000 --> 04:26.000 And so this is going to be merged into whatever your route operating system is. 04:26.000 --> 04:31.000 And it will end up basically looking like, it's kind of an overlay. 04:31.000 --> 04:38.000 So it looks like as if both files, both binaries, always existed in your routeFS. 04:38.000 --> 04:43.000 All right. Now you got your extension image, and how do you even get it on the node? 04:43.000 --> 04:47.000 For those demos, I'm going to use FlatCar, which can be instructed via 04:47.000 --> 04:52.000 Yammer configuration to do provisioning time downloads. 04:52.000 --> 04:55.000 And in this example, in the first example, I'm going to show. 04:55.000 --> 05:01.000 I downloaded from interesting GitHub releases page, a Kubernetes Sussex, 05:01.000 --> 05:05.000 and put it on my node. 05:05.000 --> 05:07.000 And this is what's going to look like. 05:07.000 --> 05:11.000 This is the first demo, and you'll see how well the Wi-Fi works. 05:11.000 --> 05:15.000 All right. So, who? Can you folks read that? 05:15.000 --> 05:18.000 It's much smaller here, sorry. 05:19.000 --> 05:22.000 So this is slightly different, and that's why I'm bringing it up again. 05:22.000 --> 05:29.000 I will drop the extension file here, which is an entirely custom path. 05:29.000 --> 05:35.000 The implementation I'm using on the node system, the Sussex, is not a very of that path. 05:35.000 --> 05:39.000 And the reason I do that is because I want to make it a very manually. 05:39.000 --> 05:43.000 So I can show you around before the whole thing is being merged. 05:43.000 --> 05:50.000 So this is the point where things get exciting, because the download starts now. 05:50.000 --> 05:57.000 All right. So it took like more than a minute when I was sitting there during the previous talk. 05:57.000 --> 06:06.000 I hope this will be faster, but I can bridge the time by checking on my cluster API stuff. 06:07.000 --> 06:12.000 Oh, here, stuff is being provisioned. So cluster API generates this stuff for you automatically. 06:12.000 --> 06:21.000 So you get a fully operational cluster. It's still downloading. 06:21.000 --> 06:25.000 Yeah. Oh, wow. 20 seconds. Okay. That's good. 06:25.000 --> 06:35.000 I'm going to give that. All right. So as I said, this is now actually let me. 06:36.000 --> 06:43.000 Do this. This has been downloaded to a custom path. 06:43.000 --> 06:50.000 It's just a file there. Since this is a file system image, I can just go ahead and mount it. 06:50.000 --> 07:10.000 Okay. So if you look at mount now, you see it's basically kind of like a file system structure. 07:10.000 --> 07:15.000 If you do this recursively, you'll see that this is in the folder on file system. 07:15.000 --> 07:21.000 Literally only contains the stuff I need to run Kubernetes. 07:21.000 --> 07:30.000 And so two things might be interesting in particular. 07:30.000 --> 07:37.000 I can find my short cuts here. The first is the ship unit files. 07:37.000 --> 07:42.000 So, you know, the whole thing starts up as soon as the Sussex becomes operational. 07:42.000 --> 07:52.000 And we ship metadata, extension metadata that tells Sussex what to do. 07:52.000 --> 07:57.000 In this case, it just says that it will run on any. 07:57.000 --> 08:01.000 It will run on any operating system, basically any distribution version. 08:01.000 --> 08:05.000 It doesn't have any dependencies on the actual operating system image. 08:05.000 --> 08:10.000 It's got, it's built for the X86 65 architecture. 08:10.000 --> 08:16.000 And it tells the service manager to please reload all of the service configurations as soon as this is merged. 08:16.000 --> 08:21.000 And we need to do that because we ship system the units and we want these units to activate. 08:21.000 --> 08:34.000 So, if I now, if I now make system the aware of this by sum linking it into a directory, 08:34.000 --> 08:46.000 that system the Sussex is to be off, then it will pop up in the, in the Sussex list. 08:46.000 --> 08:49.000 And you see a number of Sussex here. And the reason for that is, 08:49.000 --> 08:52.000 if I use the Sussex for a lot of stuff, not just for Kubernetes. 08:52.000 --> 08:58.000 And the extensions right here, it's not active yet, it's basically staged, but not activated. 08:58.000 --> 09:13.000 So, if I check for Kubernetes by no reason you are already on on. 09:13.000 --> 09:20.000 Let's check for the Kubernetes, and then the binary is not being found. 09:20.000 --> 09:28.000 And if I ask John a color, but it knows about Kubernetes, it doesn't know anything. 09:28.000 --> 09:36.000 I will now merge the thing. 09:36.000 --> 09:38.000 And that's usually what happens at boot, right? 09:38.000 --> 09:41.000 And as soon as this is merged, multiple things happen. 09:41.000 --> 09:47.000 First of all, we actually have Cubelet in the local file system. 09:47.000 --> 09:55.000 Oh, I forgot to pin, no, the demo effect. 09:55.000 --> 09:59.000 Okay, we have Kubernetes by no reason in the file system. 09:59.000 --> 10:02.000 And John will cut it and knows about it. 10:02.000 --> 10:07.000 And Cubelet, it's just crying into the log right now because it didn't come with the configuration. 10:07.000 --> 10:09.000 So, it can't run any clusters, but it's there. 10:09.000 --> 10:14.000 It's being executed to service starts. 10:23.000 --> 10:31.000 So, learning from the previous waiting period, I'll start my next demo right away. 10:31.000 --> 10:34.000 So, it's ready even when I need it. 10:34.000 --> 10:41.000 So, this was basically a single-note demo that's image-based way of provisioning stuff. 10:41.000 --> 10:46.000 And the only relation to Kubernetes is, it's being used, it's used to Kubernetes. 10:46.000 --> 10:48.000 So, there's no cluster yet. 10:48.000 --> 10:53.000 Before we go to that, we'll look into in-place updates. 10:53.000 --> 10:57.000 And that works on the file system image as well. 10:57.000 --> 11:00.000 So, as I mentioned, there's a complementary service called. 11:00.000 --> 11:04.000 So, update can be configured to update any given image. 11:04.000 --> 11:06.000 It is semantic versioning aware. 11:06.000 --> 11:11.000 And all it needs to consume images from is an HDPS endpoint. 11:11.000 --> 11:16.000 The kind of an index file in it, the sharsams in it, so it knows what files are available. 11:16.000 --> 11:19.000 It knows what files are in the local file system. 11:19.000 --> 11:23.000 It checks regularly that HDPS endpoint for new versions, 11:23.000 --> 11:25.000 matching the semantic versioning pattern. 11:25.000 --> 11:27.000 And it also is summing over. 11:27.000 --> 11:33.000 So, it basically can update the summing that I used to make my personal Sussex 11:33.000 --> 11:37.000 or known to systemly Sussex and will update that. 11:37.000 --> 11:40.000 So, basically both put together a load, some kind of get-ups. 11:40.000 --> 11:43.000 And that's what we're doing with the Sussex Bakery, 11:43.000 --> 11:48.000 which in which you can regularly or on demand build or rebuild Sussex, 11:48.000 --> 11:54.000 publish them as GitHub releases and then instruct your note configuration 11:54.000 --> 11:58.000 to just go there, check for updates and download and stage updates. 11:58.000 --> 12:02.000 As soon as new updates become available. 12:02.000 --> 12:07.000 So, this is a single note, this is update demo and we have booted. 12:07.000 --> 12:10.000 So, I learned from a previous mistakes. 12:10.000 --> 12:15.000 This is slightly further automated in that. 12:15.000 --> 12:21.000 In this configuration, we do a little more stuff. 12:22.000 --> 12:24.000 We create the sum link. 12:24.000 --> 12:28.000 So, this is basically merged already when the note boot. 12:28.000 --> 12:32.000 And we add a configuration for Sussex update. 12:32.000 --> 12:35.000 So, that Sussex update is aware of that file, 12:35.000 --> 12:38.000 aware of the endpoint that consumes new versions from. 12:38.000 --> 12:43.000 And basically, when it is all of these updates for us. 12:43.000 --> 12:46.000 The one thing we don't do is we don't start Sussex update, 12:46.000 --> 12:49.000 because that's my demo. 12:49.000 --> 12:56.000 So, oops, yeah. 12:56.000 --> 13:06.000 So, we have... 13:06.000 --> 13:08.000 We have Kubernetes installed. 13:08.000 --> 13:10.000 We have Q-blade, again, crying in its log, 13:10.000 --> 13:12.000 because it came without the config. 13:12.000 --> 13:15.000 This is the access merged. 13:20.000 --> 13:22.000 And there. 13:22.000 --> 13:25.000 And we have basically provisioned the Sussex and activated the Sussex. 13:25.000 --> 13:28.000 And we know what to update it. 13:32.000 --> 13:36.000 Here's our custom Sussex store. 13:36.000 --> 13:38.000 And that has a single file in it. 13:38.000 --> 13:41.000 And that was the file, the whole system was provisioned it. 13:42.000 --> 13:49.000 And the sum link in it is the extensions, 13:49.000 --> 13:52.000 where Sussex looks for its extensions. 13:52.000 --> 13:56.000 Points to the 131.3 version. 13:56.000 --> 14:02.000 Now, I'm going to start... 14:02.000 --> 14:12.000 Sussex manually. 14:12.000 --> 14:19.000 Usually, that will be done that went fast. 14:19.000 --> 14:23.000 That's usually been done by a timer unit, 14:23.000 --> 14:27.000 so you would set up a timer unit appropriately to start this. 14:33.000 --> 14:36.000 I, of course, need to start system usage update, 14:36.000 --> 14:38.000 and not system usage. 14:38.000 --> 14:40.000 That's running on the time. 14:40.000 --> 14:41.000 Sorry for that. 14:41.000 --> 14:43.000 Okay, so we start system usage update. 14:43.000 --> 14:47.000 That'll take a bit longer because it doesn't go into the background 14:47.000 --> 14:48.000 when downloading new updates. 14:48.000 --> 14:51.000 I made sure that it has an update to find, 14:51.000 --> 14:54.000 because I want to demo something, right? 14:54.000 --> 14:57.000 So, let's happening right now in the background. 14:57.000 --> 14:58.000 It just goes there. 14:58.000 --> 15:02.000 This is the same kind of waiting period that we have 15:02.000 --> 15:04.000 when we started the instance. 15:04.000 --> 15:07.000 It's usually faster, it's a lot faster in the cloud, 15:07.000 --> 15:10.000 because the Kubernetes Sussex is below 100 megabytes. 15:10.000 --> 15:14.000 So, this usually doesn't need like 30 seconds, 15:14.000 --> 15:16.000 it's just because of the Wi-Fi here. 15:16.000 --> 15:21.000 That takes a little longer and it'll install that Sussex. 15:21.000 --> 15:24.000 Let me see what my, ah, nice, 15:24.000 --> 15:26.000 so at least that one has been deployed. 15:26.000 --> 15:28.000 We have, okay. 15:28.000 --> 15:30.000 System usage update has finished. 15:30.000 --> 15:32.000 We can even check. 15:39.000 --> 15:42.000 We can even check the log output and we'll see. 15:42.000 --> 15:44.000 It went there, it found a new version, 15:44.000 --> 15:46.000 which is 134. 15:46.000 --> 15:49.000 And it put it there right next to 15:49.000 --> 15:52.000 where the other stuff resides. 15:52.000 --> 15:54.000 There's a new file there now. 15:54.000 --> 15:58.000 And if you check the, the sum link here, 15:58.000 --> 16:01.000 it updated the sum link because we told it to do so. 16:01.000 --> 16:03.000 Now, when I run, 16:08.000 --> 16:10.000 cube let's test that version. 16:10.000 --> 16:11.000 What version did I see? 16:11.000 --> 16:12.000 So, of course, the old one, 16:12.000 --> 16:14.000 because we, we staged the update, 16:14.000 --> 16:15.000 but we didn't activate it yet. 16:15.000 --> 16:17.000 There's a number of ways you can do this 16:17.000 --> 16:20.000 because Kubernetes is a complex workload 16:20.000 --> 16:22.000 to think the way I will use 16:23.000 --> 16:25.000 is a node reboot. 16:25.000 --> 16:27.000 And we're getting to that later. 16:27.000 --> 16:29.000 That's nothing to be scared of. 16:29.000 --> 16:32.000 You would usually want to act a vacuum agent out 16:32.000 --> 16:35.000 and make sure like it's properly 16:35.000 --> 16:37.000 according to before you do that. 16:37.000 --> 16:38.000 And that's my last demo actually. 16:38.000 --> 16:39.000 So. 16:44.000 --> 16:45.000 That's the new version. 16:45.000 --> 16:47.000 We updated in place. 16:53.000 --> 16:56.000 Now, how does that? 16:56.000 --> 17:00.000 First of all, let's wrap up on the demo on extension images. 17:00.000 --> 17:04.000 So, this is basically a self-contained Kubernetes 17:04.000 --> 17:06.000 that you can just grab and throw on 17:06.000 --> 17:08.000 basically any operating system. 17:08.000 --> 17:09.000 It'll come up. 17:09.000 --> 17:13.000 You'll need to only do minor customizations 17:13.000 --> 17:16.000 and specifications to make it work for your use case. 17:16.000 --> 17:19.000 It is, it's relatively easy to generate. 17:19.000 --> 17:22.000 If you want to make squashFS on a sub directory 17:22.000 --> 17:25.000 and you get the image, it is easy to compose 17:25.000 --> 17:28.000 into running systems or into new provisionings. 17:28.000 --> 17:31.000 And it's easy to manage because all of those tools 17:31.000 --> 17:33.000 are integrated in system D. 17:33.000 --> 17:37.000 And it comes with a complementary update mechanism 17:37.000 --> 17:40.000 and they're just relatively likely already supported. 17:40.000 --> 17:44.000 Because it has been part of system D for a few years now. 17:44.000 --> 17:45.000 Kubernetes deployments. 17:45.000 --> 17:48.000 I need to rush because I only have 10 minutes left. 17:48.000 --> 17:53.000 We are, we'll be provisioning like a full blown Kubernetes. 17:53.000 --> 17:55.000 That is configured. 17:55.000 --> 17:57.000 And that actually works as a cluster locally 17:57.000 --> 18:00.000 and curing more images using this method. 18:00.000 --> 18:02.000 So, basically, take a distro image. 18:02.000 --> 18:04.000 In my case, that's a curing image locally 18:04.000 --> 18:06.000 that I use with this snapshot option. 18:06.000 --> 18:09.000 So, changes will not be written back to the image. 18:09.000 --> 18:11.000 Create an instance from it. 18:11.000 --> 18:14.000 Grab the Sussex from a separate source. 18:14.000 --> 18:19.000 Compose it at provisioning time 18:19.000 --> 18:23.000 and have the Kubernetes cluster ready. 18:23.000 --> 18:24.000 Okay. 18:24.000 --> 18:27.000 So, that's that. 18:27.000 --> 18:29.000 This is again downloading. 18:29.000 --> 18:33.000 Now, the benefit here is that you have separation of concerns. 18:33.000 --> 18:37.000 You can have operating system folks care about data 18:37.000 --> 18:39.000 to keep it up to date. 18:39.000 --> 18:42.000 You can have Kubernetes folks all yourself depending 18:42.000 --> 18:44.000 on the scale of your operation. 18:44.000 --> 18:47.000 Focus on the Kubernetes bits entirely. 18:47.000 --> 18:53.000 And, nonetheless, some kind of loose coupling between two components. 18:53.000 --> 18:56.000 It even allows you switching distro's. 18:56.000 --> 18:59.000 Basically, upgrading distro versions or switching to a whole new distro 18:59.000 --> 19:02.000 without much concern about your Kubernetes. 19:02.000 --> 19:06.000 Because you manage both independently and Kubernetes itself. 19:06.000 --> 19:08.000 I mean, it's just a bunch of go binaries. 19:08.000 --> 19:10.000 It's pretty independent from the distro 19:10.000 --> 19:12.000 that you're deploying it to. 19:12.000 --> 19:16.000 You can, you can in place update both separately. 19:16.000 --> 19:21.000 Like, leave the operating system updates with the 19:21.000 --> 19:22.000 operating system folks. 19:22.000 --> 19:26.000 Have your own Kubernetes updates and the indicators you need. 19:26.000 --> 19:32.000 And, don't worry about the interdependence of both parts, right? 19:32.000 --> 19:36.000 And, for larger deployments, if you do it the traditional way, 19:36.000 --> 19:42.000 is building images from doctors to images with Kubernetes included. 19:42.000 --> 19:46.000 This approach will actually allow you to cut down on the version matrix a lot. 19:46.000 --> 19:48.000 So, what do I mean, my luck? 19:48.000 --> 19:53.000 Well, imagine you maintaining a Kubernetes image for your classes to deploy 19:53.000 --> 19:59.000 with based on Ubuntu 22 Kubernetes 129 on AWS. 19:59.000 --> 20:02.000 Maybe another one on an Azure, maybe another one on GCP. 20:02.000 --> 20:05.000 But then you also want to ship Kubernetes 130. 20:05.000 --> 20:08.000 So, you have six images and what's about, what about 31? 20:08.000 --> 20:11.000 That's nine images, but what about you 124? 20:11.000 --> 20:14.000 That's 18 images, but maybe you want to ship our nine. 20:14.000 --> 20:19.000 And so, this is what I mean by, by kind of exploding version matrix, right? 20:19.000 --> 20:27.000 You don't have that with the Sussex approach, because you combine these at provisioning time. 20:27.000 --> 20:32.000 And you basically can take control of the parts that you want to take control of, 20:32.000 --> 20:37.000 which is Kubernetes and leave this to maintenance to the lower level. 20:37.000 --> 20:41.000 There's this inspection that exists, and you can leverage it with that. 20:41.000 --> 20:44.000 Okay. 20:44.000 --> 20:50.000 Nice. So, this put it. 20:50.000 --> 21:00.000 We can go and fetch the, no, no coffee, not right now. 21:00.000 --> 21:03.000 I'm too excited already. 21:03.000 --> 21:09.000 So, we can go and fetch the, the cube config here. 21:09.000 --> 21:14.000 And I think I'm going to shorten this just a little bit because, 21:14.000 --> 21:17.000 I'm running out of time. 21:17.000 --> 21:25.000 So, the full demo, which is published in the flatcard demos repository in the flatcard. 21:25.000 --> 21:33.000 I can get up actually includes a full blown local Kubernetes cluster. 21:33.000 --> 21:38.000 And I want to like take a shortcut here and only work with the control plane. 21:38.000 --> 21:42.000 No, you see the other windows, those would have been the, the worker notes, 21:42.000 --> 21:47.000 but we're going to go with the control plane in this example. 21:47.000 --> 22:02.000 And we'll quickly provision a, see an eye because right now our notes are already, 22:02.000 --> 22:12.000 and we'll only get those notes, this note ready. 22:13.000 --> 22:18.000 Follow four. 22:18.000 --> 22:21.000 This is too much demo effect for me. 22:21.000 --> 22:27.000 Okay. I'm interesting. 22:43.000 --> 22:46.000 Five minutes left. 22:46.000 --> 22:55.000 The idea here is that you, 22:55.000 --> 22:58.000 I can do this on the note actually. 22:58.000 --> 23:03.000 So, 23:03.000 --> 23:13.000 you have a pretty, you have a pretty happy cube that here. 23:13.000 --> 23:22.000 And cube ADM basically trying to do its thing and it's waiting for the, 23:22.000 --> 23:29.000 the C and I to be deployed. 23:29.000 --> 23:35.000 On this version and the idea was here that I basically show you that from the Kubernetes side of things, 23:35.000 --> 23:43.000 but it's kind of stuck from the, from the slow Wi-Fi and me running out of time to 23:43.000 --> 23:48.000 update that to the newer version, 23:48.000 --> 23:59.000 131.4, which I only can show you on the command line now. 23:59.000 --> 24:01.000 Okay. 24:01.000 --> 24:06.000 So, this basically pulls in new Kubernetes, in the instance, the publishing, 24:06.000 --> 24:08.000 can happen via GitHub, which is what we do. 24:08.000 --> 24:12.000 We have a GitHub action that basically rebuilds periodically, 24:12.000 --> 24:16.000 and then publishes new versions, and then you can activate in place, 24:16.000 --> 24:20.000 and that would have been the demo, sorry for that. 24:20.000 --> 24:22.000 Operating at scale. 24:22.000 --> 24:25.000 So, basically you don't want to log in, you don't want to reboot manually, 24:25.000 --> 24:30.000 because that distributes, that interferes with workloads, 24:30.000 --> 24:33.000 and it's also manual work, so you don't want to do that. 24:33.000 --> 24:37.000 So, for the last scenario, and that fortunately concluded, 24:37.000 --> 24:43.000 deployed a cluster API cluster on Azure. 24:43.000 --> 24:45.000 So, that's here now. 24:51.000 --> 24:53.000 Here's the version that I deployed. 24:53.000 --> 24:56.000 You see that it's a slightly lower Kubernetes version, 24:56.000 --> 25:00.000 and that's because the cluster API Azure automation uses that by default. 25:00.000 --> 25:15.000 The thing I'm going to do now is... 25:15.000 --> 25:20.000 I'm going to pre-empt us a little bit. 25:20.000 --> 25:25.000 Yeah, there's a dash F missing. 25:25.000 --> 25:28.000 And let it do its job so we can take a look later, 25:28.000 --> 25:31.000 and then show you the slides in between. 25:31.000 --> 25:34.000 So, cluster API is a really big thing. 25:34.000 --> 25:38.000 You would only use it for large operations at scale. 25:38.000 --> 25:40.000 It's not really an API. 25:40.000 --> 25:43.000 Kind of a misnomer, it's a cluster management system. 25:43.000 --> 25:47.000 And it uses a management cluster, which holds all the state for you, 25:47.000 --> 25:52.000 so you can go and provision workload clusters from that. 25:52.000 --> 25:58.000 One good example, which is being used is, if you host a hosted Kubernetes offer, 25:58.000 --> 26:02.000 and sell that or provide that to other users or customers, 26:02.000 --> 26:07.000 then cluster API is one kind of mechanism or tool they could use that. 26:07.000 --> 26:10.000 Now, these clusters have an immense scale. 26:10.000 --> 26:14.000 They are automatically managed from this management cluster. 26:14.000 --> 26:18.000 And at this point, the latest, you don't want to log in on the worker clusters, 26:18.000 --> 26:20.000 and reboot them. 26:20.000 --> 26:23.000 What can we do that safely? 26:23.000 --> 26:29.000 In cluster API, we ship cluster port with in-place updates prepared, 26:29.000 --> 26:32.000 but disabled by default. 26:32.000 --> 26:36.000 And if you prepare a new deploy and new cluster API workload cluster, 26:36.000 --> 26:41.000 which is what I did in the other screen, you would first install the provider, 26:41.000 --> 26:47.000 in my case, Azure, in the management cluster, generate a workload cluster configuration, 26:47.000 --> 26:51.000 which is a huge YAML file, and then apply that config to your management cluster, 26:51.000 --> 26:53.000 which will then spawn your workload cluster. 26:53.000 --> 26:59.000 Now, after step two, before step three, I patched that configuration 26:59.000 --> 27:03.000 to not just prepare and place updates, but enabled them. 27:03.000 --> 27:07.000 So those nodes are just sitting there now and just waiting for reboot. 27:07.000 --> 27:12.000 And they have a way of signaling that they just create a flag file in the fire system, 27:12.000 --> 27:19.000 which is a known path for cluster reboot operators to pick this up. 27:19.000 --> 27:27.000 And what I just did in the screen here, I deployed QD. 27:27.000 --> 27:34.000 QD is a node refresh operator, where you can basically instruct to safely reboot nodes. 27:34.000 --> 27:39.000 It's a CNCF project, just as cluster API and Fletcar are. 27:39.000 --> 27:44.000 And it'll handle safely draining your node, rebooting your node, and uncorning it again, 27:44.000 --> 27:46.000 so it's populated. 27:46.000 --> 27:51.000 Yeah, it's very flexible, configurable. 27:51.000 --> 27:54.000 So in a good bit of a time when the next target is starting now, 27:54.000 --> 27:58.000 so you're going to have to take the rest outside for questions, 27:58.000 --> 28:00.000 if anyone has any fear. 28:00.000 --> 28:02.000 Makes sense. 28:02.000 --> 28:08.000 That node updated.