WEBVTT 00:30.000 --> 01:00.000 I'll try to hear a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little 01:00.000 --> 01:09.740 little bit of a little bit of much a little bit more of a little bit and putting some little bit of a little bit of a little bit of慢慢 like a little bit 01:09.780 --> 01:18.000 of a little bit of a little bit of a little bit 01:18.000 --> 01:45.000 Okay, so now we have Jonathan, who will explain us about all the things that he's doing with the Federation of Education for AI and HPC infrastructure. 01:46.000 --> 01:48.000 Hi, everyone. 01:48.000 --> 01:58.000 So I'm Jonathan Kalmos, and today I'm going to present to you a Federation of Education for AI and HPC infrastructure. 01:58.000 --> 02:00.000 All right, so it would be a background. 02:00.000 --> 02:05.000 I'm part of the Applied System team at Nvidia. 02:05.000 --> 02:11.000 And together we work on bringing up the next generation super-computer for AI. 02:12.000 --> 02:26.000 So essentially we focus on providing a set of best practices for things ranging from like performance, reliability and in this case security. 02:26.000 --> 02:32.000 And over the past few years we stood up numerous super-computers. 02:32.000 --> 02:37.000 Many of them ranked on the Todd 500 list. 02:37.000 --> 02:50.000 The latest one being EOS, which we submitted in 2023, and it's number nine in the world. 02:50.000 --> 02:57.000 So for those who are not super-family with our super-plot architecture, 02:58.000 --> 03:03.000 super-plot is basically a scalable unit that you're able to deploy in your data center. 03:03.000 --> 03:06.000 It's comprised of several racks. 03:06.000 --> 03:15.000 We have one rack that's dedicated for the control plane, where you have logging nodes, management nodes, storage nodes. 03:15.000 --> 03:20.000 And all the other racks are dedicated for compute. 03:20.000 --> 03:24.000 So it's where you're going to have your GPU heavy application. 03:25.000 --> 03:30.000 All of these racks are linked through various interconnects. 03:30.000 --> 03:38.000 So on the compute racks we have a memory fabric, which is very high speed, which is called multi-nodeinvillink. 03:38.000 --> 03:43.000 And that allows all the GPU to communicate very functionally within the rack. 03:43.000 --> 03:50.000 And then across the racks we have a lot of other interconnects, so we have an infinite fabric for storage. 03:50.000 --> 03:56.000 We have one for compute, for things like AI collectives. 03:56.000 --> 04:01.000 We have an internet network for managing the nodes. 04:01.000 --> 04:07.000 And we have one for compute and storage, which is not performance critical. 04:07.000 --> 04:11.000 Things like NFS. 04:11.000 --> 04:17.000 All right, so when it comes to educating users on all of these fabrics and various services, 04:18.000 --> 04:21.000 we had a lot of requirements. 04:21.000 --> 04:30.000 So first off we needed the authentication and authorization of our posics account that's tied to our corporate identities. 04:30.000 --> 04:35.000 I then video all our employees are storing the Microsoft Active Directory. 04:35.000 --> 04:43.000 So we need to be able to fetch those accounts, permissions, et cetera, and comply with policies. 04:43.000 --> 04:49.000 We also needed a unified solution that worked across all our services. 04:49.000 --> 04:54.000 In our case we have SSH, a storm for job scheduling. 04:54.000 --> 04:57.000 We have GitLab for continuous integration. 04:57.000 --> 04:59.000 NFS and Luster for storage. 04:59.000 --> 05:05.000 UFM, which is for a sharp, collective overflow to switching fabric. 05:05.000 --> 05:08.000 In case you're familiar with that. 05:08.000 --> 05:12.000 And then we have IMAX, which is for multi-nodeinvillink, which I just mentioned. 05:12.000 --> 05:18.000 We also needed something where you can authenticate on prem, as well as in the cloud. 05:18.000 --> 05:20.000 Short-lived credentials. 05:20.000 --> 05:25.000 We want to rotate these credentials often, so we want short-lived credentials. 05:25.000 --> 05:29.000 And we didn't want any client-side magnification. 05:29.000 --> 05:34.000 So we want our users to just be able to connect to our data center with plain SSH. 05:34.000 --> 05:41.000 We don't want maintaining clients on their side and having to upgrade that and fix vulnerabilities. 05:41.000 --> 05:49.000 And then we also wanted multi-factor authentication and single sign-on where we are applicable. 05:49.000 --> 05:53.000 And other things like we wanted it to be easily auditable. 05:53.000 --> 05:58.000 These are friendly, very minimal data center footprints. 05:58.000 --> 06:04.000 And so when we check out all the solution that exists out there, we sell on a free APA, 06:04.000 --> 06:09.000 which is basically the only option if you try to comply with all these requirements. 06:10.000 --> 06:13.000 And that's when we went with. 06:13.000 --> 06:18.000 And so Kerbos, which powers free APA obviously. 06:18.000 --> 06:26.000 All right, so to give you a little idea of the overall architecture. 06:26.000 --> 06:32.000 So we have free APA, which has a cross-force trust with our Active Directory, 06:32.000 --> 06:35.000 the corporate Active Directory that we have in video. 06:36.000 --> 06:44.000 And from there, our clients, the users, are able to authenticate to our login note through Active Directory. 06:44.000 --> 06:47.000 From there, they'll get an issue to tick it. 06:47.000 --> 06:53.000 And then with this ticker, they can actually submit a job through sperm. 06:53.000 --> 06:58.000 At this point, we'll register their credential and they're ticked inside service that we call civil. 06:58.000 --> 07:01.000 I'll talk about that later in the talk. 07:01.000 --> 07:05.000 And then at some point in time, the job is going to get scheduled. 07:05.000 --> 07:08.000 It's going to end up on some compute note. 07:08.000 --> 07:10.000 We're going to retrieve the credentials. 07:10.000 --> 07:19.000 And from there, the application can access very services like storage or all the things I mentioned. 07:19.000 --> 07:25.000 On the continuation side of things, it's slightly different. 07:25.000 --> 07:34.000 Here, our users are actually running pipeline from GitLab, which is outside of a cluster boundary. 07:34.000 --> 07:39.000 And GitLab is authenticated with OpenID Connect through enter ID. 07:39.000 --> 07:42.000 So here, what we do is slightly different. 07:42.000 --> 07:48.000 We effectively do a protocol transition from OA to Kerberos. 07:48.000 --> 07:54.000 So effectively, what we do is we impersonate the user as if he was actually submitting from the login note. 07:54.000 --> 07:58.000 Even though it's pipeline that's been triggered through GitLab. 07:58.000 --> 08:04.000 Again, this is a chief through civil, which I'll talk about later. 08:04.000 --> 08:09.000 All right, so as far as the support of authentication flow that we have. 08:09.000 --> 08:13.000 First off, we have password authentication. 08:13.000 --> 08:21.000 This is the classical Kerberos pre-auth, where you issue a password, you get ticket out of it. 08:21.000 --> 08:27.000 So here we authenticate your active directory. 08:27.000 --> 08:33.000 What we do, though, is we add a second factor in the. 08:33.000 --> 08:36.000 So we add a second factor as an SSH key. 08:36.000 --> 08:41.000 So you need your basically corporate credentials as well as an SSH key. 08:41.000 --> 08:46.000 And the public key is stored in ID view inside free IPA. 08:46.000 --> 08:56.000 So we can rotate the keys of our users and force them to update their second factor, basically. 08:56.000 --> 09:00.000 Note that you usually want Kerberos armoring when you do these kind of things, 09:00.000 --> 09:10.000 and also validate TGTs to avoid some sort of class of attacks that exist out there. 09:10.000 --> 09:15.000 The second thing we support is where we call active directory single sign-on. 09:15.000 --> 09:20.000 It's essentially where you client is already domain join. 09:20.000 --> 09:24.000 So you have a corporate laptop, which is already bound to AD. 09:24.000 --> 09:30.000 And you already have a ticket by the virtue of authenticating on your Windows laptop. 09:30.000 --> 09:37.000 In this case, we allow our users to come in with their existing ticket through what we call, 09:37.000 --> 09:41.000 what is called as unconstrained delegation. 09:41.000 --> 09:48.000 So we'll delegate their TGT to our login nodes and from there same flow applies. 09:48.000 --> 09:55.000 The downside of that is you usually need KCM on the login nodes to refresh the credentials, 09:55.000 --> 09:59.000 because once delegated the credential won't get refreshed. 09:59.000 --> 10:05.000 And so to avoid our users having to refresh manually their tickets, we use KCM. 10:06.000 --> 10:14.000 The second thing is that on Windows at least SSPI and constraint delegation doesn't work 10:14.000 --> 10:25.000 if you have credentialed guard turned on, which is a bummer, but you need to turn it off for this flow to your apply. 10:25.000 --> 10:29.000 Slight variation is entry-acloud turbo stress. 10:29.000 --> 10:35.000 This one is when your client is actually authenticated through the cloud. 10:35.000 --> 10:41.000 So he has a PRT from entry-ID, but he's still has a leg on your corporate network. 10:41.000 --> 10:46.000 So as far as I know it's only available through Windows Hello for Business, 10:46.000 --> 10:53.000 but what Windows will do is we'll automatically convert the PRT to turbo stick it. 10:53.000 --> 10:56.000 And from there we, it's the same flow as I explained before. 10:56.000 --> 11:01.000 You delegate to tick it and you can reuse that on your infrastructure. 11:01.000 --> 11:05.000 And lastly we support cloud authentication. 11:05.000 --> 11:10.000 This one is achieved through the external IDP. 11:10.000 --> 11:16.000 Support that free AP has has been describing in earlier talks. 11:17.000 --> 11:22.000 Here we, so free IPA does the open ID connect device code flow, 11:22.000 --> 11:29.000 which is entry-ID, which effectively grants you a carburett ticket. 11:29.000 --> 11:32.000 Now there are some challenges with this approach. 11:32.000 --> 11:38.000 Essentially, it's not AD, but free AP, we use a ticket. 11:38.000 --> 11:44.000 We have to switch back to the Active Directory identity. 11:44.000 --> 11:47.000 And for this, we use a local association. 11:47.000 --> 11:51.000 Once you're a session on the login node, we re-assess you locally, 11:51.000 --> 11:57.000 so as to change your identity with the Active Directory one. 11:57.000 --> 11:59.000 So some benefits. 11:59.000 --> 12:03.000 Using SSH will put you in new login session. 12:03.000 --> 12:07.000 I will go through PAM, so you'll get a new kernel of hearing, things they did. 12:07.000 --> 12:10.000 You'll go through turbo certification. 12:11.000 --> 12:16.000 And SSH will also copy or take it, which is good. 12:16.000 --> 12:20.000 If you were to use something like case, you wouldn't work. 12:20.000 --> 12:23.000 So we had to invent if here. 12:23.000 --> 12:27.000 In the future, we would like probably to use system D, run zero for that, 12:27.000 --> 12:31.000 but we have to upstream a carburett support for it. 12:31.000 --> 12:34.000 And other challenges you have with this approach. 12:34.000 --> 12:40.000 Microsoft on devices, which are platform-assist those, 12:40.000 --> 12:46.000 so that's Mac, Mac with Safari, and Windows essentially. 12:46.000 --> 12:50.000 The device could flow, well, not work with single sign-on, 12:50.000 --> 12:54.000 so every time you do this flow, you will have to re-authenticate in your browser. 12:54.000 --> 12:57.000 You won't keep the cookie in your browser. 12:58.000 --> 13:04.000 And to counteract that, we suggest our users to use SSH multiplexing, 13:04.000 --> 13:07.000 so that they only have to authenticate once, 13:07.000 --> 13:16.000 and they're controlled versus basically to a limit that's below their ticket lifeline. 13:16.000 --> 13:21.000 All right, so quickly on the user experience, it's fairly simple. 13:21.000 --> 13:26.000 The first flow, you get a password, and you have your SSH key. 13:26.000 --> 13:31.000 Second one is AD single sign-on, or the cloud carburett stress. 13:31.000 --> 13:36.000 You use SSH minus K to do just a CPI, uncontrained delegation, 13:36.000 --> 13:39.000 or SSPI delegation. 13:39.000 --> 13:43.000 Last one is the Entra Multifactor authentication. 13:43.000 --> 13:48.000 This one will prompt you to do the challenge in your browser, 13:48.000 --> 13:53.000 and then here, you'll get veto to whatever your policy requires. 13:53.000 --> 13:56.000 And as a result of that, you get your usual tickets, 13:56.000 --> 14:00.000 that allows you to access various resources on the cluster. 14:00.000 --> 14:04.000 And for us, we just, then something to explain the job 14:04.000 --> 14:07.000 with a container image that will inherit the telekering. 14:07.000 --> 14:12.000 We've already a ticket in it, and everything just works. 14:12.000 --> 14:15.000 All right, so I wanted to talk a little bit about civil, 14:15.000 --> 14:20.000 which is a project we developed for bridging the gap of batch 14:20.000 --> 14:24.000 and continuous integration workflows. 14:24.000 --> 14:27.000 So civil is kind of a small project. 14:27.000 --> 14:30.000 It's the mix of C and Rust. 14:30.000 --> 14:33.000 And you can see some mostly of three components. 14:33.000 --> 14:37.000 There is a server that you host alongside the free APA. 14:37.000 --> 14:43.000 There is a CLI that you use to delegate or impersonate user credentials. 14:43.000 --> 14:47.000 And then we have a slim plugin that will automatically do that 14:47.000 --> 14:51.000 for you as part of the job lifecycle. 14:51.000 --> 14:58.000 If you're familiar with AUX, this improves AUX on number of ways. 14:58.000 --> 15:03.000 And you also add support for impersonation through protocol transition. 15:03.000 --> 15:04.000 I'll describe that. 15:04.000 --> 15:09.000 So it can basically forge TGT as if it was a KDC. 15:09.000 --> 15:12.000 And we're still working on the full documentation for it, 15:12.000 --> 15:16.000 but if you're curious, please check it out. 15:16.000 --> 15:18.000 Okay, so the way it works. 15:18.000 --> 15:20.000 So it's a little bit complicated. 15:20.000 --> 15:22.000 I'll try to summarize. 15:22.000 --> 15:26.000 So your user authenticate on the login node as usual. 15:26.000 --> 15:29.000 It goes through the AD authentication. 15:29.000 --> 15:30.000 It gets a ticket. 15:30.000 --> 15:35.000 From there you will ask for them to submit a job. 15:35.000 --> 15:38.000 So I'm going to allocate some credentials. 15:38.000 --> 15:45.000 And from there, our plugin, the spank plugin, in slurim for civil. 15:45.000 --> 15:48.000 We'll get your ticket. 15:48.000 --> 15:51.000 And we'll send it to our civil server. 15:51.000 --> 15:54.000 Our civil server in turns stores it in KTM, 15:54.000 --> 15:58.000 so that the ticket will continuously get refreshed 15:58.000 --> 16:01.000 until the job gets scheduled. 16:01.000 --> 16:05.000 And at some point in time, the job will get launched 16:05.000 --> 16:07.000 on the computer nodes. 16:07.000 --> 16:09.000 And same thing on the computer, 16:09.000 --> 16:11.000 we have our civil plugin, 16:11.000 --> 16:13.000 which will retrieve the ticket, 16:13.000 --> 16:17.000 so that the job can have all the credentialing needs. 16:17.000 --> 16:22.000 We'll also spawn site-card process. 16:22.000 --> 16:25.000 There will keep this ticket alive. 16:25.000 --> 16:29.000 And it does that by reaching out to civil D. 16:30.000 --> 16:32.000 There's a reason for that. 16:32.000 --> 16:38.000 So that we don't do excessive request to the KDCs. 16:38.000 --> 16:43.000 Pearl, because it could involve some cross-round interaction here. 16:43.000 --> 16:46.000 So we only hit the civil D when all the nodes 16:46.000 --> 16:51.000 basically start to run the job. 16:51.000 --> 16:56.000 And the other thing is that if you take it word to expire, 16:56.000 --> 16:58.000 while your job is still in queue, 16:58.000 --> 17:01.000 what we do is we notify the users. 17:01.000 --> 17:05.000 And we ask them to be really authenticated on the cluster. 17:05.000 --> 17:09.000 The way we do that is we have a small wrapper around K&IT 17:09.000 --> 17:12.000 that summits your ticket through civil. 17:12.000 --> 17:16.000 And we'll also have a hand plug-in in civil, 17:16.000 --> 17:18.000 so that if you adjust access to the cluster, 17:18.000 --> 17:24.000 we'll automatically forward your ticket to our server. 17:24.000 --> 17:30.000 And for CI-CD, what we do instead. 17:30.000 --> 17:34.000 So we do something called as protocol transition. 17:34.000 --> 17:39.000 It involves the service for user extension of Kerberos. 17:39.000 --> 17:43.000 So here we have a user that's authenticated against GitLab 17:43.000 --> 17:45.000 through OpenIG Connect. 17:45.000 --> 17:48.000 And this in terms of authentication against Entra. 17:48.000 --> 17:53.000 And for the GitLab summits the pipeline to our custom 17:53.000 --> 17:57.000 GitLab runner that we developed. 17:57.000 --> 18:03.000 This runner is going to use civil to essentially do cross-ground 18:03.000 --> 18:04.000 protocol transition. 18:04.000 --> 18:08.000 So it's going to get a service for you to self-ticket 18:08.000 --> 18:11.000 for itself impersonating the user. 18:11.000 --> 18:15.000 And with that, it's going to get a service for user to 18:15.000 --> 18:18.000 proxy ticket to the civil server. 18:18.000 --> 18:21.000 At which point you will be able to request basically the 18:21.000 --> 18:25.000 forgury of a brand new TGT that represent the user. 18:25.000 --> 18:30.000 And basically impersonating it to without 18:30.000 --> 18:35.000 involvement of the user. 18:35.000 --> 18:39.000 The thing to note here is that the actual ticket that's 18:39.000 --> 18:42.000 being generated by civil doesn't exist in the KDC. 18:42.000 --> 18:47.000 It's is effectively just a shadow principle of what 18:47.000 --> 18:50.000 exists in Active Directory. 18:50.000 --> 18:53.000 But we make sure that basically those two principle maps 18:53.000 --> 19:00.000 to the same user so that we are allowed to do that. 19:00.000 --> 19:03.000 And yeah, that's pretty much it. 19:03.000 --> 19:05.000 This is a little bit more involved. 19:05.000 --> 19:08.000 So if you have questions about that, I'm happy to answer. 19:08.000 --> 19:10.000 Otherwise, just check out the code. 19:10.000 --> 19:13.000 It's fairly simple. 19:13.000 --> 19:15.000 It's a conclusion. 19:15.000 --> 19:19.000 So free IP bridges the gap between corporate identities, 19:19.000 --> 19:22.000 cloud authentication, and kind of the traditional 19:22.000 --> 19:25.000 physics account that you have on HPC clusters. 19:25.000 --> 19:29.000 We were able to implement all our authentication and 19:29.000 --> 19:32.000 to end across all our cluster resources. 19:32.000 --> 19:35.000 The only thing we had to do is develop these small civil 19:35.000 --> 19:38.000 server to address some of the scenarios that 19:38.000 --> 19:41.000 couldn't be covered by standard Kerberos and free 19:41.000 --> 19:43.000 IPA. 19:43.000 --> 19:47.000 And as the next step, we would like to look into 19:47.000 --> 19:50.000 Kerberizing our slurmo authentication itself. 19:50.000 --> 19:53.000 Right now, Slurmo does its own authentication across 19:53.000 --> 19:55.000 the nodes. 19:55.000 --> 20:00.000 Maybe also look into MPI authentication. 20:00.000 --> 20:04.000 MPI is a very well-eversed in HPC. 20:04.000 --> 20:07.000 So we would like to try to authenticate that. 20:07.000 --> 20:10.000 And I mentioned it, but we would like to replace this local 20:10.000 --> 20:14.000 SSH in our multi-factor authentication thing with 20:14.000 --> 20:18.000 system DRUN zero in the future. 20:18.000 --> 20:20.000 And that's it. 20:20.000 --> 20:23.000 And before I take any questions, if you're also curious 20:23.000 --> 20:28.000 about all the other types of authentication that we have, 20:28.000 --> 20:30.000 I have some backup slide. 20:30.000 --> 20:34.000 You have the SSH one, the NFS one, the last one, 20:34.000 --> 20:36.000 the SHARP and multi-nodding link. 20:36.000 --> 20:40.000 There's a really standard Kerberos stuff. 20:40.000 --> 20:43.000 The only thing I would note is that for LASTER, 20:43.000 --> 20:46.000 we did a lot of work with the community to 20:46.000 --> 20:49.000 bring it on par with an FS. 20:49.000 --> 20:52.000 And so if you're curious about this one, 20:52.000 --> 20:55.000 feel free to look at the LASTER user group conference, 20:55.000 --> 20:58.000 where we describe all the work we did there. 20:58.000 --> 21:01.000 And with that, if you have any questions. 21:01.000 --> 21:16.000 So thank you very much for this talk. 21:16.000 --> 21:21.000 So we did a review of SIPL code. 21:21.000 --> 21:28.000 And Simon and I, we believe that what you are trying to achieve 21:28.000 --> 21:32.000 could be achieved with the standard Kerberos methods 21:32.000 --> 21:34.000 through the SPU. 21:34.000 --> 21:39.000 What perhaps we are missing is... 21:39.000 --> 21:42.000 The slide difference is that the S4 is the first step 21:42.000 --> 21:44.000 to actually get a full TGT. 21:44.000 --> 21:46.000 And I think that's what you don't like. 21:46.000 --> 21:48.000 You don't need to do a full TGT. 21:48.000 --> 21:50.000 So what we do is product authorization, 21:50.000 --> 21:54.000 you get a ticket to yourself, S4 to yourself. 21:54.000 --> 21:56.000 Yeah, the thing is we don't want that because 21:56.000 --> 22:00.000 they are users are able to SSHU the remote or 22:00.000 --> 22:04.000 basically you need a full TGT on the remote 22:04.000 --> 22:09.000 notes to then have the user be able to access any ticket. 22:09.000 --> 22:11.000 Yeah, this is not on the user side. 22:11.000 --> 22:13.000 This is on the SIPL as a service side. 22:13.000 --> 22:14.000 Oh yeah. 22:14.000 --> 22:16.000 We can do this product authorization there. 22:16.000 --> 22:20.000 We don't need full TGT or the user. 22:20.000 --> 22:24.000 And at every place where you need full TGT, 22:24.000 --> 22:27.000 you need our RBCD rule and product authorization. 22:27.000 --> 22:28.000 That's all you need. 22:28.000 --> 22:29.000 Yeah. 22:29.000 --> 22:31.000 The only problem is that you don't have a head of time 22:31.000 --> 22:33.000 with the user once, right? 22:33.000 --> 22:35.000 So you don't know what service the user is going to 22:35.000 --> 22:36.000 access. 22:36.000 --> 22:40.000 But we have pack information where we can put 22:40.000 --> 22:43.000 additional details that can be used 22:43.000 --> 22:46.000 during the authorization of RBCD rules. 22:46.000 --> 22:50.000 So we can solve this much more secure 22:50.000 --> 22:53.000 without the ability to leak any of those things. 22:53.000 --> 22:54.000 Yeah. 22:54.000 --> 22:56.000 I would be interested to have that even in the KDC. 22:56.000 --> 23:00.000 I looked at maybe doing like MIT, 23:00.000 --> 23:02.000 Cabro's plugin for that type of things. 23:02.000 --> 23:03.000 Yeah. 23:03.000 --> 23:04.000 I think that would be very useful. 23:04.000 --> 23:08.000 I think CI and Procultransition in general is kind of 23:08.000 --> 23:09.000 not addressed by free idea. 23:09.000 --> 23:13.000 It needs a bit more context than you get 23:13.000 --> 23:18.000 in the raw KDC with metrics to get the authentication 23:18.000 --> 23:20.000 indicator support. 23:20.000 --> 23:21.000 Okay. 23:21.000 --> 23:24.000 Which, for example, we'll allow you to limit 23:24.000 --> 23:28.000 GSA API authentication in SSH 23:28.000 --> 23:31.000 to the point that only users who used 23:31.000 --> 23:35.000 enter ID to authenticate allow it to pass through. 23:35.000 --> 23:36.000 Yeah. 23:36.000 --> 23:39.000 I know there's several patches that are not upstream. 23:39.000 --> 23:42.000 There's like, I think the GSSR key that's not there is 23:43.000 --> 23:46.000 I think the SSH upstream, just as if you are only 23:46.000 --> 23:49.000 is limited to storing the ticket in a file, 23:49.000 --> 23:51.000 not in the kernel keyring. 23:51.000 --> 23:53.000 There's a bunch of patches I know that debut 23:53.000 --> 23:55.000 and red hat have. 23:55.000 --> 23:59.000 That would be great to have all of that in open SSH upstream. 23:59.000 --> 24:02.000 But yeah, we don't necessarily need that. 24:02.000 --> 24:03.000 Okay. 24:06.000 --> 24:07.000 We still have time. 24:07.000 --> 24:09.000 So any other question? 24:12.000 --> 24:17.000 Okay. 24:17.000 --> 24:18.000 So thank you. 24:18.000 --> 24:21.000 And I'm pretty sure we are going to fix those things 24:21.000 --> 24:22.000 together. 24:22.000 --> 24:23.000 And yeah. 24:23.000 --> 24:33.000 Thank you. 24:53.000 --> 24:55.000 Thank you. 25:23.000 --> 25:25.000 Thank you. 25:45.000 --> 25:47.000 Okay. 25:49.000 --> 25:50.000 Yeah. 25:50.000 --> 25:51.000 Yeah. 25:51.000 --> 25:53.000 I'm not sure if I can get this one. 25:53.000 --> 25:55.000 I'm not sure if I can get this one. 25:55.000 --> 25:57.000 I'm not sure if I can get this one. 25:57.000 --> 25:59.000 Oh yeah. 26:03.000 --> 26:05.000 Two out of three. 26:11.000 --> 26:13.000 Yeah almost let me just check.