WEBVTT

00:00.000 --> 00:07.000
Let's see.

00:07.000 --> 00:10.000
All right, hey everyone.

00:10.000 --> 00:11.000
My name is J. P. Lair.

00:11.000 --> 00:14.000
I'm the fourth presenter that has this logo on this slide,

00:14.000 --> 00:16.000
so apologies for stopping the dev room,

00:16.000 --> 00:18.000
but apparently we're somewhat active here.

00:18.000 --> 00:20.000
So I'm going to talk about programming

00:20.000 --> 00:21.000
as fantastic as needed.

00:21.000 --> 00:24.000
And in for us, you know, you can fill in the blanks

00:24.000 --> 00:26.000
whatever you think in for us.

00:26.000 --> 00:29.000
I think it's needed, but I would hope it's somebody else's job.

00:29.000 --> 00:32.000
But I guess what?

00:32.000 --> 00:36.000
I've kind of anyway.

00:36.000 --> 00:38.000
Why do we actually care?

00:38.000 --> 00:40.000
Why do we care about stuff?

00:40.000 --> 00:42.000
So we test upstream LVM.

00:42.000 --> 00:43.000
For two reasons.

00:43.000 --> 00:46.000
One is we have many people working immediately

00:46.000 --> 00:48.000
in upstream LVM, and of course we want

00:48.000 --> 00:49.000
to support the development, right?

00:49.000 --> 00:51.000
We want to support our upstream developers.

00:51.000 --> 00:53.000
So most of our actually back-end work

00:53.000 --> 00:55.000
for Amy GPU is done upstream.

00:55.000 --> 00:58.000
But also we want to guard downstream rockham.

00:58.000 --> 01:01.000
Because we actually merge upstream changes

01:01.000 --> 01:03.000
into downstream rockham three times a day.

01:03.000 --> 01:06.000
And whenever something breaks upstream

01:06.000 --> 01:09.000
and it hits a certain window where we pull

01:09.000 --> 01:11.000
in the changes for the merges,

01:11.000 --> 01:13.000
we break our internal build of rockham too.

01:13.000 --> 01:14.000
And we don't want to have that.

01:14.000 --> 01:17.000
And this is why we actually test upstream

01:17.000 --> 01:19.000
and test upstream more and more and more.

01:19.000 --> 01:22.000
And so we're bringing up more upstream testing.

01:22.000 --> 01:24.000
Now, in this presentation,

01:24.000 --> 01:27.000
I'm going to talk about my journey

01:27.000 --> 01:32.000
when I kind of inherited the then existing build

01:32.000 --> 01:34.000
bots and, you know,

01:34.000 --> 01:38.000
towards bringing online more build bots and working towards

01:38.000 --> 01:43.000
also potentially bringing online pre-competesting

01:43.000 --> 01:46.000
when you open a PR, such that we actually run on GPU

01:46.000 --> 01:49.000
tests if you touch stuff that we care about.

01:49.000 --> 01:51.000
Okay, so since it's about my journey,

01:51.000 --> 01:54.000
first things first, LVM testing landscape,

01:55.000 --> 01:57.000
there's three different technologies in use here.

01:57.000 --> 01:59.000
So, one is you have GitHub with

01:59.000 --> 02:01.000
a GitHub actions and a GitHub runners

02:01.000 --> 02:02.000
for some of the testing.

02:02.000 --> 02:05.000
So, for example, I think Lib CXX fully relies

02:05.000 --> 02:09.000
on GitHub runners and GitHub actions for the testing.

02:09.000 --> 02:11.000
Then there's built-kite,

02:11.000 --> 02:14.000
the current pre-compet testing is actually done.

02:14.000 --> 02:17.000
There's a GitHub workflow that puts together a string,

02:17.000 --> 02:20.000
which is a built-kite pipeline definition,

02:20.000 --> 02:22.000
and then sends it off to built-kite

02:22.000 --> 02:25.000
to test your PR changes through built-kite,

02:25.000 --> 02:28.000
and then report the result back to GitHub.

02:28.000 --> 02:30.000
I think it's going to go away at some point,

02:30.000 --> 02:32.000
but it's still there right now.

02:32.000 --> 02:35.000
And this is pre-compet, and then there's built-pot,

02:35.000 --> 02:39.000
and built-pot is the whole post-compet fleet.

02:39.000 --> 02:42.000
And this talk is mostly about built-pot.

02:42.000 --> 02:45.000
Simply because that's where I put

02:45.000 --> 02:48.000
most of the most of my energy so far in,

02:48.000 --> 02:53.000
and pre-compet is a little more new.

02:53.000 --> 02:55.000
So, first of all, built-pot.

02:55.000 --> 02:57.000
Let's talk a little bit about terminology,

02:57.000 --> 02:59.000
and I have this slide.

02:59.000 --> 03:01.000
There was a talk from David Spickett

03:01.000 --> 03:03.000
at the 2022 LVM Deaf meeting.

03:03.000 --> 03:04.000
If you're interested in the built-pot,

03:04.000 --> 03:06.000
I would highly recommend this talk,

03:06.000 --> 03:08.000
because it goes into much more detail

03:08.000 --> 03:11.000
what it is being a built-pot maintainer.

03:11.000 --> 03:14.000
But just for terminology-wise,

03:14.000 --> 03:15.000
so built-pot is orchestrated.

03:15.000 --> 03:17.000
You have a built-master, that's a server,

03:17.000 --> 03:20.000
that's run by Galena, so thank you, Galena.

03:20.000 --> 03:23.000
And then you have what is called builders.

03:23.000 --> 03:26.000
Builders are kind of the logical entity

03:26.000 --> 03:31.000
that typically would build a configuration of LVM.

03:31.000 --> 03:33.000
And these builders then run on a worker,

03:33.000 --> 03:35.000
and a worker is more or less the physical,

03:35.000 --> 03:39.000
machine or the actual thing that does the computing.

03:39.000 --> 03:41.000
So you can have a builder to work

03:41.000 --> 03:42.000
and mapping of one to one.

03:42.000 --> 03:43.000
You can have one to end.

03:43.000 --> 03:45.000
You can have, you know, you see it there.

03:46.000 --> 03:50.000
In our case, we always run a single builder on a single worker,

03:50.000 --> 03:55.000
so there's no, like we, or more crazy setup.

03:55.000 --> 03:59.000
If you scan the QR code, that's the link to David's talk.

03:59.000 --> 04:00.000
Just beware.

04:00.000 --> 04:03.000
I have another, I think couple of slides more with us QR codes.

04:03.000 --> 04:06.000
So this is the YouTube link to his talk.

04:06.000 --> 04:12.000
Oh, one thing to remember is the,

04:13.000 --> 04:16.000
the specification of the setup of these builders and these workers.

04:16.000 --> 04:19.000
That's maintained and what is called the LVM Zorque repository.

04:19.000 --> 04:21.000
So there's a separate repository for that.

04:21.000 --> 04:24.000
Okay, more terminology.

04:24.000 --> 04:27.000
In the repository, you will find building blocks

04:27.000 --> 04:30.000
that allow you to, you know, write your build paths.

04:30.000 --> 04:33.000
So for example, there's something that's called an annotated builder.

04:33.000 --> 04:35.000
That's kind of you build your own builder thing.

04:35.000 --> 04:37.000
So you have to script everything

04:37.000 --> 04:40.000
and tell it exactly the steps that this thing should do.

04:41.000 --> 04:44.000
And then there's more convenience kind of off the shelf things.

04:44.000 --> 04:47.000
So one, for example, is the OpenMP builder.

04:47.000 --> 04:52.000
This is kind of a standard way of building OpenMP in LVM.

04:52.000 --> 04:56.000
And the way this looks like is you have some sort of like Python stuff

04:56.000 --> 05:02.000
where you say, I have, you know, I have a builder name on a worker

05:02.000 --> 05:04.000
and it has a certain build directory.

05:04.000 --> 05:09.000
This has to be unique across the whole fleet of everything.

05:09.000 --> 05:12.000
So that's typically then why you put the built directory

05:12.000 --> 05:14.000
to be the same like the builder name.

05:14.000 --> 05:17.000
And then it depends on certain projects and you invoke a script.

05:17.000 --> 05:20.000
And so that's one of the annotated builders.

05:20.000 --> 05:22.000
OpenMP builder wouldn't fit you.

05:22.000 --> 05:26.000
Then anyway, our current build pot fleet roughly looks like this.

05:26.000 --> 05:31.000
So this is production we have about eight production bots right now.

05:31.000 --> 05:36.000
And I'm going to take a, now, talk you through how we arrived here.

05:36.000 --> 05:41.000
So let's first look at what I put the assignment here and what I call the inherited time

05:41.000 --> 05:45.000
is the initial time when I actually inherited to take care of these build bots.

05:45.000 --> 05:48.000
Then there's an intermediate time where I played around.

05:48.000 --> 05:50.000
Then there's the current time which is now.

05:50.000 --> 05:52.000
And then there's a bright future, right?

05:52.000 --> 05:54.000
Everybody likes the bright future.

05:54.000 --> 05:57.000
Okay, so let's, let's, we start with the inherited time.

05:57.000 --> 06:00.000
In heritage time was we had three machines basically.

06:00.000 --> 06:04.000
One for hip, one for openMP and one that basically just let the same thing

06:04.000 --> 06:06.000
as the, as one as the openMP bot.

06:06.000 --> 06:15.000
Then the third machine mirrored what, what the openMP machine was doing.

06:15.000 --> 06:20.000
And then when you wanted to make changes, you would bring up a PR to, again,

06:20.000 --> 06:21.000
Zork.

06:21.000 --> 06:22.000
You would land it.

06:22.000 --> 06:26.000
And then from Zork it would like trickle down to the builders.

06:26.000 --> 06:29.000
For the staging bots that would happen every two hours.

06:29.000 --> 06:32.000
For the production bots that need to send Galena and email that she needs to,

06:32.000 --> 06:34.000
bring it into the production bots.

06:34.000 --> 06:40.000
And this merging into staging again every two hours unless there's a problem.

06:40.000 --> 06:42.000
In the Zork repository.

06:42.000 --> 06:44.000
But you don't know.

06:44.000 --> 06:46.000
Because there was no linting or testing or anything.

06:46.000 --> 06:50.000
So it was like, I did submit a change to Zork eight hours ago.

06:50.000 --> 06:51.000
It didn't trickle down.

06:51.000 --> 06:52.000
Did I do wrong?

06:52.000 --> 06:54.000
Did somebody else put up a patch?

06:54.000 --> 06:57.000
And so sometimes it's just gotten email from Galena,

06:57.000 --> 06:59.000
telling you, oh, I landed your patch.

06:59.000 --> 07:02.000
Because there was something broken for commits before yours.

07:02.000 --> 07:04.000
And there was kind of a mess.

07:04.000 --> 07:06.000
But anyway, so typically it worked.

07:06.000 --> 07:08.000
And the builders didn't need to do like,

07:08.000 --> 07:10.000
didn't need too much attention.

07:10.000 --> 07:11.000
So everything's fine.

07:11.000 --> 07:12.000
There are three machines.

07:12.000 --> 07:16.000
But then we discussed, there's two machines doing the same thing.

07:16.000 --> 07:17.000
That's kind of annoying.

07:17.000 --> 07:19.000
Because we were bringing in more stuff.

07:19.000 --> 07:22.000
And for example, Joseph Hubert started his work on Lipsy for GPU.

07:22.000 --> 07:24.000
And so we wanted to test that, right?

07:24.000 --> 07:25.000
Okay.

07:26.000 --> 07:29.000
Does it work?

07:29.000 --> 07:31.000
The intermediate time.

07:31.000 --> 07:32.000
Okay.

07:32.000 --> 07:33.000
So we had this set up.

07:33.000 --> 07:35.000
And then Lipsy came along.

07:35.000 --> 07:38.000
Okay. So let's just make this thing also test Lipsy.

07:38.000 --> 07:39.000
Lipsy on GPU.

07:39.000 --> 07:42.000
So Lipsy on GPU is tested on a MD GPU.

07:42.000 --> 07:48.000
And then there was a change that broke our internal rockham build.

07:48.000 --> 07:49.000
But on Suzy.

07:49.000 --> 07:51.000
Suzy Linux 15.

07:51.000 --> 07:54.000
What does the default GCC version anybody knows?

07:55.000 --> 07:58.000
7.5.

07:58.000 --> 08:03.000
Typically developers do not use GCC 75 on the regular.

08:03.000 --> 08:08.000
So they use things from C++ 17 that are not present in GCC 75, right?

08:08.000 --> 08:09.000
And so that would break.

08:09.000 --> 08:12.000
So we brought online more builders,

08:12.000 --> 08:14.000
specifically a slas builder.

08:14.000 --> 08:17.000
So we would actually catch these things when they land.

08:17.000 --> 08:21.000
We also then put in well eight, well nine.

08:22.000 --> 08:25.000
And one that also does does flank testing.

08:25.000 --> 08:27.000
So we test flying too.

08:27.000 --> 08:30.000
And that's great.

08:30.000 --> 08:35.000
But bringing all these machines online was done manually.

08:35.000 --> 08:37.000
And you can, you know,

08:37.000 --> 08:39.000
I leave it up to you to think who,

08:39.000 --> 08:42.000
who that person is that's represented by two people here.

08:46.000 --> 08:47.000
Okay.

08:47.000 --> 08:49.000
But one of the things is you notice there's different

08:49.000 --> 08:50.000
icons here, right?

08:50.000 --> 08:53.000
So this, this and that up above,

08:53.000 --> 08:55.000
that's a container, right?

08:55.000 --> 08:56.000
It's a, it's a box.

08:56.000 --> 08:57.000
It's a container.

08:57.000 --> 08:58.000
And this is bare metal.

08:58.000 --> 09:02.000
So we would actually bring in the new builders more and more containerized.

09:05.000 --> 09:09.000
And that has advantages to some extent.

09:09.000 --> 09:13.000
So a couple of them is we have more machines that are larger.

09:13.000 --> 09:16.000
And with containers, we can nice or nice,

09:16.000 --> 09:19.000
these separate different builders with different OSs,

09:19.000 --> 09:23.000
put them on one machine and can organize for throughput,

09:23.000 --> 09:25.000
basically.

09:25.000 --> 09:28.000
And that's one, actually one of the reasons.

09:28.000 --> 09:32.000
And those bring us to the current time.

09:32.000 --> 09:34.000
Well,

09:34.000 --> 09:41.000
okay, so you're this.

09:41.000 --> 09:43.000
Let's see, I don't know what's coming next.

09:43.000 --> 09:45.000
There we go.

09:45.000 --> 09:47.000
Okay, I have to now can talk.

09:47.000 --> 09:51.000
So then I was working on actually containerizing the lower half two.

09:51.000 --> 09:54.000
And that's nice because I was actually the hit bot.

09:54.000 --> 09:57.000
That's the one that I'm the least familiar with.

09:57.000 --> 09:59.000
And then we found the problem there.

09:59.000 --> 10:02.000
In the sense that it went red.

10:02.000 --> 10:06.000
And we couldn't reproduce the issue locally.

10:06.000 --> 10:11.000
So we put it into staging back and I put up a container,

10:11.000 --> 10:17.000
created it because how the hit bot was set up was very documented

10:17.000 --> 10:20.000
and only internally documented.

10:20.000 --> 10:24.000
And then since I couldn't reproduce the issue then locally,

10:24.000 --> 10:26.000
in this containerized environment, we were like,

10:26.000 --> 10:28.000
okay, the rock conversion on the host is different.

10:28.000 --> 10:33.000
So let's update the rock conversion on the actual built-up or worker.

10:33.000 --> 10:34.000
Okay, great.

10:34.000 --> 10:35.000
Sure.

10:35.000 --> 10:36.000
Let's do this.

10:36.000 --> 10:39.000
So it runs updates and then it has to restart services.

10:39.000 --> 10:42.000
Okay, and then connection time down.

10:42.000 --> 10:45.000
Oops.

10:45.000 --> 10:47.000
Okay, I have IPMI information.

10:47.000 --> 10:52.000
I can log into the node and that IPMI information apparently is outdated.

10:52.000 --> 10:55.000
So I couldn't log into that node again, but I lost the builder.

10:55.000 --> 10:57.000
And thank you.

10:57.000 --> 11:00.000
I containerized this and I actually had put together some

11:00.000 --> 11:02.000
Ansible playbooks to deploy these things.

11:02.000 --> 11:08.000
So I can actually, I was actually able to build on the work that I did.

11:08.000 --> 11:11.000
And in about 30 minutes, turn this thing into a container.

11:11.000 --> 11:13.000
And now it's back into production, back in options,

11:13.000 --> 11:16.000
so we're testing again, also our hip stuff,

11:16.000 --> 11:22.000
and we're moving more and more stuff towards this more automated approach

11:22.000 --> 11:25.000
of containerized builders, deployment,

11:25.000 --> 11:28.000
via Ansible onto the actual machines.

11:28.000 --> 11:34.000
So let's talk about the bright future.

11:34.000 --> 11:37.000
Maybe just the future, but hopefully it's bright.

11:37.000 --> 11:41.000
And so this is where I would like to move our testing towards,

11:41.000 --> 11:45.000
because I recognize some of the problems that we have.

11:45.000 --> 11:50.000
So first is for post commit, I would like to get everything containerized

11:50.000 --> 11:54.000
and then make a distinction between what's called slow bots and fast bots.

11:54.000 --> 11:59.000
Fast bots are basically build only only little tests, nothing else.

11:59.000 --> 12:04.000
But the ideas that those things are fast enough to test every single commit,

12:04.000 --> 12:08.000
nothing batched, every single commit.

12:08.000 --> 12:12.000
Because if the testing batched, which is, you know, you have three or four commits,

12:12.000 --> 12:16.000
and your bot turns red, you don't know which commit that breaks.

12:16.000 --> 12:19.000
So as a maintainer, you then have to go locally,

12:19.000 --> 12:21.000
revert, check this, this, you know,

12:21.000 --> 12:25.000
is this the breaking change, re-apply, revert something else.

12:25.000 --> 12:28.000
And that's kind of annoying.

12:28.000 --> 12:32.000
The slow bots, that's, they will be actually batching things,

12:32.000 --> 12:37.000
but we'll have probably like a turnaround time between 45 and 45 minutes

12:37.000 --> 12:38.000
and an hour.

12:38.000 --> 12:41.000
So we could actually run some workloads through them.

12:41.000 --> 12:44.000
And see, you know, what is the impact on, for example,

12:44.000 --> 12:47.000
a spec CPU suite or whatever.

12:47.000 --> 12:53.000
And then, of course, there's the bright future with getup,

12:53.000 --> 12:57.000
actions, runner, controller, uncubanetties.

12:57.000 --> 13:01.000
And now, so that would give us pre-commit build

13:01.000 --> 13:05.000
and pre-commit test on GPU.

13:05.000 --> 13:09.000
And that's what I'm currently working on, and it's,

13:09.000 --> 13:12.000
that's also been a journey, but that's a different topic.

13:12.000 --> 13:18.000
Okay, so, lessons learned, best practices.

13:18.000 --> 13:22.000
Mostly lessons learned for me, I guess.

13:22.000 --> 13:25.000
When I inherited the single builder,

13:25.000 --> 13:28.000
it's easy enough to maintain a single thing.

13:28.000 --> 13:31.000
And it's quite well documented, actually.

13:31.000 --> 13:36.000
I think.

13:36.000 --> 13:39.000
When I started testing changes to the build,

13:39.000 --> 13:42.000
but was a nightmare, because it was basically non-existent.

13:42.000 --> 13:44.000
That's now much, much better.

13:44.000 --> 13:46.000
So I don't remember who landed the changes,

13:46.000 --> 13:47.000
but thank you so much.

13:47.000 --> 13:50.000
It's working great.

13:51.000 --> 13:55.000
And also, production builders would send you emails

13:55.000 --> 13:58.000
if the build breaks, but only if you actually

13:58.000 --> 14:01.000
marked them as sending emails in this file,

14:01.000 --> 14:04.000
that's something I learned about six weeks ago.

14:04.000 --> 14:06.000
So this is why I put this here.

14:06.000 --> 14:08.000
If you want to have your builder sending you emails

14:08.000 --> 14:12.000
as the maintainer, they have to be in this file.

14:12.000 --> 14:16.000
And the other, the next thing is that I learned is,

14:16.000 --> 14:19.000
you want to have your builds or the builders, actually,

14:19.000 --> 14:22.000
the build, but it's as reproducible as possible.

14:22.000 --> 14:25.000
Because you have a contributor asking

14:25.000 --> 14:27.000
how can it reproduce that issue?

14:27.000 --> 14:29.000
How can it reproduce the failing build?

14:29.000 --> 14:35.000
And then you say, ah, I have this machine, you know?

14:35.000 --> 14:40.000
And so I've now containerized all the environments

14:40.000 --> 14:42.000
that we use.

14:42.000 --> 14:46.000
And you will find the Dockerfiles through this link here.

14:47.000 --> 14:51.000
And also, I started to put together C-Make cache files

14:51.000 --> 14:55.000
that are entry in LVM trunk in the off-load project.

14:55.000 --> 14:58.000
And they are used as the build configs in our build pots.

14:58.000 --> 15:02.000
So you can actually download the Dockerfile build

15:02.000 --> 15:03.000
the Docker locally.

15:03.000 --> 15:07.000
And then simply do a ninja dash capital C linked to the C-Make cache file

15:07.000 --> 15:10.000
and get the exact build config that our build is run.

15:10.000 --> 15:15.000
So that should make it fairly easy to at least reproduce the build environment.

15:15.000 --> 15:19.000
All be it you might not have an AMD GPU to potentially then execute the test

15:19.000 --> 15:23.000
the same way we do it.

15:23.000 --> 15:27.000
And then finally, one of the things that I realized in the end is that

15:27.000 --> 15:30.000
if you're actually running a fleet of build pots,

15:30.000 --> 15:33.000
that's harder, but that's not harder because of build pots,

15:33.000 --> 15:36.000
but because you're actually managing a fleet of stuff, right?

15:36.000 --> 15:37.000
And that's a different thing.

15:37.000 --> 15:40.000
You have to think not only about, oh, how do I make this, you know,

15:40.000 --> 15:44.000
reproducible or whatever, but how do I manage eight machines,

15:44.000 --> 15:47.000
16 deployments, whatever?

15:47.000 --> 15:52.000
And that's, I think you should automate deployments

15:52.000 --> 15:56.000
through Ansible, for example, because that can serve as documentation,

15:56.000 --> 15:59.000
but also make your life easier in the long run.

15:59.000 --> 16:03.000
And if you found any of this interesting, we're hiring.

16:03.000 --> 16:04.000
All right?

16:04.000 --> 16:07.000
So come talk to me, come talk to other folks from AMD.

16:07.000 --> 16:10.000
I think I'm supposed to show you this.

16:10.000 --> 16:13.000
And thank you so much for having me to take questions.

16:13.000 --> 16:18.000
Thank you.

16:18.000 --> 16:26.000
How specific is build pots to build?

16:26.000 --> 16:29.000
How specific is build pot to LLVM?

16:29.000 --> 16:33.000
I think, according to the website, not at all.

16:33.000 --> 16:36.000
It could be used for some totally different project.

16:36.000 --> 16:40.000
It could be used for some totally different project.

16:40.000 --> 16:45.000
And it's because it's more a toolbox to build your own CI.

16:45.000 --> 16:48.000
Kind of thing.

16:48.000 --> 16:52.000
Yeah, build pot might be older than LVM was a comment here.

16:52.000 --> 16:54.000
Yes.

16:59.000 --> 17:00.000
We use C-Kesh.

17:00.000 --> 17:03.000
The question was whether we use any techniques to, yes,

17:03.000 --> 17:07.000
there's no way around C-Kesh for us.

17:07.000 --> 17:08.000
Yes.

17:12.000 --> 17:15.000
The question is, do we also use it for kernel or misa changes?

17:15.000 --> 17:16.000
We don't.

17:16.000 --> 17:19.000
This is completely what I showed.

17:19.000 --> 17:24.000
It's just LVM upstream and nothing else.

17:24.000 --> 17:26.000
No, this is not rockham.

17:26.000 --> 17:29.000
This is pure upstream.

17:29.000 --> 17:30.000
Yeah.

17:30.000 --> 17:32.000
Well, we do internally, that's a whole different story.

17:32.000 --> 17:34.000
But this is pure upstream.

17:34.000 --> 17:36.000
There's nothing rockham specific.

17:36.000 --> 17:37.000
We need this.

17:37.000 --> 17:40.000
We need the current fusion driver for the actually getting access to the GPUs.

17:40.000 --> 17:42.000
And then we have the HSA runtime on top of that.

17:42.000 --> 17:46.000
So we can build offloading tests, GPU offloading tests.

17:46.000 --> 17:47.000
But that's it.

17:47.000 --> 17:50.000
The build pots are not actually having large rockham installations.

17:50.000 --> 17:54.000
Yes.

17:54.000 --> 17:56.000
The HSA runtime is fixed.

17:56.000 --> 17:58.000
And we updated every now and then.

17:58.000 --> 18:02.000
And then, you know, that's reflected in the builder description.

18:02.000 --> 18:05.000
So we note which rockham version that builder is running.

18:06.000 --> 18:09.000
And so you could, you know, install the same thing.

18:09.000 --> 18:10.000
Thank you.

18:10.000 --> 18:11.000
Cool.

18:16.000 --> 18:17.000
All right.

18:17.000 --> 18:18.000
Thank you.