WEBVTT 00:00.000 --> 00:22.000 So I think we can start, so thanks for joining, then I mean assuming that you're not here for me, but so two years ago I came to share a little project I was working on trying to solve a problem that I was facing. 00:22.000 --> 00:26.000 So the idea is, I'm maintaining quite a lot of git repositories. 00:26.000 --> 00:28.000 Is it moving? 00:38.000 --> 00:39.000 Do this this way. 00:39.000 --> 00:42.000 So basically I'm mentioning quite a lot of git repositories. 00:42.000 --> 00:49.000 Those git repositories contain quite a lot of informations and those information can get audited pretty quickly. 00:49.000 --> 00:56.000 And so my problem was how can I identify those information and then to be able to do whatever was needed? 00:56.000 --> 01:01.000 Sometimes it's just updating a version, sometimes it's updating an installation links. 01:01.000 --> 01:08.000 So it was really about being able to identify what to monitor and what to do when we find something that is audited. 01:08.000 --> 01:15.000 So at the beginning I was working on a Jenkins project where we have like thousands of git repositories. 01:15.000 --> 01:22.000 So we already use scripts like Python Ruby and so on to catch those audited information. 01:22.000 --> 01:32.000 We were already using as well to like renovate but renovate but it under but the problem was it was not flexible enough for us to be able to identify all those changes. 01:32.000 --> 01:41.000 So we started working on a new project, which is like, I'm pretty bad at naming things, so just to see like that of this stuff. 01:41.000 --> 01:45.000 The best way to describe this is it's a declarative dependency management. 01:45.000 --> 01:54.000 So the idea is really to be able to explain, so to explain what we are monitoring and what we want to do. 01:54.000 --> 01:59.000 So the core has been pretty stable since the beginning of the project five years ago. 01:59.000 --> 02:05.000 So we have like five main resources, the source which defines where an information is coming from. 02:05.000 --> 02:19.000 So for example, in this case we are monitoring the Java version 17, but I could be like Java, I could be a lot of different things. 02:19.000 --> 02:24.000 We have like a plugin ecosystem, so we can really try to adapt to the different scenarios. 02:24.000 --> 02:30.000 Conditions mean this is a condition that needs to pass before moving to the next source. 02:30.000 --> 02:37.000 So that would be, doesn't make sense to update, I don't know, a hand chart if the image is not available, the current image is not available. 02:37.000 --> 02:42.000 So those are things you want to check, the target is what you need to monitor. 02:42.000 --> 02:49.000 So for example, in this case, that's a target of kind of Dockerfile, so we are monitoring Dockerfile with a specific Docker instruction. 02:49.000 --> 02:55.000 And so if for some reason the Java version is not set to the latest version of Java defined by the source, 02:55.000 --> 03:03.000 then update rely is designed to update that information, so we can move to the next, so we can just update information. 03:03.000 --> 03:13.000 As Sam resource is Git repository is that we want to interact with, so it are fetching information from a Git repository or updating a Git repository. 03:13.000 --> 03:18.000 And finally the action is that doing something one of the target is updated. 03:18.000 --> 03:23.000 So the idea here is really to define a pipeline about how to update files. 03:23.000 --> 03:28.000 And then run the commands from a CI, from the machine, from wherever it wants. 03:28.000 --> 03:36.000 Yeah, we made the same. So this is an example on the Jenkins project where we tried to monitor all the Docker images that we are building, 03:36.000 --> 03:41.000 and we have the core images for different Java version, different architectures, for example. 03:41.000 --> 03:49.000 So this diagram was generated by of the CI, so the way it works is, you just show the different components of that specific pipeline. 03:49.000 --> 03:52.000 And there is a cure code if you want to see what the manifest looks like because it's pretty long one. 03:52.000 --> 03:57.000 But what we do here is we have one source where we monitor the Java version, as I show in the previous example. 03:57.000 --> 04:08.000 We have one condition, which is check that we have a Docker image available for specific target for all the different architectures that we are monitoring. 04:08.000 --> 04:15.000 And finally we have a lot of targets, and those targets represents different files that we are monitoring, like, 04:15.000 --> 04:21.000 a Docker file, a packer because we are using Packer, so we monitor HGL file, we monitor. 04:21.000 --> 04:30.000 And so yeah, we need to be able to monitor all those different files, and then open one PR, so that's someone can just validate that. 04:30.000 --> 04:34.000 We want to use that latest version. 04:34.000 --> 04:41.000 So as I said, the core remained the same, it's always the same, like running kitsch code, fetch information, and so on. 04:41.000 --> 04:47.000 We have kind of built over time, kind of plugin ecosystem, where each plugin is dedicated to a specific problem. 04:47.000 --> 04:56.000 So for example, the yellow file is about defining what kind of file you want to do, what are the key, want to monitor, and what they can value you are expecting there. 04:56.000 --> 05:00.000 And so all those plugins work, so we work with different git providers as well. 05:01.000 --> 05:20.000 This approach about like being very flexible about what we need to do is on one side it's like quite simple, because if one day you're not monitoring the version 17, but you want to monitor the version 21, just one variable to change, so you're not like having to go deep in some complex scripts. 05:20.000 --> 05:33.000 The other part that I like here as well is you just have one process, so you know that you always monitoring a specific piece of information, so you group all those changes at once, so the thing know what you do. 05:33.000 --> 05:43.000 The counter side of this is you have to write manifest, so obviously that's, I mean, the less manifest you write the better we all, we are all, but. 05:44.000 --> 05:56.000 And more importantly, what is really challenging with this approach is that you need to know what you are monitoring, so you need to know an advance what all the five, so if someone in the team tomorrow decide to, I don't know, had a new verify. 05:56.000 --> 06:07.000 Someone, that person also need to know that the data automation to be able to catch those updates, so on one side it bring a lot of flexibility, but on your side it also. 06:07.000 --> 06:15.000 For the team to change the mindset about all we're thinking about, okay, I'm bringing something to the git repository, is that for good that version become outdated. 06:15.000 --> 06:21.000 And if it's the case try to do something that could be sure that tomorrow it's still working. 06:22.000 --> 06:42.000 Um, over time the project evolved as a need, as only developed as well, so we, we started to think about like, okay, we have to write manifest to define what or approach, what the procedural looks like, what if we could detect automatically based on some git repositories, what we need to monitor. 06:42.000 --> 06:54.000 So for people familiar with tip and about, you know, who's familiar with tip and about a renovating room, okay, so that's that's pretty much the closest idea here is that we let up the July to analyze a git repository. 06:54.000 --> 07:01.000 To detect like it has a map and file or go go modify it for go and then it generates the manifest and apply them straight away. 07:01.000 --> 07:09.000 Um, the purpose was not to replace renovate or tip and about the purpose was more to be like, um, something I would work well with it. 07:09.000 --> 07:14.000 But yeah, that was it turns out that, um, we could also. 07:14.000 --> 07:24.000 Automate different, up to make it more easier to to make different, again, scenarios. So this one is an example of a very small manifest where we want to to make this in the hand project. 07:24.000 --> 07:33.000 And so if you have this more manifest that tells us like monitor any hand dependencies in that git repository, um, do whatever needs to be done. 07:33.000 --> 07:47.000 So the, at this moment we're not any more about defining advanced what we need to update, but we're trying to guess which can with a different set of challenges because each ecosystem, um, as its own set of rules. 07:47.000 --> 07:52.000 So for example, I mean, you know, that in the Docker, the Docker ecosystem, you can use whatever version you want. 07:52.000 --> 07:56.000 And so those the different version can mean different things to different people. 07:56.000 --> 08:04.000 Um, so it can be rich for you to identify what should be the next version. So that can be the different, it's really come with a different set of challenges. 08:04.000 --> 08:15.000 But it was kind of you sort of from time to time. So the biggest benefit of this one is like the same with tools like depend about is, um, you don't necessarily know need to know what you want to update. 08:15.000 --> 08:23.000 Just let the tool detect those, but then over time you also need to be able to to limit some updates or to say like this, this dependency I don't really care. 08:23.000 --> 08:30.000 I'm fine to update that once a month or I want to group those informations together. 08:30.000 --> 08:39.000 Um, but on the other side, that's the same approach with all of those where it's really noisy because you quickly get like hundreds of dependencies in your project. 08:39.000 --> 08:45.000 Um, and you need to to decide when to what to do when something change. 08:45.000 --> 08:55.000 Um, those are the main main approach where we try to be as flexible as possible to be able to identify all those policies. 08:55.000 --> 08:59.000 But what we noticed of the time is that we ended up copypasting the same. 08:59.000 --> 09:06.000 Yeah, we'll manifest over many different repositories. So we were wondering what if we could just have like an update policy. 09:06.000 --> 09:17.000 Um, and that's what we did. So we know what we have is that we have the ability to publish those policies to a registry, like Docker Hub, GHGR, the Taiyo for example. 09:17.000 --> 09:24.000 And we can version those policies. So we just try to have like generate update policies for the different projects. 09:24.000 --> 09:30.000 Um, and so we now have like one fine name of this, you like compose that channel and then there we can list like all the policies we want to do. 09:30.000 --> 09:37.000 We can just change the parameters. So it became a lot easier to scale about the different repositories work. 09:37.000 --> 09:42.000 So the way a policy, I mean, if you're curious, there is a good repository name of this. 09:42.000 --> 09:49.000 Eli slash policies where we have like a lot of policies we use for various open source project. We are working on. 09:49.000 --> 09:53.000 But a policy like a bit similar to him. 09:54.000 --> 10:03.000 Um, um, um, um, um, hurting artifact, where we specify metadata. And then we can, yeah, publish that to, to our registry. 10:03.000 --> 10:18.000 So what we like with this is we moved from, um, now we have like one repository where we maintain all the update policies. So we really have like few people looking into those policies and 10:18.000 --> 10:22.160 You can more is your to to change those a bit policies over time because you can version 10:22.160 --> 10:23.160 them. 10:23.160 --> 10:26.800 So you can also have a different policies if you're on prediction environments or if 10:26.800 --> 10:30.800 you're in a deaf repository. 10:30.800 --> 10:34.680 The control side of this is you just have more templating in place. 10:34.680 --> 10:38.680 So you can end up with very complex policies if you're depending on what you're trying to 10:38.680 --> 10:39.680 do. 10:39.680 --> 10:45.960 So the last part I want to finish here is I just want to thank all the people who contribute. 10:45.960 --> 10:50.280 So this project started what I was working on a Jenkins project as a side project and over 10:50.280 --> 10:54.760 time we started to see a lot more people contributing to this small project. 10:54.760 --> 11:00.840 We also have sponsoring from time to time so thanks to perfect HQ is also sponsoring 11:00.840 --> 11:05.200 the project which gives us the ability to have stickers and some to sponsor some of 11:05.200 --> 11:06.480 some of some stuff. 11:06.480 --> 11:09.880 So yeah that's that was nice to see this project evolving. 11:09.880 --> 11:15.640 If you're curious we have a chat where we regularly interact with the different people. 11:15.640 --> 11:20.880 I'm not sure I'm going to have that kind of time to go on this project so I'm going to 11:20.880 --> 11:23.880 just keep this one that I get. 11:23.880 --> 11:30.800 So yeah I'm going to stop here. 11:30.800 --> 11:38.040 So yeah that was it took a quick update on how this project evolved over the past few years. 11:38.040 --> 11:39.040 That's it. 11:39.040 --> 11:40.040 Yes. 11:40.040 --> 11:53.200 How do you deal with like tricky versions of Russian numbers if you have like timestamps 11:53.200 --> 11:54.560 or something like that? 11:54.560 --> 12:00.360 So the thing is when we specify your source information you specify for example Java let's 12:00.360 --> 12:05.320 say don't have an example that I show and then we also have way to filter the version and 12:05.320 --> 12:10.400 then you can have different roles like semantic versioning dates. 12:10.400 --> 12:15.280 We have a few of them like that but the challenge you have here if you go with a declarative 12:15.280 --> 12:19.920 approach and that you know what you're looking for you can respecify I'm looking for the 12:19.920 --> 12:23.600 latest version matching a date on the line you can specify some constraints. 12:23.600 --> 12:28.000 When you go with the auto discovery which is like like the button above it that's a tricky 12:28.000 --> 12:32.720 part because you don't always know when the meaning of the version and sometimes you can 12:32.720 --> 12:34.880 have version that are very similar. 12:34.880 --> 12:39.600 So for example some projects use a version as a date and that date could be like a huge 12:39.600 --> 12:43.320 major version in the context of semantic versioning and so you have when you have like 12:43.320 --> 12:49.080 very old project you can become really tricky to know what should be the latest version 12:49.080 --> 12:51.880 and so that's why the declarative approach is really useful. 12:51.880 --> 12:55.920 So typically if you have internal dependencies and you know what you're monitoring you can 12:55.920 --> 13:01.280 just specify I'm looking for the latest information based on some date for example and you 13:01.280 --> 13:04.160 can also specify if you want to transform that information. 13:04.160 --> 13:10.160 So if you're for example some project to be version where they have the project name dash 13:10.160 --> 13:13.960 and semantic version that means as it's not really semantic versioning because they have 13:13.960 --> 13:17.760 the project name in front of it so you have like extra things to do if you want to know 13:17.760 --> 13:20.680 what should be the latest version but that definitely something went out. 13:20.680 --> 13:27.240 Can you tell it to stop upgrading when you definitely know that there's a major version? 13:27.240 --> 13:28.240 Yes. 13:28.320 --> 13:33.280 That's something we can as well do we have plenty of example once again that that one comes 13:33.280 --> 13:37.760 with one questions about if you know let's say you want to stop at the version 20 so you put 13:37.760 --> 13:42.640 some rules I don't give me a version after 20 but then at some point someone need to update 13:42.640 --> 13:47.440 the update policy to know that no you're looking for a version 21 for example and that's 13:47.440 --> 13:53.280 where the policy is we is useful is because you have one policy that says don't go over 13:53.280 --> 13:57.600 20 and then the next version will be don't go over 21 and so then you have the control 13:57.600 --> 14:01.600 to decide which gets the repository is monitoring the version 20 which gives the repository 14:01.600 --> 14:03.600 is monitoring the version 21. 14:03.600 --> 14:06.600 Can you send that it would all automatically break the R? 14:06.600 --> 14:07.600 Yes. 14:07.600 --> 14:08.600 Yes exactly. 14:08.600 --> 14:13.600 So that's the idea I can show an example do I have here? 14:13.600 --> 14:14.600 One minute. 14:14.600 --> 14:19.600 Okay one minute and then I finished. 14:19.600 --> 14:26.600 For example if I go, I think you can send from, I think you can see I. 14:26.600 --> 14:30.600 The term. 14:30.600 --> 14:37.600 If I go for performance so there is a bunch of policies so the thing is it's not only 14:37.600 --> 14:43.600 used to track I mean we can use it to track a lot of different kind of information so for example 14:43.600 --> 14:47.600 here we specify the git repository is we want to interact with. 14:47.600 --> 14:50.600 Now we have the action at the bottom. 14:50.600 --> 14:51.600 Yes here. 14:51.600 --> 14:55.600 So here we specify what should be the title of the PR what should be the labels of the PR. 14:55.600 --> 14:59.600 You have like a few configuration you can do like for example you can also open. 14:59.600 --> 15:04.600 A PR on the upstream repository so for example if you fork your repository you can automatically 15:04.600 --> 15:05.600 open it upstream. 15:05.600 --> 15:10.600 You have quite a lot of functionalities usually we use it all the time so if someone needs something 15:10.600 --> 15:14.600 just add the feature depending on the needs and then starting from there. 15:14.600 --> 15:18.600 Yeah that's what you are asking me can we filter version. 15:18.600 --> 15:23.600 Yeah definitely you just have to specify what's the version you're looking for you know and then you can do 15:23.600 --> 15:28.600 things like remove the V prefix and sound so it's really you have if you think if you know what you're looking 15:28.600 --> 15:31.600 for it's very powerful to do. 15:31.600 --> 15:33.600 It's over. 15:33.600 --> 15:34.600 Next. 15:34.600 --> 15:35.600 Thanks. 15:35.600 --> 15:39.600 Thank you.