WEBVTT 00:00.000 --> 00:15.840 Hi everyone, my name is Andrei Andriyushan, I'm a DevOps engineer at Analog Devices and I'm excited 00:15.840 --> 00:27.000 to talk about our process for refining the release strategy of a custom Linux distra. 00:27.000 --> 00:36.720 I will share how we've optimized the release process for a Kuiper, which is Analog Devices 00:36.720 --> 00:40.840 Linux custom Linux distribution. 00:40.840 --> 00:54.840 And I will start by presenting what was before, sorry, I will start by telling you what 00:54.840 --> 01:05.320 is a custom Linux distra, what is Kuiper, basically Kuiper serves as the primary distribution 01:05.320 --> 01:13.240 for product evaluation boards, offering reference designs, device drivers and a variety 01:13.240 --> 01:17.960 of development capabilities. 01:17.960 --> 01:32.360 So includes pre-build boot files for over 140 FBGA systems, sorry, base projects and 01:32.360 --> 01:46.000 30 plus Raspberry Pi designs, and provide access to 130 plus device drivers for seamless 01:46.000 --> 01:47.840 integration. 01:47.840 --> 01:56.080 Also supports application like libio, scopy, I also love scope, which can run directly 01:56.080 --> 02:08.840 on embedded boards, or via wired wireless network, USB or serial port. 02:08.840 --> 02:21.160 Note access is available from Windows, Linux, and Mac host enabling the data gathering and 02:21.160 --> 02:30.520 post-processing using Python, MATLAB, C-sharp and more. 02:30.560 --> 02:41.000 Also is compatible with silings, Intel, Raspberry Pi, and was recently tested on NXP. 02:41.000 --> 02:53.800 Platforms, basically Kuiper simplifies the accessibility for embedded system applications. 02:53.800 --> 03:03.240 Now by the end of this talk, I will want you to get insights into what was the old 03:03.240 --> 03:11.120 really slow of Kuiper Linux in order to have a better understanding of the changes that 03:11.120 --> 03:17.920 were made and how they impacted the performance, which are some of the major improvements 03:17.920 --> 03:31.360 that we needed to integrate in why, and also the current state of our custom Linux distribution. 03:31.360 --> 03:41.480 The traditional process was slow, rigid, and printed delays due to lack of modularity, 03:41.480 --> 03:45.160 and the high-intended dependence between components. 03:45.160 --> 03:52.440 All the software components were built during the image creation, which meant a longer 03:52.440 --> 04:00.840 build time, and another reason for the long release cycle was that the software components 04:00.840 --> 04:09.520 were released through Kuiper image, and with no modularity, a bug found in one of those components 04:09.520 --> 04:13.920 would delay all of them in the end. 04:13.920 --> 04:21.120 Starting from this, we identified the necessary improvements that needed to be done in order 04:21.120 --> 04:27.600 to bring Kuiper to the next level. 04:27.600 --> 04:38.080 Now we came up with a few ideas that we then implemented, then we then implemented, sorry, 04:38.080 --> 04:46.360 and the key modification are creating a stabilization branch for each release component, 04:46.360 --> 05:01.000 breaking down the monolith structure to a more modular one, also is better for our software 05:01.000 --> 05:07.920 components, because we can separate them during the release, and each component would 05:07.920 --> 05:10.880 have a separate release. 05:10.880 --> 05:17.640 Also we would create debion packaging for those components, like libraries, user application, 05:17.640 --> 05:24.720 those that I previously mentioned, like Scope, Libion, and other stuff. 05:24.720 --> 05:35.000 Also, a modular configuration for custom images based on the other change invention. 05:35.000 --> 05:42.600 So we expanded the component level testing with a modular custom Docker image, because 05:42.600 --> 05:51.180 now if our image is more concise, we can create Docker images and test those components 05:51.180 --> 05:57.800 directly on them, and after that we will be sure there will be integrated faster and 05:57.800 --> 06:09.040 cheaper, and also improvements were made on the CI site by optimization, like running 06:09.040 --> 06:17.880 on GitHub, introducing PR templates and code owners for a better collaboration. 06:17.880 --> 06:23.680 Now let's start with the improvements. 06:23.680 --> 06:29.800 I will first talk about the stabilization branch that we introduced. 06:29.800 --> 06:36.000 This was added in order to isolate the release development. 06:36.000 --> 06:42.240 This approach allows us to focus more on preparing the release, while the main branch 06:42.240 --> 06:48.120 continues to accommodate new features and updates. 06:48.160 --> 06:58.200 The result, faster, usual resolution, more focused testing, and a stable release pipeline. 06:58.200 --> 07:06.160 Based on the diagram below, you can see that the master branch serves as the main branch 07:06.160 --> 07:08.880 development. 07:08.880 --> 07:18.560 This branch, the features and the fixes are initially introduced in master, and also 07:18.560 --> 07:25.400 in the next table, which is just a stabilization branch, that is forked and prepared for 07:25.400 --> 07:29.440 the release branch. 07:29.440 --> 07:39.880 After the next table branch is created by branching for master, this branch is dedicated 07:39.880 --> 07:49.280 just for stabilization, ensuring that the code is closely tested and prepared for release. 07:49.280 --> 07:55.920 Fixing updates are often pushed, I mean, are pushed both on the main master branch and 07:55.920 --> 07:59.920 also next table. 07:59.920 --> 08:07.520 Once the next table branch is considered ready for release, release branch or release tag 08:07.520 --> 08:10.080 is created from it. 08:10.080 --> 08:21.400 But during testing, during testing, we find bugs, and that's why we need mostly all the 08:21.400 --> 08:24.600 time to create release patches. 08:24.600 --> 08:34.920 That's why new commits with fixes are uploaded on the release branch. 08:34.920 --> 08:42.360 After release is finished, as you can see, the next stable branch is overwritten with the 08:42.360 --> 08:49.880 latest changes from master, which includes also the other features added been while, 08:49.880 --> 09:03.320 in my release stabilization, and is prepared for the next release cycle. 09:03.320 --> 09:13.480 Next, we move the way from a monolithic structure for monolithic release process and separate 09:13.480 --> 09:19.480 separate the entire release into smaller independent components. 09:19.480 --> 09:27.960 In the past, everything was bundled together and a bug found in one of the components in one 09:27.960 --> 09:32.720 area could delay the entire release. 09:32.720 --> 09:41.960 Now components like software tools, libraries, and boot files are managed separately independently. 09:41.960 --> 09:49.440 By breaking down their release this way, we've eliminated unnecessary interdependence, 09:49.440 --> 09:52.960 between components. 09:52.960 --> 10:00.320 Each component can be developed, tested, and released on its own schedule, leading to 10:00.320 --> 10:06.200 faster iteration and reducing risk. 10:06.200 --> 10:12.440 Each software component has a deviant package created for it. 10:12.440 --> 10:25.880 It will be uploaded on our Linux repository, and this way, we've simplified the distribution 10:25.880 --> 10:38.120 and made it easier for users to install and update their components and utilities like 10:38.120 --> 10:39.960 libraries. 10:39.960 --> 10:50.840 We're all back to a previous version, if that's one doesn't work with the current state 10:50.840 --> 10:54.720 of the cipher. 10:54.720 --> 11:02.200 But one of the most significant steps that we've taken to increase flexibility is the 11:02.200 --> 11:09.320 introduction of a modular and customization of a modular file. 11:09.320 --> 11:17.160 This files allows users to build custom images by selecting the version and the components 11:17.160 --> 11:21.000 that they want the image to include. 11:21.000 --> 11:27.800 For instance, if you're working on a project that doesn't need scopey, it needs just 11:27.800 --> 11:36.000 libio, you can specify the configuration file to install just libio, not the other components. 11:36.960 --> 11:43.200 The nice thing is that you can also specify the version of the component that you want to 11:43.200 --> 11:45.880 be installed. 11:45.880 --> 11:54.920 This approach not only makes the release process more modular, but also gives users the freedom 11:54.920 --> 12:03.120 to create their images how they like, ensuring that no unnecessary components are included 12:03.120 --> 12:05.120 in the image. 12:05.120 --> 12:18.880 Not just regular users, benefit from this, also internal teams because this internal 12:18.880 --> 12:28.880 image is customizable and they can integrate their product more faster because we added 12:28.960 --> 12:39.120 an extra script and this way new components or additional dependencies can be easily added. 12:39.120 --> 12:51.280 There's a stage in the image creation where you can install everything you want at extra 12:51.280 --> 12:56.400 libraries or dependencies or run your own best script. 12:56.400 --> 13:07.920 Let's say that install something is external not only analog devices components. 13:07.920 --> 13:16.800 Now on testing site to improve reliability and compatibility, we've implemented a robust testing 13:16.880 --> 13:21.520 process that works at two levels. 13:21.520 --> 13:28.800 First, each component is tested individually to ensure its stand-alone functionality. 13:29.840 --> 13:37.360 This allows us to quickly catch and address issues specific to that component. 13:38.160 --> 13:44.720 Second, the components are tested as part of the pool-chiper image. 13:45.600 --> 13:52.720 But with the twist, because we've introduced Docker lighter images that are built specific 13:52.720 --> 13:54.640 for that software component. 13:54.640 --> 14:03.440 This way, this Docker images replicate the chiper environment during the development process. 14:04.160 --> 14:13.280 By testing within this controlled environment, we ensure that the software components are compatible 14:13.360 --> 14:18.640 with chiper during development cycle. 14:20.480 --> 14:27.600 This approach allows us to catch integration issues faster and 14:27.600 --> 14:32.800 rather than later stages, which saves significant time and effort. 14:34.240 --> 14:39.280 Now, if you're realized that you've seen familiar with chiper, 14:39.280 --> 14:43.440 maybe you've participated on my colleagues presentation from today 14:43.440 --> 14:48.160 about advanced test harness infrastructure for validating 14:48.160 --> 14:52.320 arm and FPGA base systems, which is here. 14:53.840 --> 14:59.760 He also presented how chiper is integrated in this testing process and how it helps. 15:00.080 --> 15:13.040 Now, to optimize our CI pipeline and make the development process more accessible and efficient, 15:13.600 --> 15:16.560 we transition from Jenkins to GitHub actions. 15:18.000 --> 15:20.400 This move brought several benefits. 15:21.280 --> 15:25.040 First, bid logs are now available to the users. 15:25.760 --> 15:30.160 Increasing this by transparency. 15:31.120 --> 15:37.520 Additionally, fresh bills are available to customers and users 15:38.160 --> 15:46.080 after a push is made on the chiper repo, and they can choose and download those images after 15:47.040 --> 15:54.880 like, I don't know, two hours a commit was pushed on the full image one. 15:56.160 --> 16:11.600 But also, we've expanded the build configuration, because previously, just a 32-bit architecture was supported, 16:11.760 --> 16:17.520 and now, we have 32 and 42 bid architectures. 16:17.520 --> 16:25.520 And for this two-type, we have also a basic image, which includes just, 16:27.200 --> 16:31.360 which it doesn't include decks of the environment, it's just the basics, 16:31.360 --> 16:37.040 but we also have the full image that includes all the software components and stuff. 16:37.440 --> 16:45.760 For this, to further improve collaboration, we added also pull request templates. 16:46.320 --> 16:53.120 Because it's an open source project, you can go search through the code and see a bug, 16:53.120 --> 16:57.920 you can actually create a PR in order to fix that bug. 16:58.880 --> 17:02.960 And to be sure that we know from star what we have there, 17:03.920 --> 17:13.840 you need to check a pull request template, and also to be sure the right person's look at that PR, 17:13.840 --> 17:19.040 we also added code owners to streamline the reviews and approvals. 17:21.920 --> 17:31.600 This changes like the build process faster, more flexible, and more user friendly, 17:31.600 --> 17:36.880 while also encouraging better collaboration among developers. 17:42.480 --> 17:47.680 Now, let's take a closer look at the current state of Kuiper 17:48.640 --> 17:51.120 of our and our release process. 17:53.280 --> 17:55.200 By using a configuration file, 17:56.160 --> 18:03.120 users can now custom and create their image mapped to their specific requirements. 18:04.160 --> 18:12.000 They can select only the components they need, and ensure, and this ensures lightweight and efficient builds. 18:13.920 --> 18:20.880 We've shifted from building each component independently to, we've shifted, yeah, 18:21.120 --> 18:25.120 and releasing as a deviant package, not through Kuiper itself. 18:26.240 --> 18:30.960 This packages are then uploaded to our Linux packaging system, 18:32.080 --> 18:35.600 which allows for faster and flexible builds. 18:36.720 --> 18:45.760 This changes also simplifies updates for users, and it makes easier to install and roll back 18:47.680 --> 18:50.400 to the version of the component that you need. 18:51.840 --> 18:56.240 By dividing the release cycle of individual components, 18:56.800 --> 19:01.840 we've insured that bugs that are found in ones of their components, 19:01.840 --> 19:09.440 one delay and affect the whole release process for the entire image. 19:13.120 --> 19:15.840 This approach simplifies the overall process, 19:16.560 --> 19:22.080 minimizes bottlenecks, and increases development efficiency. 19:24.080 --> 19:31.600 So, this is the end of my presentation, and if you have any questions, 19:32.160 --> 19:42.000 you can write me on this email address, or you can ask me here, or link to any questions, 19:42.000 --> 19:45.600 as come alongers, yeah, that's it. 19:54.400 --> 19:55.040 Yeah, sure. 19:56.640 --> 20:06.480 So my question is now, what you've described is that we moved from a whole distribution release, 20:06.560 --> 20:09.760 we're rolling release cycle for this script, 20:09.760 --> 20:13.280 because we're using individual components individually, 20:13.280 --> 20:19.200 and for how is it decided which components in which question go into the whole release them? 20:20.560 --> 20:27.680 Yeah, well previously, we released Kuiper, all of the components were installed. 20:27.680 --> 20:35.360 So, if a bug, as I said, is found in one component, then we couldn't release all of them, 20:35.360 --> 20:41.440 because we needed to fix that, but now, as we release the components separately, 20:43.440 --> 20:51.120 in the end, we can release the Kuiper image itself, and if one component is not compatible, 20:51.120 --> 20:58.640 it will be released after that, but we can release with later version of that component. 20:59.520 --> 21:08.240 So, yeah, it's more modularized now, it's not a whole, as it was before. 21:14.240 --> 21:23.120 No. No, you can have Kuiper as a whole, but also individually components. 21:29.520 --> 21:32.720 Yeah, we have sometimes them. 21:34.720 --> 21:39.440 What kind of framework do you use in the previous version of the new version, 21:39.440 --> 21:44.560 whether all components, components, or did you use anything in the resource store? 21:46.560 --> 21:53.520 Well, most of the tools are open source, because we integrated analog devices, 21:54.400 --> 22:05.280 software tools, and basically, Kuiper is created to integrate easily the analog devices tools, 22:05.280 --> 22:12.880 and we work open source. So, this is based on Debian, yes, and you still use the Debian option 22:12.880 --> 22:16.560 records for the components that you are not interested in changing the new. 22:17.520 --> 22:25.920 Well, we use a lighter version of Debian, and we install just what we need in order to make our products. 22:33.760 --> 22:35.360 Yes, yes, yes. 22:46.560 --> 22:54.640 Then, thanks a lot. Thank you.