WEBVTT 00:00.000 --> 00:17.920 Okay, so I'm going to start off with your session. 00:17.920 --> 00:27.120 So Stefan and myself, we're going to be talking about the issues of maintaining an Android 00:27.120 --> 00:36.320 release and how you interact your changes with the upstream AUSP. 00:36.320 --> 00:42.720 So it's called forking, Android considered harmful, and as you can guess the basic idea behind 00:42.720 --> 00:50.480 this is that we should try to avoid forking wherever possible. 00:50.480 --> 00:59.840 So the agenda then, Stefan is going to talk a little bit about the different ways that 00:59.840 --> 01:07.240 you can cope with a changing upstream as a changing AUSP and how you can avoid or how you 01:07.240 --> 01:13.440 typically interact with that and different ways of interacting with that. 01:13.440 --> 01:18.720 And then I'm going to come in and talk about why forking is a bad idea and some ideas 01:18.720 --> 01:31.520 I'd like to discuss both in this session and also offline about what we can do about this. 01:31.520 --> 01:36.720 So quick and true to myself, I think I'm reasonably well known. 01:36.720 --> 01:42.040 I've been doing this stuff for quite a long time. 01:42.040 --> 01:43.280 Main topic is of interest. 01:43.280 --> 01:47.240 The point where software meets hardware, I've always been interested in the interface 01:47.240 --> 01:52.480 with devices and I like to see actual hardware if possible. 01:52.480 --> 01:58.760 I've experienced with Linux and its kernel, Yopto and Android. 01:58.760 --> 02:04.360 I'm very interested in open source and I'm very interested in the idea of communities. 02:04.360 --> 02:06.160 So that's me. 02:06.160 --> 02:09.400 Thank you, hand over to Stefan. 02:09.400 --> 02:11.680 Hi, I'm Steve M. Lankfeld. 02:11.680 --> 02:13.400 I'm living in Germany. 02:13.480 --> 02:15.080 I'm working for Innovax. 02:15.080 --> 02:20.600 We are doing projects and consultancy and I'm doing this and that stuff, Linux and that and what 02:20.600 --> 02:25.200 embedded stuff since 10 years now and yeah, main interest and that it's a system, 02:25.200 --> 02:29.960 things can't be developed and built system, perfect stack and performance analysis set 02:29.960 --> 02:35.040 for myself and the short introduction to the problem is again you. 02:35.040 --> 02:36.040 Sorry, good. 02:36.040 --> 02:39.760 Next slide is slide. 02:39.800 --> 02:43.840 Okay, so yeah, there's P developers, Dalema. 02:47.040 --> 02:50.480 So the first point is the basic one. 02:50.480 --> 02:58.480 So in order to make a product out of an AWS P, you basically need to take a copy of 02:58.480 --> 03:02.320 a USB and then modify it in some way. 03:02.320 --> 03:04.640 So what do we need to do this? 03:04.640 --> 03:09.720 So typically you're adding in board support for whichever hardware platform you're using. 03:10.360 --> 03:19.160 Often you need to put in some custom how support we need to add in services, libraries, 03:19.160 --> 03:29.600 apps and we may also want to change the upstream ASP code, we may want to change the 03:29.600 --> 03:34.760 way the framework works in some ways because we need some different launchary behaviors. 03:34.760 --> 03:38.360 We will of course need to add in some SC policy on all kinds of stuff. 03:38.360 --> 03:45.480 So in order to do anything with AUSP apart from just build an emulator, you need to hike 03:45.480 --> 03:47.320 it around a bit. 03:47.320 --> 03:51.320 So how do you handle that? 03:51.320 --> 03:57.280 And that we go into the section and how we are handling with that. 03:57.280 --> 04:01.600 They are typically so we have to look at two different things. 04:01.600 --> 04:08.000 So one thing as at the one point often have to fork something or at least to patch some 04:08.000 --> 04:09.000 existing code. 04:09.000 --> 04:15.680 So patching framework code for example to fix a bug or back for the patch for a fix or adding 04:15.680 --> 04:19.440 new features in framework but of course this should not be done too much because 04:19.440 --> 04:21.880 and plotting is harder to do versions. 04:21.880 --> 04:27.760 So one thing we always do is fork or patch existing code from the ASP source tree or from 04:27.760 --> 04:32.560 the window tree you get from the SOC window and the better thing is the additions. 04:32.560 --> 04:38.240 So a lot of things we add to the ASP tree for example, how implementations, kind of 04:38.240 --> 04:46.040 drivers, additional apps and other things and both forking and additional patching and 04:46.040 --> 04:48.040 adding code. 04:48.040 --> 04:52.680 I do different tasks that we need to do as ASP developers and in the options I will now 04:52.680 --> 04:55.600 present also done differently. 04:55.600 --> 04:59.160 So we will go to the options that I have seen in the white. 04:59.400 --> 05:00.160 One question? 05:00.160 --> 05:10.440 Okay, no speakers, this mic is only for the so I have to speak louder, you are saying 05:10.440 --> 05:17.440 I should have been louder, okay, okay I tried to speak louder, so now we look at the 05:17.440 --> 05:24.640 four different methods I have seen in the ASP in the community or in projects, how 05:24.640 --> 05:29.280 we deal with that, so we look at the good, the bad and the ugly nothings. 05:29.280 --> 05:36.440 We use at the copy script option I have seen which we are using patches, we are looking 05:36.440 --> 05:46.440 at the local manifest feature from the repo tool and we are looking at the way with copying 05:46.440 --> 05:51.080 and extending the upstream repository repo manifest XML manifest. 05:51.080 --> 05:55.240 That does it for options, we are now going through. 05:55.240 --> 06:00.600 So using a copy script, so we have the upstream for example the upstream ASP source 06:00.600 --> 06:05.840 tree from Google or from a BSP vendor and you are the downstream, you are the downstream 06:05.840 --> 06:11.760 project and in with this option what you are doing is you have kind of in the bad case 06:11.760 --> 06:15.800 you have a separate archive with code, a customer has a separate archive with code or you 06:15.800 --> 06:20.880 have a git repository and in this git repository you have files from the 06:20.880 --> 06:27.760 ASP copied and modified, this is maintained by you, this git repository is maintained 06:27.760 --> 06:33.480 by you and then you are white, a weak me file or you have a weak me file where you specify 06:33.480 --> 06:40.960 the upstream manifest version from Google or from your BSP vendor and the version, so someone 06:40.960 --> 06:47.240 who is working on the code then knows on which version you have based on the copied files 06:47.240 --> 06:53.040 and modified, and then to be very convenient you have a copy script, so you place also a 06:53.040 --> 06:58.960 script into the git repository to then copy the patch files over to the actual ASP source 06:58.960 --> 07:04.640 tree where it was actually built, and how does it check out local like, so you first do 07:04.640 --> 07:09.920 your repo in it from the upstream, except all manifest from Google for example, then you 07:09.920 --> 07:15.840 do a repo sync to download on all the code, then you click git clone the downstream repository 07:15.840 --> 07:20.680 so you code or the code of your customer or your downstream project and then you execute 07:20.680 --> 07:26.920 this copy script, and with this way then you have a patched MSP source tree for your device 07:26.920 --> 07:35.640 or for your customer, and just I will not go much into the pull of cons, of all these 07:35.640 --> 07:41.760 options but I personally don't like this option, but I've seen it, for example, I've seen 07:41.760 --> 07:46.800 it in one upstream git repository here and I've seen it by customers, the problem is it's 07:46.800 --> 07:53.520 very hard to sync the files, the modified files in the ASP source tree and the modified files 07:53.520 --> 07:58.720 in the downstream git repository, it's very hard to track these changes, but I've seen it 07:58.720 --> 08:06.640 and in simple case it works, what also I have seen in the wide is patches on top, 08:06.640 --> 08:12.480 I call it patches on top, how does a downstream project does it, it maintains a git repository 08:12.480 --> 08:19.600 of patches, so not modified files, but patches get patches in the other descriptions, 08:19.600 --> 08:25.600 10 patches, 100 patches, then it also writes a written file to specify the upstream, manifest 08:25.600 --> 08:30.600 an upstream version from Google or the BSP vendor, and also it has a convenient script to 08:30.600 --> 08:38.200 apply the patches on the ASP source tree, how does it look like again, we put in it from 08:38.200 --> 08:44.280 the upstream, it's a manifest, we put sync, you get cloned the downstream project, you 08:44.280 --> 08:48.600 change the directory in the downstream project, and then you execute the convenient script 08:48.600 --> 08:53.320 to apply all the patches, and then the patches are applied to ASP source tree and then you 08:53.400 --> 09:03.400 can build and you are your device enabled, and I've also seen some examples that 09:04.360 --> 09:11.160 in the reload project, in the Pinephone project, and also Chris has a small project, this also 09:11.160 --> 09:21.000 uses patches, so let's before we go to the start option, we look at repo, manifest trick 09:21.000 --> 09:27.400 now that you may know or may not know, it's the removed project trick, this is repo, manifest, 09:28.680 --> 09:36.280 manifest for a small project, you see these lines are adding git repository, so here we are 09:36.280 --> 09:42.360 talking about the addition, but below here we see the, we move project line, that is about 09:42.360 --> 09:47.640 forking, so how we can remove every repository from the upstream and replace it with our own fork, 09:47.640 --> 09:54.520 and this works with the remove project, it's an alt tag, you write down the name or the path 09:54.520 --> 10:02.920 of the repository, then it's been moved, and then since always only adds a new fork of it, 10:02.920 --> 10:08.520 you see that it's a fork because the remote is pointing to cool project, it's not pointing 10:08.520 --> 10:17.240 to the upstream anymore, and cool project is our host on GitHub, so with this, you can do the 10:17.240 --> 10:23.720 option 3, it's the local manifest, the local manifest is a feature by the repo 2, how there's 10:23.720 --> 10:28.600 a downstream project use it, it would mean times a git repository, we'll see local manifest 10:28.600 --> 10:38.040 X and O, so for example, there's some git repository's added, some removed, this X and O 10:38.040 --> 10:44.120 one is in this git repository, again I read me file to specify the versions because you 10:44.200 --> 10:50.200 develop and needs to know that, and then it also adds and forks some repositories of the AOSP, 10:50.200 --> 10:55.240 and plays it on GitHub or somewhere, how does he check out, look like it's again a bit, it's now a 10:55.240 --> 11:01.800 bit shorter, we put in it from the upstream, then we do git clone from the downstream and project, 11:01.800 --> 11:08.600 and the important point is that we clone into the path dot repo slash local manifest, this is 11:08.600 --> 11:15.560 the repo, look in the manifest feature, if X and O files are in this folder, then it's picked up by 11:15.560 --> 11:21.400 repo, and then we do a repo sync and the whole source tree from the upstream source, and now also 11:21.400 --> 11:30.440 from the downstream source, and here also some examples, I've seen from one Raspberry Pi and Android 11:31.160 --> 11:38.840 and also here one from Snap Remote of. Now we go to the force option, but before we do the 11:38.840 --> 11:47.400 force option, another repo trick you might know of it's include, it's a small trick to make it 11:47.400 --> 11:58.920 more easy to maintain a copied or maintain your manifest, the include trick you see here is that 11:59.000 --> 12:05.240 if you have an XML manifest, you can specify the include tag, and if it's included, another text file 12:05.240 --> 12:14.360 is included in this manifest, for example here I copied the upstream manifest from Google versions 12:15.320 --> 12:26.040 13, and then my own additions, my own forks new projects I added in another X and O file, and with this, 12:26.040 --> 12:34.600 you can easily maintain a downstream fork and yeah, and with this we come to the force option 12:34.600 --> 12:42.440 if the option copy and extend the upstream manifest, so now you as a downstream project, you maintain 12:42.440 --> 12:49.320 a own manifest repository, you are not referring to a request, a manifest repository from Google 12:49.320 --> 12:56.680 or the BSP vendor, in that you copy and include the upstream manifest from Google, and you also 12:56.680 --> 13:02.360 again add forks, you add and add and forks some repositories. How to check out it's now where we 13:02.360 --> 13:08.440 easy repo in it from your downstream manifest repository repo sync, and that's it, and you have 13:08.520 --> 13:14.760 Cloncy source code, and examples, the developer does it, project, sell it on does it, and 13:15.960 --> 13:22.120 I cannot spell it, collects OS does it too, and this is the force options, how you maintain your 13:22.120 --> 13:28.040 downstream fork of Android, and why it, and with that I hand over to Chris to talk about 13:28.120 --> 13:40.200 why forking is bad. Okay, so all of the examples are seen really, are in some way rather 13:40.200 --> 13:50.440 forking the upstream project, so yeah, why is this a bad thing, so you may say well it's open source, 13:50.440 --> 13:53.880 we can do whatever we like with it, we can hack this around whatever. 13:54.120 --> 14:05.800 However, forking any project is always about idea, it has a maintenance burden, it's the main problem, 14:05.800 --> 14:15.400 so once you made the fork, you are now responsible for that code for applying security fixes, 14:15.400 --> 14:21.880 bug fixes, whatever else, so you have taken on the responsibility for maintaining this code. 14:23.880 --> 14:37.320 All you can do, you can, instead of backwarding stuff at here into your fork, you can continually 14:37.320 --> 14:44.360 re-base your fork on the upstream version, but that also has some overhead, because when you do the 14:44.360 --> 14:47.400 re-base, you're better have some conflicts and you can have to resolve them and so on. 14:47.720 --> 14:56.280 Next point is something slightly more subtle, I think, when you make a fork and you start 14:56.280 --> 15:05.400 making changes to the code base, those changes tend not to be well separated, so developers often 15:05.400 --> 15:10.520 don't differentiate too much why they're making the changes and they'll be changes to get 15:10.520 --> 15:15.560 commits, but they don't necessarily relate to each other. So you end up with a bit of a mismatch 15:15.560 --> 15:23.080 of stuff, so here's our ESP, here's our fork, we change some stuff, you can look in the 15:23.080 --> 15:26.920 Git logs to find out exactly what we change, but it's not really that obvious. 15:29.480 --> 15:33.800 As a result of all this, we typically end up when you're writing, when you're creating 15:34.360 --> 15:43.640 your own Android device, with a barrier which prevents you migrating to a new version. 15:43.640 --> 15:48.360 So it's quite common to find Android devices running old versions of Android, for example, 15:48.360 --> 15:54.200 Android 9 and running old versions of kernels and, yeah, it's kind of a nightmare. 15:54.840 --> 16:01.640 So to summarize, forking equals technical debt, we would like not to fork if we can avoid it. 16:04.760 --> 16:06.920 So, how do you do this? 16:09.720 --> 16:16.120 Fundamental rule then is don't change a USB code base, including do not change the manifest. 16:17.960 --> 16:24.760 What can you do? Well, you can add stuff, so you can add stuff into the local manifest, 16:25.800 --> 16:31.720 and this is the big thing here really is to enhance patching. So instead of 16:32.200 --> 16:39.320 changing the code in situ, create a series of patches and apply those patches at built-in. 16:41.160 --> 16:49.080 Additional benefit, if you do this in a clever way, you can organize these patches into groups, 16:49.080 --> 16:54.040 which are going to call layers, which can be applied independently of each other, and you have 16:54.040 --> 17:00.520 a much more modular system. My inspiration for this really came from a number of places, 17:00.600 --> 17:04.760 I'm going to document two here, I'm going to mention first of all, Glodroid. 17:06.280 --> 17:12.840 So Glodroid, as we mentioned, is an ASP distro for various boards, including Raspberry Pi's. 17:14.520 --> 17:21.320 And it's interesting if you look at Glodroid 2.0, they change the way that they are interacting 17:21.320 --> 17:26.440 with the upstream ASP, and they're using what they call a mono repository approach, 17:27.080 --> 17:33.720 which is more or less what I just said. This is a little small, and I don't expect you to read 17:33.720 --> 17:40.840 all of it, only one go. The key thing is, no more forking. Okay, so there are ideas, 17:40.840 --> 17:47.640 what I just said in a couple of slides back, instead we will take ASP, we will create a set of patches, 17:48.280 --> 17:53.480 and then we have some build-time tools which will apply those patches to the ASP. This makes it 17:53.560 --> 17:58.840 much more easy to maintain those patches. Also, notice this comment here, it says, 17:58.840 --> 18:04.040 decouple devices or device groups from each other. In other words, we group those patches together 18:04.040 --> 18:14.920 into different functional areas, one for the changes to the ASP code base, some are device specific. 18:15.000 --> 18:24.440 So, that's my first inspiration. My second source of inspiration is from York to Project. 18:25.800 --> 18:32.840 How many people here are familiar with York 2? Okay, about half. Good, right. 18:33.640 --> 18:43.000 So, York 2 is better than Android, yes? I think that was a yes, I definitely heard a yes there. 18:45.400 --> 18:52.200 Okay, so York 2 essentially does the same thing as the ASP build system, 18:52.200 --> 18:59.480 except it does it better in my opinion. Just maybe in a few concepts then, for those of you 18:59.480 --> 19:06.680 are not familiar with it, maybe in York 2 concepts on to ASP concepts. So, in York 2, we have 19:06.680 --> 19:14.680 everything called BitBake, which does the same job as soon. In York 2, we have recipes which are 19:14.680 --> 19:22.120 BB files, bitbake files. They are more or less exactly the same in concept as Android BB files and 19:22.120 --> 19:31.000 make files. And I'm not going to go into this in too much more detail, but York 2 also has classes, 19:31.320 --> 19:37.240 bitbake is written in Python, the classes are basically bits of Python code. 19:38.520 --> 19:45.800 That's where the real logic is implemented in the ASP, the main build logic is implemented in 19:45.800 --> 19:58.280 the build their entries in build make and also in build soon. So, we have this metadata, which 19:58.360 --> 20:07.560 are recipes, classes, and config files. And the key concept I want to borrow from York 2 here 20:07.560 --> 20:15.480 is the idea of a layer. So, we have the called meta layers. So, a meta layer is just a collection 20:15.480 --> 20:24.040 of metadata, recipes, classes and so on. And the key thing here is there are several different types 20:24.120 --> 20:32.040 of layer. So, we have BSP layers. For example, Raspberry Pi would be a good example. We have 20:32.040 --> 20:37.800 distros, distros is a slightly complicated thing which I don't want to go into now, but for example, 20:37.800 --> 20:44.760 AGL, automatic grade Linux is A distro. So, if you want to build an AGL version of York 2, 20:44.760 --> 20:52.040 you just include that layer. And it has what they call software. So, these are typically libraries. 20:52.040 --> 21:01.080 So, meta, Qt would be a good example. Qt is the graphics library. And then when you're building 21:01.080 --> 21:05.640 a product, you just patched together the layers you want. You add in the board support, whichever 21:05.640 --> 21:10.760 board you want. You add in the distro for whatever you want it to do basically. And then you add 21:10.760 --> 21:20.520 in software libraries. Kind of neat. There is an official list of layers, the layer index. 21:21.160 --> 21:25.400 And if you go look there, you'll see there's four or five hundred layers you have to choose from. 21:26.280 --> 21:32.600 So, you can build things together quite neatly. So, why can't we do something similar with Android? 21:35.880 --> 21:43.640 Okay, so this is, okay, I'll misspell that as well. This is a very early stage, but I kind of want 21:43.640 --> 21:52.120 to get to feedback either now or later on. So, how do we apply layers to Android in two minutes? 21:54.120 --> 21:59.640 Essentially, the ASP layer basically is a collection of local manifests, of patches, 21:59.640 --> 22:06.200 and also some metadata explaining what the layer does. And also documenting incompatibilities, 22:06.280 --> 22:09.560 so which version of Android does it work with, that kind of thing. 22:13.720 --> 22:20.360 We do my things up into layers. We can then combine these layers in different ways to add in 22:20.360 --> 22:26.040 what to make whatever we want. So, we can have a board support layer, we can have some house, 22:26.040 --> 22:33.560 we can have build system changes, and so on. And so, we don't want my wonderful diet. 22:34.520 --> 22:38.760 Anybody who see my lectures will know my diagrams are a rubbish, and this is a good example of that. 22:39.640 --> 22:45.800 But the basic idea is that essentially when you import the layers that you want, 22:45.800 --> 22:52.680 so this will be in a directory somewhere on your system. Here I have three layers. I have my BSP, 22:52.680 --> 22:58.920 which is the hardware support. I have a Genog Audio, how and I've got a launcher. And I've 22:58.920 --> 23:05.560 expanded out what you might see under one of these. So, for the audio how, we have some changes 23:05.560 --> 23:11.160 to the hardware directory, a bunch of patches. We have, of course, some SC policy, because you 23:11.160 --> 23:18.120 would need SC policy, a bunch of patches. So, you download the layers you want. You then apply 23:18.120 --> 23:25.320 some magic command to apply the patch, that patches AUSP in a control way, and you build a product. 23:26.040 --> 23:34.680 So, this brings me onto another concept, really, which is a product. So, a product is a collection 23:34.680 --> 23:43.640 of layers, a layers of magic metadata. The product then is a single entity, most likely a file, 23:43.640 --> 23:50.920 possibly in JSON format or something. So, I can then say, go get me the product description from 23:51.000 --> 24:01.080 some URL. It will grab the product description. It will download everything, apply the patch 24:01.080 --> 24:13.480 here and away we go. So, to make this work, we need some tooling, TBD. And just as one final thing, 24:13.480 --> 24:18.680 then the idea is that if I would say, I want to get a product. I want to build, I don't know, 24:18.760 --> 24:24.760 an automotive Android for a Raspberry Pi. Somebody somewhere has written the JSON file, which 24:24.760 --> 24:32.440 explains how to do that. So, I just say, build Android product, point to that JSON file. It will then 24:32.440 --> 24:37.480 read the pilot file, do a repo in it for me to get whichever version of Android, it specifies. 24:38.760 --> 24:46.360 Download the local manifest into the directory, do some checks, do a repo sink in order to bring in 24:46.440 --> 24:53.480 all of the local manifest changes, apply the patches, and then you type M and go away for a long 24:53.480 --> 25:02.520 holiday while it builds it for you. So, that's the basic idea. As I said, this is very much 25:02.520 --> 25:10.200 a work in progress. I would very much value feedback from everybody here. Do you have time for 25:10.200 --> 25:17.480 like maybe one question? Yeah, I'm in charge. So, we have time for one question. 25:22.840 --> 25:32.280 Anybody, I totally overwhelmed. Okay, this is a good reason then to do the wrap up now. 25:32.280 --> 25:38.440 Thank you very much, and let me know what you think.