WEBVTT 00:00.000 --> 00:19.440 All right, so let's start good morning everyone, this and like me, my name is Harin, I'm 00:19.440 --> 00:26.760 part of my school engineering team, my main focus is working on security features of my 00:26.760 --> 00:32.240 school server and live my school client and today I'm going to talk about how you can 00:32.240 --> 00:38.840 extend my school servers functionality by implementing components. I'll show you small demos 00:38.840 --> 00:45.760 about some toy components that I have written which is available on GitHub which helps you 00:45.760 --> 00:53.960 check the password whether it appears at any database breach or not. So with that, let's start. 00:53.960 --> 00:59.960 I'll just skip over. So small agenda, the overview of my 00:59.960 --> 01:07.640 skill component architecture followed by two specific use cases, the password breach 01:07.640 --> 01:12.480 check component which utilizes an already existing service that my 01:12.480 --> 01:18.640 skill provides and then one more refinement on how you can achieve component 01:18.640 --> 01:23.760 changing or broadcasting a service called to multiple components and some small 01:23.760 --> 01:29.440 library to help you back. We'll be with that. At the end, we'll see a couple of points 01:29.440 --> 01:33.280 which may help you write your own component as well. 01:33.280 --> 01:43.960 Okay, so my skill component infrastructure is actually inspired by service oriented architecture. 01:43.960 --> 01:48.520 It has a registry which allows producer and consumer to talk to each other. The good part 01:48.520 --> 01:54.680 about registry is that it's type agnostic. So it's not tied to a specific type. It can 01:54.680 --> 02:00.320 actually work on any service name and then producer and consumer can talk to each other. 02:00.320 --> 02:09.160 Very easy to reuse modular code. So you know, easy to maintain. One of the core thing 02:09.160 --> 02:13.200 about this infrastructure that is that it has strong dependency tracking. So when you're 02:13.200 --> 02:19.360 loading a component, it makes sure that all the dependent services or all the dependent 02:19.360 --> 02:25.080 implementation that the component uses are also available. If they are not, your component 02:25.080 --> 02:30.360 loading will fail and once the component is loaded and defines all the dependency, those 02:30.360 --> 02:37.200 dependencies are logged in till the time the component is again uninstalled. Also, we 02:37.200 --> 02:42.880 design it in such a way that it's not strictly dependent on my skill server. So in theory, 02:42.880 --> 02:52.640 if any binary implements a small self, self-contained library called live-man chassis, 02:52.640 --> 02:58.000 it can load any component using that and start using the functionality. So it really provides 02:58.080 --> 03:11.560 good useability. There are no versions in components. No versions. No versioning. 03:11.560 --> 03:18.760 So if you, you have a be careful when you're writing your services. If you need one more 03:18.840 --> 03:24.840 service is 2 years later, that got to be a new service, not a version 2 of the same one. 03:24.840 --> 03:37.160 Yeah. All right. Okay. So four major modules within a component architecture services, 03:37.160 --> 03:43.080 which are essentially containing API definition, street of it as header files, which defines 03:43.160 --> 03:50.120 how your API is will look like. Examples are event tracking services, password validation 03:50.120 --> 03:55.640 services, and there are 200 plus services in the server. So you can take a look in source code. 03:56.680 --> 04:00.920 A component is something that implements one or more services. So a component can have 04:00.920 --> 04:06.040 n number of services implemented. It can actually define that I as a component implements 04:06.040 --> 04:13.080 service 1, 2, 3, 4, and then it's essentially a shared library. So it can be loaded by something 04:13.080 --> 04:20.360 called dynamic loader. So in very simple terms dynamic loader will do a deal open, get the content 04:20.360 --> 04:25.320 from the shared library, and then interface it with the registry, which is the final part of 04:25.480 --> 04:33.480 the server. It essentially stores the kind of a map of implementation and actual function reference. 04:33.480 --> 04:39.880 And it utilizes it whenever a consumer of the service comes in and says, hey, I want 04:39.880 --> 04:44.120 service password validation. Okay. Let me go and look. If I find the implementation name 04:44.120 --> 04:48.680 password validation, I'll return the reference along with the reference count incremented. So that 04:50.120 --> 04:54.200 that is the component in question is not an installed while someone is using it. 04:55.320 --> 05:01.240 So this is the, this is basically the heart of component architecture in my scale. Now, 05:04.520 --> 05:09.160 registry and dynamic loader together forms live main chassis. Remember in one of the previous 05:09.160 --> 05:16.040 I said that we want to make it as modular possible. So if you use live main chassis, which is 05:16.040 --> 05:21.560 static library within any binary, you should be here theoretically, you should be able to load 05:21.800 --> 05:28.920 components. That's what mySQLD binary does. It embeds a shared this static library within 05:28.920 --> 05:35.080 itself. And with the help of this static library, it can load multiple types of component. 05:35.080 --> 05:41.560 So different services provided by different shared library is loaded alongside the server using 05:41.560 --> 05:50.520 live main chassis. Not just this, you can also have multiple implementation of same service 05:50.520 --> 05:58.920 loaded in parallel. So think of it as if, if let's say you have more than one implementation 05:58.920 --> 06:06.600 of a connection event tracking, you can actually have all of them loaded. And then you can choose 06:06.680 --> 06:12.120 whether to use one or first or second or third so on and so forth. So like registry provide 06:12.120 --> 06:21.320 that kind of infrastructure. And once this is all done, modules within executable or 06:21.320 --> 06:28.120 in fact other components can use any other component service using the registry. So server doesn't 06:28.120 --> 06:35.640 come into picture. The only thing that server does is to make sure that live main chassis is present 06:35.720 --> 06:41.480 and then every consumer will go to registry, identify the reference at once and directly talk to 06:41.480 --> 06:52.360 the producer of that component implementation. At the moment, in mySQL server, we have now 200 06:52.360 --> 06:57.720 plus services. So any functionality that is extending mySQL, mySQL server, we are using component 06:57.720 --> 07:04.680 architecture now. So we have more than 200 services, you can take a look at mySQL's component 07:04.760 --> 07:11.320 services space. You will find a lot of header files. A single header file can contain multiple 07:12.040 --> 07:18.280 service implementation. So what's out for that? And since we are continuously including new 07:18.280 --> 07:23.320 features, we are also adding to this list every now and then. So last I checked yesterday, 07:23.880 --> 07:31.320 we were somewhere around 240. So yeah, I mean, as and when we add new features, this count will increase. 07:31.400 --> 07:36.600 What this means for you is that all these services or all these hooks are already available in 07:36.600 --> 07:42.680 the server and server already emits this event. All you need is an implementation which listens 07:42.680 --> 07:50.680 to this or implement this services and make something useful out of it. In no particular order, 07:50.680 --> 07:56.040 I just, I'm just listing some of the services which I think are really good, really cool, 07:56.120 --> 08:01.880 provides this very good access. Take a look at few of them like table access. You can now 08:01.880 --> 08:06.840 access any SQL table from component or you can actually execute any SQL command through component. 08:10.440 --> 08:18.920 Right, so that's all about the overview of component infrastructure. Now, once it's there, 08:18.920 --> 08:23.880 let's take a look how you can use it to extend functionality which is not available out of the box. 08:26.360 --> 08:34.440 So for this part, I'm going to take example of password validation service. So server 08:34.440 --> 08:41.640 implement, so provides this service and there are two APIs within this service. The first 08:41.640 --> 08:46.840 API is what is used when you are creating a user or altering an existing user's password. 08:47.400 --> 08:53.240 It goes to this component and checks whether that password satisfies certain criteria or not. 08:53.960 --> 08:59.080 My skills are also provides a default implementation called component-related password and that 08:59.080 --> 09:05.080 that implementation essentially checks. Okay, does the password have N number of characters? Do you 09:05.080 --> 09:12.280 have mixed case characters, special cases, numbers, so on and so forth? It also can support a local 09:12.280 --> 09:17.080 dictionary based search saying that okay, I have a dictionary of weak password. If I see any 09:17.080 --> 09:22.200 password within that dictionary, I will straight away reject it. Apart from this, there is one more 09:22.200 --> 09:27.240 API which essentially provide the information about password strength, so it gives you score out 09:27.240 --> 09:32.840 of 100, 0 being, it's really bad password. You shouldn't be using that and 100 being, okay, 09:32.840 --> 09:38.040 I haven't seen that password anywhere before and it satisfied all the criteria, so yeah, you can 09:38.040 --> 09:45.640 go ahead and use it. This is the part that we are going to focus on today in next couple of slides. 09:45.720 --> 09:55.240 Right, how many of you have heard of have I been born? Great, how many of you have searched 09:55.240 --> 10:02.680 your password on have I been born and found that it's actually listed there? Yeah, good, all right, 10:03.480 --> 10:09.240 so yeah, so okay, so this will help. So this is, so have I been born is one of the largest 10:09.400 --> 10:15.880 database of breached password. It basically captures password which appears in various breaches. 10:15.880 --> 10:20.840 Continuously updated has millions and millions of password, real world password, that's the main 10:20.840 --> 10:28.760 part, these are not made of password, these are real words. So and it provides an API, so you can of 10:28.760 --> 10:33.880 course, you can go to haliband.com and then search it and it will show you yes, your password has been 10:33.960 --> 10:38.920 appeared in so-and-so breached blah, blah, blah, but it also provides an API to search your password 10:38.920 --> 10:44.600 programmatically. So if you provide a shavan prefix, just the first five characters of shavan 10:44.600 --> 10:51.400 hash of your password, it will send a range of password where this prefix appears. It doesn't mean 10:51.400 --> 10:57.400 that your password is necessarily there. It just says that if I see this prefix, here is a list of 10:57.400 --> 11:04.920 password that I have which happened to have this prefix and then you can, you can then compare with 11:04.920 --> 11:09.560 the suffix and see if your password actually appeared and also it returns account, we'll see that 11:09.560 --> 11:17.080 in the next slide. On an average you will have more than 400 password for any shavan five 11:17.080 --> 11:23.720 character prefix that you throw to this API. So it's really safe, it's not like by using shavan 11:23.720 --> 11:30.040 you are exposing your password to potential problems. We are not sending the complete shavan hash, 11:30.040 --> 11:40.840 it's just accepts the first five character. So what I did was it's a small component which I wrote, 11:41.720 --> 11:50.200 it's compatible with the latest mySQL 9.0, but again it's not production code, so play with it, 11:50.200 --> 11:55.640 throw around it, do it what you want, but yeah, be careful if you want to use it in production 11:55.640 --> 12:01.960 scenario. So whenever user supplies a password, in this case I actually chose a password which will 12:01.960 --> 12:07.720 pass mySQL's validate password criteria because it has a location lowercase characters, it has 12:07.720 --> 12:12.840 numbers, it's a special character, more than eight characters, everything. You take this password 12:13.720 --> 12:19.960 supplied to the component and it will search for this password using the prefix and have 12:19.960 --> 12:27.160 I been pawned and then again this being really really bad password, have I been pawned will return 12:27.160 --> 12:34.120 with yeah, I have seen this and again this is not just the only reply we will get we will get 400 12:34.120 --> 12:41.320 plus password prefixes along with account that this password has appeared 1118 time 12:41.480 --> 12:49.080 on all the databases that I have seen so far on password bridges shouldn't really use this password 12:49.080 --> 12:53.880 and based on that the plugin or sorry the component will reply saying okay you can't use this 12:53.880 --> 13:02.120 password even if it satisfied some of the criteria of you know how it should be composed etc. 13:02.440 --> 13:25.160 Right so let's take a look at the demo. So what I have done is this is a mySQL instance and 13:25.160 --> 13:31.160 this basically doesn't have any password validation component installed. So if you create a user 13:31.400 --> 13:38.120 it will allow very lousy password like me cd. Now once you install the component and 13:38.120 --> 13:46.920 validate that yeah it is actually present you can try and create user or alter an existing user 13:46.920 --> 13:55.640 password with weak password something like this right so once you do that it will it will immediately 13:55.720 --> 14:02.680 stop you from using the password e fgh because again it's really really weak. 14:04.760 --> 14:11.640 If you further go and check server's log it will also log information about that this password 14:11.640 --> 14:17.080 has actually appeared so many times it doesn't give you the actual shawan hash it doesn't log 14:17.080 --> 14:23.720 the entire hash but it tells you it okay it has appeared 454 times as per the records of her 14:23.720 --> 14:30.920 I've been pawned. I know that this is not logging user name so it's not you know super helpful 14:30.920 --> 14:40.680 but we are again limited by what's provided by server as of now. Right so let's let's now go back 14:40.680 --> 14:49.080 and let's see if if we can do something about this okay we can't use e fgh can we use a variation 14:49.080 --> 14:57.640 of word password for example um that variation will have special characters it will have sorry 15:05.160 --> 15:14.440 yeah then you sure okay right so what I just did was I used a variation of word password which 15:14.520 --> 15:22.040 replaces a with at the red sign o with zero so on and so forth. Turns out that's even more popular 15:22.040 --> 15:29.880 so that appears for what more than 600000 times so the component will not let you use that. 15:29.960 --> 15:46.280 So I can't I can't put the user name information that okay this password was used by 15:47.080 --> 15:52.600 user herine or her user herine was being created with such a password and this is why I 15:53.240 --> 16:00.840 didn't it because the components API is an yeah the API doesn't take user name as argument it simply takes 16:00.840 --> 16:09.560 of incoming password as argument so ultimately user will get there that password I mean you know 16:09.560 --> 16:15.160 the password operation failed or the create user operation failed and then they can take a look at the 16:15.800 --> 16:20.920 would you can extend the component if you want it's possible yeah but the API changes and it 16:20.920 --> 16:25.480 doesn't version so you have to come with another another name for a component which is almost the same 16:25.480 --> 16:33.000 as the previous one with a V2 or something. No no it's a different service it's a different service 16:33.000 --> 16:37.640 implementation that's okay but there's the end of the result is you you change your adjusting the 16:37.640 --> 16:43.480 API some idea it's a make it. No no I'm not I'm not saying that I'm going to change the component 16:43.560 --> 16:48.920 so we say PIs I'm saying there are other APIs available really to error log which may provide 16:48.920 --> 16:54.440 more information than this so within my component I will implement one more service which will 16:54.440 --> 17:00.520 let me find this information from server and then displayed but again probably you can do that 17:00.520 --> 17:12.840 and extend this component and then contribute to that yeah right so okay moving forward so now 17:13.640 --> 17:19.320 I'll use a password which is not available or which is not present in any of the 17:20.280 --> 17:26.840 breach database of the password and it successfully actually works you can create a user with that 17:26.840 --> 17:32.520 password you can also use that password for an existing user basically you can change existing 17:32.520 --> 17:38.920 user's password it works because go to the API bin porn provide the prefix returns the result 17:38.920 --> 17:44.440 but the suffix doesn't match so yeah so that's that's pretty much it about this component 17:48.120 --> 17:53.320 but that's just the first part now what we have seen is that if you have this component it can 17:53.960 --> 17:59.880 help you check that passwords status online but what if I want to have both 18:00.600 --> 18:06.120 component related password as well as the password breach check working simultaneously so not only 18:06.120 --> 18:12.200 I will check for the weak password online I will also put the constraint of how your password should 18:12.200 --> 18:19.960 be you know created should it have certain length criteria character criteria zone and so forth 18:21.000 --> 18:27.480 so so for that what you need is you have one service you have multiple implementation and each 18:27.480 --> 18:33.240 implementation gives you different different functionality or different use case and then 18:33.800 --> 18:40.600 you need to somehow broadcast this call to all this implementation and one by one get the result back 18:40.600 --> 18:45.640 whenever you encounter a failure stop it requires something called an anchor component 18:46.520 --> 18:50.680 not an official name I mean I made it up so my call it anchor component because 18:51.720 --> 18:57.560 it is aware that there can be multiple implementation of the service that I provide I should use 18:58.520 --> 19:04.680 some mechanism to broadcast this call to all other components get the result back aggregate them 19:04.680 --> 19:11.400 and see whether I should reply success of failure to my skill server in turn it uses something called 19:11.400 --> 19:16.200 a reference caching service this is another service that exists right now within my skill 19:16.200 --> 19:25.880 component infrastructure right so what I did was I write a small static library just a couple 19:25.880 --> 19:31.880 of files it it basically is a type agnostic library that lets you broadcast your service called 19:31.880 --> 19:37.400 to multiple component implementation yes very simple API you initialize the initialize it 19:37.400 --> 19:43.000 and then there is a broadcast call which lets you it will find all the implementation of a given 19:43.000 --> 19:48.760 service type and then return the handled back to the caller as a call back and then the caller can 19:48.760 --> 19:55.400 decide what to do so if you see here this is what the caller will do this is my call back 19:55.400 --> 20:03.800 in in password breach component that I have return so with each for each service it will get a 20:03.800 --> 20:10.680 call back it can call the validate which is the password service and if this service returns error 20:10.680 --> 20:24.120 it will return an error we will see this in action soon all right so this is the same demo 20:24.200 --> 20:31.640 we reached to the stage where both this users were able to use a password which didn't appear 20:31.640 --> 20:40.360 in have I been points later base now what we are going to do is we already have password breach 20:40.360 --> 20:46.040 check component we are now also going to install my skill validate password component which 20:46.040 --> 20:53.320 comes with my skill packages right and this comes with certain criteria like you know 8 characters 20:53.400 --> 21:00.760 mixed case number special character so on and so forth once you install this let's try to create 21:00.760 --> 21:07.960 a password which possibly does not satisfy the criteria see this is the same password that we use 21:07.960 --> 21:16.520 before if you you see is the same password that was earlier okay because it was just checked by 21:16.520 --> 21:21.720 validate of password breach component now it is being checked even by validate password component 21:21.800 --> 21:25.800 and this doesn't have any special character it has a per case lower case it has a number but it 21:25.800 --> 21:31.080 doesn't have any special character and that's one of the criteria put forward by this component so 21:31.080 --> 21:37.080 this call basically goes fast through password breach check which checks online comes back says 21:37.080 --> 21:42.120 hey is there any other components that implements the same service yes let me go back and ask that 21:42.120 --> 21:47.080 service if that service says yes I will allow the password if it says no I will block the operation here 21:47.960 --> 21:57.960 if you go to air log I don't think we will see any additional error because this was just 21:57.960 --> 22:03.320 it I mean the valid the password breach check did not really throw an error for those times 22:07.640 --> 22:12.840 so now we will try a password which satisfies criteria of 22:13.400 --> 22:20.600 validate password component but not the password breach check it is the second scenario we have 22:20.600 --> 22:26.680 already seen this password will be rejected here this was rejected here it's again rejected here 22:28.680 --> 22:34.120 so it's this is just to showcase that the validation happens against both the component it's 22:34.120 --> 22:39.400 it's not like that we are validating only against a single component and if you go back and see 22:39.400 --> 22:43.960 the error log you will see that yeah one more message appears that okay you're trying to use the 22:43.960 --> 22:51.800 password which has appeared in the breach you can't really use the password now what what we'll do 22:51.800 --> 23:01.080 now is we'll tweak the password little bit we'll add you'll change it so instead of first 23:01.080 --> 23:07.160 ms I have added dollar sign which now make sure that this password passes both the components 23:07.160 --> 23:12.200 criteria it is not appearing in any breach and it also contains all the mandatory characters 23:12.200 --> 23:19.240 that are you know enforced by validate password and as you do that it passes and you know you're 23:19.240 --> 23:25.800 able to use that password you can create a user or you can change existing users password using that 23:28.280 --> 23:35.240 so yeah that's that's pretty much it about the demo the final piece of the puzzle writing your 23:35.320 --> 23:40.200 own component these are some of the things that I have learned through my experience with component 23:40.200 --> 23:47.880 and just you know would be good to share if you're writing your own component follow an example 23:48.760 --> 23:54.200 I would say component related password is a very good example it's very small but it has all the 23:54.200 --> 23:59.720 right ingredient that you would need it has various services that it uses so you know how to 24:00.280 --> 24:07.400 specify dependencies and how to use them in your component and it also provides certain services 24:07.400 --> 24:13.400 more than one service actually so you can you can take a look you can copy paste and then play 24:13.400 --> 24:20.040 around with this stuff there are extensive macros provided by component infrastructure header files 24:20.040 --> 24:26.280 please use them that they will make your life very very easy defining various mandatory parts of 24:26.280 --> 24:33.480 this component in fra this macros are very useful I mean it's like 10 lines of code is can 24:33.480 --> 24:39.640 encapsulate it with just one macros so please use it if you're writing your own service and 24:39.640 --> 24:45.000 this will involve hacking into server adding certain services here and there make sure that you 24:45.000 --> 24:51.960 use only basic types so integer floats characters so and so forth if you have a complex object 24:52.120 --> 24:58.120 that you want to use or return or pass around between your APIs use them as a opaque pointer 24:58.120 --> 25:04.360 so only your component should be operating on that you should not expose that through service 25:04.360 --> 25:12.840 APIs one of the very important point is how long you want to keep handle of or keep a 25:12.840 --> 25:18.840 reference of another service so if you define an explicit dependency component is such 25:19.160 --> 25:24.920 will by default hold all dependency references for the lifespan of your component that is one way 25:25.720 --> 25:34.120 but if you are dynamically taking reference to any service be careful it's expensive so doing it 25:34.120 --> 25:39.960 few times is okay but if you are doing it hundreds and thousands of time that will put pressure on 25:39.960 --> 25:51.320 the entire live-manjasi infrastructure on the so there is a dedicated call to acquire a service 25:51.320 --> 25:56.440 and it's always recommended that you acquire a service or you already hold the service 25:56.440 --> 26:03.000 reference before you call any of the API of the service so once you do an acquire behind this 26:03.000 --> 26:09.320 in live-manjasi will increment a reference make sure that that component is not an installed 26:09.480 --> 26:14.360 and this is really about it keep it for the long time that means that other component cannot 26:14.360 --> 26:19.720 go away till the time your component releases the reference so it's kind of striking the balance 26:21.560 --> 26:27.480 right and if you so okay so component first you start state less if you want to maintain a state 26:27.480 --> 26:32.280 pass your own opaque pointer but if you are doing that make sure that you have a destructive API otherwise 26:32.360 --> 26:38.760 you will leak memory very easily so that's about it this are some of the references you can 26:39.160 --> 26:45.480 you can take a look at the doxygen documentation of server it provides a nice summary of all the 26:45.480 --> 26:51.240 components that we have it also gives you an example of how you can write your own component 26:51.800 --> 26:59.240 and the last two is the implementation that I have used in this presentation so that's it thank you 27:03.160 --> 27:15.320 we have time for questions how questions long last one first my place I have this issue 27:15.320 --> 27:21.160 I have to ask you to ask your plug-ins is still around plug-ins you say want to go away 27:22.200 --> 27:25.960 you will be replacing your plug-ins with the components so that people can use this 27:26.920 --> 27:44.200 we are already doing that yeah so slowly all right I'll write about it but we can not 27:44.200 --> 27:49.880 straight away remove plug-ins or the API is because we may break things left and right so right 27:49.880 --> 27:55.480 now we are deprecating things that okay so validate pass what is one prime example the plug-ins 27:55.480 --> 28:02.280 service is now deprecated probably we will remove it sometime in future but it's very it would be very 28:02.280 --> 28:06.920 ugly to remove it right away because just because we have the component in place because if somebody 28:06.920 --> 28:12.840 else is using it there might be an issue sure sure sure sure sure sure 28:25.480 --> 28:36.360 I'm not sure I understand if you can please repeat I'm not sure I understand if you can 28:36.360 --> 28:42.200 please repeat I'm not sure I understand if you can please repeat I'm not sure if you are running 28:42.200 --> 28:53.080 just serving an environment but you don't have to have access to that yes yes you can 28:55.480 --> 29:01.480 okay thank you thank you