WEBVTT 00:00.000 --> 00:14.160 Okay. So, Ricardo, we'll explain how to use of intermediary in the situation of a package 00:14.160 --> 00:20.000 of a tool in Python, of course. I think that will be a really interesting. Please. 00:20.000 --> 00:31.360 Thank you. So, hi, everyone. I'm Ricardo Mallocetti. I work in absolutely the elastic 00:31.360 --> 00:37.600 where I maintain open-ended stuff. And also, I'm a maintainer for the upstream open-ended 00:37.600 --> 00:45.760 Python, Python 3B projects. I'm this talk. I want to show you, like, I'll open-ended 00:45.760 --> 00:52.320 Python implementation works. And so, we see, I'll be able to automatically recognize 00:52.320 --> 00:58.800 an install instrumentation libraries with open-delamity bootstrap. Then we'll discover 00:58.800 --> 01:04.160 our open-ended instrument, is able to leverage the Python site module to enable 01:04.160 --> 01:11.200 automating instrumentation of your applications. We'll then see, like, our end-to-point 01:11.600 --> 01:17.520 used to discover components in different packages. And later, we'll see, like, our 01:17.520 --> 01:22.960 instrumentation libraries are able to patch for particle. And you're already at, like, a 01:22.960 --> 01:31.440 spoilers on talks ago. But I'll start, like, giving people new to open-delamity and 01:31.440 --> 01:37.600 the visibility, like, a bit more context on this thing. So, how many of you already 01:37.600 --> 01:47.040 knows, like, open-delametry, at least the word, okay, okay, many hands. And how many of you 01:47.040 --> 01:58.240 are already using open-delametry? Okay, still a bit cool. Okay, so, observability is the ability 01:58.960 --> 02:06.000 to understand the internal state of a system by examining its outputs. And this means, like, 02:06.000 --> 02:12.240 being able to understand what a software system is doing, by looking at a telemetry data, 02:12.240 --> 02:18.960 it exports. And this telemetry data, usually, is like, it's about the problem programs as a 02:18.960 --> 02:24.160 cushion, like, in terms of states, and also about the communication between components. 02:25.440 --> 02:31.200 And the open-delametry is an open source and vendor-neutral observability framework, providing 02:31.200 --> 02:37.840 both a specification and a lot of implementation different languages. I noted to create a 02:37.840 --> 02:44.240 manage-based telemetry data. And this data could be, like, traces for tracing as a cushion 02:44.240 --> 02:52.640 of the software parts. So, calls into and from your Python application, also think, like, 02:52.640 --> 02:58.240 database calls, HTTP calls, RPC calls, and stuff like that, but also, like, internal function calls. 02:59.200 --> 03:05.760 And then we have metrics to capture some measurement at runtime. For example, like, in memory usage, 03:06.320 --> 03:11.040 or, like, the number of requests and the time spent saving them and stuff like that. 03:12.240 --> 03:20.640 And then we have lots, but can be, like, some structured or unstructured text with a timestamp. 03:21.520 --> 03:28.320 And in open telemetry, this free concept traces metrics and logs are called signals. 03:29.040 --> 03:37.600 There is another signal in development for profiling data. And in practice, with observability, 03:37.600 --> 03:43.040 you can do something, but look like this. You can see, like, distributed transaction between 03:43.120 --> 03:50.880 different services. In the picture, we have free services. The one I read is the main one. 03:51.520 --> 03:56.960 And we can select calls. And also, like, it's communication with the one in blue, 03:57.520 --> 04:04.720 the middle of the screen. And with the one you read in the, at the end of the screen. 04:04.720 --> 04:09.840 So, the idea, like, to see, like, the whole life cycle of requests and insider application. 04:10.800 --> 04:16.640 OK, this was enough for introduction. Now, let's look at some open-demory Python specific stuff. 04:18.000 --> 04:25.040 So, the open-demory Python project consists of two different repositories. One is named open-demory Python. 04:25.040 --> 04:32.160 And that one is called the open-demory Python country. But open-demory Python contains, like, a few different 04:32.240 --> 04:39.120 package. The main one are the open-demory API. But defines the interface, 04:40.000 --> 04:47.040 exposes to users. And this is not tied for specific implementation. And then we have the SDK, 04:48.320 --> 04:55.280 by the same implementation on the API we've seen before. And then we have another important project. 04:55.280 --> 05:02.080 Package is the semantic convention package, by contents, contains all the article definition 05:02.080 --> 05:06.960 that have been populated by these semantic libraries. This semantic convention defined 05:06.960 --> 05:15.120 practice what data you get in traces logs and metrics. And the content of this package is generated 05:15.120 --> 05:21.200 from a specific cache. And then we have the other repo, that is open-demory Python trip. 05:21.920 --> 05:29.120 By contents, all the instrumentation libraries, when it contains also the open-demory instrumentation 05:29.120 --> 05:34.000 package and the open-demory to destroy package. Each instrumentation library, 05:34.800 --> 05:44.000 specific to a Python package, is a different package itself. And that sits inside the instrumentation 05:44.000 --> 05:50.720 libraries. We also have an instrumentation generic directory for a generic specific instrumentation. 05:52.160 --> 05:56.960 And then we have the open-demory instrumentation package, but the implements what we call 05:56.960 --> 06:02.240 like auto instrumentation or zero-cond instrumentation. But it's the ability to instrument application 06:02.240 --> 06:08.560 without touching the application code. Finally, we have the open-demory to destroy package, 06:08.560 --> 06:16.160 like contains like basic default configuration, like deciding to configure like where to 06:16.160 --> 06:23.200 support the data. And also the support in protocols. And please note that the project 06:23.200 --> 06:29.200 likes a lot on environment variables for configuration. And so the destroy uses environment 06:29.200 --> 06:38.640 levels a lot of suits. Enough for the instrument introduction. Let's look a bit of the 06:39.600 --> 06:47.200 thing's work. So for a demo, we use like a very basic flask application. This is just an 06:47.200 --> 06:55.440 endpoint mounted on the root part. But it is returning as thing, all the worst thing. So I guess 06:55.440 --> 07:01.280 you are familiar with something like that. So first we're going to stop like a virtual environment 07:01.280 --> 07:07.760 activated and install flask. And we're just installing an open telemetry destroy because it 07:07.840 --> 07:14.480 depends on the API SDK and instrumentation packages. So we get everything we need for the just 07:14.480 --> 07:22.720 for demo. So the first tool we'll take a look at is the open-demory bootstrap. And open-demory 07:22.720 --> 07:28.400 bootstrap is a CLI tool from the open-demory instrumentation package, but is able to list 07:28.400 --> 07:34.160 or install an instrumentation library available for your project dependencies. The first thing 07:34.320 --> 07:40.480 but here requires is an updated list of some special libraries and the library instrument. 07:41.280 --> 07:46.240 And this is generated at each full request. So every time an instrumentation library is added, 07:47.200 --> 07:54.720 the open telemetry Python-con3 repository is getting updated. A bunch of instrumentation 07:54.720 --> 08:00.640 libraries depend only on the standard library and so they're always list as available to install. 08:01.600 --> 08:08.400 And in practice, this is how the file containing the demo thing between the Python packages 08:08.400 --> 08:15.040 on one side and the instrumentation libraries on the other side looks like. So for each instrumentation 08:15.040 --> 08:23.600 library, that has some dependencies. Open telemetry bootstrap searches the models you have installed 08:23.600 --> 08:29.360 in your virtual environment. For a distribution of the dependencies dependency, that has a 08:29.440 --> 08:37.280 much in version. If it finds a library, but can be instrumented, it lists the instrumentation 08:37.280 --> 08:47.280 library as available to install. In the case, you're asking the tool to install this library. 08:47.840 --> 08:54.800 It also calls a course pip check to check for any version conflict. And in specific case, 08:54.800 --> 09:01.520 for the flash library, we see that we're looking for a flashed version of a bigger equal than 09:01.520 --> 09:09.520 one.to. And if it is found, it will like list or install the open telemetry instrumentation 09:09.520 --> 09:21.200 flask version at a package at version 0.50 m0. And there, you have the default instrumentation, 09:21.200 --> 09:28.160 but there's the instrumentation, but does not have any dependencies. And so this one are stored 09:28.160 --> 09:39.760 only by default. Now, let's try this tool against our virtual environment. Mine, this is the output 09:39.760 --> 09:47.600 of the list common, even if calling the install action, because the output of the action. 09:48.320 --> 09:55.600 A lot bigger and doesn't fit in the slide. So, like we can see, we have the common packages, 09:55.600 --> 10:00.160 but that does not have any requirements, like I said in Cairo and the WISGi, 10:00.160 --> 10:05.200 open LDS and the special packages. But we also have the open telemetry instrumentation flask, 10:05.200 --> 10:11.040 because in the sounds like the go, we installed flask, but is already at one. And later, 10:12.000 --> 10:18.320 installed in our virtual environment. Okay, now let's talk at the next tool, 10:18.320 --> 10:23.360 but is open telemetry instrument, but is the one that remains of the instrumentation. 10:24.720 --> 10:30.720 So, our instrumentation is the ability to set up a telemetry like tracing matrix 10:30.800 --> 10:42.400 log is metering without touching the code. So, the tool is like a wrapper, but wraps your application. 10:43.200 --> 10:48.960 So, question, resilience. Oh, many of you already know the set customized module. 10:51.200 --> 10:58.240 Okay, one pair, two persons. Cool. So, you'll learn something new, hopefully. 10:59.040 --> 11:04.480 Okay, by default, if you don't know it, like a set module of your Python installation, 11:04.480 --> 11:11.280 tries to load from your Python part, a model named set customize, and open them to Python, 11:11.840 --> 11:18.160 auto instrumentation uses like leverage system in order to initialize itself. 11:18.160 --> 11:23.680 If you want to learn more, there's the documentation, the standard library documentation. 11:24.560 --> 11:33.280 And a quick example to demonstrate how the set customized work. First, we'll tell the Python 11:33.280 --> 11:42.640 temperature to print the word string, and it works. And then, we're doing the same, 11:42.640 --> 11:48.320 but adding the current directory as the first element of the Python part variable. 11:48.320 --> 11:55.520 Yeah, unfortunately, the library changes, and we see another string printed. 11:57.280 --> 12:03.840 And this work, because in the current part, there is a five module named the set customize, 12:03.840 --> 12:11.520 a five name set customized.pi, but contains the print. And so, yeah, yeah, this is like magic. 12:11.760 --> 12:20.560 So, I open the letter instrument, words by like wrapping application. 12:22.160 --> 12:27.920 We've the open the letter instrument command. And this command, first thing is puts 12:29.200 --> 12:35.040 the part of the open parameter Python set customize module as a first entry Python part. 12:35.440 --> 12:41.040 And also, that's like a bunch of things under root. First thing is it loads the 12:41.040 --> 12:47.520 alters, but loads the distra, the open parameter is the package, by default. And also, 12:47.520 --> 12:52.800 the alters, the loads the configuration, but are usually provided by the distra. And we'll see 12:52.800 --> 13:01.040 our later. And the alters, the loads, the instrumentations. And I'll explicitly disable the 13:01.040 --> 13:06.880 random variable again. All distraimmentation founds are enabled and loaded by the fold. 13:07.600 --> 13:14.720 And all these alters are using like Python at importance in order to work. And we'll see later 13:14.720 --> 13:23.280 what we are. So, if we run our Flask application wrapped by open parameter instrument 13:24.160 --> 13:31.920 and do a request, we'll see like atrace like this, because the installed Flask instrumentation 13:32.720 --> 13:39.280 library has been loaded and it's working. Please note that here I'm configuring an 13:39.280 --> 13:46.400 open that instrument to the drop matrix analogs and print the traces to the console. And so, 13:46.400 --> 13:50.160 we'll see something like that. This is like useful only for debugging or developing. 13:51.040 --> 14:02.480 And if we look a bit into the output, we'll see like the attributes key. And this key is 14:02.480 --> 14:12.080 in values like the depends on what the semantic convention packages and specification define. 14:12.640 --> 14:21.600 And like, not important to look at specific keys, but this, like, follow more or less the 14:21.600 --> 14:28.880 HTTP semantic convention. Now, let's take a look at 10 different points. Do you already know 14:28.880 --> 14:35.200 anyone already knows what are the entry points in Python? Okay, a bunch of hands. Cool. 14:36.160 --> 14:42.480 Okay, so I enter points are mechanism where packages use like to provide the discoverability 14:43.360 --> 14:49.280 over some very cold. And this is the way they're open-termitri Python project implement plugins. 14:50.800 --> 14:58.480 And this project, and open the open-termitri visthro package. And also in other packages, 14:58.480 --> 15:06.240 we configure entry points in the pyproject.com all metadata file. This is an example of 15:06.240 --> 15:16.880 the configurator entry points. Like, we have a group. In this case, open-termitri configuration 15:16.880 --> 15:25.360 greater. And every group has a like a bunch of couples of name and controversies. In this case, 15:25.440 --> 15:34.080 we have a configurator name with the value of the open-termitri configurator class inside the 15:34.080 --> 15:47.920 open-termitri visthro package. And in practice, like, other code, but we need to pop with the open-termitri 15:47.920 --> 15:57.920 configurator group. Just called the entry points function. We're importing it from the open-termitri 15:57.920 --> 16:05.360 utility, imported method data model. But the re-entry mutation is from, importantly, 16:05.360 --> 16:09.760 under-score method data models. So it's a different package, not the one in the standard library, 16:09.760 --> 16:15.760 because it changed a lot between Python version. I mean, it's a part like white range of Python 16:15.840 --> 16:23.440 versions. So we're just looping over the all the entry points it's finding. And we're calling 16:23.440 --> 16:30.160 this is like a bit hard to read, but like we are loading the entry point. And then we are creating 16:30.160 --> 16:36.560 an instance of the open-termitri configurator class. We are calling the configurator method of the 16:36.640 --> 16:44.640 instance. And, you know, just for debugging, we are planning like, but we're loaded with code. 16:47.280 --> 16:55.840 So, let's begin. We are seeing today, that is like monkey patching or patching for particle. 16:55.840 --> 17:04.080 You know, how many of you were already there and seen the pair's term? Okay, so a lot. 17:05.120 --> 17:12.080 So, I can say before you, that if you use any tracing or instrumentation libraries or something 17:12.080 --> 17:17.520 or anything like that, you're already monkey patching your code in production. So sorry. 17:18.320 --> 17:26.720 So, monkey patching is a technique used to dynamically change the behavior of some code at 17:26.720 --> 17:34.320 runtime. And it used by most of the instrumentation libraries in the open-termitri Python project 17:34.320 --> 17:39.440 to wrap other classics, methods or function. Like for example, imagine we are still 17:39.440 --> 17:44.720 renting like a function. For each code, we want to know of course the function parameters, 17:44.800 --> 17:51.440 and also what it returns. And the instrumenter code, my ear loops, for example, 17:51.440 --> 17:57.600 like so libraries like request, I guess. We can use, we can use, but most of the time, 17:57.600 --> 18:04.160 like libraries does not provide and you can, and you can do. And, thanks to Python, 18:04.160 --> 18:09.520 many being a dynamic language, we can monkey patch the functions and methods again. 18:10.480 --> 18:15.840 Instead of calling the original function, we can have like our wrapper, it called before, 18:15.840 --> 18:23.680 and after the original code. And yeah, this wrapper function is usually like the one 18:23.680 --> 18:32.560 I will create traces, matrix, and maybe sometimes ever, sometimes. So, like we're using like 18:32.640 --> 18:40.000 most of the time, the RAPT library by Graham Dubb, Dumpeton. And, you know, my 18:40.000 --> 18:43.920 person suggesting is whatever you need to monkey patch, they create something just as 18:43.920 --> 18:53.120 RAPT because it's also provided of tools. Yeah, great. And we see like a couple of examples, 18:53.120 --> 19:00.240 on how to use like a couple of different way to instrument things. For example, is using like 19:01.200 --> 19:08.240 toy instrumentation library, what it created for another talk, and this is 19:08.240 --> 19:15.120 something like a couple of methods from the Azgirf library. Azgirf is an implementation of the Azgir 19:15.120 --> 19:22.240 protocol that is used by Django, at least also I think also all the Azgir servers. 19:22.480 --> 19:31.360 So, this specific instrumentation, the Azgirf instrumentals, implement like a class, but is 19:31.360 --> 19:36.800 in editing from the basis to mentor class, but is provided by open standard Python. And 19:36.800 --> 19:42.240 it implements like at least a couple of methods. First one is an underscore instrument, 19:42.880 --> 19:50.000 but is the method that is like applying the actual monkey patching. And it's using like function 19:50.080 --> 19:58.800 from RAPT called RAP function RAPA. RAPA, where yeah, first parameter is the function we want to 19:58.800 --> 20:07.120 the model we want to look into. Second parameter is the actual in this case I think is a function 20:07.120 --> 20:14.640 we want to RAP. And the third parameter is our function we provide as a RAPA, and we see it on 20:14.720 --> 20:18.960 the next slide. And then we have also like an underscore instrument, 20:20.160 --> 20:26.880 method, but of course I'm RAPA, but it is an IPR also from open standard Python, but like 20:26.880 --> 20:35.360 removal RAPA and the rest day to the original code. And yeah, so our RAPA code, 20:35.920 --> 20:45.920 you know the important bits here, it's the first parameter, well after self, because it's like 20:46.800 --> 20:51.920 developed function, so the original function, and so we can call it and get it's output. 20:52.720 --> 20:59.280 And this example we are doing something I think a bit silly, but is like printing, like sending 20:59.360 --> 21:09.360 an event, adding an event to a trace of the function, adding its stack trace, and this is 21:09.360 --> 21:15.840 like not like the usual way you do for tracing, because it's maybe like a lot of data if you 21:15.840 --> 21:23.440 call the function a lot, but it was like down for again research purposes, because like I was 21:23.440 --> 21:33.200 divided by the bagging, how many times a single application was going from a single to single 21:33.200 --> 21:43.760 and the other way around. So again, like last line is returning what the original function returns. 21:44.480 --> 21:58.320 Then like another example of how we can use RAPA to RAPA things, we are using the object proxy, 21:58.320 --> 22:08.720 but it is a class, but you can use to wrap classes instead of functional methods, and the usage 22:08.800 --> 22:19.840 is back like you replace the original value with instance of this class passing the original class 22:19.840 --> 22:27.120 you want to wrap as a first parameter, and this is from Maria called this is from Maria in 22:27.120 --> 22:35.440 its documentation, for the botto call library, but it's like what botto free library used to connect 22:35.520 --> 22:50.160 to AWS stuff uses under the root, and the idea here is that like we are like hooking into the 22:50.160 --> 22:57.360 the under I termited, because this class is just like an iterator where you get like events 22:58.320 --> 23:06.640 from an API, and so we are able to like consume them with iterator, but also 23:08.080 --> 23:13.840 process things, and so we are able to call our own code, and the color doesn't see the difference, 23:13.840 --> 23:22.800 because we are yielding the original value. Okay, so we used was the last topic 23:23.440 --> 23:29.200 conclusions, to conclude like we are seeing like a bunch of tricks, they are open to having the 23:29.200 --> 23:38.080 pattern project users to work under the root, and maybe like for the user you may be like 23:38.080 --> 23:42.800 now it's more clear what you are doing under the root, and we are seeing like how we are able to 23:42.800 --> 23:47.040 automatically recognize and install instrumentation libraries, we will open the telemetry boost up 23:47.440 --> 23:57.120 command, how the site and site customize modules can be used to run our code before the application 23:57.120 --> 24:04.960 one, and our interpoints are used to discover components in different packages, and how you can 24:04.960 --> 24:12.480 use RAPT to patch for a party code. If you are interested in the open telemetry Python project, 24:12.560 --> 24:19.520 we are always looking for help, I live some links, like the open telemetry IO, which is the open telemetry 24:19.520 --> 24:27.040 website, the two repos, so the telemetry Python and open telemetry Python on Twitter, and then we have 24:27.040 --> 24:36.000 a select channel on the CNCF as lucky instance, and we also do like a weekly call to discuss topics, 24:36.640 --> 24:44.240 yeah, last one is my master dynamic, if you want to get in touch, if you have time, I'm glad 24:44.240 --> 24:52.400 you take any questions, thanks. Thank you very much. I heard a microphone do you have a question? 24:54.160 --> 24:56.160 No, yes. 25:06.000 --> 25:21.200 Thank you. Thank you very much for this talk. I'm sorry to continue, so guys, 25:22.560 --> 25:26.080 guys leaving, please love it more silent, please. Thank you. 25:28.160 --> 25:30.000 Thank you very much for your talk. 25:30.960 --> 25:37.280 I have been using production your tool to instrument. I'm sorry, I can't hear it. 25:40.160 --> 25:43.280 That would be really a little bit louder. Does it work now? 25:43.280 --> 25:48.880 Yeah, thanks. Thank you very much for your talk. So I've been using the bootstrap and 25:48.880 --> 25:52.640 holding instrumentation in production for a while, and I've noticed that there's a little bit 25:52.640 --> 25:58.160 of results in terms of which dependencies it pulls in. One example can be, I'm not using 25:58.160 --> 26:06.560 AWS as a cloud, but some other dependencies are pulling in S3 of libraries because of dependency 26:06.560 --> 26:11.360 leakage. Is there any way for me to run the command in a way that I would be able to understand 26:11.360 --> 26:18.640 which dependencies are pulled why, like the interconnection between which open telemetry 26:18.640 --> 26:27.200 dependencies are pulled and why are they pulled? Okay, so you're asking, if you're able 26:28.480 --> 26:37.200 to discover which instrumentation library gets installed, depending on your packages you have already 26:37.200 --> 26:43.440 installed, your dependencies? Which package is pulling the dependency? Yeah, like, 26:43.440 --> 26:50.400 ever seen on this slide, you can grow like, oh, you mean dependencies of this 26:50.480 --> 26:58.320 instrumentation? Yeah, exactly. So I have a dependency. The bootstrap is pulling in lambda hours. 26:59.360 --> 27:07.440 Okay, so like, if you need to see like the dependencies of each instrumentation, you can look at the 27:07.440 --> 27:16.560 by project.com all five. If you want to look like which packages this instrumentation 27:16.640 --> 27:23.280 wraps, you can look inside the Python 3B repo, the mod is called boostrap and the score 27:23.280 --> 27:31.360 gen and where is the map I've shown before, where you can see like the libraries and the instrumentation. 27:34.160 --> 27:38.560 And we'll later guess. Thank you, may you can talk later.