WEBVTT 00:00.000 --> 00:13.080 Thank you, so let's repeat for the stream. Today I'm going to tell you a little bit about 00:13.080 --> 00:21.320 the site evideo.bg and how it was born and how we made it possible. The title of the 00:21.320 --> 00:29.800 stock is scaling to 12k live streams and 12k you will understand from where it comes. 00:29.800 --> 00:38.120 So first of all, who am I? I'm Vladimir. I'm mostly a participant in DevOps and security 00:38.120 --> 00:46.960 guys. I like play and do automation and some tooling. I do some architecture and everything 00:46.960 --> 00:56.040 that gets thrown by sites. I work for the company information services. The company itself 00:56.040 --> 01:02.800 is the national integrator for the Republic of Bulgaria. We have different kinds of services. 01:02.800 --> 01:13.880 We are a eidus qualified. We are also a member of Uritus. We have platform service, regular 01:13.880 --> 01:21.600 infrastructure services, on-premises, clubs, everything. We also do stock as a service for the 01:21.600 --> 01:30.840 public institutions in Bulgaria. One of some of our major projects are in health care, transportation, 01:30.840 --> 01:40.000 financial and judiciary branches of the government. The problem that we have, there is legislation 01:40.040 --> 01:46.560 and the legislation says any and all ballot counting processes and activities should be recorded, 01:46.560 --> 01:55.440 streamed and dark-eyeed in real-time. The requirements for that simple question that simple 01:55.440 --> 02:01.920 problem are first of all, there is a legislation and the legislation says, just live streaming 02:01.920 --> 02:07.080 should be done recording, filling of the forms, counting of the ballots, basically the whole 02:07.160 --> 02:16.640 process that starts when the poll closes. The recordings should be available online. The recording 02:16.640 --> 02:23.720 should be archived and it is absolutely mandatory with several small exceptions and the exceptions 02:23.720 --> 02:33.640 are mainly for specific settings like for example elder care facilities, medical facilities, 02:33.640 --> 02:41.880 gel facilities and similar. Basically where there is sensitive context, we are not doing 02:41.880 --> 02:52.600 any monitoring, but it is mandated in the law itself. The law came into force in April, 02:52.600 --> 03:00.360 in February, a little bit less than two months before the next election. The whole system 03:00.840 --> 03:11.960 had to be designed, deployed, created, operated in that short time. The devices that are provided 03:11.960 --> 03:19.160 are going to be operated by the polling station chairman, at least that is defined in the law. 03:22.040 --> 03:29.720 The person people operating it are usually not really trained and there is no bonuses rewards 03:29.800 --> 03:37.480 but also no penalties if something doesn't happen right. Which basically means untrained personnel, 03:37.480 --> 03:42.360 uncertain conditions, the only thing that we can control is the device itself. 03:43.880 --> 03:49.000 The scope, we have close to 12,000 polling stations in Bulgaria, 03:49.880 --> 03:55.400 all will be streaming or at least, should be streaming. They start mostly simultaneously, 03:56.360 --> 04:00.440 they stream for two to six hours on average from the election day itself, 04:02.040 --> 04:11.400 and those were the things that were known from the start. On top of that, we added some more 04:12.200 --> 04:18.520 some more of the requirements. We want a really nice audio channel so you can hear the audio, 04:18.600 --> 04:26.200 which is usually quite important. We also wanted to do a parallel recording or local storage, 04:26.200 --> 04:35.240 so we have a backup in case there is no connectivity. As I said, this needed to be 04:35.240 --> 04:42.520 live and operational in less than two months. The solution for it is simple, nothing strange, 04:42.520 --> 04:49.400 nothing ordinary, just the scale and the timing were a bit problematic. So, first of all, 04:49.400 --> 04:57.080 we need a camera. Mobile phone will do, in fact, the mobile phone, it might sound strange to some 04:57.080 --> 05:04.200 people but otherwise it's a very fully featured device. It has a built-in UPS in the form of battery. 05:05.160 --> 05:14.040 It has usually a decent camera that you can use as a monitor. It has microphones, 05:14.040 --> 05:19.240 if you don't know, the microphones in recent forms are really nice, really good, really crisp, 05:20.040 --> 05:27.880 they also come sometimes with an array of microphones. The mobile phone itself has multiple connectivity 05:27.960 --> 05:37.640 options, including words via USB-C, for example. More and more devices have hardware 05:37.640 --> 05:44.280 accelerated colleagues, so it's easier to do this in hardware than to write the software and 05:44.280 --> 05:50.920 back the CPU to the top. They're usually pretty simple to handle or almost anybody can 05:51.000 --> 06:01.080 handle a phone. He usually does any day of the week. The mobile phones have a very, very well 06:01.880 --> 06:07.320 accepted solution for remote management in the form of mobile device management software. 06:08.520 --> 06:17.400 Additionally, we slapped a little bit of RFID attacks and QR codes, so it's easier to ask it 06:17.560 --> 06:26.360 manage them. The devices that we chose, they were based on Android Go, the Android version was 13, 06:28.440 --> 06:36.360 and they were quite inexpensive in fact. If we go back to the previous requirement that this should 06:36.360 --> 06:44.120 be live in two months, try to find 12,000 falls with a lead time of less than 30 days. 06:44.760 --> 06:52.040 I really challenge you, it's not a pleasant task. The other pieces of the puzzle that we're using 06:52.040 --> 06:59.400 is mobile application of course. We need to do some kind of streaming, so we developed in 06:59.400 --> 07:07.400 house application that's doing this. We are not using anything fancy, just of the shelf libraries 07:08.360 --> 07:15.640 and our custom logic. The application itself needs to do the recording to, the application itself 07:15.640 --> 07:29.160 needs to be configurable by some remote means. We chose to do this via the MDM. We decided to specify 07:29.160 --> 07:37.000 QR code as a means of initial configuration, starting stopping streaming and other tasks. 07:37.960 --> 07:46.440 The QR codes themselves are encrypted, and so the data is mostly secure. If you have enough time, 07:46.440 --> 07:52.040 of course probably you could break it, but you usually have the QR code for something like 07:52.520 --> 08:00.920 24 hours before it needs to be used, so we're relatively sure that it can be broken in that time. 08:02.760 --> 08:10.520 Of course the applications need to evolve. We're constantly improving them with recently added 08:11.880 --> 08:16.920 health monitoring, the application itself will hit the phone, the state of everything that's 08:16.920 --> 08:23.080 happening, the audio, the positioning, the orientation. We fought it much more logging since the 08:23.080 --> 08:29.080 initial implementation. What I forgot to mention, the system has been already used for 08:30.600 --> 08:36.840 four different major elections. I know three, sorry. Three major elections that are covering the 08:36.840 --> 08:45.480 whole country, several several smaller elections that are happening due to certain conditions. 08:46.760 --> 08:52.440 One of the major elections was a combined election for parliament and the European parliament. 08:53.560 --> 09:02.360 So it was, I think it was pretty well tested. Better encoding profiles were chosen, so in order to 09:02.360 --> 09:11.160 lower a bit the CPU usage and the temperature. The performance has been improved. All in all, 09:11.560 --> 09:20.760 we're developing it because, yeah, it's good to be developed. We also need a mobile device management 09:20.760 --> 09:29.560 solution. We found a big plan that's open source. It provides the director for monitoring 09:29.560 --> 09:36.760 logging of the devices themselves. It supports different profiles, but that's usually all of them do. 09:38.040 --> 09:46.600 It stores the configuration. It supports messaging. It's as synchronous. It doesn't block 09:46.600 --> 09:52.600 all the devices calling tweets and if it needs to push the push information the other way, 09:52.600 --> 10:01.640 it also happens as synchronous thing. We can also do via the MDM, forced updates and 10:01.640 --> 10:10.600 other different tasks. The MDM itself allows us to lock down the device itself, so it's not so 10:10.600 --> 10:16.280 the user cannot easily break it. Of course, excluding dropping the phone itself on the floor. 10:16.680 --> 10:25.400 Some results, you have run already eight different elections that I said. Both combined, 10:25.400 --> 10:33.560 both single, small, large. The graphs that you see on the top, it's just the statistics of 10:34.440 --> 10:43.400 using the website that's caused the results. The bottom picture is a screenshot from our internal 10:43.400 --> 10:53.000 control plane. The top row is listing the active sections that have been active for the 10:53.000 --> 11:02.360 election itself. The bottom row is the current active ones. For the public reception of the 11:02.360 --> 11:10.840 system itself, at least we want to think that's very nice. But it enforced a stricter process 11:10.840 --> 11:19.720 controls. There are more than 50 official complaints in the court for which the sole basis. 11:19.720 --> 11:24.520 Well, not the sole basis, but the main basis is the video monitoring that we're doing. 11:26.120 --> 11:33.320 There is already one of the cases, has reached the final judgements and conviction. 11:34.360 --> 11:40.360 Many of the cases are still in investigation and more or less constantly in new cases are coming. 11:41.320 --> 11:47.800 So, all in all, I can say that the system has increased the water engagement and the public 11:47.800 --> 11:55.640 engagement in the process itself. The pictures that you're seeing are the top one is from the 11:55.640 --> 12:02.680 already finished court case, that has reached the judgment. The bottom one is again from 12:02.840 --> 12:10.920 very possible manipulation for the process itself. That's also being investigated. 12:12.680 --> 12:17.960 Of course, you cannot do such a system without testing and of course I guarantee you the testing 12:17.960 --> 12:25.640 will not be enough. We did quite a lot of testing and then some more and then again some more. 12:26.920 --> 12:31.880 We were doing testing in the development process on a freshly provisioned environment 12:32.200 --> 12:39.720 in a clean in a clean state. In the polling places, we also did some testing. 12:41.160 --> 12:49.000 We did performance testing continuing to wire out some details and I guarantee you when 12:49.000 --> 12:53.720 you deploy a system in the world, all the testing will not save you because the world can 12:53.720 --> 13:02.200 throw at you very different stuff that will break you. Now we come to the infrastructure, 13:02.200 --> 13:08.440 that's the basic stuff that we are going to need. So, the calculations are brought to the 13:08.440 --> 13:16.440 simple 12K in-bound streams, megabit and a half for each total of 80 gigabits in-bound, 13:16.600 --> 13:26.280 outbound and certain. We have seen several thousand online viewers but that's frankly said 13:27.800 --> 13:31.800 not the thing that we are monitoring due to the way the infrastructure is deployed. 13:34.200 --> 13:42.840 We unit 1220 to 40 terabytes of storage space for the recordings of a single election cycle, 13:43.480 --> 13:51.000 at least for the big ones. You need sufficient connectivity to all Bulgarian telco providers 13:51.640 --> 13:56.600 because as we already know, we are using just a phone and the phone needs to be able to talk to 13:56.600 --> 14:02.920 its mothership. We are not using Wi-Fi, we are using 4G for the connectivity. 14:05.640 --> 14:10.680 From the infrastructure itself, we do require quite nice performance, 14:11.640 --> 14:18.520 especially on the storage sites. We decided that we want to use 14:21.240 --> 14:27.000 it's going to like this large scale or largely scale infrastructure and for that we need it 14:27.720 --> 14:36.920 relatively well established provider. We need to do quick scale outs because frankly, 14:37.000 --> 14:43.480 we can provision infrastructure preemptively but there is not a real point to do that. 14:44.280 --> 14:50.360 The later you do it, if you do it in the cloud as we do, the less money you will pay. 14:52.520 --> 14:59.960 We needed quite good support, at least mostly in terms of networking, troubleshooting and security. 15:00.840 --> 15:05.800 We require multiple locations so we can easily fail over and of course, 15:05.800 --> 15:11.400 security and details protection are almost for this kind of world we live in. 15:13.560 --> 15:19.800 So, the components, if you cannot tell from the picture, it shows GCP, 15:21.640 --> 15:27.800 the components are relatively simple. We have a control plane that's running on the compute 15:27.800 --> 15:35.480 engine on the bottom. We have a cloud test kill in the form of postgres that's our DV layer. 15:36.840 --> 15:42.760 We have a video on the month that's passing mostly through cloud services. 15:44.520 --> 15:49.000 The storage is on persistent disks. We use a shared cache, 15:49.960 --> 15:57.880 reducing effects. We have on ingest and terestream clusters that are again running on top of compute 15:57.880 --> 16:05.160 engine and then we're using the firewall options that GCP provides us. 16:06.920 --> 16:12.600 Probably the first interesting note is the ingest notes. We try to run in with 16:12.600 --> 16:24.200 multi-rtimp and then we set out on media antics. Our main goal usually is to try to use as much 16:24.200 --> 16:31.960 open source software as possible and to contribute back whatever we can. In our case for the 16:31.960 --> 16:40.200 application server, we're using engine x and topon resting. For our team p ingestion, we're using 16:40.200 --> 16:51.400 media antics now before we were using multi-rtimp and for processing control and some communication 16:51.400 --> 17:00.600 we're using rediscellary and FMPEC. The streamer note is extremely simple. 17:00.600 --> 17:05.800 Engine x and open resty serving static files even though they are 17:06.680 --> 17:11.560 HOS generated. It's absolutely fine. Basically we're streaming HOS. 17:13.320 --> 17:17.560 The protection of the assets themselves so not to allow 17:18.440 --> 17:27.880 easy way to do those ourselves is done in a layer approach. We're doing caching on the HOS chunks 17:27.880 --> 17:33.000 in memory for improved performance and to lower the note from the ingest notes. 17:33.880 --> 17:40.840 The control plane is in-house developed mainly in PHP. It handles internal eventing, 17:43.960 --> 17:48.360 provisioning of the elections themselves, provisioning of polling places, 17:48.360 --> 17:55.960 defaults, controls, who goes where who can stream what keys should be used and so on. Basically 17:55.960 --> 18:02.200 that's the thing that is making the process possible. The other things around it are just talking 18:02.200 --> 18:10.200 to it, getting data and doing it's bidding. The last part of the process is the data retrieval 18:10.200 --> 18:17.720 because as you remember we have also a local recording and sometimes there could be no online 18:17.720 --> 18:24.360 recording because there is no connectivity, any kind of connectivity. The data retrieval itself 18:24.920 --> 18:31.400 is super simple. Unpack the phone, plug it to a USB-C and application written in 18:32.760 --> 18:42.040 open Pascal. Pascal. Application written in Pascal just downloads whatever is needed from there. 18:43.080 --> 18:52.840 Quite simple rules. Again we have a very involved processing chain that's mostly concerned 18:52.840 --> 18:59.480 with making absolutely sure that the files are good, right and proper. If they are broken files 18:59.480 --> 19:04.200 we try to automatically fix them. If not they go to broken and somebody has to fix them manually. 19:05.400 --> 19:13.320 The files need to be arranged somehow in a file structure. There are a lot of manual checks, 19:13.320 --> 19:20.360 a lot of process of manual processes and does the reason why we have to verify this. If you have 19:20.360 --> 19:29.560 a 12,000 devices in a single room, go find number 42. If you don't have RFID, it's a total mess. If 19:29.560 --> 19:38.840 you have RFID, it's a gain mess but not total. Just takes a little time. Other interesting pieces 19:38.840 --> 19:44.360 of the puzzle or things that we learned, don't reinvent the wheel. Try to use platform services 19:44.360 --> 19:50.440 if you are using cloud stuff. Do test, do automate and always have a very good backup plan. 19:52.120 --> 19:59.560 Implement pre-flight checks. If possible, automate them, run them twice and then get somebody 19:59.560 --> 20:09.320 to look over them manually. What we use, Google Cloud Services, Ansible, Terraform to stand up 20:09.320 --> 20:19.880 the infrastructure cloud flare for DNS and some white caching. That's it. Google Cloud Platform, 20:19.880 --> 20:27.400 the GSP provides us very nice services in the form of firewall caching and global distribution 20:27.400 --> 20:35.640 of facets and if you have such use case it's quite nice. Of course they are issues as always. 20:35.720 --> 20:46.200 The provisioning of the devices is manual, the upgraded self is a gain manual. Some things 20:46.200 --> 20:53.640 can be automated but most no. If you start too many devices at single, at a single point of 20:53.640 --> 21:03.640 time, all the telco providers will be very happy with you. You can see phones going to the 21:03.720 --> 21:11.320 base station for miles and miles and miles away in a city environment, which is not really nice. 21:13.000 --> 21:22.520 Even though we do training videos, we print materials that are used during the process themselves. 21:22.520 --> 21:28.840 The people don't use them, don't read them, forget them, turn the phones upside down. That's one 21:28.840 --> 21:34.200 of the reasons why we implemented self monitoring of device positioning. If the device is not 21:34.200 --> 21:41.720 positioned properly, it will talk back to you. The first time you hear this, it's quite scary. 21:45.320 --> 21:54.680 People really like the devices. The package that we ship is phone stand and instructions. The 21:54.760 --> 22:02.760 stands are some Chinese brand and they require very deep technical knowledge how to operate them. 22:02.760 --> 22:10.360 Sorry that that's literally true. They require technical knowledge. On the process of downloading 22:10.360 --> 22:16.680 the files from the phones, choose quality cables. That will save you a lot of grief. 22:17.320 --> 22:26.040 Truth-server-vide stickers with nice antennas. We can recommend the UHF ones, not the NFC ones. 22:28.040 --> 22:35.240 And that's mostly it's what next. We have a hundred and twenty-terabytes of videos in Bulgarian. 22:37.240 --> 22:42.280 That's some options that we talked about. I don't know if you can think something else. 22:42.520 --> 22:49.800 Talk to us. We'll try to make some deal to get to you that data. 22:51.800 --> 22:57.560 For example, yesterday I saw that the VOC guys are trying to do live transcriptions with AI. 22:58.440 --> 23:04.920 Maybe we can try that too. Questions? The QR code leads to a feedback page. 23:05.560 --> 23:08.680 That's my email address. Go for questions. 23:09.080 --> 23:11.080 Yes. 23:11.080 --> 23:15.880 For how long do you have to keep the videos and you have to keep them available for people for the 23:15.880 --> 23:26.920 whole time? The question was how long should we keep the HEF to keep the ERT facts basically? 23:26.920 --> 23:35.240 The streams and the recordings from the device itself. The law does not strictly define that. 23:39.640 --> 23:47.160 If you want, I can try to find the line in the law that starts at the whole process, 23:47.160 --> 23:49.240 and it's less than two sentences. 23:51.400 --> 23:52.600 Next. 23:52.600 --> 23:53.160 Yes. 23:53.160 --> 23:58.440 I didn't follow how RFID helps you out when you have to have the devices in one move. 23:59.800 --> 24:04.520 How RFID helps with too many devices. So if you are searching for a device, 24:04.680 --> 24:11.880 and it's in a big pile of other devices, the RFID helps you to narrow down where exactly to look. 24:13.720 --> 24:20.920 The folks, we pack them in boxes, large boxes, 60 phones per box, in their own box. 24:22.120 --> 24:28.520 So basically you have a stack, and then okay, this is in this box, pull it out. 24:28.520 --> 24:32.200 In this section of the box, let's start taking out the device. 24:32.760 --> 24:40.120 The other way around this, pull every four and three, the barcode with the unique number on it. 24:41.400 --> 24:43.160 Okay, next. Yes. 24:51.800 --> 24:55.240 So the question was, back up plans, what kind of back up plans, right? 24:55.960 --> 24:59.800 Okay, so first back up plan, walk a recording. 25:00.760 --> 25:11.480 Second back up plan, keep the life recordings on the cloud instances for some amount of time on top of that. 25:12.760 --> 25:21.720 Third back up plan, keep the recordings in a bucket, because string from a bucket is kind of a requirement for the 25:21.800 --> 25:23.080 serious service that we use. 25:25.080 --> 25:30.280 And then after you download everything down on the local storage and check everything, 25:30.280 --> 25:33.800 just after that delete everything. And yes, we had to use that. 25:35.800 --> 25:36.520 Yes. 25:36.520 --> 25:41.160 Two more good questions. One, but there's no penalties if someone doesn't turn on. 25:41.160 --> 25:44.840 Is there something on the first slide? Is that a question? 25:44.840 --> 25:51.080 So they have what 27 or 20 important requirements of data? How do we be the multi-sync activist 25:51.080 --> 25:57.880 watch it? You have 50 court cases? Did they call it themselves in a room for one more time? 25:57.880 --> 26:06.360 Okay, yes. Okay, so first question, sorry, I forgot the first question. 26:06.360 --> 26:10.760 Penalties, yes. The first question was, what happens if there are no penalties? 26:10.840 --> 26:17.320 Well, there are no penalties, nothing. People can not turn the phone, can break it, 26:17.320 --> 26:22.680 can do whatever they want. I think that that will be changing shortly, though. 26:25.000 --> 26:33.720 The second question was, how do the democracy activists watch this? In at least what we know is 26:33.800 --> 26:42.840 two ways. The first way is about downloads everything. There is one specific person that we know 26:42.840 --> 26:55.800 is doing this. The second option is, the polling places, there is statistics where there are 26:55.880 --> 27:03.000 suspected forgeries, fraudulences and so on, and those are the most monitored. 27:04.520 --> 27:14.840 Usually that happens by an organization that says, okay, we need 50 people. You are going to 27:14.840 --> 27:25.000 monitor this five, this X amount of stations. Otherwise, as we said, 20 to 40 terabytes of 27:25.080 --> 27:33.480 video, that's roughly seven or half to eight years per election cycle. So single person, it's hard. 27:35.400 --> 27:43.400 Otherwise, we found out that people mostly watch the polling places around them. 27:44.920 --> 27:52.280 To see how the process works, and there are a lot of people that are doing that and just jumping 27:52.280 --> 28:02.840 through the video, and some things get captured just by chance. Okay, there was another question. 28:11.880 --> 28:20.040 I'm not sure. I have talked with some Polish guys during my work, not specifically on that project. 28:20.520 --> 28:29.880 They were telling me that something about this was thought about in Poland, but was not accepted. 28:29.880 --> 28:32.840 Otherwise, sorry, I don't have information. 28:33.480 --> 28:41.320 I have a couple of words about storage. Did you use something as thick and potable or just huge 28:41.320 --> 28:51.080 files share of each note? The huge file share, not going to work. I always won't work. 28:51.640 --> 28:58.840 We used local persistent disks, and after that, we also told them to GCS, and from GCS, we get them down. 29:03.800 --> 29:10.680 Do you have primary pick-up stream, or just a primary pick-up stream, or just a primary pick-up stream? 29:13.240 --> 29:20.520 The device is the foes themselves due RTP, and then to the outside world, we do HLS, 29:21.400 --> 29:28.680 any Anto streamers can reach any Anto in just notes, so it doesn't really matter where you hit it. 29:33.560 --> 29:35.560 How much does it cost? 29:36.120 --> 29:39.960 Estimation about the whole project, not sure. 29:47.800 --> 29:51.560 Okay, let's try to do a simple estimation. 29:51.960 --> 29:52.920 Hmm. 29:59.400 --> 30:04.040 Okay, 5 to 10 million, maybe. I will really depend on... 30:04.040 --> 30:14.680 Well, thank you. Plus, the hardware that will be installed, is every election after this. 30:14.680 --> 30:24.520 It's much cheaper than the cost. It's enough to be able to buy another devices and develop some. 30:25.320 --> 30:36.280 Okay, so let's repeat it for the stream itself. About 2.5 million Euro, plus the devices cost. 30:37.240 --> 30:43.640 The costs are very high upfront because development testing can solve and so on, but after that, 30:44.280 --> 30:50.920 running kit for each and every cycle, it's much much more cheaper, up until we need to replace the devices. 30:51.960 --> 30:55.240 The devices themselves, the Android foes, where something like 30:57.720 --> 30:59.720 100 Euro, 31:00.120 --> 31:03.560 64, I'm completely out of numbers. 31:04.040 --> 31:14.360 60-old Euro port device, so the devices themselves were even not that expensive, but that was one of the reasons 31:14.360 --> 31:19.640 why we were looking for such. Do we have a time for another question? 31:22.680 --> 31:23.880 Okay, so go. 31:23.880 --> 31:29.400 Yeah, the phone from the devices, they do not listen if you're with battery spawning, 31:29.400 --> 31:32.520 because I would expect that's gonna happen after. 31:32.520 --> 31:38.920 Did we, so the question is, did we notice any battery-related issues? 31:40.120 --> 31:48.600 Up until now, not really, we found during the unpacking and downloading the device data, 31:49.160 --> 31:53.640 we found several phones that were quite hot because they were left on. 31:56.200 --> 32:05.480 We had total failure rate of less than half a percent, maybe, even less. 32:05.480 --> 32:15.480 Something like less than 50 devices have failed, and some of those have failed by being 32:15.560 --> 32:21.000 repeatedly hit, or fallen, or rock, or something like, basically, physically that. 32:23.160 --> 32:32.120 Otherwise, no, not really, but, well, you have to have in mind that those devices are not 32:32.120 --> 32:34.040 actively used and abused. 32:34.040 --> 32:40.280 Well, that's what I would expect to be from, because when you keep the charge battery in a storage 32:40.760 --> 32:47.000 and leave it, then you kill as well, and, you know, I'm not so good at battery soap, 32:47.000 --> 32:49.160 I don't really know how battery is behaved. 32:49.160 --> 32:52.360 I recommend to our splash by slowly. 32:52.360 --> 32:54.840 Okay, will you check? Anyone else? 32:54.840 --> 32:58.360 We have a lot of connections, but there is no time to go. 33:03.560 --> 33:04.360 Anyone else? 33:07.320 --> 33:07.800 Yes, sure. 33:10.840 --> 33:13.960 How do you manage the inventory of 7.2? 33:15.480 --> 33:17.640 How do we manage the inventory? 33:17.640 --> 33:19.400 Well, thank God for the RFID. 33:21.240 --> 33:26.200 Otherwise, the inventory management is built into the control plane. 33:26.920 --> 33:28.600 It has an inventory module. 33:29.560 --> 33:33.160 It's nothing too fancy, nothing too special. 33:34.360 --> 33:39.560 Probably, if the person is asking about physically managing the inventory, 33:40.520 --> 33:42.120 that's another question. 33:42.120 --> 33:44.120 It's not so easy. 33:46.120 --> 33:52.680 But 12,000 folks are also not such an extreme volume. 33:53.400 --> 33:57.400 It's something like 8 to 10 cubic meters. 34:04.280 --> 34:07.960 Yeah, sorry, we didn't took a photo of this, 34:08.840 --> 34:12.280 but on the next cycle, I promise I will have a photo. 34:13.320 --> 34:14.040 Anyone else? 34:16.600 --> 34:17.320 Okay. 34:17.880 --> 34:21.560 And you mentioned you used WiFi, you have a project. 34:22.280 --> 34:24.760 Did you have some issue of this mobile project? 34:25.800 --> 34:29.880 I mean, some problems with your connectivity in some places. 34:29.880 --> 34:33.480 Well, yeah, okay, so the question is, 34:33.480 --> 34:37.640 did we have any connectivity issues, specifically with 4G? 34:38.920 --> 34:43.160 Do you know how many different labs, 34:43.160 --> 34:51.640 well, labs, companies, sites, there are that are claiming best network in something? 34:53.320 --> 34:57.000 A lot of, well, there are at least three because each and every 34:57.000 --> 35:00.040 token provider is using different ones and everyone is saying 35:00.040 --> 35:01.320 that's what we have the best network. 35:01.640 --> 35:03.640 That's it. 35:05.240 --> 35:10.840 On the first cycle, there were some issues that on the second cycle got corrected. 35:13.160 --> 35:17.320 What did the operators themselves, the telcos themselves are doing? 35:17.320 --> 35:22.440 I don't know, but probably they are doing either prioritization 35:22.440 --> 35:25.400 or just physically moving cell towers. 35:25.960 --> 35:36.840 At least some years ago, I know that each and every telco provider had at least 12 mobile 35:36.840 --> 35:41.400 cells to deployment either due to demand or brokerage. 35:45.640 --> 35:46.920 Okay, anybody else? 35:50.040 --> 35:51.400 Then, thank you. 35:55.400 --> 35:57.400 Thank you.