WEBVTT 00:00.000 --> 00:18.760 Hello everyone, I'm Juliana Sims. I work at the Spritly Institute where we're working 00:18.760 --> 00:24.080 on a lot of cool social web stuff. My colleagues are going to have some talks in this room 00:24.080 --> 00:32.200 after me so you can hear more about that then. This work is funded by the NGI-0 and trust grant 00:32.200 --> 00:37.080 through NELNET, so it is not actually Spritly Work, but it is using Spritly Technology. 00:37.080 --> 00:42.840 I'm going to talk to you about how I am. I'm buttering Ludovic's precious shepherd. 00:42.840 --> 00:51.320 Now, I'm integrating the Goblin's library with it. I feel like I had a better segue than 00:51.320 --> 01:01.320 that, but yeah, so I'm porting the shepherd to Goblins. Who here is heard of the shepherd? 01:01.320 --> 01:06.320 If you haven't, go back in time 30 minutes and you'll learn everything you need to know. 01:06.320 --> 01:09.640 So yeah, I'm porting the shepherd to Goblins. I don't really need to explain what that 01:09.640 --> 01:16.560 is and its system, process manager, written in Giles game uses fibers to implement a concurrency 01:16.560 --> 01:25.320 system derived by the actor model. Goblins is a Giles library that implements the object 01:25.320 --> 01:30.960 capability security model, which provides security guarantees as the name would suggest, 01:30.960 --> 01:37.520 and it also uses the VAT model of concurrency, which is similar to the actor model with 01:37.520 --> 01:43.120 some variations. Again, my colleague Jessica Talon will explain that a little more, I think, 01:43.120 --> 01:48.120 later today, so I don't have to go into too much detail there. 01:48.120 --> 01:52.960 And I'm just going to be talking about what this project is, what its goals are, some benefits, 01:52.960 --> 01:58.800 where it's at, where it's going. So, first of all, what we're actually doing is discussed 01:58.800 --> 02:04.640 in more detail. I have a design dock up. That's a GitLab repo I use because of, I like 02:04.640 --> 02:09.360 the force push and savannah makes that tricky. savannah is where the official shepherd repose 02:09.440 --> 02:15.040 stored, but the Goblins port design dock, if you see all the way down at the end of the URL, 02:15.040 --> 02:21.040 you can go either to the GitLab repo under Spritly or the official savannah repo on the 02:21.040 --> 02:26.800 WIP, Goblins of FI branch, and you can read the design dock for more detail. It's kind of 02:26.800 --> 02:31.040 a living document. I think a couple of the things I said in there, I've already decided I want 02:31.040 --> 02:37.200 to change, but it'll give you an overview at a little more lower level of what's happening. 02:37.200 --> 02:42.320 And basically what's happening is, I'm taking all of the bespoke shepherd actors that 02:42.320 --> 02:48.560 Ludific has worked so hard to make what you're very nice to work with, it should add. 02:48.560 --> 02:54.000 And I'm changing the multi Goblins actors, and I have a little joke here, but calm is a 02:54.000 --> 02:59.760 Goblins thing. So, instead of using CSP and implementing actors directly, we take advantage 02:59.760 --> 03:05.520 of the Goblins VAT model and our actors, and the concurrency are managed by Goblins. 03:07.200 --> 03:11.280 There are other code changes necessary to sort of translate between how Goblins does things 03:11.280 --> 03:15.840 and how traditional scheme does things. I'll mention a little more later, but that's the 03:15.840 --> 03:21.600 core of the work is just translating code into a slightly different format. And this has some benefits. 03:21.600 --> 03:25.440 So, first of all, it makes a lot of bugs, Spritly's problem. A lot of the concurrency issues. 03:25.440 --> 03:30.400 Ludific was just talking about our supposed to be handled in Goblins, and so if we run into 03:30.400 --> 03:36.000 some of those at least, I've realized not all of them, but at least some of them, that's a bug 03:36.000 --> 03:42.560 in Goblins and not in the shepherd. I think I hope that it will reduce the required knowledge 03:42.560 --> 03:48.960 for developers to contribute because you won't need to understand the actor model and concurrency 03:48.960 --> 03:55.200 sequential processes and how it's implemented. You'll just need to understand how you use Goblins, 03:55.200 --> 04:02.880 and I'm hoping that will help. And it will also provide a distributed debugger. I threw that 04:02.960 --> 04:07.520 point in there at the end of Ludice's slide when he was talking about debugging. I don't know 04:08.960 --> 04:13.920 exactly what that's going to look like yet, because I'll get there. And yeah, hopefully it'll 04:14.960 --> 04:23.760 produce more ergonomic code because a lot of Goblins' asyncronicity and concurrency looks just like 04:23.760 --> 04:29.040 synchronous code. So, here's a little bit of an example of the code itself, and it's just an example of 04:29.760 --> 04:36.160 the code for starting and stopping a service. I think the shepherd equivalent, it's in the service 04:36.160 --> 04:43.920 controller. I'm merging different parts of the code base. So, here's the before, and I didn't test 04:44.720 --> 04:51.920 if you could see all of this, but as you can see, we have, this is a branch for a match statement, 04:51.920 --> 04:57.440 so I just pulled it out to give you some context. We're creating a channel, spawning a fiber to 04:59.520 --> 05:03.760 tell another part of the program that the service has started, and then we're doing some 05:03.760 --> 05:10.720 bookkeeping announcing to users what's going on. And so you have all of this code for mostly bookkeeping, 05:10.720 --> 05:17.600 and then we have this code, which does pretty much the same thing. Instead of using a channel, 05:17.760 --> 05:24.240 as we see here, and during get message and all of that, we instead use the Goblins concept of a promise, 05:24.240 --> 05:29.360 and so we spawn a promise, and then we return a resolver, and the caller will, 05:30.400 --> 05:37.280 is handling the actual, starting of the process, and will fulfill that, and so that's an example 05:37.280 --> 05:43.760 of what this code will look like in practice. This let on form is a macro around the Goblins on, 05:44.560 --> 05:51.760 I think that's a procedure, that handles promises, and I think I talked about promises a little bit, 05:52.240 --> 05:55.840 they're like JavaScript promises, again, hang around, you'll learn more about that, 05:56.560 --> 06:03.840 but their mechanism for concurrency and asynchronous code evaluation. And so this will not just 06:03.840 --> 06:11.440 change the code base, right? It will have benefits for users. One of the local level things, 06:11.440 --> 06:18.240 so if you have your machine and you are a single user and you have your shepherd, 06:19.040 --> 06:25.600 one of the benefits is that it will give you the ability to control your user shepherd services 06:26.800 --> 06:31.120 through the system Damon, the way the shepherd works right now, you spawn a separate shepherd 06:31.120 --> 06:37.120 Damon under the control of the system Damon to manage your user level services, and 06:37.120 --> 06:47.200 this work the certain aspects of the Ocaps paradigm enables you to just control the system 06:47.200 --> 06:52.960 shepherd, but only the aspects of the, or the only, only the specific services that you spawn 06:52.960 --> 06:59.200 yourself that you know about. And this will also give us the ability to have different shepherd 06:59.200 --> 07:03.760 Damon's on different machines talk to each other. That's sort of the headline feature, the thing 07:03.840 --> 07:09.600 that I'm most excited about and got most excited about in this project. And this will use the Ocaps 07:09.600 --> 07:14.640 and protocol, which Goblin's implement, and it allows you to abstract away the network layer, 07:14.640 --> 07:20.480 so you can connect your, your shepherd Damon's, your machines, over arbitrary network boundaries. 07:22.080 --> 07:27.200 I mentioned plan 9 because if I don't mention plan 9 in relationship to this project, I get sad, 07:27.200 --> 07:30.960 because I think plan 9 is very cool, and that's a big part of what inspired me to want to work on this. 07:31.760 --> 07:38.080 And it will also facilitate existing deployments of the shepherd on, as the example here, says, 07:38.720 --> 07:44.240 high performance computing clusters, HPCs, which I usually call super computers, and I think that's 07:44.240 --> 07:50.560 incredibly cool. So we're going to, uh, sprightly is going to implement a, uh, UNIX domain socket 07:50.560 --> 07:55.920 net layer, so that the existing networking infrastructure that HPCs use will continue to work over Ocaps 07:56.800 --> 08:03.040 and, um, I sort of have this, like, last point of pointing towards things this can be useful for 08:03.040 --> 08:11.120 that it isn't currently deployed for, because, uh, I'm not aware of any system quite like this, 08:11.120 --> 08:17.680 and so, uh, one, one idea that comes up is orchestrating your, your networks of servers and managing 08:17.680 --> 08:23.360 services on remote machines from maybe a central, uh, uh, machine, because you'll be able to start 08:23.360 --> 08:28.880 and stop and register and control services from any machine, and the really fun thing is that 08:29.440 --> 08:33.280 the user local benefits and the network benefits are the exact same thing, because Goblin's 08:33.280 --> 08:38.800 allows you to abstract away the network and it allows you to write code that operates locally 08:38.800 --> 08:46.160 and over the network and looks exactly the same. Uh, so I'll get to showing you that code in a second. 08:46.720 --> 08:51.200 I should probably have put that first, but just as a click overview of where we are on the project 08:51.280 --> 08:57.760 right now, uh, not quite as far as I would have hoped to have been, life and the world and things have 08:57.760 --> 09:03.600 slowed me down, but, um, so we have the core service registry ported to Goblin's, we have, um, 09:04.880 --> 09:08.960 the, I shouldn't have said core with that one. We have the service registry ported, which is what 09:08.960 --> 09:13.120 actually keeps track of the service object. It's a bookkeeping device essentially, but it's a 09:13.120 --> 09:18.240 great point of abstraction, um, for a lot of different functionality, and we have the core service 09:18.240 --> 09:24.320 actor, which represents the actors. I've merged the controller and the service. I think I'm probably 09:24.320 --> 09:29.520 going to separate those again, which I'll again discuss later. I still, in terms of porting this 09:29.520 --> 09:35.360 core service actor, I still need to get actions. Um, Shepherd has this cool ability where, you know, 09:35.360 --> 09:44.400 you type her status or you type, um, I'm blanking on examples, but you can send commands to the 09:44.480 --> 09:49.600 services and Shepherd allows you to define arbitrary commands and these are called actions. 09:49.600 --> 09:54.720 So I need to port the functionality for that. I've been putting it off because I kind of want to do 09:54.720 --> 09:59.040 some trickery using Goblin's. We'll see if I actually do that. And I also need to do the actual process 09:59.040 --> 10:03.840 management. It was funny that Ludific was talking about how hard can it be to spawn and forks and 10:03.840 --> 10:09.280 processes and keep track of a signal, um, I've been thinking the same thing and I'm probably 10:09.360 --> 10:14.480 overest of any myself, but you know, and then I've started documenting these code changes and what the 10:14.480 --> 10:20.480 API could look like for users. So we're about to see a rough example of what the API could look like 10:20.480 --> 10:27.760 for users. And this is some code I'm using to test that things work. I've borrowed services 10:27.760 --> 10:32.720 from the Shepherd's Test Suite and torn them out since I don't have the full system set up to 10:32.720 --> 10:39.360 give you a quick run view of what's happening. This is all, um, boilerplate. It's going to be 10:39.360 --> 10:43.040 handled in the Shepherd Proper. You don't have to worry about it. And we see here we're registering 10:43.040 --> 10:47.440 some services. We're only worried about the test service and test two service today. 10:48.480 --> 10:53.680 Essentially test service creates a file and puts, uh, I think, food in it. Test two service 10:53.680 --> 10:57.920 changes the text in food to bar and when either of them stop they delete that file. 10:58.880 --> 11:05.280 And so here we see, so we register the services and then, um, once they're registered, we ask the 11:05.280 --> 11:12.000 service registry to send us a list of registered services and then we operate on to the services. 11:12.000 --> 11:18.720 This API is strongly subject to change. It's sort of, uh, as I'm trying to figure out how 11:18.720 --> 11:22.480 these various pieces will fit together and doing that so you can ignore the details of that bit. 11:22.480 --> 11:26.080 It's not important. And then you see down here, once we've registered everything, we have 11:26.080 --> 11:30.800 letons again because we're doing this asynchronously and we're starting service two, which depends 11:30.800 --> 11:36.080 on service one. And then we're stopping service one, which should because it is a dependent of service 11:36.080 --> 11:41.520 two stop service two. And the arrow operator is really important because that is an asynchronous, 11:42.400 --> 11:48.800 uh, message to these service actors. And that arrow means that this could be over the network 11:48.800 --> 11:54.240 as well as locally. And then we have some promised stuff to deal with waiting for things to print. 11:54.240 --> 12:02.480 So let me show you this in action. It's not, uh, super exciting, but 12:03.920 --> 12:09.440 as you can see here, this is the list of services. We have four services, a little bit bigger, 12:09.440 --> 12:17.600 absolutely. Is that good? Baker. Um, so we have the list of services there and then we have 12:17.600 --> 12:21.200 a little message telling us that we're starting these services and that we're stopping these 12:21.360 --> 12:26.640 services. And it's all working over goblins. And these services could be on this machine, 12:26.640 --> 12:31.280 as they are, in this case, where they could be on a completely different machine just as easily. 12:32.480 --> 12:37.360 Um, I think I'm running lower on time than I thought I would be. So I might skip this part. 12:38.240 --> 12:43.840 There's some issues around the port with fibers and goblins, translating promises to values, 12:43.840 --> 12:47.760 goblins loves promises. They're very similar to JavaScript promises. The actual um, 12:48.720 --> 12:53.840 the Ocaps promises inspired JavaScript promises. Um, some issues around compile time. 12:53.840 --> 12:58.080 Object construction because garlics to compile things into bytecode ahead of time. And if you 12:58.080 --> 13:01.840 try to define an object, which is a lambda at the compile time it causes issues. 13:02.480 --> 13:08.560 World events have made this difficult. Um, I probably was wrong about being able to merge the, uh, 13:08.560 --> 13:14.160 actor controller and other parts of the actor. And then there's a lot of things I have to not do. 13:14.160 --> 13:18.320 But I think it would be cool. Uh, Ocaps likes process isolation. You want to spawn your 13:18.320 --> 13:24.240 services in a cold dead world, as we say. And that's sort of out of scope serialization, 13:24.240 --> 13:28.400 which is a feature goblins provides where you can serialize your actor objects. And then you can 13:28.400 --> 13:33.280 revive them. And it's the same actor object. You can upgrade them. All this stuff. That's out of scope. 13:33.280 --> 13:37.520 Um, I tried a little bit to switch replacements to using become, which is a thing where an actor 13:37.600 --> 13:44.720 becomes a different kind of actor. That didn't, um, prove necessary capability grants is where you 13:44.720 --> 13:49.920 handle objects to other places. That might come in some, but anyway, I just get really excited about 13:49.920 --> 13:55.200 problems. I think it's great. Uh, so the next steps finish up what I've started on, essentially, 13:55.200 --> 14:00.720 and refine the public API, uh, make sure that, you know, you don't have to use add-up to 14:01.440 --> 14:06.720 actually access your objects that there's a, uh, traditional scheme values API that you can use. 14:07.520 --> 14:12.640 To do stuff, um, that's the talking about service references, how users are going to be able to 14:12.640 --> 14:18.400 refer to their services in their configuration files, um, getting problems to the values. And right now, 14:18.400 --> 14:23.200 like I said, I've copied like parts of a test file for testing all of this, and I want to get proper 14:23.200 --> 14:27.840 testing. And then in terms of functionality, the things that are left, we need service actions, 14:27.840 --> 14:33.440 we need spawning and monitoring, uh, integrating the CLI, and then it'll just mostly be a matter of 14:33.440 --> 14:38.160 exposing the new functionality the goblins provides, and I don't know how complex that'll be. 14:38.160 --> 14:43.840 But we'll see. So that's it for this talk about chef and goblins. Um, 14:43.840 --> 14:48.640 spritally is going to be talking more about their work in this room after me. So stick around if you want 14:48.640 --> 14:54.160 to hear more about that. And, uh, thank you to Enelmet for funding this work. Thank you to the organizers 14:54.160 --> 15:11.760 of this room. And, yeah. Do we have time for questions? So, yes. 15:11.760 --> 15:29.280 Yeah. Yeah. So that was just, uh, stay repeat the question? Yes. Um, because you can, uh, control 15:29.280 --> 15:33.600 and monitor services on different machines, you can have dependencies across the network boundary. 15:33.680 --> 15:50.720 Yeah. Yes. Yes. So, uh, the, the question was given that you can have machines, uh, 15:50.720 --> 15:55.360 uh, dependencies on different machines, could you end up with different shepherd damans having a different 15:55.360 --> 16:00.080 perspective of the network. And the answer to that is yes. That is something I'm thinking about 16:00.080 --> 16:04.160 constantly in the back of my mind because I'm not sure exactly how to handle that. You could 16:04.160 --> 16:09.360 imagine, for example, that you have an application which needs a SQL database. And you don't care 16:09.360 --> 16:13.440 what the SQL database is. You have some abstract API you're using. It could be anywhere on your 16:13.440 --> 16:19.760 network. Um, so how do we make sure that you're getting the right kind of SQL database? That's sort 16:19.760 --> 16:23.440 of thing. So yeah, this is something I'm actively thinking about. It's one of the big issues that 16:23.440 --> 16:30.000 needs a little bit more design like on it. Any other questions? Oh, it looks like we have 16:30.080 --> 16:38.640 you get switched to the microphone. Um, yes. Um, um, I may have such a problem with this, but um, 16:38.640 --> 16:46.320 with this part, are there changes, shepherd's public API? Okay. So the question was, will this 16:46.320 --> 16:52.960 port change shepherd's public API? The public API will remain backwards compatible, as well as 16:52.960 --> 16:57.760 the protocol for communication between the herd and shepherd command lines and the shepherd 16:57.760 --> 17:04.160 damen proper. So all of your code will continue to work exactly as it has. Um, uh, I would like 17:04.160 --> 17:09.200 to expose a new API that allows you to take advantage of some of the nice cities of goblins, but 17:09.200 --> 17:26.960 I might not. We'll see. We'll see how things go. Yeah. Yes. Yeah. So the question was about my 17:27.040 --> 17:33.360 mention of plan nine and asking if machines will be able to use physical parts of another computer, 17:33.360 --> 17:39.280 the example is an audio card. Um, the answer is that this work will not enable that, but you could 17:39.280 --> 17:44.400 potentially use it to set up some sort of system that would do that. So that's not within scope 17:44.400 --> 17:50.480 for this project sharing physical resources, but um, I can imagine some ways you might be able to 17:50.480 --> 17:57.200 make that happen. Yeah. I don't see any more questions. So I'm going to hand it off. Thank you 17:57.200 --> 18:00.320 all so much. Thank you for that today.