WEBVTT

00:00.000 --> 00:10.880
Okay so hello everyone nice to see a bigger room than last year was more convenient

00:10.880 --> 00:19.440
last year because it almost happened so basically shortly about me I'm pushing some

00:19.440 --> 00:24.080
stuff to get up you can find some repos in the presentation with some links to the

00:24.800 --> 00:32.560
things that they do basically I want to discuss topic how we got here and from the point of

00:32.560 --> 00:38.000
you of how the whole ecosystem looks at least from my perspective how I feel it looks like

00:40.080 --> 00:46.080
and then I would like to show a little bit about story how I came from Java to closure

00:47.120 --> 00:51.200
you're actually moving glass of developers from the Java to move to closure so they all go into

00:51.200 --> 00:58.000
the same story so I want to tell you what was my motivation before the things that I'm doing

00:58.800 --> 01:03.600
then I'll talk about a little bit of architecture patterns this comes back to the library from

01:03.600 --> 01:09.600
the I presented last year and so last year we were almost at basically entering to production and

01:09.600 --> 01:14.480
now we're only one year in production and we learn lots of stuff and we figure out lots of

01:14.480 --> 01:21.440
stuff that how things can be done that works for us and I'll talk about you know I will

01:21.440 --> 01:27.280
experience in production with with our library that we have built and then I'll hopefully write some

01:27.280 --> 01:33.120
conclusion I said most of the stuff is based my experience if you if you think something I'm telling

01:33.120 --> 01:37.440
you will not fit your environment I would have be happy to discuss with you later on so we are

01:37.440 --> 01:44.400
basically building application which has around 10,000 active users and about 15 to 30 maybe up to

01:44.400 --> 01:54.240
100 million records so basically this is the scope where this is built so what I have figured out

01:54.240 --> 01:59.200
that lots of people are confusing simply these you know like we have all this nice framework

01:59.200 --> 02:05.360
hello world they look so beautiful everything is nice in on presentation but I don't think the

02:05.360 --> 02:12.400
people until they actually start working on something feel the complexity behind it so and I see

02:12.400 --> 02:17.760
everywhere people I just designing systems to which I put from check boxes and it reminds me you know

02:17.760 --> 02:24.240
on academia they are usually doing these papers where they are basically have to fit pick the

02:24.240 --> 02:28.960
topic that is kind of popular but not too too strange so they can actually get a funding and I see

02:28.960 --> 02:33.760
we have the same thing in software happening over and over again that people there are so much

02:33.760 --> 02:38.320
architectural patterns in the company they have to put them check boxes to get the project approved

02:38.320 --> 02:42.160
and then they do stuff and then start to in the end figure out how to actually implement a

02:42.160 --> 02:49.280
business logic because they actually never even took it to consideration before and from from my

02:49.280 --> 02:54.480
perspective at least we have so much library so much stuff happening and it's all like horrifying

02:54.480 --> 02:59.840
when I see what people are pulling into the project just to complete some task which looks like

02:59.920 --> 03:07.440
super simple to complete so from me this faulty looks very very horrifying and the other consideration

03:07.440 --> 03:12.960
which came very in focus at least over last year especially now with door are coming in place

03:12.960 --> 03:21.440
is the security and here what at least my fear is but also I mean I also the Uncle Bob was talking

03:21.440 --> 03:25.120
about it basically that you know we will come to the point where something horrible will happen

03:25.120 --> 03:29.200
and we will be blamed as a developers and then government will start regulating what we are

03:29.280 --> 03:34.560
actually using for development and recently I was article that basically US government is saying

03:34.560 --> 03:39.440
engineers to stop using CNZ plus plus for development which is for me I mean it's not like we are

03:39.440 --> 03:44.480
using this stuff because it's fun we obviously have use cases we are using it at least I hope so

03:44.480 --> 03:52.240
but yeah coming back to the first point it is also questionable so when I started coding

03:52.240 --> 03:56.640
I was always like this open source and the decision it uses Java E was perfect everything

03:57.520 --> 04:05.040
I love it so I use lots of spring boot at that time and what was when it designed

04:05.040 --> 04:11.840
nice system we have these three layers in multiplayer architecture and the testing was kind of always

04:11.840 --> 04:17.280
this weird thing that would come last and it was very very hard to actually squeeze it into you

04:17.280 --> 04:21.520
had to mock all this stuff you have this spring boot data thing you have some functions there

04:22.160 --> 04:27.040
you are mocking implementation some H2 database weird things you have never really felt that

04:27.040 --> 04:32.080
I have actually tested the functionality I was only basically testing and writing to test the code actually

04:32.080 --> 04:36.160
and usually what's noticeable when I would do some refactoring which didn't change the business

04:36.160 --> 04:39.680
logic I would have to always throw away half of the test because they were completely used

04:39.680 --> 04:46.720
because they were basically following the code so then I started to think how can I organize the

04:46.720 --> 04:52.080
code to actually be able to test to my test actually testing this logic because if I do

04:52.080 --> 04:58.000
or internal refactoring they should not actually change the implementation itself and I stumbled

04:58.000 --> 05:04.560
at that point on the SQL S pattern and basic even sourcing so in Java I brought a small

05:04.560 --> 05:09.760
library that will handle this to me and based on this already I could see the pattern

05:09.760 --> 05:14.160
where I could actually have a small percentage layer which was generic I would have some

05:14.160 --> 05:19.920
handlers and small routing code upfront and basically I could test my business logic completely

05:19.920 --> 05:24.400
from outside so my internal service would completely be isolated I could test it from outside

05:24.400 --> 05:28.240
and whatever I changed it side it would not affect the business logic from the outside

05:29.840 --> 05:34.880
and in 2019 I got opportunity to work a new green field project and I said okay let's start

05:34.880 --> 05:39.360
something different so we need some dynamic language because we need to be related to data driven as well

05:39.360 --> 05:46.640
so we decided to the closure and it was amazing so the difference between the closure and most

05:46.640 --> 05:51.360
of that language it's all about data and all there are a couple of functions but whatever you do

05:51.360 --> 05:55.680
you always use map produce you are putting some stuff into map getting some stuff about it

05:55.680 --> 06:01.920
to map and filtering so it was super easy to make very complex things with this basic acronyms

06:01.920 --> 06:10.320
and this was basically the for me very very nice revelation after that and yeah I figured

06:10.320 --> 06:14.880
out that architecture actually is important because it is actually affecting how we putting stuff

06:14.880 --> 06:21.920
in the code so you know people read some business requirements they see I know we have some

06:21.920 --> 06:26.000
graphs let's use graph database and they put a graph database well it is very easy to put it there

06:26.000 --> 06:30.240
but you know two years later to maintain it is when when the pain comes or some times

06:30.240 --> 06:36.000
years database and we did the same thing we started with postgres as a data storage and use

06:36.000 --> 06:39.760
force because we had some search of course we had use open search and then figured out this is

06:39.760 --> 06:46.880
not so cheap or easy to maintain and basically then after why we started thinking okay do we really

06:46.880 --> 06:51.840
need anything more than for example postgres so postgres like can store noise build data some

06:51.840 --> 06:57.760
JSON this structure data it can store also it has a very powerful query capabilities and so on so

06:58.640 --> 07:04.240
it is very important to think what this how much data you will have and what are actual things

07:04.240 --> 07:10.240
that you require and start small from up from down to up and then add new stuff afterwards not

07:10.240 --> 07:18.640
upfront then there's always this topic whether to go microservices small it and there we

07:18.640 --> 07:22.640
started microservices because everybody is doing it so it was fun to add stuff we had long

07:22.640 --> 07:27.280
free to services and then you figured out that this thing does work so and then we started

07:27.280 --> 07:32.240
config things together and we came in the end we have now around 20 services and we have

07:32.240 --> 07:37.200
basic 3 criteria by something should be isolated so first thing is scalability so there's some

07:37.200 --> 07:41.840
surface service some part of code that means huge amount of resources then you extracted outside

07:41.840 --> 07:47.040
because it's helped with scale basically independently and it's cheaper to scale with small part

07:48.000 --> 07:52.400
the other reason why we extract something is because it's using some weird library and this is

07:52.400 --> 07:56.560
mostly coming to updating and security so for example I know we use PDF box and microservices

07:56.560 --> 08:02.320
trying some documents or we use Chrome to render some HTML documents we extracted this

08:02.320 --> 08:07.680
is separate service because then when there's a bug coming in or security vulnerability we just

08:07.680 --> 08:11.600
replay this one service and this was also the second criteria we used for extracting something out

08:12.320 --> 08:19.680
and the third but this is less important is when we see two teams using working with the same code

08:19.680 --> 08:23.840
then we kind of tried to split it together because it resolves less conflict and it's less

08:23.840 --> 08:29.200
less in the negotiation everything that remains outside of this we put together in kind of a

08:29.200 --> 08:33.760
monolithic service and this is worked fast really best so first you have you can really test

08:33.760 --> 08:39.040
huge scope in one of these service and you still have this capability of updating stuff independently

08:39.600 --> 08:46.320
and what helped us and basically this is how we built our whole library is to design

08:47.200 --> 08:52.240
things about more about state transition then actually mutation so we are building a big aggregate

08:52.240 --> 08:57.680
so if you have a group of six and and if we put it together at one station because it is in memory

08:57.680 --> 09:03.360
it is easy to update you can easily test it instead of you know having some slacks then joining

09:03.360 --> 09:08.400
data in memory trying to figure out what is what and trying to connect this together so it was easier

09:08.400 --> 09:12.960
you just want things together one big station you can store now and post this two gigabytes

09:12.960 --> 09:18.560
station in the field so if you really need all the data to make any decision then just put it put it together

09:22.160 --> 09:31.680
so yeah it is the core this is our library so this is the only thing that used for

09:31.680 --> 09:35.680
almost entire system is that there are some outliers somewhere in the edge but most of the

09:35.680 --> 09:41.520
stuff this is the whole thing we use the most important features we have is decorative dependencies

09:41.520 --> 09:47.360
solutions so you in your handler you need to define what things you need and then you are picking

09:47.360 --> 09:51.600
the data from the from the other services so you declare what you need and then the system is

09:51.600 --> 09:56.720
pulling this for you this is very important because it makes API very simple everything will be the same

09:56.720 --> 10:02.720
you you hide this resolution code into one small place and you just declare what you need instead of

10:02.720 --> 10:07.920
everybody somewhere in the middle of the code pulling some dependency and and then trying to figure out

10:09.040 --> 10:15.200
later on what was pulled there and how can I actually reflect on this thing so then this came

10:15.200 --> 10:20.240
with nice combination of the colors API so we have our own front and client basically it's 300

10:20.240 --> 10:26.080
lines of code it can create absolutely all of our services so we've wrote it four years ago it's

10:26.080 --> 10:33.520
close to the same it works till now perfectly working and I'm in a small graph on the right side

10:33.520 --> 10:39.840
basically all of our service is the same thing so we resolve the dependencies we will command handler

10:39.840 --> 10:44.320
command handler is because of that very pure so you have all these data coming from the dependencies

10:45.200 --> 10:51.120
you fetch your current state of your aggregate you process it and then your command handler

10:51.120 --> 10:56.800
basically speeds out a couple of units which we are then storing back to the to the database and as

10:56.800 --> 11:03.360
last step we are using then the outbox pattern to store things that need to be distributed to other

11:03.360 --> 11:08.000
services so for example you create a user and you need to send the email so you would create a user

11:08.000 --> 11:11.680
and you will store in database okay I need to send the email then this will be then string to the other

11:11.680 --> 11:16.720
service of course this email service is this 2% for example where you don't have a pure command handler

11:16.800 --> 11:20.880
and it would as far as possible from this code because you know this thing is dangerous and hard to

11:20.880 --> 11:26.480
test so we then isolate these things out and usually use some words you know libraries to actually send

11:26.480 --> 11:35.120
emails and so and what we discovered now in production if I have a problem in the system everything

11:35.120 --> 11:39.440
is the same I can just go to database I can figure out okay what is my flow what was my

11:39.440 --> 11:43.520
what was the other service doing and I can pinpoint I don't even know what the service is doing

11:43.520 --> 11:47.520
but I can already figure out where the code might be that the second handling from just observing

11:47.520 --> 11:52.160
from the database I don't even go to logs when I have a problem I take a request ID I can trace

11:52.160 --> 11:57.440
in the database back to the service handler that actually handled this thing I pulled the data

11:57.440 --> 12:04.400
and I write a unit test and I can figure out what the problem is and yeah so we are actually working

12:04.400 --> 12:11.120
on the huge banking system so we have the workflow for the long processing we have then the

12:11.120 --> 12:16.880
calculation engine which is calculating with multiple dimensions the consolidation tree we have

12:16.880 --> 12:22.000
a document rendering so we managed to fit this whole system into the with with this same library

12:22.000 --> 12:28.240
with this same set of principles and the other thing that was you know benefit as well was the

12:28.240 --> 12:32.880
testing as said most of the stuff is pure we really have huge amount we have tests we have services

12:33.680 --> 12:38.800
that have 20 commands and have 4,000 unit tests because we can really test everything that is

12:39.280 --> 12:44.880
inside and we are deploying now 10 times per day to production throughout the day because we have

12:44.880 --> 12:49.280
so much that people just don't even care and they they make a poor request they send it for approval

12:49.280 --> 12:58.800
we get the proof goes to production nobody's brain actually about this anymore so what was how

12:58.800 --> 13:04.400
biggest pain point that basically came like middle of last year was dependencies so we had our

13:04.400 --> 13:10.240
library we had a full control what is using nobody's actually pulling dependencies and we try to

13:10.240 --> 13:16.320
really minimize this as much as possible we are still working on to drop some things mostly things are

13:16.320 --> 13:22.320
like that we are using like a key recipe for the connection pool because we didn't try to

13:22.320 --> 13:26.560
write this on our own but all of the dependencies we have from outside we have for them we have

13:26.560 --> 13:31.040
build them at least on our own because if the problem comes we need to be able to to fix them on

13:31.040 --> 13:35.680
our own so basically that's why we that's main motivation why we're trying to minimize and

13:37.760 --> 13:42.960
other dependencies are like a small closure wrapper we have a small wrapper around a Java HTTP client

13:42.960 --> 13:47.520
before we were using some weird HTTP client but now as Java nicely implemented the client there

13:47.520 --> 13:54.640
is no need to do that so we just use a Java HTTP client and we are doing basically a white listen library

13:54.640 --> 14:00.640
so when we talk with a Java spring boot white listening was no go so we had a topic security

14:00.640 --> 14:06.480
person always there it is impossible to white list libraries that we are going to use with this

14:06.480 --> 14:11.280
approach it was we can do quite listing without problems we know what is running in our system we

14:11.280 --> 14:18.640
know everything what is where and yeah we have also built some pipeline jobs where I can

14:18.640 --> 14:22.640
pick a button up the entire system it goes over all the system fetches all the new dependencies

14:22.640 --> 14:27.440
set up automatically pull requests and in one hour I can up the entire system so some critical

14:27.440 --> 14:36.640
moment really it is like lock project and so on are basically not a problem anymore so what I can

14:36.640 --> 14:40.640
what can I recommend use as much as possible what language offer and I think we should all go

14:40.640 --> 14:45.120
in this direction to move you know these basic things for example for me the Java doesn't have a

14:45.120 --> 14:50.400
JSON implementation it's a horrible thing I go to Python I go to no JSON I have a JSON there I mean

14:50.400 --> 14:54.640
everything is working with JSON this is like for me they should all they should make it extendable

14:54.720 --> 14:59.200
of course we can make better implementation but at least if I open a Java language I should be able

14:59.200 --> 15:05.440
to serialization without adding some external dependencies so now Java has HTTP client I think

15:05.440 --> 15:10.720
this was lots of misery I remember this moving HTTP client from a passage to be client from version

15:10.720 --> 15:22.080
4.4 to 4.5 plus MS and these kind of things are now gone away and really try to stay to the basic

15:22.080 --> 15:32.640
things and higher complexity be higher complexity because between a small layer and focus on

15:32.640 --> 15:38.720
writing a pure business logic that can actually be testing yeah I said the architecture is very

15:38.720 --> 15:42.080
important it doesn't influence all in your components you're going to use you have to really

15:42.080 --> 15:47.600
think about how the implementation behind you be all to the story details this I mentioned and yes

15:48.560 --> 15:53.440
excellent it's also interesting that you're doing a for bank which a notorious for having

15:53.440 --> 15:59.920
crappy software right yes so have you considered using geeks for dependencies I know I know people

16:01.280 --> 16:06.400
I'm just going to squeeze it in yes so we had to implement last year kind of exit strategy

16:06.400 --> 16:12.000
we were sighted already actually try geeks but then it took us two days to implement it we didn't

16:12.000 --> 16:17.280
have more time so unfortunately not to talk with us yes I will go so the next speaker can

16:17.280 --> 16:23.200
set up who is the next Peter picker yeah he's not a speaker where's the speaker for the next one

16:24.000 --> 16:30.160
okay you can set up and please repeat the questions and I also you know we'll

16:30.160 --> 16:35.920
this is just back to back I'm going to ask you to because it's getting more busy to start squeezing in

16:35.920 --> 16:45.840
okay so we don't have empty chairs in between anyway next question sorry

16:53.600 --> 16:58.240
we have some property testing in some some especially in calculation because it's nicely

16:58.240 --> 17:08.480
actually but most of the stuff is actually standard you didn't test but also property

17:08.480 --> 17:11.920
is comes to an especially in calculation but you need to figure out what combination of

17:11.920 --> 17:16.800
you're going to spit out what and this is helpful there as well

17:35.600 --> 17:39.360
so first we try to figure out do we can we fix it somehow on our own without actually

17:39.360 --> 17:42.640
employees especially in front and you know it sounds small things you can write a function

17:42.640 --> 17:46.960
at five point you only to include dependencies we do the review of course within the team we

17:46.960 --> 17:51.520
discuss what can be done we overchecked the pennies that later there and then we try to see

17:51.520 --> 17:56.240
we've taken causation security so for example there's lots of this PDF rendering libraries

17:56.240 --> 18:00.480
which you can use but we figure out okay we can render HTML with the Chrome right so we then

18:00.480 --> 18:06.000
rely on Chrome and so but it's mostly discussion within the team and of course for code review

18:06.000 --> 18:09.120
where everybody will then review if there's somebody at the some dependence in song

18:13.120 --> 18:16.720
most of the studies that we use we have forked and we have our own builds and all I mean we

18:16.720 --> 18:20.720
use polyclone but at least we need to prove that we can actually build it so to be able to

18:20.720 --> 18:25.680
write on a fixed stuff and there was some bugs in like a Hikari CP which we had to fix this song

18:26.640 --> 18:35.680
sorry the question what the question is how we analyze how we decide which dependencies we use

18:35.680 --> 18:40.080
within the team and whether or not we build all of our dependencies the store plan

18:40.400 --> 18:42.080
oh yeah

18:46.400 --> 18:48.400
other questions

18:50.320 --> 18:51.840
yes

18:51.840 --> 18:58.880
so I've seen that basically how this is that we try to use more or less full system

18:58.880 --> 19:05.680
dusting more and the system is like getting big and you cannot really do this that is a strategy

19:06.640 --> 19:12.640
so we have more of most of the logic is tested with the unit is because the whole thing is

19:12.640 --> 19:15.760
basically more so the service is the purest I can develop the whole service however it

19:15.760 --> 19:22.480
will still be running locally we have then of course integration tests and the UITests which

19:22.480 --> 19:27.680
cover the whole system the what was the important as the scalability of testing infrastructure and

19:27.680 --> 19:32.400
infrastructure itself so we use serverless for this case and basically we are running thousands of

19:32.400 --> 19:38.160
integration and unison and UITests basically to test the whole thing in the case when you

19:38.160 --> 19:42.560
have a system then you need to define some boundaries of testing we didn't come to that for us

19:42.560 --> 19:47.920
I think we have built quite a medium-sized system and we still need to do it but then we would really

19:47.920 --> 19:52.240
have to define the boundaries of the system and how they communicate with each other and test it there

20:02.400 --> 20:13.840
and