WEBVTT 00:00.000 --> 00:02.000 You 00:30.000 --> 00:32.000 You 01:00.000 --> 01:22.000 You 01:22.000 --> 01:25.000 Here the mic but they're trying to record this. 01:25.000 --> 01:32.000 So if you have 96 cores in your system and turn all those loose on a parallel GC effort, 01:32.000 --> 01:36.000 they're all going to sit waiting for each other to get out of each other's way. 01:36.000 --> 01:40.000 So it's much more efficient if you can take some small subset of those 96 cores, 01:40.000 --> 01:48.000 put them on concurrent GC, let the others work in the cash for your service threads. 01:49.000 --> 01:58.000 My focus today will be on getting low latency for Java and making the Java execution more frugal. 01:58.000 --> 02:01.000 Several dimensions of that. 02:01.000 --> 02:11.000 We want within Amazon for concurrent garbage collection to be the default configuration within a couple of years. 02:11.000 --> 02:19.000 We want concurrent GC to run as efficiently as G1 GC and similar or even smaller heap sizes. 02:19.000 --> 02:21.000 It's not that way today. 02:21.000 --> 02:29.000 What we find is that G1 is by far the most prominent garbage collector inside of Amazon. 02:29.000 --> 02:36.000 Many services over provision the memory so that they can separate the pause times. 02:36.000 --> 02:41.000 So this is our effort, this is our focus, this is what we're working on. 02:41.000 --> 02:49.000 And I'm going to give you the story of the challenges we face as the engineers trying to deliver on this vision. 02:49.000 --> 02:58.000 So back in 2021, we were beginning to see adoption of Shenandoah inside Amazon for various services, 02:58.000 --> 03:04.000 but by far the small limited minority of services. 03:04.000 --> 03:10.000 Along the bottom here, you can barely see it, this is the G1 latency. 03:10.000 --> 03:17.000 It's pretty good all the way up to say 290, almost 300 transactions per second. 03:17.000 --> 03:24.000 When we apply Shenandoah to this workload, first of all, this is a low test program. 03:24.000 --> 03:29.000 It fails the test at 200 transactions per second. 03:29.000 --> 03:41.000 On the way to that destination, you see the latency is up there 32520 over 620 millisecond latency, 03:41.000 --> 03:44.000 even though it's a positive garbage collector. 03:44.000 --> 03:47.000 So what's happening there? 03:47.000 --> 03:56.000 Shenandoah GC cannot keep up with the pace of allocation for this heavy high allocation workload. 03:56.000 --> 04:09.000 And so even though we're not pausing for stop-to-world GC pauses, we have to pause so the garbage collector can catch up with the allocation demand of the service. 04:09.000 --> 04:14.000 So this is what motivated our work on generational Shenandoah. 04:14.000 --> 04:18.000 We needed the garbage collector to be more efficient. 04:18.000 --> 04:28.000 We needed the frugality of deploying a pauses garbage collector to match that of G1 GC. 04:28.000 --> 04:36.000 Let me just say, if you look at this chart and you run this same test with generational Shenandoah, 04:36.000 --> 04:43.000 and with generational GC, you'll see that both of these are much closer to the bottom line here, 04:43.000 --> 04:49.000 but still G1 has the better p50 latency. 04:49.000 --> 04:53.000 And for many services, p50 is what they care most about. 04:53.000 --> 05:03.000 But if you care about p99 or p59s, that's when you begin to see the benefits of a concurrent garbage collector. 05:03.000 --> 05:11.000 And when I say, we want concurrent GC to be as efficient or as frugal as G1 GC. 05:11.000 --> 05:16.000 We would like to be G1 GC across the whole line of p50 as well. 05:16.000 --> 05:19.000 We're not there yet. 05:19.000 --> 05:27.000 One of the challenges with a concurrent garbage collector is knowing when to start the garbage collection. 05:28.000 --> 05:34.000 If you start too soon, you'll spend a lot of time doing many more GC cycles than necessary. 05:34.000 --> 05:37.000 That will chew up CPU time. 05:37.000 --> 05:40.000 It will detract from the CPU available for your service. 05:40.000 --> 05:44.000 It will slow your p50 latency. 05:44.000 --> 05:49.000 If you start too late, you're going to lose the race. 05:49.000 --> 05:53.000 You're going to deplete or exhaust the free pool before it's been replenished. 05:53.000 --> 06:03.000 So you want to predict when to start it, so you finish it exactly the right time and don't do too many GC cycles. 06:03.000 --> 06:05.000 This problem goes away with G1. 06:05.000 --> 06:08.000 They just run and tell the memory is exhausted. 06:08.000 --> 06:11.000 Stop the world, collect the young. 06:11.000 --> 06:17.000 They do have the problem with the old garbage, the collection of the old region. 06:17.000 --> 06:22.000 The old generation, they use something called eye hop. 06:22.000 --> 06:27.000 And now I'll show you what to stand for. 06:27.000 --> 06:31.000 Initial, initial, keep occupancy percentage. 06:31.000 --> 06:34.000 So you set this to 40% for example. 06:34.000 --> 06:39.000 When the old generation gets up to 40%, it will start the GC. 06:39.000 --> 06:46.000 Let me make the point that G1 GC has a constraint that concurrent GC does not have. 06:47.000 --> 06:58.000 G1 GC has to keep the young generation small because it has to do a stop to the world of the young generation without violating the 200ml sec in or whatever 06:58.000 --> 07:02.000 POS time maximum has been specified. 07:02.000 --> 07:06.000 With concurrent garbage collectors, the young collection is also concurrent. 07:06.000 --> 07:10.000 So we don't have that constraint that we have to finish it in 200ml seconds. 07:10.000 --> 07:15.000 It's a concurrent phase, so we can make our young larger. 07:15.000 --> 07:30.000 Shenandoah, if you study its performance characteristics, study where it performs well, where it begins to have weaknesses, it has these two attributes. 07:30.000 --> 07:42.000 If the memory utilization is very low, for example, if you have a heap of 5% live memory, I would say low memory utilization, Shenandoah does extremely well. 07:42.000 --> 07:52.000 That's because it's got 95% of the heap as run way to allocate while it's doing the garbage collection has a lot of time to do the garbage collection before it runs out of memory. 07:52.000 --> 07:57.000 The other time Shenandoah does very well is if the allocation rate is very low. 07:57.000 --> 08:07.000 So if the allocation rate is very low, you're going to have also a lot of time to get the garbage collection done before you run out because you're not allocating very fast. 08:08.000 --> 08:16.000 So what we've done in the design of generational Shenandoah is we basically replicate the Shenandoah garbage collector twice. 08:16.000 --> 08:21.000 We have it running once on the young generation, once on the old generation. 08:21.000 --> 08:25.000 In the young generation, we have low utilization. 08:25.000 --> 08:33.000 Most objects die young. Every time we do a young collection, very few objects are still alive in the young generation. 08:33.000 --> 08:42.000 We copy those and we claim all the other garbage automatically as a by-product. 08:42.000 --> 08:50.000 In the old generation, we have very low allocation rate, promotion rate, very few objects get promoted. 08:50.000 --> 09:00.000 So Shenandoah works well on each generation independently and we run them concurrently. 09:01.000 --> 09:06.000 We use something similar to the G1 heuristics in Shenandoah. 09:06.000 --> 09:13.000 So we reclaim the garbage first when we choose which regions to garbage collect. 09:13.000 --> 09:16.000 If there's a region has a lot of light memory, we just leave it aside. 09:16.000 --> 09:21.000 We'll come back to that later after more objects have died in that region. 09:21.000 --> 09:23.000 This also lets it be very efficient. 09:23.000 --> 09:28.000 We do that both for the young regions and the old regions. 09:29.000 --> 09:35.000 We trigger old generation marking under several conditions. 09:35.000 --> 09:40.000 Very different from the way G1 does it with their eye hop heuristic. 09:40.000 --> 09:48.000 We trigger if we detect that the old generation has become fragmented, if it's spread out all over the heap, 09:48.000 --> 09:55.000 if it's infringing on areas of the heap that we would like to set aside to efficiently handle humongous objects. 09:56.000 --> 09:59.000 We'll trigger an old generation collection. 09:59.000 --> 10:02.000 We also trigger an old generation collection. 10:02.000 --> 10:09.000 If the old generation has grown more than a specific percent since the last time we did old marking. 10:09.000 --> 10:13.000 So every time we do old marking we say okay there's 10 gigabytes live. 10:13.000 --> 10:20.000 If it gets up to 10 plus 10 gig plus 12 percent or about 12. 10:21.000 --> 10:27.000 Yeah about 12 gigabytes we'll start another old-gen collection. 10:27.000 --> 10:31.000 Also if there's a shortage of available memory. 10:31.000 --> 10:37.000 Just realize this is very different from the way G1 behaves. 10:37.000 --> 10:44.000 When a generation of Shenandoah is running correctly configured properly, 10:44.000 --> 10:51.000 what you'll see is a interleaving of an increment of old generation collection, 10:51.000 --> 10:56.000 an urgent preemption of that to do a young collection, 10:56.000 --> 11:01.000 and then resume the old collection and then another increment of young collection. 11:01.000 --> 11:09.000 The reason we structured this way is because it's very urgent whenever we have to do a young collection 11:09.000 --> 11:12.000 to get that young collection done very quickly. 11:12.000 --> 11:14.000 That's the higher priority activity. 11:14.000 --> 11:21.000 It's urgent because the services cannot run until they get their allocations. 11:21.000 --> 11:24.000 And so we need to replenish the free pool very quickly. 11:24.000 --> 11:26.000 That's most important thing. 11:26.000 --> 11:29.000 Get that young collection done quickly. 11:29.000 --> 11:33.000 The old collection usually takes more time. 11:33.000 --> 11:38.000 There's a lot higher density of live objects in the old regions. 11:38.000 --> 11:45.000 So it takes a lot longer to mark old memory than it does to mark young memory. 11:45.000 --> 11:53.000 And likewise it takes longer to evacuate or compress those old generation regions. 11:53.000 --> 11:58.000 So this process goes on until you've finished marking the old. 11:58.000 --> 12:01.000 Then we do very much like G1 does. 12:02.000 --> 12:10.000 We go into something called young mixed evacuations. 12:10.000 --> 12:19.000 So at the end of old marking, we identify regions that are candidates to be collected from the old regions. 12:19.000 --> 12:30.000 And in subsequent young collections, we do the normal young garbage collection effort combined with some subset of the candidate old gen regions. 12:30.000 --> 12:45.000 We use the same evacuation cycle that the young generation is using so that we can share the cost of the barriers that support concurrent evacuation. 12:45.000 --> 12:50.000 We run as some number of those young mixed evacuations. 12:50.000 --> 12:55.000 And then we've completely evacuated the candidates from the old generation. 12:55.000 --> 12:58.000 And at some point in the future, we'll trigger another old gen collection. 12:58.000 --> 13:03.000 So this is the general way it works. 13:03.000 --> 13:06.000 How do we trigger young collections? 13:06.000 --> 13:11.000 The way this has been done is the same way traditional Shenandoah does it. 13:11.000 --> 13:15.000 We calculate headroom, which is our working safety buffer. 13:15.000 --> 13:20.000 The headroom is determined by something called an allocation spike factor. 13:20.000 --> 13:25.000 We reserve a certain percentage, maybe 5% of the young gen. 13:25.000 --> 13:30.000 It says we'd like to stay away from the pending on that memory. 13:30.000 --> 13:35.000 We'd like to try to get it garbage collection done before we use that last 5%. 13:35.000 --> 13:44.000 But in case we mis-predicted, that gives us a little extra runway to finish the GC before we're all the way out of memory. 13:44.000 --> 13:47.000 And then we get we factor in penalties. 13:47.000 --> 13:52.000 So there's something in Shenandoah called a degenerated GC cycle. 13:52.000 --> 14:04.000 Degeneration happens when the garbage collector is unable to free memory faster than the application allocates memory. 14:04.000 --> 14:08.000 So the application says I want 100 megabytes. 14:08.000 --> 14:11.000 The GC says sorry don't have it. 14:11.000 --> 14:14.000 That causes us to do a degenerated cycle. 14:14.000 --> 14:16.000 That means stop the world. 14:16.000 --> 14:18.000 That's the way it's currently implemented. 14:18.000 --> 14:19.000 Stop the world. 14:19.000 --> 14:22.000 Let the GC catch up. 14:22.000 --> 14:24.000 So it's a safe point. 14:24.000 --> 14:29.000 Thousands of threads have to stop even though only one thread needed that memory. 14:29.000 --> 14:32.000 But we do the stop the world. 14:32.000 --> 14:40.000 We do the degenerated GC cycle and then we resume concurrent execution. 14:41.000 --> 14:44.000 So those degenerated cycles are very undesirable. 14:44.000 --> 14:54.000 If we ever have one, we penalize because the degenerated GC, so that it would be more aggressive 14:54.000 --> 15:01.000 to stay with the next GC sooner so that it won't happen during the run out of memory. 15:01.000 --> 15:11.000 Basically, we trigger the collection when we are at risk of run out of memory before the GC can catch. 15:11.000 --> 15:19.000 I want to show you some experimental results that are based on this heuristic that I just described to you. 15:19.000 --> 15:27.000 For full disclosure, I'm providing the details of the configuration. 15:27.000 --> 15:34.000 I'm not going to go into all depth, but it's here in the slide if somebody wants those details. 15:34.000 --> 15:41.000 To give you an overview of what all that gobbledy group means, it's about 2,000 running threads. 15:41.000 --> 15:47.000 They're allocating at a rate of about 1.7 gigabytes per second. 15:47.000 --> 15:53.000 There's about 11 gigabytes of live old-gen memory. 15:54.000 --> 15:59.000 This runs a simulation for 20 total minutes. 15:59.000 --> 16:06.000 And every 90 seconds, we rebuild half the database approximately. 16:06.000 --> 16:12.000 Inside of this simulation, there's a customer database and there's a product database. 16:12.000 --> 16:19.000 Every other time we rebuild the customers and then we rebuild the products and then we rebuild the customers. 16:19.000 --> 16:27.000 In between those rebuild operations, we delete some customers, we add some customers, we delete some products, we add some products. 16:27.000 --> 16:29.000 So that's what's happening. 16:29.000 --> 16:33.000 Notice that this is an intermittent variable allocation rate. 16:33.000 --> 16:39.000 Every time you rebuild the database, you get a spike of allocations causes us to mispredict. 16:39.000 --> 16:44.000 It's intentionally designed to be a hard benchmark to run. 16:44.000 --> 16:52.000 In this benchmark, I'm going to share results from G1GC, yes? 16:52.000 --> 16:54.000 What did I do wrong? 16:54.000 --> 16:56.000 Thank you for the alert. 16:56.000 --> 16:58.000 I don't know what happened here. 16:58.000 --> 17:00.000 I still have it all. 17:06.000 --> 17:10.000 Okay, I thought I was asleep, I don't know. 17:10.000 --> 17:16.000 So yeah, so I'm going to share the results of G1GC out of the box. 17:16.000 --> 17:24.000 I'm comparing with some of the special configurations and experimental development branch of Gension. 17:24.000 --> 17:28.000 I'm not going to share the GNGGC results right now. 17:28.000 --> 17:33.000 But here are some of the results from this data. 17:33.000 --> 17:38.000 And the first thing I want to point out, this is showing a response time. 17:38.000 --> 17:44.000 Those are measured in micro seconds for various heap sizes of the same workload. 17:44.000 --> 17:48.000 Remember it's got 11 gigabytes of old gen. 17:48.000 --> 17:52.000 It's got allocation rate of 1.7 gigabytes per second. 17:52.000 --> 17:58.000 And what's interesting to see is as the heap size shrinks, 17:58.000 --> 18:00.000 the response time goes up. 18:00.000 --> 18:04.000 That's what you would expect because the GC has to run more frequently. 18:04.000 --> 18:10.000 It has to interfere more with the service. 18:10.000 --> 18:12.000 There's a big spike here. 18:12.000 --> 18:16.000 There's a quantum leap. 18:16.000 --> 18:20.000 That corresponds to compressed oops optimization. 18:20.000 --> 18:24.000 So what's happening here is in a 32 gigabyte, 18:24.000 --> 18:28.000 and everything to the right of that, we do not have compressed oops. 18:28.000 --> 18:33.000 And from 31 gigabytes down, we do have compressed oops. 18:33.000 --> 18:36.000 So it's like two completely different worlds. 18:36.000 --> 18:38.000 Your miles may vary. 18:38.000 --> 18:43.000 It depends on your application, how many pointers are inside your data structures and so forth. 18:43.000 --> 18:48.000 But it's very interesting to see that a 31 gigabyte heap 18:48.000 --> 18:54.000 is running with almost the same efficiency as a 48 gigabyte heap. 18:54.000 --> 19:00.000 And a 19 gigabyte heap is running with more efficiency than the 32 gigabyte heap, 19:00.000 --> 19:03.000 because of compressed oops. 19:03.000 --> 19:07.000 At the scale of Amazon, this represents hundreds of millions of dollars 19:07.000 --> 19:10.000 to be able to move into that compressed oops range. 19:10.000 --> 19:14.000 So that's something that's very important to us in terms of being efficient 19:14.000 --> 19:19.000 in the way we deploy these systems. 19:19.000 --> 19:24.000 A lot of data here shows the experiment we did. 19:24.000 --> 19:28.000 The reason, I don't expect you to digest all the details, 19:28.000 --> 19:33.000 but the reason I run this at all these different heap sizes is twofold. 19:33.000 --> 19:41.000 The typical Amazon customer says, my objective, my need, my SLA requirement, 19:41.000 --> 19:50.000 is to satisfy, for example, 50 millisecond response time at p99.9. 19:50.000 --> 19:56.000 So we have to ask, well, how big of a heap do you need for that objective? 19:56.000 --> 20:02.000 And you go over to the p99.9 column, and you say, hey, 50 milliseconds, that's easy. 20:02.000 --> 20:06.000 We can do that even with a 19 gigabyte heap size. 20:06.000 --> 20:09.000 But every customer is different. 20:09.000 --> 20:15.000 The other thing that this shows us is that typical service will configure for 20:15.000 --> 20:22.000 the intended design workload, but they get spikes. 20:22.000 --> 20:27.000 Something happens in all of a sudden, they get twice as much traffic as they plan for. 20:27.000 --> 20:32.000 So that's like shrinking the heap size in terms of how efficient it's going to operate. 20:32.000 --> 20:38.000 So you want to make sure that it's scalable across the range of different heap size. 20:38.000 --> 20:42.000 So that's why we also look at different heap size configurations. 20:42.000 --> 20:48.000 This is a G1 comparison on the same workload, all the numbers are there. 20:48.000 --> 20:53.000 What I'm showing you is how they compare. 20:53.000 --> 20:56.000 Thank you, five minutes left. 20:56.000 --> 21:07.000 So p50, p95, that's pink color means generational Shannon do what Shannon does is worse in latency at this end. 21:07.000 --> 21:11.000 In the middle, generation of Shannon does a lot better. 21:11.000 --> 21:20.000 In fact, like 94% better, if you go back just real quickly, 21:20.000 --> 21:28.000 the three nines column, those are in the hundreds or thousands of microseconds, 21:28.000 --> 21:35.000 whereas with Gen Shen, the three nines column was in the tens of microseconds. 21:36.000 --> 21:38.000 So those are huge differences. 21:38.000 --> 21:46.000 But at the p100, comes G1 is still better, bothersome to me. 21:46.000 --> 21:54.000 bothersome to anyone that says, hey, I chose this because I wanted better latency. 21:54.000 --> 21:59.000 So this motivated some changes to the way we do our triggering. 21:59.000 --> 22:04.000 In particular, we've implemented, this is not yet fully integrated. 22:04.000 --> 22:12.000 We have the branch PRs that are, some in draft, some ready to integrate, 22:12.000 --> 22:14.000 but these are the changes. 22:14.000 --> 22:21.000 Instead of sampling the allocation rate every 10 Hertz, look at it every 500 Hertz, 22:21.000 --> 22:24.000 detect acceleration of allocation rates. 22:24.000 --> 22:27.000 Don't assume it's an average that's going to stay constant. 22:27.000 --> 22:31.000 If you see it's increasing from one sample to the next, 22:31.000 --> 22:33.000 it's going to continue to increase. 22:33.000 --> 22:38.000 Do your calculation of the assumption based on the rate of increase. 22:38.000 --> 22:41.000 And finally, search number of GC threads. 22:41.000 --> 22:45.000 If you do your computation, you say, we triggered now. 22:45.000 --> 22:48.000 We're already destined to lose the race. 22:48.000 --> 22:51.000 Let's search the GC threads. 22:51.000 --> 22:57.000 When we do this, I'm kind of at the end of my time. 22:57.000 --> 23:05.000 But you see this last P100 column has almost all switched to green and significantly green. 23:05.000 --> 23:10.000 So we think we're making a very significant and worthwhile improvements. 23:10.000 --> 23:13.000 We continue to work on that. 23:13.000 --> 23:16.000 We've got a still an outlier down at the bottom. 23:16.000 --> 23:17.000 It's not a severe outlier. 23:17.000 --> 23:20.000 The numbers are reasonable. 23:21.000 --> 23:22.000 So you can't see. 23:22.000 --> 23:34.000 It's 339,000 microseconds versus, I'm sorry, I don't have it here. 23:34.000 --> 23:39.000 But it's like 100 microseconds. 23:39.000 --> 23:42.000 Next steps in our path. 23:42.000 --> 23:44.000 Big plans for the year. 23:44.000 --> 23:49.000 We're going to reduce number of safe points. 23:49.000 --> 23:50.000 Currently, there's four. 23:50.000 --> 23:53.000 We've actually reduced it to three safe points every time. 23:53.000 --> 23:54.000 Gen Shen runs. 23:54.000 --> 23:57.000 We're going to take it down to one safe point. 23:57.000 --> 23:59.000 The others will be hand shakes. 23:59.000 --> 24:03.000 We're replacing the out of memory during evacuation protocol. 24:03.000 --> 24:04.000 Right now that degenerates. 24:04.000 --> 24:06.000 We're going to do it a different way. 24:06.000 --> 24:09.000 Then we can take away the degenerated cycles. 24:09.000 --> 24:11.000 We're going to take away Shen into a pacing. 24:11.000 --> 24:16.000 We're going to replace that with the GC worker thread search. 24:17.000 --> 24:24.000 We're going to defragment the humongous memory concurrently so that we don't need any full GC cycles. 24:24.000 --> 24:26.000 We can get rid of the full GC cycles. 24:26.000 --> 24:30.000 And we're going to also prototype if we get it done. 24:30.000 --> 24:34.000 A from space invariant instead of the two space invariant that's currently implemented. 24:34.000 --> 24:37.000 That's going to give us higher throughput we believe. 24:37.000 --> 24:44.000 And better scalability, especially to things like new market textures. 24:45.000 --> 24:48.000 Gen Shen is now an open JDK 24. 24:48.000 --> 24:56.000 It's not currently in production, but we are evaluating it under experimental production workloads. 24:56.000 --> 24:59.000 Limited blast radius at this time. 24:59.000 --> 25:09.000 We're planning later this year, middle of the year, to have a GA product that's fully supported for production use. 25:09.000 --> 25:14.000 It is not in Coreto. It will be in Coreto 25. 25:14.000 --> 25:17.000 It'll be in Coreto 24. 25:17.000 --> 25:23.000 But you can get it in a nightly build of Coreto 21 on this website. 25:23.000 --> 25:31.000 That's not our officially supported Coreto product, but it is a experimental release that you can toy with. 25:31.000 --> 25:36.000 So I think we're okay for maybe some questions. 25:37.000 --> 25:46.000 If there are any questions. 25:46.000 --> 25:48.000 Okay, thank you. 25:48.000 --> 25:53.000 Are there any questions? 25:53.000 --> 25:59.000 Yes? 25:59.000 --> 26:03.000 The overhead? 26:04.000 --> 26:16.000 Okay, so the question is, how does the memory overhead compare between G1, Gc, and generational Shenandoah? 26:16.000 --> 26:22.000 We have done all these experiments using the same heap size. 26:23.000 --> 26:32.000 G1, for a given heap size, has a much more complicated and large, remembered set representation, much larger than ours. 26:32.000 --> 26:43.000 So for any given heap size, choose G1, 30 gigabytes, G1 is going to consume a larger RSS than Gen Shen. 26:43.000 --> 26:58.000 But our goal is to match the G1 sizes or even improve upon it because you can now do the young collections concurrently. 26:58.000 --> 26:59.000 Any other questions? 26:59.000 --> 27:04.000 Yes? 27:04.000 --> 27:09.000 Amazon has a rule that we cannot say things like that. 27:10.000 --> 27:15.000 You can see the GitHub repository which people are contributing. 27:15.000 --> 27:19.000 There's a small handful. 27:19.000 --> 27:25.000 Say again, handful? 27:25.000 --> 27:27.000 Anything else? 27:27.000 --> 27:29.000 Thank you very much. 27:39.000 --> 27:41.000 Thank you.