WEBVTT 00:00.000 --> 00:11.000 Welcome everybody, just two quick announcements. Please try to say for the talk. It's 00:11.000 --> 00:16.000 a fairly disruptive one. Someone in the middle says that. If you have questions later, 00:16.000 --> 00:21.000 give it to questions. Discussions can happen after them. Let's give it up for our 00:21.000 --> 00:34.600 first month's F4. Hi everyone, my name is Riga. I hope I'm audible. My name is Riga. I'm currently 00:34.600 --> 00:40.600 working with IBM. I'm a part of SF and ODF team. And with me, I have Weshnavi. She is also 00:40.600 --> 00:45.480 the part of the same team. So today we will be discussing about how we can make 00:45.480 --> 00:51.800 newbar more resilient by eliminating the single point of failure. So I would like to know 00:51.800 --> 00:59.240 how many of you I have actually heard about Nuba. Any hand. Okay, just go ahead and do. Okay, 00:59.240 --> 01:07.880 yeah, that's great because I have included introduction to Nuba. So yeah, great. So yeah, so this is 01:07.880 --> 01:13.560 the agenda for the today's topic. So we'll firstly go through the introduction to Nuba. We'll 01:13.640 --> 01:19.640 then go understand the Nuba architecture and then we'll see how this single point of failure 01:19.640 --> 01:27.400 plays role in the Nuba environment. And then what we can do to mitigate it. So what is Nuba? 01:27.400 --> 01:34.920 Let's try to understand that. Nuba is a software defined storage platform that is used to manage 01:34.920 --> 01:40.200 your object storage across like diverse environments. So what do I mean when I say diverse 01:40.520 --> 01:50.520 environments, they can be any cloud public cloud platforms such as Azure, AWS, GCP, or it could 01:50.520 --> 01:57.160 be any S3 compliant private cloud as well. It could also be any storage drives that you might 01:57.160 --> 02:03.400 have. It just needs to have a volume and a file system on top so that they can connect to Nuba. 02:04.280 --> 02:11.000 So what does this do is it makes the users job a lot easier. You can have your storage 02:11.000 --> 02:17.400 spread across different environments and you can manage them at the same time at the same platform. 02:18.440 --> 02:24.600 So Nuba has bucket policies. These are just the rules that tell where you should put the 02:24.600 --> 02:31.400 data like you can define these policies by yourself. They are customizable. So these policies 02:32.280 --> 02:38.680 they provide these multi cloud capabilities like, you know, replication, mirroring, 02:38.680 --> 02:45.400 tidying, and name space. And the goal here is to again, like I said, to simplify your job, 02:45.400 --> 02:50.920 you can manage your data across diverse environments using the same platform. 02:52.600 --> 02:59.320 So why you shouldn't use Nuba? It is again, it's a multi cloud solution. So this eliminates 02:59.400 --> 03:04.520 the vendor lock in, you know, you're not tied to a single vendor here. You can have your 03:05.480 --> 03:11.480 multi clouds. You can also have a on-prem, self-cluster as a storage back in for Nuba. 03:11.480 --> 03:17.800 You also can use your regular storage drives as well. So it's not, you know, tied to a specific 03:17.800 --> 03:24.760 vendor. Again, it provides a S3 compatible API. So whenever you put your data, you put it through 03:24.840 --> 03:32.280 your S3 API. So it's a standard, you know, de facto standard API. So again, not tied to any specific 03:32.280 --> 03:40.120 vendor. Again, this increases the high availability of your data. You can have a single bucket 03:40.120 --> 03:46.360 mirror across different platforms. So let's say one of your back in storage goes down. You still 03:46.360 --> 03:54.200 have your data available at some other place. So you still have your data. Again, it's a storage 03:54.280 --> 04:01.080 orchestrator tool. So it can, again, manage your storage. It consumes storage from diverse 04:01.080 --> 04:07.000 environments. It also provides bucket policies that are highly customizable. So you can 04:08.520 --> 04:13.720 set rules where you should put your data depending on, you know, your requirements. 04:14.760 --> 04:22.360 Again, these, these are very highly dynamic. So a lot of times our project starts with one set of 04:22.360 --> 04:28.840 requirements and then it ends up with something else. Your architecture changes, your demands might 04:28.840 --> 04:34.120 change. You may start with a block storage, then you might just start requiring objects storage, 04:34.120 --> 04:40.040 your platform requirements can also change. So in that, in those scenarios, these bucket policies 04:40.040 --> 04:48.920 are very helpful. You can just change them dynamically and use them. And again, we on top of your 04:49.000 --> 04:57.320 existing storage layer, we also get compression, de-deplication and encryption. So this also 04:57.320 --> 05:07.400 adds a security, like added layer of security. Again, we also get agility in Nuba. We can migrate 05:07.400 --> 05:15.400 the data without any disruption and services. So, you know, in imagine, you have a setup that you 05:15.400 --> 05:22.840 used, you know, you built it on-prem. Right. Now, your client requires some cloud involvement in 05:22.840 --> 05:29.000 that setup. Now, you have acquired cloud license, you have cloud storage, but how are you going 05:29.000 --> 05:35.160 to migrate the data? In Nuba, you can just use a mirrored bucket. You can, you know, whatever 05:35.160 --> 05:41.080 your existing bucket, you can just mirror it. Your data will be mirrored directly to your cloud storage. 05:41.160 --> 05:48.360 Now, you can get rid of your on-prem bucket, that you can also do. General workloads for objects 05:48.360 --> 05:54.440 storage, they can be media backed up or archival type of, you know, workload. This is just a generic 05:55.400 --> 06:03.400 thing. Oh, can you do the next? So, this is just a basic overview for Nuba, like how we usually 06:03.480 --> 06:13.400 see Nuba when you talk about your client and backend sort of a thing. So, you have your applications, 06:13.400 --> 06:22.600 your applications will directly talk to your S3 APIs, that Nuba S3 API. Your Nuba S3 API does have, 06:22.600 --> 06:28.840 you know, it processes the data. And in the back and you have all those storage, you know, 06:28.920 --> 06:36.760 back and such as AWS, Nuba will, it could also be a safe or any multiple different location, 06:37.400 --> 06:45.000 you know, your backend storage can be. This is the Nuba platform, when your client puts data 06:45.000 --> 06:52.840 in the S3 API, the Nuba S3 endpoint, it will process the data, depending on how you have configured 06:52.840 --> 06:59.080 your bucket policies, it will put the data across these diverse, you know, backend. So, for 06:59.080 --> 07:05.640 example, if you have multi-cloud buckets, your data will be spread across these multiple clouds. 07:05.640 --> 07:12.040 If you have hybrid, it would be on-prem and some other cloud and multi-site, it would be on 07:12.040 --> 07:18.600 different multiple locations. So, for the scope of this topic, we are talking about Nuba 07:18.680 --> 07:24.520 deployed on a Kubernetes platform, because that's where, you know, most of the agile workload, 07:24.520 --> 07:34.600 you know, we have seen customers deploy, right? So, next. So, this is what happens to your 07:34.600 --> 07:44.520 data when you put it in your Nuba S3 API. So, this is just a example of Mona Lisa, the picture. 07:45.480 --> 07:55.080 So, you put it in your S3 endpoint, it will firstly get in fragmented, we fragment the data in 07:55.080 --> 08:01.800 4 MB chunks, then it gets de-deplication, you know, if it's there is any repetitive data, it will 08:01.800 --> 08:10.280 be eliminated, then we compress the data for better storage utilization, and then the encryption 08:10.280 --> 08:16.760 takes place. Once you encrypt the data depending upon how you have configured the bucket 08:16.760 --> 08:27.320 policies, it will be stored across different environments. Next. So, these are the basic core components 08:27.320 --> 08:35.880 for Nuba, you have Nuba core, you have Nuba endpoint, and you have Nuba agent. Next. So, this is the 08:35.960 --> 08:42.840 architecture, we will talk about the data flow here, we have your application writing data directly 08:42.840 --> 08:50.680 to the S3 endpoint, then we again here have Nuba core, Nuba DB, and Nuba agent at demon. 08:51.480 --> 08:58.040 So, your data once it goes inside your S3 endpoint, it will do all the heavy lifting, 08:58.040 --> 09:05.240 it will, you know, compress the duplicate, encrypt the data, then it will send the metadata to 09:05.240 --> 09:11.480 your Nuba core, Nuba core will then process the metadata, you know, generally metadata contains 09:11.480 --> 09:18.440 the bucket details, where you want to put the data which bucket, and what bucket policies that 09:18.440 --> 09:25.560 bucket has. So, for example, if I have a multi-cloud bucket, I am putting data in a bucket 09:25.560 --> 09:31.720 B1, and it is a multi-cloud bucket, that information will go to the Nuba core, and Nuba DB 09:32.040 --> 09:38.680 stores a lot of bucket related metadata like where your objects are stored, what bucket, 09:38.680 --> 09:43.880 they are stored, and what is the bucket policies, all the configuration related details. 09:43.880 --> 09:49.880 So, with the help of Nuba database, Nuba core will calculate where to put the data, and it will send 09:49.880 --> 09:57.080 the instructions back to the Nuba endpoint, and then Nuba endpoint will actually store the data 09:57.160 --> 10:03.720 in your back end depending on where it is. So, once the data is stored, it will send that knowledge 10:03.720 --> 10:10.280 meant back to your Nuba core, and then Nuba core will update your Nuba database, and, you know, 10:10.280 --> 10:18.120 the cycle that is how it is completed. We here we have, you know, scalable Nuba storage node, 10:18.120 --> 10:27.000 which basically is a Nuba agent, it is a demon that keeps track of statuses of all these components 10:27.400 --> 10:32.520 in case your back end storage is not available, it will try to, you know, think that and make 10:32.520 --> 10:39.480 sure that it is available if it is not, it will market us down, and yeah, basically it keeps track 10:39.480 --> 10:47.240 of it, and updates the status, again it goes back to Nuba core, and sends what all data, 10:47.240 --> 10:53.560 you know, and I will fix they have gathered, and it again updates to Nuba DB with the current 10:53.640 --> 10:59.000 statuses of the, you know, the whole cluster, and, you know, again cycle continues. 11:00.360 --> 11:08.120 This is the data path for Nuba architecture, again Nuba DB here is, as you might have seen, 11:08.120 --> 11:14.840 it is very important, it is a core component for Nuba, because it contains all the bucket information, 11:16.200 --> 11:23.080 such as configuration, it contains account and access keys, bucket policies, again it contains 11:23.640 --> 11:29.320 the metadata related to that backing store and cloud connection information is also there, 11:30.360 --> 11:37.720 mapping of object to bucket, right, where my each object is stored at what location, that is also 11:37.720 --> 11:45.000 stored inside your Nuba DB, it also has encryption keys, it also has system status and statistics, 11:45.000 --> 11:51.640 so yeah, you might have gotten an idea like how important it is, so without this Nuba DB, 11:51.720 --> 11:58.440 we cannot, you know, access our data, because again, all the access related information is stored 11:58.440 --> 12:09.240 inside here, so what we can do to, what we can do, you know, to understand how we can fix this 12:09.240 --> 12:16.680 situation, in case something goes wrong, right, so we have Vaishnavi discussing what we can do. 12:21.640 --> 12:47.320 So, till now we discuss the architecture of the Nuba, why we should use the Nuba and what is the 12:47.400 --> 12:54.280 workflow, now we will discuss the understanding and single point of failure in Nuba, so as we discussed 12:54.280 --> 13:00.920 earlier, Nuba, Nuba metadata, Nuba DB is very important, because it knows the location of the 13:00.920 --> 13:06.440 object, where the actual object stored in the background, whenever we write any data, it is metadata 13:06.440 --> 13:12.040 gets written in the Nuba DB, so it has to go to the Nuba DB and when we want to read the data 13:12.040 --> 13:16.840 from the back end storage, we need to know the location, so again we need to go to the Nuba DB, 13:16.840 --> 13:23.160 to know the location, to read the object, so Nuba DB stores the crucial metadata, every 13:23.160 --> 13:29.880 of some Nuba database storage is scalable to handle the increase load and ensuring the flexibility, 13:29.880 --> 13:35.480 so at time passes, our storage might get increased along with the metadata, so if we want to scale 13:35.480 --> 13:40.200 out the increase the size of Nuba DB, we can do that, we can just increase the size, so that's 13:40.200 --> 13:46.040 what flexibility it is, so as mentioned, every operation of the Nuba DB has to pass to the 13:46.920 --> 13:51.880 database, so in case of any failure happens, it leads to the disconnection between the data 13:51.880 --> 13:57.640 and the client IO, which results in the system failure, again having repeating, if the data 13:57.640 --> 14:07.320 base layer is corrupted, it's obviously going to impact our overall Nuba cluster, so how you 14:07.320 --> 14:14.520 how you will identify the Nuba DB is corrupted, so was from the client side, client is unable 14:14.520 --> 14:22.120 to write the data or unable to read the data from the cluster, and then and also if the client 14:22.120 --> 14:27.960 is unable to create a new object claims from the cluster, so this is what customer will observe, 14:27.960 --> 14:33.640 but from the system administrative perspective, we will see that Nuba pod is fails to come up, 14:33.640 --> 14:39.400 how we will troubleshoot this kind of issues, so if Nuba pod is not coming up, then we can just 14:39.400 --> 14:44.760 start with checking the describe output of the animal, and then we can check the events of that 14:44.760 --> 14:49.960 specific namespace, in the namespace, in the events, if we observe a specific found error on the 14:49.960 --> 14:57.080 device, specifically on they are BD0, if the back end store register, then we will have RBD, 14:57.080 --> 15:04.280 and could not found, then there is a possibility of the database corruption, in the second scenario, 15:04.360 --> 15:10.440 if the Nuba DB pod is failed to start, we will observe these kind of messages in the Nuba DB 15:10.440 --> 15:15.880 pod log, that is could not start server, and from the database log, we will see could not 15:15.880 --> 15:24.760 locate and valid the checkpoint record, so this scenario will observe if the node get crashed 15:24.760 --> 15:32.520 unexpectedly or rebooted an expectedly, if all Nuba pods are up and running, but still object 15:32.520 --> 15:37.240 bucket claims are not getting created or still claims are impacted, then we can check from the 15:37.240 --> 15:43.400 backing store perspective, if the backing stores talk in a connecting phase, and if our devidum fails 15:43.400 --> 15:48.360 with this kind of error that is invalid page in the block, then again there is a possibility of 15:48.360 --> 15:53.720 the data loss, there might be more possibilities, but these are some general scenarios that we 15:53.720 --> 16:00.200 have seen till now, so this is how we can identify that Nuba DB is corrupted, no one wants 16:00.200 --> 16:08.200 are cluster in that scenario, so let us see how there are few ways we can mitigate this 16:08.200 --> 16:16.360 single point of failure, let us take a look one by one, so first way is using an external 16:16.360 --> 16:23.720 post-grace query, in an generally in Nuba, we use a single instance of Nuba PDC, but the external 16:23.800 --> 16:30.440 post-grace query cluster, use Nuba DB as it provide the database lever replication, so in case 16:30.440 --> 16:37.160 of any corruption happens of database, we will still have another copy available, so we will not 16:37.160 --> 16:43.560 lose the access to the database, so we can use the external post-grace query cluster to avoid 16:43.560 --> 16:50.120 the single point of failure, external post-grace query cluster also supports the SSL, let us take 16:50.440 --> 16:55.560 look the prerequisites if we want to use the external database cluster, first obviously we need 16:55.560 --> 17:01.560 to have the valid working post-grace query, accessible input for the external point, second 17:01.560 --> 17:07.480 the empty details where we can store the Nuba DB's metadata, and for that we can use this command 17:07.480 --> 17:13.240 to create specifically with this LT code locate, third one is we should have correct user name and 17:13.240 --> 17:22.760 password to create a retables in the database, so in this method we will going to clone the 17:22.760 --> 17:29.160 Nuba DB PVC periodically, so Kubernetes provide the feature called cloning which is pointing 17:29.160 --> 17:35.320 thing copy of the data, so if we create a clone of the PVC in the back inside, it will create 17:35.320 --> 17:42.120 one more PVC and it will store the point in them copy of the data on that PVC, cloning supports 17:42.760 --> 17:49.000 CSI driver as well as dynamic provisioning, so we can clone the DB PVC periodically to take a 17:49.000 --> 17:55.480 backup and whenever require we can just restore it, so we can to do that we can create Kubernetes 17:55.480 --> 18:01.560 job in the Kubernetes cluster, so basically it's system administrator responsibility to decide 18:01.560 --> 18:07.640 how many jobs they want per day and their retention period as well, because whenever as we are 18:07.640 --> 18:13.160 creating a clones it will also going to consume the space in the cluster, let's take an example 18:13.160 --> 18:20.280 if my current Nuba DB is 50 GB and I'm three backups are taken per day, then it will be 50 18:20.280 --> 18:28.040 to three, 150 GB data will require to take a backup and also if the back end storage is 18:28.040 --> 18:34.200 set then replication must be considered, so keeping all this thing in the bin we need to decide 18:34.200 --> 18:41.960 how many backups and their retention period as well, let's take a look to step to restore the 18:41.960 --> 18:49.400 DB using the clone, as we need to clone the DB PVC periodically to have the up to date backups, 18:49.400 --> 18:55.960 for restoring we can just scale down the Nuba DB state for set, delete the older corrupted data 18:55.960 --> 19:01.880 and we need to identify the latest healthy clone because that would be obviously before the data 19:01.960 --> 19:07.320 corruption happens and then we can create a new clone with the original PVC named because whenever 19:07.320 --> 19:14.760 we scale up the Nuba DB set a state full set it will look for that specific PVC name that's why 19:14.760 --> 19:20.040 we need to give the original PVC named and then we can scale up the Nuba DB state full set 19:20.040 --> 19:27.160 and we can check the port status it should be up and running next, so in the previous method 19:27.160 --> 19:33.080 actually we took a backup of the whole Nuba DB PVC but here in this scenario we will take a backup 19:33.080 --> 19:39.000 we will take a dump of the database, so in case of any failure happens we can just restore the 19:39.000 --> 19:45.240 Hindi Nuba DB dump from the back of folder let's take a look at the steps how can we do that 19:46.440 --> 19:52.680 so steps to restore the data, we should take a backup of the all Nuba related secrets first 19:52.760 --> 19:58.120 to the local folder and then back of the post-gay school database to a local folder, 19:58.120 --> 20:04.360 these just an example come on, we need to go to the Nuba DB port first and then we'll just 20:04.360 --> 20:10.120 pj dump and be core to take a backup and as we took a backup inside the port we need to copy 20:10.120 --> 20:16.680 from to the local machine, this is copy come on and if any corruption happens for the restoration 20:16.680 --> 20:24.280 we need to stop the Nuba DB service before we restore and then verify the all components are 0 20:24.280 --> 20:29.640 and then obviously we need to clear the older corrupted data and then restore from the local folder 20:29.640 --> 20:36.200 again the back of folder is on local machine, so we have to copy that inside the port and then 20:36.200 --> 20:41.640 we'll just run the pj restore command so it will just restore the healthy data from 20:42.360 --> 20:48.200 that dump to the database and then we'll again delete the older corrupted secrets and we'll 20:48.200 --> 20:53.400 restore them from the local folder and then we can destroy to restart the Nuba service it will 20:53.400 --> 21:00.040 be up and running, so these are three ways that we can follow to avoid a single point of failure 21:00.040 --> 21:02.680 in the Nuba, so anybody has any question?