# How to create a multi-cluster deployment on Kubernetes? **URL:** \ **Category:** General\ **Created:** [May 29, 2019, 8:05pm UTC](https://forum.yugabyte.com/t/how-to-create-a-multi-cluster-deployment-on-kubernetes/370 "2019-05-29T20:05:07Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![thecodeassassin-mycu](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/thecodeassassin-mycu/32/99_2.png) [@thecodeassassin-mycu](https://forum.yugabyte.com/u/thecodeassassin-mycu)\ **Post date:** [May 29, 2019, 8:05pm UTC](https://forum.yugabyte.com/t/how-to-create-a-multi-cluster-deployment-on-kubernetes/370/1 "2019-05-29T20:05:08Z")
Hi there, We are currently operating Kubernetes clusters (1.12+) in GKE and AWS (kops). We are looking to expand to Alibaba for China and Azure for Africa. We are looking for a globally distributed, strongly consistent database. We are interested in YugabyteDB primarily because of YEDIS, which allows us to start using YugaByteDB with next to no rewrites needed in the code (except we need to use smembers instead of SSCAN which can negatively affect read performance, at least on classic Redis). We would like to deploy YugaByteDB using helm charts, but we do have a couple of questions. 1. Why is there no repository available for YugaByteDB? 2. How do we create a multi-cluster setup using the Helm charts, at least on GKE so we can start testing it in multiple regions? 3. How performant is Yedis compared to the other APIs? Looking forward to having a discussion about this topic! Cheers ---
**Author:** ![ramkumar](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/ramkumar/32/64_2.png) [@ramkumar](https://forum.yugabyte.com/u/ramkumar)\ **Post date:** [May 29, 2019, 8:12pm UTC](https://forum.yugabyte.com/t/how-to-create-a-multi-cluster-deployment-on-kubernetes/370/2 "2019-05-29T20:12:06Z")
> [@thecodeassassin-mycu](#): > > - Why is there no repository available for YugaByteDB? What do you mean by no respository for YugaByteDB, we do have source code repository on github. And the docker images on dockerhub [Source Code](https://github.com/YugaByte/yugabyte-db/) [Docker image](https://hub.docker.com/r/yugabytedb/yugabyte) > [@thecodeassassin-mycu](#): > > How do we create a multi-cluster setup using the Helm charts, at least on GKE so we can start testing it in multiple regions? I am assuming you guys have setup multi region cluster in GKE which can talk to each other, right now GKE out of the box supports regional clusters and zonal clusters. > [@thecodeassassin-mycu](#): > > 1. How performant is Yedis compared to the other APIs? I would let @amitanand or @karthik share the performance comparison. ---
**Author:** ![thecodeassassin-mycu](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/thecodeassassin-mycu/32/99_2.png) [@thecodeassassin-mycu](https://forum.yugabyte.com/u/thecodeassassin-mycu)\ **Post date:** [May 29, 2019, 8:16pm UTC](https://forum.yugabyte.com/t/how-to-create-a-multi-cluster-deployment-on-kubernetes/370/3 "2019-05-29T20:16:59Z")
> [@ramkumar](#): > > What do you mean by no respository for YugaByteDB, we do have source code repository on github. And the docker images on dockerhub Many companies such as Bitnami have a public chartmuseum repository to make it easy and straightforward to install software using helm. See: [GitHub - bitnami/charts: Bitnami Helm Charts](https://github.com/bitnami/charts). That way we don’t have to clone the repo or create some kind of CI integration for this in order to keep our software up-to-date. Also it makes it more straightforward to deploy the software for everyone. > [@ramkumar](#): > > I am assuming you guys have setup multi region cluster in GKE which can talk to each other, right now GKE out of the box supports regional clusters and zonal clusters. They don’t actually talk to each other, that would probably have to be done using a service mesh or public load balancers. ---
**Author:** ![ramkumar](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/ramkumar/32/64_2.png) [@ramkumar](https://forum.yugabyte.com/u/ramkumar)\ **Post date:** [May 29, 2019, 8:22pm UTC](https://forum.yugabyte.com/t/how-to-create-a-multi-cluster-deployment-on-kubernetes/370/4 "2019-05-29T20:22:57Z")
> [@thecodeassassin-mycu](#): > > That way we don’t have to clone the repo or create some kind of CI integration for this in order to keep our software up-to-date. Also it makes it more straightforward to deploy the software for everyone. > > ![](https://canada1.discourse-cdn.com/flex027/uploads/yugabyte/original/1X/52d2e5865438326559497b48721a1b750d27144e.jpeg) ramkumar: ah, yes, i am currently working on that, should wrap that up by End of this month or so, we have forked the opensource charts repo for now [yugabyte chart](https://github.com/YugaByte/charts/tree/master/stable/yugabyte) ---
**Author:** ![karthik](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/karthik/32/13_2.png) [@karthik](https://forum.yugabyte.com/u/karthik)\ **Post date:** [May 30, 2019, 4:51am UTC](https://forum.yugabyte.com/t/how-to-create-a-multi-cluster-deployment-on-kubernetes/370/5 "2019-05-30T04:51:49Z")
Hi @thecodeassassin-mycu, > [@thecodeassassin-mycu](#): > > How do we create a multi-cluster setup using the Helm charts, at least on GKE so we can start testing it in multiple regions? I think this is the core question. @ramkumar will post the set of steps to get you going. Here is the design doc for it: [Multi-Zone and Multi-Cluster YB deployment in Kubernetes](https://forum.yugabyte.com/t/multi-zone-and-multi-cluster-yb-deployment-in-kubernetes/316) Note the following in case of multi-cluster kubernetes setup: > **the clusters have connectivity amongst themselves, that is, the pods in one cluster can resolve the FQDNs of the pods in the other cluster(s)** To answer your question on Yedis: > [@thecodeassassin-mycu](#): > > How performant is Yedis compared to the other APIs? Yedis is comparable in performance to the other APIs. Simple key-value read operations are generally sub-millisecond. However, note that no active work is happening on Yedis at the moment, please read the note here for more details: [YEDIS | YugabyteDB Docs](https://docs.yugabyte.com/latest/yedis/) ---
**Author:** ![ramkumar](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/ramkumar/32/64_2.png) [@ramkumar](https://forum.yugabyte.com/u/ramkumar)\ **Post date:** [May 30, 2019, 6:39pm UTC](https://forum.yugabyte.com/t/how-to-create-a-multi-cluster-deployment-on-kubernetes/370/6 "2019-05-30T18:39:19Z")
Here is the instructions on how you can run YugaByte db on a regional cluster (multi zone) on GKE. This same instructions can be used for multi region clusters as well. Pre-Req: 1. Create a Regional cluster with minimum 4 nodes, with 8 CPU each 2. Clone YugaByte charts repository: [GitHub - yugabyte/charts: Respository of all helm charts that are owned by YugabyteDB for database deployment acroos various K8s configurations](https://github.com/YugaByte/charts.git) 3. Make sure helm chart 2.8.0+ is installed. Installation Steps: Step1: Create Storage Class and Overrides template files `$ mkdir yb-multi-az` `$ echo 'kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: standard-##zone## provisioner: kubernetes.io/gce-pd parameters: type: pd-standard replication-type: none zone: ##zone##"' > yb-multi-az/storage-class-template.yml` `$ echo 'isMultiAz: True AZ: "##zone##" storage: master: storageClass: "standard-##zone##" tserver: storageClass: "standard-##zone##" masterAddresses: "yb-master-0.yb-masters.yb-demo-##zone1##.svc.cluster.local:7100, yb-master-0.yb-masters.yb-demo-##zone2##.svc.cluster.local:7100, yb-master-0.yb-masters.yb-demo-##zone3##.s vc.cluster.local:7100" replicas: master: 1 tserver: 1 totalMasters: 3 gflags: master: placement_cloud: "kubernetes" placement_region: "##region##" placement_zone: "##zone##" tserver: placement_cloud: "kubernetes" placement_region: "##region##" placement_zone: "##zone##"' > yb-multi-az/overrides-template.yml` Step 2: Setup RBAC `$ cd charts/stable/yugabyte/` `$ kubectl apply -f yugabyte-rbac.yaml` Step 3: Generate Zone specific overrides First fetch the zone and region labels by running this command `$ kubectl get nodes -Lfailure-domain.beta.kubernetes.io/region -Lfailure-domain.beta.kubernetes.io/zone` ex: lets say your region label is us-west1 and zone labels are us-west1-a, us-west1-b, us-west1-c you would generate the zone specific overrides like below Below command shows for one zone(us-west1-a), Repeat this step for each of the zones. `$ sed 's/##zone##/us-west1-a/g' storage-class-template.yml > yb-multi-az/storage_class_us-west1-a.yml` `$ sed 's/##zone##/us-west1-a/g; s/##region##/us-west1/g; s/##zone1##/us-west1-a/g; s/##zone2##/us-west1-b/g; s/##zone3##/us-west1-c/g' overrides-template.yml > yb-multi-az/overrides-us-west1-a.yml` Step 4: Create the storage class Note: repeat this command for all the zones `$ kubectl applly -f yb-multi-az/storage_class_us-west1-a.yml` Step 5: Initialize helm and have it use the service account we created on Step 2: `$ helm init --service-account yugabyte-helm --upgrade —wait` Step 6: Run helm install. Note: repeat this command for all the zones `$ helm install charts/stable/yugabyte --namespace yb-demo-us-west1-a --name yb-demo-us-west1-a -f yb-multi-az/overrides-us-west1-a.yml` Step 7: Wait for all the containers to come up Below command would list all the pods that are specific to yugabytedb `$ kubectl get pods --all-namespaces -lcomponent=yugabytedb` Step 8: Access the db admin UI Below command would fetch all the services which are expose via loadBalancer. `$ kubectl get svc --all-namespaces -lcomponent=yugabytedb --field-selector=metadata.name=yb-master-ui` If you run into issues please hop on our community slack channel and one of us can help debug ---
**Author:** ![karthik](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/karthik/32/13_2.png) [@karthik](https://forum.yugabyte.com/u/karthik)\ **Post date:** [June 5, 2019, 5:33am UTC](https://forum.yugabyte.com/t/how-to-create-a-multi-cluster-deployment-on-kubernetes/370/7 "2019-06-05T05:33:09Z")
@thecodeassassin-mycu - let us know if the above instructions worked for you. Please feel free to reach out if you need any help. ---
**Author:** ![thecodeassassin-mycu](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/thecodeassassin-mycu/32/99_2.png) [@thecodeassassin-mycu](https://forum.yugabyte.com/u/thecodeassassin-mycu)\ **Post date:** [June 5, 2019, 12:07pm UTC](https://forum.yugabyte.com/t/how-to-create-a-multi-cluster-deployment-on-kubernetes/370/8 "2019-06-05T12:07:52Z")
@karthik Could you perhaps post the actual yaml files? It’s hard to format them correctly. Also it’s still not really clear to me how to apply this to multiple clusters and have them appear as a single YugabyteDB. We already run a regional cluster so most of the pods are already spread out correctly so this doesn’t really solve anything for us. ---
**Author:** ![karthik](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/karthik/32/13_2.png) [@karthik](https://forum.yugabyte.com/u/karthik)\ **Post date:** [June 5, 2019, 3:26pm UTC](https://forum.yugabyte.com/t/how-to-create-a-multi-cluster-deployment-on-kubernetes/370/9 "2019-06-05T15:26:53Z")
Yes, totally @thecodeassassin-mycu @arnav - could you please post yaml files for this scenario? ---
**Author:** ![arnav](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/arnav/32/80_2.png) [@arnav](https://forum.yugabyte.com/u/arnav)\ **Post date:** [June 5, 2019, 6:51pm UTC](https://forum.yugabyte.com/t/how-to-create-a-multi-cluster-deployment-on-kubernetes/370/10 "2019-06-05T18:51:33Z")
@thecodeassassin-mycu Below is the overrides file snippet. I’ve also added comments to hopefully make the YAML more meaningful and hopefully answer your question on how the multiple helm deployments over multiple clusters combine into a single universe. ``` # To signify if the deployment spans multiple zones. isMultiAz: True # Set if you would like to deploy this deployments pods specifically on nodes in a particular zone # (affinitized by using node label: failure-domain.beta.kubernetes.io/zone) AZ: "" # Since in a single cluster spanning multiple zones, the PVC cannot be controlled by the helm deployment. # So to pin the pods to the required zone, we need to ensure that there is a storage class created just for that zone. storage: master: storageClass: "standard-" tserver: storageClass: "standard-" # The master addresses is how the YugaByteDB finds the rest of the members of the requested universe which may have been deployed # in different helm deployments (and by extension, in different zones/clusters) masterAddresses: "yb-master-0.yb-masters.yb-demo-.svc.cluster.local:7100, yb-master-0.yb-masters.yb-demo-.svc.cluster.local:7100, yb-master-0.yb-masters.yb-demo-.svc.cluster.local:7100" # The number of pods corresponding to each service that you would like to bring up in the particular helm deployment. # The totalMasters specifies the total number of masters in the universe (so essentially the required replacement factor) # The number of masters also specifies the min number of replicas in that zone, so always masters <= tservers. replicas: master: 1 tserver: 1 totalMasters: 3 # These are the flags that specifies to the master the placement information, and ensures that data is replicated according to # the user's request. gflags: master: placement_cloud: "kubernetes" placement_region: "" placement_zone: "" tserver: placement_cloud: "kubernetes" placement_region: "" placement_zone: "" ``` ---
**Author:** ![arnav](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/arnav/32/80_2.png) [@arnav](https://forum.yugabyte.com/u/arnav)\ **Post date:** [June 5, 2019, 6:55pm UTC](https://forum.yugabyte.com/t/how-to-create-a-multi-cluster-deployment-on-kubernetes/370/11 "2019-06-05T18:55:31Z")
The snippet for creating the storage class for a particular zone is the following: ``` kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: standard-us-west1-a provisioner: kubernetes.io/gce-pd parameters: type: pd-standard replication-type: none zone: us-west1-a --- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: standard-us-west1-b provisioner: kubernetes.io/gce-pd parameters: type: pd-standard replication-type: none zone: us-west1-b --- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: standard-us-west1-c provisioner: kubernetes.io/gce-pd parameters: type: pd-standard replication-type: none zone: us-west1-c --- ```