Issues with yugabytedb/yugabyte chart

General
1

I have been using your charts without issue so far in a kubernetes development environment.

Today I wanted to try an update from 2024.1.2.0-b77 to 2024.1.3.0-b105. Si I did change the value Image.tag for my helm deployment and run

helm upgrade yugabyte yugabytedb/yugabyte --namespace database --values applications/yugabytedb/values.yaml

The command finish it’s execution but the pods where never updated. I checked the describe of the statefulset and the config was correctly applied on it but the pod was still with the older version.
I did try to remove the pods one by one with kubectl -n database delete pod yb-tserver-0 and the new pod was still using the old image.
The only solution I found was to scale down and scale up with

kubectl -n database scale sts yb-tserver --replicas=0; kubectl -n database scale sts yb-tserver --replicas=3

If you have any suggestion for this issue (did you want me to report it on github?).

Also, I found that the chart tries to get the images from quay.io/yugabytedb/yugabyte but get unauthorized access and final fallback to docker hub and start the containers in the pod.

  Warning  Failed                  15m                kubelet                  Error: ErrImagePull
  Warning  Failed                  15m                kubelet                  Failed to pull image "yugabytedb/yugabyte:2024.1.3.0-b105": reading manifest 2024.1.3.0-b105 in quay.io/yugabytedb/yugabyte: unauthorized: access to the requested resource is not authorized
  Normal   BackOff                 15m                kubelet                  Back-off pulling image "yugabytedb/yugabyte:2024.1.3.0-b105"
  Normal   BackOff                 15m (x2 over 15m)  kubelet                  Back-off pulling image "yugabytedb/yugabyte:2024.1.3.0-b105"
  Warning  Failed                  15m (x2 over 15m)  kubelet                  Error: ImagePullBackOff
  Normal   BackOff                 15m (x2 over 15m)  kubelet                  Back-off pulling image "yugabytedb/yugabyte:2024.1.3.0-b105"
  Warning  Failed                  15m (x2 over 15m)  kubelet                  Error: ImagePullBackOff
  Warning  Failed                  15m                kubelet                  Error: ImagePullBackOff
  Normal   Pulling                 14m (x2 over 15m)  kubelet                  Pulling image "yugabytedb/yugabyte:2024.1.3.0-b105"
  Normal   Started                 14m                kubelet                  Started container yb-tserver
  Normal   Created                 14m                kubelet                  Created container yb-tserver
2

Hi @czotti

I’ve asked internally about this and will get back to you.

1 Like
3

This is the values.yaml for the chart before

Image:
  repository: "yugabytedb/yugabyte"
  tag: 2024.1.2.0-b77
  pullPolicy: Always
  pullSecretName: ""

and after

Image:
  repository: "yugabytedb/yugabyte"
  tag: 2024.1.3.0-b105
  pullPolicy: Always
  pullSecretName: ""
4

@czotti : To upgrade, you would typically refresh the helm charts repo and run a helm upgrade instead of just updating the tag.

However, that doesn’t explain why the images are being pulled from quay.io. Is your kubernetes cluster’s container runtime set to pull from quay.io by default, by any chance?

You can check the resulting manifest by running a helm template. When I do this at my end, I don’t see any references to quay.io. Yugabyte images are published to docker.io by default and are available without the need for a pull secret on docker.io (the quay images might need a pull secret though).

5

@sanketh I found my issue I defined the partition like that:

partition:
  master: 3
  tserver: 3

I removed this partition config (I think the default value is 0 for both) and the RollingUpdate is working correctly. Don’t ask me why I changed that I don’t remember.

For the quay.io part I checked docker manifest inspect and helm template and I see no reference to this url anywhere. I will check the configuration of cri-o on my nodes to see if I see something.

6

And I found the issues with our cluster crio is trying sometimes to pull images from quay.io.
It’s defined in the /etc/containers/registries.conf.d/crio.conf file.