Istio sidecars cause setup-credentials-job to never finish

Design Discussions
1

Hello Yugabyte Community,

When you deploy the Yugabyte helm chart in a kubernetes cluster with istio sidecar injection enabled the setup-credentials-job never reaches “Complete” status. This is because even though the setup-credentials-job succeeds without issues, the istio sidecar in that pod keeps running. Kubernetes sees that the pod still has containers running in it and thinks the job is still running and never marks it as “Complete”.

This is an issue especially when using terraform to deploy the helm chart as the deployment of the helm chart times out and causes the terraform script to fail.

I have already created an issue about this on github ( Istio sidecars cause setup-credentials-job to never finish · Issue #206 · yugabyte/charts · GitHub ) with a proposed solution:
Explicitly shut down the sidecar at the end of the setup-credentials.sh script (optionally only do this if the script had no errors) with “curl -fsI -X POST http://localhost:15020/quitquitquit”. This works perfectly in other jobs that previously had exactly this issue.

I would really appreciate some feedback on this as this has been an issue for a while!

2

Hi @leonlenz

Thank you for reporting!

Let me get someone from k8s team to check this out.

1 Like
3

Hi @leonlenz

We don’t add sidecars to that job, it looks to be something you have setup on your own, correct?

It does not make sense for us to call apis for a sidecar we are not placing there.

4

Hi @dorian_yugabyte,

Thanks for the clarification — agreed the chart isn’t adding a sidecar.

The issue shows up in a very common Istio setup: when you run Envoy sidecars for essentially all in-cluster service-to-service traffic (mTLS / policy / telemetry), Kubernetes Jobs also end up with an istio-proxy container. In that case the job’s main container can exit successfully, but the proxy can keep the Pod “alive”, so the Job never reaches “Complete” and Helm/Terraform can time out.

For anyone wondering what worked cleanly for me without any Yugabyte chart changes was upgrading to Kubernetes native sidecars and letting Istio use them:

  • Kubernetes: v1.29+ (the SidecarContainers capability is enabled by default there, and Kubernetes explicitly handles Job completion correctly with native sidecars).

  • Istio: v1.27+ (native sidecars enabled by default for eligible pods; ENABLE_NATIVE_SIDECARS defaults to true).

After moving to that combination, Jobs like “setup-credentials-job” complete normally and Terraform no longer gets stuck waiting.