Issue with tls and cert-manager

General
1

Hi! I’m trying to deploy a yugabyte cluster in k8s with helm and I enabled tls using cert-manager and a clusterIssuer.
The certificates seems to have been generated properly, but from some reason the yb-master-X pods are failing with the following error:

I0129 16:05:59.150353    95 refined_stream.cc:231] SECURE[C] kHandshake { local: 172.16.8.149:36421 remote: 172.16.11.74:7100 }: Handshake failed: Network error (yb/rpc/secure_stream.cc:914): Handshake failed: Network error (yb/rpc/secure_stream.cc:1129): Unverified certificate: unable to get local issuer certificate, address: 172.16.11.74, hostname: yb-master-2.yb-masters.database.svc.cluster.local`

here is the config I used:

tls:
  # Set to true to enable the TLS.
  enabled: true
  nodeToNode: true
  clientToServer: true
  insecure: false
  certManager:
    enabled: true
    bootstrapSelfsigned: false
    useClusterIssuer: true
    clusterIssuer: vault-issuer-db
    certificates:
      duration: 2160h # 90d
      renewBefore: 360h # 15d
      algorithm: RSA # ECDSA or RSA
      keySize: 4096
  rootCA:
    cert: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS...

In my setup I have the following certificate chain:
ROOT CA (self signed) → ica1 → icadb → server certificate
the value of rootCA.cert is the ROOT CA
in the yb-master-yugabyte-tls-cert secret, the tls.crt value contains the server certificate followed by the 2 int ca (icadb and ica1) and I verified the certificate validation against the ca chain and it’s valid.
openssl verify -CAfile rootca -untrusted <(cat icadb ica1) crt
crt: OK
version: yugabytedb/yugabyte:2024.2.1.0-b185
any idea why I’m getting this issue?

2

Just to add a little bit more details here. I validated the certificate chain using openssl in the pod:

[root@yb-master-0 certs]# cd /mnt/disk0/certs

[root@yb-master-0 certs]# ls -alh
total 24K
drwxr-xr-x 2 root root 4.0K Feb  3 17:50 .
drwxr-xr-x 6 root root 4.0K Feb  3 19:26 ..
-rw------- 1 root root 2.0K Feb  3 17:50 ca.crt
-rw------- 1 root root 5.7K Feb  3 19:26 node.yb-master-0.yb-masters.database.svc.cluster.local.crt
-rw------- 1 root root 3.2K Feb  3 19:26 node.yb-master-0.yb-masters.database.svc.cluster.local.key

[root@yb-master-0 certs]# hostname -f
yb-master-0.yb-masters.database.svc.cluster.local

[root@yb-master-0 certs]# openssl verify -trusted ca.crt -untrusted node.yb-master-0.yb-masters.database.svc.cluster.local.crt node.yb-master-0.yb-masters.database.svc.cluster.local.crt
node.yb-master-0.yb-masters.database.svc.cluster.local.crt: OK
3

Ok, I think I know what is causing the issue. It seems that if we have 1 rootCA and we generate a certificate signed by that CA, (no intermediate) it works.
I tested it using the selfsigned configuration from helm

tls:
  # Set to true to enable the TLS.
  enabled: true
  nodeToNode: true
  clientToServer: true
  # Set to false to disallow any service with unencrypted communication from joining this cluster
  insecure: false
  # Set enabled to true to use cert-manager instead of providing your own rootCA
  certManager:
    enabled: false
    # Will create own ca certificate and issuer when set to true
    bootstrapSelfsigned: true
    # Use ClusterIssuer when set to true, otherwise use Issuer
    useClusterIssuer: false
    certificates:
      # The lifetime before cert-manager will issue a new certificate.
      # The re-issued certificates will not be automatically reloaded by the service.
      # It is necessary to provide some external means of restarting the pods.
      duration: 2160h # 90d
      renewBefore: 360h # 15d
      algorithm: RSA # ECDSA or RSA
      # Can be 2048, 4096 or 8192 for RSA
      # Or 256, 384 or 521 for ECDSA
      keySize: 4096

I have tested with 1 intermediate and it also fails.
Is there a way to make it work when Intermediate CA are used?

4

Hi, @js.frerot

Thanks we will try to reproduce this internally and respond to the thread shortly.

Meanwhile some questions:
Do you have the full chain updated to cert manager issuer?
Can you try adding the full chain to rootCA as well?

Thanks
Aman

5

Hi, for the cert-manager ClusterIssuer, the value of the vault.caBundle contains the full chain (IntCaDB. IntCA1 and RootCA)

As for testing with having the full CA chain in the ca.crt file, it seems to work. I manually generated certificates and key and added the full chain in the ca.crt and the cluster is working fine.

When using cert-manager, only the rootCA is in the ca.crt, but the certificate + the certificate chain is in the node.yb-master-0.yb-masters.database.svc.cluster.local.crt and with this setup YB is not able to start.

6

Hi @js.frerot

We have isolated this issue internally and managed to reproduce this

We have an issue in the support the intermediate certs in the node.*local.crt file.
The workaround meanwhile would be to populate the rootCA manually.

7

Hi @Aman_Nijhawan, not sure the workaround is clear as I have tried to put the full CA chain (2x int + RootCA) in the following helm setting:
tls.rootCA.cert
and on the pods, only the single RootCA appears.
This is when trying to use cert-manager and providing the rootCA as follow:

tls:
  # Set to true to enable the TLS.
  enabled: true
  nodeToNode: true
  clientToServer: true
  insecure: false
  certManager:
    enabled: true
    bootstrapSelfsigned: false
    useClusterIssuer: true
    clusterIssuer: vault-issuer-db
    certificates:
      duration: 1h # 90d
      renewBefore: 10m # 15d
      algorithm: RSA # ECDSA or RSA
      keySize: 4096
  rootCA:
    cert: LS0tLS1CRUdJTiBDRVJ...

I don’t believe there is a workaround when using cert-manager.
You have to fully set the certificates manually or use the bootstrap feature which doesn’t use cert-manager.