# Issue with tls and cert-manager
**URL:** https://forum.yugabyte.com/t/issue-with-tls-and-cert-manager/4182
**Category:** General
**Created:** [February 3, 2025, 6:38pm UTC](https://forum.yugabyte.com/t/issue-with-tls-and-cert-manager/4182 "2025-02-03T18:38:19Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@js.frerot](https://forum.yugabyte.com/u/js.frerot)
#### Post date: [February 3, 2025, 6:38pm UTC](https://forum.yugabyte.com/t/issue-with-tls-and-cert-manager/4182/1 "2025-02-03T18:38:19Z")
Hi! I’m trying to deploy a yugabyte cluster in k8s with helm and I enabled tls using cert-manager and a clusterIssuer.
The certificates seems to have been generated properly, but from some reason the yb-master-X pods are failing with the following error:
```auto
I0129 16:05:59.150353 95 refined_stream.cc:231] SECURE[C] kHandshake { local: 172.16.8.149:36421 remote: 172.16.11.74:7100 }: Handshake failed: Network error (yb/rpc/secure_stream.cc:914): Handshake failed: Network error (yb/rpc/secure_stream.cc:1129): Unverified certificate: unable to get local issuer certificate, address: 172.16.11.74, hostname: yb-master-2.yb-masters.database.svc.cluster.local`
```
here is the config I used:
```auto
tls:
# Set to true to enable the TLS.
enabled: true
nodeToNode: true
clientToServer: true
insecure: false
certManager:
enabled: true
bootstrapSelfsigned: false
useClusterIssuer: true
clusterIssuer: vault-issuer-db
certificates:
duration: 2160h # 90d
renewBefore: 360h # 15d
algorithm: RSA # ECDSA or RSA
keySize: 4096
rootCA:
cert: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS...
```
In my setup I have the following certificate chain:
ROOT CA (self signed) → ica1 → icadb → server certificate
the value of rootCA.cert is the ROOT CA
in the yb-master-yugabyte-tls-cert secret, the tls.crt value contains the server certificate followed by the 2 int ca (icadb and ica1) and I verified the certificate validation against the ca chain and it’s valid.
openssl verify -CAfile rootca -untrusted \<(cat icadb ica1) crt
crt: OK
version: yugabytedb/yugabyte:2024.2.1.0-b185
any idea why I’m getting this issue?
---
### Author:  [@js.frerot](https://forum.yugabyte.com/u/js.frerot)
#### Post date: [February 3, 2025, 7:31pm UTC](https://forum.yugabyte.com/t/issue-with-tls-and-cert-manager/4182/2 "2025-02-03T19:31:57Z")
Just to add a little bit more details here. I validated the certificate chain using openssl in the pod:
```auto
[root@yb-master-0 certs]# cd /mnt/disk0/certs
[root@yb-master-0 certs]# ls -alh
total 24K
drwxr-xr-x 2 root root 4.0K Feb 3 17:50 .
drwxr-xr-x 6 root root 4.0K Feb 3 19:26 ..
-rw------- 1 root root 2.0K Feb 3 17:50 ca.crt
-rw------- 1 root root 5.7K Feb 3 19:26 node.yb-master-0.yb-masters.database.svc.cluster.local.crt
-rw------- 1 root root 3.2K Feb 3 19:26 node.yb-master-0.yb-masters.database.svc.cluster.local.key
[root@yb-master-0 certs]# hostname -f
yb-master-0.yb-masters.database.svc.cluster.local
[root@yb-master-0 certs]# openssl verify -trusted ca.crt -untrusted node.yb-master-0.yb-masters.database.svc.cluster.local.crt node.yb-master-0.yb-masters.database.svc.cluster.local.crt
node.yb-master-0.yb-masters.database.svc.cluster.local.crt: OK
```
---
### Author:  [@js.frerot](https://forum.yugabyte.com/u/js.frerot)
#### Post date: [February 3, 2025, 9:21pm UTC](https://forum.yugabyte.com/t/issue-with-tls-and-cert-manager/4182/3 "2025-02-03T21:21:21Z")
Ok, I think I know what is causing the issue. It seems that if we have 1 rootCA and we generate a certificate signed by that CA, (no intermediate) it works.
I tested it using the selfsigned configuration from helm
```auto
tls:
# Set to true to enable the TLS.
enabled: true
nodeToNode: true
clientToServer: true
# Set to false to disallow any service with unencrypted communication from joining this cluster
insecure: false
# Set enabled to true to use cert-manager instead of providing your own rootCA
certManager:
enabled: false
# Will create own ca certificate and issuer when set to true
bootstrapSelfsigned: true
# Use ClusterIssuer when set to true, otherwise use Issuer
useClusterIssuer: false
certificates:
# The lifetime before cert-manager will issue a new certificate.
# The re-issued certificates will not be automatically reloaded by the service.
# It is necessary to provide some external means of restarting the pods.
duration: 2160h # 90d
renewBefore: 360h # 15d
algorithm: RSA # ECDSA or RSA
# Can be 2048, 4096 or 8192 for RSA
# Or 256, 384 or 521 for ECDSA
keySize: 4096
```
I have tested with 1 intermediate and it also fails.
Is there a way to make it work when Intermediate CA are used?
---
### Author:  [@Aman\_Nijhawan](https://forum.yugabyte.com/u/Aman_Nijhawan)
#### Post date: [February 11, 2025, 4:03am UTC](https://forum.yugabyte.com/t/issue-with-tls-and-cert-manager/4182/4 "2025-02-11T04:03:16Z")
Hi, @js.frerot
Thanks we will try to reproduce this internally and respond to the thread shortly.
Meanwhile some questions:
Do you have the full chain updated to cert manager issuer?
Can you try adding the full chain to rootCA as well?
Thanks
Aman
---
### Author:  [@js.frerot](https://forum.yugabyte.com/u/js.frerot)
#### Post date: [February 11, 2025, 9:43pm UTC](https://forum.yugabyte.com/t/issue-with-tls-and-cert-manager/4182/5 "2025-02-11T21:43:23Z")
Hi, for the cert-manager ClusterIssuer, the value of the vault.caBundle contains the full chain (IntCaDB. IntCA1 and RootCA)
As for testing with having the full CA chain in the ca.crt file, it seems to work. I manually generated certificates and key and added the full chain in the ca.crt and the cluster is working fine.
When using cert-manager, only the rootCA is in the ca.crt, but the certificate + the certificate chain is in the node.yb-master-0.yb-masters.database.svc.cluster.local.crt and with this setup YB is not able to start.
---
### Author:  [@Aman\_Nijhawan](https://forum.yugabyte.com/u/Aman_Nijhawan)
#### Post date: [February 11, 2025, 10:55pm UTC](https://forum.yugabyte.com/t/issue-with-tls-and-cert-manager/4182/6 "2025-02-11T22:55:19Z")
Hi @js.frerot
We have isolated this issue internally and managed to reproduce this
>
>
> Jira Link: \[DB-15306\](https://yugabyte.atlassian.net/browse/DB-15306)
> \### Descri…ption
>
>
> 1. Create a root and intermediate CA and a server cert for 127.0.0.1 signed by the intermediate CA.
> 3. Place the root CA file in ca.crt, place a concatenation of server cert and intermediate CA cert in that order in node.127.0.0.1.crt
> 4. Start master and tserver with node to node encryption turned on and the appropriate gflags set.
>
> \### Observed
>
> Cluster does not form. The master reports an error similar to
>
> \`\`\`
> I0211 07:09:50.723590 55 refined\_stream.cc:231\] SECURE\[C\] kHandshake { local: 10.180.11.58:54676 remote: 10.180.8.67:7100 }: Handshake failed: Network error (yb/rpc/secure\_stream.cc:913): Handshake failed: Network error (yb/rpc/secure\_stream.cc:1128): Unverified certificate: unable to get local issuer certificate, address: 127.0.0.1, hostname: 127.0.0.1
> \`\`\`
>
> \### Expected
>
> Though this pattern is not common, it is used by cert-manager and we don't work with such certs https://github.com/cert-manager/cert-manager/pull/3433
>
> \`\`\`
> When a certificate is issued by an intermediate CA and the Issuer can provide the issued certificate's chain, the contents of tls.crt will be the requested certificate followed by the certificate chain.
> \`\`\`
>
> This pattern works with postgres when for client to node encryption in transit.
>
> \### Details
>
> ca.crt contains only the root CA as seen by
>
> \`\`\`
> openssl crl2pkcs7 -nocrl -certfile /home/sanketh/certs/intermediate\_ca/yb\_style2/ca.crt | openssl pkcs7 -print\_certs -text -noout
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number:
> 33:ef:c1:ec:a1:f8:e0:c8:ac:db:22:be:b0:29:03:fc:12:35:30:27
> Signature Algorithm: sha256WithRSAEncryption
> Issuer: CN=Root-ca
> Validity
> Not Before: Feb 11 17:31:57 2025 GMT
> Not After : Feb 9 17:31:57 2035 GMT
> Subject: CN=Root-ca
> Subject Public Key Info:
> Public Key Algorithm: rsaEncryption
> RSA Public-Key: (2048 bit)
> Modulus:
> 00:e3...
> Exponent: 65537 (0x10001)
> X509v3 extensions:
> X509v3 Basic Constraints:
> CA:TRUE
> Signature Algorithm: sha256WithRSAEncryption
> 74:68:1b:5...
> \`\`\`
>
> node.127.0.0.1.crt contains two certs
>
> \`\`\`
> openssl crl2pkcs7 -nocrl -certfile /home/sanketh/certs/intermediate\_ca/yb\_style2/node.127.0.0.1.crt | openssl pkcs7 -print\_certs -text -noout
> Certificate:
> Data:
> Version: 1 (0x0)
> Serial Number: 4097 (0x1001)
> Signature Algorithm: sha256WithRSAEncryption
> Issuer: CN=Interm.
> Validity
> Not Before: Feb 11 17:31:57 2025 GMT
> Not After : Feb 11 17:31:57 2026 GMT
> Subject: CN=127.0.0.1
> Subject Public Key Info:
> Public Key Algorithm: rsaEncryption
> RSA Public-Key: (2048 bit)
> Modulus:
> 00:d4:12:2f:f1:e9:a1:0f:de:8e:17:4b:83:3e:74:
> Exponent: 65537 (0x10001)
> Signature Algorithm: sha256WithRSAEncryption
> 6b:3a:bf:c2:a7:73:13:86:03:68:eb:f0:21:78:1b:02:f9:5b:
>
>
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number: 4096 (0x1000)
> Signature Algorithm: sha256WithRSAEncryption
> Issuer: CN=Root-ca
> Validity
> Not Before: Feb 11 17:31:57 2025 GMT
> Not After : Feb 11 17:31:57 2026 GMT
> Subject: CN=Interm.
> Subject Public Key Info:
> Public Key Algorithm: rsaEncryption
> RSA Public-Key: (2048 bit)
> Modulus:
> 00:bb:15:21:1d:42:ac:48:43:73:f1:a0:e0:af:87:
> Exponent: 65537 (0x10001)
> X509v3 extensions:
> X509v3 Basic Constraints:
> CA:TRUE
> Signature Algorithm: sha256WithRSAEncryption
> 9d:3f:d2:fa
> \`\`\`
>
>
>
> \### Issue Type
>
> kind/bug
>
> \### Warning: Please confirm that this issue does not contain any sensitive information
>
> \- \[x\] I confirm this issue does not contain any sensitive information.
>
> \[DB-15306\]: https://yugabyte.atlassian.net/browse/DB-15306?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
We have an issue in the support the intermediate certs in the node.\*local.crt file.
The workaround meanwhile would be to populate the rootCA manually.
---
### Author:  [@js.frerot](https://forum.yugabyte.com/u/js.frerot)
#### Post date: [February 12, 2025, 2:32pm UTC](https://forum.yugabyte.com/t/issue-with-tls-and-cert-manager/4182/7 "2025-02-12T14:32:26Z")
Hi @Aman_Nijhawan, not sure the workaround is clear as I have tried to put the full CA chain (2x int + RootCA) in the following helm setting:
tls.rootCA.cert
and on the pods, only the single RootCA appears.
This is when trying to use cert-manager and providing the rootCA as follow:
```auto
tls:
# Set to true to enable the TLS.
enabled: true
nodeToNode: true
clientToServer: true
insecure: false
certManager:
enabled: true
bootstrapSelfsigned: false
useClusterIssuer: true
clusterIssuer: vault-issuer-db
certificates:
duration: 1h # 90d
renewBefore: 10m # 15d
algorithm: RSA # ECDSA or RSA
keySize: 4096
rootCA:
cert: LS0tLS1CRUdJTiBDRVJ...
```
I don’t believe there is a workaround when using cert-manager.
You have to fully set the certificates manually or use the bootstrap feature which doesn’t use cert-manager.