Hi @vineet2k1 , as @nmalladi mentioned, yugabyted doesn’t have native support for enabling mTLS. Let’s break down the issues you have faced and then the solution for it.
Issues:
- When you were trying to start a node with
--secure flag and mTLS enabled, it failed. This is because enabling mTLS means client_certs are required to connect to ysqlsh and when yugabyted tries to update the password (since --secure flag was used), it fails.
- A workaround here was to not use
secure flag and just enable mTLS by explicitly specifying the required gFlags. This would work for single node. But when trying to start a 2nd node , it would fail as yugabyted doesn’t allow a node without --secure flag to join to a node with gFlags, for enabling TLS, set.
Solution:
Requirements:
Following set of certificates:
- Server Certs:
ca.crt, node.<node_hostname>.crt, node.<node_hostname>.key
- Client Certs:
ca.crt, <client_name>.crt, <client_name>.key
Please ensure that the ca.crt is same for both set of certs and same set of root-ca certificates were used to create both server and client certs.
Now to tackle the 2nd issue, we’ll first start all the nodes in a normal TLS fashion with the server certs.
Your yugabyte_start_mtls.conf would look like:
{
"data_dir": "/crdb/data/yugabyte",
"log_dir": "/crdb/log/yugabyte",
"ysql_port": 9090,
"ycql_port": 9097,
"master_rpc_port": 9092,
"tserver_rpc_port": 9093,
"master_webserver_port": 9094,
"tserver_webserver_port": 9095,
"ysql_metric_port": 9096,
"yugabyted_ui_port": 9091,
"backup_daemon": true,
"certs_dir": "/home/crdbadmin/ybdb1d/certs"
}
Start the 1st node using:
$HOME/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted start \
--secure \
--config $HOME/yugabyte_start_mtls.conf \
--advertise_address=$(hostname) \
--cloud_location=$(get_aws_region_az) \
--base_dir=$HOME/yb_open_binaries \
--fault_tolerance=region
This will start a node with only TLS enabled. The --secure flag will take care of setting the required gFlags, so no need to explicitly set them.
Add more nodes to it:
$HOME/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted start \
--secure \
--config $HOME/yugabyte_start_mtls.conf \
--advertise_address=$(hostname) \
--join aws-us-west-2a-ybdb1d-crdb-01.g.apple.com \
--cloud_location=$(get_aws_region_az) \
--base_dir=$HOME/yb_open_binaries \
--fault_tolerance=region
Once all the nodes are started, we’ll now restart the nodes (in sequential manner) while enabling mTLS.
Stop the node:
$HOME/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted stop \
--base_dir=$HOME/yb_open_binaries
Restart the node with mTLS enabled:
$HOME/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted start \
--base_dir=$HOME/yb_open_binaries \
--tserver_flags="ysql_hba_conf_csv={hostssl all all all trust clientcert=verify-full}"
Once all the nodes are restarted, you can connect to ysqlsh using the command:
./ysqlsh -h $(hostname) -p 9090 -U yugabyte \
"sslcert=/home/crdbadmin/ybdb1d/certs/<client_name>.crt sslkey=/home/crdbadmin/ybdb1d/certs/<client_name>.key sslrootcert=/home/crdbadmin/ybdb1d/certs/ca.crt sslmode=verify-full"
A little about sslmode and clientcert:
- When specifying
clientcert(specified in hba_conf) as verify-ca, the server will verify the client certs but only checks whether the client certs has been signed by the same root-ca or not whereas verify-full will ensure server checks the clients certs for correct signature as well as the username is stored in the client certificates.
If using verify-full please ensure that the client-certs have CN set as the username (yugabyte in the above example).
- Similarly using
verify-ca as sslmode(specified while running ysqlsh) will ensure client only checks for correct signature of the server certs, whereas verify-full mode will ensure that the hostname is the same as the one stored in server certs.
Enabling vs Disabling Password authentication:
Password authentication can toggled by either using trust(disable) or md5(enable) in the hba_conf while restarting the nodes.
Please note that the password was updated by yugabyted during the initial start of the 1st node. A credentials file would have been created and stored in node 1.
If you decide to enable password authentication with mTLS, please use the generated password.
Please let us know if you face any issues with this deployment.
My ask is to set the cluster mTLS enabled so exactly what shall I set for below 2 setups: