Default (insecure) cluster to --secure cluster warning!

General
1

This is a general warning for folks that encounter the following error after switching to –secure cluster.

If you run a cluster by default (insecure), and then upgrade your cluster:

  • Create Certs
  • Upload to all your nodes
  • Start your cluster with yugubyted start --secure --base_dir= …

You may encounter the issue where the :15433 general web interface is broken With errors like:

  • Something went wrong. Please try again.

And your Browser (F12 > Console ) may show errors like “index.ad425760657ed59418bb.js:2 gocql: unable to create session: unable to discover protocol version: Provided username ‘cassandra’ and/or password are incorrect”

While your cluster is running (the nodes page works), you can now use SSL remote ysql, the broken web API interface is a issue.

The only way to solve this issue (as far i discovered) is to destroy the cluster and restart.

  • yugubyted stop --base_dir= …
  • yugubyted destroy --base_dir=…
  • yugubyted start --secure --base_dir=…

After you do this, the cluster will work with In-Transit Encryption, and show all the stats correctly.

For the developers:

The issue seem to be that no password is created:

nano /home/xxxx/yuga/node/data/yugabyted_credentials.txt

When checking this file, it was never created when switchen from insecure, to secure. No file existed. And the DB connection still relies on the default (insecure cluster) yugabyte:yugabyte (user/pass), while the cluster itself is using SSL for its inner communication.

It was only when creating a cluster directly in secure mode, that this file is created, and the API errors got solved (as there was now a password set). And the DB connection uses the yugabyted_credentials.txt passwords.

There needs to be a major warning that if a cluster was run in insecure mode, that it needs to be recreated for secure mode (or whatever bugs resulting from going insecure > secure fixed). As most people will first try out the default insecure (easy mode), only to later run into this issue when they start to secure the cluster. Multiple similar posts here are a result of this.

2

Hi Benjiro,

Thanks for reporting the issue and providing a workaround. We’ll take a look the behavior of moving from --insecure to --secure node, ideally it should just have worked. As you pointed out looks like there might be some bug, I’ll file a Github ticket and we’ll investigate.

Thanks,
Nikhil Chandrappa
Lead Engineer, YugabyteDB OSS

1 Like