Cannot connect ysql

Have a 4 node deployment, have configured master on 3 nodes and tablet servers on all 4, YCQL is working but cannot use ysqlsh am getting the following error:
ysqlsh: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

tserver log files have nothing about the error, postgres is running, as follows:

/opt/yugabyte- post gres/bin/ post gres -D /mnt/yugga_vol01/pg_data -p 5433 -h -k -c logging_collector=on -c log_directory=/mnt/yugga_vol01/yb-data/tserver/logs -c shared_preload_libraries=yb_pg_metrics -c yb_pg_metrics.node_name=DEFAULT_NODE_NAME -c yb_pg_metrics.port=13000

How do I debug this further?

Hi @salil,
Did you have an older version of YB and then upgrade/install a newer version? We’ve had a backward incompatible change in our recent version for YSQL and the error you’re seeing above is a typical symptom when you upgrade.

If this is a fresh install (and not an upgrade), then you can try this to debug:

  1. Update postgresql.conf file located in $data_dir/disk-1/pg_data/postgresql.conf and add the line log_statement='all' . Please do this on the node you’re connecting to with ysqlsh .
  2. Restart tserver on that node
  3. Run ./bin/ysqlsh and check any errors in $data_dir/disk-1/yb-data/tserver/logs/postgresql*.log

Thanks @neha. I was using the flagfile option for yb-tserver, but was not giving the full path of the file. So postgres was complaining about tsserver.conf not found. I restarted tserver with full path and now I am getting a different error:

ysqlsh: could not connect to server: Connection refused
Is the server running on host “localhost” (::1) and accepting
TCP/IP connections on port 5433?
FATAL: Not found: Error loading table with oid 1260 in database with oid 1: The object does not exist: table_id: “000000010000300080000000000004ec”

Hi @salil, I think this is because you haven’t yet initialized YSQL on the cluster. Please follow the instructions here to do that:

You should be able to connect after that.

Hi @neha,
Thanks, this did help me get it working (at least can get to the ysql shell now). However the instructions as documented did not help. .

Currently the steps given are:
-start tserver
-init ysql.

However I noticed the following:

  • when I start tserver, it starts postgres with a data dir in $data_dir/disk-1/pg_data
  • The instructions ask you to init the ysql using the following data dir:

Doing this does not work as tserver initiated postgress process is not looking at this data dir!

I tried to run the initdb again giving $data_dir/disk-1/pg_data as data dir, however this command failed saying the dir already exists.

So here are the steps I performed to get ysql working:

  • killed tserver
  • removed $data_dir/disk-1/pg_data dir
  • ran initdb script with -D $data_dir/disk-1/pg_data
  • While the script was running, started tserver in another window

Glad to hear that it’s working for you now, @salil. And thank you for your feedback on the documentation. We’ll be sure to review and update it.