Error while installing 2.0.11.0

#1

I tried to upgrade 2.0.8.0 to 2.0.11.0 but yb-ctl create fails like this:

jk@tiira:~/bin2/yugabyte-2.0.11.0/bin$ ./yb-ctl create --rf 1
Creating cluster.
Waiting for cluster to be ready.
Viewing file /home/jk/yugabyte-data/node-1/disk-1/tserver.err:
sh: 1: /home/jk/bin2/yugabyte-2.0.11.0/bin/yb-tserver: not found
Viewing file /home/jk/yugabyte-data/node-1/disk-1/master.err:
sh: 1: /home/jk/bin2/yugabyte-2.0.11.0/bin/yb-master: not found
Traceback (most recent call last):
  File "./yb-ctl", line 1969, in <module>
    control.run()
  File "./yb-ctl", line 1946, in run
    self.args.func()
  File "./yb-ctl", line 1706, in create_cmd_impl
    self.wait_for_cluster_or_raise()
  File "./yb-ctl", line 1551, in wait_for_cluster_or_raise
    raise RuntimeError("Timed out waiting for a YugaByte DB cluster!")
RuntimeError: Timed out waiting for a YugaByte DB cluster!
Viewing file /tmp/tmpSY_rXD:
2020-01-31 17:22:02,367 INFO: Starting master-1 with:
/home/jk/bin2/yugabyte-2.0.11.0/bin/yb-master --fs_data_dirs "/home/jk/yugabyte-data/node-1/disk-1" --webserver_interface 127.0.0.1 --rpc_bind_addresses 127.0.0.1 --v 0 --version_file_json_path=/home/jk/bin2/yugabyte-2.0.11.0 --webserver_doc_root "/home/jk/bin2/yugabyte-2.0.11.0/www" --replication_factor=1 --yb_num_shards_per_tserver 2 --ysql_num_shards_per_tserver=2 --master_addresses 127.0.0.1:7100 --enable_ysql=true >"/home/jk/yugabyte-data/node-1/disk-1/master.out" 2>"/home/jk/yugabyte-data/node-1/disk-1/master.err" &
2020-01-31 17:22:02,407 INFO: Starting tserver-1 with:
/home/jk/bin2/yugabyte-2.0.11.0/bin/yb-tserver --fs_data_dirs "/home/jk/yugabyte-data/node-1/disk-1" --webserver_interface 127.0.0.1 --rpc_bind_addresses 127.0.0.1 --v 0 --version_file_json_path=/home/jk/bin2/yugabyte-2.0.11.0 --webserver_doc_root "/home/jk/bin2/yugabyte-2.0.11.0/www" --tserver_master_addrs=127.0.0.1:7100 --yb_num_shards_per_tserver=2 --redis_proxy_bind_address=127.0.0.1:6379 --cql_proxy_bind_address=127.0.0.1:9042 --local_ip_for_outbound_sockets=127.0.0.1 --use_cassandra_authentication=false --ysql_num_shards_per_tserver=2 --enable_ysql=true --pgsql_proxy_bind_address=127.0.0.1:5433 >"/home/jk/yugabyte-data/node-1/disk-1/tserver.out" 2>"/home/jk/yugabyte-data/node-1/disk-1/tserver.err" &
2020-01-31 17:22:02,411 INFO: Waiting for master and tserver processes to come up.
2020-01-31 17:22:13,078 INFO: PIDs found: {'tserver': [None], 'master': [None]}
2020-01-31 17:22:13,079 ERROR: Failed waiting for master and tserver processes to come up.
^^^ Encountered errors ^^^

Why is this? Error messages say that yb-tserver and yb-master not found but they are available as this listing shows:

jk@tiira:~/bin2/yugabyte-2.0.11.0/bin$ ll
total 40908
drwxr-xr-x  2 jk jk     4096 tammi 24 19:57 ./
drwxr-xr-x 10 jk jk     4096 tammi 24 20:06 ../
-rwxr-xr-x  1 jk jk     2327 tammi 24 19:57 bulk_load_cleanup.sh*
-rwxr-xr-x  1 jk jk     4056 tammi 24 19:57 bulk_load_helper.sh*
-rwxr-xr-x  1 jk jk     1516 tammi 24 19:57 configure*
-rwxr-xr-x  1 jk jk     1353 tammi 24 19:57 cqlsh*
-rwxr-xr-x  1 jk jk    97735 tammi 24 19:57 cqlsh.py*
-rwxr-xr-x  1 jk jk   726312 tammi 24 19:57 ldb*
-rwxr-xr-x  1 jk jk     4486 tammi 24 19:57 log_cleanup.sh*
-rwxr-xr-x  1 jk jk  2051392 tammi 24 19:57 log-dump*
-rwxr-xr-x  1 jk jk  1141216 tammi 24 19:57 patchelf*
-rwxrwxr-x  1 jk jk     4043 tammi 24 19:57 post_install.sh*
-rwxr-xr-x  1 jk jk   178257 tammi 24 19:57 pprof*
lrwxrwxrwx  1 jk jk       22 tammi 24 20:05 psql -> ../postgres/bin/ysqlsh*
-rwxr-xr-x  1 jk jk  2464368 tammi 24 19:57 redis-cli*
-rwxr-xr-x  1 jk jk    16952 tammi 24 19:57 sst_dump*
-rwxr-xr-x  1 jk jk 10295264 tammi 24 19:57 yb-admin*
-rwxr-xr-x  1 jk jk  4936952 tammi 24 19:57 yb-bulk_load*
-rw-rw-r--  1 jk jk     1065 tammi 24 19:57 yb-check-consistency.py
-rwxr-xr-x  1 jk jk     4218 tammi 24 19:57 yb-check-failed-tablets.sh*
-rwxr-xr-x  1 jk jk    85443 tammi 24 19:57 yb-ctl*
-rwxr-xr-x  1 jk jk  1984480 tammi 24 19:57 yb-generate_partitions_main*
-rwxr-xr-x  1 jk jk  3937568 tammi 24 19:57 yb_load_test_tool*
-rwxr-xr-x  1 jk jk  3213920 tammi 24 19:57 yb-master*
-rwxr-xr-x  1 jk jk   187904 tammi 24 19:57 yb-pbc-dump*
-rwxr-xr-x  1 jk jk     9095 tammi 24 19:57 yb-prof.py*
-rwxr-xr-x  1 jk jk  3288096 tammi 24 19:57 yb-ts-cli*
-rwxr-xr-x  1 jk jk  4170560 tammi 24 19:57 yb-tserver*
-rwxr-xr-x  1 jk jk  2918080 tammi 24 19:57 yb-ysck*
lrwxrwxrwx  1 jk jk       22 tammi 24 20:05 ysqlsh -> ../postgres/bin/ysqlsh*
-rwxr-xr-x  1 jk jk    79816 tammi 24 19:57 yugabyted*
jk@tiira:~/bin2/yugabyte-2.0.11.0/bin$

I have no problems creating v 2.0.8.0.

#2

@jk-1
Can you try and see if it works ?:
0. pkill yb-tserver && pkill yb-master (or ./bin/yb-ctl stop)

  1. download the new binary into a new directory
  2. run ./bin/post_install.sh in the new version
  3. new-binary-directory ./bin/yb-ctl start
#3

@jk-1–

It’s possible that there are processes left over from previous install. Let’s check:

/home/jk/yugabyte-data/node-1/disk-1/master.out & /home/jk/yugabyte-data/node-1/disk-1/master.err

files, and similarly, these files:

/home/jk/yugabyte-data/node-1/disk-1/tserver.out /home/jk/yugabyte-data/node-1/disk-1/tserver.err

to see why those processes didn’t start.

#4

I did
yb-ctl destroy
post_install.sh
yb-ctl create --rf 1
yb-ctl start

Ports 7000 & 9000 are up

I can log into ysqlsh and status is

jk@tiira:~/bin2/yugabyte-2.0.11.0/bin$ yb-ctl status
----------------------------------------------------------------------------------------------------
| Node Count: 1 | Replication Factor: 1                                                            |
----------------------------------------------------------------------------------------------------
| JDBC                : jdbc:postgresql://127.0.0.1:5433/postgres                                  |
| YSQL Shell          : ysqlsh                                                                     |
| YCQL Shell          : cqlsh                                                                      |
| YEDIS Shell         : redis-cli                                                                  |
| Web UI              : http://127.0.0.1:7000/                                                     |
| Cluster Data        : /home/jk/yugabyte-data                                                     |
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
| Node 1: yb-tserver (pid 8390), yb-master (pid 8387)                                              |
----------------------------------------------------------------------------------------------------
| JDBC                : jdbc:postgresql://127.0.0.1:5433/postgres                                  |
| YSQL Shell          : ysqlsh                                                                     |
| YCQL Shell          : cqlsh                                                                      |
| YEDIS Shell         : redis-cli                                                                  |
| data-dir[0]         : /home/jk/yugabyte-data/node-1/disk-1/yb-data                               |
| yb-tserver Logs     : /home/jk/yugabyte-data/node-1/disk-1/yb-data/tserver/logs                  |
| yb-master Logs      : /home/jk/yugabyte-data/node-1/disk-1/yb-data/master/logs                   |
----------------------------------------------------------------------------------------------------

So it seems to work. Thank you for your support!

I tried without post_install first because this guide does not mention it: https://docs.yugabyte.com/latest/manage/upgrade-deployment/

So do I need to run post_install always after upgrade?

#5

Yes. We’re fixing the missing docs: https://github.com/yugabyte/yugabyte-db/pull/3508

#6

post_install.sh is a one time thing that you need to do after downloading a new release bundle. [You don’t need to do it for every stop/start or destroy/start cycle.]