Kubernetes deployment Round #2

General
1

image
image1334×776 34.8 KB

I am not sure why this happening. I used a NodePort instead of the LoadBalance port to get around setting up an ingress. Maybe I still have too?

When i ran:

kubectl exec -it yb-tserver-0 /home/yugabyte/bin/cqlsh
Connection error: (‘Unable to connect to any servers’, {‘127.0.0.1’: error(111, “Tried connecting to [(‘127.0.0.1’, 9042)]. Last error: Connection refused”)})
command terminated with exit code 1

this is on bare-metal

indent preformatted text by 4 spaces

kubectl describe po yb-tserver-0
Name: yb-tserver-0
Namespace: default
Node: fleet-94b9c3-k8s-worker-0/10.4.222.185
Start Time: Tue, 11 Sep 2018 23:35:05 -0400
Labels: app=yb-tserver
controller-revision-hash=yb-tserver-578cd855
statefulset.kubernetes.io/pod-name=yb-tserver-0
Annotations:
Status: Running
IP: 10.99.1.251
Controlled By: StatefulSet/yb-tserver
Containers:
yb-tserver:
Container ID: docker://ada7f909a93f0604cccd90d7fb69db3886eb257115a28308dcc1e2299dfdb711
Image: yugabytedb/yugabyte:latest
Image ID: docker-pullable://docker.io/yugabytedb/yugabyte@sha256:8804b7f2bcae983b30b4762b2ba2c6e47db017b097aa24207c8c279c71d32306
Ports: 9000/TCP, 9100/TCP, 9042/TCP, 6379/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
Command:
/home/yugabyte/bin/yb-tserver
–fs_data_dirs=/mnt/data0
–tserver_master_addrs=yb-masters.default.svc.cluster.local:7100
–tserver_master_replication_factor=3
State: Running
Started: Tue, 11 Sep 2018 23:35:47 -0400
Ready: True
Restart Count: 0
Environment:
Mounts:
/mnt/data0 from datadir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-twj2x (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
datadir:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: datadir-yb-tserver-0
ReadOnly: false
default-token-twj2x:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-twj2x
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations:
Events:
Type Reason Age From Message

Normal Scheduled 12m default-scheduler Successfully assigned yb-tserver-0 to fleet-94b9c3-k8s-worker-0
Normal SuccessfulAttachVolume 12m attachdetach-controller AttachVolume.Attach succeeded for volume “pvc-f2937ba4-b088-11e8-8d59-506b8d2df073”
Normal SuccessfulMountVolume 12m kubelet, fleet-94b9c3-k8s-worker-0 MountVolume.SetUp succeeded for volume “default-token-twj2x”
Normal SuccessfulMountVolume 12m kubelet, fleet-94b9c3-k8s-worker-0 MountVolume.SetUp succeeded for volume “pvc-f2937ba4-b088-11e8-8d59-506b8d2df073”
Normal Pulling 12m kubelet, fleet-94b9c3-k8s-worker-0 pulling image “yugabytedb/yugabyte:latest”
Normal Pulled 12m kubelet, fleet-94b9c3-k8s-worker-0 Successfully pulled image “yugabytedb/yugabyte:latest”
Normal Created 12m kubelet, fleet-94b9c3-k8s-worker-0 Created container
Normal Started 12m kubelet, fleet-94b9c3-k8s-worker-0 Started container
indent preformatted text by 4 spaces

2

Hmm not sure about why the nodeport fails… could you also post the output of the services so we can check the bindings:

kubectl get svc

I think the process does not bind on localhost. Could you try:

kubectl exec -it yb-tserver-0 /home/yugabyte/bin/cqlsh yb-tserver-0.yb-tservers

Also, seems like the master did not get elected, would be good to see the master logs. You can find them on the yb-master pods at the following location:

# Attach to the master pod
$ kubectl exec -it yb-master-0 bash

# Logs are present here
$ vi /mnt/disk0/yb-data/master/logs/yb-master.INFO

Could you please upload the logs? We can also do a hangout if thats easier.

3

Maybe the below is a sign of firewall issues on the worker nodes, i will go check that out.
indent preformatted text by 4 spaces
kubectl exec -it yb-tserver-0 /home/yugabyte/bin/cqlsh yb-tserver-0.yb-tservers
Connection error: (‘Unable to connect to any servers’, {‘10.99.1.251’: error(111, “Tried connecting to [(‘10.99.1.251’, 9042)]. Last error: Connection refused”)})
command terminated with exit code 1
indent preformatted text by 4 spaces

/mnt/disk0/ this directory appears to be empty. - i will maybe manually delete the PV and see if that helps

Preformatted textkubectl describe svc yb-master-ui
Name: yb-master-ui
Namespace: default
Labels: app=yb-master
Annotations:
Selector: app=yb-master
Type: NodePort
IP: 10.88.0.105
Port: ui 7000/TCP
TargetPort: 7000/TCP
NodePort: ui 31535/TCP
Endpoints: 10.99.1.252:7000,10.99.2.254:7000,10.99.3.24:7000
Session Affinity: None
External Traffic Policy: ClusterPreformatted text

4

If you have time today,a hangout would be good to but please don’t me get in the way more important things

5

Wow sleep is good! The PV didn’t respect the claim i guess. Delete them, reran and I am good!

Just need a easy way to thru some load at it and then video time.

Thanks

1 Like
7

https://youtu.be/oOP3c7_ILgs - thanks for the help again

1 Like
8

Loving the demo @dlink7!! Thats awesome! Thanks for sharing.