Executing Cassandra pod CQL queries from shell script, using shell variables with kubectl exec

All we need is an easy explanation of the problem, so here it is.

I have a repetitive task that I do while testing which entails connecting to a cassandra pod and running a couple of CQL queries.

Here’s the "manual" approach:

  1. On cluster controller node, I exec a shell on the pod using kubectl:
    kubectl exec pod/my-app-cassandra-pod-name -it --namespace myns -- /bin/bash

  2. Once in the pod I execute cqlsh:
    cqlsh $(hostname -i) -u myuser
    and then enter password interactively

  3. I execute my cql queries interactively

Now, I’d like to have a bash script to automate this. My intent is to run cqlsh directly, via kubectl exec.

The problem I have is that apparently I cannot use a shell variable within the "command" section of kubectl exec. And I will need shell variables to store a) the pod’s IP, b) an id which is the input to my first query, and c) intermediate query results (the two latter ones are not added to script yet).

Here’s what I have so far, using a dummy CQL query for now:

#!/bin/bash

CASS_IP=$(kubectl exec pod/my-app-cassandra-pod-name -it --namespace myns -- /usr/bin/hostname -i)
echo $CASS_IP   # This prints out the IP address just fine, say 192.168.79.208

# The below does not work, errors provided below
kubectl exec pod/my-app-cassandra-pod-name -it --namespace myns -- /opt/cassandra/bin/cqlsh $CASS_IP -u myuser -p 'mypass' -e 'SELECT now() FROM system.local;'

# The below works just fine and returns the CQL query output
kubectl exec pod/my-app-cassandra-pod-name -it --namespace myns -- /opt/cassandra/bin/cqlsh 192.168.79.208 -u myuser -p 'mypass' -e 'SELECT now() FROM system.local;'

The output from the above is as follows, where IP is echoed, first exec’d cqlsh breaks, and second succeeds:

192.168.79.208
Warning: Timezone defined and 'pytz' module for timezone conversion not installed. Timestamps will be displayed in UTC timezone.

Traceback (most recent call last):
  File "/opt/cassandra/bin/cqlsh.py", line 2357, in <module>
    main(*read_options(sys.argv[1:], os.environ))
  File "/opt/cassandra/bin/cqlsh.py", line 2326, in main
    encoding=options.encoding)
  File "/opt/cassandra/bin/cqlsh.py", line 463, in __init__
    load_balancing_policy=WhiteListRoundRobinPolicy([self.hostname]),
  File "/opt/cassandra/bin/../lib/cassandra-driver-internal-only-3.25.0.zip/cassandra-driver-3.25.0/cassandra/policies.py", line 425, in __init__
  File "/opt/cassandra/bin/../lib/cassandra-driver-internal-only-3.25.0.zip/cassandra-driver-3.25.0/cassandra/policies.py", line 426, in <listcomp>
  File "/usr/lib64/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
command terminated with exit code 1
Warning: Timezone defined and 'pytz' module for timezone conversion not installed. Timestamps will be displayed in UTC timezone.


 system.now()
--------------------------------------
 e78e75c0-0d3e-11ed-8825-1de1a1b1c128

(1 rows)

Any ideas how to get around this? I’ve been researching this for quite a while now, but I’m stuck…

How to solve :

I know you bored from this bug, So we are here to help you! Take a deep breath and look at the explanation of your problem. We have many solutions to this problem, But we recommend you to use the first method because it is tested & true method that will 100% work for you.

Method 1

This is really a kubectl exec issue. Your variables and are only available to the shell process in your current session.

Those same variables are not available to the pod you’re connected to when exec is invoked. The $CASS_IP variable doesn’t get expanded to the value 192.168.79.208 — it just stays as $CASS_IP when it gets "consumed" by exec leading to the "unknown host" error:

socket.gaierror: [Errno -2] Name or service not known

A workaround is to create a shell script in the pod which you would invoke with kubectl exec, similar to the way you would run cqlsh which itself is just a shell script. The difference is that your shell script will contain the cqlsh command:

/opt/cassandra/bin/cqlsh $(hostname -i) \
    -u myuser -p 'mypass' \
    -e 'SELECT now() FROM system.local';

You would then run your script with:

$ kubectl exec pod/my-pod-name -it --namespace myns -- /path/to/my_script.sh

Cheers!

Note: Use and implement method 1 because this method fully tested our system.
Thank you 🙂

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply