Otdfbt

We are running cassandra on kubernetes using the yaml below. It's working fine except for nodetool. We would like to have a kubernetes cronjob that executes nodetool repair periodically but accessing the jmx port (7199) from outside of the cassandra pods is not working. I'm fairly sure the problem is that the jmx port is listening on 127.0.0.1 and I need it to listen on the pod ip or even better would be the node ip.

Has anyone setup something similar?

Inside the pod

root@cassandra-0:/# nc -vz localhost 9042
localhost [127.0.0.1] 9042 (?) open
root@cassandra-0:/# nc -vz localhost 7199
localhost [127.0.0.1] 7199 (?) open

Outside the pod

$ nc -vz localhost 9042
nc: connect to localhost port 9042 (tcp) failed: Connection refused
Connection to localhost 9042 port [tcp/*] succeeded!
$ nc -vz localhost 7199
nc: connect to localhost port 7199 (tcp) failed: Connection refused
nc: connect to localhost port 7199 (tcp) failed: Connection refused

Outside the pod using the pod ip

$ nc -vz 10.244.1.20 7199
nc: connect to 10.244.1.20 port 7199 (tcp) failed: Connection refused
$ nc -vz 10.244.1.20 9042
Connection to 10.244.1.20 9042 port [tcp/*] succeeded!

apiVersion: v1
kind: Service
metadata:
 labels:
 app: cassandra
 name: cassandra
spec:
 clusterIP: None
 ports:
 - port: 9042
 selector:
 app: cassandra
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
 name: cassandra
 labels:
 app: cassandra
spec:
 serviceName: cassandra
 replicas: .Values.global.replicas 
 selector:
 matchLabels:
 app: cassandra
 template:
 metadata:
 labels:
 app: cassandra
 spec:
 terminationGracePeriodSeconds: 1800
 containers:
 - name: cassandra
 image: cassandra:3.11.3
 imagePullPolicy: Always
 ports:
 - containerPort: 7000
 name: intra-node
 - containerPort: 7001
 name: tls-intra-node
 - containerPort: 7199
 name: jmx
 hostPort: 7199
 - containerPort: 9042
 hostPort: 9042
 name: cql
 resources:
 limits:
 cpu: "500m"
 memory: 1Gi
 requests:
 cpu: "500m"
 memory: 1Gi
 securityContext:
 capabilities:
 add:
 - IPC_LOCK
 lifecycle:
 preStop:
 exec:
 command: 
 - /bin/sh
 - -c
 - nodetool drain
 env:
 - name: MAX_HEAP_SIZE
 value: 512M
 - name: HEAP_NEWSIZE
 value: 100M
 - name: CASSANDRA_SEEDS
 value: "cassandra-0.cassandra.default.svc.cluster.local"
 - name: CASSANDRA_CLUSTER_NAME
 value: "K8Demo"
 - name: CASSANDRA_DC
 value: "DC1-K8Demo"
 - name: CASSANDRA_RACK
 value: "Rack1-K8Demo"
 - name: POD_IP
 valueFrom:
 fieldRef:
 fieldPath: status.podIP
 volumeMounts:
 - name: cassandra-data
 mountPath: /var/lib/cassandra/data
 volumes:
 - name: cassandra-data
 hostPath:
 path: /data/cassandra