how to update the etcd member list

See also questions close to this topic

• Java Spring Webflux on Kubernetes: always [or-http-epoll-1], [or-http-epoll-2], [or-http-epoll-3], [or-http-epoll-4] despite configured resource

  Small question regarding a Java 11 Spring Webflux 2.6.6+ web app, containerized and deployed using Kubernetes please.

  From the web app application logs, I am seeing things such as:

  INFO [service,1bcce5941c742568,22c0ab2133c63a77] 11 --- [or-http-epoll-2] a.b.c.SomeClass  : Some message from the reactive pipeline.
  INFO [service,67cb40974712b3f4,15285d01bce9dfd5] 11 --- [or-http-epoll-4] a.b.c.SomeClass  : Some message from the reactive pipeline.
  INFO [service,5011dc5e09de30b7,f58687695bda20f2] 11 --- [or-http-epoll-3] a.b.c.SomeClass  : Some message from the reactive pipeline.
  INFO [service,8046bdde07b13261,5c30a56a4a603f4d] 11 --- [or-http-epoll-1] a.b.c.SomeClass  : Some message from the reactive pipeline.
  

  And always, I can only see [or-http-epoll-1] [or-http-epoll-2] [or-http-epoll-3] [or-http-epoll-4] which I think stands for: [reactor-http-epoll-N]

  The problem is, no matter how much CPU I allocate from Kubernetes, it is always those 4, no less, no more.

  I tried:

    resources:
              requests:
                cpu: 1
                memory: 1G
              limits:
                cpu: 2
                memory: 2G
  
    resources:
              requests:
                cpu: 4
                memory: 4G
              limits:
                cpu: 6
                memory: 6G
  
    resources:
              requests:
                cpu: 10
                memory: 10G
              limits:
                cpu: 10
                memory: 10G
  
  

  But again, always only those 4.

  I am having a hard time understanding what is the problem here, and why am I stuck with only/always 4 "or-http-epoll-".

  Thank you
• How does Kubernetes and Terraform work seamlessly together and what role do they each undertake?

  I am a bit confused about the individual roles of Kubernetes and Terraform when using them both on a project.

  Until very recently, I had a very clear understanding of both their purposes and everything made sense to me. But, then I heard in one of Nana's videos on Terraform, that Terraform was also very advanced in orchestration and I got confused.

  Here's my current understanding of both these tools:

  • Kubernetes: Orchestration software that controls many docker containers working together seamlessly. Kubernetes makes sure that new containers are deployed based on the desired infrastructure defined in configuration files (written with the help of a tool like Terraform, as IaC).

  • Terraform: Tool for provisioning, configuring, and managing infrastructure as IaC.

  So, when we say that Terraform is a good tool for orchestration, do we mean that it's a good tool for orchestrating infrastructure states or docker containers as well?

  I hope someone can clear that out for me!
• Update node condition type in status

  I am trying to patch, clear Node conditions in OpenShift and/or Kubernetes cluster on a worker node. Patch isn't working, trying even workarounds, maybe update the key in etcd.

  Main problem that i created new node conditions and then i removed them but they are not removed from list although they are no longer there or being updated by the controller.

  $ oc describe node node1.example.com
  
  Conditions:                 
    Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
    ----             ------  -----------------                 ------------------                ------                       -------
    ExampleToRemove        False   Wed, 13 Feb 2019 15:09:42 -0500   Wed, 13 Feb 2019 11:05:57 -0500   KubeletHasSufficientDisk     kubelet has sufficient disk space available
    MemoryPressure   False   Wed, 13 Feb 2019 15:09:42 -0500   Wed, 13 Feb 2019 11:05:57 -0500   KubeletHasSufficientMemory   kubelet has sufficient memory available
    DiskPressure     False   Wed, 13 Feb 2019 15:09:42 -0500   Wed, 13 Feb 2019 11:05:57 -0500   KubeletHasNoDiskPressure     kubelet has no disk pressure
    PIDPressure      False   Wed, 13 Feb 2019 15:09:42 -0500   Wed, 13 Feb 2019 11:05:57 -0500   KubeletHasSufficientPID      kubelet has sufficient PID available
    Ready            True    Wed, 13 Feb 2019 15:09:42 -0500
  
• How to recover the etcd snapshot into a machine with different CPU Arch?

  I'm having an issue at recovering the etcd snapshot (kubeadm db) into the new machine in a different CPU Arch.

  My original machine is on the s390x arch, and the new one is the typical amd64 one.

  I used the command below to get a snapshot on my s390x machine.

  $ sudo ETCDCTL_API=3 etcdctl snapshot save snapshot.db --cacert /etc/kubernetes/pki/etcd/server.crt --cert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/ca.key
  

  The snapshot gets generated successfully, and I was able to verify it with the command below.

  $ sudo ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshot.db
  
  +---------+----------+------------+------------+
  |  HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
  +---------+----------+------------+------------+
  | 75---0e | 15-----1 |       3--3 |      36 MB |
  +---------+----------+------------+------------+
  

  But If I move the snapshot file to the amd64 machine and run the verify command again, then I see this error only.

  $ ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshot.db
  
  Error: invalid database
  

  Of course, recovering it with etcdctl snapshot restore does not work.

  The version of the etcd-client of both machines is v3.5.2.

  Can someone give me some help?

  Thanks a ton.
• Connection between pods in k8s (etcd cluster)

  I am trying to set up a cluster of etcd pods in k8s.

  I am doing this.

  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    name: local-etcd-storage-class
    namespace: etcd
  provisioner: kubernetes.io/no-provisioner
  volumeBindingMode: WaitForFirstConsumer
  allowVolumeExpansion: true
  reclaimPolicy: Delete
  
  ---
  apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    name: etcd
    namespace: etcd
  spec:
    serviceName: etcd
    replicas: 2
    selector:
      matchLabels:
        app: etcd
    template:
      metadata:
        labels:
          app: etcd
      spec:
        containers:
        - name: etcd
          image: gcr.io/etcd-development/etcd:latest
          imagePullPolicy: IfNotPresent
          ports:
          - containerPort: 2379
            name: client
          - containerPort: 2380
            name: peer
          volumeMounts:
          - name: etcd-claim-template
            mountPath: /var/run/etcd
          command:
            - /bin/sh
            - -c
            - |
              echo ${HOSTNAME}
              PEERS="etcd-0=http://etcd-0.etcd:2380,etcd-1=http://etcd-1.etcd:2380,etcd-2=http://etcd-2.etcd:2380"
              exec etcd --name ${HOSTNAME} \
                --listen-peer-urls http://0.0.0.0:2380 \
                --listen-client-urls http://0.0.0.0:2379 \
                --advertise-client-urls http://${HOSTNAME}.etcd:2379 \
                --initial-advertise-peer-urls http://${HOSTNAME}:2380 \
                --initial-cluster-token etcd-cluster-1 \
                --initial-cluster ${PEERS} \
                --initial-cluster-state new \
                --data-dir /var/run/etcd/default.etcd
    volumeClaimTemplates:
    - metadata:
        name: etcd-claim-template
      spec:
        storageClassName: local-etcd-storage-class
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 1Gi
  
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: etcd-svc
    namespace: etcd
  spec:
    selector:
      app: etcd
    clusterIP: None
    ports:
    - name: etcd-client-port
      port: 2379
      protocol: TCP
    - name: etcd-peer-port
      port: 2380
      protocol: TCP
  

  The problem I am facing is that etcd-0.etcd does not see etcd-1.etcd

  2022-05-04 12:23:21.803307 W | rafthttp: health check for peer 5ba2a9fe8d840508 could not connect: dial tcp: lookup etcd-2 on 10.245.0.10:53: no such host (prober "ROUND_TRIPPER_SNAPSHOT")
  2022-05-04 12:23:21.803509 W | rafthttp: health check for peer eba8308136b9bf9e could not connect: dial tcp: lookup etcd-1 on 10.245.0.10:53: no such host (prober "ROUND_TRIPPER_SNAPSHOT")
  2022-05-04 12:23:21.803737 W | rafthttp: health check for peer eba8308136b9bf9e could not connect: dial tcp: lookup etcd-1 on 10.245.0.10:53: no such host (prober "ROUND_TRIPPER_RAFT_MESSAGE")
  2022-05-04 12:23:21.804521 W | rafthttp: health check for peer 5ba2a9fe8d840508 could not connect: dial tcp: lookup etcd-2 on 10.245.0.10:53: no such host (prober "ROUND_TRIPPER_RAFT_MESSAGE")
  

  Is there a way to make two pods communicate? Do I need separate services for each pod replica?

  I am stuck with this though I thought it will work as it is.

  I tried to ping etcd-1 from etcd-0 but is does not see the domain.

  etcd-0 terminal

  ping etcd-1
  ping: bad address 'etcd-1'
  
  ping etcd-1.etcd
  ping: bad address 'etcd-1.etcd'
  
  ping etcd-1.etcd.etcd.svc.cluster.local
  ping: bad address 'etcd-1.etcd.etcd.svc.cluster.local'
  
  ping etcd-1.etcd.svc.cluster.local
  ping: bad address 'etcd-1.etcd.svc.cluster.local'
  

  DO I need to do something to see other pods from the same stafullset