kube-proxy seems to be unable to resolve service IP on windows

Running a hybrid cluster on 3 VMs, master node a linux, one worker is linux the other one is windows. the linux node is working correctly (at least, it can run Jenkins job and a grafana). The Windows node is unable to reach the dns from the service IP even though the dns service is up and the two pods yields no error. The kube proxy pod is also running correctly on the windows node yielding only infos in the logs:

"I1011 10:02:39.770583    7524 proxier.go:919] syncProxyRules took 2.2098ms\n","stream":"stderr","time":"2021-10-11T08:02:39.7715829Z"}
{"log":"I1011 10:02:39.770583    7524 bounded_frequency_runner.go:296] sync-runner: ran, next possible in 1s, periodic in 30s\n","stream":"stderr","time":"2021-10-11T08:02:39.7715829Z"}
{"log":"I1011 10:03:09.782792    7524 proxier.go:967] Syncing Policies\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"I1011 10:03:09.782792    7524 proxier.go:978] Policy already applied for (*winkernel.serviceInfo)(0xc00080aab0)({\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":" BaseServiceInfo: (*proxy.BaseServiceInfo)(0xc0005b35c0)({\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  clusterIP: (net.IP) (len=16 cap=16) {\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"   00000000  00 00 00 00 00 00 00 00  00 00 ff ff ac 1f ee 5b  |...............[|\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  },\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  port: (int) 3000,\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  protocol: (v1.Protocol) (len=3) \"TCP\",\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  nodePort: (int) 32083,\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  loadBalancerStatus: (v1.LoadBalancerStatus) {\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"   Ingress: ([]v1.LoadBalancerIngress) (len=1 cap=1) {\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"    (v1.LoadBalancerIngress) {\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"     IP: (string) (len=10) \"10.40.42.0\",\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"     Hostname: (string) \"\",\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"     Ports: ([]v1.PortStatus) \u003cnil\u003e\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"    }\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"   }\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  },\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  sessionAffinityType: (v1.ServiceAffinity) (len=4) \"None\",\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  stickyMaxAgeSeconds: (int) 0,\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  externalIPs: ([]string) \u003cnil\u003e,\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  loadBalancerSourceRanges: ([]string) \u003cnil\u003e,\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  healthCheckNodePort: (int) 0,\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  onlyNodeLocalEndpoints: (bool) false,\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  topologyKeys: ([]string) \u003cnil\u003e\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":" }),\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":" targetPort: (int) 3000,\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":" externalIPs: ([]*winkernel.externalIPInfo) \u003cnil\u003e,\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":" loadBalancerIngressIPs: ([]*winkernel.loadBalancerIngressInfo) (len=1 cap=1) {\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  (*winkernel.loadBalancerIngressInfo)(0xc0007daf40)({\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"   ip: (string) (len=10) \"10.40.42.0\",\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"   hnsID: (string) (len=36) \"4de2a935-88f4-4ff3-8ee6-f29b136d93b4\"\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  })\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":" },\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":" hnsID: (string) (len=36) \"42ee3d2f-523a-4501-8e44-a879f8af9192\",\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":" nodePorthnsID: (string) (len=36) \"2f2c1557-1561-40ee-accb-6e2cc9b659dc\",\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":" policyApplied: (bool) true,\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":" remoteEndpoint: (*winkernel.endpointsInfo)(0xc0003261e0)({\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  ip: (string) (len=13) \"172.31.238.91\",\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  port: (uint16) 0,\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  isLocal: (bool) false,\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  macAddress: (string) (len=17) \"00:50:56:88:37:89\",\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  hnsID: (string) (len=36) \"ef3d0218-58b7-4d1e-b30a-aa870cf1cd84\",\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  refCount: (*uint16)(0xc00093ccd0)(1),\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  providerAddress: (string) (len=11) \"10.40.32.17\",\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  hns: (winkernel.hnsV2) {\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"  }\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":" }),\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":" hns: (winkernel.hnsV2) {\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":" },\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":" preserveDIP: (bool) false,\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":" localTrafficDSR: (bool) false\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}
{"log":"})\n","stream":"stderr","time":"2021-10-11T08:03:09.7837903Z"}

when trying to nslookup anything from a pod running on the cluster this is the result:

DNS request timed out.
    timeout was 2 seconds.
Server:  UnKnown
Address:  172.31.128.10

DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
*** Request to UnKnown timed-out

Trying to query the dns directly with one of the IP from the pod running the DNS yields correct result.

The linux nodes are standard, I went through the doc https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/, cni is flannel, and there's metalLB. there are 3 subnet one /17 for services one /17 for cluster ips and one /24 for load balanced external IP. All ranges are RFC1918 ranges (the cluster is meant for internal use only). DNS is coredns with default values Windows node is setup using this tutorial https://v1-17.docs.kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-nodes/ changing the version from 1.17 to 1.21.3

windows kube proxy is spawned via

curl -L https://github.com/kubernetes-sigs/sig-windows-tools/releases/latest/download/kube-proxy.yml | sed 's/VERSION/v1.22.0/g' | kubectl apply -f -

kubectl version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T21:04:39Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:04:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

Os version for linux:

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
$ uname -a
Linux ci-d10-kube1 4.19.0-17-amd64 #1 SMP Debian 4.19.194-1 (2021-06-10) x86_64 GNU/Linux

for windows:

C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
Caption
Microsoft Windows Server 2019 Datacenter
Version
10.0.17763
BuildNumber
17763
OSArchitecture
64-bitcat 

CRI: Docker version 20.10.7, build f0df350 on linux Docker version 20.10.7, build 40ef3b6 on windows

CNI: Flannel using quay.io/coreos/flannel:v0.14.0

How many English words
do you know?
Test your English vocabulary size, and measure
how many words do you know
Online Test
Powered by Examplum