Wordpress Slowness Behind GCP HTTP Load Balancer

I am hosting a Wordpress VM on GCP (using the Marketplace image). It works great when I directly access the instance IP, and the latency is around 20 ms.

However, in order to make the site secure, I am using a GCP HTTP load balancer in front of the VM. I have pointed my domain to the load balancer.

When I access the Wordpress through the load balancer (either HTTP or HTTPS), I get a few queries that have very good latency times (around 17 ms), but then every 4 or 5 queries, there's a request that takes about 5 seconds. When I access the VM directly, this does not happen.

I enabled the load balancer log, and I observed that the VM is responding slowly every few requests. I tried to set the KeepAlive Apache2 parameter of the VM to 300, but it doesn't make any difference.

What might be the root cause of this? (I am not sure whether it's Wordpress, or the VM, or the load balancer).

Thanks