Today is a frustrating day and I am scheduled for oncall. There is one related to network that the requests from one side intermittently timeout, for example:
1 | failed to create dial connection with read/write 10s timeout: dial tcp: i/o timeout |
It turns out the root cause is network packet loss. GCP has network performance dashboard to help you monitor and spot packet loss and latency statistics, very helpful.
I am also educated by this blog, it is worth to read through.