Google Cloud Engine - Network unreachable (with NAT)
May 7
Context & issue
We would like to do maintenance on one of our Compute Engine instance (update dependencies via apt update/upgrade), but said instance does not have access to the public internet, even if there is a NAT in the network:
sudo apt update
Err:1 http://security.ubuntu.com/ubuntu jammy-jellyfish-security InRelease
Could not connect to security.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.39), connection timed out Could not connect to security.ubuntu.com:80 (185.125.190.36), connection timed out
...
ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4096ms
We weren't able to figure out why the instance is unable to access the public internet; given the below information, can you help us figure out the cause?
Info
Instance details:
Zone: us-central1-c
Networking:
- Public DNS PTR Record: None
Network interface:
| name | VPC network | VPC subnetwork | Primary internal IP address | External IP address |
|---|---|---|---|---|
| nic0 | default (project default VPC) | default | 10.128.0.1 | None |
Firewalls:
- HTTP traffic: Off
- HTTPS traffic: Off
Cloud NAT Gateway:
| Name | Region | VPC network | Router |
|---|---|---|---|
| project-nat | northamerica-northeast1 | default | project-router |
Cloud Router:
| Name | Region | VPC network | VPN Gateway |
|---|---|---|---|
| project-nat | northamerica-northeast1 | default | None |
Firewall rules:
| name | network | direction | priority | src_ranges | dest_ranges | allow | deny | disabled |
|---|---|---|---|---|---|---|---|---|
| default-allow-icmp | default | INGRESS | 65534 | 0.0.0.0/0 | icmp | FALSE | ||
| default-allow-internal | default | INGRESS | 65534 | 10.128.0.0/9 | tcp:0-65535,udp:0-65535,icmp | FALSE | ||
| default-allow-rdp | default | INGRESS | 65534 | 0.0.0.0/0 | tcp:3389 | FALSE | ||
| default-allow-ssh | default | INGRESS | 65534 | 0.0.0.0/0 | tcp:22 | FALSE | ||
| server-allow-grpc-840d2afd | default | INGRESS | 100 | 10.128.0.0/9 | tcp:50051,tcp:50052 | FALSE | ||
| server-deny-all-840d2afd | default | INGRESS | 200 | 0.0.0.0/0 | icmp,udp,tcp | FALSE | ||
| server-ssh-iap-840d2afd | default | INGRESS | 100 | 35.235.240.0/20 | tcp:22 | FALSE |
Additional Firewall rules added to test if the firewall rules were the issue (without any success):
| name | network | direction | priority | src_ranges | dest_ranges | allow | deny | disabled |
|---|---|---|---|---|---|---|---|---|
| ubuntu-repositories | default | EGRESS | 10 | (IPs of: security.ubuntu.com, archive.canonical.com, us-central1.gce.archive.ubuntu.com, etc.) | tcp:80,tcp:443 | FALSE | ||
| server-vm-egress-all-https | default | EGRESS | 10 | 0.0.0.0/0 | tcp:80,tcp:443,tcp:8443 | FALSE |
Instance network configuration:
cat /etc/hosts
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
169.254.169.254 metadata.google.internal metadata
cat /etc/resolv.conf
nameserver 127.0.0.53
options edns0
search us-central1-c.c.project-name.internal c.project-name.internal google.internal
ifconfig
ens4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1460
inet 10.128.0.2 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::4001:aff:fe80:2 prefixlen 64 scopeid 0x20<link>
ether 42:01:0a:80:00:02 txqueuelen 1000 (Ethernet)
RX packets 9167 bytes 4126195 (4.1 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 10106 bytes 1160860 (1.1 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 640 bytes 61620 (61.6 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 640 bytes 61620 (61.6 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ip rule list
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
ip route show table all
default via 10.128.0.1 dev ens4 proto dhcp src 10.128.0.2 metric 100
10.128.0.1 dev ens4 proto dhcp scope link src 10.128.0.2 metric 100
local 10.128.0.2 dev ens4 table local proto kernel scope host src 10.128.0.2
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
::1 dev lo proto kernel metric 256 pref medium
fe80::/64 dev ens4 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
local fe80::4001:aff:fe80:2 dev ens4 table local proto kernel metric 0 pref medium
ff00::/8 dev ens4 table local metric 256 pref medium
dig google.com
; <<>> DiG 9.11.3-1ubuntu1.11-Ubuntu <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34116
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;google.com. IN A
;; ANSWER SECTION:
google.com. 300 IN A 142.251.161.101
google.com. 300 IN A 142.251.161.113
google.com. 300 IN A 142.251.161.102
google.com. 300 IN A 142.251.161.138
google.com. 300 IN A 142.251.161.100
google.com. 300 IN A 142.251.161.139
;; Query time: 19 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sun May 07 21:28:29 UTC 2023
;; MSG SIZE rcvd: 135
If you need any more information, let me know. Thanks a lot!
1 answer
Accepted answer · original discussion
May 8
As per the information provided, it is observed that the Compute Engine instance is in us-central1-c zone whereas Cloud NAT Gateway and Cloud Router are in northamerica-northeast1 region. It clearly shows that they are in different regions/zones. Also, please note that NAT gateway is regional and cannot connect to the VM instances which are in different regions.
As per the official documentation
Cloud NAT gateways are associated with subnet IP address ranges in a single region and a single VPC network. A Cloud NAT gateway created in one VPC network cannot provide NAT to VMs in other VPC networks connected by using VPC Network Peering, even if the VMs in peered networks are in the same region as the gateway.
If the issue persists even after fixing the NAT config, I recommend you to perform a connectivity test and validate whether the connectivity is broken.
Hope the above information is useful to you.
1 question comment
Use comments to ask for clarification. Post a solution as an answer.
May 7