How to use ss in specific network namespace

Whilst investigating some TCP network issues in containers I tried to use ss to peek into container network TCP stack.

We are running Amazon Linux in AWS:

# uname -a
Linux 4.14.173-137.229.amzn2.x86_64 #1 SMP Wed Apr 1 18:06:08 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

ss has the following cli switch for that:

       -N NSNAME, --net=NSNAME
              Switch to the specified network namespace name.

lsns gives me the following output:

# lsns | grep net
4026531993 net       225     1 root      /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026532284 net         2 26244 root      /pause

This is pause container created for each Kubernetes pod -- it's the container that creates the network namespace.

Trying to peek into the pod network namespace by running ss:

# ss -tp -N 4026532284
Cannot open network namespace "4026532284": No such file or directory

What's interesting is ip netns list does not return any network namespaces:

# ip netns list
#

Is there any way how one can look into K8s pod network namespaces from the root network namespace i.e. from the netns 1?

# ss --version
ss utility, iproute2-ss180129

# lsns --version
lsns from util-linux 2.30.2

# rpm -qi iproute
Name        : iproute
Version     : 4.15.0
Release     : 1.amzn2.0.4
Architecture: x86_64
Install Date: Sat 07 Mar 2020 03:42:24 AM UTC
Group       : Applications/System
Size        : 1321292
License     : GPLv2+ and Public Domain
Signature   : RSA/SHA256, Fri 21 Feb 2020 09:00:29 PM UTC, Key ID 11cf1f95c87f5b1a
Source RPM  : iproute-4.15.0-1.amzn2.0.4.src.rpm
Build Date  : Fri 21 Feb 2020 07:56:50 PM UTC
Build Host  : build.amazon.com
Relocations : (not relocatable)
Packager    : Amazon Linux
Vendor      : Amazon Linux
URL         : http://kernel.org/pub/linux/utils/net/iproute2/
Summary     : Advanced IP routing and network device configuration tools

Update: Tue Dec 1 11:35:39 UTC 2020

After some struggle, I finally decided to strace this.

It turns out ss is an awesome tool, but when it comes to using it with containers it leaves a bit to be desired, but I feel there is more than one "culprit" involved.

ss does not bother to look up the actual PID of the process that creates the network namespaces, but rather goes directly to check /var/run/netns:

openat(AT_FDCWD, "/var/run/netns/4026532284", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
write(2, "Cannot open network namespace \"4"..., 70Cannot open network namespace "4026532284": No such file or directory
) = 70

Now, I suspect this is due to how iproute package creates network namespaces i.e. given the ss is shipped with iproute package the assumption ip makes about network namespaces is: "hey all network ns should be found in /var/run/netns directory, because, like, why not and also this will make lives of iproute devs easy, or whatever.

Turns out that's a false assumption made on ss/iproute side or the lack of "agreement" on modern containers tools and iproute interoperability, but it kind of explains the empty output from

ip netns list

So the way ip creates network namespaces (so they can be inspected by ss) obviously does not match how kubernetes and the likes create them, making iproute package utilities borderline useless in the grand scheme of things.

nsenter -t ${PID_FOO} -muni ss -tpi

docker run -it --rm --security-opt=seccomp:unconfined \
                    --security-opt=apparmor:unconfined \
                    --privileged --pid=host --userns=host \
debian:jessie@sha256:51cd80bb935b76fbbf49640750736abc63ab7084d5331e198326b20063e7f13c \
nsenter -t ${PID_FOO} -m -u -n -i -F ss -tpi