Routing only the private network in wireguard VPN

I have a few hosts behind a NAT router that I want to access via a wireguard VPN. I could successfully configure the private network, but there's still something that baffles me.

I want each peer to:

• access each other (172.9.9.*) via the VPN (via wg0)
• access every other host outside the VPN (via eth0).

Here's a schema of the network and current configuration:

┌─────┐    ┌──────────┐    ┌─────┐
│  S  ├────┤ Internet ├────┤  A  │
└─────┘    └───┬──────┘    └─────┘
               │
               │
          ┌────┴─────┐
          │ NAT DHCP │
       ┌──┤  Router  ├──┐
       │  └──────────┘  │
       │                │
    ┌──┴──┐          ┌──┴──┐
    │  X  │          │  B  │
    └─────┘          └─────┘

• S is the VPN server and it's accessible on the internet via static IP;
• X is a "compute server", it can access the internet, but is behind a NAT and it's IP is dynamic and not known in advance;
• A is a "remote client" that wants to connect to X;
• B is a "local client" that wants to connect to X and it's in the same local network.

I want that A and B can connect to X through S, but all of these hosts should use the VPN only when contacting each other and not when accessing the internet.

So, for instance, A can ping google.com directly, but will ping X via S.

After searching and reading documentations, it's still unclear to me if it's possible to do this without using iptables and if it's possible to do so using only the wireguard configuration.

The current configuration is the following:

## S wg0.conf

[Interface]
PrivateKey = S-private-key
Address = 172.9.9.1/24
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
ListenPort = 51820

[Peer]
# A
PublicKey = A-public-key
AllowedIPs = 172.9.9.100/32

[Peer]
# B
PublicKey = B-public-key
AllowedIPs = 172.9.9.101/32

[Peer]
# X
PublicKey = X-public-key
AllowedIPs = 172.9.9.10/32

# A wg0.conf

[Interface]
Address = 172.9.9.100/24
PrivateKey = A-private-key
DNS = 1.1.1.1

[Peer]
PublicKey = S-public-key
Endpoint = S-ip-address:51820
AllowedIPs = 0.0.0.0/0, ::/0

B's configuration is similar to A, but with IP 172.9.9.101 and different private key.

# X wg0.conf

[Interface]
Address = 172.9.9.10/24
PrivateKey = X-private-key
DNS = 1.1.1.1

[Peer]
PublicKey = S-public-key
Endpoint = S-ip-address:51820
AllowedIPs = 0.0.0.0/0, ::/0
PersistentKeepalive = 25  # To keep the server reachable

This configuration works and all the hosts can access each other via the VPN, but I want that ONLY traffic directed to hosts 172.9.9.* goes through this VPN. The other traffic shall be routed by the default gateway.

What puzzles me is that, if I change the configuration of A so that

AllowedIPs = 172.9.9.0/24

then, for A, the packages are routed as intended (e.g. I can curl ifconfig.me and get A's public IP), but if I do the same on X, it will not work and packages not going to 172.9.9.0/24 will fail to be delivered.

EDIT#1

Forgot to mention that I would also love if, when connecting to X, the local clients such as B would not send packages outside the local network, so B -> Router -> X and not B -> Router -> S -> Router -> X.