Diagnosing VSI-to-VSI Connectivity Issues in IBM Cloud VPC Using Security Groups

Recognizable Condition

Two Virtual Server Instances (VSIs) in the same IBM Cloud VPC cannot establish a TCP connection on a specific port, even though they share the same subnet or are reachable via ICMP. The symptom often appears as a timeout or connection‑refused error when an application on VSI A tries to talk to a service listening on VSI B.

Cause/Diagnostic Table

Possible Cause	What to Look For	Typical Fix
Missing inbound rule on VSI B’s Security Group	TCP packets from VSI A never reach the listening process on VSI B	Add an inbound rule allowing the port from VSI A’s IP or its Security Group
Missing outbound rule on VSI A’s Security Group	VSI A drops the outbound SYN before it leaves the interface	Add an outbound rule allowing the port to VSI B’s IP or its Security Group
Conflicting rules where a deny overrides an allow	Security Group evaluation shows a higher‑priority deny for the same traffic	Adjust rule priority or remove the conflicting deny
Default Security Group is overly restrictive	Both VSIs still using the default SG that denies all traffic	Replace or augment the default SG with a custom SG that permits required traffic

Ordered Checks

1. Verify basic IP reachability

   From VSI A, run:

   ping -c 4 <private‑IP‑of‑VSI‑B>

   If ping fails, check routing, subnet ACLs, or VPC peering before proceeding.

2. Test the specific TCP port

   From VSI A, run:

   nc -zv <private‑IP‑of‑VSI‑B> <port>

   or, if nc is not available:

   telnet <private‑IP‑of‑VSI‑B> <port>

   Note the output: Connection refused indicates the packet reached the host but was rejected; timeout suggests the packet never arrived.

3. Inspect the Security Group of the destination VSI (VSI B)

   Using the IBM Cloud CLI (requires Authorization to read Security Groups):

   ibmcloud is security-group <SG‑ID‑of‑VSI‑B> --json

   Look for an inbound rule that matches:

   • Protocol: TCP
   • Port range: the target port (or a range that includes it)
   • Source: either the private IP of VSI A, its Security Group ID, or the subnet CIDR that contains VSI A.
4. Inspect the Security Group of the source VSI (VSI A)

   Run the same CLI command for VSI A’s SG and verify an outbound rule exists with the same protocol, port, and destination (VSI B’s IP, SG, or subnet).

5. Check for rule conflicts

   In the JSON output, note the direction and action fields. If a rule with action: "deny" has a lower priority number (higher precedence) than an matching allow, the deny wins.

Fixes Tied to Findings

Add Missing Inbound Rule (Destination)

If step 3 shows no matching inbound rule:

ibmcloud is security-group-rule-add <SG‑ID‑of‑VSI‑B> \
  --direction inbound \
  --protocol tcp \
  --port-min <port> \
  --port-max <port> \
  --remote <source‑identifier> \
  --action allow

Replace <source‑identifier> with either the private IP of VSI A (e.g., 10.10.10.5/32) or the Security Group ID of VSI A (prefixed with sg-).

Add Missing Outbound Rule (Source)

If step 4 shows no matching outbound rule on VSI A:

ibmcloud is security-group-rule-add <SG‑ID‑of‑VSI‑A> \
  --direction outbound \
  --protocol tcp \
  --port-min <port> \
  --port-max <port> \
  --remote <destination‑identifier> \
  --action allow

Resolve Conflicting Deny

Identify the deny rule’s ID from the JSON, then either:

• Delete it: ibmcloud is security-group-rule-delete <SG‑ID> <rule‑ID>
• Or raise its priority number (lower precedence) using ibmcloud is security-group-rule-update with a higher priority value.

Verification

After applying the change, repeat step 2 (nc -zv or telnet) from VSI A. A successful test returns:

• Connection to <IP> <port> port [tcp/*] succeeded! (nc)
• Or a telnet session that connects and can be closed with Ctrl+] Enter.

Optionally, run a packet capture on VSI B to confirm the SYN arrives:

sudo tcpdump -i eth0 host <private‑IP‑of‑VSI‑A> and port <port>

You should see SYN packets followed by SYN‑ACK if the service is listening.

Escalation Criteria

• If after verifying both SGs the connection still fails, check VPC network ACLs, routing tables, or floating IP NAT rules.
• If packet capture shows SYN packets leaving VSI A but never arriving at VSI B, the issue may lie in the underlying physical network or a misconfigured VPC gateway; open a support ticket with IBM Cloud providing the capture and SG details.
• If you are unable to modify Security Groups due to missing IAM permissions, request the Authorization role Administrator or Operator on the VPC resource group from your account administrator.

Limitations and Practical Tips

• Security Groups are stateful: you only need to define the rule for the initiating direction of traffic. However, return traffic is automatically allowed, so inbound rules on the responder are often unnecessary if the outbound rule on the initiator is correct.
• Default Security Groups deny all traffic until you add explicit rules; do not rely on the default for production workloads.
• Changes to Security Groups are eventually consistent; allow up to a minute for propagation before retesting.
• Be cautious when widening source/destination ranges (e.g., using 0.0.0.0/0) as this can expose unintended services.