Diagnosing Vitess VTGate Routing Failures by Keyspace and Shard

The Problem: Routing Errors That Look Like Outages

When an application query fails inside Vitess, the error often names a keyspace, shard, tablet type, or cell. Messages such as unknown keyspace, unknown shard, no healthy tablet, or all hosts unavailable can appear after topology changes, tablet restarts, DDL, or resharding. In many cases the cluster is healthy overall, but VTGate—the query router—cannot find a valid target for the requested keyspace and shard.

The useful takeaway: before restarting anything, determine whether the failure is a topology visibility problem, a tablet health problem, or a stale cache problem. The checks below are ordered so that each step narrows the cause.

Recognizable Condition

You are likely looking at a VTGate routing failure when:

• Errors intermittently or consistently name a specific keyspace, shard, tablet type (primary, replica, rdonly), or cell.
• The symptoms started after a topology change, vttablet restart, DDL execution, or resharding operation.
• Only one shard is affected, or only writes (primary) or only reads (replica/rdonly) fail.

Cause and Diagnostic Table

Symptom	Likely Cause	Diagnostic Focus
Unknown keyspace or shard	Keyspace/shard not created, or VTGate cell filter excludes it	Compare GetKeyspaces and GetShard output with the cells VTGate watches
No healthy tablet	vttablet down, MySQL down, health check failing, or wrong tablet type requested	Check GetTablets, tablet status/health, and MySQL SELECT 1
All hosts unavailable	Stale VTGate healthcheck cache, network partition, or bad credentials	Inspect VTGate status/healthcheck logs and tablet logs

Ordered Checks

1. Capture the exact error. Record the full message, keyspace, shard, tablet type, cells, timestamp, and whether one shard or all shards are affected. This determines whether the problem is global or scoped.
2. Verify topology exists. Confirm the keyspace and shard are present in the topology server and that the shard has the tablet types your query needs. Run from an administrative host with access to the Vitess control client (for example, vtctldclient or the cluster's supported equivalent) and appropriate permissions:

# Replace <keyspace> and <shard> with real values
vtctldclient GetKeyspaces
vtctldclient GetShard <keyspace>/<shard>

Expected check: the shard is listed and its record includes the expected serving types. If the command returns an error or the shard is absent, the topology is missing or invisible to VTGate.

1. List tablets for the shard. Confirm host, port, tablet type, and MySQL reachability. Use the control client to list tablets, then test MySQL directly from the tablet host or a bastion:

# List tablets visible in the cell; filter for the target shard
# Exact flags vary by release
vtctldclient GetTablets --cell <cell>

Expected check: at least one primary tablet exists for writes; replica or rdonly tablets exist for reads. From the tablet host, run mysql -h <host> -P <port> -u <user> -e 'SELECT 1' to confirm MySQL is accepting connections.

A concrete diagnostic decision: if GetShard returns the shard record but GetTablets shows no primary tablet while the application is writing, the problem is tablet health, not topology. If GetShard itself fails, the shard is missing from the topology server or VTGate cannot see it because of a cell filter.

1. Check vttablet health and VTGate cache. Confirm the vttablet process is running and its health check is passing. On the VTGate host, check the healthcheck status or logs for the shard. If VTGate reports no healthy tablets but the tablets are up, the healthcheck cache may be stale.
2. Check replication and read-only status. If reads fail, verify replica lag and that replicas are not stuck in read-only mode. If writes fail, confirm the primary is writable and not in read-only mode.
3. Check DDL or resharding state. If the failure followed DDL or resharding, verify VTGate schema tracking, routing rules, and VReplication workflow state before retrying traffic.

Fixes Tied to Findings

Finding	Fix	Risk / Note
Missing keyspace/shard or wrong cells	Create or rebuild through Vitess tooling, or restart VTGate with the correct cell list	Never hand-edit the topology server; a restart may hide a stale cache if topology is already healthy
Unhealthy tablet	Repair MySQL, network, or credentials, or restart vttablet and wait for health checks to pass	Capture logs and metrics before restarting; restarts can clear transient errors but may hide root causes
Stale VTGate healthcheck cache	Restart VTGate only after confirming topology and tablets are healthy	Health checks are eventually consistent; a transient error may resolve after the healthcheck interval
Replication lag or read-only replica	Pause affected reads or follow the failover runbook	Do not force writes to a read-only replica; verify the requested tablet type first

Escalation Criteria

Escalate to the DBA or Vitess operator when any of the following are true:

• Possible data loss or corruption is suspected.
• A primary failover or resharding workflow is in progress.
• Topology inconsistency is detected (for example, tablets in the topology server that do not match running vttablets).
• Multiple cells or shards are affected.
• Security or credential exposure is possible.
• No clear cause emerges after completing the ordered checks.

When escalating, include the exact error text, keyspace and shard names, tablet list, VTGate healthcheck output, and timestamps.

Limitations and Version Sensitivity

Command names and VTGate flags vary across Vitess releases. The legacy vtctlclient and the newer vtctldclient are not always interchangeable, and flag names such as those controlling cells or healthcheck intervals may differ. Use the client and flags documented for your cluster's release, and confirm endpoint names before relying on them.

Health checks are eventually consistent. A single no healthy tablet error can clear after the healthcheck interval, so confirm persistence before changing topology. Restarting VTGate or vttablet can clear transient cache errors, but capture logs and metrics first so the root cause is not lost.

How to Check the Result

After applying a fix, re-run the original application query. Then confirm that VTGate's healthcheck status shows the expected shard and tablet type as healthy. If the symptom followed DDL or resharding, verify that VTGate schema tracking and routing rules match MySQL and that the VReplication workflow reports a healthy state before restoring full traffic.