Diagnosing Vitess 'No Healthy Tablet Available' Errors from vtgate

Recognizing the condition

Your application starts failing queries through vtgate with an error like target: commerce.80-.primary: no healthy tablet available, or gRPC UNAVAILABLE responses. Sometimes only reads fail while writes succeed, or the reverse — that asymmetry is itself a clue, because it tells you which tablet type (primary, replica, rdonly) lost health in the affected shard.

The useful takeaway up front: this error almost never means vtgate itself is broken. It means vtgate's health-check-driven view of the cluster currently contains zero serving tablets for the requested target. Your job is to find out whether that view is accurate (a real tablet/MySQL problem), stale (a discovery convergence issue), or expected (a planned reparent or resharding cutover in progress).

Note: exact error strings, flag names, and health-check behavior vary across Vitess major versions. Confirm flags against the version you actually run before changing anything.

Likely causes at a glance

Ordered checks

Work through these in order; each step narrows the layer at fault.

1. Correlate with operations. Check vtctld's workflow/reparent history for the affected keyspace and shard around the error timestamps. A PlannedReparentShard or resharding cutover intentionally creates a brief window with no serving primary. If the errors lasted seconds and line up with an operation, this is expected behavior — verify recovery and stop here.
2. List the tablets. From a host with the Vitess client tools and access to the topology service, run vtctldclient ListAllTablets (or open the vtctld UI) and look at every tablet in the affected shard. You need operator-level access to the cluster. Check for tablets in a non-serving state and for records pointing at hosts that no longer exist.
3. Check tablet health and serving state. For each tablet in the shard, confirm its reported state and health. A tablet whose health check against its local MySQL is failing will be excluded from vtgate's routing even though the process is running.
4. Verify MySQL on the tablet host. On the affected tablet's host (requires SSH or equivalent access), confirm the MySQL process is up, accepts connections, and — for replicas — check SHOW REPLICA STATUS for running replication and lag within the discovery threshold your deployment configures.
5. Compare with vtgate's view. Query vtgate's /debug/health or status endpoint and compare it against what vtctld shows. If tablets are healthy in the topology but missing or wrong in vtgate's view, you have a discovery convergence problem, not a tablet problem.
6. Read vtgate logs. Look for which tablet alias and type vtgate excluded and any health-stream errors. This often names the exact culprit directly.

Fixes tied to findings

Primary not serving

If the shard has no serving primary and no reparent is in progress, run an emergency or planned reparent from vtctld (for example EmergencyReparentShard when the old primary is unreachable). This changes cluster state — confirm you are targeting the correct keyspace/shard and that the candidate replica has the most advanced position before proceeding. A wrong reparent can lose transactions.

Replicas excluded for lag

Fix the underlying replication problem first: a stalled SQL thread, a long-running backup, or a network issue on the replica. As a temporary measure you can raise the lag tolerance vtgate uses for discovery, but do not leave it raised — a permanently high threshold masks real lag and can serve stale reads to your application.

Stale tablet records

If ListAllTablets shows a record for a dead instance, delete it with the vtctld client so discovery stops advertising it. Be conservative: manual topology edits can split-brain discovery if you delete or alter records for tablets that are actually alive. Only remove records for instances you have confirmed are gone.

vtgate's view is stale

If all tablets are healthy but one vtgate still routes to nothing, watch its /debug/health for convergence — the health stream usually catches up on its own within seconds to a minute. If a single vtgate never converges while others are fine, restarting that vtgate is a reasonable step; if all vtgates disagree with the topology, suspect the topology service itself.

Verifying the fix

Before closing the incident, run a simple read and a simple write through vtgate against the affected keyspace and shard — for example, a SELECT against a small table and an insert into a scratch table, using your normal application connection path. Confirm both succeed, and re-check /debug/health to verify the target now lists a serving tablet of each type you route to. Also re-run the failing application query, not just a synthetic one, since routing can differ per table or shard.

When to escalate

Escalate to a Vitess-experienced operator or the community (the Vitess Slack is the usual venue) when any of these hold: the primary cannot be made serving after reparent attempts; topology data appears corrupted or inconsistent between what vtctld shows and what tablets report; or errors persist after every tablet reports healthy and all vtgates have been restarted. These point to topo/etcd-level or vtgate-internal faults that routine tablet operations will not fix, and further manual intervention without deeper expertise risks making discovery worse.