Implementing Service Routing with Consul Prepared Queries

Solving Dynamic Service Discovery with Named Filters

Standard service discovery often requires applications to handle complex filtering logic—such as selecting nodes by specific tags or managing failover targets—within the application code. Consul Prepared Queries solve this by moving the filtering logic to the Consul server. Instead of querying a generic service and filtering the results manually, you define a named, reusable filter that returns a curated list of healthy endpoints via DNS or HTTP.

The primary takeaway is that Prepared Queries allow you to create a virtual service name (e.g., web-prod.service.consul) that dynamically resolves to a subset of instances based on tags, health status, and failover rules, without changing the application's connection string.

How Prepared Queries Work

A Prepared Query is a server-side definition that maps a friendly name to a specific set of criteria. When a client requests the name via DNS or the API, Consul evaluates the filter in real-time (subject to the TTL) and returns only the nodes that satisfy the requirements.

Example: Creating a Production-Specific Web Filter

In this scenario, we have a service named web. Some instances are tagged prod and others staging. We want a stable DNS entry that only points to production nodes, with a specific fallback node if all production instances fail.

Step 1: Create the Prepared Query
Run this command on a machine with the Consul CLI installed and administrative permissions:

consul prepared-query create -name web-prod -service web -tag prod -failover-node node1.dc1.consul -ttl 10s

Configuration Breakdown:

• -name web-prod: The friendly name used for DNS lookups.
• -service web: The base service to filter.
• -tag prod: Only return instances with the prod tag.
• -failover-node node1.dc1.consul: If no healthy prod nodes are found, return this specific node.
• -ttl 10s: The Time-To-Live for the result. The server caches the result for 10 seconds before re-evaluating the filter.

Step 2: Verify via DNS
Use dig to query the Consul DNS interface (default port 8600) from a local agent:

dig @127.0.0.1 -p 8600 web-prod.service.consul +short

Step 3: Verify via HTTP API
If you have the Query ID (returned during creation), you can execute it via the API:

curl -s http://localhost:8500/v1/prepared/query/execute/YOUR_QUERY_ID

Operational Limits and Constraints

While powerful, Prepared Queries have specific technical boundaries that impact architecture decisions:

• Agent Capacity: Consul agents have a default cap of 1,000 prepared queries. Exceeding this can lead to creation failures.
• TTL Minimums: The TTL cannot be set to zero. The minimum is 1 second.
• Read-Only Nature: Prepared Queries are strictly for discovery. They cannot be used to modify service metadata or change the state of the cluster.
• No Complex Aggregations: You cannot perform custom scoring or complex mathematical aggregations within the query filter.

Common Implementation Mistakes

Stale Results via High TTL

Setting a TTL that is too high (e.g., several minutes) can lead to "zombie" traffic. If a service instance becomes unhealthy or is deregistered, the Prepared Query may continue to return that IP until the TTL expires, causing connection timeouts in the application.

ACL Permission Failures

Prepared Queries are subject to Access Control Lists (ACLs). A common error is creating a query but failing to grant the application's token query:read permissions for that specific prepared query. This results in a 403 Forbidden error, which is often misdiagnosed as the service being missing.

Incorrect DNS Suffixes

Developers often attempt to query web-prod directly. Prepared Queries must be accessed using the full .service.consul suffix (or your specific Consul domain) to be routed correctly by the DNS interface.

Verification and Rollback

Checking the Result

To verify the current state of all filters, run:

consul prepared-query list

To test the failover mechanism, manually mark the production nodes as unhealthy or change their tags. Wait for the TTL to expire, then run the dig command again to ensure the failover-node is returned.

Rollback

Because creating a prepared query changes the state of the Consul server, you can remove the filter using the following command:

consul prepared-query deregister -id YOUR_QUERY_ID