Catching Silent Downtime with CloudWatch Synthetics Canaries Before Your Users Do

Your status page is green, your Lambda errors are flat, and yet customers are emailing support because the login page returns a blank screen. This is the gap between internal monitoring (metrics your services emit about themselves) and external monitoring (what a real client actually experiences). Internal metrics can't tell you that DNS broke, a certificate expired, or a CDN misconfiguration is serving errors to half your users.

Amazon CloudWatch Synthetics canaries close that gap. A canary is a small script that runs on a schedule as a Lambda function, hits your application from outside like a user would, and publishes success, failure, and latency metrics to CloudWatch. The thesis of this post: a handful of well-chosen canaries on your most critical endpoints is one of the cheapest ways to convert "users reported it" into "we knew within five minutes."

What a canary actually is

Under the hood, every canary is a versioned Lambda function managed by CloudWatch Synthetics. You write a script in Node.js (using the provided Synthetics library, which wraps Puppeteer for browser checks or plain HTTP requests) or Python (with Selenium or requests-style helpers). The canary runs on a schedule you define — as often as once per minute — and each run emits metrics such as SuccessPercent, Duration, and failure counts into the CloudWatchSynthetics namespace. Execution logs land in CloudWatch Logs, and browser-based canaries can capture screenshots on failure, which is invaluable when the failure is visual rather than a status code.

Because canaries are Lambda functions, they can also run inside your VPC to probe internal endpoints that aren't publicly reachable — useful for checking an internal admin API or a service behind a load balancer.

A worked example: a lightweight HTTP check

The simplest useful canary doesn't need a browser at all. Here's a Node.js script that performs a GET against a health endpoint, asserts a 200 status, and checks that the response body contains an expected marker — catching the "returns 200 but serves an error page" failure mode:

const synthetics = require('Synthetics');
const log = require('SyntheticsLogger');

const apiCanaryBlueprint = async function () {
    const requestOptions = {
        hostname: 'api.example.com',   // your endpoint
        method: 'GET',
        path: '/health',
        port: 443,
        protocol: 'https:'
    };

    await synthetics.executeHttpStep('Verify health endpoint', requestOptions, async (response) => {
        if (response.statusCode !== 200) {
            throw 'Failed: ' + response.statusCode;
        }
        const body = await new Promise((resolve) => {
            let data = '';
            response.on('data', (chunk) => data += chunk);
            response.on('end', () => resolve(data));
        });
        if (!body.includes('"status":"ok"')) {
            throw 'Unexpected body: ' + body.substring(0, 200);
        }
        log.info('Health check passed');
    });
};

exports.handler = async () => {
    return await apiCanaryBlueprint();
};

To deploy this, open the CloudWatch console, go to Synthetics Canaries → Create canary, choose the "Heartbeat monitoring" blueprint, and replace the script with the code above. Set the schedule to every 5 minutes. Creating the canary requires permissions to create Lambda functions and the IAM role the canary assumes; the canary role itself needs AWSLambdaBasicExecutionRole-equivalent logging permissions, plus S3 write access to the artifact bucket the wizard creates for screenshots and HAR files. No console access is needed at runtime — everything runs under the canary's role.

Alerting that doesn't cry wolf

A canary without an alarm is just a log generator. Attach a CloudWatch Alarm to the canary's SuccessPercent metric and route it to SNS (or EventBridge) so the on-call channel gets paged. The important tuning decision is the evaluation window: a single failed run often means a transient network blip between the canary's region and your endpoint, not a real outage. A common starting point is alarming when SuccessPercent drops below 100 for 2 out of 3 consecutive evaluation periods — at a 5-minute cadence, that's roughly 10–15 minutes of sustained failure before a page, trading a little detection speed for far fewer false positives.

Also consider where the canary runs. A canary in the same region as your app won't catch a regional network path problem affecting distant users. Running a second canary from another region against the same endpoint gives you a cheap cross-check: if one region's canary fails and the other passes, the problem is likely network path, not your app.

Trade-offs and limitations

Canaries are priced per run — the published rate is on the order of $0.0012 per run, with a small free tier per account per month, but check the current CloudWatch pricing page before committing, since pricing changes. A 1-minute cadence across a dozen endpoints adds up; a 5-minute cadence on critical paths and 15-minute or hourly cadence on secondary ones is a reasonable budget posture.

Two scope limitations matter. First, a canary tests exactly what the script tests — a heartbeat GET won't catch a broken checkout flow five clicks deep. Multi-step browser scripts can model those journeys, but they're more brittle and need maintenance when the UI changes. Second, canaries don't validate rendering or full client-side behavior the way real user monitoring (RUM) does; they complement rather than replace it.

Finally, if your script calls other AWS services or hits authenticated endpoints, the canary role needs those permissions, and any credentials must come from Secrets Manager or environment variables — never hard-coded in the script.

Verify it works, then break it on purpose

After creating the canary, don't trust the green checkmark. Confirm three things: the SuccessPercent metric appears in the CloudWatchSynthetics namespace, the canary's CloudWatch log group shows your log.info output, and the alarm transitions to OK. Then deliberately break the endpoint — point the canary at a path that returns 500, or deploy a bad health response — and confirm the SNS notification actually arrives. An alerting path you've never seen fire is an alerting path you don't have. Once verified, expand from the heartbeat check to one multi-step script covering your single most revenue-critical flow, and revisit cadence against your bill after the first month.