Scaling Heroku Dynos: Managing Ephemerality and Request Timeouts

The primary challenge when scaling Heroku applications is the ephemeral nature of the dyno. Because Heroku runs code in isolated Linux containers, any data written to the local filesystem is wiped during every deployment, restart, or scaling event. To scale horizontally without data loss, you must externalize all state to managed services and architect your requests to fit within the Heroku Router's strict timing constraints.

The Dyno Architecture and State

Heroku utilizes "dynos," which are isolated containers running your application code. Horizontal scaling involves adding more identical instances of a process rather than increasing the resources of a single instance. The Heroku Router acts as the load balancer, distributing incoming HTTP requests across available web dynos using a round-robin-like algorithm.

Because dynos are ephemeral, local storage cannot be used for user uploads, session data, or databases. All persistence must be moved to a backing service, such as Heroku Postgres for relational data or Heroku Redis for caching and session management.

Implementing Horizontal Scaling

To increase the capacity for concurrent requests, you can scale the number of web dynos via the Heroku CLI. This command should be run from a local terminal with the appropriate permissions for the target application.

# Scale the 'web' process to 3 instances
heroku scale web:3

To verify that the instances are active and receiving traffic, use the following commands to check the process list and monitor the logs:

# List all running dynos
heroku ps

# Stream logs to observe new dynos booting
heroku logs --tail

Resolving the H12 Gateway Timeout

The Heroku Router enforces a hard 30-second request timeout. If a web dyno does not send a response within this window, the router terminates the connection and returns an H12 Gateway Timeout error. Horizontal scaling does not resolve H12 errors if the bottleneck is a slow individual process or a blocked thread.

To diagnose if an endpoint is prone to H12 errors, you can implement a temporary test route that exceeds the limit:

// Example in Node.js/Express to trigger H12
app.get('/timeout-test', (req, res) => {
  // Sleep for 31 seconds to exceed the Heroku Router limit
  setTimeout(() => {
    res.send('This response will not be delivered');
  }, 31000);
});

The practical solution for H12 errors is to offload long-running tasks to a background worker process using a job queue (such as Sidekiq for Ruby or Bull for Node.js), returning a 202 Accepted response to the client immediately.

Verifying Filesystem Ephemerality

A common architectural mistake is relying on /tmp for temporary storage across requests or restarts. You can verify this behavior by manually creating a file and restarting the dyno.

1. Log into a running dyno and create a file:
   heroku run bash -c "touch /tmp/test_file.txt"
2. Confirm the file exists:
   heroku run bash -c "ls /tmp/test_file.txt"
3. Restart the application to trigger a container recycle:
   heroku app restart
4. Attempt to locate the file again:
   heroku run bash -c "ls /tmp/test_file.txt"

The expected result is that the second ls command fails, confirming that the local filesystem is wiped upon restart. To persist this data, you must use an external object store like Amazon S3.

Summary of Scaling Limitations

Issue	Scaling Effect	Corrective Action
Memory Leaks	Increases aggregate RAM, but leaks persist per dyno	Fix code-level memory management
Slow Requests (H12)	No effect on individual request duration	Move task to background worker
Local File Storage	Data is fragmented across dynos and lost on restart	Use external storage (S3/Postgres)