Using Travis CI Matrix Builds to Run Parallel Tests Across Environments

Problem: Long Test Times

When a project supports multiple Ruby, Node, or Python versions, a single test suite can take minutes per version. Running them sequentially turns a quick commit‑review cycle into a slow, blocking process.

Solution: Matrix Builds

Travis CI’s matrix feature automatically expands a single configuration into multiple parallel jobs, each with a distinct set of variables (language version, OS, environment flags). The result is a dramatic reduction in wall‑clock time while still validating every combination.

How to Configure a Matrix

The matrix lives under the jobs key in .travis.yml. You can define simple lists or use include and exclude blocks for fine‑grained control.

# .travis.yml
language: python
python:
  - "3.6"
  - "3.7"
  - "3.8"

# Simple matrix for language versions
jobs:
  include:
    - language: node_js
      node_js: "12"
    - language: node_js
      node_js: "14"

# Custom environment variables per job
env:
  matrix:
    - DB=postgresql
    - DB=sqlite3

# Optional: stop all jobs when one fails
matrix:
  fast_finish: true

Key points:

• Indentation matters – YAML is whitespace‑sensitive. Use two spaces per level and avoid tabs.
• Each job starts in a clean virtual machine; there is no shared state.
• Use include to add jobs that override defaults (e.g., a special OS).
• Use exclude to prune unsupported combinations.
• Use allow_failures to let known flaky jobs fail without breaking the entire build.

Concrete Example: Python + Node.js + DB Versions

Below is a full .travis.yml that tests:

• Python 3.6, 3.7, 3.8 on Linux
• Node.js 12 and 14 on Linux
• Both PostgreSQL and SQLite databases per language version

language: python
python:
  - "3.6"
  - "3.7"
  - "3.8"

# Node jobs are added via matrix include
jobs:
  include:
    - language: node_js
      node_js: "12"
    - language: node_js
      node_js: "14"

# Environment matrix for databases
env:
  matrix:
    - DB=postgresql
    - DB=sqlite3

# Stop all jobs if any fails
matrix:
  fast_finish: true

# Example install & script steps
install:
  - if [ "$TRAVIS_PYTHON_VERSION" ]; then pip install -r requirements.txt; fi
  - if [ "$TRAVIS_NODE_VERSION" ]; then npm install; fi

script:
  - if [ "$TRAVIS_PYTHON_VERSION" ]; then python -m unittest discover; fi
  - if [ "$TRAVIS_NODE_VERSION" ]; then npm test; fi

When this file is pushed to a GitHub repo linked to Travis, the dashboard shows eight distinct jobs:

• Python 3.6 + PostgreSQL
• Python 3.6 + SQLite3
• Python 3.7 + PostgreSQL
• Python 3.7 + SQLite3
• Python 3.8 + PostgreSQL
• Python 3.8 + SQLite3
• Node 12 + PostgreSQL
• Node 12 + SQLite3
• Node 14 + PostgreSQL
• Node 14 + SQLite3

Common Mistakes

• Wrong indentation – A single missing space can cause Travis to ignore the matrix entirely.
• Using build: instead of jobs: – Older configs may still use build:, but the current syntax is jobs:.
• Exceeding the free account’s concurrent build quota – Open‑source projects get 100 concurrent builds/month; commercial accounts have higher limits.
• Assuming jobs share state – Each job is isolated; environment variables set in one job do not persist to others.
• Not using fast_finish when appropriate – Without it, a single failure can still let other jobs run, wasting resources.

Limits and Quotas

• Free accounts are limited to 100 concurrent builds per month. A large matrix can quickly consume this budget.
• Each job has a maximum runtime of 60 minutes on the free plan. Long tests may need to be split or optimized.
• Matrix size cannot exceed the concurrency quota; jobs will queue if the limit is reached.
• Travis does not support nested matrices; you must flatten combinations manually.

Verification Steps

1. Run travis lint .travis.yml locally to catch YAML syntax errors.
2. Push the file to GitHub; Travis will automatically trigger a build.
3. Open the Travis CI dashboard for the repository and confirm that multiple jobs appear under the same commit.
4. Click each job to inspect the log and verify that the environment variables (e.g., DB, TRAVIS_PYTHON_VERSION) are set correctly.
5. Use travis status --remote to get a quick summary of build results from the command line.

Conclusion

Matrix builds are a powerful way to keep your test suite fast and comprehensive. By carefully configuring jobs, env, and optional flags like fast_finish, you can run dozens of parallel jobs without manual scripting. Just remember to watch your build quota, keep YAML indentation clean, and verify the matrix in the Travis UI before relying on the results for production deployments.