Automating Test Suites in Bitbucket Pipelines for Pull Requests

Preventing Regressions with PR-Triggered Pipelines

Integrating automated tests into the pull request (PR) workflow prevents broken code from reaching your main branch. The primary challenge is ensuring tests run specifically on PR events without wasting build minutes on every single commit to every feature branch. The solution is a targeted bitbucket-pipelines.yml configuration that leverages the pull-requests trigger.

Prerequisites

• A Bitbucket Cloud repository with Pipelines enabled in the repository settings.
• A Docker image that matches your application's runtime (e.g., node:18 for JavaScript or openjdk:17 for Java).
• Repository variables configured for any sensitive data (API keys, database passwords) via Repository settings > Pipelines > Repository variables. Ensure "Secure" is checked to mask these values in logs.

Configuring the Pipeline

Create a bitbucket-pipelines.yml file in the root directory of your repository. To ensure tests run only when a pull request is created or updated, use the pull-requests keyword rather than the default pipeline block.

image: node:18

pipelines:
  pull-requests:
    '**': # This wildcard ensures tests run for all pull requests
      - step:
          name: "Install and Test"
          caches:
            - node
          script:
            - npm install
            - npm test

Refining Execution for Target Branches

In larger projects, you may only want to run heavy integration tests when a PR is targeting a specific branch, such as main or develop. You can replace the wildcard with specific patterns.

pipelines:
  pull-requests:
    'main': # Only runs when the PR target branch is 'main'
      - step:
          name: "Production Readiness Suite"
          script:
            - npm install
            - npm run test:integration
    'develop':
      - step:
          name: "Development Suite"
          script:
            - npm install
            - npm test

Diagnostic Checks and Verification

Once the configuration is committed and pushed, perform the following checks to verify the automation:

1. Trigger the Event: Create a new pull request or push a new commit to an existing open pull request.
2. Monitor Execution: Navigate to the Pipelines tab in the left sidebar. You should see a run labeled with the pull request ID.
3. Inspect Logs: Click into the running pipeline. Verify that the npm install (or equivalent) completes and that the test runner output (e.g., Jest, Mocha, JUnit) is visible in the console.
4. Validate Status: Confirm the pipeline ends with a green SUCCESS status. If it fails, the PR will show a "Build failed" indicator, blocking the merge if you have "Merge checks" enabled in repository settings.

Handling Failures and Recovery

If a pipeline fails, use these recovery paths based on the cause:

Failure Scenario	Action	Risk/Note
Flaky Test / Network Timeout	Use the Rerun button in the Pipeline UI.	Does not change code; only retries the environment.
Configuration Error (YAML)	Edit bitbucket-pipelines.yml locally, commit, and push.	Each commit triggers a new run; check for syntax errors.
Credential Expiration	Update the value in Repository variables.	Requires a rerun of the failed pipeline to pick up new values.

Limitations and Considerations

• Concurrent Build Limits: Bitbucket plans have a limit on how many pipelines can run simultaneously. If multiple developers push to PRs at once, builds will enter a Queued state.
• Build Minutes: Every PR update consumes build minutes. For very large suites, consider using caches (as shown in the example) to reduce npm install time.
• State Changes: Since this pipeline only runs tests and does not deploy, no rollback of infrastructure is required. If the pipeline modifies a remote test database, ensure your test suite includes a cleanup script in the after-script section.