Choosing Artifact Storage for Large Monorepo Projects in GitLab CI/CD

Decision and Constraints

When a monorepo builds multiple Maven modules, the pipeline often needs to persist compiled JARs or other build outputs for later stages, downstream pipelines, or release processes. The choice of where to store these artifacts affects cost, retrieval speed, access control, and how long the data is retained. The decision must balance native GitLab features against the operational overhead of external services.

Options Comparison

Option	Pros	Cons	Setup Complexity
GitLab Packages (Maven repository)	Native integration, versioning, fine‑grained access control via CI/JWT tokens	SaaS plans enforce storage quotas; large binaries can increase latency	Low
Job Artifacts	Immediately available in the same pipeline; no extra configuration	Expire after the job (unless `expire_in` is set); not shareable across pipelines; size limits apply	Very Low
External S3‑compatible storage (e.g., MinIO, AWS S3)	Virtually unlimited scale, low cost per GB, configurable retention lifecycle	Requires managing credentials, bucket policies, and encryption; separate service to monitor	Medium

Trade‑offs Explanation

GitLab Packages (Maven)

This option stores artifacts as Maven packages inside GitLab’s package registry. Access is controlled by the same tokens used for Git operations, and each version is immutable, which simplifies dependency resolution. However, GitLab SaaS imposes a hard storage limit (e.g., 10 GB on the Free tier) that is shared with Container Registry and other package types. Exceeding the limit causes upload failures, and retrieving large JARs may be slower than a purpose‑built object store.

Job Artifacts

Artifacts uploaded with the `artifacts:` keyword are available for the duration of the pipeline and can be downloaded in later stages without extra steps. They are ideal for transient build outputs that are only needed within the same pipeline run. Because they are tied to a specific job, they cannot be reused by other pipelines or long‑term release processes, and they are subject to the artifact size limits defined in the instance settings.

External S3‑compatible Storage

Using an object store decouples artifact retention from GitLab’s internal quotas. You can define lifecycle rules to automatically delete old versions, and you pay only for the storage and request traffic you actually use. The downside is the need to manage access keys or IAM roles, ensure bucket policies restrict access to the intended GitLab runners, and enable server‑side encryption to protect potentially sensitive build outputs.

Concrete Implementation: Publishing Maven Outputs to an External S3 Bucket

Prerequisites

• An S3‑compatible bucket (e.g., my‑company‑maven‑artifacts) with versioning enabled.
• CI variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY defined as protected and masked in the project’s Settings → CI/CD → Variables.
• The GitLab runner must have network access to the S3 endpoint.

.gitlab-ci.yml Snippet

stages:
  - build
  - publish
  - verify

build_maven:
  stage: build
  script:
    - mvn clean package -DskipTests
  artifacts:
    paths:
      - target/**/*.jar
    expire_in: 1h

publish_to_s3:
  stage: publish
  needs: ["build_maven"]
  script:
    - aws s3 cp target/ s3://$MY_BUCKET/maven/${CI_COMMIT_SHORT_SHA}/ --recursive --exclude "" --include "*.jar"
  only:
    - branches

verify_s3_artifact:
  stage: verify
  image: maven:3.9-eclipse-temurin-17
  script:
    - aws s3 cp s3://$MY_BUCKET/maven/${CI_COMMIT_SHORT_SHA}/my‑app‑1.0.0.jar ./my-app.jar
    - mvn -f ./pom.xml dependency:get -Dartifact=my.group:my-app:1.0.0 -DrepoUrl=file://$(pwd)
    - mvn verify
  only:
    - branches

The publish_to_s3 job uploads all JARs from the target/ directory to a path that includes the short commit SHA, making each pipeline’s output uniquely addressable. The verify_s3_artifact job downloads a specific artifact and runs Maven commands to confirm it is usable.

Limitations and Practical Verification

Storage cost: While S3 is cheap, frequent uploads of large snapshots can increase request charges. Monitor usage via the bucket’s metrics or your cloud provider’s billing dashboard.

Access control: If the CI variables are accidentally exposed (e.g., echoed in a job log), anyone with repository read access could retrieve or overwrite artifacts. Ensure that aws s3 cp commands are not run with set -x or similar debugging flags that would print the variables.

Verification steps: After the publish_to_s3 job finishes, inspect its log for a line similar to:

upload: target/my-app-1.0.0.jar to s3://my-company-maven-artifacts/maven/a1b2c3d/my-app-1.0.0.jar

In the downstream verify_s3_artifact job, run:

aws s3 ls s3://$MY_BUCKET/maven/${CI_COMMIT_SHORT_SHA}/

Confirm that the expected JAR appears with the correct size and timestamp. Then execute the Maven verification commands shown above; a successful mvn verify exit code indicates the artifact was not corrupted during transfer.

When to Choose Each Option

• Use GitLab Packages when you need tight integration with GitLab’s dependency proxy, version immutability, and you stay within the storage quota of your plan.
• Use Job Artifacts for short‑lived build outputs that are only consumed later in the same pipeline (e.g., test reports, temporary binaries).
• Opt for External S3‑compatible storage when you require long‑term retention, sharing across multiple pipelines or projects, or you anticipate exceeding GitLab’s internal storage limits.