Monorepo CI/CD: Caching, Affected Builds, and Parallel Execution

Introduction

Monorepos have become the standard architecture for modern software development, enabling code sharing, atomic commits, and consistent tooling across multiple projects. However, as monorepos grow, CI/CD pipelines become increasingly expensive and slow. Running full builds and tests for every commit across all packages is unsustainable when you have dozens or hundreds of packages. The solution lies in intelligent CI/CD strategies that understand your monorepo's dependency graph, cache aggressively, and execute only what's necessary.

In this comprehensive guide, we'll explore advanced CI/CD techniques for monorepos using tools like Turborepo, Nx, and GitHub Actions. You'll learn how to implement dependency-aware builds, intelligent caching, affected package detection, and parallel execution to dramatically reduce build times while maintaining reliability. These patterns are essential for any team scaling a monorepo beyond a handful of packages.

Understanding Monorepo CI/CD: Core Concepts

The Monorepo CI/CD Challenge

Traditional CI/CD pipelines treat each repository independently. In a monorepo, you need a fundamentally different approach:

Challenge	Traditional Repo	Monorepo
Build Scope	Entire project	Only affected packages
Caching	Per-project	Cross-project with dependency awareness
Parallelization	Limited	Extensive across packages
Dependency Tracking	Simple	Complex graph traversal
Artifact Sharing	Rare	Common across packages

Dependency Graph

The foundation of monorepo CI/CD is understanding your dependency graph. Tools like Turborepo and Nx analyze package.json and import statements to build a complete dependency map:

// apps/web/package.json
{
  "dependencies": {
    "@myorg/ui": "workspace:*",
    "@myorg/utils": "workspace:*",
    "@myorg/api-client": "workspace:*"
  }
}
 
// packages/ui/package.json
{
  "dependencies": {
    "@myorg/utils": "workspace:*"
  }
}

This creates a dependency graph:

web → ui → utils
web → api-client → utils
web → utils

When utils changes, web, ui, and api-client all need to be rebuilt. When ui changes, only web and ui need rebuilding.

Architecture and Design Patterns

Pattern 1: Turborepo Configuration

Turborepo provides a turbo.json configuration that defines your build pipeline:

{
  "$schema": "https://turbo.build/schema.json",
  "globalDependencies": [".env"],
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": ["dist/**", ".next/**", "!.next/cache/**"],
      "env": ["NODE_ENV"]
    },
    "test": {
      "dependsOn": ["build"],
      "outputs": ["coverage/**"],
      "inputs": ["src/**/*.tsx", "src/**/*.ts", "test/**/*.ts", "test/**/*.tsx"]
    },
    "lint": {
      "outputs": []
    },
    "dev": {
      "cache": false,
      "persistent": true
    },
    "type-check": {
      "dependsOn": ["^build"],
      "outputs": []
    }
  }
}

Pattern 2: Nx Configuration

Nx uses a different approach with nx.json and project.json:

// nx.json
{
  "tasksRunnerOptions": {
    "default": {
      "runner": "nx-cloud",
      "options": {
        "cacheableOperations": ["build", "test", "lint", "typecheck"],
        "accessToken": "your-token"
      }
    }
  },
  "targetDefaults": {
    "build": {
      "dependsOn": ["^build"],
      "inputs": ["production", "^production"],
      "outputs": ["{projectRoot}/dist"]
    },
    "test": {
      "inputs": ["default", "^production", "{workspaceRoot}/jest.preset.js"]
    }
  }
}
 
// apps/web/project.json
{
  "targets": {
    "build": {
      "executor": "@nx/webpack:webpack",
      "outputs": ["{options.outputPath}"],
      "options": {
        "outputPath": "dist/apps/web",
        "main": "apps/web/src/main.ts",
        "tsConfig": "apps/web/tsconfig.app.json"
      }
    },
    "test": {
      "executor": "@nx/jest:jest",
      "outputs": ["{workspaceRoot}/coverage/{projectRoot}"],
      "options": {
        "jestConfig": "apps/web/jest.config.ts"
      }
    }
  }
}

Pattern 3: GitHub Actions for Monorepos

Configure GitHub Actions to detect changes and run only affected tasks:

# .github/workflows/ci.yml
name: Monorepo CI
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
 
jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      web: ${{ steps.changes.outputs.web }}
      api: ${{ steps.changes.outputs.api }}
      ui: ${{ steps.changes.outputs.ui }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v2
        id: changes
        with:
          filters: |
            web:
              - 'apps/web/**'
              - 'packages/ui/**'
              - 'packages/utils/**'
            api:
              - 'apps/api/**'
              - 'packages/utils/**'
            ui:
              - 'packages/ui/**'
 
  build-web:
    needs: detect-changes
    if: needs.detect-changes.outputs.web == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'pnpm'
      - run: pnpm install --frozen-lockfile
      - run: pnpm turbo build --filter=web
      - run: pnpm turbo test --filter=web

Step-by-Step Implementation

Setting Up Turborepo with Remote Caching

Remote caching allows your entire team and CI system to share build artifacts:

# Install Turborepo
pnpm add turbo -Dw
 
# Login to Vercel for remote caching
npx turbo login
npx turbo link
 
# turbo.json with remote caching
{
  "$schema": "https://turbo.build/schema.json",
  "remoteCache": {
    "team": "your-team-name"
  }
}

Implementing Affected Package Detection

Only build and test packages that have changed or depend on changed packages:

# Turborepo: Run only affected packages
pnpm turbo build --filter='...[origin/main]'
 
# Nx: Run only affected packages
npx nx affected --target=test --base=origin/main --head=HEAD
 
# Custom script for detecting changes
#!/bin/bash
# scripts/get-affected.sh
CHANGED_FILES=$(git diff --name-only origin/main...HEAD)
AFFECTED_PACKAGES=""
 
for file in $CHANGED_FILES; do
  # Extract package name from file path
  if [[ $file =~ ^packages/([^/]+)/ ]]; then
    pkg="${BASH_REMATCH[1]}"
    if [[ ! $AFFECTED_PACKAGES =~ $pkg ]]; then
      AFFECTED_PACKAGES="$AFFECTED_PACKAGES @$pkg"
    fi
  elif [[ $file =~ ^apps/([^/]+)/ ]]; then
    app="${BASH_REMATCH[1]}"
    if [[ ! $AFFECTED_PACKAGES =~ $app ]]; then
      AFFECTED_PACKAGES="$AFFECTED_PACKAGES @$app"
    fi
  fi
done
 
echo $AFFECTED_PACKAGES

Parallel Execution with Turborepo

Configure Turborepo to run tasks in parallel with concurrency limits:

# Run with 10 parallel tasks
pnpm turbo build --concurrency=10
 
# Run with 50% of available CPUs
pnpm turbo build --concurrency=50%
 
# Run without parallelism (for debugging)
pnpm turbo build --concurrency=1
 
# turbo.json with parallel configuration
{
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": ["dist/**"],
      "parallel": true
    },
    "test": {
      "dependsOn": ["build"],
      "parallel": true
    }
  }
}

GitHub Actions with Matrix Strategy

Run multiple packages in parallel using GitHub Actions matrix:

# .github/workflows/parallel-build.yml
name: Parallel Build
on:
  pull_request:
    branches: [main]
 
jobs:
  get-packages:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - id: set-matrix
        run: |
          PACKAGES=$(pnpm turbo run build --filter='...[origin/main]' --dry-run=json | jq -r '.tasks[].package' | jq -R -s -c 'split("\n") | map(select(. != ""))')
          echo "matrix={\"package\":$PACKAGES}" >> $GITHUB_OUTPUT
 
  build:
    needs: get-packages
    runs-on: ubuntu-latest
    strategy:
      matrix: ${{ fromJson(needs.get-packages.outputs.matrix) }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'pnpm'
      - run: pnpm install --frozen-lockfile
      - run: pnpm turbo build --filter=${{ matrix.package }}
      - run: pnpm turbo test --filter=${{ matrix.package }}

Real-World Use Cases

Use Case 1: Large-Scale SaaS Platform

A SaaS platform with 50+ packages, multiple applications, and shared libraries:

# .github/workflows/saas-ci.yml
name: SaaS Platform CI
on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]
 
jobs:
  affected:
    runs-on: ubuntu-latest
    outputs:
      web: ${{ steps.affected.outputs.web }}
      api: ${{ steps.affected.outputs.api }}
      mobile: ${{ steps.affected.outputs.mobile }}
      shared: ${{ steps.affected.outputs.shared }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: pnpm/action-setup@v2
        with:
          version: 8
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'pnpm'
      - run: pnpm install --frozen-lockfile
      - id: affected
        run: |
          echo "web=$(pnpm turbo run build --filter='...[origin/main]' --filter='./apps/web' --dry-run | grep -c 'web')" >> $GITHUB_OUTPUT
          echo "api=$(pnpm turbo run build --filter='...[origin/main]' --filter='./apps/api' --dry-run | grep -c 'api')" >> $GITHUB_OUTPUT
 
  deploy-web:
    needs: affected
    if: needs.affected.outputs.web != '0'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pnpm install --frozen-lockfile
      - run: pnpm turbo build --filter=web
      - run: pnpm turbo test --filter=web
      - uses: amondnet/vercel-action@v20
        with:
          vercel-token: ${{ secrets.VERCEL_TOKEN }}
          vercel-org-id: ${{ secrets.VERCEL_ORG_ID }}
          vercel-project-id: ${{ secrets.VERCEL_PROJECT_ID_WEB }}
 
  deploy-api:
    needs: affected
    if: needs.affected.outputs.api != '0'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pnpm install --frozen-lockfile
      - run: pnpm turbo build --filter=api
      - run: pnpm turbo test --filter=api
      - uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      - run: aws ecs update-service --cluster production --service api --force-new-deployment

Use Case 2: Micro-Frontend Architecture

Managing multiple frontend applications sharing common components:

// turbo.json
{
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": [".next/**", "!.next/cache/**", "dist/**"]
    },
    "dev": {
      "cache": false,
      "persistent": true
    },
    "lint": {
      "outputs": []
    },
    "test": {
      "dependsOn": ["build"],
      "outputs": ["coverage/**"]
    },
    "e2e": {
      "dependsOn": ["build"],
      "outputs": []
    },
    "deploy": {
      "dependsOn": ["build", "test", "e2e"],
      "cache": false
    }
  }
}

Use Case 3: Multi-Language Monorepo

Supporting multiple languages with appropriate build tools:

# .github/workflows/multi-language.yml
name: Multi-Language Monorepo
on:
  pull_request:
 
jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      frontend: ${{ steps.changes.outputs.frontend }}
      backend: ${{ steps.changes.outputs.backend }}
      infrastructure: ${{ steps.changes.outputs.infrastructure }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v2
        id: changes
        with:
          filters: |
            frontend:
              - 'apps/frontend/**'
              - 'packages/ui/**'
            backend:
              - 'apps/backend/**'
              - 'packages/shared/**'
            infrastructure:
              - 'infrastructure/**'
 
  frontend:
    needs: detect-changes
    if: needs.detect-changes.outputs.frontend == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v2
      - run: pnpm install --frozen-lockfile
      - run: pnpm turbo build --filter=frontend
      - run: pnpm turbo test --filter=frontend
 
  backend:
    needs: detect-changes
    if: needs.detect-changes.outputs.backend == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v3
        with:
          go-version: '1.21'
      - run: cd apps/backend && go build ./...
      - run: cd apps/backend && go test ./...
 
  infrastructure:
    needs: detect-changes
    if: needs.detect-changes.outputs.infrastructure == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v2
      - run: cd infrastructure && terraform init
      - run: cd infrastructure && terraform plan

Best Practices for Production

Cache everything possible: Configure remote caching for all build artifacts, test results, and lint outputs. This dramatically reduces CI time for unchanged packages.
Use affected detection: Never run full builds on PRs. Use dependency-aware affected detection to only build and test what changed.
Parallelize aggressively: Run independent packages in parallel. Use matrix strategies in CI to fan out work across multiple runners.
Optimize cache keys: Include lock files, source code hashes, and configuration in cache keys. Exclude volatile files like node_modules from cache.
Use workspace-aware tools: Tools like Turborepo and Nx understand workspace dependencies natively. Avoid custom scripts that reimplement this logic.
Monitor CI performance: Track build times, cache hit rates, and parallelization efficiency. Use metrics to identify bottlenecks.
Implement CI cost controls: Set budgets for CI minutes and alert when approaching limits. Use spot instances for non-critical builds.
Document CI configuration: Maintain clear documentation of your CI pipeline, caching strategy, and affected detection logic.

Common Pitfalls and Solutions

Pitfall	Impact	Solution
Running full builds on every PR	Slow CI, wasted resources	Implement affected detection
No remote caching	Redundant builds across team	Set up Turborepo or Nx Cloud
Incorrect dependency graph	Missing builds for affected packages	Verify package.json dependencies
Too many parallel jobs	Resource exhaustion, timeouts	Set concurrency limits
Stale cache	Incorrect build artifacts	Invalidate cache on config changes
Missing CI for shared packages	Breaking changes go undetected	Always test affected dependents

Performance Optimization

Optimize your monorepo CI/CD pipeline for maximum efficiency:

# Turborepo: Run with verbose logging for debugging
pnpm turbo build --filter='...[origin/main]' --verbose
 
# Turborepo: Prune unused packages from build
pnpm turbo prune --scope=web
 
# Nx: Run with cloud distribution
npx nx affected --target=test --base=origin/main --head=HEAD --parallel=3
 
# Custom: Measure build time
time pnpm turbo build --filter='...[origin/main]'
 
# Custom: Check cache hit rate
pnpm turbo build --filter='...[origin/main]' --dry-run=json | jq '.tasks | map(select(.cache == "HIT")) | length'

GitHub Actions Cache Optimization

- uses: actions/cache@v3
  with:
    path: |
      node_modules
      .turbo
      packages/*/node_modules
    key: ${{ runner.os }}-pnpm-${{ hashFiles('**/pnpm-lock.yaml') }}
    restore-keys: |
      ${{ runner.os }}-pnpm-

Comparison with Alternatives

Feature	Turborepo	Nx	Bazel	Pants
Language Support	JS/TS	JS/TS (extensible)	Multi-language	Multi-language
Remote Caching	Vercel	Nx Cloud	Remote Cache	Pants Remote
Affected Detection	Built-in	Built-in	Built-in	Built-in
Parallelization	Built-in	Built-in	Built-in	Built-in
Learning Curve	Low	Medium	High	High
Community	Large	Large	Large	Small
Configuration	turbo.json	nx.json	BUILD files	BUILD files

Advanced Patterns and Techniques

Dynamic Pipeline Generation

Generate pipeline configuration based on workspace structure:

// scripts/generate-turbo-config.js
const fs = require('fs');
const path = require('path');
const glob = require('glob');
 
const packages = glob.sync('packages/*/package.json').map(pkg => {
  const content = JSON.parse(fs.readFileSync(pkg, 'utf-8'));
  return { name: content.name, path: path.dirname(pkg) };
});
 
const turboConfig = {
  $schema: 'https://turbo.build/schema.json',
  pipeline: {
    build: {
      dependsOn: ['^build'],
      outputs: ['dist/**', '.next/**']
    },
    test: {
      dependsOn: ['build'],
      outputs: ['coverage/**']
    }
  }
};
 
// Add package-specific overrides
packages.forEach(pkg => {
  if (pkg.name.includes('api')) {
    turboConfig.pipeline[`${pkg.name}:deploy`] = {
      dependsOn: ['build', 'test'],
      cache: false
    };
  }
});
 
fs.writeFileSync('turbo.json', JSON.stringify(turboConfig, null, 2));

CI/CD with Deployment Preview

# .github/workflows/preview.yml
name: Preview Deployment
on:
  pull_request:
    branches: [main]
 
jobs:
  preview:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pnpm install --frozen-lockfile
      - run: pnpm turbo build --filter='...[origin/main]'
      - run: pnpm turbo test --filter='...[origin/main]'
      - uses: amondnet/vercel-action@v20
        with:
          vercel-token: ${{ secrets.VERCEL_TOKEN }}
          vercel-org-id: ${{ secrets.VERCEL_ORG_ID }}
          vercel-project-id: ${{ secrets.VERCEL_PROJECT_ID }}
          working-directory: apps/web
          github-comment: true

Testing Strategies

// Test CI configuration locally
describe('Monorepo CI Configuration', () => {
  test('turbo.json is valid', () => {
    const turbo = JSON.parse(fs.readFileSync('turbo.json', 'utf-8'));
    expect(turbo.pipeline).toBeDefined();
    expect(turbo.pipeline.build).toBeDefined();
  });
 
  test('all packages have correct dependencies', () => {
    const packages = glob.sync('packages/*/package.json');
    packages.forEach(pkg => {
      const content = JSON.parse(fs.readFileSync(pkg, 'utf-8'));
      expect(content.name).toBeDefined();
      expect(content.version).toBeDefined();
    });
  });
 
  test('affected detection works correctly', async () => {
    const affected = await getAffectedPackages('origin/main', 'HEAD');
    expect(Array.isArray(affected)).toBe(true);
  });
});

Future Outlook

Monorepo CI/CD tooling continues to evolve rapidly. Turborepo is adding features like incremental builds and better caching. Nx is expanding beyond JavaScript with plugins for Go, Rust, and Java. The trend toward remote caching and distributed execution will continue, making monorepo CI/CD more efficient and cost-effective.

The integration of AI for optimizing build order and predicting affected packages is an emerging trend. Tools will become smarter about which packages to build and test, further reducing CI times.

Cache Invalidation Strategies

Effective cache invalidation is critical for monorepo CI/CD performance. Use content-based hashing to generate cache keys that change only when the source code or dependencies change. Scope caches to specific projects and their dependencies to avoid false cache hits from unrelated changes. Implement a distributed cache (like Turborepo's remote cache or Nx Cloud) to share build artifacts across CI runners and developer machines. Monitor cache hit rates and adjust granularity to balance cache freshness against build speed.

Monorepo Dependency Graph Analysis

Visualize and analyze your monorepo's dependency graph to optimize CI/CD pipelines. Tools like Nx, Turborepo, and Bazel provide dependency graph visualization that shows which projects depend on which others. Use this graph to determine the minimum set of projects that need rebuilding when a file changes. Identify circular dependencies that prevent effective caching and parallelization. Export the dependency graph as a CI artifact for debugging pipeline decisions. Use graph analysis to plan project boundaries and identify refactoring opportunities that would improve build parallelism.

Conclusion

Effective CI/CD for monorepos requires a combination of intelligent caching, affected detection, and parallel execution. By leveraging tools like Turborepo and Nx, you can build CI/CD pipelines that scale with your monorepo while keeping build times and costs manageable.

Key takeaways:

Cache aggressively - Remote caching can reduce build times by 90%+
Build only what changed - Affected detection prevents unnecessary work
Parallelize everywhere - Run independent tasks concurrently
Monitor and optimize - Track CI metrics and continuously improve
Use purpose-built tools - Turborepo and Nx understand monorepo dependencies
Plan for scale - Design your CI/CD to handle growing numbers of packages

Start with Turborepo or Nx, configure remote caching, and implement affected detection. As your monorepo grows, add parallel execution and optimize based on your specific bottlenecks.

Minh Vo

Slaying code & making it lit fr fr 🔥 tagline