Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
CI_Source CI_Source
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Labels
    • Labels
  • Merge requests 3
    • Merge requests 3
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Operations
    • Operations
    • Metrics
    • Environments
  • Analytics
    • Analytics
    • CI/CD
    • Repository
    • Value Stream
  • Members
    • Members
  • Activity
  • Graph
  • Jobs
  • Commits
Collapse sidebar
  • Support
  • CI_SourceCI_Source
  • Merge requests
  • !6

Merged
Created Jun 01, 2021 by Ben Avison@bavisonDeveloper

Reduce disc space requirements for GitLab Runner server

  • Overview 0
  • Commits 2
  • Pipelines 2
  • Changes 1

On the Runner machine, each fork of each project gets its own directory, which is left in the state which the latest job for whatever pipeline was most recently run on it. This typically will include a large number of object and binary files, which are of no use to anyone (anything of interest will already have been packaged up into an artifact and uploaded to the main GitLab server). Address this by adding an additional job to the end of each pipeline, which does a git clean (it's worth leaving these in place to reduce the bandwidth requirement when doing a git fetch when a pipeline is next run for the fork).

The Runner machine also stores cache files for each fork of each project, at least for jobs that complete fully successfully (and there are an increasing number of these). The way our pipelines use caches, these are tarballs of pre-built source trees for each target platform. These take up less space than the temporary files noted above, but will now become the dominant user of disc space. To address this, abandon use of GitLab Runner's own cache facility, and take advantage of the fact that shell executors actually have visibility of the gitlab-runner user's whole home directory to maintain a single, cached version of each tarball, shared across all forks of all projects. This is stored within ~/cache, but namespaced under ~/cache/common to avoid collisions with any users of GitLab Runner's cache facility.

Assignee
Assign to
Reviewer
Request review from
None
Milestone
None
Assign milestone
Time tracking
Source branch: DiscSpaceReduction