- 17 Jan, 2023 1 commit
-
-
Ben Avison authored
!30 introduced its own problem: the `|` character has special meaning in YAML, so the script lines that introduced it (as part of the bash `||` operator) needed to be quoted. The result was that none of the nightly superproject pipelines have run since last month!
-
- 20 Dec, 2022 2 commits
-
-
Ben Avison authored
This re-uses the same script from the Runner machine as is used for the reworked `cppcheck` job. In an analogous way, this enables us to more easily detect whitespace errors introduced in new work, without first requiring the removal of any examples from pre-existing versions. Due to the shared framework, this job has many of the same facilities as the `cppcheck` job. You can get more information about its deductions by setting `VERBOSE` to 1 when launching a pipeline manually. You can also suppress diagnostics by commit, name, file or line; however a different variable name is (re)used for this job: WHITESPACE_WHITELIST. Rather than using a custom regular expression to detect whitespace errors, this job now uses `git diff-tree --check`, so it picks up all the same issues as `git` itself does.
-
Ben Avison authored
!29 didn't account for the possibility that the list of submodules where the default branch and the CrossCompilationSupport branch had diverged was empty. Also apply the same fix for the case where the list of submodules on the CrossCompilationSupport branch is empty, as one day that will (hopefully) be the case also. Should fix BuildHost, iMx6 and OMAP5 pipelines.
-
- 16 Dec, 2022 1 commit
-
-
Ben Avison authored
Softload jobs (and deployment jobs, and when they're re-enabled, ROM jobs) in submodule pipelines save time by using the build trees which are regenerated overnight by each superproject pipeline. The build trees are packaged as a zip file, and contain the whole `RiscOS` directory, but not other top-level files such as `.gitlab-ci.yml`, nor the `.git` directory, in order to save space. It is desirable to be able to access the version numbers of the components from this build tree in the submodule job log. This is expected to be helpful in debugging build issues, but would also form a record of what constituted "top of tree" at the time of the build: whilst this might be obvious at the time that the job runs, it could well be less so when looking back at the job log days, weeks or even longer afterwards. However, since the build zip file doesn't contain git information, this isn't directly accessible from the submodule pipeline. Instead, generate a listing of version information (using the same `git describe` command as is used when generating `prodcommit` logs) during the superproject pipeline, and print a link to it during the submodule job (with the actual URLs involved being conveyed by a file called `BuildInfo` which is inserted into the zip file). Take the opportunity to also dump information about the use of CrossCompilationSupport branches, to assist with diagnosing issues related thereto.
-
- 01 Jul, 2022 2 commits
-
-
Ben Avison authored
Sometimes cppcheck flags up issues that don't exist, due to bugs in cppcheck itself. As a compromise between inserting inline suppression comments in such cases (which is undesirable because we'd probably want to remove them again later when cppcheck is fixed) and globally suppressing the diagnostics triggered by the bugs (which is undesirable because that renders those disgnostics unable to flag true bugs), we will be introducing a centralised list of false positive diagnostics, tuned to suppress the false positives as precisely as is permitted by cppcheck (short of specifying the line numbers, which are subject to change from revision to revision). This will take some time to develop. In the meantime, as a stopgap solution, hard-code additional suppressions for select components into their included YAML files.
-
Ben Avison authored
At a later date, there may be a mechanism to permit components to opt back in to these diagnostics, but for now disable them for all components.
-
- 29 Jun, 2022 2 commits
-
-
Ben Avison authored
This involves introducing the concept of opt-in jobs, controlled by the ENABLE_JOBS environment variable, analogous to the pre-existing SUPPRESS_JOBS variable. A large bulk of components don't currently build with the GNU tools, due to deficiencies in the shared makefiles and the fact that we don't currently autobuild ELF versions of libraries for them to link against. Having softload_gnu be off by default saves us from having to explicitly disable it in dozens of .gitlab-ci.yml files only to have to come back later to re-enable it once the build infrastructure becomes available.
-
Ben Avison authored
The GitLab runner machine previously ran Ubuntu 18.04, which featured cppcheck 1.82. Some planned enhancements to the CI scripts required a newer version of cppcheck, so we have upgraded it to Ubuntu 20.04, which has cppcheck 1.90. However, the format of the diagnostics printed by cppcheck has changed in 1.90, so our code that parsed them needs adapting to match.
-
- 16 Jun, 2022 1 commit
-
-
Ben Avison authored
In other words, permit files (previous false positives primarily expected to be BASIC ones) within the Resources subdirectory of each component to lack a copyright/licence notice. Simplify the way we similarly omit the .git subdirectory from the test whilst we're at it.
-
- 16 Feb, 2022 1 commit
-
-
Ben Avison authored
The exceptions list is processed into a form that excludes the given files or directories from the `find` command that searches for files to scan. However, a missing `-print` option caused the files to be included anyway (due to the default action of the `find` command).
-
- 07 Feb, 2022 1 commit
-
-
Ben Avison authored
This job scans all the source files in a project, attempting to ensure they each have a header identifying the licence terms under which the file may be distributed. The job prints a list of the files that fail its check to the job log, and exits with failure status if the list is non-empty. There is no standard format for licence headers, however one thing they do have in common is that they all feature a copyright notice. Upon scanning all the source files hosted by ROOL, very few files failed this test, and those that did either genuinely lacked licence terms, or had unusual ones where the author had waived copyright and placed the code into the public domain. A few files used hard spaces around the word "Copyright" so the test uses `grep -w` which considers hard spaces to constitute a word boundary. Although not (yet?) in use by any of our sources, the ISO SPDX standard advocates using `SPDX-FileCopyrightText` in source files in place of a copyright declaration. Therefore, this patch pre-emptively permits this as an alternative. To cater for waived-copyright cases, a `.gitlab-ci.yml` file can define `COPYRIGHT_WHITELIST` to specify files for which this test should be skipped. This follows the same file naming pattern as `WHITESPACE_WHITELIST`.
-
- 06 Feb, 2022 3 commits
-
-
Ben Avison authored
Sometimes, a job isn't expected to pass (most often `softload_gnu`, because certain components don't support non-Norcroft toolchains). By defining this variable in a `.gitlab-ci.yml` file, we can force any of the jobs to be skipped. This only applies to submodule pipelines: it is not sensible to skip a build phase for a superproject pipeline.
-
Ben Avison authored
We're still some way out from having these jobs pass - estimated at least 70 MRs until ROMs cross-compile. In the meantime, the main effect of these jobs will be to cause the overall pipeline status to be amber (i.e. passed with warnings) which undermines its usefulness as an at-a-glance indicator of whether the other jobs all passed. In theory, their job logs could be inspected by human reviewers to check that the MR doesn't make a cross-compile build any worse than it already is, but I'm not sure that this is going to be the case in practice, or whether this overrides the benefit of being able to quickly see that the whole pipeline passes. So, comment out the lines that generate that part of the YAML for the time being. Reinstate them once the respective cross-compile builds can complete.
-
Ben Avison authored
-
- 18 Jan, 2022 1 commit
-
-
Ben Avison authored
Now that we've had chance to see how this job functions with real world MRs, rather than just with artificial test cases, it's apparent that it is far too sensitive and generates far too many false positives to be useful as it currently stands. Problems include: * When two nearby groups of lines are swapped over, the underlying `-b` and non-`-b` `git diff` commands can make different decisions about which group is fixed and which was removed from one place and inserted into another. When our job script finds those lines unchanged in the `-b` diff and changed in the non-`-b` diff, it incorrectly assumes the presence of a whitespace change. * Even minor indentation changes get flagged up as an error. Even something as simple as changing bar(); to if (foo) bar(); * In order to pass `head_whitesp` on a component that hasn't been touched recently, it's necessary to strip the trailing whitespace from the `VersionNum` file even though once the MR is accepted, this will happen automatically. Yet doing this gets picked up as a whitespace error change by `merge_whitesp`, so it's impossible to simultaneously pass all jobs! We could just throw away this job, but there is still an important case that ideally we do want to catch: where a source file was originally imported from an upstream project which includes whitespace errors - meaning that we want to preserve them to keep tracking the upstream project easy - and when one of our contributors has accidentally stripped all trailing whitespace from the file. We want this to be caught at the review stage - better still by the contributor, before any reviewers have to get involved. And a CI job can still serve this purpose well. To distinguish these different cases, we can take into account the fact that the accidental space-stripping scenario will typically alter many lines a long way away from where the new contributor was working. Thus, the job script is tightened as follows: * Lines "nearby" a diff `-b` hunk are discounted, where "nearby" is arbitrarily defined as within +/- 20 lines. * At least one line in the "removed" half of the non-`-b` hunk must have contained a whitespace error. It's impossible to exactly match up the individual "removed" and "added" lines in the general case, because there are often different numbers of lines in each half of a hunk. * A threshold number of flagged lines (currently 10) must be detected across the whole project before the test is considered to have failed, although we still print any lines that we find. * The test for introduction of whitespace errors is removed, since this duplicates the functionality of the `head_whitesp` job.
-
- 17 Dec, 2021 1 commit
-
-
Ben Avison authored
When working on an unrelated issue, it became clear that the reason why previously only the `merge_log` and `merge_whitesp` jobs appeared in detached pipelines (the ones that relate to an open MR) was really because they include a `rules` section. It is as though in the absence of a `rules` section, a default one applies, which adds the job to the pipeline only if that pipeline was due to an update of a branch or tag ref. When `rules` was present for a job, it overrides the default, and is evaluated irrespective of the the pipeline trigger. Now, it's useful to have all the jobs present in detached pipelines. The latest detached pipeline state, and its associated artifactes, are displayed at the top of the "Overview" tab of each MR page, and a new one can be easily triggered from the "Run pipeline" button at the top of the MR page's "Pipelines" tab, without having to navigate to the contributor's fork project (which may not even be public). It's also a problem that the `cleanup` job wasn't being run for detached pipelines, since over time that risks filling up the disc space on the runner machine. It turns out all of this can be implemented simply by adding a `workflow` section which defines a new default set of rules, which adds jobs to detached pipelines as well as ones for branches and tags.
-
- 01 Sep, 2021 1 commit
-
-
Ben Avison authored
Unlike all other components with CrossCompilationSupport branches, HostFS had two, and the correct one needed to be selected in place of the HAL branch. Now that its MRs have been merged, this can be removed.
-
- 08 Jun, 2021 1 commit
-
-
Ben Avison authored
From RiscOS/Env!15, we no longer use aliases, so there's no need to tell non-interactive shells to expand them.
-
- 03 Jun, 2021 2 commits
-
-
Ben Avison authored
-
Ben Avison authored
Various CI jobs run make on all the components within a project. The COMPONENT and TARGET were correctly set, but we neglected to change into the appropriate subdirectory first (where applicable).
-
- 01 Jun, 2021 2 commits
-
-
Ben Avison authored
These jobs were trying to fetch a build tree from a non-existent superproject `Products/IOMD32` when they should have referenced `Products/IOMDHAL`.
-
Ben Avison authored
On the Runner machine, each fork of each project gets its own directory, which is left in the state which the latest job for whatever pipeline was most recently run on it. This typically will include a large number of object and binary files, which are of no use to anyone (anything of interest will already have been packaged up into an artifact and uploaded to the main GitLab server). Address this by adding an additional job to the end of each pipeline, which does a `git clean` (it's worth leaving these in place to reduce the bandwidth requirement when doing a `git fetch` when a pipeline is next run for the fork). The Runner machine also stores cache files for each fork of each project, at least for jobs that complete fully successfully (and there are an increasing number of these). The way our pipelines use caches, these are tarballs of pre-built source trees for each target platform. These take up less space than the temporary files noted above, but will now become the dominant user of disc space. To address this, abandon use of GitLab Runner's own cache facility, and take advantage of the fact that shell executors actually have visibility of the gitlab-runner user's whole home directory to maintain a single, cached version of each tarball, shared across all forks of all projects. This is stored within ~/cache, but namespaced under ~/cache/common to avoid collisions with any users of GitLab Runner's cache facility.
-
- 17 May, 2021 1 commit
-
-
Ben Avison authored
Once the following have been merged: * Products/BCM2835!6 * Products/BuildHost!2 * Products/Disc!6 * Products/iMx6!2 * Products/IOMDHAL!3 * Products/OMAP3!3 * Products/OMAP4!3 * Products/OMAP5!3 * Products/Titanium!3 * Products/Tungsten!3 then we can have the pipelines for submodules fetch their source trees from the central projects rather than my forks.
-
- 14 May, 2021 1 commit
-
-
Ben Avison authored
For these versions, "git submodule update --remote" will fail for any submodules that don't specify a branch in .gitmodules, and which are currently checked out on a remote tracking branch that tracks a remote other than "origin". For our pipelines, this means almost any submodule which has a CrossCompilationSupport branch which has not yet been merged. Work around it by first doing "git submodule update" without "--remote", which will put all submodules into detacehd HEAD state.
-
- 23 Dec, 2020 1 commit
-
-
Ben Avison authored
Resolves bug introduced in commit 9c61e2f0.
-
- 11 Dec, 2020 2 commits
-
-
Ben Avison authored
If the following happens: * a GitLab runner recursively clones a superproject * one or more submodules has changes upstream and the submodule references in the superproject have been updated to point to it * a new pipeline is launched for the new superproject revision then, while the runner would fetch changes to the superproject, it wasn't doing so for the submodules. Ironically, we do both `git submodule update` and `git submodule update --remote` in different pipeline stages and the latter includes an implicit submodule fetch - but that was the later stage which we don't get as far as. To fix this, while retaining the pipeline stage order, include an explicit fetch in the earlier stage (and remove the implicit fetch in the later one, since that now merely wastes time).
-
Ben Avison authored
This one is a special case where the superproject name mismatches the corresponding Env (and thus Components) file. Our own CI script also didn't previously support the *removal* of an autogenerated YAML file. This should be a rare occurrence, but it's best to automate this also. Also support force-pushing to the submodule; this will be required in most cases where the pipeline was triggered by a force-push to *this* project.
-
- 25 Nov, 2020 1 commit
-
-
Ben Avison authored
-