Draft: Introducing async for ci-collate patch
On draft
Needs !31
Description
This major update brings asynchronous operations to ci-collate patch
, enhancing its performance and efficiency in handling network requests and file operations. The async_main
entry point is a significant addition, enabling asynchronous execution of the main functionality, leveraging Python's asyncio
and aiohttp
for non-blocking IO operations. This enhancement is particularly beneficial for fetching and processing artifacts concurrently, critical for dealing with multiple jobs in large CI pipelines.
Moreover, the update introduces a new module, async_utils.py
, which contains utilities for asynchronous retrying of network requests and artifact downloading. This module employs advanced retry strategies to handle transient network issues and GitLab API rate limits, ensuring robust artifact fetching.
Another key feature of this update is introducing the 'is_dirty' property in both ExpectationFile
and ExpectationsProcessor
classes. This property tracks whether any changes (additions or removals) have been made to the expectation files, enabling more intelligent processing and updating of files only when necessary. This change provides more precise insights into the state of expectation files, letting it easily print the affected jobs.
The pipeline and job handling have been adapted to support asynchronous operations, with changes to the CollatePipeline
and CollateJob
classes. These adaptations include asynchronous methods for expectation updates and artifact downloads, fully leveraging the new asynchronous utilities for efficient parallel processing.
This update significantly improves ci-collate, particularly for large-scale CI/CD environments where handling numerous jobs and artifacts concurrency is essential.
Performance Comparison
Pipeline 1107132
Source | Command | Mean [s] | Min [s] | Max [s] |
---|---|---|---|---|
This branch | python /var/home/guilherme/.local/bin/async-ci-collate --pipeline-url https://gitlab.freedesktop.org/mesa/mesa/-/pipelines/1107132 patch |
42.811 | 34.988 | 62.331 |
!31 MR | python /var/home/guilherme/.local/bin/ci-collate --pipeline-url https://gitlab.freedesktop.org/mesa/mesa/-/pipelines/1107132 patch |
118.440 | 76.506 | 154.863 |
origin/main | ci-collate --namespace mesa --project mesa patch 1107132 |
258.006 | 221.964 | 289.948 |
Pipeline 1120610
Source | Command | Mean [s] | Min [s] | Max [s] |
---|---|---|---|---|
This branch | python /var/home/guilherme/.local/bin/async-ci-collate --pipeline-url https://gitlab.freedesktop.org/mesa/mesa/-/pipelines/1120610 patch |
99.974 | 94.726 | 113.826 |
origin/main | ci-collate --namespace mesa --project mesa patch 1120610 |
513.000 | 433.126 | 632.704 |