Skip to content

Draft: Introducing async for ci-collate patch

Guilherme Gallo requested to merge gallo/ci-collate:async-patch into main

On draft

Needs !31

Description

This major update brings asynchronous operations to ci-collate patch, enhancing its performance and efficiency in handling network requests and file operations. The async_main entry point is a significant addition, enabling asynchronous execution of the main functionality, leveraging Python's asyncio and aiohttp for non-blocking IO operations. This enhancement is particularly beneficial for fetching and processing artifacts concurrently, critical for dealing with multiple jobs in large CI pipelines.

Moreover, the update introduces a new module, async_utils.py, which contains utilities for asynchronous retrying of network requests and artifact downloading. This module employs advanced retry strategies to handle transient network issues and GitLab API rate limits, ensuring robust artifact fetching.

Another key feature of this update is introducing the 'is_dirty' property in both ExpectationFile and ExpectationsProcessor classes. This property tracks whether any changes (additions or removals) have been made to the expectation files, enabling more intelligent processing and updating of files only when necessary. This change provides more precise insights into the state of expectation files, letting it easily print the affected jobs.

The pipeline and job handling have been adapted to support asynchronous operations, with changes to the CollatePipeline and CollateJob classes. These adaptations include asynchronous methods for expectation updates and artifact downloads, fully leveraging the new asynchronous utilities for efficient parallel processing.

This update significantly improves ci-collate, particularly for large-scale CI/CD environments where handling numerous jobs and artifacts concurrency is essential.

Performance Comparison

Pipeline 1107132

Source Command Mean [s] Min [s] Max [s]
This branch python /var/home/guilherme/.local/bin/async-ci-collate --pipeline-url https://gitlab.freedesktop.org/mesa/mesa/-/pipelines/1107132 patch 42.811 34.988 62.331
!31 MR python /var/home/guilherme/.local/bin/ci-collate --pipeline-url https://gitlab.freedesktop.org/mesa/mesa/-/pipelines/1107132 patch 118.440 76.506 154.863
origin/main ci-collate --namespace mesa --project mesa patch 1107132 258.006 221.964 289.948

Pipeline 1120610

Source Command Mean [s] Min [s] Max [s]
This branch python /var/home/guilherme/.local/bin/async-ci-collate --pipeline-url https://gitlab.freedesktop.org/mesa/mesa/-/pipelines/1120610 patch 99.974 94.726 113.826
origin/main ci-collate --namespace mesa --project mesa patch 1120610 513.000 433.126 632.704
Edited by Guilherme Gallo

Merge request reports