review job retries and improve the monitor loop
Exclude jobs without artifacts from retry flake candidates
The procedure to retry jobs included the retry on jobs that didn't produce artifacts. Those without artifacts have a retry procedure right before, so they were retried 3 times. Also, the recently added filter to avoid retries on stuck jobs, was not avoiding a retry in this step. Those two things together increase the time of the step unnecessarily.
Pipeline jobs trigger, filter to only manual
The filtering on the jobs, to evaluate if they need to be triggered, can have one filter moved up. When it is obtained a generator for the pipeline jobs, we can have there already only jobs in manual status. This will reduce to evaluating created jobs, or even running or completed jobs.
Use a pool to trigger jobs
Use a pool of threads to delegate the triggering of the manual jobs. Each of those triggers are independent so they don't need to wait for each other, but wait all of them are triggered before continue.
Skip reporting about non-interesting jobs
When refreshing the jobs' status in the pipeline, exclude the jobs that aren't considered build or test jobs in the current uprev. So, then we reduce noise in the logs.
Use a pool to refresh the job status
The refresh for each of the job status in the pipeline in progress is a practical candidate to convert to a concurrent computation. Once all are refreshed, it can continue reporting with this information.
Consider pipeline started when there aren't jobs to be triggered
When the tool is waiting for the pipeline to start, consider it started when there aren't more build jobs to trigger with the pipeline in manual status.