Skip to content

Improve CollatePipeline concurrency, thread safety and type hints

The current CollatePipeline.__job_selection is thread unsafe and has complicated type hints.

Split the job filtering functionality in to a different function _iter_jobs. When a caller only wants the jobs this new function can be used. This eliminates the need for typing.overload.

The __job_selection is NOT thread safe because it shares args_dict between the threads and each thread writes its own job_id to it. This means it is possible for job_id to be modified while other thread still uses it.

Instead make this map function to take a callable which should accept the CollateJob and int (job_id) as arguments. This makes type hints more accurate as the return type of passed callable can be made generic meaning the return type will change based on passed callable.

Make retried jobs requests run concuirrently. There was a TODO in the old code about making it concurrent.

Also rename the __job_selection to _map_jobs as that better describes its functionality.

Merge request reports

Loading