Skip to content

Optimize driver-trace-stats

Guilherme Gallo requested to merge gallo/mesa-performance-tracking:influx-opt into master

This Merge Request (MR) introduces several key improvements to the ETL process, threading utilities, and CI dependency management.

  1. ETL Process Improvements: The ETL process has been updated to use the ThreadPool for processing pipelines. This change allows for parallelized downloading of artifacts for each trace performance job, effectively eliminating the bottleneck in the process. Additionally, InfluxDB's write API has been updated to use batching, improving its performance.

  2. Threading Utilities: A new utility, SmartThreadPoolExecutor, has been introduced to enhance thread management. This utility automatically re-raises exceptions that occur inside a thread, making exception handling more transparent and manageable. It also includes a thread_exception_handler decorator for more fine-grained control over exception-protected operations.

  3. CI Dependency Management: Changes to the requirements.txt file, both at the root and in subprojects, now trigger jobs, ensuring that all Python scripts and subprojects maintain their dependencies up-to-date. Moreover, missing rules for dashboard_exporter in the CI have been added.

  4. Memory Management: Generators have been adopted in preference to lists in driver-stats/gitlab to reduce memory footprint during extensive thread execution.

  5. Dashboard Fixes: Fix datalinks between panels, fix frame timings timestamps to merge time to origin/main, and fix FPS minimum to zero.

Before After
image image
.5 FPS difference seems huge Now it is visible, but not overrated
image image
Real huge variance before. After: Higher relative differences are still very visible

The changes in this MR aim to enhance performance, improve memory management, streamline CI dependency handling, and provide better control over exception handling during multithreaded operations.

Edited by Guilherme Gallo

Merge request reports