executor: No such file or directory: '/var/run/user/0/52_54_00_11_22_01.sock.lock'
I see this crash loop in executor sometimes when running the integration testing locally. It doesn't happen every time, and I've been unable to find a consistent way to reproduce it
Oct 17 20:20:46 vivian executor[717]: 2022-10-17 20:20:46,785 [ExecutorThread-52:54:00:11:22:01] [WARNING] _report: SergentHartman/52:54:00:11:22:01 - Registration failed with status UNKNOWN. Retrying in 2 second(s)
Oct 17 20:20:46 vivian executor[717]: 10.0.2.2 - - [17/Oct/2022 20:20:46] "GET /api/v1/machines HTTP/1.1" 200 -
Oct 17 20:20:48 vivian executor[717]: Traceback (most recent call last):
Oct 17 20:20:48 vivian executor[717]: File "/usr/lib/python3.10/site-packages/valve_gfx_ci/executor/server/dut.py", line 514, in run
Oct 17 20:20:48 vivian executor[717]: self.sergent_hartman.execute_next_task(stop_event=self.stop_event)
Oct 17 20:20:48 vivian executor[717]: File "/usr/lib/python3.10/site-packages/valve_gfx_ci/executor/server/dut.py", line 290, in execute_next_task
Oct 17 20:20:48 vivian executor[717]: proc = self.machine.start_job(self._next_task(callback_port=local_port))
Oct 17 20:20:48 vivian executor[717]: File "/usr/lib/python3.10/site-packages/valve_gfx_ci/executor/server/dut.py", line 495, in start_job
Oct 17 20:20:48 vivian executor[717]: raise ValueError("The job process failed to start")
Oct 17 20:20:48 vivian executor[717]: ValueError: The job process failed to start
Oct 17 20:20:48 vivian executor[717]: 10.0.2.2 - - [17/Oct/2022 20:20:48] "GET /api/v1/machines HTTP/1.1" 200 -
Oct 17 20:20:49 vivian executor[717]: 2022-10-17 20:20:49,790 [ExecutorThread-52:54:00:11:22:01] [INFO] _next_task: SergentHartman/52:54:00:11:22:01 - Try registering the machine
Oct 17 20:20:50 vivian executor[717]: 10.0.2.2 - - [17/Oct/2022 20:20:50] "GET /api/v1/machines HTTP/1.1" 200 -
Oct 17 20:20:50 vivian executor[1139]: Traceback (most recent call last):
Oct 17 20:20:50 vivian executor[1139]: File "/usr/bin/executor", line 8, in <module>
Oct 17 20:20:50 vivian executor[1139]: sys.exit(run())
Oct 17 20:20:50 vivian executor[1139]: File "/usr/lib/python3.10/site-packages/valve_gfx_ci/executor/server/init.py", line 29, in run
Oct 17 20:20:50 vivian executor[1139]: run(config_f=args.config, socket_path=args.socket)
Oct 17 20:20:50 vivian executor[1139]: File "/usr/lib/python3.10/site-packages/valve_gfx_ci/executor/server/executor.py", line 780, in run
Oct 17 20:20:50 vivian executor[1139]: socket_lock = open(f"{socket_path}.lock", "w")
Oct 17 20:20:50 vivian executor[1139]: FileNotFoundError: [Errno 2] No such file or directory: '/var/run/user/0/52_54_00_11_22_01.sock.lock'