HIP Error: Invalid Device Function When Running PyTorch Micro Benchmark
Brief summary of the problem:
I am trying to test out my PyTorch installation by running the PyTorch micro benchmarking script. I am consistently running into RuntimeError: HIP Error: invalid device function.
Hardware description:
- CPU: Ryzen 7 7700X
- GPU: Radeon RX 7900 XTX
- System Memory: 32GB DDR5 4800 MHz
- Display(s): AORUS FI32U (32", 3840x2160, 144Hz)
- Type of Display Connection: DP
System information:
- Distro name and Version: Ubuntu 22.04.3 LTS
- Kernel version: 6.2.0-34-generic
- Custom kernel: N/A
- AMD official driver version: "Radeon™ Software for Linux® version 23.20.00.48 for Ubuntu 22.04.3 HWE with ROCm 5.7"
How to reproduce the issue:
- Installed listed Linux AMD driver and ROCm 5.7 from https://www.amd.com/en/support/linux-drivers
- Installed PyTorch with pip wheels from https://rocm.docs.amd.com/projects/radeon/en/latest/docs/install/install-pytorch.html
- git clone https://github.com/ROCmSoftwarePlatform/pytorch-micro-benchmarking
- cd pytorch-micro-benchmarking
- python micro_benchmarking_pytorch.py --network resnet50 --compile
- RuntimeError: HIP Error: invalid device function.
Log files (for system lockups / game freezes / crashes)
Traceback (most recent call last):
File "/home/taccuser/pytorch-micro-benchmarking/micro_benchmarking_pytorch.py", line 405, in
main()
File "/home/taccuser/pytorch-micro-benchmarking/micro_benchmarking_pytorch.py", line 367, in main
run_benchmarking_wrapper(copy.deepcopy(args))
File "/home/taccuser/pytorch-micro-benchmarking/micro_benchmarking_pytorch.py", line 190, in run_benchmarking_wrapper
run_benchmarking(0, params)
File "/home/taccuser/pytorch-micro-benchmarking/micro_benchmarking_pytorch.py", line 271, in run_benchmarking
inp = torch.randn(batch_size, 3, 224, 224, device="cuda")
RuntimeError: HIP error: invalid device function
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing HIP_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_HIP_DSA
to enable device-side assertions.