Skip to content

utils: Add retry logic in shell.download

Rubén Gonzalez requested to merge rgonzalez/cerbero:gsthf_download_retry into main

The tries option in wget was not working. The retry-connrefused, 'retry-on-host-error' and retry-on-http-error options are mandatory to retry correctly, only with the tries option is not enough.

Before:

$ wget http://httpbin.org/status/404 --tries=4 --timeout=10.0 --progress=dot:giga
--2021-10-08 11:47:22--  http://httpbin.org/status/404
Resolving httpbin.org (httpbin.org)... 3.209.149.47, 54.156.165.4, 54.159.86.231, ...
Connecting to httpbin.org (httpbin.org)|3.209.149.47|:80... connected.
HTTP request sent, awaiting response... 404 NOT FOUND
2021-10-08 11:47:22 ERROR 404: NOT FOUND.

After:

$ wget http://httpbin.org/status/404 --tries=4 --retry-connrefused --retry-on-host-error --retry-on-http-error=404,429,503,504 --timeout=10.0 --progress=dot:giga
--2021-10-08 11:48:10--  http://httpbin.org/status/404
Resolving httpbin.org (httpbin.org)... 3.215.162.201, 54.159.86.231, 54.156.165.4, ...
Connecting to httpbin.org (httpbin.org)|3.215.162.201|:80... connected.
HTTP request sent, awaiting response... 404 NOT FOUND
Retrying.

--2021-10-08 11:48:12--  (try: 2)  http://httpbin.org/status/404
Reusing existing connection to httpbin.org:80.
HTTP request sent, awaiting response... 404 NOT FOUND
Retrying.

--2021-10-08 11:48:14--  (try: 3)  http://httpbin.org/status/404
Reusing existing connection to httpbin.org:80.
HTTP request sent, awaiting response... 404 NOT FOUND
Retrying.

--2021-10-08 11:48:17--  (try: 4)  http://httpbin.org/status/404
Reusing existing connection to httpbin.org:80.
HTTP request sent, awaiting response... 404 NOT FOUND
Giving up.

But the retry-on-http-error requeres version 1.19.1. This version is very new and it is not included in all supported OSs.

Final solution was implementing the retry mechanism in cerbero to support all versions of wget, and also curl and urllib2.

Merge request reports