[REC-65] Internal server error (500) when pushing to internal registry

Tue, 12 Nov 2019 13:21:55 +0000

Frequently, a response "received unexpected HTTP status: 500 Internal Server Error" is received when attempting to push to internal registry, e.g.

docker push registry.kube-system.svc.rec.io:5555/bluval/porter:1.0
fe9a8b4f1dcc: Preparing
fe9a8b4f1dcc: Pushed
received unexpected HTTP status: 500 Internal Server Error

The image can be found in registry but still the push fails occasionally:

[cloudadmin@controller-1 j]$ docker push registry.kube-system.svc.rec.io:5555/bluval/porter:1.0
The push refers to repository [registry.kube-system.svc.rec.io:5555/bluval/porter]
2475b11e4b59: Layer already exists 
44ec04fa5fba: Layer already exists 
35b17a8c3a93: Layer already exists 
1.0: digest: sha256:824f041b2757f8da2980092ace0991b4b8c6689e76c3b94f677521f7157bc7b8 size: 943
[cloudadmin@controller-1 j]$ docker push registry.kube-system.svc.rec.io:5555/bluval/porter:1.0
The push refers to repository [registry.kube-system.svc.rec.io:5555/bluval/porter]
2475b11e4b59: Layer already exists 
44ec04fa5fba: Layer already exists 
35b17a8c3a93: Layer already exists 
1.0: digest: sha256:824f041b2757f8da2980092ace0991b4b8c6689e76c3b94f677521f7157bc7b8 size: 943
[cloudadmin@controller-1 j]$ docker push registry.kube-system.svc.rec.io:5555/bluval/porter:1.0
The push refers to repository [registry.kube-system.svc.rec.io:5555/bluval/porter]
2475b11e4b59: Layer already exists 
44ec04fa5fba: Layer already exists 
35b17a8c3a93: Layer already exists 
received unexpected HTTP status: 500 Internal Server Error
[cloudadmin@controller-1 j]$

Log excerpt from a registry pod:

Sep 19 11:31:59 controller-1 dockerd[2869]: time="2019-09-19T11:31:59.427553942+03:00" level=info msg="Attempting next endpoint for push after error: received unexpected HTTP status: 500 Internal Server Error"
Sep 19 12:30:22 controller-1 dockerd[2869]: time="2019-09-19T12:30:22.795423817+03:00" level=warning msg="failed to upload schema2 manifest: received unexpected HTTP status: 500 Internal Server Error - falling back to schema1"
Sep 19 12:30:22 controller-1 dockerd[2869]: time="2019-09-19T12:30:22.842725367+03:00" level=info msg="Attempting next endpoint for push after error: received unexpected HTTP status: 500 Internal Server Error"
Sep 19 12:34:19 controller-1 dockerd[2869]: time="2019-09-19T12:34:19.448281710+03:00" level=warning msg="failed to upload schema2 manifest: received unexpected HTTP status: 500 Internal Server Error - falling back to schema1"
Sep 19 12:34:22 controller-1 dockerd[2869]: time="2019-09-19T12:34:22.071897919+03:00" level=info msg="Attempting next endpoint for push after error: received unexpected HTTP status: 500 Internal Server Error"
Sep 19 12:36:06 controller-1 dockerd[2869]: time="2019-09-19T12:36:06.332029718+03:00" level=info msg="Attempting next endpoint for push after error: received unexpected HTTP status: 500 Internal Server Error"
Sep 19 12:39:53 controller-1 dockerd[2869]: time="2019-09-19T12:39:53.426861752+03:00" level=warning msg="failed to upload schema2 manifest: received unexpected HTTP status: 500 Internal Server Error - falling back to schema1"
Sep 19 12:39:53 controller-1 dockerd[2869]: time="2019-09-19T12:39:53.867425401+03:00" level=info msg="Attempting next endpoint for push after error: received unexpected HTTP status: 500 Internal Server Error"
Sep 19 12:41:21 controller-1 dockerd[2869]: time="2019-09-19T12:41:21.114190515+03:00" level=warning msg="failed to upload schema2 manifest: received unexpected HTTP status: 500 Internal Server Error - falling back to schema1"
Sep 19 12:41:21 controller-1 dockerd[2869]: time="2019-09-19T12:41:21.180209597+03:00" level=info msg="Attempting next endpoint for push after error: received unexpected HTTP status: 500 Internal Server Error"

Product release:

[cloudadmin@controller-1 j]$ cat /etc/product-release
release=TA-1
build=126

Akraino JIRA

[REC-65] Internal server error (500) when pushing to internal registry