https://jira.akraino.org This file is an XML representation of an issue en-us 9.4.5 940005 11-04-2023 https://jira.akraino.org/browse/IEC-47 Integrated Edge Cloud <p>There are two kind of failures. It seems both cases has the same issue since there is no response from voltha:</p> <p><a href="https://jenkins.akraino.org/view/iec/job/iec-type2-install-seba_on_arm-fuel-baremetal-ubuntu1804-daily-master/20/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.akraino.org/view/iec/job/iec-type2-install-seba_on_arm-fuel-baremetal-ubuntu1804-daily-master/20/console</a><br/> <a href="https://jenkins.akraino.org/view/iec/job/iec-type2-install-seba_on_arm-fuel-baremetal-ubuntu1804-daily-master/21/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.akraino.org/view/iec/job/iec-type2-install-seba_on_arm-fuel-baremetal-ubuntu1804-daily-master/21/console</a></p> IEC-47

[IEC][SEBA] CI tests fail with voltha not healthy

Bug Medium Done Done Catalin Iova Catalin Iova Fri, 22 May 2020 08:03:27 +0000 Wed, 27 May 2020 07:03:55 +0000 Wed, 27 May 2020 07:03:55 +0000 0 1 <p>Workaround occurs on <a href="https://jenkins.akraino.org/view/iec/job/iec-type2-install-seba_on_arm-fuel-baremetal-ubuntu1804-daily-master/32/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.akraino.org/view/iec/job/iec-type2-install-seba_on_arm-fuel-baremetal-ubuntu1804-daily-master/32/console</a> and voltha recovered successfully.</p> <p>First workaround applied: <a href="https://github.com/iecedge/automation-tools/commit/d63aedab1c7d70c70908228a0abbacc9898b5ca4" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/iecedge/automation-tools/commit/d63aedab1c7d70c70908228a0abbacc9898b5ca4</a></p> <p>It seems voltha coordinator remains in a state that only renew session and check its leader membership. The worker cannot get a stable voltha leader and assign the core stores keys leading to worker not finishing its initialization. This is happening after huge delays in responses from etcd and also after etcd error: "etcdserver: request timed out".<br/> The etcd error signaled few times is because the etcd leader loss the leadership of the etcd cluster during cluster initialization and a new leader is assigned. This etcd cluster leader election process can take 2-3 to tens of seconds and can happen after the voltha core created its session/lease in previous etcd leader.<br/> The voltha coordinator, leader and worker should have a recovery mechanism implemented but seems this is not fully implemented.<br/> This is also signaled on <a href="https://jira.opencord.org/browse/VOL-339" class="external-link" target="_blank" rel="nofollow noopener">https://jira.opencord.org/browse/VOL-339</a> and sub-tasks for when voltha is used with consul instead of etcd but it is a similar handling.<br/> It is difficult and come with great risks to implement the fix inside voltha core so there is a workaround available.</p> <p>Possible workarounds:</p> <p>1. After voltha and etcd-cluster are installed and ready, check voltha is healthy. If no response or voltha not healthy after two minutes, then delete vcore-0 pod. In this way voltha core is restarted after etcd-cluster is ready.</p> <p>2.etcd-cluster shall be installed before and independently by voltha pods. Voltha installation shall also wait for etcd-cluster to be ready. This shall be done by updating the voltha helm-chart too which is not a preferred workaround.</p> Development Rank 0|i0046k: