Akraino JIRA

Akraino JIRA https://jira.akraino.org This file is an XML representation of an issue en-us

9.4.5

940005

11-04-2023

[ICN-233] OpenNESS gap for Cross-Node communication https://jira.akraino.org/browse/ICN-233 Integrated Cloud Native NFV <p>Investigate Cross-Node (e.g. producer/consumer communication) gap for OpenNESS integration</p> ICN-233

OpenNESS gap for Cross-Node communication

Task High Done Done Chenjie Xu Huifeng Le Thu, 9 Jan 2020 02:54:02 +0000 Thu, 16 Jul 2020 00:22:50 +0000 Fri, 17 Jan 2020 08:01:20 +0000 0 2 <p>The gap analysis has been added to the Akraino ICN OpenNESS wiki page as following:</p> <p><a href="https://wiki.akraino.org/display/AK/OpenNESS+19.12+Investigation#OpenNESS19.12Investigation-Cross-Nodecommunication" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.akraino.org/display/AK/OpenNESS+19.12+Investigation#OpenNESS19.12Investigation-Cross-Nodecommunication</a></p> <p>By changing the common name and myURN of openvino consumer, openvino-cons-app and openvino-cons-app2 will use different version of docker image openvino-cons-app, and they both can run and consume the same producer. The logs are below:</p> <p><span class="error">[root@master consumer]</span># kubectl get pod<br/> NAME READY STATUS RESTARTS AGE<br/> openvino-cons-app 1/1 Running 0 111s<br/> openvino-cons-app2 1/1 Running 0 17s<br/> openvino-prod-app 1/1 Running 4 2d18h</p> <p><span class="error">[root@master consumer]</span># kubectl logs openvino-cons-app<br/> <span class="error">[setupvars.sh]</span> OpenVINO environment initialized<br/> 2020/01/13 02:22:53 OpenVINO Consumer Started<br/> 2020/01/13 02:22:53 WebSocket establishment successful<br/> 2020/01/13 02:22:53 Subscribed to notification: openvino-inference 1.0.0<br/> 2020/01/13 02:22:53 Subscribed to notification: terminate 1.0.0<br/> 2020/01/13 02:23:44 Rx notification – vehicle-detection-adas-0002 | CPU<br/> 2020/01/13 02:24:44 Rx notification – pedestrian-detection-adas-0002 | CPU</p> <p><span class="error">[root@master consumer]</span># kubectl logs openvino-cons-app2<br/> <span class="error">[setupvars.sh]</span> OpenVINO environment initialized<br/> 2020/01/13 02:24:21 OpenVINO Consumer Started<br/> 2020/01/13 02:24:41 WebSocket establishment successful<br/> 2020/01/13 02:24:46 Subscribed to notification: openvino-inference 1.0.0<br/> 2020/01/13 02:24:46 Subscribed to notification: terminate 1.0.0<br/> 2020/01/13 02:24:46 Rx notification – pedestrian-detection-adas-0002 | CPU</p> <p><span class="error">[root@master consumer]</span># kubectl describe pod openvino-cons-app | grep Image<br/> Image: openvino-cons-app:1.0<br/> Image ID: docker://sha256:b691b193662743554ab9c240ea5ed3dcb21765695fa0d27afaae1be24c6715d4</p> <p><span class="error">[root@master consumer]</span># kubectl describe pod openvino-cons-app2 | grep Image<br/> Image: openvino-cons-app:2.0<br/> Image ID: docker://sha256:8163d1887505ef6790902032f796ee326ed0beb958f3ba6736113214ac5aeece</p> <p>A map is used to store consumer connections as following:<br/> <a href="https://github.com/open-ness/edgenode/blob/master/pkg/eaa/main.go#L23" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/open-ness/edgenode/blob/master/pkg/eaa/main.go#L23</a></p> <p>The common name is used as the key of the map:<br/> <a href="https://github.com/open-ness/edgeapps/blob/master/openvino/consumer/cmd/main.go#L42" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/open-ness/edgeapps/blob/master/openvino/consumer/cmd/main.go#L42</a><br/> <a href="https://github.com/open-ness/edgenode/blob/master/pkg/eaa/api_consumer.go#L57" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/open-ness/edgenode/blob/master/pkg/eaa/api_consumer.go#L57</a></p> <p>As the openvino-cons-app and openvino-cons-app2 is 2 instances of the same microservice, they share the same common name. When openvino-cons-app2 starts successfully and try to register with eaa, eaa will detect whether this common name has registerd or not as following:<br/> <a href="https://github.com/open-ness/edgenode/blob/master/pkg/eaa/api_consumer.go#L37" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/open-ness/edgenode/blob/master/pkg/eaa/api_consumer.go#L37</a><br/> Because the common name has been registered by openvino-cons-app, eaa will close the connection of openvino-cons-app as following:<br/> <a href="https://github.com/open-ness/edgenode/blob/master/pkg/eaa/api_consumer.go#L41" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/open-ness/edgenode/blob/master/pkg/eaa/api_consumer.go#L41</a></p> <p>Finding a bug within openness:<br/> If we run multiple instances of same consumer, only one consumer can work. </p> <p>For example:<br/> consumer1 and consumer2 are 2 instances of the same consumer. Assume we run consumer1 firstly. After we run consumer2, consumer1 will lost the connection and restart. After consumer1 restart successfully, consumer2 will lost connection and restart. <br/> node1 node2 <br/> <br/> eaa <br/> <br/> producer1 consumer1 consumer2 </p> <p> </p> <p>The logs are below:<br/> <span class="error">[root@master consumer]</span># kubectl logs openvino-cons-app2<br/> <span class="error">[setupvars.sh]</span> OpenVINO environment initialized<br/> 2020/01/10 08:42:28 OpenVINO Consumer Started<br/> 2020/01/10 08:42:28 WebSocket establishment successful<br/> 2020/01/10 08:42:28 Subscribed to notification: openvino-inference 1.0.0<br/> 2020/01/10 08:42:28 Subscribed to notification: terminate 1.0.0<br/> 2020/01/10 08:42:30 Failed to read message from WebSocket: websocket: close 1012: New connection request, closing this connection<br/> ./start.sh: line 10: fg: no job control</p> <p><span class="error">[root@master consumer]</span># kubectl get pod -o wide<br/> NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES<br/> openvino-cons-app 1/1 Running 3 63m 10.16.0.17 node1 <none> <none><br/> openvino-cons-app2 0/1 Error 4 62m 10.16.0.18 node2 <none> <none><br/> openvino-prod-app 1/1 Running 0 74m 10.16.0.3 node1 <none> <none></p> <p>Because edge apps on different edge node all can access service eaa, the consumer can consume the service provided by producer which is on a different node.</p> <p>For example:<br/> producer1 is located in node1 and consumer2 is located on node2. The networking flow will be:<br/> producer1 -> service eaa -> pod eaa<br/> consumer2 -> service eaa -> pod eaa<br/> node1 node2 <br/> <br/> eaa <br/> <br/> producer1 consumer2 </p> <p>eaa is deployed as a deployment and only 1 eaa will be deployed:<br/> <a href="https://github.com/open-ness/edgecontroller/blob/master/kube-ovn/openness.yaml#L41" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/open-ness/edgecontroller/blob/master/kube-ovn/openness.yaml#L41</a></p> <p>Because all edge apps will access only 1 eaa, it doesn't matter that eaa is stateful. </p> <p>For example:<br/> only 1 eaa is deployed on node1. producer1 and producer2 will activate the new service with eaa. consumer1 and consumer2 will consume services stored in eaa. Because all the information are stored in only 1 eaa, there won't be issues. </p> <p> node1 node2 <br/> <br/> eaa <br/> <br/> producer1 consumer1 producer2 consumer2 <br/> </p> <p>Edge apps will access eaa through eaa.openness (name.namespace) which is a kubernetes service:<br/> <a href="https://github.com/open-ness/edgecontroller/blob/master/kube-ovn/openness.yaml#L18" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/open-ness/edgecontroller/blob/master/kube-ovn/openness.yaml#L18</a></p> <p>For example: as following links show, openvino consumer will access <a href="http://eaa.openness:443/auth" class="external-link" target="_blank" rel="nofollow noopener">http://eaa.openness:443/auth</a> for authentication.<br/> <a href="https://github.com/open-ness/edgeapps/blob/master/openvino/consumer/cmd/main.go#L24" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/open-ness/edgeapps/blob/master/openvino/consumer/cmd/main.go#L24</a><br/> <a href="https://github.com/open-ness/edgeapps/blob/master/openvino/consumer/cmd/main.go#L66" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/open-ness/edgeapps/blob/master/openvino/consumer/cmd/main.go#L66</a></p> <p>Edge applications must introduce themselves to OpenNESS framework and identify if they would like to activate new edge services or consume an existing service. Edge Application Agent (EAA) component is the handler of all the edge applications hosted by the OpenNESS edge node and acts as their point-of-contact. </p> <p>OpenNESS-awareness involves (a) authentication, (b) service activation/deactivation, (c) service discovery, (d) service subscription, and (e) Websocket connection establishment. The Websocket connection retains a channel for EAA for notification forwarding to pre-subscribed consumer applications. Notifications are generated by "producer" edge applications and absorbed by "consumer" edge applications.</p> <p>The sequence of operations for the producer application:</p> <ol> <li>Authenticate with OpenNESS edge node</li> <li>Activate new service and include the list of notifications involved</li> <li>Send notifications to OpenNESS edge node according to business logic</li> </ol> <p>The sequence of operations for the consumer application:</p> <ol> <li>Authenticate with OpenNESS edge node</li> <li>Discover the available services on OpenNESS edge platform</li> <li>Subscribe to services of interest and listen for notifications</li> </ol> Development Epic Link ICN-191 Rank 0|i00310: Sprint ICN Sprint 10