https://jira.akraino.org This file is an XML representation of an issue en-us 9.4.5 940005 11-04-2023 https://jira.akraino.org/browse/IEC-24 Integrated Edge Cloud <p>Currently we deploy SEBA on top of IEC by using the simpler approach, with three set of charts: cord-platform -&gt; seba -&gt; att-workflow.<br/> Although this method is mentioned in the Opencord Guide and it should work fine, in practice it seems to fail due to unpredictable race conditions, on both x86 and aarch64 setups (verified mostly on virtual PODs but could affect baremetal as well).</p> <p>Looking at the upstream Opencord SEBA-in-a-Box Jenkins jobs, we see that the deployment of SEBA is done at a more granular level, component by component. This is described in the Opencord Guide:<br/> <a href="https://guide.opencord.org/cord-6.1/profiles/seba/install.html#alternatively-install-seba-as-separate-components" class="external-link" target="_blank" rel="nofollow noopener">https://guide.opencord.org/cord-6.1/profiles/seba/install.html#alternatively-install-seba-as-separate-components</a></p> <p>For this purpose we will need to make use of the opencord automation-tools repo which can deploy SEBA-in-a-Box by running "make siab"</p> <p><a href="https://github.com/opencord/automation-tools/blob/master/seba-in-a-box/Makefile" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opencord/automation-tools/blob/master/seba-in-a-box/Makefile</a></p> <p>Similar work has been done for integrating PONSim support in IEC ci-management:</p> <p><a href="https://gerrit.akraino.org/r/#/c/ci-management/+/1243/" class="external-link" target="_blank" rel="nofollow noopener">https://gerrit.akraino.org/r/#/c/ci-management/+/1243/</a></p> IEC-24

[IEC][SEBA] Change SEBA deployment method to replicate upstream Opencord SIAB

Task Medium Done Done Ciprian Barbu Ciprian Barbu Release_2 Fri, 20 Sep 2019 13:10:11 +0000 Mon, 6 Apr 2020 11:51:57 +0000 Mon, 6 Apr 2020 11:51:57 +0000 0 3 <p>I've added charts-helm-charts-cord-7.0-arm64.xlsx spreadsheet, similar to the -amd64 version but pointing to the final image locations and versions for the current stable SEBA version that we have verified on aarch64.</p> <p>At the moment the scope of this Jira card is complete, although I would like to remove the old charts-seba_charts spreadsheets before closing.</p> <p>The work for switching the installation method to use automation-tools is now complete, after the following change has been merged:<br/> <a href="https://gerrit.akraino.org/r/c/iec/+/3305" class="external-link" target="_blank" rel="nofollow noopener">https://gerrit.akraino.org/r/c/iec/+/3305</a></p> <p>At this very moment, the scripts use the helm charts that have been added as git submodules in iec, which in turn uses Docker images from personal Docker hub repositories. We are working to copy these to the iecedge space.</p> <p>When this is done, I will also post a spreadsheet indicating the current Docker images locations and versions, for clarity.</p> <p>I have added a new version of the charts-patchset-13027.xls spreadsheet and removed the old version, as it missed one set of K8S PODs (kafka-0,1,2).</p> <p>The new version also contains the additional column indicating the helm template that defines the specified POD, which is helpful for better understanding how the POD comes to exist in the deployment, and what containers it defines. Granted there could be additional information showing the set of containers defined in each template, but this information can be extracted from the "Image" column in the spreadsheet, no matter if the image defines an initContainer or a regular container.</p> <p>It should suffice for achieving our goal of understanding what is the set of charts and images needed for deploying a minimal SEBA application, and help us track the changes that happened in upstream.</p> <p>I have also added a new spreadsheet called charts-iecedge-seba-1.0.0.xlsx, which is basically the same as charts-seba_charts.xlslx, but with additions and corrections. We should probably delete the old spreadsheet.</p> <p>Hi,</p> <p>Attached charts-seba_charts_v1 contains a new column added with the template for each pod.</p> <p>Valentin Radulescu</p> <p>Hi,</p> <p>Attached charts-seba_charts contains the versions used in seba for arm.</p> <p>Valentin Radulescu</p> <p>I started doing some investigation work, with the goal of better understanding what changes we need to make to our own set of charts and Docker images.</p> <p>One of the things I discovered while investigating the failed SiaB robot tests (as per <a href="https://jira.akraino.org/browse/IEC-25" title="[IEC][SEBA][PONSim] Errors while running SIAB robot tests" class="issue-link" data-issue-key="IEC-25"><del>IEC-25</del></a>), is that even for SEBA and SiaB stable (6.1.0) Opencord does not keep a fixed baseline of charts and Docker images. I find this at least strange, but there are clearly good reason to do so, probably it was the only easy way to ensure that the stable branch still builds and runs correctly, while making maintenance easy on the log run.</p> <p>This is in contrast with the situation of SEBA in IEC, since the charts and images have been ported a while ago (beginning of 2019) and have stayed mostly the same ever since. This is, by the way, a strong reason to work together with upstream and make sure SEBA is properly maintained and verified on ARM ecosystem.</p> <p>To start with, I needed to find out what was the base revision of the charts when they were first forked to our IEC SEBA charts repo <span class="error">&#91;1&#93;</span>. In truth, there was no actual fork done, instead the charts have been downloaded from <a href="https://charts.opencord.org/" class="external-link" target="_blank" rel="nofollow noopener">https://charts.opencord.org</a>, decompressed and added to a new repo. I don't know why this was done, but I believe it was a bad idea, since this is a live repo and is updated regularly.</p> <p>However, I tracked down the way the repo was populated and found out that the source of it is cord-charts-repo in opencord <span class="error">&#91;2&#93;</span>. This stores all the released charts as archives, just as charts.opencord.org, thus being a set of snapshots of the charts repository. The actual source of the charts (at least those owned by opencord), is helm-charts <span class="error">&#91;3&#93;</span>, but <span class="error">&#91;2&#93;</span> is important because it holds the "combined" charts for cord-platform and seba, which specify everything needed to install SEBA, including charts from outside of Opencord.</p> <p>Having established these, the next step was to identify the set of charts corresponding with the charts in seba-charts as described by <span class="error">&#91;1&#93;</span>. There is no exact, easy way to find it, so I just eyeballed it to be this commit <span class="error">&#91;4&#93;</span> or something around it, based on the date. I also unpacked cord-platform-6.1.0.tgz, seba-1.0.0.tgz and att-workflow-1.0.2.tgz and diffed them to their counterparts in iecedge seba_charts <span class="error">&#91;1&#93;</span> and they matched, so this was good enough for me.</p> <p>The next step was to determine all of the sub-charts that are actually used in our deployments and the images that are used, together with their location (Docker repository) and versions (Docker tags). This meant traversing all the charts and sub-charts and inspecting all the yaml files which define Helm Deployment, DaemonSet, Job and other similar Helm metadata type which actually create containers, being them ephemeral (e.g. tosca-loader jobs) or persistent.</p> <p>Also, some yaml templates are conditioned by boolean enabled-like values in the chart, which raises the question of what would happen if someone would try to enable them for aarch64 but there are no aarch64 version of those images. The short answer is that these kind of situation are not supported by default, and there will likely not be a plan to support them any time soon. So for now we will stick with supporting exactly the same images that are being tested regularly by Opencord.</p> <p>Finally, I tried to find a good way to represent all this information for the two situations we are interested in:<br/> A. SEBA deployed using the "combined" charts, on x86_64, using the base revision as identified by <span class="error">&#91;2&#93;</span>. Note that for aarch64 the actual values of chart versions and Docker images are actually the ones stored in iecedge seba_charts master branch <span class="error">&#91;5&#93;</span>, but it is easy enough to spot the differences.<br/> B. SEBA deployed using the individual charts, on x86_64, as performed by the Jenkins SiaB stable jobs <span class="error">&#91;6&#93;</span>. It is after all SiaB that we are after, and this is the actual Job that tests SiaB in upstream.</p> <p>For now I have settled on representing the charts and subcharts, together with the Docker images used and their tags, in a spreadsheet. For now I only have the representation for scenario A above, showing the values extracted from cord-charts-repo (again, not from iecedge seba_charts). I have attached this as charts-patchset-13027.xlsx</p> <p>The pattern should be easy to understand, columns A, B and C show charts and subcharts. Each chart or subchart can have their own templates and/or further subcharts. I have used different shades of blue to depict the "imbrication" level. The images are colored as green, again using different shades depending on the imbrication level. Also, each chart is clearly marked with solid outside borders.</p> <p>I hope this makes sense for someone seeing it for the first time, it does make sense to me. The next step is to do the same for scenario B and then start comparing them. This, in my opinion, will make very easy to determine what images we need to update or port on aarch64, so that scenario B can be put in effect for aarch64.</p> <p><span class="error">&#91;1&#93;</span> <a href="https://github.com/iecedge/seba_charts/commit/c65095b35c1aa82e270e0784db2dfb67f1ea09e0" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/iecedge/seba_charts/commit/c65095b35c1aa82e270e0784db2dfb67f1ea09e0</a><br/> <span class="error">&#91;2&#93;</span> <a href="https://gerrit.opencord.org/gitweb?p=cord-charts-repo.git;a=summary" class="external-link" target="_blank" rel="nofollow noopener">https://gerrit.opencord.org/gitweb?p=cord-charts-repo.git;a=summary</a><br/> <span class="error">&#91;3&#93;</span> <a href="https://gerrit.opencord.org/gitweb?p=helm-charts.git;a=summary" class="external-link" target="_blank" rel="nofollow noopener">https://gerrit.opencord.org/gitweb?p=helm-charts.git;a=summary</a><br/> <span class="error">&#91;4&#93;</span> <a href="https://gerrit.opencord.org/gitweb?p=cord-charts-repo.git;a=commit;h=b036eac6cb96094bc5792e7e0165581a07c6c65e" class="external-link" target="_blank" rel="nofollow noopener">https://gerrit.opencord.org/gitweb?p=cord-charts-repo.git;a=commit;h=b036eac6cb96094bc5792e7e0165581a07c6c65e</a><br/> <span class="error">&#91;5&#93;</span> <a href="https://github.com/iecedge/seba_charts/tree/master" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/iecedge/seba_charts/tree/master</a><br/> <span class="error">&#91;6&#93;</span> <a href="https://jenkins.opencord.org/view/SIAB/job/seba-in-a-box-build-test-stable/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opencord.org/view/SIAB/job/seba-in-a-box-build-test-stable/</a></p> Relates IEC-28 Development Rank 0|i00070: