After a long gap since the work from last year, we now finally have Jenkins job running on the IEC project in Akraino Jenkins, for testing SEBA on top of IEC.

For now, only jobs for the Fuel installer and only for baremetal PODs, since virtual PODs are running excruciatingly slow and are not reliable for running SEBA properly.

Here is the corresponding change:
https://gerrit.akraino.org/r/c/ci-management/+/3352

Here is the view of the master job in Jenkins:
https://jenkins.akraino.org/view/iec/job/iec-type2-fuel-baremetal-ubuntu1804-daily-master/

Here is the actual job that runs the tests:
https://jenkins.akraino.org/view/iec/job/iec-type2-test-seba_on_arm-fuel-baremetal-ubuntu1804-daily-master/

Note that these jobs could change names in the future.

For achieving the purpose of this task, there were a number of tasks involved.

As a foreword, I initially started experimenting with PONSim using IEC 3 nodes clusters, first on x86 and then arm64. So here are the different issues/tasks that needed to be solved.

1. Enable PONSim to run successfully on 3 nodes clusters. The were 2 issues here, one with genie-policy running on all nodes, even if it should only work on the master, and another one where the mininet pod would sometimes be scheduled to another node than the ponsimv2 components (e.g. rg). The following commit fixes this until we can make it upstream [1].
2. Build PONSim and other related components for arm64. The Docker images resulted have been pushed to a temporary location on Dockerhub, but I will push them to the iecedge hub as soon as possible. The ponsim_3node_arm64 branch in iecedge/helm-charts shows the images changed and their location [2] 
3. Installation scripts for PONSim. I tried to reuse the tools from upstream opencord, so I made this gerrit change [3]. This uses the opencord/automation-tools repo to deploy SEBA-in-a-Box. Since our Jenkins jobs already deploy SEBA, I decided to bypass the SEBA installation steps from automation-tools by faking the completion of the corresponding Makefile targets (this consists of touching various files in a temporary directory). It was also necessary to make the scripts use the iecedge/helm-charts version, previously described. It's worth noting that the installation script must be run on the K8S master.
4. Tests scripts for SEBA-in-a-Box. As with the install script, this phase uses the tools and testing framework from upstream, namley opencord/cord-tester. I referred to SIAB Opencord Jenkins jobs [4]. The test scripts are run in a Docker container [5], specifically built for running cord-tester SIAB robot tests. The resulting change for IEC is pending [6] but should be merged soon.
5. JJB related work. This consists of defining a new step in the pipeline, iec-test-{use_case}-*. For now we only have seba_on_arm use_case, which means only one such test job, for each combination of {installer} and {os}. The gerrit change is pending [7] and needs review from ci-management team.

Unfortunately there is still some work to be done. During the time spent experimenting with SEBA-in-a-Box and PONSim I've identified a number of issues, both specific to virtual pods and arm64. I will create new Jira cards to document and keep track of them. This means I cannot successfully run the test jobs in Jenkins, not unless we solve the arm64 issues and possibly move to baremetal deployment pods.

[1] https://github.com/iecedge/helm-charts/commit/561dfcff4e363bdc0b01e04a987c90a8c491c85f
[2] https://github.com/iecedge/helm-charts/tree/ponsim_3node_arm64
[3] https://gerrit.akraino.org/r/#/c/iec/+/1131/
[4] https://jenkins.opencord.org/view/SIAB/job/seba-in-a-box-build-test-latest/
[5] https://hub.docker.com/r/iecedge/cord-tester/tags
[6] https://gerrit.akraino.org/r/#/c/iec/+/1215/
[7] https://gerrit.akraino.org/r/#/c/ci-management/+/1243/