[REC-22] Create CD testrunner

Wed, 4 Sep 2019 13:38:42 +0000

Create the necessary code that can be called by contributor CD systems (such as the one already run by AT&T or by CD systems that may be setup in the future by any new REC contributors.)This story does not include actually writing any tests, but it does include writing some documentation in a new child page under https://wiki.akraino.org/display/AK/Radio+Edge+Cloud+Developer+Documentation that explains how developers can add tests such that they will be automatically run the next time an LF Jenkins job is triggered (e.g. when a new change is posted to https://gerrit.akraino.org (verify job) or when an existing change on Gerrit is merged (merge job)) and how new contributors can invoke the top level testrunner in a CD system that they setup in the future.

It was generally agreed by people in a preliminary discussion that we want to standardize on Robot Framework as the top level testrunner. If it looks like it is necessary for Robot to invoke some other tooling, that should be brought back to the team for discussion.

The code to be written needs to have the following characteristics:

Can be run on a deployed REC cluster
- The expected usage is that Jenkins will transfer the test runner to a newly deployed REC cluster and execute it on the cluster
Clones one or more git repositories containing all the test case code (exact location TBD, but it'll be on https://gerrit.akraino.org with the other TA/REC repos). The reason for the "or more" is that we may want to put tests that apply to all Telco Appliance blueprint family members into a separate repo from tests that are specifically radio related.
Executes the tests in sequence
- Stretch goal: think through whether it is possible to architect the test runner to allow for some way of parallelizing tests
Logs parse-able test results so that we can have some sort of post-processing of test results in the future.
Supports a concept of non-fatal failure and/or experimental test case so that as new test cases are added we can have the test running execute them but not count them towards the overall pass/fail status
Returns an exit code to Jenkins to report whether the test suite was successful

Akraino JIRA

[REC-22] Create CD testrunner