Improving Lyo Test Suite

SDKs Lyo
1

Hi everyone,

I decided to start the discussion here instead of GH to give a bit of a status update and give us a chance to discuss before filing actual work tickets under https://github.com/eclipse/lyo.testsuite/issues

Today I made the following updates:

  • migrated Test Suite wiki from Eclipse to the README on Github
  • updated RIO & added instructions to run on Java 8 and 11 (mostly Sesame/RDF4J and Jetty version updates to fetch from Central and run on JDK 1.8) and Eclipse 2020-12.
  • updated the Test Suite and ran it against the RIO CM server, validating the instructions (my understanding is that 80 out of 97 test executions ran without an error).

This is the test result:

image
image703×1091 223 KB

Here are my thoughts:

  1. We need to automate running the test suite. Fortunately, the launch files just start a single test with a JVM param that I have automated to run with mvn clean -Dtest=org.eclipse.lyo.testsuite.server.DynamicSuiteBuilder -Dprops=config/rio-cm/rio-cm-setup.properties test -fae. Currently the RIO CM server setup is not automated and I have little interest to work on old RIO, I just wanted to run the old stuff to get the baseline.
  2. We need to automate server startup. Either by shallow clone in git, or by publishing WAR files in GH actions or by deploying the latest version of every app to KTH or OP server.
  3. Create a text matrix. Maybe also makes sense to go away from Properties files and use YAML (https://commons.apache.org/proper/commons-configuration/apidocs/org/apache/commons/configuration2/YAMLConfiguration.html should help migrate)
  4. Create a new baseline against OSLC 2020 RefImpl.
  5. Check the failing tests and decide if they are still in line with OSLC 3.
  6. Add OSLC 3 conformance clause number and preferably mnemonics to each test method.
  7. Add more tests to cover OSLC 3 or at least to fully cover OSLC 2.

How far did you get when runnning the test suite against your server? Is anyone using the test suite now and is interested to help out with setting up regular tests against 2020 RefImpl and potentially extending the RefImpl to pass more test cases?

–Andrew

2

@jad @ndjc @jamsden any chance you could allocate some time next year to help me to improve the test suite parity with the conformance clauses in the specs and (2nd prio) to get the RefImpl CM and others to improve on the test scores of the RIO CM?

3

I have made some updates and the test suite is now able to build on all JDKs from 8 till 17-ea (next LTS which will be released in the fall of 2021):

image
image1148×1018 106 KB

The reason behind this is that I wish to update the code to use some new features but given the deep integration with Ant and JUnit, Kotlin (which is my normal choice because it allows to keep the project JVM 1.8 compatible) is not a good fit.