I decided to start the discussion here instead of GH to give a bit of a status update and give us a chance to discuss before filing actual work tickets under https://github.com/eclipse/lyo.testsuite/issues
Today I made the following updates:
- migrated Test Suite wiki from Eclipse to the README on Github
- updated RIO & added instructions to run on Java 8 and 11 (mostly Sesame/RDF4J and Jetty version updates to fetch from Central and run on JDK 1.8) and Eclipse 2020-12.
- updated the Test Suite and ran it against the RIO CM server, validating the instructions (my understanding is that 80 out of 97 test executions ran without an error).
This is the test result:
Here are my thoughts:
- We need to automate running the test suite. Fortunately, the launch files just start a single test with a JVM param that I have automated to run with
mvn clean -Dtest=org.eclipse.lyo.testsuite.server.DynamicSuiteBuilder -Dprops=config/rio-cm/rio-cm-setup.properties test -fae. Currently the RIO CM server setup is not automated and I have little interest to work on old RIO, I just wanted to run the old stuff to get the baseline.
- We need to automate server startup. Either by shallow clone in git, or by publishing WAR files in GH actions or by deploying the latest version of every app to KTH or OP server.
- Create a text matrix. Maybe also makes sense to go away from Properties files and use YAML (https://commons.apache.org/proper/commons-configuration/apidocs/org/apache/commons/configuration2/YAMLConfiguration.html should help migrate)
- Create a new baseline against OSLC 2020 RefImpl.
- Check the failing tests and decide if they are still in line with OSLC 3.
- Add OSLC 3 conformance clause number and preferably mnemonics to each test method.
- Add more tests to cover OSLC 3 or at least to fully cover OSLC 2.
How far did you get when runnning the test suite against your server? Is anyone using the test suite now and is interested to help out with setting up regular tests against 2020 RefImpl and potentially extending the RefImpl to pass more test cases?