DOORS Lyo incompatibility

Michael.C · June 16, 2022, 9:50am

Hello,

When I return a Resource with Lyo it’s formated like this

<?xml version="1.0" encoding="UTF-8"?>
<rdf:RDF
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:oslc="http://open-services.net/ns/core#"
    xmlns:j.0="http://open-services.net/ns/am#"
    xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">
  <j.0:Resource rdf:about="https://domain:8443/server-am/services/resource/c306ab60-b3f1-4a0b-97e4-bd82e820a138/46e1cf32-a19a-4c6b-8567-92e4ccffcdab">
    <dcterms:title rdf:parseType="Literal">Storekeeper</dcterms:title>
    <dcterms:identifier>46e1cf32-a19a-4c6b-8567-92e4ccffcdab</dcterms:identifier>
    <dcterms:modified rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime"
    >2020-09-23T09:13:10Z</dcterms:modified>
    <oslc:shortTitle rdf:parseType="Literal">Storekeeper</oslc:shortTitle>
  </j.0:Resource>
</rdf:RDF>

I tried to return a String like this

<rdf:RDF
	xmlns:dcterms="http://purl.org/dc/terms/"
	xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
	xmlns:oslc="http://open-services.net/ns/core#"
	xmlns:j.0="http://open-services.net/ns/am#"
	xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">
  <rdf:Description rdf:about="https://domain:8443/server-am/services/resource/c306ab60-b3f1-4a0b-97e4-bd82e820a138/46e1cf32-a19a-4c6b-8567-92e4ccffcdab">
	<dcterms:title rdf:parseType="Literal">Storekeeper</dcterms:title>
	<dcterms:identifier>46e1cf32-a19a-4c6b-8567-92e4ccffcdab</dcterms:identifier>
	<dcterms:modified rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime"
	>2020-09-23T09:13:10Z</dcterms:modified>
	<oslc:shortTitle rdf:parseType="Literal">Storekeeper</oslc:shortTitle>
	<rdf:type rdf:resource="http://open-services.net/ns/am#Resource"/>
  </rdf:Description>
</rdf:RDF>

So the first one does not work with DOORS next because I’m not getting a PUT request after the GET when trying to make a link from DOORS.
It was working with Polarion.
I read the documents about the valid OSLC format here (OSLC Architecture Management Specification Version 2.0 | Architecture Management - Open Services for Lifecycle Collaboration) and I was under the impression that both are valid.

Is this an issue with DOORS or Lyo ?
Is there a setting to have the second format in Lyo ?

Thanks you

Frej · June 16, 2022, 11:02am

Hi Michael,

The difference seems to be that one is application/xml content and the other is application/rdf+xml content.
If you specify an “Accept: xxx” header on your request, you should be able to switch from format 1 to format 2.
As for the PUT/GET, I am unsure… is your DOORSNext project configuration enabled?

-Frej.

Michael.C · June 16, 2022, 11:52am

Hello Frej,

Thanks for your advice. I logged the request from DOORS in my OSLC provider.
So DOORS is using a header with Accept application/rdf+xml, so I can’t change it.
The result of application/rdf+xml is the first request in my previous post.
If I use Postman and try with application/xml I get something again different.

<?xml version="1.0" encoding="UTF-8"?>
<rdf:RDF 0="http://open-services.net/ns/am#" xmlns:dcterms="http://purl.org/dc/terms/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:oslc="http://open-services.net/ns/core#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" rdf:about="https://PCTBOR2241:8443/server-am/services/resource/c306ab60-b3f1-4a0b-97e4-bd82e820a138/2f2b70cd-f771-4d18-a51f-a49e2e7d9fd2" rdf:datatype="http://www.w3.org/2001/XMLSchema#string">
    2f2b70cd-f771-4d18-a51f-a49e2e7d9fd2
</rdf:RDF>

I’m not using gc. I’m linking from DOORS, after selecting my ‘am’ object in the selection ui. DOORS is performing a GET but it’s only performing a PUT for adding the link in my producer with the second representation.

I think the 2 output I showed in my first post are valid application/rdf+xml but I need the second one. For the second one I returned a String from my oslc provider since I don’t know how to get it with Lyo.

jad · June 16, 2022, 9:42pm

Hi!
Both formats do seem to be valid rdf+xml.

I am a bit confused though about what you are trying to do.
For example, you say “I’m not getting a PUT request after the GET when …” Why would you expect a PUT request after a GET?
And, you also seem to be working with Jazz, Lyo and Polarion?

It would be good to understand, because I am wondering if your problem is somewhere else than the format?

jad · June 16, 2022, 10:02pm

I recall now some special hacks in Lyo Writers to handle DNG that might help.

github.com

eclipse/lyo/blob/db2e96960e0a37739a2d75e986c754b683e66516/core/oslc4j-jena-provider/src/main/java/org/eclipse/lyo/oslc4j/provider/jena/AbstractOslcRdfXmlProvider.java#L104

      
        
            						   final MultivaluedMap<String, Object> map,
            						   final OutputStream					outputStream,
            						   final Map<String, Object>			properties,
            						   final String							descriptionURI,
            						   final String							responseInfoURI,
            						   final ResponseInfo<?>				responseInfo)
            				throws WebApplicationException
            	{
            		String serializationLanguage = getSerializationLanguage(baseMediaType);
            
            
		// This is a special case to handle RDNG GET on a ServiceProvider resource.
            		// RDNG can only consume RDF/XML-ABBREV although its sometimes sends Accept=application/rdf+xml.
            		// The org.eclipse.lyo.oslc4j.alwaysXMLAbbrevOnlyProviders system property is used
            		// to turn off this special case
            		if ((objects != null && objects[0] != null) &&
            			( objects[0] instanceof ServiceProviderCatalog || objects[0] instanceof ServiceProvider ) &&
            			serializationLanguage.equals(FileUtils.langXML) &&
            			"true".equals(System.getProperty("org.eclipse.lyo.oslc4j.alwaysXMLAbbrevOnlyProviders"))) {
            			serializationLanguage = FileUtils.langXMLAbbrev;
            			log.info("Using RDF/XML-ABBREV for ServiceProvider resources");
            		}

This particular piece of code is most likely not relevant, but there seems to be 2 formats for rdf/xml. Normal and abbreviated. Could it be that your 2 formats above corrspond to these 2 formats?

Could you possibly put a breakpoint in the getSerializationLanguage() method to work out what you are working with? Can you make it choose one or the other to see what happens? (See the method OSLC4JUtils.alwaysAbbrevXML())

github.com

eclipse/lyo/blob/db2e96960e0a37739a2d75e986c754b683e66516/core/oslc4j-jena-provider/src/main/java/org/eclipse/lyo/oslc4j/provider/jena/AbstractOslcRdfXmlProvider.java#L252

      
        
            		);
            	}
            
            
	/**
            	 * Determine whether to serialize in xml or abreviated xml based upon mediatype.
            	 * <p>
            	 * application/rdf+xml yields xml
            	 * <p>
            	 * applicaton/xml yields abbreviated xml
            	 */
            	private String getSerializationLanguage(final MediaType baseMediaType) {
            
            
		if(baseMediaType == null) {
            			throw new IllegalArgumentException("Base media type can't be null");
            		}
            
            
		List<Map.Entry<MediaType, String>> mediaPairs = new ArrayList<>();
            		mediaPairs.add(new AbstractMap.SimpleEntry<>(OslcMediaType.TEXT_TURTLE_TYPE,
            													 RDFLanguages.strLangTurtle));
            		mediaPairs.add(new AbstractMap.SimpleEntry<>(OslcMediaType.APPLICATION_JSON_LD_TYPE,
            													 RDFLanguages.strLangJSONLD));

Finally, one difference I noticed above is a missing xmlDeclaration.
Could this be related to this line in the code?

github.com

eclipse/lyo/blob/db2e96960e0a37739a2d75e986c754b683e66516/core/oslc4j-jena-provider/src/main/java/org/eclipse/lyo/oslc4j/provider/jena/AbstractOslcRdfXmlProvider.java#L147

      
        
            																objects,
            																properties);
            			RDFWriterI writer = getRdfWriter(serializationLanguage, model);
            
            
			if (serializationLanguage.equals(FileUtils.langXML) || serializationLanguage.equals(FileUtils.langXMLAbbrev))
            			{
            				// XML (and Jena) default to UTF-8, but many libs default to ASCII, so need
            				// to set this explicitly
            				writer.setProperty("showXmlDeclaration",
            								   "false");
            				String xmlDeclaration = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n";
            				outputStream.write(xmlDeclaration.getBytes());
            			}
            			writer.write(model,
            						 outputStream,
            						 null);
                        Instant finish = Instant.now();
                        log.trace("writeObjectsTo - Execution Duration: {} ms", Duration.between(start, finish).toMillis());
            		}
            		catch (final Exception exception)
            		{

Just observations from old obscure logic from Lyo that I never really understood myself. But they somehow seem related to what you are facing.

andrew · June 16, 2022, 10:21pm

The second format will be returned if you are asking for application/rdf+xml (unless you set OSLC4JConstants.OSLC4J_ALWAYS_XML_ABBREV to true). If you only ask for application/xml, you will always get the first (abbreviated) format.

The abbreviated output (first in your example) is only used for Jazz. So, it’s a bit unusual that you want to generate a non-abbreviated output for the only system that needs abbreviated format support…

I guess we need a bit more info to reproduce:

what are your steps (on all the components involved)
what are the versions of software (jazz, lyo, java, etc.) you are running. I guess you are using OSLC RefImpl AM server?

Michael.C · June 20, 2022, 12:59pm

Hello, thanks for the advices.

I’m using open jdk 16 with Lyo 5.0.0.alpha2 and DOORS NEXT 7.0.2.
I based my producer on RefImpl AM server.

So when DOORS is performing a GET with the 2st example it works and DOORS is performing a PUT after.

<rdf:Description rdf:about="https://domain:8443/server-am/services/resource/c306ab60-b3f1-4a0b-97e4-bd82e820a138/46e1cf32-a19a-4c6b-8567-92e4ccffcdab">

But with this, it’s not performing a PUT after the GET

<j.0:Resource rdf:about="https://domain:8443/server-am/services/resource/c306ab60-b3f1-4a0b-97e4-bd82e820a138/46e1cf32-a19a-4c6b-8567-92e4ccffcdab">

This is even more confusing then.

I added Oslc_amDomainConstants.ARCHITECTURE_MANAGEMENT_DOMAIN in the list of types of the resource and I’m able to get the 2st form now.
But after adding some links, Lyo return sometimes the first form again.

I tried to put this line in the constructor of Application.java but with true or false, it’s not changing the output.

OSLC4JConstants.OSL4J_PROPERTY_SINGLETON.put(OSLC4JConstants.OSLC4J_ALWAYS_XML_ABBREV, true);

Michael.C · June 23, 2022, 12:40pm

So after more testing. It seems that Lyo or Jena is not taking into account the OSLC4JConstants.OSLC4J_ALWAYS_XML_ABBREV propertie. I’m not able to get
<rdf:Description … every time unless I add a custom types URI.

Also, I did not find a setting in DOORS to allow abreviated output, it’s working only if I have the plain version.

andrew · June 25, 2022, 8:44pm

I guess we will need input from @DavidJHoney or @jamsden here.

I am assuming you are trying to create a link from DNG to your AM and expecting a GET request to be followed by a PUT request?

Does DNG write some errors in the log?

The code that should do the work is under

github.com

eclipse/lyo/blob/db2e96960e0a37739a2d75e986c754b683e66516/core/oslc4j-jena-provider/src/main/java/org/eclipse/lyo/oslc4j/provider/jena/AbstractOslcRdfXmlProvider.java#L263-L271


      
          		if (OSLC4JUtils.alwaysAbbrevXML()) {
          			// application/rdf+xml will be forcefully abbreviated
          			mediaPairs.add(new AbstractMap.SimpleEntry<>(OslcMediaType.APPLICATION_RDF_XML_TYPE,
          														 FileUtils.langXMLAbbrev));
          		} else {
          			mediaPairs.add(new AbstractMap.SimpleEntry<>(OslcMediaType.APPLICATION_RDF_XML_TYPE,
          														 RDFLanguages.strLangRDFXML));
          
          		}

The value you want for this property is false.

I created a quick PR feat: Control the RDF output format manually by berezovskyi · Pull Request #277 · eclipse/lyo · GitHub that shows how we can require Jena to use the exact output format we want, instead of letting Jena choose depending on the language. I suspect that over time, Jena switched RDFXML to produce abbreviated output by default and it may have broken something.

We should also update the logic around forced/custom RDF/XML-ABBREV given that it seems that Jazz tools no longer like/require it.

jad · June 28, 2022, 10:08pm

Michael.C:

I tried to put this line in the constructor of Application.java but with true or false, it’s not changing the output.
OSLC4JConstants.OSL4J_PROPERTY_SINGLETON.put(OSLC4JConstants.OSLC4J_ALWAYS_XML_ABBREV, true);

This doesn’t seem to me to be how this constant should be set. As you can see below, the code is looking for a system property.

github.com

eclipse/lyo/blob/db2e96960e0a37739a2d75e986c754b683e66516/core/oslc4j-jena-provider/src/main/java/org/eclipse/lyo/oslc4j/provider/jena/AbstractOslcRdfXmlProvider.java#L104


      
          						   final MultivaluedMap<String, Object> map,
          						   final OutputStream					outputStream,
          						   final Map<String, Object>			properties,
          						   final String							descriptionURI,
          						   final String							responseInfoURI,
          						   final ResponseInfo<?>				responseInfo)
          				throws WebApplicationException
          	{
          		String serializationLanguage = getSerializationLanguage(baseMediaType);
          
          		// This is a special case to handle RDNG GET on a ServiceProvider resource.
          		// RDNG can only consume RDF/XML-ABBREV although its sometimes sends Accept=application/rdf+xml.
          		// The org.eclipse.lyo.oslc4j.alwaysXMLAbbrevOnlyProviders system property is used
          		// to turn off this special case
          		if ((objects != null && objects[0] != null) &&
          			( objects[0] instanceof ServiceProviderCatalog || objects[0] instanceof ServiceProvider ) &&
          			serializationLanguage.equals(FileUtils.langXML) &&
          			"true".equals(System.getProperty("org.eclipse.lyo.oslc4j.alwaysXMLAbbrevOnlyProviders"))) {
          			serializationLanguage = FileUtils.langXMLAbbrev;
          			log.info("Using RDF/XML-ABBREV for ServiceProvider resources");
          		}

eclipse/lyo/blob/db2e96960e0a37739a2d75e986c754b683e66516/core/oslc4j-core/src/main/java/org/eclipse/lyo/oslc4j/core/OSLC4JUtils.java#L90


      
          	 * property type from the resource shape and it will only be considered if
          	 * the property inferTypeFromShape is set to true. This is part of the the
          	 * fix for defect 412789.
          	 */
          	private static List<ResourceShape> shapes = new ArrayList<>();
          
          	public static boolean useStrictDatatypes() {
          		return parseBooleanPropertyOrDefault(OSLC4JConstants.OSLC4J_STRICT_DATATYPES, true);
          	}
          
          	public static boolean alwaysAbbrevXML() {
          		return parseBooleanPropertyOrDefault(OSLC4JConstants.OSLC4J_ALWAYS_XML_ABBREV, false);
          	}
          
          	/**
          	 * Returns the boolean value of org.eclipse.lyo.oslc4j.disableRelativeURIs Default is true if
          	 * not set or invalid (relative URIs will not be allowed)
          	 *
          	 * @return true if relative URIs are disabled, or if the property is not set or invalid.
          	 */
          	public static boolean relativeURIsAreDisabled() {