OSLC4J: Cannot get a ChangeRequest that contains special characters

Hello everyone,

I am trying to get a ChangeRequest from IBM Jazz using OSLC4J Library, but I have a problem when that ChangeRequest (in IBM called EWM workitem) contains special characters (e.g. quotation marks) in some of the fields.

When the workitem contains quotation marks in the Title or the Description (which are standard dcterms attributes) then the request works and the item is retrieved,

But,

When the special characters are in the extended/custom properties, for example “Requested Due Date Change Reason” custom text field, then it gives me the error HTTP 400 Bad Request caused by illegal instance of Datatype rdf:XMLLiteral (see the stacktrace at the end).

I checked the raw response and all text fields are of type XMLLiteral, so both the title and “Requested Due Date Change Reason” field are XML Literals, the only difference is that the title (and the description) are represented using dcterms and “Requested Due Date Change Reason” is rtc_ext.

StackTrace:

Caused by: org.apache.jena.datatypes.DatatypeFormatException: Lexical form ‘“test”’ is not a legal instance of Datatype[http://www.w3.org/1999/02/22-rdf-syntax-ns#XMLLiteral] Bad rdf:XMLLiteral

at org.apache.jena.graph.impl.LiteralLabelImpl.getValue(LiteralLabelImpl.java:338)
at org.apache.jena.graph.Node_Literal.getLiteralValue(Node_Literal.java:44)
at org.apache.jena.rdf.model.impl.LiteralImpl.getValue(LiteralImpl.java:98)
at org.eclipse.lyo.oslc4j.provider.jena.JenaModelHelper.handleExtendedPropertyValue(JenaModelHelper.java:1154)
at org.eclipse.lyo.oslc4j.provider.jena.JenaModelHelper.fromResource(JenaModelHelper.java:659)
at org.eclipse.lyo.oslc4j.provider.jena.JenaModelHelper.createObjectResultList(JenaModelHelper.java:528)
at org.eclipse.lyo.oslc4j.provider.jena.JenaModelHelper.fromJenaModel(JenaModelHelper.java:402)
at org.eclipse.lyo.oslc4j.provider.jena.JenaModelHelper.unmarshal(JenaModelHelper.java:369)
at org.eclipse.lyo.oslc4j.provider.jena.AbstractOslcRdfXmlProvider.readFrom(AbstractOslcRdfXmlProvider.java:305)

For me it looks like that for extended/properties you have a different mechanism (JenaModelHelper.handleExtendedPropertyValue) to read properties compared to the to the procedure for reading properties from standard domains (like dcterms).

Do you have a solution for this?

Thanks,
Ana

Hi Anna

To be able to best help you, could you possibly share:

  1. the version of OSLC4J library you are using.
  2. the raw data you are getting back from Jazz.

Also, the exception seems to occur when OSLC4J tries to handled an “ExtendedProperty”, which are properties not defined in the ChangeRequest class you are trying to Marshall into. So, could you possibly share

  1. The ChangeRequest class you are using in your code (or a link to the class, if you are using some existing library of classes from OSLC4J/Lyo)

Hello Jad,

I am using OSLC4J version 4.0.0.

Here you can see the raw response that I receive:

    <rdf:RDF
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:rtc_ext="http://jazz.net/xmlns/prod/jazz/rtc/ext/1.0/"
    xmlns:oslc="http://open-services.net/ns/core#"
    xmlns:acp="http://jazz.net/ns/acp#"
    xmlns:oslc_cm="http://open-services.net/ns/cm#"
    xmlns:oslc_cmx="http://open-services.net/ns/cm-x#"
    xmlns:oslc_pl="http://open-services.net/ns/pl#"
    xmlns:acc="http://open-services.net/ns/core/acc#"
    xmlns:rtc_cm="http://jazz.net/xmlns/prod/jazz/rtc/cm/1.0/"
    xmlns:process="http://jazz.net/ns/process#" >
    <rdf:Description rdf:about="https://server:port/ccmfed/resource/itemName/com.ibm.team.workitem.WorkItem/157769">
        <oslc:discussedBy rdf:resource="https://server:port/ccmfed/oslc/workitems/_rcQaoOrmEeuu8O7_BDi16Q/rtc_cm:comments"/>
        <rtc_cm:type rdf:resource="https://server:port/ccmfed/oslc/types/_yoMVoBT6Eeu9VOClgNyCpg/com.ibm.team.workitem.workItemType.featurerequest"/>
        <rtc_ext:archived rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</rtc_ext:archived>
        <rtc_ext:contextId rdf:datatype="http://www.w3.org/2001/XMLSchema#string">_yoMVoBT6Eeu9VOClgNyCpg</rtc_ext:contextId>
        <rtc_cm:progressTracking rdf:resource="https://server:port/ccmfed/oslc/workitems/_rcQaoOrmEeuu8O7_BDi16Q/progressTracking"/>
        <acc:accessContext rdf:resource="https://server:port/ccmfed/acclist#_yoMVoBT6Eeu9VOClgNyCpg"/>
        <oslc_cmx:priority rdf:resource="https://server:port/ccmfed/oslc/enumerations/_yoMVoBT6Eeu9VOClgNyCpg/priority/priority.literal.l5"/>
        <oslc_cmx:severity rdf:resource="https://server:port/ccmfed/oslc/enumerations/_yoMVoBT6Eeu9VOClgNyCpg/severity/severity.literal.l7"/>
        <rtc_cm:subscribers rdf:resource="https://server:port/jts/users/uic81254"/>
        <rtc_ext:automotive.wf.attribute.parentcollaborationcustomerrefid rdf:datatype="http://www.w3.org/2001/XMLSchema#string"></rtc_ext:automotive.wf.attribute.parentcollaborationcustomerrefid>
        <oslc_cm:verified rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</oslc_cm:verified>
        <acp:accessControl rdf:resource="https://server:port/ccmfed/oslc/access-control/_yoMVoBT6Eeu9VOClgNyCpg"/>
        <rtc_cm:repository rdf:resource="https://server:port/ccmfed/oslc/repository"/>
        <rtc_ext:automotive.wf.attribute.closingcomment rdf:parseType="Literal"></rtc_ext:automotive.wf.attribute.closingcomment>
        <process:projectArea rdf:resource="https://server:port/ccmfed/process/project-areas/_yoMVoBT6Eeu9VOClgNyCpg"/>
        <rtc_ext:automotive.wf.attribute.criticality rdf:resource="https://server:port/ccmfed/oslc/enumerations/_yoMVoBT6Eeu9VOClgNyCpg/automotive.wf.enumeration.criticality/automotive.wf.enumeration.criticality.literal.l7"/>
        <rtc_ext:automotive.wf.attribute.requesteddetectedby rdf:resource="https://server:port/ccmfed/oslc/enumerations/_yoMVoBT6Eeu9VOClgNyCpg/automotive.wf.enumeration.requesteddetectedby/automotive.wf.enumeration.requesteddetectedby.literal.l6"/>
        <rtc_ext:automotive.wf.attribute.requestedduedatechangereason rdf:datatype="http://www.w3.org/1999/02/22-rdf-syntax-ns#XMLLiteral">&amp;quot;test&amp;quot;</rtc_ext:automotive.wf.attribute.requestedduedatechangereason>
        <oslc_cmx:project rdf:resource="https://server:port/ccmfed/oslc/projectareas/_yoMVoBT6Eeu9VOClgNyCpg"/>
        <rtc_cm:timeSheet rdf:resource="https://server:port/ccmfed/oslc/workitems/_rcQaoOrmEeuu8O7_BDi16Q/rtc_cm:timeSheet"/>
        <dcterms:contributor rdf:resource="https://server:port/jts/users/unassigned"/>
        <dcterms:subject rdf:datatype="http://www.w3.org/2001/XMLSchema#string"></dcterms:subject>
        <oslc_cm:inprogress rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</oslc_cm:inprogress>
        <rtc_cm:filedAgainst rdf:resource="https://server:port/ccmfed/resource/itemOid/com.ibm.team.workitem.Category/_1GbSoBT6Eeu9VOClgNyCpg"/>
        <rtc_cm:modifiedBy rdf:resource="https://server:port/jts/users/uic81254"/>
        <dcterms:created rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2021-07-22T13:05:58.272Z</dcterms:created>
        <rdf:type rdf:resource="http://open-services.net/ns/cm#ChangeRequest"/>
        <rtc_ext:automotive.wf.attribute.enablecustomerexchange rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</rtc_ext:automotive.wf.attribute.enablecustomerexchange>
        <rtc_ext:automotive.wf.attribute.ccbcomment rdf:parseType="Literal"></rtc_ext:automotive.wf.attribute.ccbcomment>
        <rtc_ext:automotive.wf.attribute.collaborationpartnerdescription rdf:parseType="Literal"></rtc_ext:automotive.wf.attribute.collaborationpartnerdescription>
        <rtc_ext:automotive.wf.attribute.resource rdf:datatype="http://www.w3.org/2001/XMLSchema#string"></rtc_ext:automotive.wf.attribute.resource>
        <oslc:shortTitle rdf:parseType="Literal">Feature Request 157769</oslc:shortTitle>
        <rtc_cm:resolvedBy rdf:resource="https://server:port/jts/users/unassigned"/>
        <rtc_ext:automotive.wf.attribute.riskforsuccessfulcompletion rdf:resource="https://server:port/ccmfed/oslc/enumerations/_yoMVoBT6Eeu9VOClgNyCpg/automotive.wf.enumeration.riskforsuccessfulcompletion/automotive.wf.enumeration.riskforsuccessfulcompletion.literal.l9"/>
        <dcterms:identifier rdf:datatype="http://www.w3.org/2001/XMLSchema#string">157769</dcterms:identifier>
        <dcterms:title rdf:parseType="Literal">Test Feature</dcterms:title>
        <dcterms:creator rdf:resource="https://server:port/jts/users/uic81254"/>
        <rtc_ext:automotive.wf.attribute.resourcebookedduration rdf:datatype="http://www.w3.org/2001/XMLSchema#decimal">0</rtc_ext:automotive.wf.attribute.resourcebookedduration>
        <oslc_pl:schedule rdf:resource="https://server:port/ccmfed/oslc/workitems/_rcQaoOrmEeuu8O7_BDi16Q/schedule"/>
        <dcterms:type rdf:datatype="http://www.w3.org/2001/XMLSchema#string">Feature Request</dcterms:type>
        <rtc_ext:automotive.wf.attribute.commentbycollaborationpartner rdf:parseType="Literal"></rtc_ext:automotive.wf.attribute.commentbycollaborationpartner>
        <rtc_ext:automotive.wf.attribute.collaborationpartnerid rdf:datatype="http://www.w3.org/2001/XMLSchema#string"></rtc_ext:automotive.wf.attribute.collaborationpartnerid>
        <oslc_cm:closed rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</oslc_cm:closed>
        <rtc_ext:automotive.wf.attribute.commenttocollaborationpartner rdf:parseType="Literal"></rtc_ext:automotive.wf.attribute.commenttocollaborationpartner>
        <rtc_cm:state rdf:resource="https://server:port/ccmfed/oslc/workflows/_yoMVoBT6Eeu9VOClgNyCpg/states/com.ibm.team.workitem.FeatureRequestWorkflow/com.ibm.team.workitem.FeatureRequestWorkflow.state.s1"/>
        <dcterms:description rdf:datatype="http://www.w3.org/1999/02/22-rdf-syntax-ns#XMLLiteral">this is a &amp;quot;test&amp;quot; description</dcterms:description>
        <rtc_ext:automotive.wf.attribute.documentationofresult rdf:parseType="Literal"></rtc_ext:automotive.wf.attribute.documentationofresult>
        <oslc_cm:reviewed rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</oslc_cm:reviewed>
        <oslc_cm:fixed rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</oslc_cm:fixed>
        <oslc:instanceShape rdf:resource="https://server:port/ccmfed/oslc/shapes/workitems/_rcQaoOrmEeuu8O7_BDi16Q"/>
        <oslc:shortId rdf:datatype="http://www.w3.org/2001/XMLSchema#string">157769</oslc:shortId>
        <rtc_ext:automotive.wf.attribute.definitionofdone rdf:parseType="Literal"></rtc_ext:automotive.wf.attribute.definitionofdone>
        <rtc_ext:automotive.wf.attribute.analysissummary rdf:parseType="Literal"></rtc_ext:automotive.wf.attribute.analysissummary>
        <oslc_cm:approved rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</oslc_cm:approved>
        <oslc:serviceProvider rdf:resource="https://server:port/ccmfed/oslc/contexts/_yoMVoBT6Eeu9VOClgNyCpg/workitems/services"/>
        <rtc_ext:automotive.wf.attribute.collaborationpartnerstate rdf:datatype="http://www.w3.org/2001/XMLSchema#string"></rtc_ext:automotive.wf.attribute.collaborationpartnerstate>
        <dcterms:modified rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2021-07-22T13:05:58.517Z</dcterms:modified>
        <rtc_ext:automotive.wf.attribute.approver rdf:resource="https://server:port/jts/users/unassigned"/>
        <rtc_ext:automotive.wf.attribute.initiator rdf:datatype="http://www.w3.org/2001/XMLSchema#string"></rtc_ext:automotive.wf.attribute.initiator>
        <rtc_ext:automotive.wf.attribute.collaborationpartnerseverity rdf:datatype="http://www.w3.org/2001/XMLSchema#string"></rtc_ext:automotive.wf.attribute.collaborationpartnerseverity>
        <oslc_cm:status rdf:datatype="http://www.w3.org/2001/XMLSchema#string">New</oslc_cm:status>
    </rdf:Description>
</rdf:RDF>

I hardcoded the server name for obvious reasons, but the things that I consider important are these:

  1. <dcterms:description rdf:datatype="http://www.w3.org/1999/02/22-rdf-syntax-ns#XMLLiteral">this is a &amp;quot;test&amp;quot; description</dcterms:description>

  2. <rtc_ext:automotive.wf.attribute.requestedduedatechangereason rdf:datatype="http://www.w3.org/1999/02/22-rdf-syntax-ns#XMLLiteral">&amp;quot;test&amp;quot;</rtc_ext:automotive.wf.attribute.requestedduedatechangereason>

Related to your 3rd question, I use the ChangeRequest class from OSLC4J 4.0.0 and from what I saw in the class, yes, the extended properties are not handled there but in the AbstractResource class from which ChangeRequest is extended.

Hope this clarifies your questions,
Ana

Hi

I have now tried and can’t seem to easily reproduce this problem.
I tried with the following extendedProperties but they all seem to be unmarshalled fine by an OSLC Client.

    Map<QName, Object> props = new HashMap<QName, Object>();
    props.put(new QName("http://jazz.net/xmlns/prod/jazz/rtc/ext/1.0/", "automotive.wf.attribute.requestedduedatechangereason1"), new XMLLiteral ("test"));
    props.put(new QName("http://jazz.net/xmlns/prod/jazz/rtc/ext/1.0/", "automotive.wf.attribute.requestedduedatechangereason2"), new XMLLiteral ("\"test\""));
    props.put(new QName("http://jazz.net/xmlns/prod/jazz/rtc/ext/1.0/", "automotive.wf.attribute.requestedduedatechangereason3"), new XMLLiteral ("&amp;quot;test&amp;quot;"));
    r.setExtendedProperties(props );

Do you have access to the actual values (at the user interface) for this property? If so, what is the actual value? is it "test or “&quot;test&quot;”
Can you also modify this value removing the quotes (and other characters like “&”) until it works?

Finally, you can also breakpoint at this point in the OSLC4J code to see exactly what the problem is.
Note though that there is some code that tries to recover from such an exception (line 1150). Only if that fails, will the exception be thrown again at line 1170.

The exception is occuring at this exact point (I believe, but you need to confirm).

Hi Anna

Just wondering if you ever managed to solve this issue?

Hello Jad,

Sorry that it took me so long to respond.

The actual values in the user interface are with “ and not ;&quot and even if it works if it’s replaced with ;&quot I don’t consider it a solution because:

  • We cannot make sure that all the users will write it like this and it’s also not so user friendly;
  • There are other special characters that can cause the error, not only quotation marks (for example, ^ throws the same error).

Related to the exact point where the error occurs, you are right it’s the line that you indicated.

The error is caught at line 1150 and then it goes directly to the “else” from line 1168 and is thrown at 1170. So we checked the “if” which looks like this:

if ("false".equals(System.getProperty(AbstractOslcRdfXmlProvider.OSLC4J_STRICT_DATATYPES)))

We tried then to explicitly set this system property to false

System.setProperty(AbstractOslcRdfXmlProvider.OSLC4J_STRICT_DATATYPES,"false");

and now the Change Request is returned well, except the fields that contain special characters which are returned as UnparsableLiteral, for example:

{​​​QName@4989}​​​ "{​​​http://jazz.net/xmlns/prod/jazz/rtc/ext/1.0/}​​​custom.wf.attribute.closingcomment" -> {​​​UnparseableLiteral@5170}​​​ "unparseable literal: "<span style="background-color:rgb(255, 255, 255)">][$%§{​​​&quot;}​​​?\#&#94;°1</span>"^^<http://www.w3.org/1999/02/22-rdf-syntax-ns#XMLLiteral>"

So, the error which was thrown before is now a warning. Maybe for you that property was already set to false and that’s why you did not get the error.

It’s good now that at least we have access to the other properties but I don’t know if it’s a good workaround or not.

But if this is the solution, how should we handle the UnparsableLiterals?

Regards,
Ana

Hi

My suggestion to modify this value is simply for debugging purposes. Of course we need to solve it for real.

So I actually don’t get the exception in my own tests. Even though I am trying to consume the same xml you shared earlier.
I am trying to guess where your problem occurs. The environment property will not help :frowning:

One potential difference between our solutions is that I am running an OSLC Server that also acts as a client. Are you running a pure OSLC Client application? How does your pom.xml file look like.

Can you put a breakpoint at line 1147, and investigate the internals of literalValue ?

Hi Jad,

Yes, I am running this on an OSLC client application. The connection is made using OslcClient.java class from lyo 4.0.0.

We use Gradle instead of Maven, so these are, I think, the lyo-related dependencies that are included in build.gradle:

runtimeOnly 'org.slf4j:slf4j-simple:1.7.30'

//required for lyo version > 4.0.0
api 'org.glassfish.jersey.core:jersey-server:2.30.1'
api 'org.glassfish.jersey.containers:jersey-container-servlet:2.30.1'
api 'org.glassfish.jersey.connectors:jersey-apache-connector:2.30.1'
api 'org.glassfish.jersey.inject:jersey-hk2:2.30.1'

api ('org.eclipse.lyo:oslc-domains:4.0.0')
api('org.eclipse.lyo.clients:oslc-client:4.0.0')

This is the value of ‘literal’ from line 1147:

lexicalForm = ][$%§{​​​"}​​​?#^°1
​wellformed = false
exceptionMesg = Bad rdf:XMLLiteral)

We can see that the ‘wellformed’ flag is then checked in getValue():

Hello again

So I can now reproduce the problem, which helps proceeding.
Here’s a very simple client program that illustrates the problem, with any dependencies on your tools. I read the content from a file, and try to make a ChangeRequest out of it.

run it with mvn compile exec:java -Dexec.mainClass="com.sample.client.OslcMainApplication"

What is interesting is that dcterms:description is handled fine, but not rtc_ext:reason.
This tells me that the problem is not really in Jena or the model, but certainly in our OSLC4J libraries (good to know)

I will investigate this further.

But as a workaround, consider creating a new Java class that extends ChangeRequest and adds a property to explicitly handle this reason property. I have not tested it, but if dcterms:description is working fine, it should work for the new property.

.

1 Like

Hello,

I have added a failing test to Add a failing test case for extended property handling with invalid values by berezovskyi · Pull Request #199 ·. I have traced the problem to two points in the code:

  1. If a property has a corresponding setter (i.e. not an extended property), we fetch the value as literal (without parsing) and then check the type of the setter argument. Because the method accepts a String, we don’t even try to parse the literal as the XML literal. You get a string (but should have really gotten an exception). See JenaModelHelper.java#L779
  2. If there is no setter, an extended property is parsed to the most concrete type we can. See JenaModelHelper.java#L1147, which is when we try to instantiate an instance of an XMLLiteral class and you get an exception (as you should).

Solution: make sure your RDF either uses a string type instead of XML literals OR make sure your XML literal is well-formed and properly escaped.

1 Like

I just updated the test cases with XML input: https://github.com/eclipse/lyo/pull/199

I want to suggest testing XML literal logic with Turtle instead. When you are dealing with RDF/XML, you essentially have to deal with XML serialization of RDF and then on top of that, with the validity of the XML literal encoded inside the RDF/XML. When RDF/XML is parsed (unmarshalled), property values are unescaped. And then, if your property type is XMLLiteral, they need to be valid XML after the raw value is unescaped. On the other hand, you don’t need to escape everything in XML, just things that conflict with XML.

To get a better feel for this, try to convert the valid/invalid Turtle into RDF/XML and back. I see you are using IntelliJ, you may want to give https://plugins.jetbrains.com/plugin/12802-lnkd-tech-editor/ a try:

1 Like