XMLStreaming doesn't convert XML character references (in attributes)

[westnordost]

XMLStreaming doesn't convert XML character references. Only tested for attributes, don't know about other places in the XML where XML character references could occur.

This unit test would currently fail:

@Test fun `writer converts character references in attributes`() {
  val buffer = StringBuilder()
  val writer = xmlStreaming.newWriter(buffer)
  writer.startTag("", "test", null)
  writer.attribute(null, "attr", null, "With\nlinefeed")
  writer.endTag("", "test", null)

  assertEquals(
    "<test attr=\"With&#10;linefeed\"></test>", // or hexadecimal: &#xA
    buffer.toString()
  )
}

instead, the result looks like this:

<test attr="With
linefeed"></test>

Additional information

This unit test does not fail

@Test fun `reader converts character references in attributes`() {
  val source = "<test attr=\"With&#10;linefeed\"></test>" // or hexadecimal: &#xA
  val reader = xmlStreaming.newReader(source)
  
  forEach { 
    if (it == START_ELEMENT && localName == "test") {
      assertEquals(
        "With\nlinefeed",
        getAttributeValue(null, "attr")!!
      )
    }
  }
}

[westnordost]

Tested with another character reference, < <-> <. That works fine.

The W3C XML Validator also doesn't find anything wrong with

<test attr="With
linefeed"></test>

I tested also this, this also does not fail:

@Test fun `reader understands newlines in attributes`() {
  val source = "<test attr=\"With\nlinefeed\"></test>"
  val reader = xmlStreaming.newReader(source)
  
  forEach { 
    if (it == START_ELEMENT && localName == "test") {
      assertEquals(
        "With\nlinefeed",
        getAttributeValue(null, "attr")!!
      )
    }
  }
}

So, maybe it is working as intended? I thought I read something in the XML spec that newlines must be stripped (in a step before the parsing), but maybe that only referred to newlines outside of elements?

[westnordost]

To answer my own question, I think section 3.3.3 Attribute-Value-Normalization of the XML spec defines this:

[...] Before the value of an attribute is passed to the application or checked for validity, the XML processor MUST normalize the attribute value by applying the algorithm below [...]

[...] All line breaks MUST have been normalized on input to #xA [...]

[...] For a white space character (#x20, #xD, #xA, #x9), append a space character (#x20) to the normalized value [...]

So, it looks to me that there are two bugs:

1. XmlReader should parse attr in <test attr=\"With\nlinefeed\"></test> as "With linefeed" (replacing any number of newlines and whitespaces with a single whitespace)

2. XMLWriter should convert any whitespaces, newlines, ... in the source string also to character references to preserve them in the XML

Note: The spec reads a bit confusing in this section, so I am not 100% sure of what the spec-conformant behavior should be.

[pdvrieze]

@westnordost Another place to look is the xmlschema specification (part 2 - datatypes). It actually specifies how normalization works (and there is code in the library - iirc. in the xmlschema module) that implements this.

As to the spec.:

• 3.3.3 Step 1 refers to the fact that at higher level all line breaks must be normalized.
• Step 3 means that:
  • whitespace characters directly in the input should be normalized into spaces
  • entity references (not supported by the generic parser, except the default ones) are replaced by their content and then processed to replace the reference
  • character references are replaced by their content
• Reference is made to the attribute being cdata (or not), as we don't use DTD's we cannot know that

This means that you are correct, and serialization should write the newline as character reference. I have added appropriate changes to the codebase. Note that this does not do the CDATA replacement as there are use cases where the values must be read as a different value.