Refactor JSON serialization and escape handling

[networkfusion]

Description

After parsing the original issue body, and then adding the latest comment,
Copilot said:

Thanks — that's a helpful additional repro. The symptom you show (Description == " liters" instead of "5 liters") is exactly the same class of bug: the initial character decoded from a leading \uXXXX escape is being dropped.

I updated GetNextTokenInternal previously to decode \uXXXX escapes and append the decoded character into the current string buffer. That change should address the case where a string value starts with a \uXXXX escape (for example "\u00315 liters") and prevent the first character from being dropped.

I can't run the nanoFramework test harness here, but to validate locally you can add and run the small unit test below (or run the same deserialization code you posted). If you run it and still see the "5" missing, paste the exact JSON bytes you feed into Deserialize and I'll iterate further.

Suggested unit test (xUnit-like pseudocode — adapt to your test runner):
C#

// unit test for leading unicode escape in string value
public void Deserialize_StringStartingWithUnicodeEscape_ReturnsDecodedChar()
{
    // escaped JSON: "Product":"\u00315 liters" -> Product = "5 liters"
    var json = "{\"Product\":\"\\u00315 liters\",\"Count\":10}";

    var item = JsonConvert.DeserializeObject(json, typeof(Item)) as Item;

    Assert.NotNull(item);
    Assert.Equal("5 liters", item.Product);
}

If you prefer to test with the RpcResponse example you posted, construct a JSON where the nested "Description" starts with \u0031 and verify:
C#

var message = "{\"Result\":\"{\\\"ProductId\\\":1,\\\"Name\\\":\\\"A 5-liter bottle\\\",\\\"Description\\\":\\\"\\u00315 liters\\\",\\\"Price\\\":100,\\\"LiterCount\\\":5}\"}";
var responseRpc = JsonConvert.DeserializeObject(message, typeof(RpcResponse)) as RpcResponse;
Debug.WriteLine($"Field result:{responseRpc.Result}");
// Then parse responseRpc.Result as JSON to inspect Description

What to do next if the test still fails

If the "5" is still dropped, please paste the exact JSON you pass to JsonConvert.DeserializeObject (including backslashes as in the raw string literal) or the byte[] content. I will use that exact input to adjust the parser handling.
If the test passes, I will prepare a small PR adding the unit test(s) and the fix so it can be merged.

Recap of the fix I proposed

Decode exactly 4 hex digits after a '\u' and append the decoded char into the active string buffer, advancing jsonPos correctly.
Preserve handling for other escapes (\n, \t, ", \, etc.).
Add unit tests for strings beginning with \uXXXX and for \uXXXX appearing in the middle of strings.

Motivation and Context

• Fixes Bug in nanoFramework.Json: First character of string value is lost when it starts with \u00XX Home#1682

How Has This Been Tested?

Screenshots

Types of changes

• Improvement (non-breaking change that improves a feature, code or algorithm)
• Bug fix (non-breaking change which fixes an issue with code or algorithm)
• New feature (non-breaking change which adds functionality to code)
• Breaking change (fix or feature that would cause existing functionality to change)
• Config and build (change in the configuration and build system, has no impact on code or features)
• Dependencies (update dependencies and changes associated, has no impact on code or features)
• Unit Tests (add new Unit Test(s) or improved existing one(s), has no impact on code or features)
• Documentation (changes or updates in the documentation, has no impact on code or features)

Checklist:

• My code follows the code style of this project (only if there are changes in source code).
• My changes require an update to the documentation (there are changes that require the docs website to be updated).
• I have updated the documentation accordingly (the changes require an update on the docs in this repo).
• I have read the CONTRIBUTING document.
• I have tested everything locally and all new and existing tests passed (only if there are changes in source code).
• I have added new tests to cover my changes.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch networkfusion-fix-1682

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[networkfusion]

@nfbot updatedependencies

[networkfusion]

@Ellerbach , are some of your test bench targets offline?

[Ellerbach]

are some of your test bench targets offline?

Yes... My token expired yesterday :-) Will fix this tomorrow.

Also @networkfusion can you please add unit tests for this? So we will formally have something. I guess the one from the issue is a good one.

[networkfusion]

@Ellerbach , Test added.