Enable graph-based type checking, parallel optimizations and ILX gen in deterministic build

[majocha]

This as a follow up to #18998:

• adds tests for added dependency graph supported scenarios: script compilation mode and misordered .fs .fsi files
• updates naming and adds comments wrt to the above
• enables graph based type checking in deterministic builds
• enables parallel optimizations in deterministic builds
• enabled parallel ilx gen in deterministic builds

Necessary changes to enable determinism:

• sort things when writing PDB
• update NiceNameGenerator to produce stable names
• update generation of array literal types (multiple delayed gens would place then under "private implementation" concurrently, resulting in non-deterministic order)

There are no additional tests for determinism, it is currently tested only by running the test-determinism script locally and in the CI.

One issue needed investigation: type extensions with mismatched type argument names can break determinism:

// in one file
type LexBuffer<'Char>

// in another file
type LexBuffer<'char> with

Either char or Char will end up in the IL.

This is related to #15287 and needs work: #19033

[github-actions]

❗ Release notes required

---

✅ Found changes and release notes in following paths:

Change path	Release notes path	Description
src/Compiler	docs/release-notes/.FSharp.Compiler.Service/11.0.0.md

[majocha]

Typar name not always picked up from the source (?)

But wth is going on here? Looks like a type parameter 'Char gets randomly converted to char.

[majocha]

Good thing is I got exactly the same diff locally as in the CI, so it is already deterministic in this regard 😁

[majocha]

Ok so it's not a type parameter but actual char type. It just randomly gets emitted either as Char or char.

Edit: Nope it was because of not unified type argument names between type declaration and extensions.
NOTE: It needs an actual separate fix.

[majocha]

Hallelujah.

Ilx gen is still disabled. Only optimization and graph checking works.

portable pdb test needs updating, too.

[majocha]

TODO:

fsharp/src/Compiler/CodeGen/IlxGen.fs

Line 2337 in 12efe3b

CompilerGeneratedName("T" + string (newUnique ()) + "_" + string size + "Bytes") // Type names ending ...$T<unique>_37Bytes

and

fsharp/src/Compiler/CodeGen/IlxGen.fs

Line 2757 in 12efe3b

let ilFieldName = CompilerGeneratedName("field" + string (newUnique ()))

Apart from stabilizing these generated names, the order of of generated IL also needs to be deterministic. Specifically, in FCS.dll generated types in PrivateImplementationDetails are sometimes in different order.

Possibly relevant:

fsharp/src/Compiler/CodeGen/IlxGen.fs

Lines 2389 to 2394 in 12efe3b

	member _.GenerateRawDataValueType(cloc, size) =
	// Byte array literals require a ValueType of size the required number of bytes.
	// With fsi.exe, S.R.Emit TypeBuilder CreateType has restrictions when a ValueType VT is nested inside a type T, and T has a field of type VT.
	// To avoid this situation, these ValueTypes are generated under the private implementation rather than in the current cloc. [was bug 1532].
	let cloc = CompLocForPrivateImplementationDetails cloc
	rawDataValueTypeGenerator.Apply((cloc, size))

[majocha]

Ready for review, cc @nojaf

[majocha]

changes in this file are 100% copilot 😅 , so, caveat emptor

[majocha]

copilot generated, but adjusted by hand.

[nojaf]

Can't say this feels cathartic just yet.
You danced around a problem without context, don't do that.
Nobody in the future will have any use for that.

PS: I do holy appreciate your effort here, sorry for sounding grumpy.

[nojaf]

Why is this being skipped? There is no context explaining why this is necessary for deterministic builds. If this is a workaround, please acknowledge it and document its rationale and implications so users know what to expect.

[majocha]

Yes, this is a workaround.

I'll add a comment, but this probably needs splitting into a separate issue and properly digging in.

[majocha]

I was not 100% sure what was ging on here, but now I have a very small repro.
This is a race condition in a very specific scenario when we have two or more extensions of the same type in two independent (from graph checking pov) files.

If each extension "redefines" the type var name, we have a race during parallel checking. In effect some of the extensions will end up with a randomly wrong typar name.

This happens in FCS.dll with LexBuffer<_> which has such multiple independent extensions.

The workaround here sucks, because 1) it is not needed for sequential compilation, 2) this is still buggy for non-deterministic parallel compilation.

[nojaf]

Either that needs to be fixed, or we should reconsider the whole typar renaming. I don't remember all the details, but if it's only a cosmetic change, we should consider reverting it in favour of compilation speed.

[majocha]

I opened #19033 for this. This is possibly not hard to fix for someone more familiar with checking and it's phases. Totally alien to me, unfortunately.

[majocha]

It was a very focused change, I commented out the relevant code fragment with a link to the issue. Also added skip to the tests.

[majocha]

Can't say this feels cathartic just yet. You danced around a problem without context, don't do that. Nobody in the future will have any use for that.

PS: I do holy appreciate your effort here, sorry for sounding grumpy.

I don't mind grumpy 😄 I was expecting it. This PR is another chaotic aftermath of a fight. Your input is the best in bringing order from the chaos, I noticed.