Floating-Point Precision Causes Unexpected Intersection Errors

[GuoJiechang]

I encountered an issue when performing intersection tests with triangles where small floating-point differences lead to unexpected errors.

Triangle values to reproduce the error:

// Case that causes intersection error
__t1.a.set(0, 50.000000000000014, 50);
__t1.b.set(0, 49.999999999999986, -50);
__t1.c.set(100, 50.000000000000014, 50);
__t2.a.set(17, 50, -24.543842315673828);
__t2.b.set(18, 50, -23.144481658935547);
__t2.c.set(18, 49, -23.595157623291016);

However, if I round the floating-point values to exact integers, the issue disappears:

// Case without error
__t1.a.set(0, 50, 50);
__t1.b.set(0, 50, -50);
__t1.c.set(100, 50, 50);
__t2.a.set(17, 50, -24.543842315673828);
__t2.b.set(18, 50, -23.144481658935547);
__t2.c.set(18, 49, -23.595157623291016);

Expected Behavior:
Small floating-point differences should not cause the intersection result to be wrong if the triangles are effectively the same in real-world positioning.

Potential solution:
in src/utils.ts, function orient3D(), line 40. Round the determinate to an integer.
const det = Math.round(_matrix4.determinant());

But I'd like to know if it is the best solution and if there are any potential issues.

Thank you!

[GuoJiechang]

I just realized that the potential solution is not good, it will make (-1,1) to 0, which is not expected.
Another solution is to change the value of EPSILON, to 1e-8.

After further investigation, the artifact is caused by
CrossIntersect.intersectPlane(t2.a, t2.b, t1.a, _n1, _i1); // k
where t2.a and t2.b are almost on the plane(P1) formed by t1.a and _n1. which are supposed not to happen because t2.a should be the only vertex on the positive side of the plane P1. vector (t2.a,t2.b) should not parallel to P1.

In the source code:

    private static intersectPlane (
        a: Vector3,
        b: Vector3,
        p: Vector3,
        n: Vector3,
        target: Vector3
    ) {
        _u.subVectors(b, a);
        _v.subVectors(a, p);
        const dot1 = n.dot(_u);
        const dot2 = n.dot(_v);
        _u.multiplyScalar(-dot2 / dot1);
        target.addVectors(a, _u);
    }

dot1 is 3.9772221974154755e-16
dot2 is -6.975779546017357e-15
-dot2/dot1 is 17.53932568954895
which causes _u to be in the wrong position, resulting in the wrong intersection result.