3D Raycasting in 3.1

[Sjiun]

Description 📝

Hi, first of all thank you for all your work on this amazing project!

Usecase: I am developing an XR-App. I have to raycast not only from a screen (Desktop or in Augmented Reality on mobile) but also from a Headset plus controller. However, SimpleRaycaster.castRay() only takes a 2d-pointer as a position parameter while in 3D space a 3D origin plus a 3D direction is required to create a ray.

Edit: Since custom ThreeJS raycasters are not supported in 3.1, it became impossible to perform a raycast in a 3D environment (i.e. when in AR or VR on mobile or headset ).

Suggested solution 💡

Even though it has not worked for me as expected in the past, the most straightforward solution seems to be a re-implementation of the function SimpleRaycaster.castRayFromVector().
Up until this commit, the method existed. Now it is commented out.
Trying to fully understand the way the raycasting works internally, I got lost a couple of files in between connection handlers and multithreading controllers. So I'm unfortunately not confident I can provide more concrete feedback on how to re-implement SimpleRaycaster.castRayFromVector().

Alternative ⛕

Back in 2.4 we used to be able to work around this by utilizing the three.js Raycaster with something like this:

    const raycaster = components.get(OBC.Raycasters).get(world).three;
    raycaster.set(touchOrigin, directionFromCamera);
    const intersects = raycaster.intersectObjects(scene.children, true);

However, this thread states that the new fragments won't work with three.js raycaster anymore since geometry buffers are dropped after GPU upload.
And indeed, when upgrading to 3.1, the raycaster.intersectObjects() will throw errors like this for many objects (and also prevents copying/stringifying them):

TypeError: can't access property 0, this.array is undefined

(This happens irrespective of whether the model is loaded from a .ifc or .frag file.)

I have tried working around this by manually reconstructing the dropped meshes for later raycast use - as described in the thread mentioned above. But on first glance it seems like this either requires duplicate meshes for items or more involved replacement of all item's meshes (upon raycast, I still want to infer the item from the mesh and associated ifc-properties).

Additional context ☝️

Maybe I am missing something but for now I will be switching back to 2.4 in order to use the threejs raycast. (EDIT: docs for previous versions are here: https://thatopen.github.io/engine_past-docs/3.0.x/Tutorials/Components/Core/Raycasters)

Thank you all for your time and effort!

Validations ✅

• Read the docs.
• Check that there isn't already an issue that requests the same feature to avoid creating a duplicate.

[nikobojs]

I had the exact same bug, and spent 3 full work days on getting raycast working in fragments version 3.1.7. It was so tricky, so I ended up creating a new empty ts/vite project and copy paste the whole example code (from https://docs.thatopen.com/Tutorials/Components/Core/Raycasters). Which worked.

It did not work when I passed the mesh-array argument to the raycaster methods, but luckily, it is not a requirement for our use-case (it now works with caster.castRay(), without arguments).

In my case (nuxt / vue 3), I had to NOT save the raycaster singleton in a vue proxy/ref object, and I also needed to create the Fragments manager object like described in the raycaster example: fragments = components.get(OBC.FragmentsManager);.

Also, I think I found that the OrthoPerspectiveCamera was the only thing that worked.

Sorry for not giving a more precise solution, but maybe some of these tricks can help you. 🙏

Here is my working raycast helper:

async function castRawRay(): Promise<{
  modelIdMap: OBC.ModelIdMap;
  intersection:
    | THREE.Intersection<THREE.Object3D<THREE.Object3DEventMap>>
    | undefined;
  data: FRAGS.ItemData;
}> {
  const modelIdMap: OBC.ModelIdMap = {};
  let data: FRAGS.ItemData = {};
  const caster = casters.get(world!);
  const intersection = await caster.castRay();
  if (!intersection) {
    return { modelIdMap, intersection: undefined, data };
  }

  const inter = intersection as THREE.Intersection<
    THREE.Object3D<THREE.Object3DEventMap>
  > & { fragments?: FragmentsModel; localId?: number };

  if (inter.fragments?.modelId && inter?.localId) {
    modelIdMap[inter.fragments.modelId] = new Set([inter.localId]);
  }

  const modelId = Object.keys(modelIdMap)[0];
  let model: FragmentsModel | undefined = undefined;
  if (modelId && fragments.list.get(modelId)) {
    model = fragments.list.get(modelId)!;
    const [modelData] = await model.getItemsData([...modelIdMap[modelId]]);
    data = modelData;
  }

  return {
    modelIdMap,
    intersection,
    data,
  };
}

[Sjiun]

Thank you for sharing. If I understand correctly, your code works with the default's 2D mouse position since you're not passing any arguments to the castRay() method.

However, as described in our XR usecase above, we can't use the (2D) mouse pointer since in AR or VR we are inside 3D space. Not only looking at a 2D desktop view of the 3D viewer. So we have to point/cast at something from some 3D origin with a 3D direction.