Camera coordinate to Three.js coordinate.

[See71]

1. Seeking help with converting from the camera coordinate system to the Three.js coordinate system.
2. All coordinates are represented in the camera coordinate system. After setting the plane, origin, and scale, they are converted to world coordinates. Then, the world coordinates are transformed into the Three.js coordinate system.
3. So before converting the coordinates to Three.js, I did not perform any axis swapping or sign inversion on the x, y, or z coordinates.
4. In your project, the transformation was applied in advance, as shown _camera_read. 、 acquire_floor 、 set_origin
5. Just as you said, “you don’t **** know why,” this issue also confuses me.👊
6. Theoretically, before rendering to the frontend, the plane is first defined in the camera coordinate system to obtain the rotation part of the toWorldMatrix. Then, the origin is set in the world coordinate system to complete the translation part of the toWorldMatrix. After that, the scale is set in the world coordinate system. Finally, the world coordinates are transformed into the Three.js coordinate system.
7. However, I’m not exactly sure how to convert the coordinates into the Three.js coordinate system. As shown in the image, I drew a rectangle on the ground — I expected it to lie flat on the plane, but it doesn’t.
8. I’m looking for help — seeking some experience and methods to properly handle this.🚩
   

[Luo-ouL]

Hello,
I did not use the original project's extrinsic calibration code; instead, I rewrote it myself based on the same calibration principles. Therefore, I am not very familiar with that part of the original project's code. Nevertheless, I hope my possibly inaccurate explanation can still be helpful to you:

I think your issue might be related to the order of extrinsic calibration. The correct sequence should be as Joshua Bird described, divided into three steps:

1. First, capture a single infrared light source to calculate the rotation matrix and the direction of the translation vector between cameras.
2. Next, capture two infrared light sources to determine the scale (the magnitude of the translation vector between cameras).
3. Finally, capture a moving single infrared light source on a plane (the ground) to determine the world coordinate system (i.e., the plane and the origin).

From your description, I guess your sequence might be "plane, origin, and scale."

The key issue is that to obtain the world coordinate system, the prerequisite is that the positioning with camera 0 (i.e., the camera coordinate system you mentioned) as the "world coordinate system" must be accurate. Only then can you obtain the correct world coordinate system under the camera coordinate system (i.e., plane and origin). Since the first step only determines the direction of the translation vector between cameras, the second step is needed to determine the magnitude of the translation vector between cameras. This way, you obtain a complete pose relationship between the cameras, ensuring accurate positioning under the camera coordinate system. Only after completing the first two steps can you proceed to the third step.

Regarding the "Three.js coordinate," I am not sure what kind of coordinate system this is, as I did not use the original project's React frontend code. However, I think that if you can correctly obtain the world coordinate system, converting it to the Three.js coordinate system should not be too difficult.

As for why the first step only gives the direction of the translation vector but not its magnitude, here’s my understanding:

Taking two cameras as an example (in fact, the original code also determines camera poses pairwise). First, cameras 0 and 1 capture a moving single infrared light source within the common field of view—about a dozen valid images are sufficient. After this step, we obtain the "image points." Then, the two sets of image points are passed to cv.findFundamentalMat() to obtain the Fundamental Matrix (F matrix), which represents the relationship between image points. Next, the F matrix and the camera intrinsics are passed to cv.sfm.essentialFromFundamental() to obtain the Essential Matrix (E matrix), representing the relationship between points in the camera coordinate system. Finally, the E matrix is passed to cv.sfm.motionFromEssential() to get the possible rotation matrix (R) and the direction of the translation vector (t) between the cameras.

These steps can only yield an initial estimation of the rotation matrix (R) and the direction of the translation vector (t); they do not give the length of t. The default output length of t is 1, which is a unit vector in the t direction. This makes sense because, after converting the image points to the camera coordinate system using the camera intrinsics, you essentially have many rays. The "world point" is the intersection of these rays. However, capturing many world points from the camera's perspective does not inform the camera about the distances between points, only their relative positions. Therefore, we need two fixed-distance points (e.g., 0.2 m apart) to calibrate the scale, essentially telling the camera how far 0.2 m is.

Before watching the tutorial video recommended by Joshua Bird, I was not familiar with camera calibration. I found the video very informative, and perhaps after watching it, you will understand how to solve your problem as well.

[Luo-ouL]

Congratulations! 🎉

[jyjblrd]

Great stuff! Thank you @Luo-ouL

[See71]

Gotta give you props for open-sourcing it！ @jyjblrd 👍
By the way, I really liked the visuals in your project — and even more so the ones in your paper DVM-SLAM.
Would you mind sharing what tool you used to create them? I’d really appreciate a recommendation.🤝🤝

[jyjblrd]

Thanks! There's not much to my figures, I just use Figjam, Matplotlib and Google Slides haha. My the video clips & audio for my videos are recording using my iPhone and edited using Final Cut Pro (but I used to just use iMovie).

IMO, nothing fancy is need to make good videos and visualizations. A strong vision & coherent message is most important.

[See71]

Totally agree👍. Your presentation style is super clean — it’s just really pleasant to go through.✅ Thanks so much🌟

[See71]

Your answers are always so high-quality and detailed. It makes me feel once again how coherent and rigorous your thinking is.👍👍I'll take your advice and completely restructure the world coordinate system to get a clean one. Then I'll convert it to Three.js coordinates.💪💪I'm working on the changes right now. Just wait for my good news!✅✅Appreciate ！👏👏