Errorness Transformation of Objects

[FYRichie]

Hi, thank you for this amazing work!

I have a question about the object transformation from canonical to camera space. I have an indoor scene image (from HyperSim) which looks like this:

The results I got from the online demo is pretty good (shown in the image below)

However, when I tested the released code and followed demo_multi_object.ipynb, the result (rendered GIF) looks very differently to the online demo.

Also, when I transform the output glb mesh with the estimated scale, rotation, and translation, the estimation doesn't seem to be accurate (visualized in blender, figure below).

I think it's reasonable when the object is cropped (ex. the towel near the camera, the metal ring), but I don't understand why objects that are complete (ex. the brown box, duck, red bottle, bathtub) have weird spatial relationships. Is there anything that I'm doing wrong.

Again, thanks for your work, and I'll be grateful if you can provide any suggestions.

[JiahaoXia]

It would be great if the authors could clarify the coordinate system they are using when generating the 3D mesh.

[FYRichie]

Hi @JiahaoXia , do you think this is an issue caused by operating on the incorrect coordinate system? Because I didn't do any operation between the mesh generation and video (GIF) rendering. It follows the same code structure as the demo.

[alexsax]

Thanks for the detailed report!

In general, SAM 3D uses pytorch3d camera conventions: https://pytorch3d.org/docs/cameras.

The result in the online demo looks about like I would expect; but the local code appears a little bit worse. Are the same generations used in the rendered GIF and the Blender image? I ask because I think the GLB may be rotated or flipped 180 degrees from the 'object' coordinate space, which is expected.

However, the towel ring in the Blender image looks to be in a different location relative to the towel than it is in the rendered image from demo_multi_object.inpynb

[JiahaoXia]

Hi @JiahaoXia , do you think this is an issue caused by operating on the incorrect coordinate system? Because I didn't do any operation between the mesh generation and video (GIF) rendering. It follows the same code structure as the demo.

I am experiencing a similar issue aligning the 3D mesh with the original image, and I believe it would be helpful if the authors of SAM3D could clarify the coordinate system they use.

[alexsax]

Hi @JiahaoXia,

The predictions are in Pytorch3d camera conventions:

In general, SAM 3D uses pytorch3d camera conventions: https://pytorch3d.org/docs/cameras.

You can see how the transform is applied to take the predicted gaussians from canonical space to the camera space, via this make_scene function that's used in demo_multi_object.ipynb

Do you find that the mesh you are using is not aligned with the gaussians returned during model inference?

[FYRichie]

Hi @alexsax , thanks for your reply! The code I used to render the GIF and to get the mesh is like this:

outputs = []
for mask in all_masks:
    output = inference(img, mask, 42, pointmap)

    # for rendering GIF
    outputs.append(output)

    # for mesh demo in blender
    mesh = output["glb"]
    scale = output["scale"].cpu().squeeze().numpy()
    rotation = np.zeros((4, 4))
    rotation[:3, :3] = quaternion_to_matrix(output["rotation"]).cpu().squeeze().numpy()
    rotation[3, 3] = 1
    translation = output["translation"].cpu().squeeze().numpy()
    mesh.apply_scale(scale).apply_transform(rotation).apply_translation(translation)
    mesh.export(save_path)

# render GIF
scene_gs = make_scene(*outputs)
scene_gs = ready_gaussian_for_video_rendering(scene_gs)
video = render_video(scene_gs, r=1, fov=60, resolution=512)["color"]
# save GIF

I follow the steps in compose_transform in sam3d_objects/data/dataset/tdfy/transform_3d.py to apply the transformations. This function is used in the make_scene function (in SceneVisualizer.object_pointcloud). There might be some other errors in my usage that I haven't found (sorry that I haven't go through your code yet).

My question is: Why is the GLB may be rotated or flipped 180 degrees from the 'object' coordinate space expected?

[Constannnnnt]

Not sure if I understand this correct, but In the to_glb, a GLB which is already Z-up→Y-up rotated, which means

obj["glb"]: mesh vertices are in local object space, already Z-up→Y-up rotated.
obj["translation"], obj["rotation"], obj["scale"]: layout in canonical/camera space before that Z-up→Y-up rotation.

So make_scene and to_glb are in different frames:

Gaussians and layout → canonical / camera frame.
GLB vertices → canonical object frame, but with a fixed Y-up rotation baked in.

My solution is to transform the glb files back to z-up, apply the transformations from the gs outputs and then transform back to y-up.

[alexsax]

@Constannnnnt yes you have the right idea here, that the GLB mesh and Gaussian outputs use different coordinate frames. these are the coordinate frames of the different outputs:

• GLB mesh: Exported in Y-up coordinates (standard for glTF/GLB format). The Z-up → Y-up rotation is applied in postprocessing_utils.py
• Gaussian splats: Stay in Z-up coordinates (no rotation applied in save_ply())
• Transformation parameters (rotation, translation, scale): Computed in the original Z-up/PyTorch3D camera frame

So @FYRichie -- the upshot here is that if you apply the transformation parameters directly to GLB mesh vertices in Blender the mesh vertices have already been rotated to Y-up but the transforms haven't.
Here is an (untested) manual workaround to that might work in Blender

# Manual workaround to convert the GLB mesh coordinate (Y-up → Z-up):
import numpy as np

R_zup_to_yup = np.array([[1, 0, 0], [0, 0, -1], [0, 1, 0]])
R_yup_to_zup = R_zup_to_yup.T

# 1. Convert GLB vertices back to Z-up
vertices_zup = glb_vertices @ R_yup_to_zup
# 2. Apply your transformation in Z-up frame
# 3. Convert back to Y-up
vertices_final = transformed_vertices @ R_zup_to_yup

[alexsax]

Here is a standalone snippet that you might find useful for combining glb meshes into a Y-up scene. The function make_scene_untextured_mesh is analogous to make_scene -- except that the new function returns the scene in Y-up.

  import numpy as np
  import torch
  from copy import deepcopy
  from pytorch3d.transforms import quaternion_to_matrix
  from sam3d_objects.data.dataset.tdfy.transforms_3d import compose_transform

  _R_ZUP_TO_YUP = np.array([[1, 0, 0], [0, 0, -1], [0, 1, 0]], dtype=np.float32)
  _R_YUP_TO_ZUP = _R_ZUP_TO_YUP.T


  def make_scene_untextured_mesh(*outputs, in_place=False):
      import trimesh

      if not in_place:
          outputs = [deepcopy(output) for output in outputs]

      all_meshes = []
      for output in outputs:
          mesh = output["glb"]
          if mesh is None:
              continue

          # GLB is Y-up, transforms are Z-up; convert, apply, convert back
          vertices = mesh.vertices.astype(np.float32) @ _R_YUP_TO_ZUP
          vertices_tensor = torch.from_numpy(vertices).float().to(output["rotation"].device)
          R_l2c = quaternion_to_matrix(output["rotation"])
          l2c_transform = compose_transform(
              scale=output["scale"],
              rotation=R_l2c,
              translation=output["translation"],
          )
          vertices = l2c_transform.transform_points(vertices_tensor.unsqueeze(0))
          mesh.vertices = vertices.squeeze(0).cpu().numpy() @ _R_ZUP_TO_YUP
          all_meshes.append(mesh)

      if not all_meshes:
          return None

      if len(all_meshes) == 1:
          return all_meshes[0]

      return trimesh.util.concatenate(all_meshes)

Usage:
outputs = [inference(image, mask, seed=42) for mask in masks]
scene_mesh = make_scene_untextured_mesh(*outputs)
scene_mesh.export("scene.glb")

And here is the visualization in meshlab

[Constannnnnt]

Thank you @alexsax! I have another question: since the gaussian outputs use a coorindate from the camera space, where if the camera is tilted, the whole scene and objects might be a bit tilted (even though their poses and positions look almost correctly from the camera view). Then, how do we recover a better pose? Take the image you provided for example, the whole scene is a bit tilted and the mattress and the sofa might not be on the same plane and the floating frame might not have a local up close to world up. In this case, do depth and camera parameters help (e.g. a depth-anything 3 model)? I haven't tried this (I used a naive method: use PCA to get principal axes per object and then readjust its orientation), but would love to have some insights from you? Thank you very much!

[alexsax]

I really like P3D's figure for different coordinate frames to anchor the discussion.

So in the pic above I'm interpreting this as as "the model returns poses in the camera space, but how do we transform this to world space" -- is that right? :)

You could try estimating the ground plane, and using that to determine the direction of gravity (i.e. which way is up), and rotating that to align with your preferred axis. We have a simple estimate_plane function in the source code as a starting point.

Even after global rotaitons, the different objects might still be slightly misaligned -- that's more due to the fact that the model makes object predictions one-at-a-time and there can still be errors. My guess is that sort of physics/geometry post-optimization could help here (e.g. DSO). If you end up using a physics-based optimization I'd be very curious about the results.

[Constannnnnt]

Thank you @alexsax, much appreicated! I will let you know if I have some results! Actually I have tried to implement some methods based on SDF, like CAST and Scenethesis, but my results were before sam3d-objects and I focused on the scene language at that time, which was not ideal if the scene was complicated and occluded with many objects. I will try to integrate them if possible and update it here once I have them. Thanks again for your help!