Questions about RaycastCarver

[Aczheng-cai]

Thank you very much for open-sourcing your work. I encountered some issues while trying to reproduce it. Without modifying any part of the code, the system crashes at the beginning because there are no GS points. After repeated debugging, I found a few problems:

In sensor_.model.backProject(pixel_f, &ray_dir_C);, it seems that the camera intrinsics used in this function are somehow modified. As a last resort, I tried hard-coding them with constants for debugging, but it still didn’t work.

if (map_.template pointToVoxel<se::Safe::On>(ray_pos_W, voxel_coord)) — this condition is never satisfied. Moreover, the input point coordinate ray_pos_W has strange values: [ -6.2095e+07, -5.93713e+12, 1.75429e+13 ].

I suspect the root cause might be that voxel_keys.size is 0, which results in no GS points being generated.

Could you please advise me on how to fix these issues so that the program can run properly?
`
template<typename MapT, typename SensorT>
std::vectorse::OctantBase* RaycastCarver<MapT, SensorT>::operator()()
{
TICK("fetch-frustum")
// Fetch the currently allocated Blocks in the sensor frustum.
// i.e. the fetched blocks might contain blocks outside the current valid sensor range.
std::vectorse::OctantBase* fetched_block_ptrs = se::fetcher::frustum(map_, sensor_, T_WS_);
TOCK("fetch-frustum")

TICK("create-list")
se::OctantBase* root_ptr = octree_.getRoot();

const int num_steps = ceil(config_.band / (2 * map_.getRes()));

const Eigen::Vector3f t_WS = T_WS_.topRightCorner<3, 1>();

#pragma omp declare reduction(merge : std::set se::key_t : omp_out.insert(omp_in.begin(), omp_in.end()))
std::setse::key_t voxel_key_set;

// #pragma omp parallel for reduction(merge : voxel_key_set)
for (int x = 0; x < depth_img_.width(); ++x) {
for (int y = 0; y < depth_img_.height(); ++y) {
const Eigen::Vector2i pixel(x, y);
const float depth_value = depth_img_(pixel.x(), pixel.y());
// Only consider depth values inside the valid sensor range
if (depth_value < sensor_.near_plane || depth_value > (sensor_.far_plane + config_.band * 0.5f)) {
continue;
}

        Eigen::Vector3f ray_dir_C;
        const Eigen::Vector2f pixel_f = pixel.cast<float>();
        // std::cout<<pixel_f<<std::endl;
        sensor_.model.backProject(pixel_f, &ray_dir_C);
        std::cout<<ray_dir_C<<std::endl;
        std::cout<<((depth_value * ray_dir_C).homogeneous()).template head<3>()<<std::endl;

        const Eigen::Vector3f point_W = (T_WS_ * (depth_value * ray_dir_C).homogeneous()).template head<3>();
        const Eigen::Vector3f reverse_ray_dir_W = (t_WS - point_W).normalized();

        const Eigen::Vector3f ray_origin_W = point_W - (config_.band * 0.5f) * reverse_ray_dir_W;
        const Eigen::Vector3f step = (reverse_ray_dir_W * config_.band) / num_steps;

        Eigen::Vector3f ray_pos_W = ray_origin_W;
        for (int i = 0; i < num_steps; i++) {
            Eigen::Vector3i voxel_coord;

            if (map_.template pointToVoxel<se::Safe::On>(ray_pos_W, voxel_coord)) {
                const se::OctantBase* octant_ptr = se::fetcher::block<typename MapT::OctreeType>(voxel_coord, root_ptr);
                if (octant_ptr == nullptr) {
                    se::key_t voxel_key;
                    se::keyops::encode_key(voxel_coord, octree_.max_block_scale, voxel_key);
                    voxel_key_set.insert(voxel_key);
                }
            }
            ray_pos_W += step;
        }
    }
}
// Allocate the Blocks and get pointers only to the newly-allocated Blocks.
std::vector<key_t> voxel_keys(voxel_key_set.begin(), voxel_key_set.end());
TOCK("create-list")

TICK("allocate-list")
std::vector<se::OctantBase*> allocated_block_ptrs = se::allocator::blocks(voxel_keys, octree_, octree_.getRoot(), true);
TOCK("allocate-list")
std::cout<<voxel_keys.size()<<std::endl;
TICK("combine-vectors")
// Merge the previously-allocated and newly-allocated Block pointers.
allocated_block_ptrs.reserve(allocated_block_ptrs.size() + fetched_block_ptrs.size());
allocated_block_ptrs.insert(allocated_block_ptrs.end(), fetched_block_ptrs.begin(), fetched_block_ptrs.end());
TOCK("combine-vectors")
return allocated_block_ptrs;

}`

[goldoak]

The internal RaycastCarver is not the issue. First, please verify that your dataset structure is correct according to the guidelines in the Download Datasets section. Also, ensure that you have modified the relevant configuration file as outlined in the Usage Example section. If the corresponding reader cannot load the images into memory, then there are no GS points to be assigned.

[Aczheng-cai]

The internal RaycastCarver is not the issue. First, please verify that your dataset structure is correct according to the guidelines in the Download Datasets section. Also, ensure that you have modified the relevant configuration file as outlined in the Usage Example section. If the corresponding reader cannot load the images into memory, then there are no GS points to be assigned.

This is the structure of my data：
├── room0
│   ├── cameras.json
│   ├── info_semantic.json
│   ├── results
│   ├── room0_mesh.ply
│   └── traj.txt
This is the structure of yaml：
`%YAML:1.2

SPDX-FileCopyrightText: 2021 Smart Robotics Lab, Imperial College London, Technical University of Munich

SPDX-FileCopyrightText: 2024 Smart Robotics Lab, Technical University of Munich

SPDX-License-Identifier: CC0-1.0

map:
dim: [15, 15, 15]
res: 0.01

data:

tsdf

truncation_boundary_factor: 8
max_weight: 100

sensor:
width: 1200
height: 680
fx: 600.0
fy: 600.0
cx: 599.5
cy: 339.5
near_plane: 0.4
far_plane: 6.0

reader:
reader_type: "replica"
sequence_path: "/home/zwc/dataset/Replica/room0/"
ground_truth_file: "/home/zwc/dataset/Replica/room0/traj.txt"
fps: 0.0
drop_frames: false
verbose: 0

app:
enable_ground_truth: true
optim_params_path: "/home/zwc/下载/GSFusion/parameter/optimization_params_replica.json"
ply_path: "/home/zwc/下载/GSFusion/point_cloud"
mesh_path: "/home/zwc/下载/GSFusion/mesh"
slice_path: ""
structure_path: ""
integration_rate: 1
rendering_rate: 1
meshing_rate: 0
max_frames: -1
log_file: "/tmp/log.tsv"
`
I visualized the images and depth maps, and I made sure that the input data exists.

[goldoak]

Please debug the app/src/main.cpp file. Ensure the reader successfully reads images in L142-152. Also, please paste your information in an appropriate format for easy reading.

[Aczheng-cai]

Please debug the app/src/main.cpp file. Ensure the reader successfully reads images in L142-152. Also, please paste your information in an appropriate format for easy reading.
The read_ok return value in the code segment you mentioned is OK. I apologize for any inconvenience caused by the formatting issues in my reply.

[Aczheng-cai]

gaussianModel.Get_xyz().sizes() is 0

[goldoak]

If you're still unable to identify the problem, I recommend rebuilding your code and environment from scratch, or trying it on a different laptop. Unfortunately, I don't have a better solution at the moment.

[aravindhankrishnan]

Thank you for this implementation, really appreciate it!

I observe the same issue on Replica dataset room0. I debugged a bit and identified one cause.

In singleres_tsdf_gs_updater_impl.hpp, center_data.weight is always 0 and this code gets executed

  if (center_data.weight != 1) {
           continue;
   }

which means that scales, positions, and colors are not populated. This leads to valid_positions being empty and the model not getting updated.

I debugged further on why center_data.weight is always 0. I notice that map_.getData() in map_impl.hpp is hitting this condition regularly (around line 70)

     if (!pointToVoxel<Safe::On>(point_W, voxel_coord)) {
            return DataType();
        }
   

Inside pointToVoxel() the contains(point_W) condition is always false because point_W is producing large values beyond the bounds (lower 0 and upper 20).

Tracking this back to sensor_.model.backProject(p2d, &center); in singleres_tsdf_gs_updater_impl.hpp, where center is getting populated, I notice that cu_ in PinholeCamera<DISTORTION_T>::backProject is set to 1 and cv_ is set to 0. However, when PinholeCamera object is initialized in the constructor, I do notice (also validated by printing) that cu_ and cv_ are initialized properly.

There seems to be some corruption happening somewhere.

I also notice that sensor.T_BS in main.cpp contains garbage values after initialization and I had to set it manually to identity.

   se::PinholeCamera sensor(config.sensor);
       sensor.T_BS = Eigen::Matrix4f::Identity();

I have noticed two instances of the sensor object getting corrupted (a) sensor.model.cu_ / cv_ (b) sensor.T_BS

I disabled optimizations in CMakeLists.txt and built in Debug mode. The issue still persists. Hopefully, this info is helpful and looking forward to hearing from you.

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O0 -g" CACHE STRING "C++ flags" FORCE)
set(CMAKE_C_FLAGS   "${CMAKE_C_FLAGS}   -O0 -g" CACHE STRING "C flags"   FORCE)

[goldoak]

Thank you for your efforts in debugging this issue. Could you let me know if you consistently encounter this problem in the Replica dataset room0 scene? Also, do you notice the same issue occurring in other datasets or scenes? The value corruptions you mentioned appear to stem from the Supereight2 mapping backbone. Now that the Supereight2 has undergone several updates, I will try to find some time to incorporate the new modifications into GSFusion.

[aravindhankrishnan]

Yes, I see this consistently every time I run the code. I confirmed that this issue happens on TUM datasets too.

[goldoak]

This may be an issue related to a specific environment. I have successfully run experiments on all Replica scenes and tested the code on the TUM-RGBD dataset using the system configurations listed in the README.md. If possible, consider compiling GSFusion on another machine to see if the problem persists.

[aravindhankrishnan]

What is your Ubuntu version and GPU you are using? I'm not clear how a different machine would resolve this issue.

[aravindhankrishnan]

A request : Is it possible to provide a docker image?

[goldoak]

Hi, I've added a new Dockerfile that builds and runs GSFusion in a controlled, reproducible environment. You can follow the guide here in README to build the Docker image. The running command can be found in Usage Example.