Tokyo-PointCloud-PointNetpp-VAE

Project is not closed, there will be improvements in the near future. Here's a useful method which should produce clickable references in any Markdown editor:

At the end of each header, add an empty anchor with a chosen name — e.g. . At the start of the document, list the headers with a link to their anchors — e.g. Foo. So this:

Table of contents

1. Data
2. Model
   

Introduction to the problem at hand

We find classic buildings boring, so what if we could create a AI algorithm to add funky elements to existing building design, or better yet give us new buildings from nothing.
Enter Variational Autoencoders (VAEs). We started out thinking about wake-sleep networks, however VAEs are more modern, and they are easier to find resources about.

Data

We are using the Tokyo 3D Point Cloud Dataset. It is a dataset of 3D point clouds of buildings in Tokyo. The dataset is available at: https://3dview.tokyo-digitaltwin.metro.tokyo.lg.jp/ There are multiple version of the dataset, the one I downloaded came in the following form:
dataset:
    - ku (district)1
         - data
             - data0.b3dm (it is a 3D mesh of a small part of the district, lets call them blocks, included many buildings inside)
             - data1.b3dm
         - tileset.json
     - ku (district)2
     - ...

dataset to master_b3dm

I copied all of the b3dm files for all blocks under the same folder using dataset2master.py so iteration is simpler

dataset-b3dm-from-chiyoka-ku.mp4

master_b3dm to master_glb

Our end goal is to have a dataset of 3D point clouds, but there is no real way (at least that I know of) converting from b3dm directly to .ply so we need to convert the b3dm files to glb files.
I did this using Cesium GS's 3d-tiles-tools, you can see the code under b3dm2glb.py

master_glb to blocks_mesh_ply

My b3dm2ply and glb2ply script failed due to a processing issue with the trimesh library, so I used aspose 3ds glb to ply converter.

blocks_mesh_ply to buildings_mesh_ply

I used block2building.ipynb to seperate buildings in each block so we could feed individual buildings to our NN.
I first tried seperating the mesh into connected components and saving them. Didn't work, it instead gave me a single wall for a building. The results:

https://github.com/user-attachments/assets/5d4b216f-4fa4-4ced-b4e4-d3edee2dd1ef
I tried using the trimesh library's fill holes function but it failed.
I then clustered faces based on the center of each triangle (the building block of a mesh) and used sklearn's DBSCAN algorithm to cluster them. Didn't work. The results:

submeshes_mesh_obj.mp4

Finally, I clustered based on edge proximity. To find the min edge distance between 2 components, we created a distance matrix for every possible combination of componenti.vertexa and componentj.vertexb and found the distance for all combinations.
We used DBSCAN again with epsilon = 0.5 and joined the walls that are clustered together and saved them to buildings_mesh_ply
Worked. The results:

https://github.com/user-attachments/assets/a086d58b-de5c-403e-82fd-2dfc1a092018

buildings_mesh_ply to buildings_pointcloud_ply

I used the mesh2pointcloud_ply.py to convert the buildings to point clouds by using open3D's poisson sampling dots on the surfaces of the mesh.

building_pointcloud_ply_building1.mp4

Model

I tried 3 version of VAEs, all of them combined with PointNet++ architecture. This is the architecture

1) Classic VAE with Chamfers Distance and KL Divergence
The results (the red points are the input, green is the predicted output):

normal_VAE.mp4

2. Disentangled VAE with Beta=3, Chamfers Distance and KL Divergence
   The results:

B3_VAE.mp4

3. Classic VAE with Chamfers Distance, KL Divergence, and a loss function I desinged (Distance Between 2 Closest Points)
   The results:

VAE_with_Distance_Loss.mp4

To Dos and Observations: - We see that first model's output is very scattered, custom lost function in the 3rd helps us out, but not good enough. Modify the last layer to improve the point distribution. - In the second model, we see that the model predicts a unit cube. Train models without standarization. - Train models with data augmentation. - Make your model bigger if necessary.