| type |
title_main |
title_follow |
author1 |
author2 |
author3 |
author4 |
author5 |
author6 |
link_1_url |
link_1_color |
link_1_text |
link_2_url |
link_2_color |
link_2_text |
show_news_updates |
news_update_1 |
news_update_2 |
article_v2 |
SceneSense: |
Diffusion models for 3D Occupancy Synthesis from Partial Observation |
Alec Reed |
Brendan Crowe |
Doncey Albin |
Lorin Achey |
Bradley Hayes |
Chris Heckman |
|
yellowgreen |
GitHub |
|
blue |
ArXiv |
true |
|
2024-06-30 SceneSense accepted to IROS 2024! |
<style>
body {
margin: 0;
padding: 0;
position: relative;
overflow: auto;
}
#cube-container {
position: fixed;
top: 0;
left: 0;
width: 100%;
height: 100%;
z-index: -1;
perspective: 1000px;
pointer-events: none; /* Ensure the container doesn't block interaction */
}
.cube {
position: absolute;
transform-style: preserve-3d;
}
.cube-face {
position: absolute;
width: 100%;
height: 100%;
opacity: 0.8;
border: 1px solid black; /* Add black border to each face */
}
</style>
<script>
// Gen a rand int bw min and max (inclusive)
function getRandomInt(min, max) {
return Math.floor(Math.random() * (max - min + 1)) + min;
}
// Gen rand color
function getRandomColor() {
var colors = ['#FF0000', '#00FF00', '#FFFFFF']; // Red, Green, White, for different occupancies
return colors[Math.floor(Math.random() * colors.length)];
}
// Draw cube at a random position on the sides of the page
function drawRandomCube() {
var cube = document.createElement('div');
cube.classList.add('cube');
var cubeSize = 20 + 30 * Math.random(); // Random size between 20 and 50px
// Set cube size and random rotation
cube.style.width = cubeSize + 'px';
cube.style.height = cubeSize + 'px';
cube.style.transform = 'rotateX(' + getRandomInt(-180, 180) + 'deg) rotateY(' + getRandomInt(-180, 180) + 'deg)';
var cubeColor = getRandomColor();
// Create cube faces
var faces = ['front', 'back', 'left', 'right', 'top', 'bottom'];
faces.forEach(function (face) {
var faceElement = document.createElement('div');
faceElement.classList.add('cube-face');
faceElement.style.backgroundColor = cubeColor;
switch (face) {
case 'front':
faceElement.style.transform = 'translateZ(' + (cubeSize / 2) + 'px)';
break;
case 'back':
faceElement.style.transform = 'rotateY(180deg) translateZ(' + (cubeSize / 2) + 'px)';
break;
case 'left':
faceElement.style.transform = 'rotateY(-90deg) translateZ(' + (cubeSize / 2) + 'px)';
break;
case 'right':
faceElement.style.transform = 'rotateY(90deg) translateZ(' + (cubeSize / 2) + 'px)';
break;
case 'top':
faceElement.style.transform = 'rotateX(90deg) translateZ(' + (cubeSize / 2) + 'px)';
break;
case 'bottom':
faceElement.style.transform = 'rotateX(-90deg) translateZ(' + (cubeSize / 2) + 'px)';
break;
}
cube.appendChild(faceElement);
});
// Randomly place the cube on the sides of the window borders
var side = Math.random() < 0.5 ? 'left' : 'right';
var x = side === 'left' ? getRandomInt(0, 0.1 * window.innerWidth - cubeSize) : getRandomInt(0.9 * window.innerWidth, window.innerWidth - cubeSize);
var y = getRandomInt(0, window.innerHeight - cubeSize);
cube.style.left = x + 'px';
cube.style.top = y + 'px';
document.getElementById('cube-container').appendChild(cube);
}
// Create cube container
var cubeContainer = document.createElement('div');
cubeContainer.id = 'cube-container';
document.body.appendChild(cubeContainer);
// Using the function above, draw 10 random 3D cubes on the page
for (var i = 0; i < 10; i++) {
drawRandomCube();
}
</script>
We present SceneSense, a novel generative 3D diffusion model for synthesizing 3D occupancy information from observations. SceneSense uses a running occupancy map and a single RGB-D camera to generate predicted geometry around the platform, even when the geometry is occluded or out of view. The architecture of our framework ensures that the generative model never overwrites observed free or occupied space, making SceneSense a low risk addition to any robotic planning stack.
Our occupancy in-painting method ensures that observed space remains intact while integrating SceneSense predictions. Drawing inspiration from image inpainting techniques like image diffusion and guided image synthesis, our approach continuously incorporates known occupancy information during inference. To execute occupancy in-painting, we select a portion of the occupancy map for diffusion, generating masks for occupied and unoccupied voxels. These masks guide the diffusion process to modify only relevant voxels while introducing noise at each step. This iterative process, depicted below, enhances scene predictions' accuracy while preventing the model from altering observed geometry.
@misc{reed2024scenesense,
title={SceneSense: Diffusion Models for 3D Occupancy Synthesis from Partial Observation},
author={Alec Reed and Brendan Crowe and Doncey Albin and Lorin Achey and Bradley Hayes and Christoffer Heckman},
year={2024},
eprint={2403.11985},
archivePrefix={arXiv},
primaryClass={cs.RO}
}
