KHR_gaussian_splatting

[weegeekps]

Update: I've split out KHR_gaussian_splatting_spz_2 into it's own PR. There's been many changes since I wrote the summary, and while I'm leaving it at the moment. It will be updated soon.

---

This extension proposal, KHR_spz_gaussian_splats_compression, allows for efficient storage of 3D Gaussian splats data within glTF using the SPZ compression library from Niantic Spatial. The extension is applied to a primitive. The SPZ binary blob is stored as a buffer within the glTF file, and implementations can use the SPZ library to either decompress and then map the compressed Gaussians into placeholder attributes on the primitive or directly decompress into their rendering pipeline if preferred. Content creators have the flexibility to choose to use no Spherical Harmonics, or up to all 3 degrees of spherical harmonics depending on their use case.

We are currently working on an implementation in the CesiumJS engine based on this draft that we hope to have released soon.

[DRx3D]

It appears that this extension is tied to the Niantic Spatial library. Is it necessary to specify a version or other unique identifier to ensure that the desired algorithm and calling sequence is used?

Also note that the license for the Niantic Spatial library is MIT.

[weegeekps]

Is it necessary to specify a version or other unique identifier to ensure that the desired algorithm and calling sequence is used?

Good question. The SPZ library packs along a version number within the binary data that we store in the buffer, so it's unnecessary to have a version number stored in the glTF metadata.

[weegeekps]

Hmm, that's odd. Let me go back through it and see what's up.

[lexaknyazev]

This setup corresponds to the back-to-front order but the "Sorting and Alpha Blending" section explicitly mentions front-to-back that generally requires separate blending factors for color and alpha components as well as a color buffer that preserves alpha values.

Even if all these details are non-normative, they must be self-consistent.

[NorbertNopper-Huawei]

should be gl.blendFunc(gl.GL_ONE, gl.ONE_MINUS_SRC_ALPHA);

[fire]

We are very excited about the standardization of 3D Gaussian Splatting (3DGS) within the glTF 2.0 ecosystem.

@fire and @lyuma have begun an initial implementation of the 3DGS extension using a Godot Engine 4.6 Beta 3 gdscript addon.

We successfully packed degree 3 Spherical Harmonics into the Godot Engine vertex attributes, although we encountered some challenges debugging color reproduction during our tests.

Additionally, we were wondering about the FOV to focal length calculation, currently that is a hard coded value and we are also missing sorting in our Godot 3DGS implementation.

Having more reference material would be beneficial, as there are currently limited samples available for validation.

Implementation: https://github.com/V-Sekai-fire/godot-3dgs

Special thanks to @javagl for providing the glTF 3DGS samples!

[javagl]

encountered some challenges debugging color reproduction during our tests.

I assume that this refers to the splats in the screenshot being nearly white?

(I had a short look at the shader code in the linked repo, but without much more context about what things like CUSTOM or WEIGHTS(?) are, I wouldn't even dare to guess what could be the reason for that)

Coincidentally, I recently added colors to that grid, to illustrate the orientation of each of these "cubes"...

... i.e. the splats at the corners now indicate which corner they are, with (0,0,0)=(nearly)black, (1,0,0)=red, ..., (1,1,1)=white.

_gltf_with_grid_instanced 2026-01-12.zip

Leading to that point;

Having more reference material would be beneficial, as there are currently limited samples available for validation.

I could imagine that people trying to implement this extension would appreciate additional test data.

These "unit cube" and "unit spherical-harmonics" data sets certainly aimed at having a "minimal" data set where the baseline correctness (aka "Seems to work...?!" 🙂 ) can be inspected visually. But they are still just a first shot. If you have ideas for further test data sets that could be useful, just drop me a note.

Dedicated tests with splats with different rotations could be relevant, e.g. something like this:

But that's just a guess (and this should be in a form that can more easily and precisely described and validated (visually)).

[fire]

(I had a short look at the shader code in the linked repo, but without much more context about what things like CUSTOM or WEIGHTS(?) are, I wouldn't even dare to guess what could be the reason for that)

1. There's an implicit COLOR gamma transfer function
2. I suspect there's a coordinate convention problem between webgl2 and godot engine's conventions

Here's the corresponding parts.

The "vertex format" correspond to these sections in the specification.

#version 300 es
precision highp float;

uniform mat4 u_projection_matrix;
uniform mat4 u_view_matrix;
uniform vec2 u_focal_length;
uniform vec3 u_camera_position;

// This uniform controls the SH degree at RUNTIME.
// Set from JavaScript to 1, 2, or 3.
uniform int u_sh_degree;

// All attributes are now always declared. The shader program requires a fixed set of inputs.
in vec2 a_quad_vertex;
in vec3 a_glob_position;
in vec3 a_glob_scale;
in vec4 a_glob_rotation;
in float a_glob_opacity;

// L=0
in vec3 a_sh_0;
// L=1
in vec3 a_sh_1;
in vec3 a_sh_2;
in vec3 a_sh_3;
// L=2
in vec3 a_sh_4;
in vec3 a_sh_5;
in vec3 a_sh_6;
in vec3 a_sh_7;
in vec3 a_sh_8;
// L=3
in vec3 a_sh_9;
in vec3 a_sh_10;
in vec3 a_sh_11;
in vec3 a_sh_12;
in vec3 a_sh_13;
in vec3 a_sh_14;
in vec3 a_sh_15;

out vec3 v_color;
out float v_opacity;
out vec3 v_cov2d_inv_upper;
out vec2 v_center_px;

// --- SPHERICAL HARMONICS CONSTANTS ---
const float SH_C0 = 0.28209479177f;
const float SH_C1 = 0.4886025119f;
const float SH_C2_0 = 1.09254843059f;
const float SH_C2_1 = 0.31539156525f;
const float SH_C2_2 = 0.54627421529f;
const float SH_C3_0 = 0.5900435899f;
const float SH_C3_1 = 2.8906114426f;
const float SH_C3_2 = 0.4570457996f;
const float SH_C3_3 = 0.3731763325f;

We are trying to bitpack the inputs from the gltf2 into our limited vertex format by treating float32s as 16 bit halfs. It's a mess.

uniform vec2 u_focal_length = vec2(2, 2);

// This uniform controls the SH degree at RUNTIME.
// Set from JavaScript to 1, 2, or 3.
uniform int u_sh_degree = 3;

// All attributes are now always declared. The shader program requires a fixed set of inputs.
// VERTEX_ID & 3 // in vec2 a_quad_vertex;
// VERTEX // in vec3 a_glob_position;
// NORMAL // in vec3 a_glob_scale;
// TANGENT // in vec4 a_glob_rotation;
// COLOR.w // in float a_glob_opacity;

// L=0
// COLOR.rgb // in vec3 a_sh_0;
/*
// L=1
in vec3 a_sh_1;
in vec3 a_sh_2;
in vec3 a_sh_3;
// L=2
in vec3 a_sh_4;
in vec3 a_sh_5;
in vec3 a_sh_6;
in vec3 a_sh_7;
in vec3 a_sh_8;
// L=3
in vec3 a_sh_9;
in vec3 a_sh_10;
in vec3 a_sh_11;
in vec3 a_sh_12;
in vec3 a_sh_13;
in vec3 a_sh_14;
in vec3 a_sh_15;
*/

varying vec3 v_color;
varying float v_opacity;
varying vec3 v_cov2d_inv_upper;
varying vec2 v_center_px;

// --- SPHERICAL HARMONICS CONSTANTS ---
const float SH_C0 = 0.28209479177f;
const float SH_C1 = 0.4886025119f;
const float SH_C2_0 = 1.09254843059f;
const float SH_C2_1 = 0.31539156525f;
const float SH_C2_2 = 0.54627421529f;
const float SH_C3_0 = 0.5900435899f;
const float SH_C3_1 = 2.8906114426f;
const float SH_C3_2 = 0.4570457996f;
const float SH_C3_3 = 0.3731763325f;

vec2 a_quad_vertex = vec2( 
float(VERTEX_ID & 1) * 2.0 - 1.0,
float(VERTEX_ID & 2) * 1.0 - 1.0);
vec3 a_glob_position = VERTEX;

// We hacked in the parameters for guassian splats here
// See the next section for the translation.

vec2 UVy = unpackHalf2x16(floatBitsToUint(UV.y));
vec2 UV2y = unpackHalf2x16(floatBitsToUint(UV2.y));
vec2 CUSTOM0y = unpackHalf2x16(floatBitsToUint(CUSTOM0.y));
vec2 CUSTOM1y = unpackHalf2x16(floatBitsToUint(CUSTOM1.y));
vec2 CUSTOM2y = unpackHalf2x16(floatBitsToUint(CUSTOM2.y));
vec2 WEIGHTSy = unpackHalf2x16(floatBitsToUint(BONE_WEIGHTS.y));
vec2 WEIGHTSw = unpackHalf2x16(floatBitsToUint(BONE_WEIGHTS.w));
vec2 CUSTOM3y = unpackHalf2x16(floatBitsToUint(CUSTOM3.y));

// Variables from the original list from @weegeekps

vec3 a_glob_scale = vec3(unpackHalf2x16(floatBitsToUint(UV.x)), UVy.x);
float a_glob_opacity = COLOR.w;
vec3 a_sh_0 = COLOR.rgb / SH_C0;
vec3 a_sh_1 = vec3(unpackHalf2x16(floatBitsToUint(UV2.x)), UV2y.x);
vec3 a_sh_2 = vec3(unpackHalf2x16(floatBitsToUint(CUSTOM0.x)), CUSTOM0y.x);
vec3 a_sh_3 = vec3(CUSTOM0y.y, unpackHalf2x16(floatBitsToUint(CUSTOM0.z)));
vec3 a_sh_4 = vec3(unpackHalf2x16(floatBitsToUint(CUSTOM0.w)), UVy.y);
vec3 a_sh_5 = vec3(unpackHalf2x16(floatBitsToUint(CUSTOM1.w)), UV2y.y);
vec3 a_sh_6 = vec3(unpackHalf2x16(floatBitsToUint(CUSTOM1.x)), CUSTOM1y.x);
vec3 a_sh_7 = vec3(CUSTOM1y.y, unpackHalf2x16(floatBitsToUint(CUSTOM1.z)));
vec3 a_sh_8 = vec3(unpackHalf2x16(floatBitsToUint(CUSTOM2.x)), CUSTOM2y.x);
vec3 a_sh_9 = vec3(CUSTOM2y.y, unpackHalf2x16(floatBitsToUint(CUSTOM2.z)));
vec3 a_sh_10 = vec3(unpackHalf2x16(floatBitsToUint(CUSTOM2.w)), WEIGHTSw.x);
vec3 a_sh_11 = vec3(unpackHalf2x16(floatBitsToUint(CUSTOM3.x)), CUSTOM3y.x);
vec3 a_sh_12 = vec3(CUSTOM3y.y, unpackHalf2x16(floatBitsToUint(CUSTOM3.z)));
vec3 a_sh_13 = vec3(unpackHalf2x16(floatBitsToUint(CUSTOM3.w)), WEIGHTSw.y);
vec3 a_sh_14 = vec3(unpackHalf2x16(floatBitsToUint(BONE_WEIGHTS.x)), WEIGHTSy.x);
vec3 a_sh_15 = vec3(WEIGHTSy.y, unpackHalf2x16(floatBitsToUint(BONE_WEIGHTS.z)));

Edited:

3. Also render_mode blend_mix, depth_draw_never, cull_disabled, skip_vertex_transform, unshaded, fog_disabled; does a lot of hardcoded magic in Godot Engine.
4. We ran out of space so rotation (quaternion) was encoded as a Basis (3x3 matrix) encoded as mat3(TANGENT, BINORMAL, NORMAL)
5. Dedicated tests with splats with different rotations could be relevant, e.g. something like this This would help debug the coordinate conventions!

[javagl]

Dedicated tests with splats with different rotations could be relevant, e.g. something like this

This would help debug the coordinate conventions!

Yeah... the coordinate systems. I've been going back and forth on some of that (and just yesterday opened javagl/JSplat#6 ). Let's hope that for glTF, this does not cause toooo many headaches, given that the coordinate system in the source is clear and unambiguous, and it's ""only"" about converting this to the coordinate system of the engine. (Skipping some considerations about spherical harmonics here...)

The example from the screencap was created with some preliminary experiments, and I hope that I can get the I/O aligned for all formats one day. Right now, the bar here is pretty low: All variants look the same when loaded in my viewer, and obviously, this might be a case where an error in the reader and the writer just cancel each other out 😬 (some variants are at least "upside down" in BabylonJS...). However, the current output of that "rotations2D" part is attached here, even though that is "underspecified" if it's supposed to be something that people can rely on for tests...

rotations2D_2026-01-12.zip

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

✅ Ronald909
✅ weegeekps
❌ keyboardspecialist
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[weegeekps]

I just pushed some updates to the spec and schema:

• Updated the TOC.
• Addressed multiple review comments.
• Resolved all resolved conversation threads.
• Removed the broken shaders.

I will be adding the corrected shaders in a little while. I believe I found an issue with the texture atlasing that I want to resolve.

[fire]

We were looking for a test splat to check for incorrect sort order

[javagl]

@fire When an implementation is using a sort order that is reversed, then nearly every data set* will look completely wrong. I'm not entirely sure what a test case could look like in order to visually validate the sort order, somewhat "holistically". I.e. I can hardly imagine a data set where one can see 100 splats, and see whether exactly two of them are just swapped in terms of their sort order. But I could imagine some test cases that could be interesting.
(A test data set that is related to this question may be the one from CesiumGS/cesium#13118 (comment) , but that may be too specific (and it's only the glTF+SPZ one there)).
I'll try to add some possible test cases here a bit later today.

---

* Except for the ones that only contain a single (or very few non-overlapping) splats - which largely applies to the ones that I posted here until now...

[javagl]

Something like this, maybe...?

This is just a bunch of "disc-shaped" splats, arranged to visually check the sort order...

depthSortingTest 2026-01-16.zip

Maybe there should be dedicated tests with "long, spikey" splats that are poking through "disc-shaped" splats...?

[weegeekps]

I've added some clarity to the Spherical Harmonic section and removed the shader examples. To ensure we are covering everything, I've also included details about the Condon-Shortly phase which newcomers to Spherical Harmonics may not know about.

[javagl]

(Minor glitches in the last table, just mentioning).

Is the plan to add some reference shader later? It's probably not necessary from a normative point of view, but could be helpful if someone wants to implement things from scratch. If so, it could make sense to align whatever is supposed to be added here with what is done in the glTF-Sample-Viewer. (That is: It would be nice if the shaders here could (basically) "be" the ones that are used in the sample viewer, give or take some minor details)

[weegeekps]

Is the plan to add some reference shader later?

No, we discussed this earlier. We can't include a shader normatively anyhow, and the truth is that shader implementations are going to have enough engine and use-case specific needs that an example will probably be misleading. There are quite a lot of really solid production examples of 3D Gaussian Splatting shaders that are open source, so for the purposes of the shaders it's best if people use those as examples.