Gaussian Splatting, Three Years In: What Shipped, What Still Breaks

The research problem turned into a delivery problem

When Kerbl, Kopanas, Leimkühler and Drettakis presented 3D Gaussian Splatting at SIGGRAPH 2023, the headline was speed: rendering a radiance field at interactive rates on ordinary rasterization hardware, where NeRF-style methods had needed a multi layer perceptron evaluated dozens of times per pixel (the original paper, project page). That argument is settled. Nobody spends 2026 writing a volumetric ray marcher to look at a captured room.

What replaced it is duller and harder. The interesting engineering now sits in three places: how a splat scene is stored and exchanged, how it gets sorted and blended inside a frame that other things also want, and what a battery powered headset can actually pay for. Three events this year show all three moving at once, and they change what “we support Gaussian splatting” means in a product spec.

This piece is the state of play, written for people who have to ship something rather than cite something.

What the rasterizer does, and where the cost hides

The representation is worth restating precisely, because most of the limitations follow from it.

A scene is a set of Gaussians. Each has a position (a mean), a covariance that says how it is stretched and oriented in space, an opacity, and spherical harmonic coefficients that encode colour as a function of view direction. Degree three harmonics give 48 coefficients per Gaussian, which is why a naive file is so large and why view-dependent effects like reflections survive without a separate material model.

Rendering projects every Gaussian into screen space, sorts the contributions in depth order, and alpha blends front to back, either per pixel or per tile. There is no network to evaluate, no ray marching, no per sample density query. That is the whole reason it runs at frame rate, and it is also why the two real costs are fill rate from overdraw and the depth sort itself. Sorting tens of millions of primitives every frame is the part that does not parallelize as nicely as the projection does.

Training follows the same shape. Points come from structure from motion, usually COLMAP. The optimizer renders from a training view, compares against the photograph with an L1 term plus a structural similarity term, backpropagates through the rasterizer, and periodically runs adaptive density control: clone Gaussians where reconstruction is thin, split them where detail is missing, prune the ones whose opacity has collapsed. A scene that took hours to fit with a neural field now converges in minutes on one GPU.

If you want the scale of that, the billion Gaussian experiments are a reasonable place to start. The practical fact is that training stopped being the bottleneck somewhere around 2024. Delivery never was anything else.

PLY was never a format, and 2026 finally fixed it

For two years the de facto container was a PLY file: raw float buffers of position, scale, rotation, opacity and harmonics, with no statement of how to interpret them. It loaded because every viewer happened to agree on the same ad hoc layout. It said nothing about colour space, kernel shape, projection method, or sort order. It also stored everything uncompressed, which is how a single room ends up at hundreds of megabytes.

Niantic Spatial’s contribution was to treat that as a codec problem rather than a research problem. SPZ shipped in late 2024, described at the time as “JPG for 3D Gaussian splats”, and the pitch was simple: the same scene at roughly a tenth of the PLY size, small enough to move off a phone (the format repository). Adoption has been quiet and broad. Niantic reports that Photoshop users alone created around 800,000 SPZ files in a two month window earlier this year, which is the kind of number that only appears once a format has stopped being a choice and started being a default.

SPZ 4, released in May, is the version that matters for anything large. The original wrapped the whole payload in one gzip stream, so encode and decode ran on a single core, and multi million point captures spent minutes waiting. Version 4 replaces that with six parallel ZSTD streams, one per attribute (positions, colours, scales, rotations, alphas, harmonics), plus a table of contents so a loader can find each attribute without scanning the file first. The old 10 million point cap is gone (release notes).

Their benchmark numbers are worth quoting because they are the ones a build pipeline feels. On an 8.5 GB PLY with 34 million points, encode went from 3 minutes 26 seconds to 1 minute 8 seconds, with files 2.5 percent smaller. On a 7.1 million point scene, size dropped from 254 MB to 128 MB and time to first render fell from 3,450 ms to 1,660 ms. The second number is the one that decides whether a splat scene can be streamed into an app the way a texture is.

Research compression goes much further. HAC, from ECCV 2024, treats the unorganised anchors as a point cloud with spatial structure, models that structure with a binary hash grid, and entropy codes the quantised attributes against it. The reported result is more than 75 times smaller than vanilla 3DGS at better fidelity, and more than 11 times smaller than Scaffold-GS (paper). Self Organizing Gaussians takes the other useful angle: sort the attributes into a 2D grid during training, enforce local smoothness, then hand the result to an ordinary image codec (project page). Both are lossy, and both move work into decode, which is a real cost on a headset with a fixed memory budget.

Then the piece that makes this a format story rather than a codec story. In February, Khronos published a release candidate for KHR_gaussian_splatting, an extension that stores splats inside glTF 2.0 as a point primitive with position, rotation, scale, opacity and harmonics, plus interpretation rules for rendering splats rather than triangles. The specification defines a kernel, a colorSpace, an optional projection and an optional sortingMethod, is deliberately designed to be extended, and allows a viewer without support to fall back to showing the data as a point cloud (press release, specification). The repository now records the extension as complete and ratified.

Two things about that are more interesting than the announcement. First, compression was left out of the base extension on purpose and handed to vendor extensions, with Niantic’s SPZ and Qualcomm’s L-GSC already proposed. Qualcomm proposing a mobile compression scheme is not a neutral data point: the mobile case is the XR case, and the vendor that supplies the silicon in most standalone headsets is positioning itself at the codec layer. Second, the contributor list and the early adopters tell you who this is actually for: Cesium and Bentley, Esri, Niantic, NVIDIA, Huawei, Autodesk, with CesiumJS, ArcGIS, Scaniverse and XGRIDS shipping against the release candidate. Khronos is also coordinating with the Alliance for OpenUSD so glTF and OpenUSD schemas stay aligned.

That is the difference between a demo and an asset. A splat that lives in a glTF scene graph can carry a transform, sit next to terrain and photogrammetry, appear in a digital twin alongside meshes, and be animated by the same tooling. The XR consequence is indirect but real: the content you will be asked to display will increasingly arrive as a layer in a scene, not as a standalone viewer app with its own camera controls.

What a headset actually pays

The number that matters on a standalone device is not frames per second in a demo, it is where the frame budget goes when the scene gets heavy. A September study from Alpen-Adria-Universität Klagenfurt is the most useful public measurement here (Ghanbari, Amirpour and Timmerer, paper page).

They built an interactive tool on Quest 3 that exposes render resolution scale, splat density and frame rate cap, then swept 180 conditions: six resolution scales from 0.5 to 1.0, ten density levels from 10 to 100 percent, three camera distances, ten repetitions each, all captured through Meta’s OVR Metrics Tool at a 72 Hz target.

Three findings matter for anyone planning a splat feature.

Resolution scale dominates GPU cost, because pixel count grows quadratically while the Gaussian count does not change. Halving render scale is not a 2x saving in quality terms, it is a 4x saving in pixels, and the measurement shows that dominating everything else.

Reducing splat density is only an effective lever when GPU time is already close to the 72 Hz budget. If you have headroom, throwing away Gaussians buys nothing and costs fidelity. If you are over budget, it is the fastest thing to cut after resolution, and it is cheaper than giving up resolution at close range.

Power draw tracks total GPU work per unit time, not cost per frame, which produces the counterintuitive result that a lower resolution scale can draw more power by letting the frame rate climb. On a headset that is thermally limited, that is the difference between a comfortable 40 minute session and a warm one. It also means frame rate caps are a power feature, not just a latency feature.

Two platform facts sit on top of that. Stereo doubles the pixel work, so the numbers above are the generous single view case, and Meta’s own Spatial SDK now exposes Gaussian splats as a scene feature for loading and positioning captured content (documentation). Splats on Quest are no longer a plugin experiment; they are a supported content type sitting next to meshes, which puts them in the same memory and bandwidth conversation as everything else in the frame. The framebuffer budget deep dive is the depressing arithmetic of that conversation.

The browser path is moving in parallel, and it is the clearest demonstration of how much of the cost was ever really the sort. SuperSplat Editor 3.0 rebuilt its renderer and data model on WebGPU: projection, frustum culling, compaction, depth sorting and draw submission all run on the GPU, with a compute projector, a GPU radix sort and indirect drawing, so the CPU sort worker is gone. On a 4.4 million splat scene loaded from a 990 MB PLY, idle JavaScript heap fell from 1,557 MB to 105 MB, and rendering a 1080p video went from 1,673 MB to 142 MB (release post).

They also shipped the pragmatic hack worth stealing: a stochastic alpha mode that renders Gaussians opaque and depth tested with per fragment coverage, which is order independent, so interacting with a huge scene does not pay for sorting. When the scene settles, a fully sorted frame is drawn. Exactness when still, speed when moving. That is a reasonable model for how splats will be handled inside XR compositors too, and it echoes the same argument about who owns the frame that runs through the spatial compositor fight.

For XR developers on the web stack, the practical note is that this performance is WebGPU only. The editor requires it, and it is the same compute path that makes neural and splat rendering plausible in a headset browser at all, which the WebGPU and WebGL comparison covers in more detail.

What still breaks

Four problems have not been solved, and each one has a product consequence.

Lighting. Spherical harmonics encode how a Gaussian looks from a direction, not how it responds to light. A captured room cannot receive new lighting, so a splat sofa will not pick up the glow of a virtual lamp, and a virtual object will not cast a shadow onto a splat floor. This is the same blending problem the compositor discussion runs into: something has to decide how real and virtual surfaces share illumination, and today that something is usually a compromise.

Inverse rendering. Recovering albedo, normals and materials so a capture can be relit is an active field with real progress and honest numbers. GS-IR (CVPR 2024) optimises in around 13 minutes and reaches about 20.4 dB PSNR on relighting; R3DG takes about 56 minutes for 26.6 dB; GeoSplatting (ICCV 2025) reports 9 minutes and 29.5 dB by guiding the decomposition with geometry (paper). Those are research numbers on research scenes, minutes per scene rather than milliseconds, and none of it is in a shipping runtime yet.

Geometry and physics. Gaussians do not form a watertight surface, so there is no mesh to collide with, no navmesh to path on, and no clean depth to occlude a virtual object behind a captured chair. Surface aligned variants recover meshes at some fidelity cost, and reconstruction pipelines increasingly output both a splat layer for looks and a mesh for behaviour. That pairing is probably the shape of production content for a while.

Editing and dynamics. Removing an object from a splat scene leaves a hole, because the capture has no notion of objects. Selection tools help an artist clean a scene, they do not decompose it. Dynamic capture is further along than most people assume, with 4D Gaussian approaches rendering video driven scenes in real time (CVPR 2024), but storage and compute scale with duration, and on device reconstruction is still research territory, with SLAM systems like Gaussian Splatting SLAM pointing at where tracking and mapping are going.

None of these are reasons to avoid the technique. They are the reason a splat is a layer in a scene rather than a replacement for the scene.

What to do if you ship XR this quarter

Concretely, from the evidence above.

Do not ship a PLY. Use SPZ, or a glTF asset with KHR_gaussian_splatting once your engine supports it, and keep the uncompressed original out of the build. If your pipeline generates captures weekly, the SPZ 4 encode speed matters more than the last few percent of size.

Budget resolution scale first, then density, and cap the frame rate on thermal grounds rather than treating 72 or 90 as a target to exceed. Measure with OVR Metrics at a fixed rate instead of trusting an average, because a scene that averages 85 fps and stutters is worse than one that holds 72.

Sort less. Either sort on the GPU with a compute pass, or use an order independent approximation while the user is moving, and take the quality hit where it is invisible.

Plan for the lighting mismatch early. If captured content sits next to lit virtual objects, decide now whether splats are treated as unlit backdrops or as something the compositor relights, because retrofitting that decision later rewrites your render graph. How passthrough is composited is the constraint you are working inside, and eye tracking is where the foveation savings will have to come from once resolution scale runs out.

And expect the content to arrive as data. The format war is over in the sense that matters: there is a ratified baseline in glTF and at least two serious compression schemes attached to it. The next fight is over who composites the result, which is the same fight the rest of XR is having this year.

---

Sources

• Kerbl, Kopanas, Leimkühler, Drettakis: 3D Gaussian Splatting for Real-Time Radiance Field Rendering (SIGGRAPH 2023) and project page
• Niantic Spatial: SPZ 4 is here, leaner, faster, and more future-proof (May 2026)
• Niantic: spz format repository
• Khronos: Khronos Announces glTF Gaussian Splatting Extension (February 3, 2026)
• Khronos: KHR_gaussian_splatting specification
• Chen et al.: HAC, Hash-grid Assisted Context for 3D Gaussian Splatting Compression (ECCV 2024)
• Morgenstern et al.: Self Organizing Gaussians, Compact 3D Scene Representation via Self-Organizing Gaussian Grids (ECCV 2024)
• Ghanbari, Amirpour, Timmerer: 3D Gaussian Splatting Rendering Performance Trade-offs on the Meta Quest 3 (2026)
• Meta: Use Gaussian splats in the Spatial SDK
• PlayCanvas: New in SuperSplat, Editor 3.0 rebuilt on WebGPU
• Liang et al.: GS-IR, 3D Gaussian Splatting for Inverse Rendering (CVPR 2024)
• Ye et al.: GeoSplatting, geometry guided Gaussian Splatting for physically based inverse rendering (ICCV 2025)
• Wu et al.: 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (CVPR 2024)
• Matsuki et al.: Gaussian Splatting SLAM (CVPR 2024)