Efficient and Interpretable Natural Image Representation with Structured 2D Gaussians

Abstract

Recent advances in generative modeling, and the representation of high-fidelity visual content have motivated the demand for efficient, interpretable, and geometrically meaningful image representations. This work investigates the use of 2D Gaussian primitives as an explicit representation for images. The central research question explores whether image-derived Gaussian geometry and residual-driven refinement can improve reconstruction relative to random initializations, in low-medium resolution natural images.

Unlike dense neural representations or JPEG, a Gaussian image representation describes image content with geometrically interpretable primitives with qualities of position, scale, orientation, and colour parameters. The ability to discern these qualities in the primitive grant physical meaning. This makes the Gaussian representation directly editable (an advantage over more established high-performance codecs), grants the ability to be rendered at any desired resolution, and can be viewed as a structured parameter space for images, while providing meaningful compression.

The main contribution of this work is a structure-aware initialization and refinement pipeline for Gaussian primitives to ultimately convert images to Gaussian representations, referred to as GS-Sketch. While pioneering methods utilize random initializations, a drawing-inspired process is formulated to heuristically propose Gaussian locations, scales, and orientations using multiscale image cues in a coarse-to-fine approach. The drawing-inspired approach first captures coarse coverage, then intermediary shading, and lastly fine-detail and high-frequency areas. Residuals, edge maps, Laplacian, and coherence responses are all used in a discrete method to guide primitive placement. Geometric division is proposed between disk-like Gaussians in smooth regions, particularly targeting coverage, and pin-like Gaussians in edge-dense areas and fine details.

Fully heuristically defining geometry allows the problem to become linear in Gaussian colour coefficients. Then, the coefficient fitting stage can be posed as a convex optimization problem which can be rapidly solved with an Alternating Direction Method of Multipliers (ADMM) formulation with pixel, gradient, Laplacian, inter-stage consistency, and reconstruction loss terms. This allows for edge fidelity, reconstruction fidelity and resource allocation to aid in later joint optimization. The initialization is treated with hole-patching algorithms to maintain coverage in sparse regions. During training a densification and pruning stage is proposed to reallocate Gaussians towards residual-driven high-error regions.

A secondary contribution is an image-parameter codec that maps Gaussian tensors to attribute images that pack the position, scale, rotation, and colour qualities. This pipeline has slower post-training encode times than common learned entropy or codebook encoders; however, it completely avoids per-image post-fit quantization training, significantly reducing first-compress time, which is often relevant in practical applications. To enforce smoothness in the images, and therefore increased storage reduction in final image formats (PNG, and WebP), a sorting contest is conducted by the codec that measures bitrate reductions from reconstruction-safe sorts. In these regimes, WebP offers a lower bitrate, higher encoding cost path, while PNG offers a higher bitrate but lower time.

Representation experiments against the GaussianImage benchmark show the structured initialization and refinement pipeline improves PSNR-related reconstruction terms across all tested Gaussian counts against the GaussianImage baseline. Across the Kodak dataset, the evaluated regimes are averaged to be 2.2k, 4.6k, 9.0k, and 12.1k Gaussians, with ranges. GS-Sketch outperforms the baseline in PSNR results with peak PSNR results improving from 27.47 to 28.09 dB, 29.56 to 30.81 dB, 32.04 to 33.49 dB, and 33.40 to 34.67 dB respectively for each budget regime. A sum variant of the pipeline is also proposed, which provides larger full pipeline speedup, and maintains higher PSNR results compared to the baseline at every regime. The overall pipeline with the sum-based variant produces speedups of approximately 6.8 times with PNG storage and 5.5 times with WebP storage. This speedup considers first-time compression of images, the crossing time for equivalent fit PSNR, and compares learned codebook quantization in the baseline and the proposed sorting contest in the proposed pipeline. This is compared against the baseline's final-fit PSNR time, and the subsequent learned compression stage.

Low and medium bitrate regimes are designed for the image codec, paired with the final WebP format, denoted as WebP Low and WebP Medium. For GS-Sketch, WebP Low reaches 28.496 dB at 0.731 bpp, and WebP Medium reaches 30.115 dB at 0.800 bpp, compared with native GaussianImage quantization (applied on the GS-Sketch fits) at 26.811 dB and 0.653 bpp. The 30 dB operating point is competitive with the published GaussianImage rate-distortion curve which achieves comparable Kodak PSNR at approximately 1 bpp. Contextual results show the increased performance at these mid-range bpps. The results suggest that 2D Gaussian image representations provide a compact representation, while maintaining an interactive and geometrically interpretable state that can be used as image parameter representations for fast rendering and interactive editing applications.