# Instant Neural Graphics Primitives with a Multiresolution Hash Encoding THOMAS MÜLLER, NVIDIA, Switzerland ALEX EVANS, NVIDIA, United Kingdom CHRISTOPH SCHIED, NVIDIA, USA ALEXANDER KELLER, NVIDIA, Germany Trained for 1 second 15 seconds 1 second 15 seconds reference 60 seconds ![](images/2022_InstantNGP/a0c2ee786850b825374f7a5b327dc911764a07fec5f5204ec0b895b43e8248d5.jpg) Fig. 1. We demonstrate instant training of neural graphics primitives on a single GPU for multiple tasks. In Gigapixel image we represent a gigapixel image by a neural network. SDF learns a signed distance function in 3D space whose zero level-set represents a 2D surface. Neural radiance caching (NRC) [Müller et al. 2021] em lo s a neural network that is trained in real-time to cache costl li htin calculations. Lastl , NeRF [Mildenhall et al. 2020] uses 2D ima es and their camera poses to reconstruct a volumetric radiance-and-density field that is visualized using ray marching. In all tasks, our encoding and its eficient implementation provide clear benefits: rapid training, high quality, and simplicity. Our encoding is task-agnostic: we use the same implementation and hyperparameters across all tasks and only vary the hash table size which trades of quality and performance. Tokyo gigapixel photograph ©Trevor Dobson (CC BY-NC-ND 2.0), Lego bulldozer 3D model ©Håvard Dalen (CC BY-NC 2.0) Neural graphics primitives, parameterized by fully connected neural net-works, can be costly to train and evaluate. We reduce this cost with a versatile new input encoding that permits the use of a smaller network without sac- rificing quality, thus significantly reducing the number of floating point and memory access operations: a small neural network is augmented by a Authors’ addresses: Thomas Müller, NVIDIA, Zürich, Switzerland, tmueller@nvidia. com; Alex Evans, NVIDIA, London, United Kingdom, alexe@nvidia.com; Christoph Schied, NVIDIA, Seattle, USA, cschied@nvidia.com; Alexander Keller, NVIDIA, Berlin, Germany, akeller@nvidia.com. multiresolution hash table of trainable feature vectors whose values are op- timized through stochastic gradient descent. The multiresolution structure allows the network to disambiguate hash collisions, making for a simple architecture that is trivial to parallelize on modern GPUs. We leverage this parallelism by implementing the whole system using fully-fused CUDA ker nels with a focus on minimizing wasted bandwidth and compute operations. We achieve a combined speedup of several orders of magnitude, enabling training of high-quality neural graphics primitives in a matter of seconds, and rendering in tens of milliseconds at a resolution of 1920×1080. CCS Concepts: • Computing methodologies → Massively parallel algorithms; Vector / streaming algorithms; Neural networks. Additional Key Words and Phrases: Image Synthesis, Neural Networks, En-codings, Hashing, GPUs, Parallel Computation, Function Approximation. ## ACM Reference Format: Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Trans. Graph. 41, 4, Article 102 (July 2022), 15 pages. https://doi.org/10. 1145/3528223.3530127 ## 1 INTRODUCTION Computer graphics primitives are fundamentally represented by mathematical functions that parameterize appearance. The quality and performance characteristics of the mathematical representation are crucial for visual fidelity: we desire representations that remain fast and compact while capturing high-frequency, local detail. Func- tions represented by multi-layer perceptrons (MLPs), used as neural graphics primitives, have been shown to match these criteria (to varying degree), for example as representations of shape [Martel et al. 2021; Park et al. 2019] and radiance fields [Liu et al. 2020; Mildenhall et al. 2020; Müller et al. 2020, 2021]. The important commonality of the these approaches is an encoding that maps neural network inputs to a higher-dimensional space, which is key for extracting high approximation quality from com-pact models. Most successful among these encodings are trainable, task-specific data structures [Liu et al. 2020; Takikawa et al. 2021] that take on a large portion of the learning task. This enables the use of smaller, more eficient MLPs. However, such data structures rely on heuristics and structural modifications (such as pruning, splitting, or merging) that may complicate the training process, limit the method to a specific task, or limit performance on GPUs where control flow and pointer chasing is expensive. We address these concerns with our multiresolution hash encod- ing, which is adaptive and eficient, independent of the task. It is configured by just two values—the number of parameters T and the desired finest resolution $N _ { \mathrm { m a x } ^ { - } }$ —yielding state-of-the-art quality on a variety of tasks (Figure 1) after a few seconds of training. Key to both the task-independent adaptivity and eficiency is a multiresolution hierarchy of hash tables: • Adaptivity: we map a cascade of grids to corresponding fixed-size arrays of feature vectors. At coarse resolutions, there is a 1:1 mapping from grid points to array entries. At fine resolutions, the array is treated as a hash table and indexed using a spatial hash function, where multiple grid points alias each array entry. Such hash collisions cause the colliding training gradients to average, meaning that the largest gradients—those most relevant to the loss function—will dominate. The hash tables thus automatically prioritize the sparse areas with the most important fine scale detail. Unlike prior work, no structural updates to the data structure are needed at any point during training. • Eficiency: our hash table lookups are $O ( 1 )$ and do not require control flow. This maps well to modern GPUs, avoiding execution divergence and serial pointer-chasing inherent in tree traversals. The hash tables for all resolutions may be queried in parallel. We validate our multiresolution hash encoding in four representa- tive tasks (see Figure 1): (1) Gigapixel image: the MLP learns the mapping from 2D coordinates to RGB colors of a high-resolution image. (2) Neural signed distance functions (SDF): the MLP learns the mapping from 3D coordinates to the distance to a surface. (3) Neural radiance caching (NRC): the MLP learns the 5D light field of a given scene from a Monte Carlo path tracer. (4) Neural radiance and density fields (NeRF): the MLP learns the 3D density and 5D light field of a given scene from image observations and corresponding perspective transforms. The source code and data pertaining to this paper can be found at https://nvlabs.github.io/instant-ngp. In the following, we first review prior neural network encodings (Section 2), then we describe our encoding (Section 3) and its implementation (Section 4), followed lastly by our experiments (Section 5) and discussion thereof (Section 6). ## 2 BACKGROUND AND RELATED WORK Early examples of encoding the inputs of a machine learning model into a higher-dimensional space include the one-hot encoding [Harris and Harris 2013] and the kernel trick [Theodoridis 2008] by which complex arrangements of data can be made linearly separable. For neural networks, input encodings have proven useful in the at- tention components of recurrent architectures [Gehring et al. 2017] and, subsequently, transformers [Vaswani et al. 2017], where they help the neural network to identify the location it is currently pro- cessing. Vaswani et al. [2017] encode scalar positions $x \in \mathbb { R }$ as a multiresolution sequence of L ∈ N sine and cosine functions $$ \begin{array} { c } { \operatorname { e n c } ( x ) = \left( \sin ( 2 ^ { 0 } x ) , \sin ( 2 ^ { 1 } x ) , \dots , \sin ( 2 ^ { L - 1 } x ) , \right. } \\ { \left. \cos ( 2 ^ { 0 } x ) , \cos ( 2 ^ { 1 } x ) , \dots , \cos ( 2 ^ { L - 1 } x ) \right) . } \end{array}\tag{1} $$ This has been adopted in computer graphics to encode the spatiodirectionally varying light field and volume density in the NeRF algorithm [Mildenhall et al. 2020]. The five dimensions of this light field are independently encoded using the above formula; this was later extended to randomly oriented parallel wavefronts [Tancik et al. 2020] and level-of-detail filtering [Barron et al. 2021a]. We will refer to this family of encodings as frequency encodings. Notably, frequency encodings followed by a linear transformation have been used in other computer graphics tasks, such as approximating the visibility function [Annen et al. 2007; Jansen and Bavoil 2010]. Müller et al. [2019; 2020] suggested a continuous variant of the one-hot encoding based on rasterizing a kernel, the one-blob en-coding, which can achieve more accurate results than frequency encodings in bounded domains at the cost of being single-scale. Parametric encodings. Recently, state-of-the-art results have been achieved by parametric encodings which blur the line between clas- sical data structures and neural approaches. The idea is to arrange additional trainable parameters (beyond weights and biases) in an auxiliary data structure, such as a grid [Chabra et al. 2020; Jiang et al. 2020; Liu et al. 2020; Mehta et al. 2021; Peng et al. 2020a; Sun et al. 2021; Tang et al. 2018; Yu et al. 2021a] or a tree [Takikawa et al. 2021], and to look-up and (optionally) interpolate these parameters depending on the input vector $\mathbf { x } \in \mathbb { R } ^ { d }$ . This arrangement trades a larger memory footprint for a smaller computational cost: whereas for each gradient propagated backwards through the network, every weight in the fully connected MLP network must be updated, for the trainable input encoding parameters (“feature vectors”), only a very small number are afected. For example, with a trilinearly interpolated 3D grid of feature vectors, only 8 such grid points need to be updated for each sample back-propagated to the encoding. In this way, although the total number of parameters is much higher for a parametric encoding than a fixed input encoding, the number of FLOPs and memory accesses required for the update during training is not increased significantly. By reducing the size of the MLP, such parametric models can typically be trained to convergence much faster without sacrificing approximation quality. (a) No encoding ![](images/2022_InstantNGP/e31c276ff098ae58dfb2f778b0bed683de0ba119063669c461a9682f10a4bed8.jpg) 411 k + 0 parameters 11:28 (mm:ss) / PSNR 18.56 (b) Frequency [Mildenhall et al. 2020] ![](images/2022_InstantNGP/4f8bca4b3017f217e1b551e7c41fa9c0ff87eb526b46fbcaff0b5a771b876f29.jpg) 438 k + 0 12:45 / PSNR 22.90 (c) Dense grid Single resolution ![](images/2022_InstantNGP/01c8ac6ee4b5dbdfe491e01c18a23e8c700c3f61b606378fc90e617426f25fad.jpg) 10 k + 33.6 M1:09 / PSNR 22.35 (d) Dense grid Multi resolution ![](images/2022_InstantNGP/f30621dbdbf87b19642735e531f901d1390c49bc9f3774d10ad0f849775fa366.jpg) 10 k + 16.3 M 1:26 / PSNR 23.62 (e) Hash table (ours) T = 214 ![](images/2022_InstantNGP/2d7dfc8e0b59457c1da3edc26d5ac70a96e66efd6b6eb35fc6e7cbe9e6872769.jpg) 10 k + 494 k 1:48 / PSNR 22.61 (f) Hash table (ours) ![](images/2022_InstantNGP/fe74a1165f421e27228a3d87d42487c89c56e14c0ac1e8f48d4e8c4e2b5d830f.jpg) 10 k + 12.6 M 1:47 / PSNR 24.58 Fig. 2. A demonstration of the reconstruction quality of diferent encodings and parametric data structures for storing trainable feature embeddings. Each configuration was trained for 11 000 steps using our fast NeRF implementation (Section 5.4), varying only the input encoding and MLP size. The number of trainable parameters (MLP weights + encoding parameters), training time and reconstruction accuracy (PSNR) are shown below each image. Our encoding (e) with a similar total number of trainable parameters as the frequency encoding configuration (b) trains over 8× faster, due to the sparsity of updates to the parameters and smaller MLP. Increasing the number of parameters (f) further improves reconstruction accuracy without significantly increasing training time. Another parametric approach uses a tree subdivision of the do- main $\mathbb { R } ^ { d }$ , wherein a large auxiliary coordinate encoder neural network (ACORN) [Martel et al. 2021] is trained to output dense feature grids in the leaf node around x. These dense feature grids, which have on the order of 10 000 entries, are then linearly interpolated, as in Liu et al. [2020]. This approach tends to yield a larger degree of adaptivity compared with the previous parametric encodings, albeit at greater computational cost which can only be amortized when suficiently many inputs x fall into each leaf node. Sparse parametric encodings. While existing parametric encodings tend to yield much greater accuracy than their non-parametric predecessors, they also come with downsides in eficiency and versa- tility. Dense grids oftrainable features consume much more memory than the neural network weights. To illustrate the trade-ofs and to motivate our method, Figure 2 shows the efect on reconstruction quality of a neural radiance field for several diferent encodings. Without any input encoding at all (a), the network is only able to learn a fairly smooth function of position, resulting in a poor ap- proximation of the light field. The frequency encoding (b) allows the same moderately sized network (8 hidden layers, each 256 wide) to represent the scene much more accurately. The middle image (c) pairs a smaller network with a dense grid of $1 2 8 ^ { 3 }$ trilinearly inter-polated, 16-dimensional feature vectors, for a total of 33.6 million trainable parameters. The large number of trainable parameters can be eficiently updated, as each sample only afects 8 grid points. However, the dense grid is wasteful in two ways. First, it allocates as many features to areas of empty space as it does to those areas near the surface. The number of parameters grows as $O ( N ^ { 3 } )$ , while the visible surface of interest has surface area that grows only as $O ( N ^ { 2 } )$ ). In this example, the grid has resolution 1283, but only 53 807 (2.57%) of its cells touch the visible surface. Second, natural scenes exhibit smoothness, motivating the use of a multi-resolution decomposition [Chibane et al. 2020; Hadadan et al. 2021]. Figure 2 (d) shows the result of using an encoding in which interpolated features are stored in eight co-located grids with resolutions from $1 6 ^ { 3 }$ to $1 7 3 ^ { 3 }$ , each containing 2-dimensional feature vectors. These are concatenated to form a 16-dimensional (same as (c)) input to the network. Despite having less than half the number of parameters as (c), the reconstruction quality is similar. If the surface of interest is known a priori, a data structure such as an octree [Takikawa et al. 2021] or sparse grid [Chabra et al. 2020; Chibane et al. 2020; Hadadan et al. 2021; Jiang et al. 2020; Liu et al. 2020; Peng et al. 2020a] can be used to cull away the unused features in the dense grid. However, in the NeRF setting, surfaces only emerge during training. NSVF [Liu et al. 2020] and several concurrent works [Sun et al. 2021; Yu et al. 2021a] adopt a multi stage, coarse to fine strategy in which regions of the feature grid are progressively refined and culled away as necessary. While efective, this leads to a more complex training process in which the sparse data structure must be periodically updated. Our method—Figure 2 (e,f)—combines both ideas to reduce waste. We store the trainable feature vectors in a compact spatial hash table, whose size is a hyper-parameter T which can be tuned to trade the number of parameters for reconstruction quality. It neither relies on progressive pruning during training nor on a priori knowledge of the geometry of the scene. Analogous to the multi-resolution grid in (d), we use multiple separate hash tables indexed at diferent resolutions, whose interpolated outputs are concatenated before being passed through the MLP. The reconstruction quality is comparable to the dense grid encoding, despite having 20× fewer parameters. Unlike prior work that used spatial hashing [Teschner et al. 2003] for 3D reconstruction [Nießner et al. 2013], we do not explicitly handle collisions of the hash functions by typical means like probing, bucketing, or chaining. Instead, we rely on the neural network to learn to disambiguate hash collisions itself, avoiding control flow divergence, reducing implementation complexity and improving performance. Another performance benefit is the predictable mem- ory layout of the hash tables that is independent of the data that is represented. While good caching behavior is often hard to achieve with tree-like data structures, our hash tables can be fine-tuned for low-level architectural details such as cache size. ![](images/2022_InstantNGP/c7c8181a6676196d73f7bdee58ff0c0164534c3bf63a3cb526bf73b0ddfb8abf.jpg) ![](images/2022_InstantNGP/0e6cf6f6eb2e46f5a6929c1f0e3274f7b1f0a76729a992ca6e32607081ab7605.jpg) Fig. 3. Illustration of the multiresolution hash encoding in 2D. (1) for a given input coordinate x, we find the surrounding voxels at L resolution levels and assign indices to their corners by hashing their integer coordinates. (2) for all resulting corner indices, we look up the corresponding F-dimensional feature vectors from the hash tables $\theta _ { l }$ and (3) linearly interpolate them according to the relative position of x within the respective l-th voxel. (4) we concatenate the result of each level, as well as auxiliary inputs $\boldsymbol { \xi } \in \mathbb { R } ^ { E }$ , roducin the encoded MLP in ut $\boldsymbol { y } \in \mathbb { R } ^ { L F + E }$ , which (5) is evaluated last. To train the encoding, loss gradients are backpropagated through the ${ \mathrm { ~ \cal ~ M L P ~ } } ( 5 )$ , the concatenation (4), the linear interpolation (3), and then accumulated in the looked-up feature vectors. Table 1. Hash encoding parameters and their ranges in this paper. Only the hash table size T and max. resolution $N _ { \mathrm { m a x } }$ need to be tuned to the task.
ParameterSymbolValue
Number of levelsL16
Max. entries per level (hash table size)T $2 ^ { 1 4 } \ : \mathrm { t o } \ : 2 ^ { 2 4 }$
Number of feature dimensions per entryF2
Coarsest resolution $N _ { \mathrm { m i n } }$ 16
Finest resolution $N _ { \mathrm { m a x } }$ 512 to 524288
## 3 MULTIRESOLUTION HASH ENCODING Given a fully connected neural network m(y; Φ), we are interested in an encoding of its inputs $\mathbf { y } = \operatorname { e n c } ( \mathbf { x } ; \theta )$ that improves the approximation quality and training speed across a wide range of applications without incurring a notable performance overhead. Our neural net- work not only has trainable weight parameters Φ, but also trainable encoding parameters θ. These are arranged into L levels, each con- taining up to T feature vectors with dimensionality F . Typical values for these hyperparameters are shown in Table 1. Figure 3 illustrates the steps performed in our multiresolution hash encoding. Each level (two of which are shown as red and blue in the figure) is inde-pendent and conceptually stores feature vectors at the vertices of a grid, the resolution of which is chosen to be a geometric progression between the coarsest and finest resolutions $[ N _ { \mathrm { m i n } } , N _ { \mathrm { m a x } } ]$ $$ N _ { l } : = \left\lfloor N _ { \mathrm { m i n } } \cdot b ^ { l } \right\rfloor ,\tag{2} $$ $$ b : = \exp \left( \frac { \ln N _ { \mathrm { m a x } } - \ln N _ { \mathrm { m i n } } } { L - 1 } \right) .\tag{3} $$ $N _ { \mathrm { m a x } }$ is chosen to match the finest detail in the training data. Due to the large number of levels L, the growth factor is usually small. Our use cases have $b \in \left[ 1 . 2 6 , 2 \right]$ Consider a single level l. The input coordinate $\mathbf { x } \in \mathbb { R } ^ { d }$ is scaled by that level’s grid resolution before rounding down and up $\lfloor \mathbf { x } _ { l } \rfloor : =$ $\lfloor \mathbf { x } \cdot N _ { l } \rfloor , \lceil \mathbf { x } _ { l } \rceil : = \lceil \mathbf { x } \cdot N _ { l } \rceil$ ⌊xl⌋ and ⌈xl⌉ span a voxel with $2 ^ { d }$ integer vertices in $\mathbb { Z } ^ { d }$ . We map each corner to an entry in the level’s respective feature vector array, which has fixed size ofat mostT. For coarse levels where a dense grid requires fewer than T parameters, i.e. $( N _ { l } + 1 ) ^ { d } \leq T ,$ this mapping is 1:1. At finer levels, we use a hash function $h : \mathbb { Z } ^ { d } \to \mathbb { Z } _ { T }$ to index into the array, efectively treating it as a hash table, although there is no explicit collision handling. We rely instead on the gradient-based optimization to store appropriate sparse detail in the array, and the subsequent neural network $m ( \mathbf { y } ; \Phi )$ for collision resolution. The number of trainable encoding parameters T is therefore $O ( T )$ and bounded by $T \cdot L \cdot F$ which in our case is always T · 16 · 2 (Table 1). We use a spatial hash function [Teschner et al. 2003] of the form $$ h ( \mathbf { x } ) = \left( \bigoplus _ { i = 1 } ^ { d } x _ { i } \pi _ { i } \right) \quad \bmod T ,\tag{4} $$ where ⊕ denotes the bit-wise XOR operation and $\pi _ { i }$ are unique, large prime numbers. Efectively, this formula XORs the results of a per-dimension linear congruential (pseudo-random) permutation [Lehmer 1951], decorrelating the efect of the dimensions on the hashed value. Notably, to achieve (pseudo-)independence, only $d - 1$ of the d dimensions must be permuted, so we choose $\pi _ { 1 } : = 1$ for better cache coherence, $\pi _ { 2 } = 2 6 5 4 4 3 5 7 6 1$ , and $\pi _ { 3 } = 8 0 5 4 5 9 8 6 1$ Lastly, the feature vectors at each corner are d-linearly interpo- lated according to the relative position of x within its hypercube, i.e. the interpolation weight is $\mathbf { w } _ { l } : = \mathbf { x } _ { l } - \lfloor \mathbf { x } _ { l } \rfloor$ Recall that this process takes place independently for each of the L levels. The interpolated feature vectors of each level, as well as auxiliary inputs $\bar { \xi \in \mathbb { R } ^ { E } }$ (such as the encoded view direction and textures in neural radiance caching), are concatenated to produce $\mathbf { y } \in \mathbb { R } ^ { L F + E }$ , which is the encoded input $\operatorname { e n c } ( \mathbf { x } ; \theta )$ to the ${ \mathrm { M L P } } m ( \mathbf { y } ; \Phi )$ Performance vs. quality. Choosing the hash table size T provides a trade-of between performance, memory and quality. Higher values of T result in higher quality and lower performance. The memory footprint is linear in T, whereas quality and performance tend to scale sub-linearly. We analyze the impact ofT in Figure 4, where we report test error vs. training time for a wide range of T -values for three neural graphics primitives. We recommend practitioners to use T to tweak the encoding to their desired performance characteristics. ![](images/2022_InstantNGP/65ea5434f9ee1a27d85a8b6007f20e17cd5c631e4403233a019aa23f90945dfc.jpg) ![](images/2022_InstantNGP/49f12c9589ea7bdf337cd9444db814f96b02d9891722a4cafacecb2dafab716e.jpg) ![](images/2022_InstantNGP/3dc0408e4e4e85176f8aa7a99d3a0b220d3a670401998eb971881203674a204e.jpg) Fi . 4. The main curves lot test error over trainin time for var in hash table size T which determines the number of trainable encodin arameters. Increasing T improves reconstruction, at the cost of higher memory usage and slower training and inference. A performance clif is visible at $\dot { T } > 2 ^ { 1 9 }$ where the cache of our RTX 3090 GPU becomes oversubscribed (particularly visible for SDF and NeRF). The plot also shows model convergence over time leading up to the final state. This highlights how high quality results are already obtained after only a few seconds. Jumps in the convergence (most visible towards the end of SDF training) are caused by learning rate decay. For NeRF and Gigapixel image, training finishes after 31 000 steps and for SDF after 11 000 steps. ![](images/2022_InstantNGP/a6d32a32d42fedd744655970f529993abefda5b40127ff9f1be67e54d50fe878.jpg) ![](images/2022_InstantNGP/0f298a7a59295f4006c92cbd2ca064b190fc2b9b0e0219d4b121e5b69a97d664.jpg) ![](images/2022_InstantNGP/d94eb2536256aebc7ad01f727193038059726fdd1a42578eb1caaf93943185d4.jpg) Fi . 5. Test error over trainin time for fixed values of feature dimensionalit F as the number of hash table levels L is varied. To maintain a rou hl e ua trainable parameter count, the hash table size T is set according to $F \cdot T \cdot L = 2 ^ { 2 4 }$ for SDF and NeRF, whereas gigapixel image uses $2 ^ { 2 8 } .$ . Since $( F = 2 , L = 1 6 )$ is near the best-case performance and quality (top-left) for all applications, we use this configuration in all results. $F = 1$ is slow on our RTX 3090 GPU since atomic half- recision accumulation is onl eficient for 2D vectors but not for scalars. For NeRF and Gi a ixel ima e, trainin finishes after 31 000 ste s whereas SDF completes at 11 000 steps. The hyperparameters L (number of levels) and F (number of feature dimensions) also trade of quality and performance, which we analyze for an approximately constant number of trainable encoding parameters θ in Figure 5. In this analysis, we found $( F = 2 , L = 1 6 )$ to be a favorable Pareto optimum in all our applications, so we use these values in all other results and recommend them as the default. Implicit hash collision resolution. It may appear counter-intuitive that this encoding is able to reconstruct scenes faithfully in the presence of hash collisions. Key to its success is that the diferent resolution levels have diferent strengths that complement each other. The coarser levels, and thus the encoding as a whole, are injective—that is, they sufer from no collisions at all. However, they can only represent a low-resolution version of the scene, since they ofer features which are linearly interpolated from a widely spaced grid of points. Conversely, fine levels can capture small features due to their fine grid resolution, but sufer from many collisions—that is, disparate points which hash to the same table entry. Nearby inputs with equal integer coordinates ⌊xl⌋ are not considered a collision; a collision occurs when diferent integer coordinates hash to the same index. Luckily, such collisions are pseudo-randomly scattered across space, and statistically unlikely to occur simultaneously at every level for a given pair of points. When training samples collide in this way, their gradients average. Consider that the importance to the final reconstruction of such samples is rarely equal. For example, a point on a visible surface of a radiance field will contribute strongly to the reconstructed image (having high visibility and high density, both multiplicatively afecting the magnitude of gradients) causing large changes to its table entries, while a point in empty space that happens to refer to the same entry will have a much smaller weight. As a result, the gradients of the more important samples dominate the collision average and the aliased table entry will naturally be optimized in such a way that it reflects the needs of the higher-weighted point. The multiresolution aspect of the hash encoding covers the full range from a coarse resolution $N _ { \mathrm { m i n } }$ that is guaranteed to be collision- free to the finest resolution $N _ { \mathrm { m a x } }$ that the task requires. Thereby, it guarantees that all scales at which meaningful learning could take place are included, regardless of sparsity. Geometric scaling allows covering these scales with only $O \big ( \log { ( N _ { \mathrm { m a x } } / N _ { \mathrm { m i n } } ) } \big )$ many levels, which allows picking a conservatively large value for $N _ { \mathrm { m a x } }$ Online adaptivity. Note that if the distribution of inputs x changes over time during training, for example if they become concentrated in a small region, then finer grid levels will experience fewer colli- sions and a more accurate function can be learned. In other words, the multiresolution hash encoding automatically adapts to the training data distribution, inheriting the benefits of tree-based encod- ings [Takikawa et al. 2021] without task-specific data structure maintenance that might cause discrete jumps during training. One of our applications, neural radiance caching in Section 5.3, continually adapts to animated viewpoints and 3D content, greatly benefitting from this feature. d-linear interpolation. Interpolating the queried hash table entries ensures that the encoding enc(x; θ), and by the chain rule its composition with the neural network m(enc(x; θ); Φ), are continuous. Without interpolation, grid-aligned discontinuities would be present in the network output, which would result in an undesirable blocky appearance. One may desire higher-order smoothness, for exam-ple when approximating partial diferential equations. A concrete example from computer graphics are signed distance functions, in which case the gradient ∂m(enc(x; θ); Φ)/∂x, i.e. the surface normal, would ideally also be continuous. If higher-order smoothness must be guaranteed, we describe a low-cost approach in Appendix A, which we however do not employ in any of our results due to a small decrease in reconstruction quality. ## 4 IMPLEMENTATION To demonstrate the speed of the multiresolution hash encoding, we implemented it in CUDA and integrated it with the fast fully-fused MLPs of the tiny-cuda-nn framework [Müller 2021].1 We release the source code of the multiresolution hash encoding as an update to Müller [2021] and the source code pertaining to the neural graphics primitives at https://github.com/nvlabs/instant-ngp. Performance considerations. In order to optimize inference and backpropagation performance, we store hash table entries at half precision (2 bytes per entry). We additionally maintain a master copy of the parameters in full precision for stable mixed-precision parameter updates, following Micikevicius et al. [2018]. To optimally use the GPU’s caches, we evaluate the hash tables level by level: when processing a batch of input positions, we sched- ule the computation to look up the first level of the multiresolution hash encoding for all inputs, followed by the second level for all inputs, and so on. Thus, only a small number of consecutive hash tables have to reside in caches at any given time, depending on how much parallelism is available on the GPU. Importantly, this structure of computation automatically makes good use of the available caches and parallelism for a wide range of hash table sizes T. On our hardware, the performance of the encoding remains roughly constant as long as the hash table size stays below $T \le 2 ^ { 1 9 }$ Beyond this threshold, performance starts to drop significantly; see Figure 4. This is explained by the 6 MB L2 cache of our NVIDIA RTX 3090 GPU, which becomes too small for individual levels when $2 \cdot T \cdot F > 6 \cdot 2 ^ { 2 0 }$ , with 2 being the size of a half-precision entry. The optimal number of feature dimensions F per lookup depends on the GPU architecture. On one hand, a small number favors cache locality in the previously mentioned streaming approach, but on the other hand, a large F favors memory coherence by allowing for F-wide vector load instructions. $F = 2$ gave us the best cost-quality trade-of (see Figure 5) and we use it in all experiments. Architecture. In all tasks, except for NeRF which we will describe later, we use an MLP with two hidden layers that have a width of 64 neurons, rectified linear unit (ReLU) activation functions on their hidden layers, and a linear output layer. The maximum resolu- tion $N _ { \mathrm { m a x } }$ is set to 2048 × scene size for NeRF and signed distance functions, to half of the gigapixel image width, and $2 ^ { \overset { \smile } { 1 9 } }$ in radiance caching (large value to support close-by objects in expansive scenes). Initialization. We initialize neural network weights according to Glorot and Bengio [2010] to provide a reasonable scaling of activations and their gradients throughout the layers of the neural network. We initialize the hash table entries using the uniform dis- tribution $\mathcal { U } ( - 1 0 ^ { - 4 } , 1 0 ^ { - 4 } )$ to provide a small amount of randomness while encouraging initial predictions close to zero. We also tried a variety of diferent distributions, including zero-initialization, which all resulted in a very slightly worse initial convergence speed. The hash table appears to be robust to the initialization scheme. Training. We jointly train the neural network weights and the hash table entries by applying Adam [Kingma and Ba 2014], where we set $\beta _ { 1 } = 0 . 9 , \beta _ { 2 } = 0 . 9 9 , \epsilon = 1 0 ^ { - 1 5 }$ , The choice of $\beta _ { 1 }$ and $\beta _ { 2 }$ makes only a small diference, but the small value of $\epsilon = \dot { 1 } 0 ^ { - 1 5 }$ can signifi-cantly accelerate the convergence of the hash table entries when their gradients are sparse and weak. To prevent divergence after long training periods, we apply a weak L2 regularization (factor $1 0 ^ { - 6 } )$ to the neural network weights, but not to the hash table entries. When fitting gigapixel images or NeRFs, we use the $\mathcal { L } ^ { 2 }$ loss. For signed distance functions, we use the mean absolute percentage error (MAPE), defined as $\frac { | { \mathrm { p r e d i c t i o n - t a r g e t } } | } { | { \mathrm { t a r g e t } } | + 0 . 0 1 }$ , and for neural radiance caching we use a luminance-relative $\mathcal { L } ^ { 2 }$ loss [Müller et al. 2021]. We observed fastest convergence with a learning rate of $1 0 ^ { - 4 }$ for signed distance functions and $\stackrel { \sim } { 1 0 } \stackrel { - 2 } { 1 }$ otherwise, as well a a batch size $\cot 2 ^ { 1 4 }$ for neural radiance caching and $2 ^ { 1 8 }$ otherwise. Lastly, we skip Adam steps for hash table entries whose gradient is exactly 0. This saves ∼ 10% performance when gradients are sparse, which is a common occurrence with T ≫ BatchSize. Even though this heuristic violates some of the assumptions behind Adam, we observe no degradation in convergence. Non-spatial input dimensions $\boldsymbol { \xi } \in \mathbb { R } ^ { E }$ . The multiresolution hash encoding targets spatial coordinates with relatively low dimension-ality. All our experiments operate either in 2D or 3D. However, it is frequently useful to input auxiliary dimensions $\boldsymbol { \xi } \in \mathbb { R } ^ { E }$ to the neural network, such as the view direction and material parameters when learning a light field. In such cases, the auxiliary dimensions can be encoded with established techniques whose cost does not scale superlinearly with dimensionality; we use the one-blob encod- ing [Müller et al. 2019] in neural radiance caching [Müller et al. 2021] and the spherical harmonics basis in NeRF, similar to concurrent work [Verbin et al. 2021; Yu et al. 2021a]. ![](images/2022_InstantNGP/966eca8b91221f467cf085af5cf2de7362775147c9a6246a62d3003df0972860.jpg) Fig. 6. Approximating an RGB image of resolution 20 000 × 23 466 (469 M RGB pixels) with our multiresolution hash encoding. With hash table sizes T of $2 ^ { 1 2 } .$ $2 ^ { 1 \overline { { 7 } } }$ , and $\dot { 2 } ^ { \dot { 2 } 2 }$ the models shown have 117 k, 2.7 M, and 47.5 M trainable parameters respectively. With only 3.4% of the degrees of freedom of the input, the last model achieves a reconstruction PSNR of 29.8 dB. “Girl With a Pearl Earring” renovation ©Koorosh Orooj (CC BY-SA 4.0) ## 5 EXPERIMENTS To highlight the versatility and high quality of the encoding, we com-pare it with previous encodings in four distinct computer graphics primitives that benefit from encoding spatial coordinates. ## 5.1 Gigapixel Image Approximation Learning the 2D to RGB mapping of image coordinates to colors has become a popular benchmark for testing a model’s ability to represent high-frequency detail [Martel et al. 2021; Müller et al. 2019; Sitzmann et al. 2020; Tancik et al. 2020]. Recent breakthroughs in adaptive coordinate networks (ACORN) [Martel et al. 2021] have shown impressive results when fitting very large images—up to a bil lion pixels—with high fidelity at even the smallest scales. We target our multiresolution hash encoding at the same task and converge to high-fidelity images in seconds to minutes (Figure 4). For comparison, on the Tokyo panorama from Figure 1, ACORN achieves a PSNR of 38.59 dB after 36.9 h of training. With a similar number of parameters $( T = 2 ^ { 2 4 } )$ , our method achieves the same PSNR after 2.5 minutes of training, peaking at 41.9 dB after 4 min. Figure 6 showcases the level of detail contained in our model for a variety of hash table sizes T on another image. It is dificult to directly compare the performance of our encoding to ACORN; a factor of ∼10 stems from our use of fully fused CUDA kernels, provided by the tiny-cuda-nn framework [Müller 2021]. The input encoding allows for the use of a much smaller MLP than with ACORN, which accounts for much of the remaining 10×–100× speedup. That said, we believe that the biggest value-add of the multiresolution hash encoding is its simplicity. ACORN relies on an adaptive subdivision of the scene as part of a learning curriculum, none of which is necessary with our encoding. ## 5.2 Signed Distance Functions Signed distance functions (SDFs), in which a 3D shape is represented as the zero level-set of a function of position x, are used in many applications including simulation, path planning, 3D modeling, and video games. DeepSDF [Park et al. 2019] uses a large MLP to represent one or more SDFs at a time. In contrast, when just a single SDF needs to be fit, a spatially learned encoding, such as ours can be employed and the MLP shrunk significantly. This is the application we investigate in this section. As baseline, we compare with NGLOD [Takikawa et al. 2021], which achieves state-of-the-art results in both quality and speed by prefixing its small MLP with a lookup from an octree of trainable feature vectors. Lookups along the hierarchy of this octree act similarly to our multiresolution cas- cade of grids: they are a collision-free analog to our technique, with a fixed growth factor b = 2. To allow meaningful comparisons in terms of both performance and quality, we implemented an opti- mized version of NGLOD in our framework, details of which we describe in Appendix B. Details pertaining to real-time training of SDFs are described in Appendix C. In Figure 7, we compare NGLOD with our multiresolution hash encoding at roughly equal parameter count. We also show a straightforward application of the frequency encoding [Mildenhall et al. 2020] to provide a baseline, details of which are found in Appendix D. By using a data structure tailored to the reference shape, NGLOD achieves the highest visual reconstruction quality. How- ever, even without such a dedicated data structure, our encoding approaches a similar fidelity to NGLOD in terms of the intersectionover-union metric (IoU2) with similar performance and memory cost. Furthermore, the SDF is defined ev- erywhere within the training volume, as opposed to NGLOD, which is only defined within the octree (i.e. close to the surface). This permits the use of certain SDF rendering techniques such as approximate soft shadows from a small number of of-surface distance samples [Evans 2006], as shown in the adjacent figure. ![](images/2022_InstantNGP/6e7041bff82906e035e3e28ea12f761951158eaa9b5168f0c44c9e01a41351b5.jpg) To emphasize diferences between the compared methods, we visualize the SDF using a shading model. The resulting colors are sensitive to even slight changes in the surface normal, which empha- sizes small fluctuations in the prediction more strongly than in other graphics primitives where color is predicted directly. This sensitivity reveals undesired microstructure in our hash encoding on the scale of the finest grid resolution, which is absent in NGLOD and does not disappear with longer training times. Since NGLOD is essentially a collision-free analog to our hash encoding, we attribute this artifact to hash collisions. Upon close inspection, similar microstructure can be seen in other neural graphics primitives, although with significantly lower magnitude. ![](images/2022_InstantNGP/c66154ffe55c36d8c866802438f6c58dfb50017db742dc2fb5d44e867dafcf04.jpg) Fig. 7. Neural signed distance functions trained for 11 000 steps. The frequency encoding [Mildenhall et al. 2020] struggles to capture the sharp details on these intricate models. NGLOD [Takikawa et al. 2021] achieves the highest visual quality, at the cost of only training the SDF inside the cells of a close-fiting octree. Our hash encodin exhibits similar numeric ualit in terms of intersection over union (IoU) and can be evaluated an where in the scene. However it also exhibits visually undesirable surface roughness that we atribute to randomly distributed hash collisions. Bearded Man ©Oliver Laric (CC BY-NC-SA 2.0) ![](images/2022_InstantNGP/b3de8a4f8108a5e2de23614dd2e395bdb2219eb4eb570ac6c33c2cdcc7a7d459.jpg) Fig. 8. Summary of the neural radiance caching application [Müller et al. 2021]. The MLP m enc(x; θ); Φ is tasked with predicting photorealistic pixel colors from feature bufers independentlyfor each pixel. The feature bufers contain, among other variables, the world-space position x, which we propose to encode with our method. Neural radiance caching is a challenging application, because it is supervised online during real-time rendering. The training data are a sparse set of light paths that are continually spawned from the camera view. As such, the neural network and encoding do not learn a general mapping from features to color, but rather they continually overfit to the current shape and lighting. To support animated content, training has a budget of one millisecond per frame. Multiresolution hash encodin (Ours), T = 15, 133 FPS Triangle wave encoding [Müller et al. 2021], 147 FPS ![](images/2022_InstantNGP/9aa49abcbde2947e1521cfea3e746ed1dea930abdf7cd6b0e1b781fbc1ab1a47.jpg) Fi . 9. Neural radiance cachin [Müller et al. 2021] ains much im roved ualit from the multiresolution hash encodin with onl a mild erformance penalty: 133 versus 147 frames per second at a resolution of 1920×1080px. To demonstrate the online adaptivity of the multiple hash resolutions vs. the prior triangle wave encoding, we show screenshots from a smooth camera motion that starts with a far-away view of the scene (left) and zooms onto a close-by view of an intricate shadow (right). Throughout the camera motion, which takes just a few seconds, the neural radiance cache continually learns from sparse camera paths, enabling the cache to learn (“overfit”) intricate detail at the scale of the content that the camera is momentarily observing. ![](images/2022_InstantNGP/517cea36a087f15da28473f157142ad556c49f496440e23907845153a04a25f5.jpg) Fig. 10. The efect of the MLP size on test error vs. training time (31 000 training steps) on the Lego scene. Other scenes behave almost identically. Each curve represents a diferent MLP depth, where the color MLP has $N _ { \mathrm { l a y e r s } }$ hidden layers and the density MLP has 1 hidden layer; we do not observe an improvement with deeper density MLPs. The curves sweep the number of neurons the hidden layers of the density and color MLPs from 16 to 256. Informed by this analysis, we choose $N _ { \mathrm { l a y e r s } } = 2$ and $N _ { \mathrm { n e u r o n s } } = 6 4 .$ ## 5.3 Neural Radiance Caching In neural radiance caching [Müller et al. 2021], the task of the MLP is to predict photorealistic pixel colors from feature bufers; see Fig- ure 8. The MLP is run independently for each pixel (i.e. the model is not convolutional), so the feature bufers can be treated as per-pixel feature vectors that contain the 3D coordinate x as well as additional features. We can therefore directly apply our multiresolution hash encoding to x while treating all additional features as auxiliary encoded dimensions ξ to be concatenated with the encoded position, using the same encoding as Müller et al. [2021]. We integrated our work into Müller et al.’s implementation of neural radiance caching and therefore refer to their paper for implementation details. For photorealistic rendering, the neural radiance cache is typ- ically queried only for indirect path contributions, which masks its reconstruction error behind the first reflection. In contrast, we would like to emphasize the neural radiance cache’s error, and thus the improvement that can be obtained by using our multiresolution hash encoding, so we directly visualize the neural radiance cache at the first path vertex. Figure 9 shows that—compared to the triangle wave encoding of Müller et al. [2021]—our encoding results in sharper reconstruction while incurring only a mild performance overhead of 0.7 ms that reduces the frame rate from 147 to 133 FPS at a resolution of 1920 × 1080px. Notably, the neural radiance cache is trained online—during rendering—from a path tracer that runs in the background, which means that the 0.7 ms overhead includes both training and runtime costs of our encoding. ## 5.4 Neural Radiance and Density Fields (NeRF) In the NeRF setting, a volumetric shape is represented in terms of a spatial (3D) density function and a spatiodirectional (5D) emission function, which we represent by a similar neural network architec- ture as Mildenhall et al. [2020]. We train the model in the same ways as Mildenhall et al.: by backpropagating through a diferentiable ray marcher driven by 2D RGB images from known camera poses. ![](images/2022_InstantNGP/29c2432542c8f277bd23a5fca689059c3b1e682a399efc4935de238d1b4c7d3d.jpg) Fig. 11. Feeding the result of our encoding through a linear transformation (no neural network) versus an MLP when learning a NeRF. The models were trained for 1 min. The MLP allows for resolving specular details and reduces the amount of background noise caused by hash collisions. Due to the small size and eficient implementation of the MLP, it is only 15% more expensive—well worth the significantly improved quality. Model Architecture. Unlike the other three applications, our NeRF model consists of two concatenated MLPs: a density MLP followed by a color MLP [Mildenhall et al. 2020]. The density MLP maps the hash encoded position $\mathbf { y } = \operatorname { e n c } ( \mathbf { x } ; \theta )$ to 16 output values, the first of which we treat as log-space density. The color MLP adds view-dependent color variation. Its input is the concatenation of • the 16 output values of the density MLP, and • the view direction projected onto the first 16 coeficients of the spherical harmonics basis (i.e. up to degree 4). This is a natural frequency encoding over unit vectors. Its output is an RGB color triplet, for which we use either a sigmoid activation when the training data has low dynamic-range (sRGB) or an exponential activation when it has high dynamic range (linear HDR). We prefer HDR training data due to the closer resemblance to physical light transport. This brings numerous advantages as has also been noted in concurrent work [Mildenhall et al. 2021]. Informed by the analysis in Figure 10, our results were generated with a 1-hidden-layer density MLP and a 2-hidden-layer color MLP, both 64 neurons wide. Accelerated ray marching. When marching along rays for both training and rendering, we would like to place samples such that they contribute somewhat uniformly to the image, minimizing wasted computation. Thus, we concentrate samples near surfaces by maintaining an occupancy grid that coarsely marks empty vs. nonempty space. In large scenes, we additionally cascade the occupancy grid and distribute samples exponentially rather than uniformly along the ray. Appendix E describes these procedures in detail. At HD resolutions, synthetic and even real-world scenes can be trained in seconds and rendered at 60 FPS, without the need of caching of the MLP outputs [Garbin et al. 2021; Wizadwongsa et al. 2021; Yu et al. 2021b]. This high performance makes it tractable to add efects such as anti-aliasing, motion blur and depth of field by brute-force tracing of multiple rays per pixel, as shown in Figure 12. Table 2. Peak signal to noise ratio (PSNR) of our NeRF implementation with multiresolution hash encoding (“Ours: Hash”) vs. NeRF [Mildenhall et al. 2020], mip-NeRF [Barron et al. 2021a], and NSVF [Liu et al. 2020], which require ∼hours to train (values taken from the respective papers). To demonstrate the comparatively rapid training of our method, we list its results after training for 1 s to 5 min. For each scene, we mark the methods with least error using gold , silver , and bronze medals. To analyze the degree to which our speedup originates from our optimized implementation vs. from our hash encoding, we also report PSNR for a nearly identical version of our implementation, in which the hash encoding has been replaced by the frequency encoding and the MLP correspondingly enlarged to match Mildenhall et al. [2020] (“Ours: Frequency”; details in Appendix D). It approaches NeRF’s quality after training for just ∼5 min, yet is outperformed by our full method after training for 5 s–15 s, amounting to a 20–60× improvement that can be atributed to the hash encoding
MICFicUsCHAIRHoTDOGMATERIALSDRUMSSHIPLEGOavg.
Ours: Hash (1 s)26.0921.3021.5521.6322.0717.7620.3818.8321.202
Ours: Hash (5 s)32.6030.3530.7733.4226.6023.8426.3830.1329.261
Ours: Hash (15 s)34.7632.2632.9535.5628.2525.2328.5633.6831.407
Ours: Hash (1 min)35.92 ●33.05 ●34.34 ●36.7829.3325.8230.20 ●35.63 ●32.635 ●
Ours: Hash (5 min)36.2233.5135.0037.4029.78 ●26.0231.10 36.3933.176
mip-NeRF (~hours)36.5133.2935.1437.4830.7125.48 ●30.4135.7033.090
NSVF (~hours)34.2731.2333.1937.14 ●32.6825.1827.9332.2931.739
NeRF (~hours)32.9130.1333.0036.1829.6225.0128.6532.5431.005
Ours: Frequency (5 min)31.8928.7431.0234.8628.9324.1828.0632.7730.056
Ours: Frequency (1 min)26.6224.7228.5132.6126.3621.3324.3228.8826.669
![](images/2022_InstantNGP/ac551826ee9980ec12f8a3c6153a5c667525d2f8a417f9e37c0c5452f5329b4b.jpg) Fig. 12. NeRF reconstruction of a modular synthesizer and large natural 360 scene. The left image took 5 seconds to accumulate 128 samples at 1080p on a single RTX 3090 GPU, allowing for brute force defocus efects. The right image was taken from an interactive session running at 10 frames per second on the same GPU. Comparison with direct voxel lookups. Figure 11 shows an ablation where we replace the entire neural network with a single linear matrix multiplication, in the spirit of (although not identical to) concurrent direct voxel-based NeRF [Sun et al. 2021; Yu et al. 2021a]. While the linear layer is capable of reproducing view-dependent efects, the quality is significantly compromised as compared to the MLP, which is better able to capture specular efects and to resolve hash collisions across the interpolated multiresolution hash tables (which manifest as high-frequency artifacts). Fortunately, the MLP is only 15% more expensive than the linear layer, thanks to its small size and eficient implementation. Comparison with high-quality ofline NeRF. In Table 2, we compare the peak signal to noise ratio (PSNR) our NeRF implementation with multiresolution hash encoding (“Ours: Hash”) with that of NeRF [Mildenhall et al. 2020], mip-NeRF [Barron et al. 2021a], and NSVF [Liu et al. 2020], which all require on the order of hours to train. In contrast, we list results of our method after training for 1 s to 5 min. Our PSNR is competitive with NeRF and NSVF after just 15 s of training, and competitive with mip-NeRF after 1 min to 5 min of training. On one hand, our method performs best on scenes with high geometric detail, such as Ficus, Drums, Ship and Lego, achieving the best PSNR of all methods. On the other hand, mip-NeRF and NSVF outperform our method on scenes with complex, view-dependent reflections, such as Materials; we attribute this to the much smaller MLP that we necessarily employ to obtain our speedup of several orders of magnitude over these competing implementations. Next, we analyze the degree to which our speedup originates from our eficient implementation versus from our encoding. To this end, we additionally report PSNR for a nearly identical ver-sion of our implementation: we replace the hash encoding by the frequency encoding and enlarge the MLP to approximately match the architecture of Mildenhall et al. [2020] (“Ours: Frequency”); see Appendix D for details. This version of our algorithm approaches NeRF’s quality after training for just ∼ 5 min, yet is outperformed by our full method after training for a much shorter duration (5 s–15 s), amounting to a 20–60× improvement caused by the hash encoding and smaller MLP. For “Ours: Hash”, the cost ofeach training step is roughly constant at ∼6 ms per step. This amounts to 50 k steps after 5 min at which point the model is well converged. We decay the learning rate after 20 k steps by a factor of 0.33, which we repeat every further 10 k steps. In contrast, the larger MLP used in “Ours: Frequency” requires ∼30 ms per training step, meaning that the PSNR listed after 5 min corresponds to about 10 k steps. It could thus keep improving slightly iftrained for extended periods oftime, as in the ofline NeRF variants that are often trained for several 100 k steps. While we isolated the performance and convergence impact of our hash encoding and its small MLP, we believe an additional study is required to quantify the impact of advanced ray marching schemes (such as ours, coarse-fine [Mildenhall et al. 2020], or DONeRF [Nef et al. 2021]) independently from the encoding and network architecture. We report additional information in Section E.3 to aid in such an analysis. ![](images/2022_InstantNGP/ae5856ec9e4c170c8a73fd9819636e1f50fa5707a1a9a07fee757af9c7318b76.jpg) (a) Ofline rendered reference ![](images/2022_InstantNGP/bc6747a89a7769284a11e69a8cbaadef06d94d23acb860cecf4c29dfe8cc4f16.jpg) (b) Hash (ours), trained for 10 s Rendered in 32 ms (2 samples per pixel) ![](images/2022_InstantNGP/d9227209ddc58abfa278b4126f44ba927e5ca16da650a716cb6c7649295eee41.jpg) (c) Path tracer Rendered in 32 ms (16 samples per pixel) Fig. 13. Preliminary results of training a NeRF cloud model (b) from real-time path tracing data. Within 32 ms, a 1024×1024 image of our model convincingly approximates the ofline rendered ground truth (a). Our model exhibits less noise than a GPU path tracer that ran for an equal amount of time (c). The cloud data is ©Walt Disney Animation Studios (CC BY-SA 3.0) ## 6 DISCUSSION AND FUTURE WORK Concatenation vs. reduction. At the end of the encoding, we concatenate rather than reduce (for example, by summing) the F -di-mensional feature vectors obtained from each resolution. We prefer concatenation for two reasons. First, it allows for independent, fully parallel processing of each resolution. Second, a reduction of the dimensionality of the encoded result y from LF to F may be too small to encode useful information. While F could be increased proportionally, it would make the encoding much more expensive. However, we recognize that there may be applications in which reduction is favorable, such as when the neural network is signifi- cantly more expensive than the encoding, in which case the added computational cost of increasing F could be insignificant. We thus argue for concatenation by default and not as a hard-and-fast rule. In our applications, concatenation, coupled with F = 2 always yielded by far the best results. Choice ofhash function. A good hash function is eficient to com pute, leads to coherent look-ups, and uniformly covers the feature vector array regardless of the structure of query points. We chose our hash function for its good mixture of these properties and also experimented with three others: (1) The PCG32 [O’Neill 2014] RNG, which has superior statistical properties. Unfortunately, it did not yield a higher-quality reconstruction, making its higher cost not worthwhile. (2) Ordering the least significant bits of Zd by a space-filling curve and only hashing the higher bits. This leads to better look-up coherence at the cost of worse reconstruction quality. However, the speed-up is only marginally better than setting π1 := 1 as done in our hash, and is thus not worth the reduced quality. (3) Even better coherence can be achieved by treating the hash function as a tiling of space into dense grids. Like (2), the speedup is small in practice with significant detriment to quality. Alternatively to hand-crafted hash functions, it is conceivable to optimize the hash function in future work, turning the method into a dictionary-learning approach. Two possible avenues are (1) developing a continuous formulation of indexing that is amenable to analytic diferentiation or (2) applying an evolutionary optimization algorithm that can eficiently explore the discrete function space. Microstructure due to hash collisions. The salient artifact of our encoding is a small amount of “grainy” microstructure, most visible on the learned signed distance functions (Figure 1 and Figure 7). The graininess is a result of hash collisions that the MLP is unable to fully compensate for. We believe that the key to achieving state- of-the-art quality on SDFs with our encoding will be to find a way to overcome this microstructure, for example by filtering hash table lookups or by imposing an additional smoothness prior on the loss. Generative setting. Parametric input encodings, when used in a generative setting, typically arrange their features in a dense grid which can then be populated by a separate generator network, typically a CNN such as StyleGAN [Chan et al. 2021; DeVries et al. 2021; Peng et al. 2020b]. Our hash encoding adds an additional layer of complexity, as the features are not arranged in a regular pattern through the input domain; that is, the features are not bijective with a regular grid of points. We leave it to future work to determine how best to overcome this dificulty. Other applications. We are interested in applying the multiresolution hash encoding to other low-dimensional tasks that require accurate, high-frequency fits. The frequency encoding originated from the attention mechanism of transformer networks [Vaswani et al. 2017]. We hope that parametric encodings such as ours can lead to a meaningful improvement in general, attention-based tasks. Heterogenous volumetric density fields, such as cloud and smoke stored in a VDB [Museth 2013, 2021] data structure, often include empty space on the outside, a solid core on the inside, and sparse detail on the volumetric surface. This makes them a good fit for our encoding. In the code released alongside this paper, we have included a preliminary implementation that fits a radiance and density field directly from the noisy output of a volumetric path tracer. The initial results are promising, as shown in Figure 13, and we intend to pursue this direction further in future work. ## 7 CONCLUSION Many graphics problems rely on task specific data structures to exploit the sparsity or smoothness of the problem at hand. Our multi-resolution hash encoding provides a practical learning-based alternative that automatically focuses on relevant detail, indepen- dent of the task. Its low overhead allows it to be used even in time- constrained settings like online training and inference. In the context of neural network input encodings, it is a drop-in replacement, for example speeding up NeRF by several orders of magnitude and matching the performance of concurrent non-neural 3D reconstruc- tion techniques. Slow computational processes in any setting, from lightmap baking to the training of neural networks, can lead to frustrating work- flows due to long iteration times [Enderton and Wexler 2011]. We have demonstrated that single-GPU training times measured in seconds are within reach for many graphics applications, allowing neural approaches to be applied where previously they may have been discounted. ## ACKNOWLEDGMENTS We are grateful to Andrew Tao, Andrew Webb, Anjul Patney, David Luebke, Fabrice Rousselle, Jacob Munkberg, James Lucas, Jonathan Granskog, Jonathan Tremblay, Koki Nagano, Marco Salvi, Nikolaus Binder, and Towaki Takikawa for profound discussions, proofread- ing, feedback, and early testing. We also thank Arman Toorians and Saurabh Jain for the factory robot dataset in Figure 12 (right). ## REFERENCES Thomas Annen, Tom Mertens, Philippe Bekaert, Hans-Peter Seidel, and Jan Kautz. 2007. Convolution Shadow Maps. In Rendering Techniques, Jan Kautz and Sumanta Pattanaik (Eds.). The Eurographics Association. https://doi.org/10.2312/EGWR EGSR07/051-060 Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin Brualla, and Pratul P. Srinivasan. 2021a. Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. arXiv (2021). https://jonbarron.info/mipnerf Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hed man. 2021b. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. arXiv:2111.12077 (Nov. 2021). Rohan Chabra, Jan E. Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Love-grove, and Richard Newcombe. 2020. Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction. In Computer Vision – ECCV 2020 Andrea Vedaldi Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 608–625. Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. 2021. Eficient Geometry-aware 3D Generative Adversarial Networks. arXiv:2112.07945 (2021). arXiv:2112.07945 [cs.CV] Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. 2020. Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W. Taylor, and Joshua M. Susskind. 2021. Unconstrained Scene Generation with Locally Condi tioned Radiance Fields. arXiv (2021). Eric Enderton and Daniel Wexler. 2011. The Workflow Scale. In Computer Graphics International Workshop on VFX, Computer Animation, and Stereo Movies. Alex Evans. 2006. Fast Approximations for Global Illumination on Dynamic Scenes. In ACM SIGGRAPH 2006 Courses (Boston, Massachusetts) (SIGGRAPH ’06). Association for Com utin Machiner , New York, NY, USA, 153–171. htt s://doi.or /10.1145 1185657.1185834 Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. 2021. FastNeRF: High-Fidelity Neural Rendering at 200FPS. arXiv:2103.10380 (March 2021). Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional Sequence to Sequence Learning. In Proceedings ofthe 34th International Conference on Machine Learning - Volume 70 (Sydney, NSW, Australia) (ICML’17). JMLR.org, 1243—-1252. Xavier Glorot and Yoshua Bengio. 2010. Understanding the Dificulty of Training Deep Feedforward Neural Networks. In Proc. 13th International Conference on Artificial Intelligence and Statistics (Sardinia, Italy, May 13–15). JMLR.org, 249–256. Saeed Hadadan, Shuhong Chen, and Matthias Zwicker. 2021. Neural radiosity. ACM Transactions on Graphics 40, 6 (Dec. 2021), 1—-11. https://doi.org/10.1145/3478513. 3480569 David Mone Harris and Sarah L. Harris. 2013. 3.4.2 - State Encodin s. In Digital Design and Computer Architecture (second ed.). Morgan Kaufmann, Boston, 129–131. https://doi.org/10.1016/B978-0-12-394424-5.00002-1 Jon Jansen and Louis Bavoil. 2010. Fourier Opacity Mapping. In Proceedings of the 2010 ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (Washington, D.C.) (I3D ’10). Association for Computing Machinery, New York, NY, USA, 165—-172. https://doi.org/10.1145/1730804.1730831 Chiyu Max Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas Funkhouser. 2020. Local Implicit Grid Representations for 3D Scenes. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 (June 2014). Derrick H. Lehmer. 1951. Mathematical Methods in Large-scale Computing Units. In Proceedings ofthe Second Symposium on Large Scale Digital Computing Machinery. Harvard University Press, Cambridge, United Kingdom, 141–146. Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. 2018. Noise2Noise: Learning Image Restoration without Clean Data. arXiv:1803.04189 (March 2018). Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. 2020. Neural Sparse Voxel Fields. NeurIPS (2020). https://lingjie0206.github.io/papers/ NSVF/ Julien N.P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, and Gordon Wetzstein. 2021. ACORN: Adaptive Coordinate Networks for Neural Representation. ACM Trans. Graph. (SIGGRAPH) (2021). Ishit Mehta, Michaël Gharbi, Connell Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chandraker. 2021. Modulated Periodic Activations for Generalizable Local Functional Representations. In IEEE International Conference on Computer Vision. IEEE. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsbur , Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed Precision Training. arXiv:1710.03740 (Oct. 2018). Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul Srinivasan, and Jonathan T. Barron. 2021. NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images. arXiv:2111.13679 (Nov. 2021). Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019. Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines. ACM Trans. Graph. 38, 4, Article 29 (July 2019), 14 pages. https://doi.org/10.1145/3306346.3322980 Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ra mamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View S nthesis. In ECCV. Thomas Müller. 2021. Tiny CUDA Neural Network Framework. https://github.com/nvlabs/tiny-cuda-nn. Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. 2019. Neural Importance Sampling. ACM Trans. Graph. 38, 5, Article 145 (Oct. 2019), 19 pages. https://doi.org/10.1145/3341156 Thomas Müller, Fabrice Rousselle, Alexander Keller, and Jan Novák. 2020. Neural Control Variates. ACM Trans. Graph. 39, 6, Article 243 (Nov. 2020), 19 pages. https: //doi.org/10.1145/3414685.3417804 Thomas Müller, Fabrice Rousselle, Jan Novák, and Alexander Keller. 2021. Real-time Neural Radiance Caching for Path Tracing. ACM Trans. Graph. 40, 4, Article 36 (Aug. 2021), 16 pages. https://doi.org/10.1145/3450626.3459812 Ken Museth. 2013. VDB: High-Resolution Sparse Volumes with Dynamic Topology. ACM Trans. Graph. 32, 3, Article 27 (July 2013), 22 pages. https://doi.org/10.1145/ 2487228.2487235 Ken Museth. 2021. NanoVDB: A GPU-Friendly and Portable VDB Data Structure For Real-Time Renderin And Simulation. In ACM SIGGRAPH 2021 Talks (Virtual Event, USA) (SIGGRAPH ’21). Association for Com utin Machiner , New York, NY, USA, Article 1, 2 pages. https://doi.org/10.1145/3450623.3464653 Thomas Nef, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H. Mueller, Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, and Markus Steinberger. 2021. DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks. Computer Graphics Forum 40, 4 (2021). https://doi.org/10.1111/cgf.14340 Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. 2013. Real Time 3D Reconstruction at Scale Using Voxel Hashing. ACM Trans. Graph. 32, 6, Article 169 (nov 2013), 11 pages. https://doi.org/10.1145/2508363.2508374 Fakir S. Nooruddin and Greg Turk. 2003. Simplification and Repair of Polygonal Models Using Volumetric Techniques. IEEE Transactions on Visualization and Computer Graphics 9, 2 (apr 2003), 191––205. https://doi.org/10.1109/TVCG.2003.1196006 Melissa E. O’Neill. 2014. PCG: A Family of Simple Fast Space-Eficient Statistically Good Algorithms for Random Number Generation. Technical Report HMC-CS-2014-0905. Harvey Mudd College, Claremont, CA. Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Love-grove. 2019. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. arXiv:1901.05103 (Jan. 2019). Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. 2020a. Convolutional Occupancy Networks. In European Conference on Computer Vision (ECCV). Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. 2020b. Convolutional Occupancy Networks. (2020). arXiv:2003.04618 [cs.CV] Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically Based Rendering: From Theory to Implementation (3rd ed.) (3rd ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. 1266 pages. Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. 2020. Implicit Neural Representations with Periodic Activation Functions. In Proc. NeurIPS Cheng Sun, Min Sun, and Hwann-Tzong Chen. 2021. Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction. arXiv:2111.11215 (Nov. 2021). Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. 2021. Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes. (2021). Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. 2020. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. NeurIPS (2020). htt s://bmild. ithub.io/fourfeat/index.htm Danhang Tang, Mingsong Dou, Peter Lincoln, Philip Davidson, Kaiwen Guo, Jonathan Taylor, Sean Fanello, Cem Keskin, Adarsh Kowdle, Sofien Bouaziz, Shahram Izadi, and Andrea Ta liasacchi. 2018. Real-Time Com ression and Streamin of 4D Per formances. ACM Trans. Graph. 37, 6, Article 256 (dec 2018), 11 pages. https: //doi.or /10.1145/3272127.3275096 Matthias Teschner, Bruno Heidelberger, Matthias Müller, Danat Pomeranets, and Markus Gross. 2003. O timized S atial Hashin for Collision Detection of De formable Objects. In Proceedings ofVMV’03, Munich, Germany. 47–54. Sergios Theodoridis. 2008. Pattern Recognition. Elsevier. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. arXiv:1706.03762 (June 2017). Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. 2021. Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields. arXiv:2112.03907 (Dec. 2021). Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn. 2021. NeX: Real-time View Synthesis with Neural Basis Expansion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2021a. Plenoxels: Radiance Fields without Neural Networks. arXiv:2112.05131 (Dec. 2021). Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. 2021b. PlenOctrees for Real-time Rendering of Neural Radiance Fields. In ICCV. ## A SMOOTH INTERPOLATION One may desire smoother interpolation than the d-linear interpolation that our multiresolution hash encoding uses by default. In this case, the obvious solution would be using a d-quadratic or d-cubic interpolation, both of which are however very expensive due to requiring the lookup of $_ { 3 } d$ and $4 ^ { d }$ instead of $2 ^ { d }$ vertices, respectively. As a low-cost alternative, we recommend applying the smoothstep function, $$ S _ { 1 } ( x ) = x ^ { 2 } ( 3 - 2 x ) ,\tag{5} $$ to the d-linear interpolation weights. Crucially, the derivative of the smoothstep, $$ S _ { 1 } ^ { \prime } ( x ) = 6 x ( 1 - x ) ,\tag{6} $$ vanishes at 0 and at 1, causing the discontinuity in the derivatives of the encoding to vanish by the chain rule. The encoding thus becomes $C ^ { 1 }$ -smooth. However, by this trick, we have merely traded discontinuities for zero-points in the individual levels which are not necessarily more desirable. So, we ofset each level by half of its voxel size $1 / ( 2 N _ { l } )$ which prevents the zero derivatives from aligning across all levels. The encoding is thus able to learn smooth, non-zero derivatives for all spatial locations x. For higher-order smoothness, higher-order smoothstep functions $S _ { n }$ can be used at small additional cost. In practice, the computational cost of the 1st order smoothstep function $S _ { 1 }$ is hidden by memory bottlenecks, making it essentially free. However, the reconstruction quality tends to decrease as higher-order interpolation is used. This is why we do not use it by default. Future research is needed to explain the loss of quality. ## B IMPLEMENTATION DETAILS OF NGLOD We designed our implementation of NGLOD [Takikawa et al. 2021] such that it closely resembles that of our hash encoding, only dif- fering in the underlying data structure; i.e. using the vertices of an octree around ground-truth triangle mesh to store collision-free feature vectors, rather than relying on hash tables. This results in a notable diference to the original NGLOD: the looked-up fea- ture vectors are concatenated rather than summed, which in our implementation serendipitously resulted in higher reconstruction quality compared to the summation of an equal number of trainable parameters. The octree implies a fixed growth factor $b = 2 ,$ which leads to a smaller number of levels than our hash encoding. We obtained the most favorable performance vs. quality trade-of at a roughly equal number of trainable parameters as our method, through the following configuration: (1) the number of feature dimensions per entry is $F = 8 ,$ (2) the number of levels is $L = 1 0 ;$ , and (3) look-ups start at level $l = 4 .$ The last point is important for two reasons: first, it matches the coarsest resolution of our hash tables $2 ^ { 4 } = 1 6 = N _ { \operatorname* { m i n } }$ , and second, it prevents a performance bottleneck that would arise when all threads of the GPU atomically accumulate gradients in few, coarse entries. We experimentally verified that this does not lead to reduced quality, compared to looking up the entire hierarchy. ## C REAL-TIME SDF TRAINING DATA GENERATION In order to not bottleneck our SDF training, we must be able to generate a large number of ground truth signed distances to high- resolution meshes very quickly (∼millions per second). ## C.1 Eficient Sampling of 3D Training Positions Similar to prior work [Takikawa et al. 2021], we distribute some (1/8th) of our training positions uniformly in the unit cube, some (4/8ths) uniformly on the surface of the mesh, and the remainder (3/8ths) perturbed from the surface of the mesh. The uniform samples in the unit cube are trivial to generate using any pseudorandom number generator; we use a GPU implementation of PCG32 [O’Neill 2014]. To generate the uniform samples on the surface ofthe mesh, we compute the area of each triangle in a preprocessing step, normalize the areas to represent a probability distribution, and store the corre- sponding cumulative distribution function (CDF) in an array. Then, for each sample, we select a triangle proportional to its area by the inversion method—a binary search of a uniform random number over the CDF array—and sample a uniformly random position on that triangle by standard sample warping [Pharr et al. 2016]. Lastly, for those surface samples that must be perturbed, we add a random 3D vector, each dimension independently drawn from a logistic distribution (similar shape to a Gaussian, but cheaper to compute) with standard deviation r/1024, where r is the bounding radius of the mesh. Octree sampling for NGLOD. When training our implementation of Takikawa et al. [2021], we must be careful to rarely generate training positions outside of octree leaf nodes. To this end, we replace the uniform unit cube sampling routine with one that creates uniform 3D positions in the leaf nodes of the octree by first rejection sampling a uniformly random leaf node from the array of all nodes and then generating a uniform random position within the node’s voxel. Fortunately, the standard deviation $r / 1 0 2 4$ of our logistic perturbation is small enough to almost never leave the octree, so we do not need to modify the surface sampling routine. ## C.2 Eficient Signed Distances to the Triangle Mesh For each sampled 3D position x, we must compute the signed distance to the triangle mesh. To this end, we first construct a triangle bounding volume hierarchy (BVH) with which we perform eficient unsigned distance queries; O log $N _ { \mathrm { t r i a n g l e s } } )$ on average. Next, we sign these distances by tracing 32 “stab rays” [Nooruddin and Turk 2003], which we distribute uniformly over the sphere using a Fibonacci lattice that is pseudorandomly and independently ofset for every training position. If any of these rays reaches infinity, the corresponding position x is deemed “outside” of the object and the distance is marked positive. Otherwise, it is marked negative.3 For maximum eficiency, we use NVIDIA ray tracing hardware through the OptiX 7 framework, which is over an order of magnitude faster than using the previously mentioned triangle BVH for ray- shape intersections on our RTX 3090 GPU. ## D BASELINE MLPS WITH FREQUENCY ENCODING In our signed distance function (SDF), neural radiance caching (NRC), and neural radiance and density fields (NeRF) experiments, we use an MLP prefixed by a frequency encoding as baseline. The respective architectures are equal to those in the main text, except that the MLPs are larger and that the hash encoding is replaced by sine and cosine waves (SDF and NeRF) or triangle waves (NRC). The following table lists the number of hidden layers, neurons per hidden layer, frequency cascades (each scaled by a factor of 2 as per Vaswani et al. [2017]), and adjusted learning rates.
PrimitiveHidden layersNeuronsFrequenciesLearning rate
SDF812810 $3 \cdot 1 0 ^ { - 4 }$
NRC36410 $1 0 ^ { - 2 }$
NeRF7/1256 / 25616 / 410-3
For NeRF, the first listed number corresponds to the density MLP and the second number to the color MLP. For SDFs, we make two ad- ditional changes: (1) we optimize against the relative $\mathcal { L } ^ { 2 }$ loss [Lehtinen et al. 2018] instead of the MAPE described in the main text, and (2) we perturb training samples with a standard deviation of $r / 1 2 8$ as opposed to the value of $r / 1 0 2 4$ from Appendix C.1. Both changes smooth the loss landscape, resulting in a better reconstruction with the above configuration. Notably, even though the above configurations have fewer param- eters and are slower than our configurations with hash encoding, they represent favorable performance vs. quality trade-ofs. An equal parameter count comparison would make pure MLPs too expensive due to their scaling with $O ( n ^ { 2 } )$ as opposed to the sub-linear scaling of trainable encodings. On the other hand, an equal throughput com- parison would require prohibitively small MLPs, thus underselling the reconstruction quality that pure MLPs are capable of. We also experimented with Fourier features [Tancik et al. 2020] but did not obtain better results compared to the axis-aligned fre- quency encodings mentioned previously. ## E ACCELERATED NERF RAY MARCHING The performance of ray marching algorithms such as NeRF strongly depends on the marching scheme. We utilize three techniques with imperceivable error to optimize our implementation: (1) exponential stepping for large scenes, (2) skipping of empty space and occluded regions, and (3) compaction of samples into dense bufers for eficient execution. ## E.1 Ray Marching Step Size and Stopping In synthetic NeRF scenes, which we bound to the unit cube $[ 0 , 1 ] ^ { 3 } ,$ we use a fixed ray marching step size equal to $\Delta t : = \sqrt { 3 } / 1 0 2 4 ;$ ; √3 represents the diagonal of the unit cube. In all other scenes, based on the intercept theorem4, we set the step size proportional to the distance t along the ray $\Delta t : = t / 2 5 6$ clamped to the interval $\left[ { \sqrt { 3 } } / { 1 0 2 4 } , s \cdot { \sqrt { 3 } } / { 1 0 \bar { 2 } 4 } \right]$ , where s is size of the largest axis of the scene’s bounding box. This choice of step size exhibits exponential growth in t, which means that the compu tation cost grows only logarithmically in scene diameter, with no perceivable loss of quality. Lastly, we stop ray marching and set the remaining contribution to zero as soon as the transmittance of the ray drops below a threshold; in our case $\epsilon = 1 0 ^ { - 4 }$ Related work. Mildenhall et al. [2019] already identified a nonlinear step size as benefitial: they recommend sampling uniformly in the disparity-space of the average camera frame, which is more aggressive than our exponential stepping, requiring on one hand only a constant number of steps, but on the other hand can lead to a loss of fidelity compared to exponential stepping [Nef et al. 2021]. In addition to non-linear stepping, some prior methods propose to warp the 3D domain of the scene towards the origin, thereby improving the numerical properties of their input encodings [Barron et al. 2021b; Mildenhall et al. 2020; Nef et al. 2021]. This causes rays to curve, which leads to a worse reconstruction in our implementation. In contrast, we linearly map input coordinates into the unit cube before feeding them to our hash encoding, relying on its exponential multiresolution growth to reach a proportionally scaled maximum resolution $N _ { \mathrm { m a x } }$ with a constant number of levels (variable L as in Equation (3)) or logarithmically many levels L (constant b). ## E.2 Occupancy Grids To skip ray marching steps in empty space, we maintain a cascade of K multiscale occupancy grids, where K = 1 for all synthetic NeRF scenes (single grid) and $K \in [ 1 , 5 ]$ for larger real-world scenes (up to 5 grids, depending on scene size). Each grid has a resolution of 1283, spanning a geometrically growing domain $[ - 2 ^ { k - 1 } + 0 . 5 , 2 ^ { k - 1 } + 0 . 5 ] ^ { 3 }$ that is centered around (0.5, 0.5, 0.5). Each grid cell stores occupancy as a single bit. The cells are laid out in Morton (z-curve) order to facilitate memory-coherent traversal by a digital diferential analyzer (DDA). During ray marching, whenever a sample is to be placed according to the step size from the previous section, the sample is skipped if its grid cell’s bit is low. Which one of the K grids is queried is determined by both the sample position x and the step size Δt: among the grids covering x, the finest one with cell side-length larger than Δt is queried. Updating the occupancy grids. To continually update the occu-pancy grids while training, we maintain a second set of grids that have the same layout, except that they store full-precision floating point density values rather than single bits. We update the grids after every 16 training iterations by performing the following steps. We (1) decay the density value in each grid cell by a factor of 0.95, (2) randomly sample M candidate cells, and set their value to the maximum of their current value and the density component of the NeRF model at a random location within the cell, and (3) update the occupancy bits by thresholding each cell’s density with $t = 0 . 0 1 \cdot 1 0 2 4 / { \sqrt { 3 } } ,$ , which corresponds to thresholding the opacity of a minimal ray marching step by $1 - \exp ( - 0 . 0 1 ) \approx 0 . 0 1$ The sampling strategy of the M candidate cells depends on the training progress since the occupancy grid does not store reliable information in early iterations. During the first 256 training steps, we sample $M = K \cdot 1 2 8 ^ { 3 }$ cells uniformly without repetition. For sub-sequent training steps we set $M = K \cdot 1 2 8 ^ { 3 } / 2$ which we partition into two sets. The first $M / 2$ cells are sampled uniformly among all cells. Rejection sampling is used for the remaining samples to restrict selection to cells that are currently occupied. Related work. The idea of constraining the MLP evaluation to occupied cells has already been exploited in prior work on trainable, cell-based encodings [Liu et al. 2020; Sun et al. 2021; Yu et al. 2021a,b]. In contrast to these papers, our occupancy grid is independent from the learned encoding, allowing us to represent it more compactly as a bitfield (and thereby at a resolution that is decoupled from that of the encoding) and to utilize it when comparing against other methods that do not have a trained spatial encoding, e.g. “Ours: Frequency” in Table 2. Empty space can also be skipped by importance sampling the depth distribution, such as by resampling the result of a coarse prediction [Mildenhall et al. 2020] or through neural importance sampling [Müller et al. 2019] as done in DONeRF [Nef et al. 2021]. Table 3. Batch size, number of rays per batch, and number of samples per ray for our full method (“Ours: Hash”), our implementation of frequency encoding NeRF (“Ours: Freq.”) and mip-NeRF. Since the values corresponding to our method vary by scene, we report minimum and maximum values over the synthetic scenes from Table 2.
MethodBatch size二 Samples per rayX Rays per batch
Ours: Hash256 Ki3.1 to 25.710 Ki to 85 Ki
Ours: Freq.256 Ki2.5 to 929 Ki to 105 Ki
mip-NeRF1Mi128 coarse + 128 fine4Ki
## E.3 Number of Rays Versus Batch Size The batch size has a significant efect on the quality and speed of NeRF convergence. We found that training from a larger number of rays, i.e. incorporating more viewpoint variation into the batch, converged to lower error in fewer steps. In our implementation where the number of samples per ray is variable due to occupancy, we therefore include as many rays as possible in batches of fixed size rather than building variable-size batches from a fixed ray count. In Table 3, we list ranges of the resulting number of rays per batch and corresponding samples per ray. We use a batch size of 256 Ki, which resulted in the fastest wall-clock convergence in our experiments. This is 4× smaller than the batch size chosen in mip- NeRF, likely due to the larger number of samples each of their rays requires. However, due to the myriad other diferences across implementations, a more detailed study must be carried out to draw a definitive conclusion. Lastly, we note that the occupancy grid in our frequency-encoding baseline (“Ours: Freq.”; Appendix D) produces even fewer samples than when used alongside our hash encoding. This can be explained by the slightly more detailed reconstruction of the hash encoding: when the extra detail is finer than the occupancy grid resolution, its surrounding empty space can not be efectively culled away and must be traversed by extra steps.