Point cloud

Point cloud is a set of data points in three-dimensional space, each defined by Cartesian coordinates (X, Y, Z) and optionally by extra attributes such as color, intensity, surface normal, or a timestamp. The points sample the external surfaces of objects or an environment rather than describing them as continuous surfaces, so a point cloud is a discrete, unstructured representation of three-dimensional geometry.

Point clouds are produced by LiDAR sensors, depth cameras, other 3D scanners, and photogrammetry software that measure many points on the surfaces around a device. In virtual reality (VR) and augmented reality (AR) they are a common intermediate form of spatial data: head-worn and handheld devices reconstruct the geometry of a room as a point cloud, then use it for tracking, spatial mapping, occlusion, and 3D reconstruction. The same representation underlies volumetric capture and recent radiance-field rendering methods.

How a point cloud is structured

A point cloud is a collection of independent points with no inherent connectivity or ordering, which distinguishes it from a polygon mesh (vertices joined into faces) or a raster image (pixels on a fixed 2D grid). Because the data is unstructured it captures three-dimensional geometry without enforcing a fixed spatial layout, at the cost of leaving surface topology implicit.

Each point always carries its position. Depending on the capture method and use case, points may also store RGB color, a reflectance or laser-return intensity value, a surface normal, a timestamp, and per-point classification labels. A cloud can be dense (millions of closely spaced points covering a surface) or sparse (a thinner sampling, such as the feature points an AR device tracks), and that density distinction drives how the data is processed, stored, and compressed.

Capture methods

The main ways to generate a point cloud differ in range, density, and the conditions they need.

• LiDAR (light detection and ranging) emits laser pulses and measures the return time to compute distance, producing accurate measurements at long range but comparatively sparse data.
• Depth cameras (for example structured-light or time-of-flight sensors such as Microsoft Kinect) combine an infrared illuminator with an imager to recover per-pixel depth, which back-projects to a point cloud.
• Photogrammetry and structure-from-motion reconstruct a 3D scene from many overlapping 2D photographs, yielding dense, colored reconstructions but requiring good lighting and textured surfaces.

LiDAR and photogrammetry trade off against each other: LiDAR is strong at long-range accuracy with sparser output, while photogrammetry can produce dense colored reconstructions but depends on lighting and surface texture.

File formats

Several formats are used to exchange and archive point clouds.

Format	Origin	Notes
PLY (Polygon File Format / Stanford Triangle Format)	Developed at Stanford University by Greg Turk and colleagues in the mid-1990s	Designed to store data from 3D scanners; holds vertices with optional color, transparency, surface normals and texture coordinates, plus faces; has ASCII and binary variants.
LAS (LASer)	Maintained by the American Society for Photogrammetry and Remote Sensing (ASPRS)	Open binary format for interchange and archiving of LiDAR point cloud data; each record stores X, Y, Z plus fields such as GPS time, intensity, return number, color and classification. Version 1.4 R15 was released on 9 July 2019.
PCD (Point Cloud Data)	Native format of the Point Cloud Library (PCL)	Header declares per-point FIELDS (for example x y z rgb), SIZE, TYPE and COUNT; stores data as ASCII, binary, or binary_compressed.

Conversion to a mesh

For rendering and physics, a point cloud is often converted into a polygon surface. Common surface-reconstruction approaches build a triangle network directly from the points (Delaunay triangulation, alpha shapes, or ball pivoting) or first convert the points into a volumetric distance field and then extract a surface with the marching cubes algorithm. In AR specifically, a point cloud assembled across frames is used as the input to generate a triangle mesh that gives a more complete and renderable model of the surroundings, which the application can then use for occlusion and collision against virtual objects.

Use in AR tracking and spatial mapping

Markerless AR systems build and track against a point cloud as part of SLAM (simultaneous localization and mapping). Google's ARCore detects visually distinct features in the camera image, called feature points, and uses them to compute its change in location; it combines this visual information with the device IMU to estimate the camera pose over time. ARCore then looks for clusters of feature points that lie on common horizontal or vertical surfaces, such as tables or walls, and exposes those surfaces to applications as geometric planes. Because plane detection depends on feature points, untextured surfaces such as a blank white wall may not be detected reliably.

Apple's ARKit exposes a comparable sparse cloud of tracked points through its raw feature points, an ARPointCloud that Apple documents as the current intermediate results of the scene analysis the framework uses for world tracking; its main intention is a debug visualization of what the underlying tracking algorithm is processing rather than a stable map for applications to build on. By combining many such per-frame observations, an AR system builds a spatial map of the environment that lets the application reason about the 3D layout of the space. The technique generalizes across markerless inside-out tracking systems used on devices such as HoloLens, Magic Leap, and the Apple Vision Pro, and historically on Google's Project Tango.

Use with depth-sensing headsets

Devices with a dedicated depth sensor turn raw depth into denser geometry. The LiDAR Scanner that Apple introduced on the 2020 iPad Pro feeds ARKit, whose scene-reconstruction feature aggregates depth data across frames (available at 60 Hz, tied to each AR frame) into a 3D mesh. The mesh is assembled from several smaller submeshes stored in ARMeshAnchor objects; regions the device is close to are tessellated more finely than distant ones, and areas the device is no longer facing are progressively coarsened or discarded. One independent analysis reported roughly 30,000 mesh vertices on average for a detailed room. Beyond the reconstructed mesh, ARKit also gives developers a LiDAR-derived depth map directly: on LiDAR-capable devices the scene depth API populates each AR frame with an ARDepthData object containing a per-pixel depth map (in meters) and a confidence map, updated at 60 Hz, which back-projects to a point cloud.

Use in radiance-field rendering

Point clouds also seed modern photorealistic scene capture. The method 3D Gaussian Splatting, presented by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis in ACM Transactions on Graphics in July 2023, starts from the sparse points produced during camera calibration (a structure-from-motion point cloud) and represents the scene with 3D Gaussians, then optimizes them to reproduce the input views and renders the result in real time. The studio Magnopus describes the pipeline the same way (estimate a point cloud by structure from motion, convert each point to a Gaussian) and reports that the real-time rasterization and detail quality make the approach useful for virtual reality backdrops and immersive scenes, citing Meta's experiments using Gaussian splatting for Codec Avatars toward photorealistic telepresence.

Compression for immersive media

Because dense point clouds are large, the Moving Picture Experts Group (MPEG) standardized two point cloud compression methods within the ISO/IEC 23090 (MPEG-I) family for immersive media. Video-based Point Cloud Compression (V-PCC) targets dense point clouds by segmenting the cloud into surface patches, projecting them into 2D atlases, and coding those with a video codec; Geometry-based Point Cloud Compression (G-PCC) targets sparse point clouds by encoding geometry and attributes directly in 3D using an octree. The overview paper by Graziosi and colleagues (2020) lists the target applications as 6-degree-of-freedom VR/AR, immersive telepresence, autonomous-vehicle LiDAR data, and cultural-heritage archival, and notes the first versions were planned for release in 2020.

See also

• Photogrammetry
• Depth sensing
• Spatial mapping
• SLAM
• 3D mesh