Sensor Data Formats Reference

Overview

This document describes the pixel data formats produced by each sensor type in the UnrealCV sensor system. Understanding these formats is essential for correctly interpreting captured data in downstream processing pipelines.

Related Documents:

• Sensor System Architecture - Sensor system architecture

• UnrealCV+ Plugin - UnrealCV+ recording feature summary

Lit (RGB) Data

Sensor: ULitCamSensor

Format: FColor (8-bit sRGB)

Data Layout:

Pixel[0]        Pixel[1]        Pixel[2]     ...
+---+---+---+   +---+---+---+   +---+---+---+
| R | G | B |   | R | G | B |   | R | G | B |
+---+---+---+   +---+---+---+   +---+---+---+
(Row-major order, scanline by scanline)

Range: 0-255 per channel (8-bit)

Color Space: sRGB

Dimensions: Width × Height × 4 bytes per pixel

Python Decoding:

import numpy as np
from PIL import Image

# lit_0001.png is already in PNG format
img = Image.open('lit_0001.png')
rgb_array = np.array(img)  # Shape: (Height, Width, 3)

# Or read raw bytes
with open('lit_0001.png', 'rb') as f:
    data = f.read()

Depth Data

Sensor: UDepthCamSensor

Format: float32 (32-bit float) or float16 (16-bit half-float)

Data Layout:

Pixel[0]        Pixel[1]        Pixel[2]     ...
+-----------+   +-----------+   +-----------+
|  4 bytes  |   |  4 bytes  |   |  4 bytes  |
|  (float)  |   |  (float)  |   |  (float)  |
+-----------+   +-----------+   +-----------+
(Row-major order, scanline by scanline)

Depth Modes:

PlaneDepth (Default):

Distance from the camera’s near plane to the pixel’s depth value. Formula: Depth = distance(near_plane, pixel_world_position)

DistToCamCenter:

Euclidean distance from camera center to pixel world position.

Range: 0.0 to infinity (clamped by far plane)

Units: Unreal units (cm by default)

Python Decoding:

import numpy as np

# Float16 PNG
depth_png = Image.open('depth_0001.png')
depth_array = np.array(depth_png).astype(np.float32) / 1000.0  # if stored as mm

# Raw float32
with open('depth_0001.raw', 'rb') as f:
    data = np.frombuffer(f.read(), dtype=np.float32)
depth_array = data.reshape((height, width))

Normal Data

Sensor: UNormalCamSensor

Format: FColor (8-bit encoded normals)

Data Layout:

Channel   Meaning              Range
-------   ------------------   -----
R         Normal X             0-255 (mapped to -1.0 to 1.0)
G         Normal Y             0-255 (mapped to -1.0 to 1.0)
B         Normal Z             0-255 (mapped to  0.0 to 1.0)
A         Unused               255

Encoding Formula:

Color.R = (Normal.X + 1.0) * 127.5
Color.G = (Normal.Y + 1.0) * 127.5
Color.B = Normal.Z * 255.0
Color.A = 255

Decoding Formula:

Normal.X = (Color.R / 255.0) * 2.0 - 1.0
Normal.Y = (Color.G / 255.0) * 2.0 - 1.0
Normal.Z = Color.B / 255.0
Normal = normalize(Normal)

Python Decoding:

import numpy as np
from PIL import Image

normal_img = Image.open('normal_0001.png')
normal_array = np.array(normal_img).astype(np.float32) / 255.0

normal_array[:, :, 0] = normal_array[:, :, 0] * 2.0 - 1.0  # X
normal_array[:, :, 1] = normal_array[:, :, 1] * 2.0 - 1.0  # Y
normal_array[:, :, 2] = normal_array[:, :, 2]               # Z

Optical Flow Data

Sensor: UFlowCamSensor

Format: FColor (8-bit encoded flow vectors)

Data Layout:

Channel   Meaning              Range           Scale
-------   ------------------   --------------  ------
R         Flow X (horizontal)  0-255           See below
G         Flow Y (vertical)    0-255           See below
B         Mask                0 or 255        Valid/Invalid
A         Unused               255

Encoding:

Flow vectors are encoded with an offset to support negative values:

Color.R = flow_x * scale + 128
Color.G = flow_y * scale + 128
Color.B = 255 if valid else 0

Default Scale: 1.0 (adjustable in material)

Range: Approximately -128 to +127 per pixel

Python Decoding:

import numpy as np
from PIL import Image

flow_img = Image.open('flow_0001.png')
flow_array = np.array(flow_img).astype(np.float32)

# Decode flow vectors
flow_x = (flow_array[:, :, 0] - 128.0)  # Horizontal
flow_y = (flow_array[:, :, 1] - 128.0)  # Vertical

# Mask (valid pixels)
valid_mask = flow_array[:, :, 2] > 127

Segmentation Mask

Sensor: UAnnotationCamSensor

Format: FColor (8-bit annotation colors)

Data Layout:

Channel   Meaning              Range
-------   ------------------   -----
R         Annotation Red       0-255
G         Annotation Green     0-255
B         Annotation Blue     0-255
A         Unused               255

Color Generation:

Colors are generated by FColorGenerator using bit manipulation:

Index 0:  R=1,   G=2,   B=4   (0x010204)
Index 1:  R=1,   G=4,   B=8   (0x010408)
Index 2:  R=1,   G=8,   B=16  (0x010810)
Index 3:  R=1,   G=16,  B=32  (0x011020)
...

Color-to-ID Mapping:

// From FObjectAnnotator::GetAnnotationColors()
TMap<FString, FColor> Colors = FObjectAnnotator::GetAnnotationColors();

// Iterate to get actor-color mappings
for (const auto& Pair : Colors)
{
   FString ActorName = Pair.Key;
   FColor Color = Pair.Value;
   int32 ID = (Color.R << 0) | (Color.G << 8) | (Color.B << 16);
}

Python Decoding:

import numpy as np
from PIL import Image
from collections import defaultdict

# Get color-to-actor mapping from metadata
with open('metadata.json', 'r') as f:
    metadata = json.load(f)

seg_img = Image.open('seg_0001.png')
seg_array = np.array(seg_img)

# Create mask for specific actor
actor_color = metadata['foreground_color']  # e.g., [1, 2, 4]
mask = np.all(seg_array == actor_color, axis=-1)

One Object Mask

Sensor: UOneObjMaskCamSensor

Format: FColor (8-bit)

Description: Binary mask for a single specified actor. The actor appears white, everything else appears black.

Data Layout:

Channel   Meaning              Range
-------   ------------------   -----
R         Mask Value          0 (background) or 255 (actor)
G         Mask Value          0 (background) or 255 (actor)
B         Mask Value          0 (background) or 255 (actor)
A         Unused               255

Python Decoding:

import numpy as np
from PIL import Image

mask_img = Image.open('one_obj_mask_0001.png')
mask_array = np.array(mask_img)[:, :, 0]  # Single channel

# Boolean mask
is_foreground = mask_array > 127

One Object Lit

Sensor: UOneObjLitCamSensor

Format: FColor (8-bit sRGB)

Description: Alpha-only visibility data for the selected actor. Every RGB pixel is written as 0,0,0 by the GPU capture-output pass; downstream postprocessing does not rewrite the PNG. Opaque occluders suppress the selected actor’s alpha contribution, while translucent materials and Groom strands retain continuous material-opacity or hair-coverage values at visible pixels.

The supported new implementation is fixed at native 1x resolution. Hard binary 3x3 neighborhoods receive a small spatial filter, while any neighborhood that already contains intermediate alpha preserves its native center value. No supersampled, native-mask, or resolved auxiliary render targets are allocated.

unrealcv.OneObjLit.InvertAlpha defaults to 1 for compatibility with the original output: background visibility is 255 and foreground visibility is 0. Set it to 0 for conventional foreground alpha (background 0, foreground 255). The setting affects only alpha; RGB remains zero.

The legacy implementation and the explicit oneobjlit_legacy command use the same direct GPU RGB-zeroing output pass while preserving their captured alpha polarity. Both implementations write directly into the sensor render target before asynchronous readback.

Data Layout: Same as Lit data

Python Decoding:

import numpy as np
from PIL import Image

lit_img = Image.open('one_obj_lit_0001.png')
lit_array = np.array(lit_img)

# Default compatibility mode: background visibility
alpha = lit_array[:, :, 3]

To capture with the previous ShowOnlyList implementation, use:

vget /camera/0/oneobjlit_legacy output.png ActorName

The legacy command renders only the selected actor, does not account for occlusion by other actors, and preserves the old SceneColor alpha polarity (selected foreground near 0, empty background near 255). Its RGB channels are also zero. oneobjlit remains the supported occlusion-aware output.

Shadow Catcher

Sensor: UShadowCatcherCamSensor

Format: FColor (8-bit sRGB)

Description: Object composited over white background with baked shadows. Used for matting workflows.

Data Layout: Same as Lit data

Stencil Mask

Sensor: UStencilMaskCamSensor

Format: FColor (8-bit encoded stencil values)

Data Layout:

Channel   Meaning              Range
-------   ------------------   -----
R         Stencil Value       0-255
G         Stencil Value       0-255
B         Stencil Value       0-255
A         Unused               255

Metadata JSON

File: metadata.json

Schema:

{
   "scene": {
      "scene_id": "scene_0001",
      "timestamp": "2025-01-15T10:30:00Z"
   },
   "camera": {
      "cid": "CID-FusionCamPawn-00",
      "location": {"x": 100.0, "y": 200.0, "z": 150.0},
      "rotation": {"pitch": 0, "yaw": 45, "roll": 0},
      "fov": 90.0,
      "projection": "perspective"
   },
   "objects": [
      {
         "id": "foreground_001",
         "category": "Human",
         "annotation_color": [1, 2, 4],
         "location": {"x": 0, "y": 0, "z": 0},
         "bounds": {"min": [-50, -50, 0], "max": [50, 50, 200]}
      }
   ],
   "trajectory": {
      "type": "rotate_left_45",
      "frame_count": 121,
      "fps": 30
   },
   "occlusion": {
      "ratio": 0.35,
      "occluders": ["chair_001", "table_001"]
   }
}

Data Format Summary Table

Sensor	Format	Channels	Typical Range
LitCamSensor	PNG/FColor	3 or 4	0-255 (sRGB)
DepthCamSensor	RAW/Float	1	0.0 - far_plane
NormalCamSensor	PNG/FColor	3	Encoded -1 to 1
FlowCamSensor	PNG/FColor	3	-128 to +127
AnnotationCamSensor	PNG/FColor	3	0-255 (color ID)
OneObjMaskCamSensor	PNG/FColor	1	0 or 255
OneObjLitCamSensor	PNG/FColor	4	0-255 (RGBA)
ShadowCatcherCamSensor	PNG/FColor	3 or 4	0-255 (sRGB)
StencilMaskCamSensor	PNG/FColor	3	0-255 (stencil)

File Extensions by Format

Extension	Format
.png	8-bit RGB
.hdr	32-bit HDR
.exr	Float EXR
.raw	Raw binary
.json	Metadata

Common Processing Patterns

Pattern 1: Load All Modalities

import numpy as np
from PIL import Image

modalities = ['lit', 'depth', 'normal', 'seg']

data = {}
for mod in modalities:
    if mod == 'depth':
        # Depth might be float32
        with open(f'{mod}_0001.raw', 'rb') as f:
            data[mod] = np.frombuffer(f.read(), dtype=np.float32)
            data[mod] = data[mod].reshape((480, 854))
    else:
        img = Image.open(f'{mod}_0001.png')
        data[mod] = np.array(img)

Pattern 2: Apply Segmentation Mask

# Get foreground segmentation
seg = np.array(Image.open('seg_0001.png'))

# Load lit image
lit = np.array(Image.open('lit_0001.png'))

# Create masked view
foreground_mask = np.all(seg == foreground_color, axis=-1)
foreground = lit.copy()
foreground[~foreground_mask] = 0

Pattern 3: Compute Object Masks

from PIL import Image
import numpy as np

# Load segmentation
seg = np.array(Image.open('seg_0001.png'))

# Get unique colors
unique_colors = np.unique(seg.reshape(-1, seg.shape[2]), axis=0)

# Create masks for each object
for i, color in enumerate(unique_colors):
    mask = np.all(seg == color, axis=-1)
    obj_mask = Image.fromarray((mask * 255).astype(np.uint8))
    obj_mask.save(f'object_{i:04d}_mask.png')

See Also

• Sensor System Architecture - Sensor system architecture and APIs

• UnrealCV+ Plugin - UnrealCV+ dataset and recording overview