Sensor System Architecture

Overview

The Sensor System provides a unified multi-pass rendering orchestrator for capturing ground truth data from Unreal Engine scenes. UFusionCamSensor manages 5 specialized sensor types in a single render pass, enabling simultaneous capture of RGB, depth, normals, optical flow, and segmentation data.

Header: Source/UnrealCV/Public/Sensor/CameraSensor/FusionCamSensor.h

Architecture Diagram

+---------------------------+
| UFusionCamSensor         |  <-- Orchestrator Component
+---------------------------+
| - FusionSensors[5]       |  <-- Array of specialized sensors
| - PreviewCamera          |
| - MovieQualityRenderer   |
+---------------------------+
| LitCamSensor     | RGB    |
| DepthCamSensor   | Depth  |
| NormalCamSensor  | Normals|
| AnnotationCamSensor| Seg  |
| FlowCamSensor    | Flow   |
+---------------------------+

+---------------------------+
| UBaseCameraSensor         |  <-- Base Class
+---------------------------+
| - SceneCaptureComponent2D |
| - TextureRenderTarget2D   |
| - FRHIGPUTextureReadback |
+---------------------------+
| - Capture()               |  <-- Sync capture
| - CaptureFast()           |  <-- Async capture
| - CaptureFastToFile()     |  <-- Async file write
+---------------------------+

UFusionCamSensor

UFusionCamSensor is the main orchestrator component that manages multiple specialized sensors for ground truth capture.

Component Hierarchy:

UFusionCamSensor (PrimitiveComponent)
├── UCameraComponent PreviewCamera
├── UBaseCameraSensor FusionSensors[0..4]
│   ├── ULitCamSensor
│   ├── UDepthCamSensor
│   ├── UAnnotationCamSensor
│   ├── UNormalCamSensor
│   └── UFlowCamSensor
├── UMovieQualityRenderComponent
└── Additional Sensors
    ├── UAnnotationCamSensor OneObjectMaskCamSensor
    ├── ULitCamSensor OneObjectLitCamSensor
    ├── UShadowCatcherCamSensor
    └── UStencilMaskCamSensor

Sensor Enumerations

ELitMode

UENUM(BlueprintType)
enum class ELitMode : uint8
{
   Lit,    // Standard lit rendering
   Slow    // Alternative slow path
};

EDepthMode

UENUM(BlueprintType)
enum class EDepthMode : uint8
{
   PlaneDepth,      // Distance from camera plane
   DistToCamCenter   // Distance to camera center point
};

ESegMode

UENUM(BlueprintType)
enum class ESegMode : uint8
{
   AnnotationComponent,  // Uses AnnotationComponent
   VertexColor,         // Uses vertex colors
   CustomStencil         // Uses custom stencil buffer
};

EPresetFilmSize

UENUM(BlueprintType)
enum class EPresetFilmSize : uint8
{
   F640x480,   // 640x480 (480p)
   F720p,      // 1280x720
   F1080p      // 1920x1080
};

Sensor Accessors

UFUNCTION(BlueprintCallable, Category = "unrealcv")
UDepthCamSensor* GetDepthCamSensor() const;

UFUNCTION(BlueprintCallable, Category = "unrealcv")
UNormalCamSensor* GetNormalCamSensor() const;

UFUNCTION(BlueprintCallable, Category = "unrealcv")
UAnnotationCamSensor* GetAnnotationCamSensor() const;

UFUNCTION(BlueprintCallable, Category = "unrealcv")
ULitCamSensor* GetLitCamSensor() const;

UFUNCTION(BlueprintCallable, Category = "unrealcv")
UFlowCamSensor* GetFlowCamSensor() const;

UFUNCTION(BlueprintCallable, Category = "unrealcv")
UShadowCatcherCamSensor* GetShadowCatcherCamSensor() const;

UFUNCTION(BlueprintCallable, Category = "unrealcv")
UStencilMaskCamSensor* GetStencilMaskCamSensor() const;

UFUNCTION(BlueprintCallable, Category = "unrealcv")
UMovieQualityRenderComponent* GetMovieQualityRenderer() const;

UFUNCTION(BlueprintCallable, Category = "unrealcv")
TArray<UBaseCameraSensor*> GetSensors() const;

Sensor Data Capture

RGB / Lit Data:

UFUNCTION(BlueprintPure, Category = "unrealcv")
void GetLit(TArray<FColor>& LitData, int& InOutWidth, int& InOutHeight,
             ELitMode LitMode = ELitMode::Lit);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SaveLitToFile(const FString& Filename);

Depth Data:

UFUNCTION(BlueprintPure, Category = "unrealcv")
void GetDepth(TArray<float>& DepthData, int& InOutWidth, int& InOutHeight,
              EDepthMode DepthMode = EDepthMode::PlaneDepth);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SaveDepthToFile(const FString& Filename);

Normal Data:

UFUNCTION(BlueprintPure, Category = "unrealcv")
void GetNormal(TArray<FColor>& NormalData, int& Width, int& Height);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SaveNormalToFile(const FString& Filename);

Optical Flow:

UFUNCTION(BlueprintPure, Category = "unrealcv")
void GetFlow(TArray<FColor>& FlowData, int& Width, int& Height);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SaveFlowToFile(const FString& Filename);

Segmentation:

UFUNCTION(BlueprintPure, Category = "unrealcv")
void GetSeg(TArray<FColor>& ObjMaskData, int& Width, int& Height,
            ESegMode SegMode = ESegMode::AnnotationComponent);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SaveSegToFile(const FString& Filename);

Object-Specific Capture:

UFUNCTION(BlueprintPure, Category = "unrealcv")
void GetOneObjMask(AActor* Actor, TArray<FColor>& Data, int& InOutWidth, int& InOutHeight);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SaveOneObjMaskToFile(AActor* Actor, const FString& Filename);

UFUNCTION(BlueprintPure, Category = "unrealcv")
void GetOneObjLit(AActor* Actor, TArray<FColor>& Data, int& InOutWidth, int& InOutHeight);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SaveOneObjLitToFile(AActor* Actor, const FString& Filename);

UFUNCTION(BlueprintPure, Category = "unrealcv")
void GetShadowCatcher(AActor* Actor, TArray<FColor>& Data, int& InOutWidth, int& InOutHeight);

UFUNCTION(BlueprintPure, Category = "unrealcv")
void GetStencilMask(AActor* Actor, TArray<FColor>& Data, int& InOutWidth, int& InOutHeight);

Camera Transform

UFUNCTION(BlueprintPure, Category = "unrealcv")
FVector GetSensorLocation();

UFUNCTION(BlueprintPure, Category = "unrealcv")
FRotator GetSensorRotation();

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetSensorLocation(FVector Location);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetSensorRotation(FRotator Rotator);

Film Size / Resolution

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetFilmSize(int Width, int Height);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
float GetFilmWidth();

UFUNCTION(BlueprintCallable, Category = "unrealcv")
float GetFilmHeight();

FOV / Projection

UFUNCTION(BlueprintCallable, Category = "unrealcv")
float GetSensorFOV();

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetSensorFOV(float FOV);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetProjectionType(ECameraProjectionMode::Type ProjectionType);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetOrthoWidth(float OrthoWidth);

Exposure Settings

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetExposureMethod(EAutoExposureMethod ExposureMethod);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetExposureBias(float ExposureBias);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetAutoExposureSpeed(float ExposureSpeedDown, float ExposureSpeedUp);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetAutoExposureBrightness(float MinBrightness, float MaxBrightness);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetApplyPhysicalCameraExposure(int ApplyPhysicalCameraExposure);

Rendering Settings

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetLitCaptureSource(ESceneCaptureSource CaptureSource);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetReflectionMethod(EReflectionMethod::Type Method);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetGlobalIlluminationMethod(EDynamicGlobalIlluminationMethod::Type Method);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetMotionBlurParams(float MotionBlurAmount, float MotionBlurMax,
                         float MotionBlurPerObjectSize, int MotionBlurTargetFPS);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetFocalParams(float FocalDistance, float FocalRegion);

Lens Artifacts

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetChromaticAberration(float Intensity);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetVignetteIntensity(float Intensity);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetFilmGrain(float Intensity, float TexelSize = 1.0f);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetConvolutionBloom(EBloomMethod Method, UTexture2D* KernelTexture,
                         float Intensity = 1.0f);

Async Capture Control

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetAsyncCaptureNextFrame(bool bEnabled);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
void SetUseFastCapture(bool bInUseAsync);

UFUNCTION(BlueprintCallable, Category = "unrealcv")
bool GetUseFastCapture() const;

Parameter Getters

UFUNCTION(BlueprintPure, Category = "unrealcv")
EReflectionMethod::Type GetReflectionMethod() const;

UFUNCTION(BlueprintPure, Category = "unrealcv")
EDynamicGlobalIlluminationMethod::Type GetGlobalIlluminationMethod() const;

UFUNCTION(BlueprintPure, Category = "unrealcv")
EAutoExposureMethod GetExposureMethod() const;

UFUNCTION(BlueprintPure, Category = "unrealcv")
void GetAutoExposureSpeed(float& OutExposureSpeedDown, float& OutExposureSpeedUp) const;

UFUNCTION(BlueprintPure, Category = "unrealcv")
void GetMotionBlurParams(float& OutMotionBlurAmount, float& OutMotionBlurMax,
                         float& OutMotionBlurPerObjectSize, int& OutMotionBlurTargetFPS) const;

UFUNCTION(BlueprintPure, Category = "unrealcv")
void GetFocalParams(float& OutFocalDistance, float& OutFocalRegion) const;

UFUNCTION(BlueprintPure, Category = "unrealcv")
float GetChromaticAberration() const;

UFUNCTION(BlueprintPure, Category = "unrealcv")
float GetVignetteIntensity() const;

UFUNCTION(BlueprintPure, Category = "unrealcv")
void GetBloomParams(EBloomMethod& OutBloomMethod, float& OutBloomIntensity) const;

UBaseCameraSensor

Base class for all camera sensors, providing common capture functionality.

Header: Source/UnrealCV/Public/Sensor/CameraSensor/BaseCameraSensor.h

Sensor Transform

FVector GetSensorLocation()
{
   return this->GetComponentLocation();
}

void SetSensorLocation(FVector Location)
{
   this->SetWorldLocation(Location);
}

FRotator GetSensorRotation()
{
   return this->GetComponentRotation();
}

void SetSensorRotation(FRotator Rotator)
{
   this->SetWorldRotation(Rotator);
}

FOV / Film Size

float GetFOV() { return this->FOVAngle; }
void SetFOV(float FOV) { this->FOVAngle = FOV; }

void SetFilmSize(int Width, int Height);
int GetFilmWidth();
int GetFilmHeight();

Texture Target Management

virtual void InitTextureTarget(int FilmWidth, int FilmHeight);

void SetPostProcessMaterial(TScriptInterface<IBlendableInterface> PostProcessMaterial);

bool CheckTextureTarget();

Visibility Control

void SetShowOnlyList(const TArray<TWeakObjectPtr<UPrimitiveComponent>>& InShowOnlyComponents);

void HideActor(AActor* Actor);
void ShowActor(AActor* Actor);

Sync Capture

void Capture(TArray<FColor>& ImageData, int& Width, int& Height);
void Capture(TArray<FFloat16Color>& ImageData, int& Width, int& Height);

void ReadCaptureResults(TArray<FColor>& Data);

Async Capture

void SetUseFastCapture(bool bInUseFast) { bUseFastCapture = bInUseFast; }
bool GetUseFastCapture() const { return bUseFastCapture; }
bool bAsyncCaptureNextFrame;

virtual void LaunchCapture();

void CaptureFastToFile(const FString& Filename);

void CaptureFast(TArray<FColor>& ImageData, int& Width, int& Height);
void CaptureFast(TArray<FFloat16Color>& ImageData, int& Width, int& Height);

void ConfigureMaxQualityLumen();

Capture Format

enum class ECaptureFormat : uint8
{
   Invalid = 0,
   F16 = 1,      // Float16 (HDR)
   UInt8 = 2,    // 8-bit color
};

void CheckCaptureCache(ECaptureFormat Format);
virtual void CopyBackCapture(ECaptureFormat Format);
void CleanCaptureCache();

Specialized Sensors

Sensor Class	Data Type	Output Format
ULitCamSensor	RGB	FColor
UDepthCamSensor	Depth	float
UNormalCamSensor	Normals	FColor
UAnnotationCamSensor	Segmentation	FColor
UFlowCamSensor	Optical Flow	FColor
UShadowCatcherCamSensor	Shadow Composite	FColor
UStencilMaskCamSensor	Stencil Mask	FColor

Capture Pipeline

Sync Path (Blocking):

Game Thread
├── Validate capture request
├── Read pixels from GPU
├── Block until readback complete
└── Return pixel data

Async Path (Non-Blocking):

Frame 1                 Frame 2                 Frame 3
─────────               ─────────               ─────────
Game Thread
├── Enqueue render cmd
└── Return immediately

                     Render Thread
                     ├── GPU render
                     └── Readback start

                                             Game Thread
                                             ├── Readback ready
                                             ├── PNG encode
                                             └── Async file write

Visibility Modes

The sensor system supports multiple visibility modes for layer isolation:

Show Only List:

• Only specified components visible

• Used for one-object capture modes

Hide Actor:

• Hide specific actors from capture

• Used for background-only capture

Show Actor:

• Re-enable previously hidden actors

Integration with Recording

The sensor system integrates with AFusionCamCaptureActor for high-throughput recording. See UnrealCV+ Plugin for the UnrealCV+ recording feature summary.

See Also

• Annotation System Architecture - Annotation system (used by AnnotationCamSensor)

• UnrealCV+ Plugin - UnrealCV+ recording and camera feature summary

• Sensor Data Formats Reference - Sensor output formats