Python API Reference

This document provides detailed API reference for UnrealCV Python client.

Basic Usage

class unrealcv.Client(endpoint, type='inet')

Bases: object

Client can be used to send request to a game and get response Currently only one client is allowed at a time More clients will be rejected

connect(timeout=1, start_receive_thread=True): Try to connect to server, return whether connection successful

disconnect(): Disconnect from server

isconnected(): Check whether client is connected to server

receive(): Receive packages, Extract message from packages Call self.message_handler if got a message Also check whether client is still connected

request(message, timeout=5)

Send a request to server and wait util get a response from server or timeout.

Parameters:

message (str or list) – UnrealCV command to interact with the game. When message is a list of commands, the commands will be sent in batch. More info can be seen from http://docs.unrealcv.org/en/latest/reference/commands.html
timeout (int) – when timeout is larger than 0, the request will be sent synchronously, and the response will be returned when timeout is -1, the request will be sent asynchronously, and no response will be returned

Returns:

plain text message from server

Return type:

str

Examples

>>> client = Client('localhost', 9000)
>>> client.connect()
>>> client.request('vget /camera/0/location')
'100.0 -100.0 100.0'
>>> # timeout -1 means async request, no response will be returned
>>> client.request('vset /camera/0/location 100 100 100', -1)

request_async(message): Send request without waiting for any reply

request_batch(batch)

Send a batch of requests to server and wait util get all responses from server. :param batch: a list of requests, each request is a string, such as [‘command1’, ‘command2’, …] :type batch: list

Returns:: a list of responses, such as [‘response1’, ‘response2’, …]
Return type:: list

Examples

>>> client.request_batch(['vget /camera/0/location', 'vget /camera/0/rotation'])
['100.0 -100.0 100.0', '0.0 0.0 0.0']

request_batch_async(batch)

Send a batch of requests to server without waiting for any reply.

batchlist: a list of requests, each request is a string, such as [‘command1’, ‘command2’, …]

Return type:: None

send(message): Send message out, return whether the message was successfully sent

class unrealcv.SocketMessage(payload)

Bases: object

Define the format of a message. This class is defined similar to the class FNFSMessageHeader in UnrealEngine4, but without CRC check. The magic number is from Unreal implementation See https://github.com/EpicGames/UnrealEngine/blob/dff3c48be101bb9f84633a733ef79c91c38d9542/Engine/Source/Runtime/Sockets/Public/NetworkMessage.h

classmethod ReceivePayload(sock): Return only payload, not the raw message, None if failed. sock: a blocking socket for read data.

classmethod WrapAndSendPayload(sock, payload): Send payload, true if success, false if failed

unrealcv.build_keyframe_groom_wind_json(direction_keys, strength_keys, indent=2): Build a deterministic JSON payload for keyframed Groom wind.

UnrealCV Client

The client module exposes the main Client interface used to connect to UnrealCV.

UnrealCV API

The UnrealCV API provides high-level functions to interact with Unreal Engine.

The reference below is generated from the local client/python package used by this branch, so newly added public methods are included automatically. Most command-building methods accept return_cmd=True; this returns the UnrealCV command without sending it and is useful for inspection, batching, and tests.

Current 5.2 additions include positioned object spawning and bone queries through unrealcv.api.UnrealCv_API.set_new_obj(), unrealcv.api.UnrealCv_API.spawn_object_from_path(), and unrealcv.api.UnrealCv_API.get_obj_bones().

The recording, annotation, pak, panoramic, scene occupancy, and CID helpers are also documented below. They target the extended UnrealCV+ server surface and require a server build that registers their corresponding commands; the base command inventory is listed in Command System.

UnrealCV API module for interacting with Unreal Engine.

This module provides a high-level interface for communicating with Unreal Engine through UnrealCV. It includes functionality for: - Camera control and image capture - Object manipulation and querying - Environment control - Scene configuration

Example

>>> from unrealcv import api
>>> client = api.UnrealCv_API(port=9000, ip='127.0.0.1', resolution=(640, 480))
>>> # Get an RGB image from camera 0
>>> image = client.get_image(0, 'lit')
>>> # Get object names and locations
>>> objects = client.get_objects()
>>> for obj in objects:
>>>     location = client.get_obj_location(obj)

class unrealcv.api.MsgDecoder

Bases: object

Message decoder for UnrealCV server responses.

This class provides methods to decode various types of responses from the UnrealCV server, including images, vectors, colors, and other data formats.

decode_map

Mapping of data types to their decoder functions: - vertex_location: Decodes vertex coordinates - color: Decodes RGB color values - rotation: Decodes rotation angles - location: Decodes position coordinates - bounds: Decodes bounding box coordinates - scale: Decodes scale factors - png: Decodes PNG images - bmp: Decodes BMP images - npy: Decodes NumPy array data

Type:: dict

Example

>>> decoder = MsgDecoder()
>>> # Decode a color string
>>> color = decoder.string2color("(R=255,G=128,B=64)")
>>> print(color)  # [255, 128, 64]
>>>
>>> # Decode position coordinates
>>> pos = decoder.string2floats("100.0 200.0 300.0")
>>> print(pos)  # [100.0, 200.0, 300.0]

bpstring2floats(res)

Decode a string of numbers into a list of floats for blueprint.

Parameters:: res (str) – The response string.
Returns:: The list of floats or a single float.
Return type:: list or float

bpvector2floats(res)

Decode a vector string for blueprint.

Parameters:: res (str) – The response string.
Returns:: The list of vectors.
Return type:: list

cmd2key(cmd)

Extract the last word of the command as key.

Parameters:: cmd (str) – The command string.
Returns:: The key extracted from the command.
Return type:: str

decode(cmd, res)

Universal decode function that selects appropriate decoder based on command.

Parameters:

cmd (str) – Command string.
res (str) – Response data.

Returns:

Decoded data in appropriate format.

Return type:

Any

Example

>>> res = decoder.decode("vget /object/cube/color", "(R=255,G=128,B=64)")
>>> print(res)  # [255, 128, 64]

decode_bmp(res)

Decode BMP image data.

Parameters:: res (bytes) – Raw BMP image data.
Returns:: RGB image array of shape (H, W, 3) in uint8 format.
Return type:: np.ndarray
Raises:: ValueError – If image decoding fails.

decode_depth(res, inverse=False)

Decode a depth image.

Parameters:

res (str) – The response string.
inverse (bool) – Whether to inverse the depth. Default is False.

Returns:

The decoded depth image.

Return type:

np.ndarray

decode_img(res, mode, inverse=False)

Decode an image.

Parameters:

res (str) – The response string.
mode (str) – The image format (e.g., ‘png’, ‘bmp’, ‘npy’).
inverse (bool) – Whether to inverse the depth. Default is False.

Returns:

The decoded image.

Return type:

np.ndarray

Note: The depth image should use the ‘npy’ mode to decode.

decode_npy(res)

Decode a NPY image.

Parameters:: res (str) – The response string.
Returns:: The decoded image.
Return type:: np.ndarray

decode_png(res)

Decode a PNG image.

Parameters:: res (str) – The response string.
Returns:: The decoded image.
Return type:: np.ndarray

decode_vertex(res)

Decode vertex locations.

Parameters:: res (str) – Multi-line string of vertex coordinates.
Returns:: List of vertex coordinates [x, y, z].
Return type:: list[list[float]]

Example

>>> vertices = decoder.decode_vertex("0.0 0.0 0.0\n1.0 1.0 1.0")
>>> print(vertices)  # [[0.0, 0.0, 0.0], [1.0, 1.0, 1.0]]

empty(res)

Return the response as is.

Parameters:: res (str) – The response string.
Returns:: The response string.
Return type:: str

string2color(res)

Decode color string to RGB values.

Parameters:: res (str) – Color string in format “(R=255,G=128,B=64)”.
Returns:: RGB color values as [r, g, b].
Return type:: list[int]

Example

>>> color = decoder.string2color("(R=255,G=128,B=64)")
>>> print(color)  # [255, 128, 64]

string2floats(res)

Convert space-separated string of numbers to float list.

Parameters:: res (str) – Space-separated numbers string.
Returns:: List of floating point numbers.
Return type:: list[float]

Example

>>> floats = decoder.string2floats("1.0 2.5 3.7")
>>> print(floats)  # [1.0, 2.5, 3.7]

string2list(res)

Convert a string to a list.

Parameters:: res (str) – The response string.
Returns:: The list of strings.
Return type:: list

string2vector(res)

Decode a vector string.

Parameters:: res (str) – The response string.
Returns:: The vector as a list of floats.
Return type:: list

class unrealcv.api.UnrealCv_API(port, ip, resolution, mode='tcp')

Bases: object

A class to interact with UnrealCV, a toolkit for using Unreal Engine (UE) in Python.

annotate_object(actor_name, return_cmd=False): Annotate a single actor by name.

annotate_world(return_cmd=False): Annotate the current world.

batch_cmd(cmds, decoders, **kwargs)

Execute a batch of commands.

Parameters:

cmds (list) – The list of commands.
decoders (list) – The list of decoder functions.
**kwargs – Additional arguments for the decoder functions.

Returns:

The list of results.

Return type:

list

Examples

>>> cmds = ['vget /camera/0/rotation', 'vget /camera/0/location']
>>> decoders = [self.decoder.string2floats, self.decoder.string2floats]
>>> results = self.batch_cmd(cmds, decoders)
>>> print(results)  # [[0, 0, 90], [100.0, 200.0, 300.0]]

build_color_dict(objects, batch=True)

Build a color dictionary for objects.

Parameters:

objects (list) – The list of object names.
batch (bool) – Whether to use batch commands. Default is True.

Returns:

The color dictionary.

Return type:

dict

Example

>>> objects = ['tree', 'house']
>>> color_dict = api.build_color_dict(objects)
>>> color_dict['tree'] = [255, 0, 0]
>>> color_dict['house'] = [0, 255, 0]

build_pose_dic(objects)

Build a pose dictionary for objects.

Parameters:: objects (list) – The list of object names.
Returns:: The pose dictionary.
Return type:: dict

camera_info()

Get the camera information.

Returns:: The camera information.
Return type:: dict

capture_panoramic(cam_id, path, width=None, height=None, return_cmd=False, timeout=5): Capture a panoramic equirectangular image to file.

capture_panoramic_depth(cam_id, path, width=None, height=None, return_cmd=False, timeout=5): Capture a panoramic depth preview image to a server-side file.

capture_panoramic_mask(cam_id, path, width=None, height=None, return_cmd=False, timeout=5): Capture a panoramic object-mask image to a server-side file.

capture_panoramic_normal(cam_id, path, width=None, height=None, return_cmd=False, timeout=5): Capture a panoramic world-normal image to a server-side file.

check_connection(): Check the connection to the UnrealCV server and reconnect if necessary.

clear_annotation_cache(return_cmd=False): Clear annotation component cache.

clear_world_annotation(return_cmd=False): Remove world annotation.

config_ue(resolution=(320, 240), low_quality=False, disable_all_screen_messages=True)

Configure Unreal Engine settings.

Parameters:

resolution (tuple) – The resolution of the display window.
low_quality (bool) – Whether to set the rendering quality to low. Default is False.
disable_all_screen_messages (bool) – Whether to disable all screen messages. Default is True.

connect(ip, port, mode='tcp')

Connect to the UnrealCV server.

Parameters:

ip (str) – The IP address to connect to.
port (int) – The port to connect to.
mode (str) – The connection mode, either ‘tcp’ or ‘unix’. Default is ‘tcp’.

Returns:

The connected client.

Return type:

unrealcv.Client

destroy_obj(obj)

Destroy an object, removing it from the scene.

Parameters:: obj (str) – The object name.

get_bbox(object_mask, obj, normalize=True)

Get the bounding box of an object.

Parameters:

object_mask (np.ndarray) – The object mask image.
obj (str) – The object name.
normalize (bool) – Whether to normalize the bounding box coordinates. Default is True.

Returns:

The mask and bounding box of the object.

Return type:

tuple

Example

>>> object_mask = api.get_image(0, 'mask')
>>> mask, box = api.get_bbox(object_mask, 'tree')

get_cam_fov(cam_id)

Get the camera field of view (fov).

Parameters:: cam_id (int) – The camera ID.
Returns:: The field of view.
Return type:: float

get_cam_location(cam_id, newest=True, return_cmd=False, syns=True)

Get the camera location.

Parameters:

cam_id (int) – The camera ID.
newest (bool) – Whether to get the newest location from UnrealCV. Default is True.
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.
syns (bool) – Whether to synchronize the location with the internal state. Default is True.

Returns:

The camera location [x, y, z].

Return type:

list

get_cam_pose(cam_id, mode='hard')

Get the camera pose. The mode can be ‘hard’ or ‘soft’. The ‘hard’ mode will get the pose from UnrealCV, while the ‘soft’ mode will get the pose from self.cam.

Parameters:

cam_id (int) – The camera ID.
mode (str) – The mode to get the pose, either ‘hard’ or ‘soft’. Default is ‘hard’.

Returns:

The camera pose [x, y, z, roll, yaw, pitch].

Return type:

list

get_cam_rotation(cam_id, newest=True, return_cmd=False, syns=True)

Get the camera rotation.

Parameters:

cam_id (int) – The camera ID.
newest (bool) – Whether to get the newest rotation from UnrealCV. Default is True.
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.
syns (bool) – Whether to synchronize the rotation with the internal state. Default is True.

Returns:

The camera rotation [pitch, yaw, roll].

Return type:

list

get_camera_config()

Get the camera configuration(the location, rotation, and fov of each camera).

Returns:: The camera configuration.
Return type:: dict

get_camera_fast_capture(cam_id, return_cmd=False): Get the fast capture mode status of a camera.

get_camera_id_map(): Pair legacy camera names with stable CID identifiers.

get_camera_list()

Get the list of cameras.

Returns:: The list of camera names.
Return type:: list

get_camera_list_cid(return_cmd=False): Get stable UnrealCV+ camera identifiers (CID format).

get_camera_list_legacy(return_cmd=False): Get legacy camera names returned by vget /cameras_legacy.

get_camera_num()

Get the number of cameras.

Returns:: The number of cameras.
Return type:: int

get_depth(cam_id, inverse=False, return_cmd=False, show=False)

Get the depth image from a camera.

Parameters:

cam_id (int) – The camera ID.
inverse (bool) – Whether to inverse the depth. Default is False.
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.
show (bool) – Whether to display the image. Default is False.

Returns:

The depth image.

Return type:

np.ndarray

get_distance(pos_now, pos_exp, n=2)

Get the distance between two points.

Parameters:

pos_now (list) – The current position.
pos_exp (list) – The expected position.
n (int) – The dimension. Default is 2.

Returns:

The distance between the two points.

Return type:

float

get_image(cam_id, viewmode, mode='bmp', return_cmd=False, show=False, inverse=False)

Get an image from a camera.

Parameters:

cam_id (int) – The camera ID.
viewmode (str) – The view mode (e.g., ‘lit’, ‘normal’, ‘object_mask’, ‘depth’).
mode (str) – The image format (e.g., ‘bmp’, ‘png’, ‘npy’). Default is ‘bmp’.
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.
show (bool) – Whether to display the image. Default is False.
inverse (bool) – Whether to inverse the depth. Default is False.

Returns:

The image.

Return type:

np.ndarray

get_image_multicam(cam_ids, viewmode='lit', mode='bmp', inverse=True)

Get images from multiple cameras with the same view mode.

Parameters:

cam_ids (list) – The list of camera IDs.
viewmode (str) – The view mode (e.g., ‘lit’, ‘depth’, ‘normal’, ‘object_mask’). Default is ‘lit’.
mode (str) – The image format (e.g., ‘bmp’, ‘npy’, ‘png’). Default is ‘bmp’.
inverse (bool) – Whether to inverse the depth. Default is True.

Returns:

The list of images.

Return type:

list

get_image_multimodal(cam_id, viewmodes=None, modes=None)

Get multimodal images from a camera, such as RGB-D images (default).

Parameters:

cam_id (int) – The camera ID.
viewmodes (list) – The list of view modes. Default is [‘lit’, ‘depth’].
modes (list) – The list of image formats. Default is [‘bmp’, ‘npy’].

Returns:

The concatenated image.

Return type:

np.ndarray

get_img_batch(cam_info)

Get multiple images from multiple cameras or viewmodes, the most flexible function to get images.

Parameters:: cam_info (dict) – The camera information, such as {cam_id: {viewmode: {‘mode’: ‘bmp’, ‘inverse’: True, ‘img’: None}}}
Returns:: The updated camera information with images.
Return type:: dict

get_is_paused()

Check if the game is paused.

Returns:: True if the game is paused, False otherwise.
Return type:: bool

get_mask(object_mask, obj, threshold=3)

Get the mask of a specific object in the object mask image.

Parameters:

object_mask (np.ndarray) – The object mask image.
obj (str) – The object name.
threshold (int) – The color threshold. Default is 3.

Returns:

The mask of the object.

Return type:

np.ndarray

get_mounted_paks(return_cmd=False): Get all currently mounted pak files.

get_obj_bboxes(object_mask, objects, return_dict=False)

Get the bounding boxes of multiple objects.

Parameters:

object_mask (np.ndarray) – The object mask image.
objects (list) – The list of object names.
return_dict (bool) – Whether to return the bounding boxes as a dictionary. Default is False.

Returns:

The list or dictionary of bounding boxes.

Return type:

list or dict

get_obj_bones(obj, bone_names=None, space='component', return_cmd=False)

Get skeletal bone transforms as JSON-compatible dictionaries.

Parameters:

obj (str) – Object name returned by vget /objects.
bone_names (str | list | tuple | None) – Optional comma-separated names or a name collection.
space (str) – component or world.
return_cmd (bool) – Return the command without executing it.

Returns:

Bone names and transforms.

Return type:

list[dict]

get_obj_bounds(obj, return_cmd=False)

Get the bounds of an object.

Parameters:

obj (str) – The object name.
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns:

The bounds of the object [min_x, min_y, min_z, max_x, max_y, max_z].

Return type:

list

get_obj_color(obj, return_cmd=False)

Get the color of an object in the object mask.

Parameters:

obj (str) – The object name.
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns:

The color of the object [r, g, b].

Return type:

list

get_obj_location(obj, return_cmd=False)

Get the location of an object.

Parameters:

obj (str) – The object name.
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns:

The location of the object [x, y, z].

Return type:

list

get_obj_pose(obj)

Get the pose of an object.

Parameters:: obj (str) – The object name.
Returns:: The pose of the object [x, y, z, roll, yaw, pitch].
Return type:: list

get_obj_rotation(obj, return_cmd=False)

Get the rotation of an object.

Parameters:

obj (str) – The object name.
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns:

The rotation of the object [roll, yaw, pitch].

Return type:

list

get_obj_scale(obj, return_cmd=False)

Get the scale of an object.

Parameters:

obj (str) – The object name.
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns:

The scale of the object [scale_x, scale_y, scale_z].

Return type:

list

get_obj_size(obj, box=True)

Get the size of an object.

Parameters:

obj (str) – The object name.
box (bool) – Whether to return the bounding box size. Default is True.

Returns:

The [size_x, size_y, size_z] or volume (float) of the 3D bounding box.

Return type:

list or float

get_obj_uclass(obj, return_cmd=False)

Get the Unreal class name of an object.

Parameters:

obj (str) – The object name.
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns:

The Unreal class name of the object.

Return type:

str

get_objects()

Get all object names in the map.

Returns:: The list of object names.
Return type:: list

get_pak_assets(package_path, return_cmd=False): Get assets available under a package path.

get_pak_assets_in_pak(pak_file_path, return_cmd=False): List Unreal asset package paths discovered inside a pak file.

get_pak_files(pak_file_path, return_cmd=False): List raw file entries recorded in a pak file index.

get_scene_occupancy(profile='lingo_vis', origin_cm=None, yaw_degrees=0.0, include_dynamic=False, method='bounds'): Return a scene occupancy grid from an extended UnrealCV server.

get_scene_occupancy_spec(profile='lingo_vis', method='bounds'): Return the selected scene occupancy grid contract as decoded JSON.

get_vertex_locations(obj, return_cmd=False)

Get the vertex locations of an object.

Parameters:

obj (str) – The object name.
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns:

The vertex locations.

Return type:

list

init_map(): Initialize the map by getting camera configurations and building a color dictionary for objects.

is_pak_mounted(pak_file_path, return_cmd=False): Check whether a pak file is mounted.

is_recording(cam_id, return_cmd=False): Check if a camera is currently recording.

load_pak_asset(asset_path, return_cmd=False): Load an asset from a mounted pak package path.

message_handler(message)

Handle messages from the UnrealCV server.

Parameters:: message (str) – The message from the server.

mount_pak(pak_file_path, pak_order=0, return_cmd=False): Mount a pak file at runtime.

move_cam(cam_id, loc)

Move the camera to a location with physics simulation.

Parameters:

cam_id (int) – The camera ID.
loc (list) – The target location [x, y, z].

move_cam_forward(cam_id, yaw, distance, height=0, pitch=0)

Move the camera forward as a mobile robot at specific height.

Parameters:

cam_id (int) – The camera ID.
yaw (float) – The delta angle between the camera and the x-axis.
distance (float) – The absolute distance from the last location to the next location.
height (float) – The height of the camera. Default is 0.
pitch (float) – The pitch angle. Default is 0.

Returns:

Whether the movement was successful.

Return type:

bool

register_camera(cam_id, obj_name=None)

Register a camera in self.cam (dict).

Parameters:

cam_id (int) – The camera ID.
obj_name (str) – The object name. Default is None.

register_pak_assets(package_path, category, return_cmd=False): Register pak assets into the UnrealCV asset pool.

save_image(cam_id, viewmode, path, return_cmd=False)

Save an image from a camera.

Parameters:

cam_id (int) – The camera ID.
viewmode (str) – The view mode (e.g., ‘lit’, ‘depth’).
path (str) – The path to save the image.
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns:

The command if return_cmd is True, otherwise the image directories.

Return type:

str

save_scene_occupancy(path, profile='lingo_vis', origin_cm=None, yaw_degrees=0.0, include_dynamic=False, method='bounds'): Build and save a scene occupancy grid on an extended UnrealCV server.

scan_pak_assets(mount_point, force_rescan=True, return_cmd=False): Scan assets from a mounted pak mount point.

set_annotation_cache_enabled(enabled, return_cmd=False): Enable or disable annotation component cache.

set_cam_fov(cam_id, fov)

Set the camera field of view (fov).

Parameters:

cam_id (int) – The camera ID.
fov (float) – The field of view.

Example

>>> api.set_cam_fov(0, 90) # set the fov of the camera 0 to 90 degrees.

set_cam_location(cam_id, loc)

Set the camera location.

Parameters:

cam_id (int) – The camera ID.
loc (list) – The camera location [x, y, z].

set_cam_pose(cam_id, pose)

Set the camera pose.

Parameters:

cam_id (int) – The camera ID.
pose (list) – The camera pose [x, y, z, pitch, yaw, roll].

set_cam_rotation(cam_id, rot, rpy=False)

Set the camera rotation.

Parameters:

cam_id (int) – The camera ID.
rot (list) – The camera rotation [roll, yaw, pitch].
rpy (bool) – Whether the rotation is in roll-pitch-yaw format. Default is False.

set_camera_fast_capture(cam_id, enabled, return_cmd=False): Set the fast capture mode of a camera.

set_camera_panoramic_resolution(cam_id, cubemap_resolution, return_cmd=False): Set the cubemap resolution used for panoramic capture.

set_global_time_dilation(time_dilation, return_cmd=False)

Set the global time dilation, which affects the simulation speed of the game.

Parameters:

time_dilation (float) – The time dilation factor.
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

set_hide_obj(obj, return_cmd=False)

Hide an object, making it invisible but still present in the physics engine.

Parameters:

obj (str) – The object name.
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

set_hide_objects(objects)

Hide multiple objects, making them invisible but still present in the physics engine.

Parameters:: objects (list) – The list of object names.

set_keyboard(key, duration=0.01)

Simulate a keyboard action.

Parameters:

key (str) – The key to press, such as ‘Up’, ‘Down’, ‘Left’, ‘Right’
duration (float) – The duration to press the key. Default is 0.01.

Example

>>> api.set_keyboard('Up', 0.1) # press the up key for 0.1 seconds
>>> api.set_keyboard('Left', 0.1) # press the left key for 0.1 seconds

set_map(map_name, return_cmd=False)

Change to a new level map.

Parameters:

map_name (str) – The name of the map.
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

set_max_FPS(max_fps, return_cmd=False)

Set the maximum frames per second (FPS) for the Unreal Engine.

Parameters:

max_fps (int) – The maximum FPS to set.
return_cmd (bool) – Whether to return the command string instead of executing it. Default is False.

Returns:

The command string if return_cmd is True, otherwise None.

Return type:

str

set_new_camera()

Spawn a new camera.

Returns:: The object name of the new camera.
Return type:: str

set_new_obj(class_name, obj_name, location=None, return_cmd=False)

Spawn a new object.

Parameters:

class_name (str) – The class name of the object.
obj_name (str) – The object name.
location (list | tuple | None) – Optional world location [x, y, z].
return_cmd (bool) – Return the command without executing it.

Returns:

The object name of the new object.

Return type:

str

Example

>>> api.set_new_obj('Cube_C', 'cube_1', location=[100, 200, 300])

set_obj_color(obj, color, return_cmd=False)

Set the color of an object to render in the object mask.

Parameters:

obj (str) – The object name.
color (list) – The color to set [r, g, b].
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Example

>>> api.set_obj_color('tree', [255, 0, 0]) # set the color of the tree to red

set_obj_location(obj, loc)

Set the location of an object in the scene with async mode.

Parameters:

obj (str) – The object name.
loc (list) – The location to set [x, y, z].

Example

>>> api.set_obj_location('tree', [100, 200, 300]) # set the location of the tree to [100, 200, 300]

set_obj_rotation(obj, rot)

Set the rotation of an object in the scene with async mode.

Parameters:

obj (str) – The object name.
rot (list) – The rotation to set [roll, yaw, pitch].

set_obj_scale(obj, scale=None, return_cmd=False)

Set the scale of an object.

Parameters:

obj (str) – The object name.
scale (list) – The scale to set [scale_x, scale_y, scale_z]. Default is [1, 1, 1].
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

set_pause(return_cmd=False)

Pause the game simulation.

Parameters:: return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Example

>>> api.set_pause() # every object will not move

set_resume(return_cmd=False)

Resume the game simulation.

Parameters:: return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Example

>>> api.set_resume() # every object will move again.

set_show_obj(obj, return_cmd=False)

Show an object, making it visible.

Parameters:

obj (str) – The object name.
return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

set_show_objects(objects)

Show multiple objects, making them visible.

Parameters:: objects (list) – The list of object names.

spawn_free_camera(return_cmd=False)

Spawn a new free camera at the world origin (0, 0, 0).

Parameters:: return_cmd (bool) – Whether to return the command string instead of executing it. Default is False.
Returns:: The camera ID of the newly spawned camera, or command string if return_cmd is True.
Return type:: int

spawn_object_from_path(asset_path, obj_name=None, annotate=True, return_cmd=False)

Spawn an object directly from an asset path.

Parameters:

asset_path (str) – Full asset path, for example /Game/Props/Chair.Chair.
obj_name (str | None) – Optional spawned actor name.
annotate (bool) – Whether to use the auto-annotation variant. Default is True.
return_cmd (bool) – Whether to return the command string instead of executing it.

Returns:

Spawned object name.

Return type:

str

start_simple_recording(cam_id, output_folder, fps, duration_seconds, record_options=None, return_cmd=False): Start simple recording without camera movement.

stop_recording(cam_id, return_cmd=False): Stop an active recording for a camera.

unmount_pak(pak_file_path, return_cmd=False): Unmount a pak file.

Environment Launcher

The launcher module provides utilities to start and manage Unreal Engine environments.

UnrealCV launcher module for starting and managing Unreal Engine environments.

This module provides utilities for: - Launching Unreal Engine binaries locally or in Docker containers - Managing environment configurations - Handling multi-GPU setups - Supporting headless rendering

The module supports both local and containerized environments across Windows, Linux, and macOS.

Example

>>> from unrealcv.launcher import RunUnreal
>>> from unrealcv.api import UnrealCv_API
>>>
>>> # Launch local environment
>>> launcher = RunUnreal('path/to/UnrealBinary')
>>> ip, port = launcher.start()
>>>
>>> # Connect API
>>> api = UnrealCv_API(port, ip, resolution=(640, 480))
>>>
>>> # Use environment...
>>>
>>> # Cleanup when finished
>>> api.client.disconnect()
>>> launcher.close()

class unrealcv.launcher.RunUnreal(ENV_BIN, ENV_MAP=None)

Bases: object

Launches and manages Unreal Engine environments with UnrealCV support.

This class handles: - Local and Docker-based environment launching - Multi-GPU support - Headless rendering - Resolution and port configuration - Cross-platform compatibility

Parameters:

ENV_BIN (str) – The path to the executable Unreal Engine binary.
ENV_MAP (str, optional) – The name of the Unreal Engine map. Default is None.

Example

>>> from unrealcv.launcher import RunUnreal
>>> from unrealcv.api import UnrealCv_API
>>>
>>> # Launch a local Unreal environment
>>> launcher = RunUnreal('path/to/UnrealBinary')
>>> ip, port = launcher.start()  # Launch with default settings
>>>
>>> # Launch headless on GPU 1 with custom resolution
>>> ip, port = launcher.start(offscreen=True,
...                          gpu_id=1,
...                          resolution=(640, 480))
>>>
>>> # Connect to environment with UnrealCV API
>>> unrealcv = UnrealCv_API(port, ip, resolution=(640, 480))
>>>
>>> # Use the UnrealCV API...
>>>
>>> # Cleanup
>>> unrealcv.client.disconnect()
>>> launcher.close()

close(): Close the Unreal Engine environment.

isPortFree(ip, port)

Check if the port is free.

Parameters:

ip (str) – The IP address.
port (int) – The port number.

Returns:

True if the port is free, False otherwise.

Return type:

bool

modify_permission(path)

Modify the permission of the environment path.

Parameters:: path (str) – The path to modify.

parse_path(path)

Parse the path to get the root path and the relative path to the binary.

Parameters:: path (str) – The path to parse.
Returns:: The root path and the binary path.
Return type:: tuple

read_port()

Read the port number from unrealcv.ini.

Returns:: The port number.
Return type:: int

set_ue_options(cmd_exe=[], opengl=False, offscreen=False, nullrhi=False, gpu_id=None)

Set options for running the Unreal Engine environment.

Parameters:

cmd_exe (list) – The command to execute.
opengl (bool) – Whether to use OpenGL rendering. Default is False. If False, Vulkan is used.
offscreen (bool) – Whether to render offscreen. Default is False. This is useful for headless servers.
nullrhi (bool) – Whether to use null RHI. Default is False. This turns off the graphics rendering to save resources when rendering not needed.
gpu_id (int, optional) – The GPU ID to use. Default is None. This is useful for multi-GPU systems.

Returns:

The command with options.

Return type:

list

signal_handler(signum, frame)

Handle signals to close the environment.

Parameters:

signum (int) – The signal number.
frame (frame) – The current stack frame.

start(docker=False, resolution=(640, 480), display=None, opengl=False, offscreen=False, nullrhi=False, gpu_id=None, local_host=True, sleep_time=8, log_file_path=None)

Start the Unreal Engine environment.

Parameters:

docker (bool) – Whether to use Docker. Default is False.
resolution (tuple) – The resolution of the image. Default is (640, 480).
display (str, optional) – Specify the displaynumber. Default is None.
opengl (bool) – Whether to use OpenGL rendering. Default is False (use Vulkan).
offscreen (bool) – Whether to render offscreen. Default is False.
nullrhi (bool) – Whether to use null RHI (turn off the graphics rendering). Default is False.
gpu_id (int, optional) – The GPU ID to use. Default is None.
local_host (bool) – Whether to use local host networking. Default is True.
sleep_time (int) – The time to wait for the environment to launch. Default is 8 seconds.

Returns:

The IP and port of the UnrealCV Server in the environment.

Return type:

tuple

write_port(port)

Write the port number to unrealcv.ini.

Parameters:: port (int) – The port number to write.

write_resolution(resolution)

Set the UnrealCV camera resolution by writing to unrealcv.ini.

Parameters:: resolution (tuple) – The resolution to set.

Automation Tools

Tools for building plugins and packaging model zoo binaries.

class unrealcv.automation.DockerBinary(binary_path): Bases: UE4BinaryBase

class unrealcv.automation.LinuxBinary(binary_path): Bases: UE4BinaryBase

class unrealcv.automation.MacBinary(binary_path): Bases: UE4BinaryBase

class unrealcv.automation.UE4Automation(engine)

Bases: object

UE4 engine wrapper

build_plugin(plugin_descriptor, output_folder, overwrite=False)

Use RunUAT script to build the plugin

Parameters:

plugin_descriptor (str) –
output_folder (str) –
overwrite (bool) – Whether the compiled binary folder should be overwriten?

install(plugin_folder, overwrite=False)

Install the plugin to UE4 engine folder

Parameters:: plugin_folder (str) – The plugin folder with compiled binaries

package(project_descriptor, output_folder, overwrite=False)

Package an UE4 project

Parameters:

project_descriptor (str) – UE4 project file name ending with .uproject
overwrite (bool) – Overwrite existing files

unrealcv.automation.UE4Binary(binary_path)

Return a platform-dependent binary for user.

Examples

>>> binary = UE4Binary('./WindowsNoEditor/RealisticRendering.exe')  # For windows
>>> binary = UE4Binary('./LinuxNoEditor/RealisticRendering/Binaries/RealisticRendering') # For Linux
>>> binary = UE4Binary('./MacNoEditor/RealisticRendering.app') # For mac
>>> with binary: # Automatically launch and close the binary
>>>     client.request('vget /unrealcv/status')
>>> # Or alternatively
>>> binary.start()
>>> client.request('vget /unrealcv/status')
>>> binary.close()

class unrealcv.automation.UE4BinaryBase(binary_path)

Bases: object

UE4BinaryBase is the base class for all platform-dependent classes, it is different from UE4Binary which serves as a factory to create a platform-dependent binary wrapper. User should use UE4Binary instead of UE4BinaryBase

Binary is a python wrapper to control the start and stop of a UE4 binary. The wrapper provides simple features to start and stop the binary, mainly useful for automate the testing.

Usage:

bin = UE4Binary('/tmp/RealisticRendering/RealisticRendering')
with bin:
    client.request('vget /camera/0/lit test.png')

class unrealcv.automation.WindowsBinary(binary_path): Bases: UE4BinaryBase

unrealcv.automation.get_platform_name(): ‘ Python and UE4 use different names for platform, in this script we will use UE4 platform name exclusively

Groom Wind Helpers

Helpers for building extended UnrealCV Groom wind payloads.

unrealcv.groom_wind.build_keyframe_groom_wind_json(direction_keys, strength_keys, indent=2): Build a deterministic JSON payload for keyframed Groom wind.