Python API Reference

This document provides detailed API reference for UnrealCV Python client.

Basic Usage

class unrealcv.Client(endpoint, type='inet')

Bases: object

Client can be used to send request to a game and get response Currently only one client is allowed at a time More clients will be rejected

connect(timeout=1)

Try to connect to server, return whether connection successful

disconnect()

Disconnect from server

isconnected()

Check whether client is connected to server

raw_message_handler(raw_message)

receive()

Receive packages, Extract message from packages Call self.message_handler if got a message Also check whether client is still connected

receive_loop_queue()

request(message, timeout=5)

Send a request to server and wait util get a response from server or timeout.

Parameters

• message (str or list) – UnrealCV command to interact with the game. When message is a list of commands, the commands will be sent in batch. More info can be seen from http://docs.unrealcv.org/en/latest/reference/commands.html

• timeout (int) – when timeout is larger than 0, the request will be sent synchronously, and the response will be returned when timeout is -1, the request will be sent asynchronously, and no response will be returned

Returns

plain text message from server

Return type

str

Examples

>>> client = Client('localhost', 9000)
>>> client.connect()
>>> client.request('vget /camera/0/location')
'100.0 -100.0 100.0'
>>> # timeout -1 means async request, no response will be returned
>>> client.request('vset /camera/0/location 100 100 100', -1)

request_async(message)

Send request without waiting for any reply

request_batch(batch)

Send a batch of requests to server and wait util get all responses from server. :param batch: a list of requests, each request is a string, such as [‘command1’, ‘command2’, …] :type batch: list

Returns

a list of responses, such as [‘response1’, ‘response2’, …]

Return type

list

Examples

>>> client.request_batch(['vget /camera/0/location', 'vget /camera/0/rotation'])
['100.0 -100.0 100.0', '0.0 0.0 0.0']

request_batch_async(batch)

Send a batch of requests to server without waiting for any reply.

batchlist

a list of requests, each request is a string, such as [‘command1’, ‘command2’, …]

Returns

Return type

None

send(message)

Send message out, return whether the message was successfully sent

class unrealcv.SocketMessage(payload)

Bases: object

Define the format of a message. This class is defined similar to the class FNFSMessageHeader in UnrealEngine4, but without CRC check. The magic number is from Unreal implementation See https://github.com/EpicGames/UnrealEngine/blob/dff3c48be101bb9f84633a733ef79c91c38d9542/Engine/Source/Runtime/Sockets/Public/NetworkMessage.h

classmethod ReceivePayload(sock)

Return only payload, not the raw message, None if failed. sock: a blocking socket for read data.

classmethod WrapAndSendPayload(sock, payload)

Send payload, true if success, false if failed

fmt = 'I'

magic = 2653651905

UnrealCV Client

class unrealcv.Client(endpoint, type='inet')

Bases: object

Client can be used to send request to a game and get response Currently only one client is allowed at a time More clients will be rejected

connect(timeout=1)

Try to connect to server, return whether connection successful

disconnect()

Disconnect from server

isconnected()

Check whether client is connected to server

raw_message_handler(raw_message)

receive()

Receive packages, Extract message from packages Call self.message_handler if got a message Also check whether client is still connected

receive_loop_queue()

request(message, timeout=5)

Send a request to server and wait util get a response from server or timeout.

Parameters

• message (str or list) – UnrealCV command to interact with the game. When message is a list of commands, the commands will be sent in batch. More info can be seen from http://docs.unrealcv.org/en/latest/reference/commands.html

• timeout (int) – when timeout is larger than 0, the request will be sent synchronously, and the response will be returned when timeout is -1, the request will be sent asynchronously, and no response will be returned

Returns

plain text message from server

Return type

str

Examples

>>> client = Client('localhost', 9000)
>>> client.connect()
>>> client.request('vget /camera/0/location')
'100.0 -100.0 100.0'
>>> # timeout -1 means async request, no response will be returned
>>> client.request('vset /camera/0/location 100 100 100', -1)

request_async(message)

Send request without waiting for any reply

request_batch(batch)

Send a batch of requests to server and wait util get all responses from server. :param batch: a list of requests, each request is a string, such as [‘command1’, ‘command2’, …] :type batch: list

Returns

a list of responses, such as [‘response1’, ‘response2’, …]

Return type

list

Examples

>>> client.request_batch(['vget /camera/0/location', 'vget /camera/0/rotation'])
['100.0 -100.0 100.0', '0.0 0.0 0.0']

request_batch_async(batch)

Send a batch of requests to server without waiting for any reply.

batchlist

a list of requests, each request is a string, such as [‘command1’, ‘command2’, …]

Returns

Return type

None

send(message)

Send message out, return whether the message was successfully sent

UnrealCV API

The UnrealCV API provides high-level functions to interact with Unreal Engine.

UnrealCV API module for interacting with Unreal Engine.

This module provides a high-level interface for communicating with Unreal Engine through UnrealCV. It includes functionality for: - Camera control and image capture - Object manipulation and querying - Environment control - Scene configuration

Example

>>> from unrealcv import api
>>> client = api.UnrealCv_API(port=9000, ip='127.0.0.1', resolution=(640, 480))
>>> # Get an RGB image from camera 0
>>> image = client.get_image(0, 'lit')
>>> # Get object names and locations
>>> objects = client.get_objects()
>>> for obj in objects:
>>>     location = client.get_obj_location(obj)

class unrealcv.api.MsgDecoder

Bases: object

Message decoder for UnrealCV server responses.

This class provides methods to decode various types of responses from the UnrealCV server, including images, vectors, colors, and other data formats.

decode_map

Mapping of data types to their decoder functions: - vertex_location: Decodes vertex coordinates - color: Decodes RGB color values - rotation: Decodes rotation angles - location: Decodes position coordinates - bounds: Decodes bounding box coordinates - scale: Decodes scale factors - png: Decodes PNG images - bmp: Decodes BMP images - npy: Decodes NumPy array data

Type

dict

Example

>>> decoder = MsgDecoder()
>>> # Decode a color string
>>> color = decoder.string2color("(R=255,G=128,B=64)")
>>> print(color)  # [255, 128, 64]
>>>
>>> # Decode position coordinates
>>> pos = decoder.string2floats("100.0 200.0 300.0")
>>> print(pos)  # [100.0, 200.0, 300.0]

bpstring2floats(res)

Decode a string of numbers into a list of floats for blueprint.

Parameters

res (str) – The response string.

Returns

The list of floats or a single float.

Return type

list or float

bpvector2floats(res)

Decode a vector string for blueprint.

Parameters

res (str) – The response string.

Returns

The list of vectors.

Return type

list

cmd2key(cmd)

Extract the last word of the command as key.

Parameters

cmd (str) – The command string.

Returns

The key extracted from the command.

Return type

str

decode(cmd, res)

Universal decode function that selects appropriate decoder based on command.

Parameters

• cmd (str) – Command string.

• res (str) – Response data.

Returns

Decoded data in appropriate format.

Return type

Any

Example

>>> res = decoder.decode("vget /object/cube/color", "(R=255,G=128,B=64)")
>>> print(res)  # [255, 128, 64]

decode_bmp(res)

Decode BMP image data.

Parameters

res (bytes) – Raw BMP image data.

Returns

RGB image array of shape (H, W, 3) in uint8 format.

Return type

np.ndarray

Raises

ValueError – If image decoding fails.

decode_depth(res, inverse=False)

Decode a depth image.

Parameters

• res (str) – The response string.

• inverse (bool) – Whether to inverse the depth. Default is False.

Returns

The decoded depth image.

Return type

np.ndarray

decode_img(res, mode, inverse=False)

Decode an image.

Parameters

• res (str) – The response string.

• mode (str) – The image format (e.g., ‘png’, ‘bmp’, ‘npy’).

• inverse (bool) – Whether to inverse the depth. Default is False.

Returns

The decoded image.

Return type

np.ndarray

Note: The depth image should use the ‘npy’ mode to decode.

decode_npy(res)

Decode a NPY image.

Parameters

res (str) – The response string.

Returns

The decoded image.

Return type

np.ndarray

decode_png(res)

Decode a PNG image.

Parameters

res (str) – The response string.

Returns

The decoded image.

Return type

np.ndarray

decode_vertex(res)

Decode vertex locations.

Parameters

res (str) – Multi-line string of vertex coordinates.

Returns

List of vertex coordinates [x, y, z].

Return type

list[list[float]]

Example

>>> vertices = decoder.decode_vertex("0.0 0.0 0.0\n1.0 1.0 1.0")
>>> print(vertices)  # [[0.0, 0.0, 0.0], [1.0, 1.0, 1.0]]

empty(res)

Return the response as is.

Parameters

res (str) – The response string.

Returns

The response string.

Return type

str

string2color(res)

Decode color string to RGB values.

Parameters

res (str) – Color string in format “(R=255,G=128,B=64)”.

Returns

RGB color values as [r, g, b].

Return type

list[int]

Example

>>> color = decoder.string2color("(R=255,G=128,B=64)")
>>> print(color)  # [255, 128, 64]

string2floats(res)

Convert space-separated string of numbers to float list.

Parameters

res (str) – Space-separated numbers string.

Returns

List of floating point numbers.

Return type

list[float]

Example

>>> floats = decoder.string2floats("1.0 2.5 3.7")
>>> print(floats)  # [1.0, 2.5, 3.7]

string2list(res)

Convert a string to a list.

Parameters

res (str) – The response string.

Returns

The list of strings.

Return type

list

string2vector(res)

Decode a vector string.

Parameters

res (str) – The response string.

Returns

The vector as a list of floats.

Return type

list

class unrealcv.api.UnrealCv_API(port, ip, resolution, mode='tcp')

Bases: object

A class to interact with UnrealCV, a toolkit for using Unreal Engine (UE) in Python.

batch_cmd(cmds, decoders, **kwargs)

Execute a batch of commands.

Parameters

• cmds (list) – The list of commands.

• decoders (list) – The list of decoder functions.

• **kwargs – Additional arguments for the decoder functions.

Returns

The list of results.

Return type

list

Examples

>>> cmds = ['vget /camera/0/rotation', 'vget /camera/0/location']
>>> decoders = [self.decoder.string2floats, self.decoder.string2floats]
>>> results = self.batch_cmd(cmds, decoders)
>>> print(results)  # [[0, 0, 90], [100.0, 200.0, 300.0]]

build_color_dict(objects, batch=True)

Build a color dictionary for objects.

Parameters

• objects (list) – The list of object names.

• batch (bool) – Whether to use batch commands. Default is True.

Returns

The color dictionary.

Return type

dict

Example

>>> objects = ['tree', 'house']
>>> color_dict = api.build_color_dict(objects)
>>> color_dict['tree'] = [255, 0, 0]
>>> color_dict['house'] = [0, 255, 0]

build_pose_dic(objects)

Build a pose dictionary for objects.

Parameters

objects (list) – The list of object names.

Returns

The pose dictionary.

Return type

dict

camera_info()

Get the camera information.

Returns

The camera information.

Return type

dict

check_connection()

Check the connection to the UnrealCV server and reconnect if necessary.

config_ue(resolution=(320, 240), low_quality=False, disable_all_screen_messages=True)

Configure Unreal Engine settings.

Parameters

• resolution (tuple) – The resolution of the display window.

• low_quality (bool) – Whether to set the rendering quality to low. Default is False.

• disable_all_screen_messages (bool) – Whether to disable all screen messages. Default is True.

connect(ip, port, mode='tcp')

Connect to the UnrealCV server.

Parameters

• ip (str) – The IP address to connect to.

• port (int) – The port to connect to.

• mode (str) – The connection mode, either ‘tcp’ or ‘unix’. Default is ‘tcp’.

Returns

The connected client.

Return type

unrealcv.Client

destroy_obj(obj)

Destroy an object, removing it from the scene.

Parameters

obj (str) – The object name.

get_bbox(object_mask, obj, normalize=True)

Get the bounding box of an object.

Parameters

• object_mask (np.ndarray) – The object mask image.

• obj (str) – The object name.

• normalize (bool) – Whether to normalize the bounding box coordinates. Default is True.

Returns

The mask and bounding box of the object.

Return type

tuple

Example

>>> object_mask = api.get_image(0, 'mask')
>>> mask, box = api.get_bbox(object_mask, 'tree')

get_cam_fov(cam_id)

Get the camera field of view (fov).

Parameters

cam_id (int) – The camera ID.

Returns

The field of view.

Return type

float

get_cam_location(cam_id, newest=True, return_cmd=False, syns=True)

Get the camera location.

Parameters

• cam_id (int) – The camera ID.

• newest (bool) – Whether to get the newest location from UnrealCV. Default is True.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

• syns (bool) – Whether to synchronize the location with the internal state. Default is True.

Returns

The camera location [x, y, z].

Return type

list

get_cam_pose(cam_id, mode='hard')

Get the camera pose. The mode can be ‘hard’ or ‘soft’. The ‘hard’ mode will get the pose from UnrealCV, while the ‘soft’ mode will get the pose from self.cam.

Parameters

• cam_id (int) – The camera ID.

• mode (str) – The mode to get the pose, either ‘hard’ or ‘soft’. Default is ‘hard’.

Returns

The camera pose [x, y, z, roll, yaw, pitch].

Return type

list

get_cam_rotation(cam_id, newest=True, return_cmd=False, syns=True)

Get the camera rotation.

Parameters

• cam_id (int) – The camera ID.

• newest (bool) – Whether to get the newest rotation from UnrealCV. Default is True.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

• syns (bool) – Whether to synchronize the rotation with the internal state. Default is True.

Returns

The camera rotation [pitch, yaw, roll].

Return type

list

get_camera_config()

Get the camera configuration(the location, rotation, and fov of each camera).

Returns

The camera configuration.

Return type

dict

get_camera_list()

Get the list of cameras.

Returns

The list of camera names.

Return type

list

get_camera_num()

Get the number of cameras.

Returns

The number of cameras.

Return type

int

get_depth(cam_id, inverse=False, return_cmd=False, show=False)

Get the depth image from a camera.

Parameters

• cam_id (int) – The camera ID.

• inverse (bool) – Whether to inverse the depth. Default is False.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

• show (bool) – Whether to display the image. Default is False.

Returns

The depth image.

Return type

np.ndarray

get_distance(pos_now, pos_exp, n=2)

Get the distance between two points.

Parameters

• pos_now (list) – The current position.

• pos_exp (list) – The expected position.

• n (int) – The dimension. Default is 2.

Returns

The distance between the two points.

Return type

float

get_image(cam_id, viewmode, mode='bmp', return_cmd=False, show=False, inverse=False)

Get an image from a camera.

Parameters

• cam_id (int) – The camera ID.

• viewmode (str) – The view mode (e.g., ‘lit’, ‘normal’, ‘object_mask’, ‘depth’).

• mode (str) – The image format (e.g., ‘bmp’, ‘png’, ‘npy’). Default is ‘bmp’.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

• show (bool) – Whether to display the image. Default is False.

• inverse (bool) – Whether to inverse the depth. Default is False.

Returns

The image.

Return type

np.ndarray

get_image_multicam(cam_ids, viewmode='lit', mode='bmp', inverse=True)

Get images from multiple cameras with the same view mode.

Parameters

• cam_ids (list) – The list of camera IDs.

• viewmode (str) – The view mode (e.g., ‘lit’, ‘depth’, ‘normal’, ‘object_mask’). Default is ‘lit’.

• mode (str) – The image format (e.g., ‘bmp’, ‘npy’, ‘png’). Default is ‘bmp’.

• inverse (bool) – Whether to inverse the depth. Default is True.

Returns

The list of images.

Return type

list

get_image_multimodal(cam_id, viewmodes=['lit', 'depth'], modes=['bmp', 'npy'])

Get multimodal images from a camera, such as RGB-D images (default).

Parameters

• cam_id (int) – The camera ID.

• viewmodes (list) – The list of view modes. Default is [‘lit’, ‘depth’].

• modes (list) – The list of image formats. Default is [‘bmp’, ‘npy’].

Returns

The concatenated image.

Return type

np.ndarray

get_img_batch(cam_info)

Get multiple images from multiple cameras or viewmodes, the most flexible function to get images.

Parameters

cam_info (dict) – The camera information, such as {cam_id: {viewmode: {‘mode’: ‘bmp’, ‘inverse’: True, ‘img’: None}}}

Returns

The updated camera information with images.

Return type

dict

get_is_paused()

Check if the game is paused.

Returns

True if the game is paused, False otherwise.

Return type

bool

get_mask(object_mask, obj, threshold=3)

Get the mask of a specific object in the object mask image.

Parameters

• object_mask (np.ndarray) – The object mask image.

• obj (str) – The object name.

• threshold (int) – The color threshold. Default is 3.

Returns

The mask of the object.

Return type

np.ndarray

get_obj_bboxes(object_mask, objects, return_dict=False)

Get the bounding boxes of multiple objects.

Parameters

• object_mask (np.ndarray) – The object mask image.

• objects (list) – The list of object names.

• return_dict (bool) – Whether to return the bounding boxes as a dictionary. Default is False.

Returns

The list or dictionary of bounding boxes.

Return type

list or dict

get_obj_bounds(obj, return_cmd=False)

Get the bounds of an object.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns

The bounds of the object [min_x, min_y, min_z, max_x, max_y, max_z].

Return type

list

get_obj_color(obj, return_cmd=False)

Get the color of an object in the object mask.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns

The color of the object [r, g, b].

Return type

list

get_obj_location(obj, return_cmd=False)

Get the location of an object.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns

The location of the object [x, y, z].

Return type

list

get_obj_pose(obj)

Get the pose of an object.

Parameters

obj (str) – The object name.

Returns

The pose of the object [x, y, z, roll, yaw, pitch].

Return type

list

get_obj_rotation(obj, return_cmd=False)

Get the rotation of an object.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns

The rotation of the object [roll, yaw, pitch].

Return type

list

get_obj_scale(obj, return_cmd=False)

Get the scale of an object.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns

The scale of the object [scale_x, scale_y, scale_z].

Return type

list

get_obj_size(obj, box=True)

Get the size of an object.

Parameters

• obj (str) – The object name.

• box (bool) – Whether to return the bounding box size. Default is True.

Returns

The [size_x, size_y, size_z] or volume (float) of the 3D bounding box.

Return type

list or float

get_obj_uclass(obj, return_cmd=False)

Get the Unreal class name of an object.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns

The Unreal class name of the object.

Return type

str

get_objects()

Get all object names in the map.

Returns

The list of object names.

Return type

list

get_vertex_locations(obj, return_cmd=False)

Get the vertex locations of an object.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns

The vertex locations.

Return type

list

init_map()

Initialize the map by getting camera configurations and building a color dictionary for objects.

message_handler(message)

Handle messages from the UnrealCV server.

Parameters

message (str) – The message from the server.

move_cam(cam_id, loc)

Move the camera to a location with physics simulation.

Parameters

• cam_id (int) – The camera ID.

• loc (list) – The target location [x, y, z].

move_cam_forward(cam_id, yaw, distance, height=0, pitch=0)

Move the camera forward as a mobile robot at specific height.

Parameters

• cam_id (int) – The camera ID.

• yaw (float) – The delta angle between the camera and the x-axis.

• distance (float) – The absolute distance from the last location to the next location.

• height (float) – The height of the camera. Default is 0.

• pitch (float) – The pitch angle. Default is 0.

Returns

Whether the movement was successful.

Return type

bool

register_camera(cam_id, obj_name=None)

Register a camera in self.cam (dict).

Parameters

• cam_id (int) – The camera ID.

• obj_name (str) – The object name. Default is None.

save_image(cam_id, viewmode, path, return_cmd=False)

Save an image from a camera.

Parameters

• cam_id (int) – The camera ID.

• viewmode (str) – The view mode (e.g., ‘lit’, ‘depth’).

• path (str) – The path to save the image.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns

The command if return_cmd is True, otherwise the image directories.

Return type

str

set_cam_fov(cam_id, fov)

Set the camera field of view (fov).

Parameters

• cam_id (int) – The camera ID.

• fov (float) – The field of view.

Example

>>> api.set_cam_fov(0, 90) # set the fov of the camera 0 to 90 degrees.

set_cam_location(cam_id, loc)

Set the camera location.

Parameters

• cam_id (int) – The camera ID.

• loc (list) – The camera location [x, y, z].

set_cam_pose(cam_id, pose)

Set the camera pose.

Parameters

• cam_id (int) – The camera ID.

• pose (list) – The camera pose [x, y, z, pitch, yaw, roll].

set_cam_rotation(cam_id, rot, rpy=False)

Set the camera rotation.

Parameters

• cam_id (int) – The camera ID.

• rot (list) – The camera rotation [roll, yaw, pitch].

• rpy (bool) – Whether the rotation is in roll-pitch-yaw format. Default is False.

set_global_time_dilation(time_dilation, return_cmd=False)

Set the global time dilation, which affects the simulation speed of the game.

Parameters

• time_dilation (float) – The time dilation factor.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

set_hide_obj(obj, return_cmd=False)

Hide an object, making it invisible but still present in the physics engine.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

set_hide_objects(objects)

Hide multiple objects, making them invisible but still present in the physics engine.

Parameters

objects (list) – The list of object names.

set_keyboard(key, duration=0.01)

Simulate a keyboard action.

Parameters

• key (str) – The key to press, such as ‘Up’, ‘Down’, ‘Left’, ‘Right’

• duration (float) – The duration to press the key. Default is 0.01.

Example

>>> api.set_keyboard('Up', 0.1) # press the up key for 0.1 seconds
>>> api.set_keyboard('Left', 0.1) # press the left key for 0.1 seconds

set_map(map_name, return_cmd=False)

Change to a new level map.

Parameters

• map_name (str) – The name of the map.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

set_max_FPS(max_fps, return_cmd=False)

Set the maximum frames per second (FPS) for the Unreal Engine.

Parameters

• max_fps (int) – The maximum FPS to set.

• return_cmd (bool) – Whether to return the command string instead of executing it. Default is False.

Returns

The command string if return_cmd is True, otherwise None.

Return type

str

set_new_camera()

Spawn a new camera.

Returns

The object name of the new camera.

Return type

str

set_new_obj(class_name, obj_name)

Spawn a new object.

Parameters

• class_name (str) – The class name of the object.

• obj_name (str) – The object name.

Returns

The object name of the new object.

Return type

str

Example

>>> api.set_new_obj('Cube_C', 'cube_1')

set_obj_color(obj, color, return_cmd=False)

Set the color of an object to render in the object mask.

Parameters

• obj (str) – The object name.

• color (list) – The color to set [r, g, b].

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Example

>>> api.set_obj_color('tree', [255, 0, 0]) # set the color of the tree to red

set_obj_location(obj, loc)

Set the location of an object in the scene with async mode.

Parameters

• obj (str) – The object name.

• loc (list) – The location to set [x, y, z].

Example

>>> api.set_obj_location('tree', [100, 200, 300]) # set the location of the tree to [100, 200, 300]

set_obj_rotation(obj, rot)

Set the rotation of an object in the scene with async mode.

Parameters

• obj (str) – The object name.

• rot (list) – The rotation to set [roll, yaw, pitch].

set_obj_scale(obj, scale=[1, 1, 1], return_cmd=False)

Set the scale of an object.

Parameters

• obj (str) – The object name.

• scale (list) – The scale to set [scale_x, scale_y, scale_z].

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

set_pause(return_cmd=False)

Pause the game simulation.

Parameters

return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Example

>>> api.set_pause() # every object will not move

set_resume(return_cmd=False)

Resume the game simulation.

Parameters

return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Example

>>> api.set_resume() # every object will move again.

set_show_obj(obj, return_cmd=False)

Show an object, making it visible.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

set_show_objects(objects)

Show multiple objects, making them visible.

Parameters

objects (list) – The list of object names.

class unrealcv.api.UnrealCv_API(port, ip, resolution, mode='tcp')

Bases: object

A class to interact with UnrealCV, a toolkit for using Unreal Engine (UE) in Python.

batch_cmd(cmds, decoders, **kwargs)

Execute a batch of commands.

Parameters

• cmds (list) – The list of commands.

• decoders (list) – The list of decoder functions.

• **kwargs – Additional arguments for the decoder functions.

Returns

The list of results.

Return type

list

Examples

>>> cmds = ['vget /camera/0/rotation', 'vget /camera/0/location']
>>> decoders = [self.decoder.string2floats, self.decoder.string2floats]
>>> results = self.batch_cmd(cmds, decoders)
>>> print(results)  # [[0, 0, 90], [100.0, 200.0, 300.0]]

build_color_dict(objects, batch=True)

Build a color dictionary for objects.

Parameters

• objects (list) – The list of object names.

• batch (bool) – Whether to use batch commands. Default is True.

Returns

The color dictionary.

Return type

dict

Example

>>> objects = ['tree', 'house']
>>> color_dict = api.build_color_dict(objects)
>>> color_dict['tree'] = [255, 0, 0]
>>> color_dict['house'] = [0, 255, 0]

build_pose_dic(objects)

Build a pose dictionary for objects.

Parameters

objects (list) – The list of object names.

Returns

The pose dictionary.

Return type

dict

camera_info()

Get the camera information.

Returns

The camera information.

Return type

dict

check_connection()

Check the connection to the UnrealCV server and reconnect if necessary.

config_ue(resolution=(320, 240), low_quality=False, disable_all_screen_messages=True)

Configure Unreal Engine settings.

Parameters

• resolution (tuple) – The resolution of the display window.

• low_quality (bool) – Whether to set the rendering quality to low. Default is False.

• disable_all_screen_messages (bool) – Whether to disable all screen messages. Default is True.

connect(ip, port, mode='tcp')

Connect to the UnrealCV server.

Parameters

• ip (str) – The IP address to connect to.

• port (int) – The port to connect to.

• mode (str) – The connection mode, either ‘tcp’ or ‘unix’. Default is ‘tcp’.

Returns

The connected client.

Return type

unrealcv.Client

destroy_obj(obj)

Destroy an object, removing it from the scene.

Parameters

obj (str) – The object name.

get_bbox(object_mask, obj, normalize=True)

Get the bounding box of an object.

Parameters

• object_mask (np.ndarray) – The object mask image.

• obj (str) – The object name.

• normalize (bool) – Whether to normalize the bounding box coordinates. Default is True.

Returns

The mask and bounding box of the object.

Return type

tuple

Example

>>> object_mask = api.get_image(0, 'mask')
>>> mask, box = api.get_bbox(object_mask, 'tree')

get_cam_fov(cam_id)

Get the camera field of view (fov).

Parameters

cam_id (int) – The camera ID.

Returns

The field of view.

Return type

float

get_cam_location(cam_id, newest=True, return_cmd=False, syns=True)

Get the camera location.

Parameters

• cam_id (int) – The camera ID.

• newest (bool) – Whether to get the newest location from UnrealCV. Default is True.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

• syns (bool) – Whether to synchronize the location with the internal state. Default is True.

Returns

The camera location [x, y, z].

Return type

list

get_cam_pose(cam_id, mode='hard')

Get the camera pose. The mode can be ‘hard’ or ‘soft’. The ‘hard’ mode will get the pose from UnrealCV, while the ‘soft’ mode will get the pose from self.cam.

Parameters

• cam_id (int) – The camera ID.

• mode (str) – The mode to get the pose, either ‘hard’ or ‘soft’. Default is ‘hard’.

Returns

The camera pose [x, y, z, roll, yaw, pitch].

Return type

list

get_cam_rotation(cam_id, newest=True, return_cmd=False, syns=True)

Get the camera rotation.

Parameters

• cam_id (int) – The camera ID.

• newest (bool) – Whether to get the newest rotation from UnrealCV. Default is True.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

• syns (bool) – Whether to synchronize the rotation with the internal state. Default is True.

Returns

The camera rotation [pitch, yaw, roll].

Return type

list

get_camera_config()

Get the camera configuration(the location, rotation, and fov of each camera).

Returns

The camera configuration.

Return type

dict

get_camera_list()

Get the list of cameras.

Returns

The list of camera names.

Return type

list

get_camera_num()

Get the number of cameras.

Returns

The number of cameras.

Return type

int

get_depth(cam_id, inverse=False, return_cmd=False, show=False)

Get the depth image from a camera.

Parameters

• cam_id (int) – The camera ID.

• inverse (bool) – Whether to inverse the depth. Default is False.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

• show (bool) – Whether to display the image. Default is False.

Returns

The depth image.

Return type

np.ndarray

get_distance(pos_now, pos_exp, n=2)

Get the distance between two points.

Parameters

• pos_now (list) – The current position.

• pos_exp (list) – The expected position.

• n (int) – The dimension. Default is 2.

Returns

The distance between the two points.

Return type

float

get_image(cam_id, viewmode, mode='bmp', return_cmd=False, show=False, inverse=False)

Get an image from a camera.

Parameters

• cam_id (int) – The camera ID.

• viewmode (str) – The view mode (e.g., ‘lit’, ‘normal’, ‘object_mask’, ‘depth’).

• mode (str) – The image format (e.g., ‘bmp’, ‘png’, ‘npy’). Default is ‘bmp’.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

• show (bool) – Whether to display the image. Default is False.

• inverse (bool) – Whether to inverse the depth. Default is False.

Returns

The image.

Return type

np.ndarray

get_image_multicam(cam_ids, viewmode='lit', mode='bmp', inverse=True)

Get images from multiple cameras with the same view mode.

Parameters

• cam_ids (list) – The list of camera IDs.

• viewmode (str) – The view mode (e.g., ‘lit’, ‘depth’, ‘normal’, ‘object_mask’). Default is ‘lit’.

• mode (str) – The image format (e.g., ‘bmp’, ‘npy’, ‘png’). Default is ‘bmp’.

• inverse (bool) – Whether to inverse the depth. Default is True.

Returns

The list of images.

Return type

list

get_image_multimodal(cam_id, viewmodes=['lit', 'depth'], modes=['bmp', 'npy'])

Get multimodal images from a camera, such as RGB-D images (default).

Parameters

• cam_id (int) – The camera ID.

• viewmodes (list) – The list of view modes. Default is [‘lit’, ‘depth’].

• modes (list) – The list of image formats. Default is [‘bmp’, ‘npy’].

Returns

The concatenated image.

Return type

np.ndarray

get_img_batch(cam_info)

Get multiple images from multiple cameras or viewmodes, the most flexible function to get images.

Parameters

cam_info (dict) – The camera information, such as {cam_id: {viewmode: {‘mode’: ‘bmp’, ‘inverse’: True, ‘img’: None}}}

Returns

The updated camera information with images.

Return type

dict

get_is_paused()

Check if the game is paused.

Returns

True if the game is paused, False otherwise.

Return type

bool

get_mask(object_mask, obj, threshold=3)

Get the mask of a specific object in the object mask image.

Parameters

• object_mask (np.ndarray) – The object mask image.

• obj (str) – The object name.

• threshold (int) – The color threshold. Default is 3.

Returns

The mask of the object.

Return type

np.ndarray

get_obj_bboxes(object_mask, objects, return_dict=False)

Get the bounding boxes of multiple objects.

Parameters

• object_mask (np.ndarray) – The object mask image.

• objects (list) – The list of object names.

• return_dict (bool) – Whether to return the bounding boxes as a dictionary. Default is False.

Returns

The list or dictionary of bounding boxes.

Return type

list or dict

get_obj_bounds(obj, return_cmd=False)

Get the bounds of an object.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns

The bounds of the object [min_x, min_y, min_z, max_x, max_y, max_z].

Return type

list

get_obj_color(obj, return_cmd=False)

Get the color of an object in the object mask.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns

The color of the object [r, g, b].

Return type

list

get_obj_location(obj, return_cmd=False)

Get the location of an object.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns

The location of the object [x, y, z].

Return type

list

get_obj_pose(obj)

Get the pose of an object.

Parameters

obj (str) – The object name.

Returns

The pose of the object [x, y, z, roll, yaw, pitch].

Return type

list

get_obj_rotation(obj, return_cmd=False)

Get the rotation of an object.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns

The rotation of the object [roll, yaw, pitch].

Return type

list

get_obj_scale(obj, return_cmd=False)

Get the scale of an object.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns

The scale of the object [scale_x, scale_y, scale_z].

Return type

list

get_obj_size(obj, box=True)

Get the size of an object.

Parameters

• obj (str) – The object name.

• box (bool) – Whether to return the bounding box size. Default is True.

Returns

The [size_x, size_y, size_z] or volume (float) of the 3D bounding box.

Return type

list or float

get_obj_uclass(obj, return_cmd=False)

Get the Unreal class name of an object.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns

The Unreal class name of the object.

Return type

str

get_objects()

Get all object names in the map.

Returns

The list of object names.

Return type

list

get_vertex_locations(obj, return_cmd=False)

Get the vertex locations of an object.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns

The vertex locations.

Return type

list

init_map()

Initialize the map by getting camera configurations and building a color dictionary for objects.

message_handler(message)

Handle messages from the UnrealCV server.

Parameters

message (str) – The message from the server.

move_cam(cam_id, loc)

Move the camera to a location with physics simulation.

Parameters

• cam_id (int) – The camera ID.

• loc (list) – The target location [x, y, z].

move_cam_forward(cam_id, yaw, distance, height=0, pitch=0)

Move the camera forward as a mobile robot at specific height.

Parameters

• cam_id (int) – The camera ID.

• yaw (float) – The delta angle between the camera and the x-axis.

• distance (float) – The absolute distance from the last location to the next location.

• height (float) – The height of the camera. Default is 0.

• pitch (float) – The pitch angle. Default is 0.

Returns

Whether the movement was successful.

Return type

bool

register_camera(cam_id, obj_name=None)

Register a camera in self.cam (dict).

Parameters

• cam_id (int) – The camera ID.

• obj_name (str) – The object name. Default is None.

save_image(cam_id, viewmode, path, return_cmd=False)

Save an image from a camera.

Parameters

• cam_id (int) – The camera ID.

• viewmode (str) – The view mode (e.g., ‘lit’, ‘depth’).

• path (str) – The path to save the image.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Returns

The command if return_cmd is True, otherwise the image directories.

Return type

str

set_cam_fov(cam_id, fov)

Set the camera field of view (fov).

Parameters

• cam_id (int) – The camera ID.

• fov (float) – The field of view.

Example

>>> api.set_cam_fov(0, 90) # set the fov of the camera 0 to 90 degrees.

set_cam_location(cam_id, loc)

Set the camera location.

Parameters

• cam_id (int) – The camera ID.

• loc (list) – The camera location [x, y, z].

set_cam_pose(cam_id, pose)

Set the camera pose.

Parameters

• cam_id (int) – The camera ID.

• pose (list) – The camera pose [x, y, z, pitch, yaw, roll].

set_cam_rotation(cam_id, rot, rpy=False)

Set the camera rotation.

Parameters

• cam_id (int) – The camera ID.

• rot (list) – The camera rotation [roll, yaw, pitch].

• rpy (bool) – Whether the rotation is in roll-pitch-yaw format. Default is False.

set_global_time_dilation(time_dilation, return_cmd=False)

Set the global time dilation, which affects the simulation speed of the game.

Parameters

• time_dilation (float) – The time dilation factor.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

set_hide_obj(obj, return_cmd=False)

Hide an object, making it invisible but still present in the physics engine.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

set_hide_objects(objects)

Hide multiple objects, making them invisible but still present in the physics engine.

Parameters

objects (list) – The list of object names.

set_keyboard(key, duration=0.01)

Simulate a keyboard action.

Parameters

• key (str) – The key to press, such as ‘Up’, ‘Down’, ‘Left’, ‘Right’

• duration (float) – The duration to press the key. Default is 0.01.

Example

>>> api.set_keyboard('Up', 0.1) # press the up key for 0.1 seconds
>>> api.set_keyboard('Left', 0.1) # press the left key for 0.1 seconds

set_map(map_name, return_cmd=False)

Change to a new level map.

Parameters

• map_name (str) – The name of the map.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

set_max_FPS(max_fps, return_cmd=False)

Set the maximum frames per second (FPS) for the Unreal Engine.

Parameters

• max_fps (int) – The maximum FPS to set.

• return_cmd (bool) – Whether to return the command string instead of executing it. Default is False.

Returns

The command string if return_cmd is True, otherwise None.

Return type

str

set_new_camera()

Spawn a new camera.

Returns

The object name of the new camera.

Return type

str

set_new_obj(class_name, obj_name)

Spawn a new object.

Parameters

• class_name (str) – The class name of the object.

• obj_name (str) – The object name.

Returns

The object name of the new object.

Return type

str

Example

>>> api.set_new_obj('Cube_C', 'cube_1')

set_obj_color(obj, color, return_cmd=False)

Set the color of an object to render in the object mask.

Parameters

• obj (str) – The object name.

• color (list) – The color to set [r, g, b].

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Example

>>> api.set_obj_color('tree', [255, 0, 0]) # set the color of the tree to red

set_obj_location(obj, loc)

Set the location of an object in the scene with async mode.

Parameters

• obj (str) – The object name.

• loc (list) – The location to set [x, y, z].

Example

>>> api.set_obj_location('tree', [100, 200, 300]) # set the location of the tree to [100, 200, 300]

set_obj_rotation(obj, rot)

Set the rotation of an object in the scene with async mode.

Parameters

• obj (str) – The object name.

• rot (list) – The rotation to set [roll, yaw, pitch].

set_obj_scale(obj, scale=[1, 1, 1], return_cmd=False)

Set the scale of an object.

Parameters

• obj (str) – The object name.

• scale (list) – The scale to set [scale_x, scale_y, scale_z].

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

set_pause(return_cmd=False)

Pause the game simulation.

Parameters

return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Example

>>> api.set_pause() # every object will not move

set_resume(return_cmd=False)

Resume the game simulation.

Parameters

return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

Example

>>> api.set_resume() # every object will move again.

set_show_obj(obj, return_cmd=False)

Show an object, making it visible.

Parameters

• obj (str) – The object name.

• return_cmd (bool) – Whether to return the command instead of executing it. Default is False.

set_show_objects(objects)

Show multiple objects, making them visible.

Parameters

objects (list) – The list of object names.

class unrealcv.api.MsgDecoder

Bases: object

Message decoder for UnrealCV server responses.

This class provides methods to decode various types of responses from the UnrealCV server, including images, vectors, colors, and other data formats.

decode_map

Mapping of data types to their decoder functions: - vertex_location: Decodes vertex coordinates - color: Decodes RGB color values - rotation: Decodes rotation angles - location: Decodes position coordinates - bounds: Decodes bounding box coordinates - scale: Decodes scale factors - png: Decodes PNG images - bmp: Decodes BMP images - npy: Decodes NumPy array data

Type

dict

Example

>>> decoder = MsgDecoder()
>>> # Decode a color string
>>> color = decoder.string2color("(R=255,G=128,B=64)")
>>> print(color)  # [255, 128, 64]
>>>
>>> # Decode position coordinates
>>> pos = decoder.string2floats("100.0 200.0 300.0")
>>> print(pos)  # [100.0, 200.0, 300.0]

bpstring2floats(res)

Decode a string of numbers into a list of floats for blueprint.

Parameters

res (str) – The response string.

Returns

The list of floats or a single float.

Return type

list or float

bpvector2floats(res)

Decode a vector string for blueprint.

Parameters

res (str) – The response string.

Returns

The list of vectors.

Return type

list

cmd2key(cmd)

Extract the last word of the command as key.

Parameters

cmd (str) – The command string.

Returns

The key extracted from the command.

Return type

str

decode(cmd, res)

Universal decode function that selects appropriate decoder based on command.

Parameters

• cmd (str) – Command string.

• res (str) – Response data.

Returns

Decoded data in appropriate format.

Return type

Any

Example

>>> res = decoder.decode("vget /object/cube/color", "(R=255,G=128,B=64)")
>>> print(res)  # [255, 128, 64]

decode_bmp(res)

Decode BMP image data.

Parameters

res (bytes) – Raw BMP image data.

Returns

RGB image array of shape (H, W, 3) in uint8 format.

Return type

np.ndarray

Raises

ValueError – If image decoding fails.

decode_depth(res, inverse=False)

Decode a depth image.

Parameters

• res (str) – The response string.

• inverse (bool) – Whether to inverse the depth. Default is False.

Returns

The decoded depth image.

Return type

np.ndarray

decode_img(res, mode, inverse=False)

Decode an image.

Parameters

• res (str) – The response string.

• mode (str) – The image format (e.g., ‘png’, ‘bmp’, ‘npy’).

• inverse (bool) – Whether to inverse the depth. Default is False.

Returns

The decoded image.

Return type

np.ndarray

Note: The depth image should use the ‘npy’ mode to decode.

decode_npy(res)

Decode a NPY image.

Parameters

res (str) – The response string.

Returns

The decoded image.

Return type

np.ndarray

decode_png(res)

Decode a PNG image.

Parameters

res (str) – The response string.

Returns

The decoded image.

Return type

np.ndarray

decode_vertex(res)

Decode vertex locations.

Parameters

res (str) – Multi-line string of vertex coordinates.

Returns

List of vertex coordinates [x, y, z].

Return type

list[list[float]]

Example

>>> vertices = decoder.decode_vertex("0.0 0.0 0.0\n1.0 1.0 1.0")
>>> print(vertices)  # [[0.0, 0.0, 0.0], [1.0, 1.0, 1.0]]

empty(res)

Return the response as is.

Parameters

res (str) – The response string.

Returns

The response string.

Return type

str

string2color(res)

Decode color string to RGB values.

Parameters

res (str) – Color string in format “(R=255,G=128,B=64)”.

Returns

RGB color values as [r, g, b].

Return type

list[int]

Example

>>> color = decoder.string2color("(R=255,G=128,B=64)")
>>> print(color)  # [255, 128, 64]

string2floats(res)

Convert space-separated string of numbers to float list.

Parameters

res (str) – Space-separated numbers string.

Returns

List of floating point numbers.

Return type

list[float]

Example

>>> floats = decoder.string2floats("1.0 2.5 3.7")
>>> print(floats)  # [1.0, 2.5, 3.7]

string2list(res)

Convert a string to a list.

Parameters

res (str) – The response string.

Returns

The list of strings.

Return type

list

string2vector(res)

Decode a vector string.

Parameters

res (str) – The response string.

Returns

The vector as a list of floats.

Return type

list

Environment Launcher

The launcher module provides utilities to start and manage Unreal Engine environments.

UnrealCV launcher module for starting and managing Unreal Engine environments.

This module provides utilities for: - Launching Unreal Engine binaries locally or in Docker containers - Managing environment configurations - Handling multi-GPU setups - Supporting headless rendering

The module supports both local and containerized environments across Windows, Linux, and macOS.

Example

>>> from unrealcv.launcher import RunUnreal
>>> from unrealcv.api import UnrealCv_API
>>>
>>> # Launch local environment
>>> launcher = RunUnreal('path/to/UnrealBinary')
>>> ip, port = launcher.start()
>>>
>>> # Connect API
>>> api = UnrealCv_API(port, ip, resolution=(640, 480))
>>>
>>> # Use environment...
>>>
>>> # Cleanup when finished
>>> api.client.disconnect()
>>> launcher.close()

class unrealcv.launcher.RunDocker(path2env, image='zfw1226/unreal:latest')

Bases: object

check_image(target_images='zfw1226/unreal:latest')

close()

get_ip()

get_path2UnrealEnv()

Get the path to the Unreal environment. Default path to UnrealEnv is in user home directory under .unrealcv

Windows: C:Users<username>.unrealcvUnrealEnv Linux: /home/<username>/.unrealcv/UnrealEnv Mac: /Users/<username>/.unrealcv/UnrealEnv

Custom path can be set using the environment variable UnrealEnv.

start(ENV_BIN='/RealisticRendering_RL/RealisticRendering/Binaries/Linux/RealisticRendering', ENV_DIR_DOCKER='/UnrealEnv', options='', host_net=False)

Start the Unreal Engine environment in Docker.

Parameters

• ENV_BIN (str) – The path to the Unreal Engine binary.

• ENV_DIR_DOCKER (str) – The path to the Unreal environment in Docker. Default is ‘/UnrealEnv’.

• options (str) – Additional options for the Unreal Engine binary. Default is ‘’.

• host_net (bool) – Whether to use host networking. Default is False.

Returns

The IP address of the Docker container.

Return type

str

class unrealcv.launcher.RunUnreal(ENV_BIN, ENV_MAP=None)

Bases: object

Launches and manages Unreal Engine environments with UnrealCV support.

This class handles: - Local and Docker-based environment launching - Multi-GPU support - Headless rendering - Resolution and port configuration - Cross-platform compatibility

Parameters

• ENV_BIN (str) – The path to the executable Unreal Engine binary.

• ENV_MAP (str, optional) – The name of the Unreal Engine map. Default is None.

Example

>>> from unrealcv.launcher import RunUnreal
>>> from unrealcv.api import UnrealCv_API
>>>
>>> # Launch a local Unreal environment
>>> launcher = RunUnreal('path/to/UnrealBinary')
>>> ip, port = launcher.start()  # Launch with default settings
>>>
>>> # Launch headless on GPU 1 with custom resolution
>>> ip, port = launcher.start(offscreen=True,
...                          gpu_id=1,
...                          resolution=(640, 480))
>>>
>>> # Connect to environment with UnrealCV API
>>> unrealcv = UnrealCv_API(port, ip, resolution=(640, 480))
>>>
>>> # Use the UnrealCV API...
>>>
>>> # Cleanup
>>> unrealcv.client.disconnect()
>>> launcher.close()

close()

Close the Unreal Engine environment.

get_path2UnrealEnv()

isPortFree(ip, port)

Check if the port is free.

Parameters

• ip (str) – The IP address.

• port (int) – The port number.

Returns

True if the port is free, False otherwise.

Return type

bool

modify_permission(path)

Modify the permission of the environment path.

Parameters

path (str) – The path to modify.

parse_path(path)

Parse the path to get the root path and the relative path to the binary.

Parameters

path (str) – The path to parse.

Returns

The root path and the binary path.

Return type

tuple

read_port()

Read the port number from unrealcv.ini.

Returns

The port number.

Return type

int

set_ue_options(cmd_exe=[], opengl=False, offscreen=False, nullrhi=False, gpu_id=None)

Set options for running the Unreal Engine environment.

Parameters

• cmd_exe (list) – The command to execute.

• opengl (bool) – Whether to use OpenGL rendering. Default is False. If False, Vulkan is used.

• offscreen (bool) – Whether to render offscreen. Default is False. This is useful for headless servers.

• nullrhi (bool) – Whether to use null RHI. Default is False. This turns off the graphics rendering to save resources when rendering not needed.

• gpu_id (int, optional) – The GPU ID to use. Default is None. This is useful for multi-GPU systems.

Returns

The command with options.

Return type

list

signal_handler(signum, frame)

Handle signals to close the environment.

Parameters

• signum (int) – The signal number.

• frame (frame) – The current stack frame.

start(docker=False, resolution=(640, 480), display=None, opengl=False, offscreen=False, nullrhi=False, gpu_id=None, local_host=True, sleep_time=8)

Start the Unreal Engine environment.

Parameters

• docker (bool) – Whether to use Docker. Default is False.

• resolution (tuple) – The resolution of the image. Default is (640, 480).

• display (str, optional) – Specify the displaynumber. Default is None.

• opengl (bool) – Whether to use OpenGL rendering. Default is False (use Vulkan).

• offscreen (bool) – Whether to render offscreen. Default is False.

• nullrhi (bool) – Whether to use null RHI (turn off the graphics rendering). Default is False.

• gpu_id (int, optional) – The GPU ID to use. Default is None.

• local_host (bool) – Whether to use local host networking. Default is True.

• sleep_time (int) – The time to wait for the environment to launch. Default is 5 seconds.

Returns

The IP and port of the UnraelCV Server in the environment.

Return type

tuple

write_port(port)

write_resolution(resolution)

Set the UnrealCV camera resolution by writing to unrealcv.ini.

Parameters

resolution (tuple) – The resolution to set.

class unrealcv.launcher.RunUnreal(ENV_BIN, ENV_MAP=None)

Bases: object

Launches and manages Unreal Engine environments with UnrealCV support.

This class handles: - Local and Docker-based environment launching - Multi-GPU support - Headless rendering - Resolution and port configuration - Cross-platform compatibility

Parameters

• ENV_BIN (str) – The path to the executable Unreal Engine binary.

• ENV_MAP (str, optional) – The name of the Unreal Engine map. Default is None.

Example

>>> from unrealcv.launcher import RunUnreal
>>> from unrealcv.api import UnrealCv_API
>>>
>>> # Launch a local Unreal environment
>>> launcher = RunUnreal('path/to/UnrealBinary')
>>> ip, port = launcher.start()  # Launch with default settings
>>>
>>> # Launch headless on GPU 1 with custom resolution
>>> ip, port = launcher.start(offscreen=True,
...                          gpu_id=1,
...                          resolution=(640, 480))
>>>
>>> # Connect to environment with UnrealCV API
>>> unrealcv = UnrealCv_API(port, ip, resolution=(640, 480))
>>>
>>> # Use the UnrealCV API...
>>>
>>> # Cleanup
>>> unrealcv.client.disconnect()
>>> launcher.close()

close()

Close the Unreal Engine environment.

get_path2UnrealEnv()

isPortFree(ip, port)

Check if the port is free.

Parameters

• ip (str) – The IP address.

• port (int) – The port number.

Returns

True if the port is free, False otherwise.

Return type

bool

modify_permission(path)

Modify the permission of the environment path.

Parameters

path (str) – The path to modify.

parse_path(path)

Parse the path to get the root path and the relative path to the binary.

Parameters

path (str) – The path to parse.

Returns

The root path and the binary path.

Return type

tuple

read_port()

Read the port number from unrealcv.ini.

Returns

The port number.

Return type

int

set_ue_options(cmd_exe=[], opengl=False, offscreen=False, nullrhi=False, gpu_id=None)

Set options for running the Unreal Engine environment.

Parameters

• cmd_exe (list) – The command to execute.

• opengl (bool) – Whether to use OpenGL rendering. Default is False. If False, Vulkan is used.

• offscreen (bool) – Whether to render offscreen. Default is False. This is useful for headless servers.

• nullrhi (bool) – Whether to use null RHI. Default is False. This turns off the graphics rendering to save resources when rendering not needed.

• gpu_id (int, optional) – The GPU ID to use. Default is None. This is useful for multi-GPU systems.

Returns

The command with options.

Return type

list

signal_handler(signum, frame)

Handle signals to close the environment.

Parameters

• signum (int) – The signal number.

• frame (frame) – The current stack frame.

start(docker=False, resolution=(640, 480), display=None, opengl=False, offscreen=False, nullrhi=False, gpu_id=None, local_host=True, sleep_time=8)

Start the Unreal Engine environment.

Parameters

• docker (bool) – Whether to use Docker. Default is False.

• resolution (tuple) – The resolution of the image. Default is (640, 480).

• display (str, optional) – Specify the displaynumber. Default is None.

• opengl (bool) – Whether to use OpenGL rendering. Default is False (use Vulkan).

• offscreen (bool) – Whether to render offscreen. Default is False.

• nullrhi (bool) – Whether to use null RHI (turn off the graphics rendering). Default is False.

• gpu_id (int, optional) – The GPU ID to use. Default is None.

• local_host (bool) – Whether to use local host networking. Default is True.

• sleep_time (int) – The time to wait for the environment to launch. Default is 5 seconds.

Returns

The IP and port of the UnraelCV Server in the environment.

Return type

tuple

write_port(port)

write_resolution(resolution)

Set the UnrealCV camera resolution by writing to unrealcv.ini.

Parameters

resolution (tuple) – The resolution to set.

class unrealcv.launcher.RunDocker(path2env, image='zfw1226/unreal:latest')

Bases: object

check_image(target_images='zfw1226/unreal:latest')

close()

get_ip()

get_path2UnrealEnv()

Get the path to the Unreal environment. Default path to UnrealEnv is in user home directory under .unrealcv

Windows: C:Users<username>.unrealcvUnrealEnv Linux: /home/<username>/.unrealcv/UnrealEnv Mac: /Users/<username>/.unrealcv/UnrealEnv

Custom path can be set using the environment variable UnrealEnv.

start(ENV_BIN='/RealisticRendering_RL/RealisticRendering/Binaries/Linux/RealisticRendering', ENV_DIR_DOCKER='/UnrealEnv', options='', host_net=False)

Start the Unreal Engine environment in Docker.

Parameters

• ENV_BIN (str) – The path to the Unreal Engine binary.

• ENV_DIR_DOCKER (str) – The path to the Unreal environment in Docker. Default is ‘/UnrealEnv’.

• options (str) – Additional options for the Unreal Engine binary. Default is ‘’.

• host_net (bool) – Whether to use host networking. Default is False.

Returns

The IP address of the Docker container.

Return type

str

Automation Tools

Tools for building plugins and packaging model zoo binaries.

class unrealcv.automation.DockerBinary(binary_path)

Bases: unrealcv.automation.UE4BinaryBase

close()

start()

class unrealcv.automation.LinuxBinary(binary_path)

Bases: unrealcv.automation.UE4BinaryBase

close()

start()

class unrealcv.automation.MacBinary(binary_path)

Bases: unrealcv.automation.UE4BinaryBase

close()

start()

class unrealcv.automation.UE4Automation(engine)

Bases: object

UE4 engine wrapper

build_plugin(plugin_descriptor, output_folder, overwrite=False)

Use RunUAT script to build the plugin

Parameters

• plugin_descriptor (str) –

• output_folder (str) –

• overwrite (bool) – Whether the compiled binary folder should be overwriten?

install(plugin_folder, overwrite=False)

Install the plugin to UE4 engine folder

Parameters

plugin_folder (str) – The plugin folder with compiled binaries

package(project_descriptor, output_folder, overwrite=False)

Package an UE4 project

Parameters

• project_descriptor (str) – UE4 project file name ends with *.uproject

• overwrite (bool) – Overwrite existing files

unrealcv.automation.UE4Binary(binary_path)

Return a platform-dependent binary for user.

Examples

>>> binary = UE4Binary('./WindowsNoEditor/RealisticRendering.exe')  # For windows
>>> binary = UE4Binary('./LinuxNoEditor/RealisticRendering/Binaries/RealisticRendering') # For Linux
>>> binary = UE4Binary('./MacNoEditor/RealisticRendering.app') # For mac
>>> with binary: # Automatically launch and close the binary
>>>     client.request('vget /unrealcv/status')
>>> # Or alternatively
>>> binary.start()
>>> client.request('vget /unrealcv/status')
>>> binary.close()

class unrealcv.automation.UE4BinaryBase(binary_path)

Bases: object

UE4BinaryBase is the base class for all platform-dependent classes, it is different from UE4Binary which serves as a factory to create a platform-dependent binary wrapper. User should use UE4Binary instead of UE4BinaryBase

Binary is a python wrapper to control the start and stop of a UE4 binary. The wrapper provides simple features to start and stop the binary, mainly useful for automate the testing.

Usage:

bin = UE4Binary(‘/tmp/RealisticRendering/RealisticRendering’) with bin:

client.request(‘vget /camera/0/lit test.png’)

class unrealcv.automation.WindowsBinary(binary_path)

Bases: unrealcv.automation.UE4BinaryBase

close()

start()

unrealcv.automation.get_platform_name()

‘ Python and UE4 use different names for platform, in this script we will use UE4 platform name exclusively

unrealcv.automation.get_plugin_version(plugin_descriptor)

unrealcv.automation.UE4Binary(binary_path)

Return a platform-dependent binary for user.

Examples

>>> binary = UE4Binary('./WindowsNoEditor/RealisticRendering.exe')  # For windows
>>> binary = UE4Binary('./LinuxNoEditor/RealisticRendering/Binaries/RealisticRendering') # For Linux
>>> binary = UE4Binary('./MacNoEditor/RealisticRendering.app') # For mac
>>> with binary: # Automatically launch and close the binary
>>>     client.request('vget /unrealcv/status')
>>> # Or alternatively
>>> binary.start()
>>> client.request('vget /unrealcv/status')
>>> binary.close()

class unrealcv.automation.UE4Automation(engine)

Bases: object

UE4 engine wrapper

build_plugin(plugin_descriptor, output_folder, overwrite=False)

Use RunUAT script to build the plugin

Parameters

• plugin_descriptor (str) –

• output_folder (str) –

• overwrite (bool) – Whether the compiled binary folder should be overwriten?

install(plugin_folder, overwrite=False)

Install the plugin to UE4 engine folder

Parameters

plugin_folder (str) – The plugin folder with compiled binaries

package(project_descriptor, output_folder, overwrite=False)

Package an UE4 project

Parameters

• project_descriptor (str) – UE4 project file name ends with *.uproject

• overwrite (bool) – Overwrite existing files