Figure 1: SPEAR is a Python library that can connect to, and programmatically control, any Unreal Engine (UE) application via a modular plugin architecture. SPEAR exposes over 14K unique UE functions, representing an order-of-magnitude increase in programmable functionality over existing simulators. We demonstrate the flexibility of SPEAR by using it to control 6 distinct embodied agents (each with a different action space) across several Epic Games sample projects: a person and a car from CitySample (top); a flying robot from StackOBot (bottom far left); multiple agents in a resource collecting game called CropoutSample (bottom center left); as well as a person with parkour skills and a quadruped robot from GameAnimationSample (bottom right).

Figure 2: SPEAR includes a customizable camera sensor that can render 1920x1080 photorealistic beauty images (left) directly into a user's NumPy array at 73 frames per second - an order of magnitude faster than existing UE plugins - while also providing ground truth image modalities that are not available in any existing UE-based simulator. For example, the SPEAR camera sensor can render all of the image modalities in the Hypersim dataset, i.e., depths, surface normals, instance and semantic IDs (right top), and a non-diffuse intrinsic image decomposition (right bottom), as well as material IDs and physically based shading parameters.

Figure 3: (a, b): We demonstrate several fundamental concepts in the SPEAR programming model with a simple example program that spawns a set of coordinate axes in an indoor environment. In our programming model, graphs of UE work are specified as transactions. In particular, the user specifies a transaction by defining a begin_frame context followed by an end_frame context. Within each context, the user specifies a graph of UE work simply by implementing it as Python code. Any C++ function or variable that is visible to UE's reflection system (e.g., SetActorScale3D, RootComponent) can be accessed as though it was a native Python function or attribute. For improved efficiency, we provide an asynchronous variant for each function in SPEAR (e.g., call_async.K2_GetComponentLocation) that avoids synchronizing with UE. (c): It is trivial to expose new C++ functions and variables to UE's reflection system, and therefore to SPEAR, simply by adding a UFUNCTION or UPROPERTY annotation next to the function or variable in any C++ header file.

Figure 4: We demonstrate the flexibility of our programming model by using it to implement a variety of synchronization strategies found in existing simulators. For each strategy, we implement a simplified OpenAI Gym step function, which takes an action as input, steps a simulation forward, and returns an observation as output. In our programming model, it is straightforward to implement AirSim's approach for single-stepping, UnrealCV+'s approach for batched commands, Habitat 2.0's approach for double-buffered observations, CARLA's synchronous and asynchronous stepping modes, and co-simulation via an external physics simulator (e.g., MuJoCo) with user-defined sub-stepping.

Figure 5: We demonstrate the flexibility of SPEAR by using it to programmatically manipulate the ElectricDreams sample project from Epic Games. (a): We control UE's procedural content generation (PCG) system by translating the main PCG entity in this scene (the rock structure in the center of each image) from left to right. Notice how the rock structure automatically harmonizes with the rest of the scene in a convincing way (e.g., the water adjusts around the rock, logs appear and connect with nearby structures), even when it is being driven by our simple programmatic control. (b): We simulate time-of-day changes by controlling the orientation of the scene's sky light.

Figure 6: SPEAR can be used in co-simulation applications with external physics simulators. In this application, we interactively control the MuJoCo physics simulator using the default MuJoCo viewer, e.g., by applying a force to the leftmost chair (red arrow). In real-time as the MuJoCo simulation is running, we query the state of the MuJoCo scene (inset images), and we use SPEAR to update the state of a corresponding UE scene (large images).

Figure 7: We demonstrate the generality of the SPEAR camera sensor by using it to render synchronized multi-view images of a detailed human character in the MetaHumans sample project from Epic Games.

Figure 8: We demonstrate the flexibility of SPEAR by using it to implement an agentic natural-language scene editing application, where a vision-and-language coding assistant manipulates a UE scene by iteratively writing SPEAR programs in response to user text prompts.

Table 1: Comparison of programmable functionality across UE simulators. We report the number of hand-crafted functions and variables each simulator provides, as well as the number of underlying UE functions and variables it exposes (higher is better). For reference, we also report the total lines of Python and C++ code in each simulator codebase (lower is better). SPEAR provides an order of magnitude more programmable functionality than existing UE-based simulators while maintaining a modest code footprint.

Table 2: Comparison of rendering performance under various configurations when rendering the image on the right at 1920x1080 resolution. For each configuration, we measure the total end-to-end time required to deliver a rendered image to a user's Python program, and we report frame time (ms) and frames per second (FPS). As baselines, we include a standalone UE executable that does not communicate with Python, a standalone executable that does not communicate with Python but does the extra UE work necessary to do so (e.g., rendering an extra view of the scene to an off-screen buffer), and UnrealCV+. We report the performance of SPEAR with and without asynchronous communication and shared memory, and with different amounts of rendering latency. We find that asynchronous communication and shared memory both improve performance, as does increasing rendering latency, and that SPEAR is 9-21x faster than UnrealCV+ while also supporting more photorealistic rendering.

Table 3: Comparison of rendering performance across simulators when rendering the images on the right at 1920x1080 resolution. For each simulator, we measure the total end-to-end time required to deliver a rendered image to a user's Python program, and we report frame time (ms) and frames per second (FPS). We ensure the rendering speed of each simulator is as similar as possible when running as a standalone executable that does not communicate with Python (SPEAR: 89 FPS; CARLA: 90 FPS; AirSim: 93 FPS). Under these conditions, any differences we observe when measuring total end-to-end rendering time are attributable to differences in communication overhead. We find that SPEAR is 12x faster than AirSim, and is 10% faster than CARLA under matched amounts of rendering latency.