Unity ML-Agents Toolkit

Unity ML-Agents Toolkit
Information
Type	Software development kit
Industry	Video games, simulation, machine learning research
Developer	Unity Technologies
Written In	C# (Unity package), Python (trainers)
Operating System	Windows, macOS, Linux (training)
License	Apache License 2.0
Supported Devices	Unity runtime platforms (inference through Sentis)
Release Date	19 September 2017 (Beta 0.1); 30 April 2020 (Release 1)
Website	https://github.com/Unity-Technologies/ml-agents

The Unity ML-Agents Toolkit (ML-Agents) is an open-source project from Unity Technologies that lets games and simulations built with the Unity engine serve as environments for training intelligent agents with deep reinforcement learning, imitation learning and other machine learning methods. Its full name is the Unity Machine Learning Agents Toolkit; Unity's September 2017 announcement used "ML-Agents toolkit" as the short form. The toolkit has two parts: a C# package for Unity (com.unity.ml-agents), which turns a scene into a learning environment and runs trained models inside it, and companion Python packages that contain the training algorithms.

Unity's documentation lists uses for trained agents that include controlling non-player character (NPC) behavior, including in multi-agent and adversarial settings, "automated testing of game builds" and evaluating game design decisions before release. The code is released under the Apache License 2.0, and as of November 2026 the current Unity package is version 4.2.0, released on 30 October 2026. Published VR and mixed reality research has used the toolkit to train a redirected walking controller, an agent in a virtual reality exercise game and an agent that places 3D user interface elements around a mixed reality user.

Reviewed 1 November 2026. Release history and versions checked against the GitHub releases, the ML-Agents 4.2 and Sentis changelogs, package.json, the Unity package registry and PyPI; features against the 4.2 manual and Unity's 2017, 2020 and 2021 blog posts; research claims against the Juliani, Cohen, Chang, Lu and Gu papers and the Elor abstract and project repository. About review dates.

History

Open beta (2017-2020)

Unity announced the toolkit as an open beta in a blog post by Arthur Juliani dated 19 September 2017, and the project's first GitHub release, "ML-Agents Beta 0.1", carries the same date. The post described a software development kit that turns games and simulations created in the Unity Editor into environments where agents can be trained with deep reinforcement learning, evolutionary strategies or other machine learning methods through a Python API. It named three groups it expected to benefit: academic researchers studying multi-agent behavior in competitive and cooperative scenarios, industry researchers interested in large-scale parallel training for robotics, autonomous vehicles and other industrial applications, and game developers who want to fill virtual worlds with intelligent agents. Unity wrote that the beta was "more focused on" people interested in Unity as a platform for machine learning research and that it would increasingly support game development.

In the 2017 design, each agent was linked to a Brain that decided its actions in one of four modes: External, in which decisions came from TensorFlow or another library through an open socket to the Python API; Internal (experimental), which ran a trained model embedded in the project through TensorFlowSharp; Player, which used player input; and Heuristic, which used hand-coded behavior. An Academy object in each scene contained the Brains. The announcement described curriculum learning through custom environment parameters that can be set every time the environment is reset, and listed imitation learning as "Coming Soon".

The notes for Beta 0.2 (December 2017) refer to a Proximal Policy Optimization (PPO) trainer, which gained optional automatic normalization of states in that release. Later betas added behavioral cloning and LSTM models (0.3.0, March 2018), a curiosity-based intrinsic reward and training inside the Editor (0.4.0, June 2018), generative adversarial imitation learning or GAIL (0.9.0, August 2019), a Soft Actor-Critic (SAC) trainer (0.10.0, September 2019) and self-play for adversarial scenarios (0.14.0, February 2020).

Version 0.4.0 also moved communication between Unity and Python to gRPC and Protobuf. In 0.5.0 (September 2018) the Python code became a package named mlagents, and a new gym-unity package wrapped Unity environments in the gym interface. Version 0.6.0 turned Brains into ScriptableObject assets that can be reused across scenes and added a Demonstration Recorder, which records an agent's actions and observations in the Editor for later training. In 0.7.0 (February 2019) the Unity Inference Engine replaced TensorFlowSharp for running trained models; the 0.12.0 notes call it "Unity Inference Engine (Barracuda)". Version 0.14.0 moved the Unity code into a Unity package, com.unity.ml-agents, and the examples into a separate project folder. The notes for 0.8.0, which added training with several concurrent Unity simulations, thank Jam City for helping Unity evaluate the release's performance gains on Jam City's game Snoopy Pop.

Version 1.0 (2020)

Release 1, published on 30 April 2020, contained version 1.0.0 of the C# package and version 0.16.0 of the Python packages ml-agents, ml-agents-envs and gym-unity; its notes introduced separate version numbers for the packages that make up a release and recommend using packages from the same release together. GitHub marks the earlier 0.x betas as pre-releases; Release 1 is the first entry without that label. In an announcement dated 12 May 2020, Unity said the C# SDK had reached its first major version and was available as a Preview package through the Unity Package Manager, "on track to become a Verified package" for Unity 2020.2. The post named integrations with games, Jam City's Snoopy Pop and Carry Castle's Source of Madness, and research benchmarks built with ML-Agents: Unity's own Obstacle Tower environment and challenge, the Arena multi-agent environments, the Animal AI Olympics and the Marathon Environments.

PyTorch trainers and version 2.0 (2020-2021)

Release 7 (21 September 2020) added experimental PyTorch trainers, enabled with a --torch option and requiring PyTorch 1.6.0 or later. Release 8 (14 October 2020) added a Random Network Distillation (RND) intrinsic reward signal to the PyTorch trainers. Release 10 (20 November 2020) made the PyTorch trainers the default while keeping a TensorFlow option, and Release 13 (24 February 2021) removed the TensorFlow trainers. Release 15 (17 March 2021) added the MA-POCA trainer, which its notes say "enables Agents to learn how to work together in groups", and a BufferSensor for observing a variable number of entities. Release 17 (27 April 2021) shipped version 2.0.0 of the C# package with breaking API changes, Unity 2019.4 as the minimum supported version and Burst as the default inference device. Unity's version 2.0 announcement of 5 May 2021 said the 1.0 package "was verified for the 2020.2 Editor release" and that 2.0 was on track to be verified for the 2021.2 release; it also moved the Grid Sensors component and Match-3 game boards from the companion extensions package into the main package.

Sentis and Inference Engine (2022-2026)

Version 2.3.0-exp.3 of the C# package, part of Release 20 (29 November 2022), raised the minimum Unity version to 2021.3 and added a Custom Trainers plugin system for defining new trainers. Release 21 (9 October 2023), with version 3.0.0-exp.1, moved ML-Agents to Sentis 1.2.0-exp.2 and deprecated Barracuda; it also raised the minimum Unity version to 2022.3, updated the trainers to PyTorch 1.13.1 and deprecated Python 3.8 and 3.9. Version 3.0.0 (Release 22, October 2024) required Unity 2023.2 and updated the trainers to PyTorch 2.1.1. Version 4.0.0, dated 28 August 2025 in the changelog and published as Release 23 on 2 September 2025, moved to Inference Engine 2.2.1, required Unity 6000.0, merged the extensions package com.unity.ml-agents.extensions into the main package and made Unity's package documentation the primary developer documentation.

The 4.x updates that followed list Inference Engine 2.4.1 (4.0.1, December 2025) and 2.5.0 (4.0.2, February 2026), then Sentis 2.6.1 (4.0.3, April 2026), so ML-Agents' notes name the runtime inference library at different times as the Unity Inference Engine (Barracuda), Sentis and Inference Engine. The inference package's own changelog, published under the identifier com.unity.ai.inference, lists version 2.2.0 (15 May 2025) as the "First version of Inference Engine" and says that version 2.4.0 (22 October 2025) "Renamed Inference Engine to Sentis in package name and documentation". Version 4.1.0 (27 July 2026) moved the Python side from gym to gymnasium. The changelog lists only minor changes for version 4.2.0 (30 October 2026), among them a trainer fix for PPO and POCA value estimates that were being overwritten by returns in the training buffer.

Version history

The table lists selected releases. Dates for betas and numbered releases are GitHub release dates; versions without a GitHub release use the date in the package changelog.

Release	Date	Selected changes
Beta 0.1	19 September 2017	First open beta; four Brain modes; Python API
Beta 0.3.0	15 March 2018	Behavioral cloning, LSTM models, simultaneous training of several Brains, on-demand decisions
Beta 0.4.0	16 June 2018	Training in the Editor, curiosity reward, gRPC and Protobuf communication
Beta 0.5.0	11 September 2018	Python package mlagents, gym-unity wrapper, action masking, minimum Unity 2017.4
Beta 0.6.0	14 December 2018	Brains as ScriptableObject assets; Demonstration Recorder
Beta 0.7.0	28 February 2019	Unity Inference Engine replaces TensorFlowSharp
Beta 0.8.0	12 April 2019	Training with several concurrent Unity simulations
Beta 0.9.0	1 August 2019	GAIL; pre-training for the PPO trainer; varying the environment during training
Beta 0.10.0	30 September 2019	Soft Actor-Critic trainer
Beta 0.14.0	14 February 2020	Self-play; Unity package com.unity.ml-agents
Release 1	30 April 2020	C# package 1.0.0; Python packages 0.16.0
Release 7	21 September 2020	Experimental PyTorch trainers
Release 10	20 November 2020	C# package 1.6.0; PyTorch trainers become the default
Release 13	24 February 2021	TensorFlow trainers removed
Release 15	17 March 2021	MA-POCA trainer; BufferSensor
Release 17	27 April 2021	C# package 2.0.0; minimum Unity 2019.4
Release 20	29 November 2022	C# package 2.3.0-exp.3; Custom Trainers plugin; minimum Unity 2021.3
Release 21	9 October 2023	C# package 3.0.0-exp.1; Sentis replaces Barracuda; minimum Unity 2022.3
Release 22	5 October 2024	C# package 3.0.0; minimum Unity 2023.2; PyTorch 2.1.1
Release 23	2 September 2025	C# package 4.0.0; Inference Engine 2.2.1; minimum Unity 6000.0
4.1.0	27 July 2026	Python side moves from gym to gymnasium
4.2.0	30 October 2026	Minor changes and fixes; current version as of November 2026

How it works

Unity's documentation describes the toolkit's main components as the Learning Environment, which is the Unity scene with its agents, set up with the ML-Agents Unity SDK; the Python Low-Level API in the mlagents_envs package, which interacts with and controls a learning environment from outside Unity; the External Communicator, which lives inside the Learning Environment and connects it to the Python API; and the Python Trainers in the mlagents package, which hold the machine learning algorithms and run through a single command-line utility, mlagents-learn. Unity also provides wrappers for gymnasium, the Farama Foundation interface formerly known as OpenAI gym, and for PettingZoo, a gym-like API for multi-agent environments.

In the documentation's terms, an agent receives observations, which can be numeric or visual, takes actions and receives reward signals. Each agent is linked to a Behavior of one of three types: Learning (not yet defined and about to be trained), Heuristic (hard-coded rules) or Inference (a trained neural network), and several agents can share one Behavior. An Academy keeps the agents in sync and controls environment-wide settings, and Side Channels exchange other data, such as environment parameters, between Unity and Python outside the machine learning loop. The documentation's running example is a hypothetical medic NPC in a war-themed multiplayer game, which has to learn to avoid getting injured and to find and revive wounded players.

The C# package does not contain the machine learning algorithms, which are part of the companion Python package; after training, the trained behavior is embedded in the Unity scene. Training produces models in the open ONNX format (.onnx files), which run inside Unity through Sentis. Unity's documentation states that Sentis supports all Unity runtime platforms and offers Compute Shader, Burst and Pixel Shader inference devices, and that for most models generated with the toolkit, CPU inference with Burst "will be faster than GPU inference"; it recommends GPU inference only for the ResNet visual encoder or for large numbers of agents with visual observations. ML-Agents supports only models created with Unity's trainers, and Unity does not support inference with these models outside Unity.

Unity's example environment reference describes 17 example learning environments, which are in the repository's Project/Assets/ML-Agents/Examples folder, among them 3DBall, GridWorld, Push Block, Wall Jump, Walker, Pyramids, Soccer Twos, Match 3, Cooperative Push Block and Dungeon Escape. Unity provides pre-trained models and training configurations for the environments that demonstrate specific features, and it does not accept contributed environments into the repository.

Training methods

The documentation lists two deep reinforcement learning algorithms, PPO (the default) and SAC, plus MA-POCA and self-play for multi-agent scenarios and two imitation learning methods, behavioral cloning (BC) and GAIL. Training uses an extrinsic reward defined in the environment, enabled by default, and can add intrinsic reward signals from GAIL, curiosity or RND.

Method	Type	Description in Unity's documentation and announcements	Added
PPO	Reinforcement learning	Default algorithm, described as more general purpose and stable than many other reinforcement learning algorithms	By Beta 0.2 (December 2017)
SAC	Reinforcement learning (off-policy)	Learns from an experience replay buffer; often needs "5-10 times less samples" than PPO but more model updates; suggested for slower environments of about 0.1 seconds per step or more	Beta 0.10.0 (September 2019)
MA-POCA	Cooperative multi-agent reinforcement learning	Multi-Agent POsthumous Credit Assignment: trains a centralized critic that acts as a "coach" for the whole group; agents can be added to or removed from the group during an episode	Release 15 (March 2021)
Self-play	Adversarial multi-agent training	Two interacting agents with inverse reward signals; an agent competes against fixed, past versions of its opponent, which in symmetric games can be itself	Beta 0.14.0 (February 2020)
Behavioral cloning	Imitation learning	Trains the policy to mimic the actions in a set of demonstrations; available on the PPO and SAC trainers	Beta 0.3.0 (March 2018)
GAIL	Imitation learning	Imitation learning "from as few as 5-10 demonstrations", according to the 0.9.0 notes	Beta 0.9.0 (August 2019)
Curiosity	Intrinsic reward	Encourages exploration in sparse-reward environments	Beta 0.4.0 (June 2018)
RND	Intrinsic reward	Helps exploration in sparse or rare reward environments; implemented following the paper "Exploration by Random Network Distillation"	Release 8 (October 2020)

Other documented features include curriculum learning, environment randomization, on-demand decisions, training across several concurrent Unity instances and plugins for custom trainers. Demonstrations can be recorded in the Unity Editor or a build and saved as assets. According to the documentation, on the sparse-reward Pyramids example, "using 6 episodes of demonstrations can reduce training steps by more than 4 times" when imitation learning is combined with reinforcement learning.

Research papers

Unity staff described the platform in "Unity: A General Platform for Intelligent Agents" by Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar and Danny Lange, an arXiv preprint first posted in September 2018 and revised in May 2020 to describe version 1.0. The paper sorts simulators into four categories (environment, environment suite, domain-specific platform and general platform) and argues that Unity with ML-Agents is a general platform. It describes the SDK's sensors, agents and Academy, the Python API with its gym wrappers, and the training features then included (PPO, SAC, GAIL, behavioral cloning, self-play, curiosity and LSTM support), with baseline PPO and SAC results averaged over five runs on the example environments. It states that simulations can be run "up to one hundred times real-time", with the speed-up in practice depending on computing resources and environment complexity. Its Table 2 reports the time for env.step() calls made through the Python API on the same machine, averaged over 1,000 simulation steps:

Environment	Observation	Agents	Mean time per step (ms)	Standard deviation (ms)
Basic	Vector (1)	1	0.803	0.005
3D Ball	Vector (8)	12	5.05	0.039
GridWorld	Visual (84x84x3)	1	2.04	0.038
Visual Food Collector	Visual (84x84x3)	4	9.23	0.556

The MA-POCA trainer is described in "On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning" by Andrew Cohen and seven co-authors at Unity Technologies, an arXiv preprint first posted in November 2021 that its arXiv record lists for the RL in Games workshop at AAAI 2022. The paper calls the problem of crediting agents that leave an episode before their teammates "Posthumous Credit Assignment" and presents MA-POCA as a new architecture for the multi-agent algorithm COMA (Counterfactual Multi-Agent Policy Gradients) whose critic uses self-attention over the active agents instead of a fully connected layer with absorbing states. It compared MA-POCA, COMA and PPO over 10 seeds in four environments, three built with ML-Agents and one from the Multi-Agent Particle Environments. MA-POCA learned slightly faster than COMA in the two tasks without spawning or dying agents and "significantly outperforms" COMA, with less variance between seeds, in the two tasks with them; PPO, trained and acting in a decentralized way, converged to a local optimum in all four.

Use in VR and mixed reality research

Unity's 2020 announcement described the toolkit's machine learning library as training agents "for 2D, 3D, and VR/AR environments". The following published studies document its use in VR and mixed reality work.

Redirected walking

Yuchen Chang, Keigo Matsumoto, Takuji Narumi, Tomohiro Tanikawa and Michitaka Hirose built a redirected walking controller with the PPO trainer of the Unity ML-Agents Toolkit (Beta), reported in IEEE Access in 2021. They trained 16 agents in parallel for a total of 16 million simulation steps, which took about 2 hours on a PC with an Intel Core i7-8750H CPU, and running the trained agent took 1.81% of a frame at 60 frames per second. In simulation, the controller needed fewer resets than a conventional generalized controller in physical spaces with many obstacles, but the gain it output oscillated. In a within-subject user study, 24 participants (11 female, 13 male; mean age 25.4 years) walked in an 8 m by 8 m tracked space, with or without two 1.5 m by 1.5 m off-limit areas standing for obstacles, wearing an HTC Vive Pro connected to an MSI VR One backpack PC. The RL controller produced fewer resets than the best-performing heuristic controller (two-way ANOVA, controller effect F(1,23) = 57.14, p < 0.01), and post hoc tests found a difference between the two controllers in the environment with obstacles. The authors found no significant effect on simulator sickness scores from the Simulator Sickness Questionnaire, and an overall presence rating of 3.38 out of 6 on the iGroup Presence Questionnaire (IPQ); the abstract reports no adverse effects such as cybersickness from the oscillating gain.

Exercise game

In a paper for the 2020 IEEE 8th International Conference on Serious Games and Applications for Health (SeGAH), Aviv Elor and Sri Kurniawan described a VR exercise game built with the Unity engine, ML-Agents and the HTC Vive head-mounted display. According to the abstract, they applied PPO and GAIL to physical exercises in the game and examined the mechanics with four users, who protected a virtual butterfly with an agent acting as a cooperative "ghost arm" that visually helped them and as an independent competitor. The authors write that their results "suggest that deep learning agents are effective at learning game exercises and may provide unique insights for users". The project's public repository adds that demonstrations were recorded with a 2018 HTC Vive system and two 2018 Vive Trackers on a demonstrator's shoulder and elbow, and that 16 agents were trained in parallel with PPO; after four hours of training, the reward rose from 0.1 to 40, with +0.01 given for each frame in which the arm protected the butterfly. The repository lists the ML-Agents beta and its Reacher example among the external tools used and modified for the project.

Mixed reality interface placement

Feiyu Lu, Mengyu Chen, Hsiang Hsu, Pranav Deshpande, Cheng Yao Wang and Blair MacIntyre of JPMorgan Chase & Co. described a reinforcement learning agent that keeps a piece of 3D user interface content placed around a mixed reality user, in an extended abstract at CHI 2024. They trained the agent with PPO using the ML-Agents toolkit, which links the Unity simulation to a PyTorch backend. The agent moved the content by applying a force vector through the physics simulation, choosing a new action every five time steps, and its reward combined visibility, reachability, physicality and stability terms. Training ran six parallel instances for five million steps and took about six hours on a desktop PC with an Nvidia GeForce RTX 3080 GPU, an Intel i9 24-core CPU and 64 GB of RAM.

The preliminary evaluation was run in simulation: the agent ran for 10,000 steps in each of four Replica environments (two used for training, two for validation) while a simulated user moved through the environment at random. In the training environments, more than 91% of the interface stayed continuously visible and it avoided collisions with the environment mesh more than 94% of the time; in the two unseen validation environments, around 85% stayed visible. With moving obstacles added, the drops in visibility and physicality on the validation environments were at most 3.12%. The authors write that the larger drops in the training environments with dynamic obstacles might be a sign of overfitting. They also note that their training simulated AR and MR environments in VR, and that training in actual AR or MR spaces may bring further challenges.

XR software testing

A 2025 systematic mapping study of software testing for XR applications by Ruizhen Gu, José Miguel Rojas and Donghwan Shin, published in Automated Software Engineering, examined 34 primary studies and lists ML-Agents among the industrial tools those studies used or referenced. It describes the toolkit, with its more than 17 example environments, as well suited to evaluating XR testing approaches, including agent-based testing. The wider field is covered in Automated testing of VR applications.

Requirements and supported platforms

For the 4.x versions, Unity's installation guide requires Unity 6000.0 or later, recommends a Python 3.10.12 environment created with Conda or Mamba, and tells users to install PyTorch 2.1.1 to 2.8.0 before installing ML-Agents, because the PyPI package sets no upper limit on the PyTorch version and "PyTorch 2.9 and later include ONNX changes that break ML-Agents model export". The Unity package is added by name (com.unity.ml-agents) in the Package Manager, and the Python package is installed from PyPI as mlagents 1.1.0. The guide notes that mlagents 1.1.0 predates the 4.x Unity packages: it is compatible with them but lacks Python-side changes made in those releases, such as the PyTorch 2.8 limit, CUDA and CPU device fixes and the move from gym to gymnasium, which require installing from source. PyPI lists mlagents 1.1.0, released on 5 October 2024, as the latest version of the package, with a requirement of Python 3.10.1 to 3.10.12.

Training is limited to the Unity Editor and standalone builds on Windows, macOS and Linux with the Mono scripting backend and does not work with IL2CPP; an environment defaults to inference mode when training is not supported or not currently running. Inference runs on the end user's device, and Unity's Sentis page states that Sentis supports all Unity runtime platforms. Unity says it added in-editor analytics to the toolkit to improve the developer experience, and refers users to the Unity Privacy Policy.

Limitations

Unity's documentation lists several limits. Game physics can be sped up to at most 100 times real time. The Academy steps with FixedUpdate() rather than Update(), so game behavior implemented in Update() may be out of sync with agent decisions. Headless mode prevents the collection of visual observations. Inference performance depends on the end user's CPU or GPU, and only models created with Unity's trainers are supported for neural network behaviors. The InputActuatorComponent, part of the Input System integration, has a limited implementation of input controls and no way to customize its action space. The training documentation adds that behavioral cloning cannot generalize past the examples in its demonstrations, so it works best when demonstrations cover nearly all the states an agent can experience, or together with GAIL or an extrinsic reward. GAIL introduces a survivor bias: because it rewards similarity to the expert, the agent has an incentive to stay alive as long as possible, which can conflict with goal-oriented tasks such as the Push Block and Pyramids examples; for these, Unity recommends a low-strength GAIL reward with a sparse extrinsic reward.

Current status

As of 1 November 2026, the latest version of the Unity package is 4.2.0, released on 30 October 2026. Its package manifest names Unity 6000.0 as the minimum editor version and depends on Unity's com.unity.ai.inference package, version 2.6.1. The newest entry on the project's GitHub releases page is Release 23 (2 September 2025), which carries package 4.0.0; the installation guide points to the release/4.2.0 branch of the repository as the latest stable release. The newest Python package on PyPI remains mlagents 1.1.0 from October 2024. The documentation marks its cloud deployment guides as legacy and deprecated.

See also

• Unity AI
• How to develop VR games with AI
• Generative AI non-player characters
• XR Interaction Toolkit