Unity apps are usually developed within the Unity Editor and then published to standalone apps for iOS, Android, Windows, Mac, and so forth. We’d therefore like to think that what we’re seeing in the Editor is very close to what we’ll see once we publish the app. In many cases, this is true. However, when it comes to performance this is sometimes a bad assumption to make.

In my introductory article to performance testing I mentioned this about performance testing in the Editor:

The final issue is that code running in the Unity Editor is running in something of a debug or development mode. It’s roughly equivalent to the performance you can expect from a distributed game, but sometimes slower. For micro-benchmarks like these, it’s important to actually make a distribution build of the game.

Today I’ll show one case where performance in the Editor is slower than in a published app. As it happens, this is an important case because it’s triggered directly by the auxiliary MonoBehaviour used to capture and forward events in a “pure code” app.

The test sets up 1000 game objects and attaches one of two MonoBehaviour scripts to them. The first is a simple Updater script that has an empty Update on it:

using UnityEngine;
 
public class Updater : MonoBehaviour
{
	void Update()
	{
	}
}

The second is the full EventForwarder script that handles every Unity event type as of Unity 4.6. It’s a bit long so I won’t repeat it here, but you can find its source in the original article.

To tie these together into a test, here’s a simple script to attach to your main camera in an otherwise empty scene:

using System;
 
using UnityEngine;
 
public class TestScript : MonoBehaviour
{
	private const int NumGameObjects = 1000;
	private const float UpdateInterval = 1;
 
	private Rect drawRect;
	private bool showModeScreen;
	private float totalTime;
	private int numFrames;
	private float timeleft;
	private float fps;
 
	void Start()
	{
		drawRect = new Rect(0, 0, Screen.width, Screen.height);
		showModeScreen = true;
	}
 
	void OnGUI()
	{
		if (showModeScreen)
		{
			if (GUI.Button(new Rect(0, 0, 100, 100), "All Events"))
			{
				StartTest<EventForwarder>();
			}
			else if (GUI.Button(new Rect(100, 0, 100, 100), "Just Update"))
			{
				StartTest<Updater>();
			}
		}
		else
		{
			timeleft -= Time.deltaTime;
			totalTime += Time.timeScale / Time.deltaTime;
			numFrames++;
 
			if (timeleft <= 0)
			{
				fps = totalTime / numFrames;
				timeleft = UpdateInterval;
				totalTime = 0;
				numFrames = 0;
			}
 
			GUI.Label(drawRect, "FPS: " + fps);
		}
	}
 
	private void StartTest<TMonoBehaviour>()
		where TMonoBehaviour : MonoBehaviour
	{
		for (var i = 0; i < NumGameObjects; ++i)
		{
			var otherGameObject = new GameObject("GameObject #" + i);
			otherGameObject.AddComponent<TMonoBehaviour>();
		}
 
		showModeScreen = false;
		timeleft = updateInterval;
	}
}

I ran the tests on this computer:

  • 2.3 Ghz Intel Core i7-3615QM
  • Mac OS X 10.10.2
  • Unity 4.6.2

The first test was in the Unity Editor to establish a baseline. Here are the results:

Test FPS
All Events 32
Just Update 60

Unity Editor Performance

There is definitely a performance difference here. The “Just Update” test that only listens for Update events seems to be pegged at 60 FPS. Meanwhile, the “All Events” test is running about twice as slow at only 32 FPS. This seems like a huge gap until we run the test as a standalone app.

Here are the test results run as a Mac OS X standalone app where the build settings were set to x86_64, non-development and the run settings were set to 640×480, Fastest, and Windowed.

Test FPS
All Events 57
Just Update 668

Standalone App Performance

The 2x performance gap has widened to almost 12x! The 60 FPS cap in the Editor was hiding the full impact of the extra event handling. This leads us to ask if the only issue is the FPS cap in the Editor. So let’s adjust the test to 15,000 game objects instead of only 1,000.

Here are the updated results in the editor:

Test FPS
All Events 3
Just Update 51

Unity Editor Performance

The performance gap has now widened even further to 17x! Let’s see what happens if we run as a standalone app:

Test FPS
All Events 3
Just Update 246

Standalone App Performance

Now the performance gap is 82x! We’ve gone from 2x to 12x to 17x to 82x by only changing the way we run the app and the number of test game objects.

So what is the “true” performance difference between “All Events” and “Just Update”. Well, end users will never run the app in the Unity Editor so we can throw out those numbers. We’re left with 12x for 1,000 game objects and 82x for 15,000. Since these gaps are in the relative performance—not the absolute—we can conclude that the performance difference widens as you add more and more game objects. The reason for this is unclear as we can’t profile at this level within the Unity engine.

The main takeaway here isn’t the performance difference between “All Events” and “Just Update”. Instead, it’s to highlight the differences in testing environment. It is key to understand each environment’s performance characteristics, such as the FPS cap in the Unity Editor and that lack of a cap—event at monitor refresh rate—in standalone apps. It’s also important to not assume linear performance scaling, as we’ve seen with the varying performance gaps.

In the end you should always test in your target environment, be it a physical iPad or a designated “minimum specification” computer. Testing frequently will keep you in touch with the actual performance your end users will see. Varying your testing parameters will give you insight into the performance impacts of potential decisions, such as scaling up the number of entities in your game scene. Do not simply rely on the performance you see in the Unity Editor!

If you know of any other performance differences the Unity Editor imposes, post a comment about it!