JacksonDunstan.com

Having just concluded the series on handling errors without using exceptions, now’s a good time to check up on an assertion I made in the first part: exceptions are slow. A good question to ask is “compared to what?” So let’s put them up against good old error codes and the new Either type I presented in the series. Which is fastest when there is no error? Which is fastest when there is an error? Read on to find out!

First let’s directly test the assertion that exceptions are slow. We can do this with three simple functions:

void NoOp()
{
}
 
void TryCatch()
{
	try
	{
	}
	catch (Exception)
	{
	}
}
 
void Throw(Exception ex)
{
	try
	{
		throw ex;
	}
	catch (Exception)
	{
	}
}

NoOp does nothing and sets the baseline. TryCatch does nothing, but catches an exception. This simulates the case where there is no error and no exception is thrown. Throw does nothing except throw a pre-allocated function that’s passed to the function. This simulates the case where there is an error and an exception is thrown.

Next we run these in a tiny script:

using System;
 
using UnityEngine;
 
public class TestScript : MonoBehaviour
{
	const int NumReps = 10000000;
 
	string report;
 
	void Start()
	{
		var sw = new System.Diagnostics.Stopwatch();
		var ex = new Exception();
 
		sw.Reset();
		sw.Start();
		for (var i = 0; i < NumReps; ++i)
		{
			NoOp();
		}
		var noOpTime = sw.ElapsedMilliseconds;
 
		sw.Reset();
		sw.Start();
		for (var i = 0; i < NumReps; ++i)
		{
			TryCatch();
		}
		var tryCatchTime = sw.ElapsedMilliseconds;
 
		sw.Reset();
		sw.Start();
		for (var i = 0; i < NumReps; ++i)
		{
			Throw(ex);
		}
		var throwTime = sw.ElapsedMilliseconds;
 
		report = "Operation,Time"
			+ "\nNo-Op," + noOpTime
			+ "\nTry-Catch," + tryCatchTime
			+ "\nThrow," + throwTime
			;
		Debug.Log(report);
	}
 
	void OnGUI()
	{
		GUI.TextArea(new Rect(0, 0, Screen.width, Screen.height), report);
	}
 
	void NoOp()
	{
	}
 
	void TryCatch()
	{
		try
		{
		}
		catch (Exception)
		{
		}
	}
 
	void Throw(Exception ex)
	{
		try
		{
			throw ex;
		}
		catch (Exception)
		{
		}
	}
}

If you want to try out the test yourself, simply paste the above code into a TestScript.cs file in your Unity project’s Assets directory and attach it to the main camera game object in a new, empty project. Then build in non-development mode for 64-bit processors and run it windowed at 640×480 with fastest graphics. I ran it that way on this machine:

• 2.3 Ghz Intel Core i7-3615QM
• Mac OS X 10.11.3
• Unity 5.3.2f1, Mac OS X Standalone, x86_64, non-development
• 640×480, Fastest, Windowed

And here are the results I got:

Throwing the exception was so slow it was too hard to see the difference between NoOp and TryCatch, so I made a chart of just those two. It’s 246x slower than not throwing the exception, which is astonishingly huge! By comparison, the slowdown of just a try-catch is a relatively tame 8x. That’s still quite a lot for code that does nothing though.

Now let’s move on to a more interesting test. Next we’ll compare three different error-handling strategies: error codes, exceptions, and Either. We’ll also split Either into two parts: direct checking for the error via IsLeft and the helper Match function with some no-op lambdas. For each strategy we’ll test both success and failure cases, just like above.

Here are the test functions. All of them do integer division as a simple function that has an error case: the denominator can’t be zero.

int DivErrorCode(int a, int b)
{
	if (b == 0)
	{
		return -1;
	}
	return a / b;
}
 
int DivException(int a, int b)
{
	if (b == 0)
	{
		throw new Exception();
	}
	return a / b;
}
 
enum DivError { DenominatorZero };
 
Either<int, DivError> DivEither(int a, int b)
{
	if (b == 0)
	{
		return new Either<int, DivError>(DivError.DenominatorZero);
	}
	return new Either<int, DivError>(a / b);
}

DivErrorCode returns -1 if the denominator is zero. It’s not a very good error code, but it’ll work for this simple test. DivException creates an Exception and throws it. DivEither creates an Either with the right/error type which is a simple enum.

Here’s the test app that runs it. For success, it divides 4 by 2. For failure, it divides 4 by 0.

using System;
 
using UnityEngine;
 
public class TestScript : MonoBehaviour
{
	const int NumReps = 1000000;
 
	string report;
 
	void Start()
	{
		var sw = new System.Diagnostics.Stopwatch();
		var four = 4;
		var two = 2;
		var zero = 0;
 
		sw.Reset();
		sw.Start();
		for (var i = 0; i < NumReps; ++i)
		{
			var ret = DivErrorCode(four, two);
			if (ret == -1)
			{
			}
			else
			{
			}
		}
		var errorCodeSuccessTime = sw.ElapsedMilliseconds;
 
		sw.Reset();
		sw.Start();
		for (var i = 0; i < NumReps; ++i)
		{
			var ret = DivErrorCode(four, zero);
			if (ret == -1)
			{
			}
			else
			{
			}
		}
		var errorCodeFailureTime = sw.ElapsedMilliseconds;
 
		sw.Start();
		for (var i = 0; i < NumReps; ++i)
		{
			try
			{
				DivException(four, two);
			}
			catch (Exception)
			{
			}
		}
		var exceptionSuccessTime = sw.ElapsedMilliseconds;
 
		sw.Reset();
		sw.Start();
		for (var i = 0; i < NumReps; ++i)
		{
			try
			{
				DivException(four, zero);
			}
			catch (Exception)
			{
			}
		}
		var exceptionFailureTime = sw.ElapsedMilliseconds;
 
		sw.Reset();
		sw.Start();
		for (var i = 0; i < NumReps; ++i)
		{
			var e = DivEither(four, two);
			if (e.IsLeft)
			{
			}
			else
			{
			}
		}
		var eitherSuccessTime = sw.ElapsedMilliseconds;
 
		sw.Reset();
		sw.Start();
		for (var i = 0; i < NumReps; ++i)
		{
			var e = DivEither(four, zero);
			if (e.IsLeft)
			{
			}
			else
			{
			}
		}
		var eitherFailureTime = sw.ElapsedMilliseconds;
 
		sw.Reset();
		sw.Start();
		for (var i = 0; i < NumReps; ++i)
		{
			DivEither(four, two).Match(
				quotient => 0,
				error => 0
			);
		}
		var matchSuccessTime = sw.ElapsedMilliseconds;
 
		sw.Reset();
		sw.Start();
		for (var i = 0; i < NumReps; ++i)
		{
			DivEither(four, zero).Match(
				quotient => 0,
				error => 0
			);
		}
		var matchFailureTime = sw.ElapsedMilliseconds;
 
		report = "Error Type,Success Time,Failure Time"
			+ "\nError Code," + errorCodeSuccessTime + "," + errorCodeFailureTime
			+ "\nException," + exceptionSuccessTime + "," + exceptionFailureTime
			+ "\nEither (IsLeft)," + eitherSuccessTime + "," + eitherFailureTime
			+ "\nEither (Match)," + matchSuccessTime + "," + matchFailureTime
			;
		Debug.Log(report);
	}
 
	void OnGUI()
	{
		GUI.TextArea(new Rect(0, 0, Screen.width, Screen.height), report);
	}
 
	int DivErrorCode(int a, int b)
	{
		if (b == 0)
		{
			return -1;
		}
		return a / b;
	}
 
	int DivException(int a, int b)
	{
		if (b == 0)
		{
			throw new Exception();
		}
		return a / b;
	}
 
	enum DivError { DenominatorZero };
 
	Either<int, DivError> DivEither(int a, int b)
	{
		if (b == 0)
		{
			return new Either<int, DivError>(DivError.DenominatorZero);
		}
		return new Either<int, DivError>(a / b);
	}
}

I ran this in the same environment as the first test and got these results:

As we saw in the simple tests we started with, when an exception is thrown the performance is disastrous. It’s hard to even see the other error handling strategies in the first chart, so I once-again added another to remove that case. It’s 84x slower than the next-slowest case, so it’s best to avoid ever having to throw an exception if you’re using that strategy.

As for the others, they’re pretty densely packed compared to the gap between them and the case where an exception is thrown. That said, error codes were the fastest strategy of all. They’re followed up by exceptions (when the exception isn’t actually thrown) and then Either. Exceptions add about 50% more time compared to error codes and Either is about 2.4x slower. The failure case is worse for Either with a 3.2x slowdown.

Using the convenience Match function with Either adds more time due to the additional function calls it requires. It’s 4.4x and 5.4x slower than error codes but not a ton slower than manually checking the IsLeft property.

At this point it’s a good idea to put these numbers in perspective. They’re all in milliseconds but magnified one million times so we don’t just get a bunch of zeroes. This means that even in the case where an exception is thrown it only costs about 0.002615 milliseconds of performance. If you throw them a hundred or a thousand times per frame then you’re going to start to notice. This jives with the advice that they shouldn’t be used as part of normal control flow. You probably shouldn’t use them for anything you expect might fail, either. Instead, use either error codes or Either for much better performance.

Speaking of Either, the extra cost compared to error codes is about a hundred thousandth of a millisecond. Compared to exceptions that don’t get thrown it’s about half that: a two hundred thousandth of a millisecond. Even with the Match helper function it’s safe to say that they’re very cheap.

For more on the non-performance considerations of choosing error codes, exceptions, or Either for your error-handling strategy, see the two-part series. What do you think? Does the performance of your error-handling strategy matter to you? Let me know in the comments!