JacksonDunstan.com

Last week’s article benchmarked Unity 5.3’s new JsonUtility class against third-party alternatives LitJSON and Json.NET. JsonUtility came out the clear winner, but the question arose about how JsonUtility would fare with bigger or more complex JSON structures. Today’s article answers that question by benchmarking with more types of JSON documents to find out if JsonUtility can maintain its lead.

This week we’ll drop the simple SaveGame test class and use several classes instead. Each will target a particular aspect of JSON to see how the serializers do. Here are those target areas:

• An empty class
• A single string field
• A single array field
• Nested class instances

Here’s the source code for today’s test including the classes that test these target areas:

using System;
using System.Collections.Generic;
using System.Text;
 
using UnityEngine;
 
using LitJson;
using Newtonsoft.Json;
 
[Serializable]
public class EmptyClass
{
}
 
[Serializable]
public class StringOnly
{
	public string String;
}
 
[Serializable]
public class ArrayOnly
{
	public int[] IntArray;
}
 
[Serializable]
public class A
{
	public B B;
}
 
[Serializable]
public class B
{
	public C C;
}
 
[Serializable]
public class C
{
	public D D;
}
 
[Serializable]
public class D
{
	public E E;
}
 
[Serializable]
public class E
{
	public F F;
}
 
[Serializable]
public class F
{
	public G G;
}
 
[Serializable]
public class G
{
	public H H;
}
 
[Serializable]
public class H
{
	public I I;
}
 
[Serializable]
public class I
{
	public int Val;
}
 
class TestScript : MonoBehaviour
{
	string report = "";
 
	void Start()
	{
		var emptyClass = new EmptyClass();
		var smallString = new StringOnly { String = "hey" };
		var largeString = new StringOnly { String = new string('*', 100000) };
		var smallArray = new ArrayOnly { IntArray = new [] { 1, 2, 3 } };
		var largeArray = new ArrayOnly { IntArray = new int[1000] };
		var shallowNestedObject = new A { B = new B { C = new C() { } } };
		var deepNestedObject = new A {
			B = new B {
				C = new C {
					D = new D {
						E = new E {
							F = new F {
								G = new G {
									H = new H {
										I = new I { Val = 123 }
									} } } } } } } };
		Test("Empty Class", emptyClass, 100000);
		Test("Small String", smallString, 10000);
		Test("Large String", largeString, 100);
		Test("Small Array", smallArray, 100000);
		Test("Large Array", largeArray, 100);
		Test("Shallow Nested Object", shallowNestedObject, 10000);
		Test("Deep Nested Object", deepNestedObject, 10000);
	}
 
	void Test<T>(string label, T obj, int reps)
		where T : class
	{
		// Warm up reflection
		JsonUtility.ToJson(obj);
		JsonMapper.ToJson(obj);
		JsonConvert.SerializeObject(obj);
 
		var stopwatch = new System.Diagnostics.Stopwatch();
		string unityJson = null;
		string litJsonJson = null;
		string jsonDotNetJson = null;
		T unityObj = null;
		T litJsonObj = null;
		T jsonDotNetObj = null;
 
		stopwatch.Start();
		for (var i = 0; i < reps; ++i)
		{
			unityJson = JsonUtility.ToJson(obj);
		}
		var unitySerializeTime = stopwatch.ElapsedMilliseconds;
 
		stopwatch.Reset();
		stopwatch.Start();
		for (var i = 0; i < reps; ++i)
		{
			litJsonJson = JsonMapper.ToJson(obj);
		}
		var litJsonSerializeTime = stopwatch.ElapsedMilliseconds;
 
		stopwatch.Reset();
		stopwatch.Start();
		for (var i = 0; i < reps; ++i)
		{
			jsonDotNetJson = JsonConvert.SerializeObject(obj);
		}
		var jsonDotNetSerializeTime = stopwatch.ElapsedMilliseconds;
 
		stopwatch.Reset();
		stopwatch.Start();
		for (var i = 0; i < reps; ++i)
		{
			unityObj = JsonUtility.FromJson<T>(unityJson);
		}
		var unityDeserializeTime = stopwatch.ElapsedMilliseconds;
 
		stopwatch.Reset();
		stopwatch.Start();
		for (var i = 0; i < reps; ++i)
		{
			litJsonObj = JsonMapper.ToObject<T>(litJsonJson);
		}
		var litJsonDeserializeTime = stopwatch.ElapsedMilliseconds;
 
		stopwatch.Reset();
		stopwatch.Start();
		for (var i = 0; i < reps; ++i)
		{
			jsonDotNetObj = JsonConvert.DeserializeObject<T>(jsonDotNetJson);
		}
		var jsonDotNetDeserializeTime = stopwatch.ElapsedMilliseconds;
 
		var unitySize = Encoding.UTF8.GetBytes(unityJson).Length;
		var litJsonSize = Encoding.UTF8.GetBytes(litJsonJson).Length;
		var jsonDotNetSize = Encoding.UTF8.GetBytes(jsonDotNetJson).Length;
		var msg = string.Format(
			"{0} ({1} reps)\n" +
			"Library,Size,SerializeTime,Deserialize Time\n" +
			"Unity,{2},{3},{4}\n" +
			"LitJSON,{5},{6},{7}\n" +
			"Json.NET,{8},{9},{10}\n\n",
			label, reps,
			unitySize, unitySerializeTime, unityDeserializeTime,
			litJsonSize, litJsonSerializeTime, litJsonDeserializeTime,
			jsonDotNetSize, jsonDotNetSerializeTime, jsonDotNetDeserializeTime
		);
		report += msg;
	}
 
	void OnGUI()
	{
		GUI.TextArea(new Rect(0, 0, Screen.width, Screen.height), report);
	}
}

If you want to try out the test yourself, simply paste the above code into a TestScript.cs file in your Unity project’s Assets directory and attach it to the main camera game object in a new, empty project. Then build in non-development mode for 64-bit processors and run it windowed at 640×480 with fastest graphics. I ran it that way on this machine:

• 2.3 Ghz Intel Core i7-3615QM
• Mac OS X 10.11.2
• Unity 5.3.0f4, Mac OS X Standalone, x86_64, non-development
• 640×480, Fastest, Windowed

And here are the results I got:

Empty Class (100000 reps)

Small String (10000 reps)

Large String (100 reps)

Small Array (100000 reps)

Large Array (100 reps)

Shallow Nested Object (10000 reps)

Deep Nested Object (10000 reps)

These results are much more varied than last week, except for JSON size which never differed. Let’s take a look at them one-by-one. First up is the “empty class” test to see how each serializer handles the most trivial case. For serialization, Unity was slightly behind LitJSON but both beat Json.NET by a mile. For deserialization, Unity was way faster than either third-party library.

In the “small string” test, a single string field contains just the string “hey”. These results directly mirrored the “empty” class test with Unity slightly behind LitJSON at serialization but much faster than Json.NET. Again with deserialization, Unity was much quicker than the other serializers.

The “large string” test uses the same class as the “small string” test but with a string consisting of 100,000 asterisks (*). In both serialization and deserialization, Json.NET actually beat the others but wasn’t a lot quicker than Unity. LitJSON brought up the rear, especially in deserialization where it was far behind the others.

The “small array” test is like the string tests, except with an array instead of a string. The array is just [1,2,3] to mirror the three-character “hey” string. The results, however, didn’t mirror the “small string” test. Unity was fastest at both serialization and deserialization. LitJSON was faster than Json.NET at serialization, but slower at deserialization.

The “large array” test again uses the same class as the “small array” test, but with 1000 elements. Unity was again the fastest—by a lot—but here second place goes to Json.NET in both serialization and deserialization. LitJSON is solidly in last place for this test.

The “shallow nested object” test uses a series of classes, each containing the next. A contains a B which in turn contains a C and so on until I contains just an int. The “shallow” version just defines A, B, and C, but leaves the D as null. For serialization, LitJSON was quickest followed by Json.NET and then Unity in last place. For deserialization, Json.NET was again quickest, beating out Unity by a little and LitJSON by a lot.

The “deep nested object” test uses the same classes, but defines all the way through I and its int Val. Performance matches the “shallow” version closely. LitJSON is quickest at serialization with a tie between Unity and Json.NET for second place. Deserialization has Json.NET as fastest and then Unity and, way behind, LitJSON.

If you give three points for each first place win, two points for second place, and one point for third place, the final tally is:

Overall, LitJSON edges out Unity as the fastest serializer but Unity is far ahead as the fastest deserializer. Overall, Unity wins with 32 points over 25 for both LitJSON and Json.NET.

These aggregate scores may not apply to your particular situation though. For example, if your app makes heavy usage of large strings, Json.NET might be a better choice than Unity. You might also want the greater flexility offered by LitJSON and Json.NET compared to Unity’s simple JsonUtility class and its minimalist three functions. You might not also be able to live with the limitations of Unity’s serializer. For example, you can’t nest deeper than the I class without Unity throwing error messages and cutting off your nesting. You also can’t have arrays that are 10000 long, for example.

Which JSON serializer suits you best? Let me know what you’ve found and what you’ve chosen in the comments!