At some point, every project ends up reading or writing to the file system. If you do anything more than storing a single blob of bytes (e.g. JSON text) then you’ll need to be very careful about performance. It’s easy to accidentally write code that takes way longer to read and write than it should and you won’t get any help from the compiler or from Unity. Today’s article reveals some of these traps so you won’t fall into them!

The .NET API provides us with lots of nice classes for writing to the file system. We have the abstract Stream and its FileStream child, the File class with static functions like Open, and convenient “readers” and “writers” like BinaryReader and BinaryWriter. The C# language itself provides the using block that allows for easy closing of the streams, readers, writers, and file handles. This sort of code is easily written, safely executed, and simple to maintain:

using (var stream = File.Open("/path/to/file", FileMode.OpenOrCreate))
{
	using (var writer = new BinaryWriter(stream, Encoding.UTF8))
	{
		writer.Write("Hello, world!");
		writer.Write(123);
		writer.Write(true);
	}
}

In the end though, it all comes down to five basic operations:

  1. Open file: File.Open
  2. Read bytes: stream.Read
  3. Write bytes: stream.Write
  4. Seek to position: stream.Position or stream.Seek
  5. Close file: stream.Dispose or stream.Close

If you decompile the FileStream class you will see that stream.Position and stream.Seek do the same thing. Only the API differs. Also, stream.Dispose simply closes the file just like stream.Close.

With all of the simplicity that these streams, readers, and writers provide, it’s easy to overlook the performance implications. Each of these operations is just one function call, but the costs of them vary wildly. To put this to the test, I’ve written a little test app to try out each operation. Here’s what’s being tested today:

  • Write 20 MB, one 4 KB “chunk” at a time
  • Write 20 MB, one byte at a time
  • Read 20 MB, one 4 KB “chunk” at a time
  • Read 20 MB, one byte at a time
  • Seek the stream’s Position as many times as chunks read/written
  • Open and close the file as many times as chunks read/written

Here’s the test script:

using System;
using System.IO;
 
using UnityEngine;
 
class TestScript : MonoBehaviour
{
	const string FileName = "TestFile";
	const int FileSize = 1024 * 1024 * 20;
	const int ChunkSize = 1024 * 4;
	const int NumChunks = FileSize / ChunkSize;
 
	static readonly byte[] Chunk = new byte[ChunkSize];
 
	string report = "";
 
	void Start()
	{
		var path = Path.Combine(Application.persistentDataPath, FileName);
		if (File.Exists(path))
		{
			File.Delete(path);
		}
 
		var stopwatch = new System.Diagnostics.Stopwatch();
		long readChunkTime;
		long readByteTime;
		long writeChunkTime;
		long writeByteTime;
		long seekTime;
		long openCloseTime;
		using (var stream = File.Open(path, FileMode.OpenOrCreate))
		{
			stopwatch.Start();
			for (var i = 0; i < NumChunks; ++i)
			{
				stream.Write(Chunk, 0, ChunkSize);
			}
			writeChunkTime = stopwatch.ElapsedMilliseconds;
		}
 
		using (var stream = File.Open(path, FileMode.OpenOrCreate))
		{
			stopwatch.Start();
			for (var i = 0; i < FileSize; ++i)
			{
				stream.WriteByte(0);
			}
			writeByteTime = stopwatch.ElapsedMilliseconds;
		}
 
		using (var stream = File.Open(path, FileMode.OpenOrCreate))
		{
			stopwatch.Reset();
			stopwatch.Start();
			for (var i = 0; i < NumChunks; ++i)
			{
				var numBytesRemain = ChunkSize;
				var offset = 0;
				while (numBytesRemain > 0)
				{
					var read = stream.Read(Chunk, offset, numBytesRemain);
					numBytesRemain -= read;
					offset += read;
				}
			}
			readChunkTime = stopwatch.ElapsedMilliseconds;
		}
 
		using (var stream = File.Open(path, FileMode.OpenOrCreate))
		{
			stopwatch.Reset();
			stopwatch.Start();
			for (var i = 0; i < FileSize; ++i)
			{
				stream.ReadByte();
			}
			readByteTime = stopwatch.ElapsedMilliseconds;
		}
 
		using (var stream = File.Open(path, FileMode.OpenOrCreate))
		{
			stopwatch.Reset();
			stopwatch.Start();
			for (var i = 0; i < NumChunks; ++i)
			{
				stream.Position = stream.Position;
			}
			seekTime = stopwatch.ElapsedMilliseconds;
		}
 
		stopwatch.Reset();
		stopwatch.Start();
		for (var i = 0; i < NumChunks; ++i)
		{
			using (var stream = File.Open(path, FileMode.OpenOrCreate))
			{
			}
		}
		openCloseTime = stopwatch.ElapsedMilliseconds;
 
		File.Delete(path);
 
		report = "Operation,Time\n"
			+ "Write Chunk," + writeChunkTime + "\n"
			+ "Write Byte," + writeByteTime + "\n"
			+ "Read Chunk," + readChunkTime + "\n"
			+ "Read Byte," + readByteTime + "\n"
			+ "Seek," + seekTime + "\n"
			+ "Open+Close," + openCloseTime + "\n";
	}
 
	void OnGUI()
	{
		GUI.TextArea(new Rect(0, 0, Screen.width, Screen.height), report);
	}
}

If you want to try out the test yourself, simply paste the above code into a TestScript.cs file in your Unity project’s Assets directory and attach it to the main camera game object in a new, empty project. Then build in non-development mode for 64-bit processors and run it windowed at 640×480 with fastest graphics. I ran it that way on this machine:

  • 2.3 Ghz Intel Core i7-3615QM
  • Mac OS X 10.11.2
  • Apple SSD SM256E, HFS+ format
  • Unity 5.3.0f4, Mac OS X Standalone, x86_64, non-development
  • 640×480, Fastest, Windowed

And here are the results I got:

Operation Time
Write Chunk 37
Write Byte 1415
Read Chunk 6
Read Byte 1228
Seek 2
Open+Close 243

File I/O Performance Graph (All)

File I/O Performance Graph (Fast)

The results vary so much that it was necessary to break out the “fast” times into a separate chart just to be able to see them!

Reading and writing are tremendously more efficient if done in chunks rather than one byte at a time. While you probably won’t write just one byte, you might write a 4-byte integer or another value that would have almost as small of a chunk size. If you can, avoid this in favor of reading and writing large chunks and you’ll reap a 38x boost in writing and 205x boost in reading!

Seeking (either by setting Position or calling Seek) doesn’t read or write any bytes. It does, however, have a cost. While it’s not as much as any other operation, it does take a third as long as reading by 4 KB chunks. So it’s best to avoid seeking if possible to take advantage of buffering at various levels by reading and writing linearly through the file.

Lastly, opening and closing the file also don’t read or write any bytes but they too take time to do. Quite a lot of time, as it turns out! Opening and closing took 6.5x longer than writing the whole file by chunks and 40x longer than reading the whole file by chunks! That’s a really long time, especially considering that reading and writing are the important part and opening and closing are only requirements of the OS. So make sure that you’re not opening or closing files any more than you have to. You may even want to prefer larger files over many smaller files.

That wraps up today’s tips for file I/O performance. If you have any of your own to share, please leave a comment!