How To Fix the XML Memory “Leak”
Dealing with XML files can very easily trigger Flash to “leak” memory. Your app may only keep a tiny fraction of the XML file’s contents, but the whole file may stay in memory and never get garbage collected. Today’s article examines how this happens and how you can clean up all that unused memory.
This particular memory “leak” stems from a peculiar behavior in Flash where strings that are built from other strings do not actually have a copy of the original string’s characters. Instead, they have a “master string” field which references the characters in the original string. In a debug version of Flash Player you can see the contents of the “master string” using the flash.sampler.getMasterString. Consider the following example:
var str:String = "ABCDEFGHIJKLMNOPQRSTUV".substr(0,2); trace(str + " has master string " + flash.sampler.getMasterString(str)); // output: AB has master string ABCDEFGHIJKLMNOPQRSTUV
Even though you’re only using the “AB” string, Flash is holding the whole “ABCDEFGHIJKLMNOPQRSTUV” string in memory since that’s what “AB” was built from. While a few letters of the alphabet is just fine, the problem can get out of control when you’re dealing with huge XML files. What if all you did was save one tiny string from the XML to a variable? As it turns out, the entire XML file will stay in memory in this case. To illustrate, here’s a small example:
var xml:XML = new XML( '<people><person first="John"/><person first="Mary"/></people>' ); xmlJohn = xml.person[0].@first; trace(xmlJohn + " has master string " + flash.sampler.getMasterString(xmlJohn)); // output: John has master string <people><person first="John"/><person first="Mary"/></people>
How about JSON? Does the same problem happen there, too? Luckily, it does not:
var json:Object = JSON.parse( '{"people":[{"first":"John"},{"first":"Mary"}]}' ); jsonJohn = json.people[0].first; trace(jsonJohn + " has master string " + flash.sampler.getMasterString(jsonJohn)); // output: John has master string null
So what do you do if you want to get rid of all that master string baggage from the XML file? To solve that issue I’ve created a tiny static function that modifies the string just enough to convince the Flash Player to dump its master string. Feel free to use this in your own projects:
/** * Replace a string's "master string"-- the string it was built from-- with a single * character to save memory. * @param str String to clean * @return The input string, but with a master string only one character larger than it * @author JacksonDunstan.com/articles/2260 */ public static function cleanMasterString(str:String): String { return ("_"+str).substr(1); }
With this function the master string will be simply “_John”.
Finally, I’ve created a small example application to test strings from XML, JSON, and “cleaned” strings from XML using the above utility function. You’ll need a debug version of Flash Player to run it since it uses flash.sampler.getMasterString
.
Have a better way of cleaning master strings? Run into this problem in your own app? Spot a bug? Post a comment!
#1 by Peter Henry on June 3rd, 2013 ·
I assume you tried the following and it does not work?
return (“”+str).substr(0);
#2 by jackson on June 3rd, 2013 ·
I did, but since adding an empty string doesn’t result in a new string the master string is unchanged too. The same goes with
substr(0)
.#3 by Andrew Start on June 3rd, 2013 ·
Is the XML kept around as the master string even if you dispose of it with System.disposeXML()?
#4 by jackson on June 3rd, 2013 ·
Yes:
I believe this is because the
XML
object is garbage collected but theString
object it was built from is not.#5 by Clark on June 3rd, 2013 ·
Ahh man, an old concept application at work killed me, it used a lot of text/xml, I debugged it forever and we ended up giving up. I bet this is the problem!
Also it is entirely off topic, but have you ever used http://polygonal.github.io/ds/ ? Every few months I celebrate it as I discover something cool but I am not sure what performance is like? Well just thought I would ask. It has been around for many years now and its usefulness is staggering.
#6 by jackson on June 3rd, 2013 ·
I haven’t used it before, but I may do a performance test of it. Thanks for the link!
#7 by Gene on June 3rd, 2013 ·
var xml:XML = new XML(
”
);
var xmlJohn:String = xml.person[0].@first[0];
var map:Object={};
// <– map[xmlJohn]=xmlJohn;
trace(xmlJohn + " has master string " + flash.sampler.getMasterString(xmlJohn));
try it .. :)
#8 by jackson on June 3rd, 2013 ·
Very interesting find! I played around with it and it turns out that you can also use a
Dictionary
and you don’t need to set the string as the value in the map:This leads to an alternative way of “cleaning” strings compared to the version in the article:
That will completely remove strings’ master string and never grow the
Dictionary
beyond a temporary single entry.Thanks for the tip!
#9 by zeh on February 4th, 2014 ·
Just for completeness, I tried this variant and it seems to still keep the original master string around. The original solution with “_” and substr() still works.
#10 by benjamin guihaire on June 21st, 2013 ·
Great article ! Note that the new Scout version released by Adobe on Monday 17th has Memory Profiling capabilities.
In my experience, the XML memory leak was caused by the extensive use of describeType, and the way to reduce the number of leaks was to use the mx.utils.describeTypeCache , so no new XML is created when requesting a describeType for the same class over and over again. It also has an advantage of making the calls to describeType much faster the second time you request it for a given class.
#11 by jackson on June 21st, 2013 ·
Sounds like a handy utility, but the memory “leak” triggered by the code in the article never uses
describeType
so perhaps we’re talking about two different issues.#12 by Ivo on June 22nd, 2013 ·
From what i understand it appears this is actually an issue with the way substr() is implemented causing pointers to the original string’s location in memory and not letting it be garbage collected. This raises another interesting question, do string.slice() and string.concat() behave in the same manner if the string is not modified or do they actually create copies of the old string.
If so then according to logic it would be enough to simply append something like .slice() after .substr() to ensure a new object is created and the raw input is free for garbage collection.
#13 by jackson on June 22nd, 2013 ·
The
String
class has many functions that may or may not involve master strings. Trying them all out sounds like a good experiment.#14 by skyboy on July 1st, 2013 ·
The native JSON parser creates a new String object from scratch for every thing it parses, but if you were to use a 3rd party library you’d most likely see some different results; I know mine uses substring to process strings more quickly than the native one does where possible, others may as well. It would also be an interesting test to see if after enough memory is allocated the master string is broken up into smaller strings and freed, and what impact that has on performance.
It also would seem that the native XML parser uses substring internally to improve performance, and would be subject to the same potential pitfalls (permanent memory leak / massive performance drops in unrelated code).
#15 by Fabien on August 22nd, 2013 ·
Thank you so much Jackson, this helped a lot!
#16 by Pierre Chamberlain on August 30th, 2013 ·
If you’re going to use the attributes of an XML element as Number, int, or uint anyways, does this memory leak issue still apply? Should we clean the value as a String before casting it to a Number, int or uint?
#17 by jackson on August 30th, 2013 ·
XML is string-based, so you’re always reading
String
values from it. If all you do is convert theString
toint
or something else that has no reference to theString
object, then theString
you got from theXML
object will be unreferenced anywhere and therefore eligible for garbage collection and there won’t be a memory “leak”. The problem comes about when you keep theString
you got fromXML
in some way, such as the names in the article. As shown there, just oneString
is enough to keep the entire XML string around, so just keeping oneString
and converting another millionString
values toint
will still mean that the oneString
is keeping the whole XML string around in memory. If you use the technique in the article to “clean” that oneString
, you’ll be OK.#18 by makc3d on November 6th, 2013 ·
For the sake of completeness: alternativa3d solution.
#19 by Benny on February 20th, 2014 ·
hey,
try this,
slave = slave.substr(0,1)+slave.substr(1);
#20 by jackson on February 21st, 2014 ·
This is a really interesting approach as the resulting string has a master string that is just its first character. I’m not entirely sure what’s going on behind the scenes or which version is better—the article version, this one, or the
Dictionary
version from my comment above—but this probably warrants further investigation. Thanks for the tip!