How to deal with big strings and limited memory

By : hsmit

I have a file from which I read data. All the text from this file is stored in a String variable (a very big variable). Then in another part of my app I want to walk through this string and extract useful information, step-by-step (parsing the string).

In the meanwhile my memory gets full and an OutOfMemory exception keeps me from further processing. I think it would be better to process the data directly while reading the inputstream from the file. But for organizational aims, I would like to pass the String to another part in my application.

What should I do to keep the memory from overflowing?

By : hsmit


You must review your algorithm for dealing woth large data. You must process chunk-by-chank this data, or use random file access without storing data in memory. For example you can use StringTokenizer or StreamTokenizer as said @Zombies. You can see parser-lexer techniques: when parser parses some expression it asks to lexer to read next lexem(tokens), but doesn't reads whole input stream at once.

Others have suggested reading and processing portions of your file at a single time. If possible, one of those ways would be better.

However, if this is not possible and you are able to load the String initially into memory as you indicate but it is later parsing of this string that creates problems, you may be able to use substrings. In Java a sub-string maps on top of the original char array and just takes memory for the base Object and then the start and length int pointers.

So, when you find a portion of the string that you want to keep separately, use something like:

String piece = largeString.substring(foundStart, foundEnd);

If you instead this or code that internally does this, then the memory use will increase dramatically:

new String(largeString.substring(foundStart, foundEnd));

Note that you must use String.substring() with care for this very reason. You could have a very large string off of which you take a substring and then discard your reference to the original string. The problem is the substring still references the original large char array. The GC will not release that until the substring also is removed. In cases like this, it's useful to actually use new String(...) to ensure the unused large array will be discarded by the GC (this is one of the few cases where you should ever use new String(...)).

Another technique, if you expect to have lots of little strings around and these are likely to have the same values, but come from an external source (like a file), is to use .intern() after creating the new string.

Note: This does depend on the implementation of String which you really shouldn't have to be aware of, but in practice for large applications sometimes you do have to rely on that knowledge. Be aware that future versions of Java may change this (though not likely).

This video can help you solving your question :)
By: admin