2 replies to this topic

[Jarwulf]

Hi, I'm getting a crash in Java. Basically a txt file circa 150mb is being processed

fileHandler.shrinkFile();

goes into

public void shrinkFile() throws FileNotFoundException, IOException, FileFormatException{
		BufferedWriter outputTempFileWriter = null;
		BufferedReader inputFileReader = null;
	  
		  
			for(int i=0;i<choseFilesVector.size();i++){
				String inputFile = choseFilesVector.get(i).toString();  
				inputFileReader = new BufferedReader(new FileReader( inputFile ), readBufferSize);
			  
				txtArea.append("  Processing "+inputFile+"\n");
				txtArea.setCaretPosition(txtArea.getText().length());
							  
				//decide format
				String strLine = null;
				String format = null;
				do{

//CRASH HAPPENS HERE
					strLine = inputFileReader.readLine();	  
//CRASH HAPPENS HERE
			  
					if(strLine!=null && (strLine.split("\t").length>=4)){  
						String [] info = strLine.split("\t");
						if((info[0].trim().toLowerCase().matches("chr(x|y|[0-9]+)"))&&(info[1].trim().matches("[0-9]+"))&&(info[2].trim().matches("[0-9]+"))){
							//data of tagAlign
							format = "tagAlign";

It gets into the above loop and then simply crashes on line

strLine = inputFileReader.readLine();

the first time through giving me the error

Exception in thread "Thread-4" java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Unknown Source)
	at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)

etc etc. Maybe the file I'm processing is too big but the program has happened gigantic files before the only difference this time is that the files being processed were autoconverted from a different format.

[Bubsy]

Actually its not because the file size, but I bet this new file has a gigantic line on it.

The readLine() method will try and allocate the full line as a string in memory. If the line of text gets to big, you'll eventually run out of heap.

There's two courses of action that you can take :

• Raise the amount of memory avaliable to the JVM (It'll work, but eventually you may run on OutOfMemory erros again)
• Refactor the code without relying on the readLine() method. (Instead of reading the full string and spliting it on memory, you should create a FSM and parse the file on a more "streaming" fashion)

[Jarwulf]

Thanks, I added a newline when file generating and it seems to have worked.