4 replies to this topic

[wisestone]

I have been working on a strange problem I've been having. I am reading
a series of large files (50 mb or so) in one at a time with:
@lines = <FILE>;
or (same behavior with each)
while(<FILE>){
push(@lines, $_);
}
The first time I read a file it will read into the array in about 2
seconds. The second time I try to read a file in (the same size) it
takes about 20 seconds. Everything is declared locally inside the loop
so, everything is leaving scope. I am not sure why it is taking so much
longer the second time. I need to read the entire file in because I am doing a series of operations on the lines that is not known at compile time.

I have narrowed the problem down to a few different areas:

1. It seems that if I read the file into a large scaler by $/ = undef,
the file gets read faster. So, I assume the slow down is taking place
inside the spliting of the lines.

2. If I try to append to one large array, rather then rewritting to a
different array, the slow down does not occur. So it seems Perl has a
hard time with the memory it already has but its fine with memory it
just took from the system?

3. The problem does not seem to happen in Linux, but I'm working
Windows.

Any suggestions for a workaround? Has anyone else seen this? Thanks in
advance.

Ben

[vizskywalker]

Could you be more specific? Is this multiple file loads in the same script? Also, how much RAM do you have? I have an idea as to why this may be occuring but I need those pieces of information to help.

~Viz

[wisestone]

The file loads are in the same script and I have 2 gb of ram.

[vizskywalker]

Could you either post the script here so I can a) test it myself, and check it out, or PM it to me if it is too large to post (more than maybe 50 or so lines tops). I promise to try and help to the best of my ability.

~Viz

[wisestone]

Its seriously just reeding the file in, try:

for(1..2)
{
my @lines;
open(FILE, file1.txt);
while(<FILE>)
{
push(@lines, $_);
}
}

It doesn't have to be opening the same file again but I used that just to make it easier.