Index
 |
Strange Perl Behavior | ||
Discussion by wisestone with 5 Replies.
Last Update: July 15, 2005, 12:21 pm | |||
 |
|||
I have been working on a strange problem I've been having. I am reading
a series of large files (50 mb or so) in one at a time with:
@lines = <FILE>;
or (same behavior with each)
while(<FILE>){
push(@lines, $_);
}
The first time I read a file it will read into the array in about 2
seconds. The second time I try to read a file in (the same size) it
takes about 20 seconds. Everything is declared locally inside the loop
so, everything is leaving scope. I am not sure why it is taking so much
longer the second time. I need to read the entire file in because I am doing a series of operations on the lines that is not known at compile time.
I have narrowed the problem down to a few different areas:
1. It seems that if I read the file into a large scaler by $/ = undef,
the file gets read faster. So, I assume the slow down is taking place
inside the spliting of the lines.
2. If I try to append to one large array, rather then rewritting to a
different array, the slow down does not occur. So it seems Perl has a
hard time with the memory it already has but its fine with memory it
just took from the system?
3. The problem does not seem to happen in Linux, but I'm working
Windows.
Any suggestions for a workaround? Has anyone else seen this? Thanks in
advance.
Ben
a series of large files (50 mb or so) in one at a time with:
@lines = <FILE>;
or (same behavior with each)
while(<FILE>){
push(@lines, $_);
}
The first time I read a file it will read into the array in about 2
seconds. The second time I try to read a file in (the same size) it
takes about 20 seconds. Everything is declared locally inside the loop
so, everything is leaving scope. I am not sure why it is taking so much
longer the second time. I need to read the entire file in because I am doing a series of operations on the lines that is not known at compile time.
I have narrowed the problem down to a few different areas:
1. It seems that if I read the file into a large scaler by $/ = undef,
the file gets read faster. So, I assume the slow down is taking place
inside the spliting of the lines.
2. If I try to append to one large array, rather then rewritting to a
different array, the slow down does not occur. So it seems Perl has a
hard time with the memory it already has but its fine with memory it
just took from the system?
3. The problem does not seem to happen in Linux, but I'm working
Windows.
Any suggestions for a workaround? Has anyone else seen this? Thanks in
advance.
Ben
Tue Jul 12, 2005 Reply New Discussion
Could you be more specific? Is this multiple file loads in the same script? Also, how much RAM do you have? I have an idea as to why this may be occuring but I need those pieces of information to help.
~Viz
~Viz
Tue Jul 12, 2005 Reply New Discussion
The file loads are in the same script and I have 2 gb of ram.
Wed Jul 13, 2005 Reply New Discussion
Could you either post the script here so I can a) test it myself, and 
check it out, or PM it to me if it is too large to post (more than maybe 50 or so lines tops). I promise to try and help to the best of my ability.
~Viz
check it out, or PM it to me if it is too large to post (more than maybe 50 or so lines tops). I promise to try and help to the best of my ability.~Viz
Wed Jul 13, 2005 Reply New Discussion
Its seriously just reeding the file in, try:
for(1..2)
{
my @lines;
open(FILE, file1.txt);
while(<FILE>)
{
push(@lines, $_);
}
}
It doesn't have to be opening the same file again but I used that just to make it easier.
for(1..2)
{
my @lines;
open(FILE, file1.txt);
while(<FILE>)
{
push(@lines, $_);
}
}
It doesn't have to be opening the same file again but I used that just to make it easier.
Fri Jul 15, 2005 Reply New Discussion
 |
 |
|
 |
 |
| Reply / Comment | Ask a Question? | Share / Bookmark | E-Mail a Friend |
 Password Reminder And Password Change .htaccess I need a script to change password (4)
|
(8) Need To Discuss Ideas On Creating A Members Area A Discussion 
|