Jump to content



Welcome to AstaHost - Dear Guest , Please Register here to get Your own website. - Ask a Question / Express Opinion / Reply w/o Sign-Up!

Toggle shoutbox Shoutbox Open the Shoutbox in a popup

@  yordan : (14 April 2014 - 05:28 PM) By The Way, This Could Be An Interesting Subject For A Topic, What About Posting This Question? Let's See If Other People Have The Same Feeling Concerning Bootlists!
@  yordan : (13 April 2014 - 09:36 AM) Boot Order : Cd, [Usb,] Hard Drive :D
@  yordan : (11 April 2014 - 07:23 PM) I Simply Let The Bios Do That
@  Ritesh : (11 April 2014 - 10:23 AM) Is It Possible To Launch Fedora Live Cd Or Installation Disk From Hard Drive On Windows Platform Using Grub Mbr File.
@  Ritesh : (11 April 2014 - 10:21 AM) No U Are Not.. Btw.. I Have Question For You.
@  yordan : (10 April 2014 - 08:02 AM) You Are Partially Right.
I Was Not.
Nevertheless, I Am Again :)
@  Ritesh : (09 April 2014 - 07:33 PM) :P
@  Ritesh : (09 April 2014 - 07:33 PM) I Think U R Not..
@  yordan : (09 April 2014 - 09:28 AM) I'm The Master Of The Shoutbox!
@  yordan : (05 April 2014 - 10:32 PM) He-He
@  Ritesh : (04 April 2014 - 06:59 PM) Ha Ha Ha ....
@  yordan : (04 April 2014 - 11:15 AM) Welcome Back, Starscream!
@  yordan : (03 April 2014 - 02:31 PM) And I Hope That He Will Come Back Soon :)
@  yordan : (01 April 2014 - 02:53 PM) Nice, Ritesh Came, I'm Not Home Alone Today.
@  Ritesh : (01 April 2014 - 08:51 AM) Oh!!! Poor Dear Yordan..
@  yordan : (31 March 2014 - 10:02 AM) I'm A Poor Lonesome Cow-Boy
@  yordan : (27 March 2014 - 02:22 PM) He Is Unpatient Due To His Patients!
@  Ritesh : (27 March 2014 - 10:46 AM) :(
@  Ritesh : (27 March 2014 - 10:46 AM) He Is Busy With His Patients.
@  yordan : (26 March 2014 - 08:12 PM) Ahsani, Where Are You?

Photo
- - - - -

Bash Shell Scripting: Comparing Directories


13 replies to this topic

#1 xboxrulz

xboxrulz

    Colonel Panic

  • Members
  • 3,057 posts
  • Gender:Male
  • Location:Toronto, Ontario, Canada
  • myCENTs:59.15

Posted 26 October 2008 - 12:09 AM

Hey guys,

I've been tasked to create a bash shell script to compare directories and only list out the differences between the two directory.

Does anyone have any ideas or pointers of how to go about this problem?

Many thanks,
xboxrulz

#2 Vyoma

Vyoma

    Cosmic Overlord

  • Members
  • 571 posts
  • Gender:Male
  • Location:Denver, Colorado, US
  • Interests:Gaming, RPG, Guitar, Bikes (riding, specifically riding bullet), Sketching, Painting
  • myCENTs:45.66

Posted 26 October 2008 - 01:59 AM

Find out the difference between the file listing in two directories? Or it should even list out the difference between each files in the two directories?

If it is the difference between the file listing, it can be done easily than the other requirement. Here is what I can think top of my mind:

1. Ensure you are not in the directories that needs to be compared

cd [path-to-some-temporary-location]

2. Get the single line file listing of both directories into temporary files

dir -1 [path/first-directory] > dirlist1
dir -1 [path./second-directory] > dirlist2

3. Do a compare between these two:

diff dirlist1 dirlist2 > difference.txt

If you want th differences to be printed on scree - ofcourse, remove the redirection to difference.txt

4. Clean up! ;)

rm dirlist1 dirlist2

The difference.txt or the output (if you removed redirection) would list the files which are in the first directory and not in second, and vise versa. Put all these commands in a .bsh with a shebhang at top, change permissions to let appropriate people execute the script, and you are good to go.

Now, ofcourse, there might be possibility of stuffing all these into one command using pipes and tees (I am lazy to figure out exactly how it can be done).

Edited by Vyoma, 26 October 2008 - 02:01 AM.


#3 yordan

yordan

    Way Out Of Control - You need a life :)

  • [MODERATOR]
  • 4,745 posts

Posted 26 October 2008 - 11:22 AM

First point.
For honesty purposes, just tell me something. Is this something you want to do for a friend, or is this a homework your teacher asked you to perform ?
Because if it's a homework, your teacher "at order 0" wants you to think by yourself, an you have to do the job quite alone.
Of course, if your teacher is smart, "at order 1" he wants to see if you are able to do some googling.
And "at order 2" he wants to know if you have some help on forums.
"At order 3" the guys on the forum can do mistakes and you must fix my bugs.

Second point. Here we are at in the "open" world. That means that if you ask two people the same question, you will have two different answers. And both of them well be righ.
My first idea was exactly what vyoma says, two lists and a "diff" to see the difference. seeing that, my second thought is "OK, I will do it differently.
Here is my script, I wold like you to test it.
# Let us suppose that we want to see if the files in folder1 are also in folder2.
cd folder1
for i in `ls`
do
if [ if ../folder2/$i]
then
#do nothing because I want to know the non-existing files
present=1
else
echo "folder2/"$i file is missing
ls -l $i >> /tmp/size_of_the_impacted_files
fi
# end of job
# This shell_script is copywrited, only Yordan friends may use it

And remember, using my shell-scripts costs 15% of the money you earn with my scripts.
Regards
Yordan

#4 Vyoma

Vyoma

    Cosmic Overlord

  • Members
  • 571 posts
  • Gender:Male
  • Location:Denver, Colorado, US
  • Interests:Gaming, RPG, Guitar, Bikes (riding, specifically riding bullet), Sketching, Painting
  • myCENTs:45.66

Posted 26 October 2008 - 03:38 PM

Based on what xboxrules actually wants, that is, if his answer to my second question was yes:

Or it should even list out the difference between each files in the two directories?


Then I think yordan's solution is more elegant. All it needs next, are two things:
1. List of files in folder2 and not in folder1 - implementation will be exactly same as what yordan wrote
2. Find out the difference bettween files which are present in both folders - To implement this, in one of the loops, when you deduce that the files are present in both directories, run a 'diff folder1/$i folder2/$i'

#5 xboxrulz

xboxrulz

    Colonel Panic

  • Members
  • 3,057 posts
  • Gender:Male
  • Location:Toronto, Ontario, Canada
  • myCENTs:59.15

Posted 28 October 2008 - 12:59 AM

Find out the difference between the file listing in two directories? Or it should even list out the difference between each files in the two directories?

If it is the difference between the file listing, it can be done easily than the other requirement. Here is what I can think top of my mind:

1. Ensure you are not in the directories that needs to be compared

cd [path-to-some-temporary-location]

2. Get the single line file listing of both directories into temporary files

dir -1 [path/first-directory] > dirlist1
dir -1 [path./second-directory] > dirlist2

3. Do a compare between these two:

diff dirlist1 dirlist2 > difference.txt

If you want th differences to be printed on scree - ofcourse, remove the redirection to difference.txt

4. Clean up! ;)

rm dirlist1 dirlist2

The difference.txt or the output (if you removed redirection) would list the files which are in the first directory and not in second, and vise versa. Put all these commands in a .bsh with a shebhang at top, change permissions to let appropriate people execute the script, and you are good to go.

Now, ofcourse, there might be possibility of stuffing all these into one command using pipes and tees (I am lazy to figure out exactly how it can be done).


Does the $* variable allow the the user to enter "/home/user/path" as one variable or will it split it to home, user and path separately per $. Like home being $1, user being $2 and path being $3?

First point.
For honesty purposes, just tell me something. Is this something you want to do for a friend, or is this a homework your teacher asked you to perform ?
Because if it's a homework, your teacher "at order 0" wants you to think by yourself, an you have to do the job quite alone.
Of course, if your teacher is smart, "at order 1" he wants to see if you are able to do some googling.
And "at order 2" he wants to know if you have some help on forums.
"At order 3" the guys on the forum can do mistakes and you must fix my bugs.

Second point. Here we are at in the "open" world. That means that if you ask two people the same question, you will have two different answers. And both of them well be righ.
My first idea was exactly what vyoma says, two lists and a "diff" to see the difference. seeing that, my second thought is "OK, I will do it differently.
Here is my script, I wold like you to test it.
# Let us suppose that we want to see if the files in folder1 are also in folder2.
cd folder1
for i in `ls`
do
if [ if ../folder2/$i]
then
#do nothing because I want to know the non-existing files
present=1
else
echo "folder2/"$i file is missing
ls -l $i >> /tmp/size_of_the_impacted_files
fi
# end of job
# This shell_script is copywrited, only Yordan friends may use it

And remember, using my shell-scripts costs 15% of the money you earn with my scripts.
Regards
Yordan


It is for homework, thing is I didn't quite get how to approach this. Obviously I'm not going to copy code, that's just plain wrong. I just want ideas and adapt them.

lol, and i don't think anyone is gonna really make any money out of using a directory comparison script, but i never know lol.

Based on what xboxrules actually wants, that is, if his answer to my second question was yes:
Then I think yordan's solution is more elegant. All it needs next, are two things:
1. List of files in folder2 and not in folder1 - implementation will be exactly same as what yordan wrote
2. Find out the difference bettween files which are present in both folders - To implement this, in one of the loops, when you deduce that the files are present in both directories, run a 'diff folder1/$i folder2/$i'


It only lists out the files that are for example in folder 1 but not 2 and vice versa but the list view must be like what ls -l would list so dircmp won't cut it unfortunately...

Like this:

-rw-r--r-- 1 xboxrulz staff 16 Oct 27 23:57 differences.txt
-rw-r--r-- 1 xboxrulz staff 19 Oct 27 23:57 dirlist1
-rw-r--r-- 1 xboxrulz staff 9 Oct 27 23:57 dirlist2
drwxr-xr-x 2 xboxrulz staff 152 Oct 27 21:16 folder1
drwxr-xr-x 2 xboxrulz staff 110 Oct 27 21:17 folder2

... but showing the different files.

Thanks for the pointers guys, that just gave me some clue to this confusing work.

xboxrulz

{EDIT} P.S: I think Vyoma's idea is easier to code, let me try that out first ...

#6 Vyoma

Vyoma

    Cosmic Overlord

  • Members
  • 571 posts
  • Gender:Male
  • Location:Denver, Colorado, US
  • Interests:Gaming, RPG, Guitar, Bikes (riding, specifically riding bullet), Sketching, Painting
  • myCENTs:45.66

Posted 28 October 2008 - 05:44 AM

Does the $* variable allow the the user to enter "/home/user/path" as one variable or will it split it to home, user and path separately per $. Like home being $1, user being $2 and path being $3?
...

I am not sure about $* - will need to look it up. But basically, $1, $2 etc refer to argument list passed to the shell script.
So, if you pass:
> script.sh "/home/user/path"
Then, when you say $1, it will be equal to "/home/user/path" - it will not be split up.

...
It only lists out the files that are for example in folder 1 but not 2 and vice versa but the list view must be like what ls -l would list so dircmp won't cut it unfortunately...
...

Yes, what yordan gave would only list files that are in folder 1 but not in folder 2, and implementing the other would give you the other set too - correct.
(Is there a dircmp command? Not sure there is something like that). So, if you want the listing to be something like what 'ls -l' would give, just use the result of yordan's solution to get it. ;)

Instead of what the code (yordan's)
...
else
echo "folder2/"$i file is missing
ls -l $i >> /tmp/size_of_the_impacted_files
fi
# end of job
...
write it as
...
else
ls -l $i | tail -1
fi
# end of job
...
The "ls -l $i | tail -1" will give each of the missing file in the "ls -l" form. (It is a bit tricky here- you might want to prefix $i with the folder path it is present in).

...
{EDIT} P.S: I think Vyoma's idea is easier to code, let me try that out first ...

Just try out a code and see if it works - if it does not, post it and we can help. ;)

#7 yordan

yordan

    Way Out Of Control - You need a life :)

  • [MODERATOR]
  • 4,745 posts

Posted 28 October 2008 - 11:31 AM

By the way, yes, $1 works.
However, your script should first check that $1 is not empty and $2 is not empty.
And the, in my script you replace the word "folder1" by $1 and the word "folder2" by $2
$* should work, it is supposed to pass $1, $2, $2 and $4.
But then your script must manage that, and say something like "one=$1; shift; two=$2; three=$3; if [ -Z $3 ] " etc...

#8 docduke

docduke

    Advanced Member

  • Members
  • 152 posts

Posted 28 October 2008 - 07:58 PM

I'm not sure whether I'm one of yordan's friends, but I am definitely someone who needs more exposure to scripts, so I tried using it. The result was bash: [: missing `]'

I visited Advanced Bash-Scripting Guide and the man page for bash, to remind me how to construct an if-test. It reminded me of enough that I thought I could fix it. I tested it, and bash no longer complains. [Gee, the forum just cleaned up my alternative term for "complains": b**ches. I had no idea it did that! ;)]

In one short line, a running script is

for i in `ls'; do if [ -e ../u/$i ]; then echo "first"; else echo "second'; fi done

The things that needed to be fixed were (1) second if -> -e (or -a), (2) blank before the closing bracket (]) and (3) "done" at the end of the script.

Thanks for the exercise -- it's been a long time since I've used all those script elements! ;)

#9 xboxrulz

xboxrulz

    Colonel Panic

  • Members
  • 3,057 posts
  • Gender:Male
  • Location:Toronto, Ontario, Canada
  • myCENTs:59.15

Posted 28 October 2008 - 07:59 PM

Ok, I had to scrap it since my professor highly discouraged using diff because it's too specific and too sensitive to differences.

He told me to use
for file in $(ls -a $1)
if (! -d $1/$file)
to check the differences of the directories

So I ended up with scripting this.
#This code is GPL v3
for file in $(ls -a $1)
   do
		if [ ! -e $2/$file ]
			then
				ls -ld $1/$file
		fi
   done
   for file in $(ls -a $2) 
   do
		if [ ! -e $1/$file ]
			then
				ls -ld $2/$file
		fi
   done
and this script works flawlessly.

Thanks for the pointers guys, really helped,
xboxrulz

#10 yordan

yordan

    Way Out Of Control - You need a life :)

  • [MODERATOR]
  • 4,745 posts

Posted 28 October 2008 - 08:55 PM

for i in `ls'; do if [ -e ../u/$i ]; then echo "first"; else echo "second'; fi done

Just kidding, but two remarks.
Putting everything in a single line seems nicer, but it's far less easy to read it, and find an error if there happen to be one.
Instead, writing
if [ something ]
then
do the first step
else
do the second thing
fi

The second way is more readable.
And, of course, it's easier to see the error.
I see for i in `ls'; do
it should be for i in `ls`
and of course bash, exactly as ksh , sh and cs, hates reading standard quotes (') instead of back quotes (`) ;)

#11 docduke

docduke

    Advanced Member

  • Members
  • 152 posts

Posted 29 October 2008 - 01:26 AM

Just kidding, but two remarks.
Putting everything in a single line seems nicer, but it's far less easy to read it, and find an error if there happen to be one.


Agreed. But I'm lazy, and I was typing your script directly into a console. When bash complained, <up-arrow> got me that one-line version complete with semicolons, which I could easily edit to identify and correct the bugs.

Then, when I had running code, I was lazy again, and just copied the running code and pasted it into my post. That way, I could be sure I wouldn't break anything by "pretty-printing" the code.

Anyway, this thread is doing what it is supposed to -- teaching each of us little bits of useful information.

I am now scratching my head over xboxrulz's final version, with for file in $(ls -a $1). I am trying to figure out how that is interpreted, and will have to try it.

Very educational thread. Thanks guys! ;)

#12 xboxrulz

xboxrulz

    Colonel Panic

  • Members
  • 3,057 posts
  • Gender:Male
  • Location:Toronto, Ontario, Canada
  • myCENTs:59.15

Posted 29 October 2008 - 09:09 PM

Docduke,

for file in $(ls -a $1) basically tells the script to grab all (-a) the file/folder entries including all hidden folders in the folder $1 (which is a variable argument).

xboxrulz

#13 Guest_Den_*

Guest_Den_*
  • Guests

Posted 15 June 2012 - 04:52 AM

Maybe this is a new function, but now you can compare directories simply whit

diff /direone/ /diretwo/

it compares the files in each on and the contents.

#14 yordan

yordan

    Way Out Of Control - You need a life :)

  • [MODERATOR]
  • 4,745 posts

Posted 15 June 2012 - 08:10 AM

Maybe this is a new function, but now you can compare directories simply whit

diff /direone/ /diretwo/

it compares the files in each on and the contents.

Quite true.
Unfortunately, the teacher asked the student not to use "diff", in order to learn how to manage recursive commands and how to pass variables to scripts.



Reply to this topic



  


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users