If your copying files using cp or rsync or whatever method. Sometimes you want to see the new files or changed files in the past N seconds.

You can do that like so

find /data -ls > /tmp/OLD
# wait a few minutes
find /data -ls /tmp/NEW

Now you can see the difference using “diff” or even “comm”
diff /tmp/OLD /tmp/NEW

Diff will show you the before and after. When looking at different files and what changed, we are just interested in seeing whats new, what changed. We are not interested in the old value (if you are then keep using “diff”). comm can help us isolate only whats new.
comm -13 /tmp/OLD /tmp/NEW

If you look at what comm does, it seperates whats unique to each file into different columns. So for example: all of the data unique to file1 (/tmp/OLD) will go in first column. all of the data unique to file2 (/tmp/NEW) will go in 2nd column. All of the data unique to both files (that appears in both /tmp/OLD and /tmp/NEW) will go in the 3rd column. using -1 (unique to file1),-2 (unique to file2), or -3 (unique to file3), we can suppress different outputs. For example from the above we only want to see whats new, so thats what is unique to /tmp/NEW (what lines only exist in /tmp/NEW in other words). So that means we only want the 2nd column. With comm, we dont tell it what we want, we tell it what we dont want. So we tell it to supress 1 and 3, leaving us with 2 – which is what is unique to file2 (/tmp/NEW).

Help output of comm might help

# comm --help
Usage: comm [OPTION]... FILE1 FILE2
Compare sorted files FILE1 and FILE2 line by line.

With no options, produce three-column output.  Column one contains
lines unique to FILE1, column two contains lines unique to FILE2,
and column three contains lines common to both files.

  -1              suppress column 1 (lines unique to FILE1)
  -2              suppress column 2 (lines unique to FILE2)
  -3              suppress column 3 (lines that appear in both files)

  --check-order     check that the input is correctly sorted, even
                      if all input lines are pairable
  --nocheck-order   do not check that the input is correctly sorted
  --output-delimiter=STR  separate columns with STR
      --help     display this help and exit
      --version  output version information and exit

Note, comparisons honor the rules specified by `LC_COLLATE'.

Examples:
  comm -12 file1 file2  Print only lines present in both file1 and file2.
  comm -3 file1 file2  Print lines in file1 not in file2, and vice versa.

Report comm bugs to bug-coreutils@gnu.org
GNU coreutils home page: <http://www.gnu.org/software/coreutils/>
General help using GNU software: <http://www.gnu.org/gethelp/>
Report comm translation bugs to <http://translationproject.org/team/>
For complete documentation, run: info coreutils 'comm invocation'

TIP: its best to sort the data for both files prior to running diff or comm. comm is more picky on it though.
find /data -ls | sort > /tmp/OLD
# wait a few minutes
find /data -ls | sort >/tmp/NEW
# Now you can see the difference using "diff" or even "comm"
diff /tmp/OLD /tmp/NEW
comm -13 /tmp/OLD /tmp/NEW

Cool thats pretty cool. We can add some tweaks, like making ls output better.
find /data -type f -exec ls -lisa -B1 \; | sort > /tmp/OLD
# wait a few minutes
find /data -type f -exec ls -lisa -B1 \; | sort > /tmp/NEW
# Now you can see the difference using "diff" or even "comm"
diff /tmp/OLD /tmp/NEW
comm -13 /tmp/OLD /tmp/NEW

ls -lisa -B1  shows entire contents of folders, so we dont want to run that on folders, we only run it on files. Either way we are only curious in seeing what files are new and get bigger. folders dont really make a difference unless there is something in them.

Now we can add loop

WATCH LOOP

watch is an interesting program that basically gives you the ability to do an infinite loop that nicely shows you data at the same time. I will use that to its advantage here.

This loops the following: ls all of the files with find and save that to /tmp/NEW, now diff /tmp/NEW and /tmp/OLD (first run of this fails because /tmp/OLD doesnt exist yet), now make a copy of /tmp/NEW to /tmp/OLD (move/rename would work as well), wait a few seconds, repeat… the second time around /tmp/OLD is the old find output, and /tmp/NEW is the new find output, so we can diff & comm them.

# simply fill out the DIR variable, or run as is to see difference in current working directory. these are one line commands, just copy paste them (need only edit the DIR var)

# comm file diff
DIR="."; watch -d -n10 "echo 'Changing files:'; find "${DIR}" -type f -exec ls -lisa -B1 {} \; | sort > /tmp/wNEW; comm -31 /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD;"

# diff file diff
DIR="."; watch -d -n10 "echo 'Changing files:'; find $DIR -type f -exec ls -lisa -B1 {} \; | sort > /tmp/wNEW; diff /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD;"

If ls freaks out you can just use the simpler find.
# simply fill out the DIR variable, or run as is to see difference in current working directory. these are one line commands, just copy paste them (need only edit the DIR var)

# comm file diff
DIR="."; watch -d -n10 "echo 'Changing files:'; find "${DIR}" -ls {} \; | sort > /tmp/wNEW; comm -31 /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD;"

# diff file diff
DIR="."; watch -d -n10 "echo 'Changing files:'; find $DIR -ls {} \; | sort > /tmp/wNEW; diff /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD;"

WHILE LOOP

Not all systems support “watch”. However all shells should have a while loop.

Here is one with the intricate better looking ls output

# comm (pick a line and use it, the second line clears the screen trying to imitate "watch" command, the first line makes the screen scroll)
DIR="."; INTS=10; while true; do echo 'Changing files:'; find "${DIR}" -type f -exec ls -lisa -B1 {} \; | sort > /tmp/wNEW; comm -31 /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD; sleep $INTS; done;
DIR="."; INTS=10; while true; do clear; echo 'Changing files:'; find "${DIR}" -type f -exec ls -lisa -B1 {} \; | sort > /tmp/wNEW; comm -31 /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD; sleep $INTS; done;

# diff (pick a line and use it, the second line clears the screen trying to imitate "watch" command, the first line makes the screen scroll)
DIR="."; INTS=10; while true; do echo 'Changing files:'; find "${DIR}" -type f -exec ls -lisa -B1 {} \; | sort > /tmp/wNEW; diff /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD; sleep $INTS; done;
DIR="."; INTS=10; while true; do clear; echo 'Changing files:'; find "${DIR}" -type f -exec ls -lisa -B1 {} \; | sort > /tmp/wNEW; diff /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD; sleep $INTS; done;

Here is one with the generic find  included ls output
# comm (pick a line and use it, the second line clears the screen trying to imitate "watch" command, the first line makes the screen scroll)
DIR="."; INTS=10; while true; do echo 'Changing files:'; find "${DIR}" -ls | sort > /tmp/wNEW; comm -31 /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD; sleep $INTS; done;
DIR="."; INTS=10; while true; do clear; echo 'Changing files:'; find "${DIR}" -ls | sort > /tmp/wNEW; comm -31 /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD; sleep $INTS; done;

# diff (pick a line and use it, the second line clears the screen trying to imitate "watch" command, the first line makes the screen scroll)
DIR="."; INTS=10; while true; do echo 'Changing files:'; find "${DIR}" -ls | sort > /tmp/wNEW; diff /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD; sleep $INTS; done;
DIR="."; INTS=10; while true; do clear; echo 'Changing files:'; find "${DIR}" -ls | sort > /tmp/wNEW; diff /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD; sleep $INTS; done;

The end

Leave a Reply

Your email address will not be published. Required fields are marked *