Looping: Loop through a directory

for i in $(ls);
do
    process $i
done

for i in `ls`;
do
   process $i
done

Loop to find the total size of a list of files
total=0
for size in `ll | gawk '{print $5}'`;
do
let total=$total+$size;
done
echo $total

Simple printf, awk and sed

Replace a char in a string 

echo $SN | sed 's/-/_/g'

Extract a column

 [root@mail slaves]# ll -tr | grep -v total
-rw-------  1 root  root  18196 Mar  1 17:22 example.com.hosts
-rw-r--r--  1 root  root    172 Mar  6 16:55 example.com.hosts.bak
-rw-------  1 named named   358 Mar 11 08:34 sec.example.com
[root@mail slaves]# ll -tr | grep -v total | awk -F" " '{print $9}'
example.com.hosts
example.com.hosts.bak
sec.example.com

Replace a char every token

[root@mail filelist]# ls
f_1363024548  f_1363024566  f_1363024569  f_1363024572
f_1363024564  f_1363024567  f_1363024570  f_1363024573
f_1363024565  f_1363024568  f_1363024571

[root@mail filelist]# for f in `ls`; do
> echo $f | sed 's/f/a/g'
> done
a_1363024548
a_1363024564
a_1363024565
a_1363024566
a_1363024567
a_1363024568
a_1363024569
a_1363024570
a_1363024571
a_1363024572
a_1363024573

Replace char(s) in a file

[root@mail filelist]# cat file.txt | sed 's/ffff/1/g'
abcd1abcd1
efgh1eeef

Print strings in specified format

[root@mail filelist]# printf "%s %s\n" "abc" "def"
abc def

Replace a space in a string to a return "\n"

[root@localhost ~]# str="abc def ijk"
[root@localhost ~]# echo $str
abc def ijk

[root@localhost ~]# echo $str | sed 's/\s/\n/g'
abc
def
ijk

Extract Token from a string

note that the regex is using greedy match, if we use gender=\(.*\), it will map till the place before school="ics"

e.g str="name=nana gender=girl age=5 school=ics"
sed -n 's/.*[[:blank:]]gender=\([a-z]*\)[[:blank:]].*/\1/p'  $str

Looping: Padding zero

Padding Zero

for i in {0..1}{0..9}; do echo $i; done

for i in 000{0..1}{0..9}; do echo $i; done

for i in {0..1}{0..9}; do echo "000$i"; done

touch {0..1}{0..9}

File descriptor

List the file descriptor id for the known processes

ls /proc/<procid>/fd/ -l

How to cause CPU load 100%

Recursively generates many processes cause CPU 100%
:(){
    :|:&
}; :

Sorting and Grep

Read a big file line by line and do some grep

cat files.txt | xargs grep keyword

Appending a list of files to one file using xargs command

cat filelist | xargs cat >> bigfile.dat

Appending the content of a list of files to one file using xargs command
cat /tmp/filelist.txt | xargs cat |grep " ERROR] Failed path" >> /tmp/ed_search.err

Cut extract multiple columns
e.g Get the first column

/etc/passwd
root:x:0:0:root:/root:/bin/bash

cut -d":" -f1 /etc/passwd
root

Find out and sort the folder according to size

du --max-depth=1 /home/ | sort -nr

To sort the file on the third field (serial number) in reverse order and save the results in sorted.data, use this command:

sort -r +2 -3 company.data > sorted.data
Mel Ancholie 636496 Research
Jan Itorre 406378 Sales
Ed Jucacion 396082 Sales
Jim Nasium 031762 Marketing


Sort by field 5 numerically
$ ls -al | sort -n -k5



Cut out column 9 where the fields are separated by one or more spaces

[root@localhost locallog]# ll -r ed_search.*.log | awk -F" " '{print "/path/"$9}' >> /tmp/filelist.txt
[root@localhost locallog]# cat /tmp/filelist.txt
/path/ed_search.8.log
/path/ed_search.7.log
/path/ed_search.6.log
/path/ed_search.5.log
/path/ed_search.4.log
/path/ed_search.3.log
/path/ed_search.2.log
/path/ed_search.1.log



Eliminlate duplicate lines, sort before uniq
sort /tmp/ed_search_err_list.txt | uniq | wc -l

grep line below and above
grep -A 1 word file.txt --> Line and one line after
grep -B 1 word file.txt --> Line and one line before
grep -C 1 word file.txt --> One line before and after


grep "string" file , how to get the line number also?

grep -n "string" file

Range of characters from a string

The first parameter after the variable describes the starting position from zero inclusively.
The second parameter describes the number of character from the starting position.
[root@localhost scripts]# d=12345678
[root@localhost scripts]# echo ${d:3:2}
45
[root@localhost scripts]# echo ${d:0:2}
12

Return the number of character in a string

[root@localhost scripts]# d=12345678
[root@localhost scripts]# echo ${#d}
8

Misc usage of find

Search for file with a specific name in a set of files (-name)
-print wil print out the path of any file
find . -name "rc.conf" -print

How to apply a unix command to a set of file (-exec).
find . -name "rc.conf" -exec chmod o+r '{}' \;


Remove files by mtime, dampend by sleep to avoid impact
x=800
while [ $x -ge 15 ]
do
 echo "removing $x days old data"
find /var/FortiGuard/remove -mtime +$x -print |xargs rm -rf
 sleep 300
 x=$(( $x - 10 ))
done

How to search for a string in a selection of files (-exec grep ...).
find . -exec grep "www.athabasca" '{}' \; -print
This command will search in the current directory and all sub directories. All files that contain the string will have their path printed to standard output.

If you want to just find each file then pass it on for processing use the -q grep option. This finds the first occurrance of the search string. It then signals success to find and find continues searching for more files.

find . -exec grep -q "www.athabasca" '{}' \; -print
This command is very important for process a series of files that contain a specific string. You can then process each file appropriately. An example is find all html files with the string "www.athabascau.ca". You can then process the files with a sed script to change those occurrances of "www.athabascau.ca" with "intra.athabascau.ca".

for size in `find . -type f -newer 1239832944:7 -exec ls -ltr '{}' \; | cut -d ' ' -f6`; do let total=$total+$size; done

//touch a wosit with a particular modified date
touch -t 201205120000 /tmp/wosit
//find a file having a newer modified date than /tmp/wosit
find ./ -newer /tmp/wosit
find -type f -size +5000k -exec ls -lh {} \; <-- if 2 para, {} {}