In this post, we will be looking into the Linux filter commands. The filter commands are used to filter required things from the output of any file, which makes it easy to find what actually is required from a multiline file.
Filters are multiple sets of Linux commands which do some manipulation of the text of a file. Use of pipe along with the filter command makes it more powerful to manipulate the data, though we have multiple commands which can we used independently to pick up required text from a file.
In this post, we will be discussing the below commands. These commands are also used heavily in shell scripts.
- less
- more
- head
- tail
- sort
- cut
- sed
- awk
- wc
- grep
- uniq
- paste
- tr
1. less
less command is widely used as a file viewer command. you can open a multiline file using less command and also perform certain operations on them. It allows seeing the output line wise and page wise.
[root@devopsage ~]# less /etc/passwd
This will open /etc/passwd file and with certain options, we can view the file page wise or line by line.
Note:
- Use “Enter Key” to scroll down line by line
- Use “d” to go to the next page
- Use “b” to move cursor to the previous page
- Use “/” to search for a word in the file
- Use “v” to take that file into the VI Editor mode, where you can edit the file and save it.
2. more
more command is similar to the less command, it is also used as a file viewer command, but the only difference is that more indicate at what line of the page you have reached relative to the entire file size. It shows the percentage of file covered, which makes it easy to identify at what location of the file you are.
[root@devopsage ~]# more /etc/passwd
[root@devopsage ~]# more -p10 /etc/passwd /etc/group // to see 10 lines of each files at a time.
Note:
- Use “Enter Key” to scroll down line by line
- Use “d” to go to the next page
- Use “b” to move the cursor to the previous page
- Use “/” to search for a word in the file
- Use “v” to take that file into the VI Editor mode, where you can edit the file and save it.
3. head
head command is used to print the starting portion of any file, by default it shows top 10 lines of any file.
[root@devopsage ~]# head /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
[root@devopsage ~]#
We can also customize it to see any number of top line by using below command.
[root@devopsage ~]# head -2 /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
[root@devopsage ~]#
Note: in above command (2) can be replaced by any number to see the top “n” numbers of the line for a particular file.
4. tail
tail command works exactly opposite of the head command. it displays the bottom lines of the files, by default it displays bottom 10 lines of a file.
[root@devopsage ~]# tail /etc/passwd // displays bottom 10 lines
[root@devopsage ~]# tail -2 /etc/passwd // displays bottom 2 lines.
tail command has some more scenarios where it frequently used like if you want to monitor any log files use can use tail -f command. So it will show 10 bottom lines and if any lines get added to the files will be monitored automatically.
[root@devopsage ~]# tail -f /etc/passwd
[root@devopsage ~]# tailf /etc/passwd // same as tail -f
5. wc
wc command is another powerful filter command which is used to for lines, words or characters count from a file.
[root@devopsage ~]# wc /etc/passwd
37 75 1940 /etc/passwd
By default wc prints 3 numbers, as shown in the above sample output. here,
- 37 is line count
- 75: is word count
- 1940 is the character count.
Note: To get the specific number we can use the below options.
[root@devopsage ~]# wc -c /etc/passwd
1940 /etc/passwd
[root@devopsage ~]# wc -w /etc/passwd
75 /etc/passwd
[root@devopsage ~]# wc -l /etc/passwd
37 /etc/passwd
[root@devopsage ~]#
6. sort
It is used to sort the line in numeric of alphanumeric order. By default sort command sorts in ascending order.
Note: “-r” option will reverse the sort order. some command example is shown below.
To sort the output in numeric or alphanumeric order. by default, it sorts in ascending order.
[root@devopsage ~]# cat test.txt
welcome to devopsage
welcome to devopsage
devopsage linux tutorials
devopsage linux tutorials
[root@devopsage ~]# sort test.txt
devopsage linux tutorials
devopsage linux tutorials
welcome to devopsage
welcome to devopsage
[root@devopsage ~]#
To sort the lines according to the number.
[root@devopsage ~]# cat test.txt
4. welcome to devopsage
2. welcome to devopsage
1. devopsage linux tutorials
3. devopsage linux tutorials
[root@devopsage ~]# sort -d test.txt
1. devopsage linux tutorials
2. welcome to devopsage
3. devopsage linux tutorials
4. welcome to devopsage
[root@devopsage ~]# sort -h test.txt // same as sort -d
1. devopsage linux tutorials
2. welcome to devopsage
3. devopsage linux tutorials
4. welcome to devopsage
[root@devopsage ~]#
To sort the file with a numeric value.
[root@devopsage ~]# cat devopsage
newyork 20
berlin 4
paris 1
amsterdam 17
[root@devopsage ~]# sort -nk2 devopsage
paris 1
berlin 4
amsterdam 17
newyork 20
[root@devopsage ~]#
Note: here,
- n: To short with a numeric value.
- k2: defines the column 2.
Also, note that “-k” will work only when the column separator is space. If the separator is something other than space then we have to use the filed terminator i.e, “-t” option.
[root@devopsage ~]# cat sortexample
newyork:20
berlin:4
paris:1
amsterdam:17
[root@devopsage ~]# sort -t ":" -nk2 sortexample
paris:1
berlin:4
amsterdam:17
newyork:20
[root@devopsage ~]#
To reverse the order of the sort.
[root@devopsage ~]# sort -r sortexample
[root@devopsage ~]# sort -r sortexample // to remove the duplicate entry.
7. uniq
uniq command removes the duplicate entries from the files, but the thing we should keep in mind is that it only removes the consecutive duplicates. If the duplicate entries are not consecutive it will not remove the duplicate as expected, so for that, we have to first sort the files and then will have to remove the duplicate entries.
In order to use sort and uniq together, we have to use pipes, sample output is shown below.
[root@devopsage ~]# cat sample
devopsage
devopsage
Cloud
devopsage
linux
linux
linux
Cloud
devopsage
Cloud
linux
[root@devopsage ~]# uniq sample
devopsage
Cloud
devopsage
linux
Cloud
devopsage
Cloud
linux
[root@devopsage ~]# sort sample | uniq
Cloud
devopsage
linux
[root@devopsage ~]#
Note: by using “-c” option we can also get the count of the duplicate entries. let’s see that also.
[root@devopsage ~]# sort sample | uniq -c
3 Cloud
4 devopsage
4 linux
[root@devopsage ~]#
Note: For doing exactly the same thing we can also use “-c” option with sort command to make it simple.
[root@devopsage ~]# sort -u sample
Cloud
devopsage
linux
8. cut
cut command is used to pick up the resulted expressions in columns and displays the output. It will extract certain fields from the files. for more understanding of cut command, refer below sample output.
[root@devopsage ~]# cat cut.txt // will be using this file for cut demonstration
apache:x:48:48:Apache:/usr/share/httpd:/sbin/nologin
mysql:x:27:27:MariaDB Server:/var/lib/mysql:/sbin/nologin
sampleuser2:x:1002:1002::/home/sampleuser2:/bin/bash
[root@devopsage ~]# cut -d ":" -f1 file.txt
apache
mysql
sampleuser2
[root@devopsage ~]#
Note: In the above command symbol -d defines the delimiter which is “:” here, and “-f1” defines the first filed. the delimiter can be any things like for space it should be like, ” “. or any symbol can be used as a delimiter. for example below sample output shows comma (“,”) as a delimiter.
[root@devopsage ~]# cat devopsage
welcome,to,devopsage,technologies
[root@devopsage ~]# cut -d "," -f1 devopsage
welcome
[root@devopsage ~]#
You want to print the only character say 1 to 4, then use the below command.
[root@devopsage ~]# cut -c1-4 file.txt
apac
mysq
samp
[root@devopsage ~]#
If you want to print only character 1 and 4 then use the below command.
[root@devopsage ~]# cut -c1,4 file.txt
ac
mq
sp
[root@devopsage ~]#
If you want to print multiple fields, use below command.
[root@devopsage ~]# cut -d ":" -f1,5 file.txt
apache:Apache
mysql:MariaDB Server
sampleuser2:
[root@devopsage ~]#
If you want to print a range of field, use below command.
[root@devopsage ~]# cut -d ":" -f1-5 file.txt
apache:x:48:48:Apache
mysql:x:27:27:MariaDB Server
sampleuser2:x:1002:1002:
9. tr
It basically changes the incoming data. The transformation can be in different ways. let’s see some example of it.
Note: tr command works only on standard input, it does not work on any file.
In the below example we will see, how we can squeeze multiple occurrences of a character into one
[root@devopsage ~]# cat tr-example
devooooooooooooooooooooooopsage
[root@devopsage ~]# cat tr-example | tr -s "o"
devopsage
[root@devopsage ~]# cat trsample
hello devopsage
[root@devopsage ~]# cat trsample | tr -s " "
hello devopsage
[root@devopsage ~]#
[root@devopsage ~]# tr -s " "
hello everyone // type this
hello everyone // output you see, ctrl+d to exit.
[root@devopsage ~]#
How we can use switching between upper and lower character.
[root@devopsage ~]# cat tr.txt
devopsage
[root@devopsage ~]# cat tr.txt | tr [a-z] [A-Z]
DEVOPSAGE
[root@devopsage ~]#
10. sed
sed command is another powerful command which searches for the word and replaces it with the word of your choice.
[root@devopsage ~]# cat sed.txt
Welcome to the world of linux
[root@devopsage ~]# sed 's/linux/devops/g' sed.txt
Welcome to the world of devops
[root@devopsage ~]#
Note: here,
- s is substitute
- g is global, i.e will replace words everywhere in the file
The above command does not make any actual changes in the files, the changes are only displayed on the terminal. To make the changes in the file, use “-i” option.
[root@devopsage ~]# sed -i 's/linux/devops/g' sed.txt
11. awk
awk command is also one of the powerful filter commands which can be used in many ways. It can be used as editing the streams which we can also do using the sed command, but awk is more powerful than the sed command.
[root@devopsage log]# awk -F":" '{ print $1 $2 $3 $4 }' /var/log/yum.log
Feb 20 143344 Installed python2-futures-3.0.5-1.el7.noarch
Feb 20 143344 Installed python-dateutil-1.5-7.el7.noarch
Feb 20 143344 Installed python2-jmespath-0.9.0-3.el7.noarch
Feb 20 143345 Installed python2-botocore-1.6.0-1.el7.noarch
Feb 20 143345 Installed python2-s3transfer-0.1.10-1.el7.noarch
Feb 24 140635 Installed bc-1.06.95-13.el7.x86_64
12. grep
grep stand for Global Regular Expression Print. It is basically used to pick out the word from the files and print the output.
[root@devopsage ~]# grep root /etc/passwd
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin
[root@devopsage ~]#
13. paste
paste command is used to join two different files horizontally. In short, it appends the content of one file to another in a order.
[root@devopsage ~]# cat name
tony
vicky
robin
[root@devopsage ~]# cat place
neywork
london
paris
[root@devopsage ~]# cat day
1st jan
2nd jan
3rd jan
[root@devopsage ~]# paste name place day > team.txt
[root@devopsage ~]# cat team.txt
tony neywork 1st jan
vicky london 2nd jan
robin paris 3rd jan
[root@devopsage ~]#