# Basics to Advance: Unleashing the Power of AWK in Advanced Linux Operations

## Introduction

If you are a beginner or intermediate in using Linux, you must know some basic commands like cd, ls, sudo, rmdir etc. And I think these commands are ok but there is a powerful command which user might ignore or don't use due to its toughness or unawareness

When it comes to command-line text processing in Linux, AWK stands out as a versatile and powerful tool. In this article, we'll take a closer look at a basic AWK command and explore its capabilities in extracting specific information from text files.

---

## Scenarios

Before you start learning any command or tool, it's important to understand how it can actually help you in real-life situations, especially in your daily tasks. If something seems fancy but doesn't have a practical use, it might not be worth learning.

Learning should be about gaining skills that make your everyday life easier and more efficient, tackling the challenges you face regularly. So, before diving in, consider whether what you're learning has a meaningful impact on your day-to-day activities.

So that's why am sharing some real use case or scenario where awk command could be really helpful.

1. **Server Log Analysis:**
    
    * **Scenario:** Analyzing Apache web server or any web server logs to extract information about the number of requests for each page.
        
        Assuming you have an Apache log file named `access.log` with entries like
        
        ```plaintext
        192.168.0.1 - - [01/Jan/2022:12:00:00 +0000] "GET /page1 HTTP/1.1" 200 1234
        192.168.0.2 - - [01/Jan/2022:12:01:00 +0000] "GET /page2 HTTP/1.1" 404 5678
        192.168.0.3 - - [01/Jan/2022:12:02:00 +0000] "GET /page1 HTTP/1.1" 200 7890
        ```
        
        You can use the awk command to extract and count the number of requests for each page
        
        **Output**
        
        ```plaintext
           2 /page1
           1 /page2
        ```
        
2. **CSV File Manipulation:**
    
    * **Scenario:** Consider you have a CSV file named `data.csv` with the following content.
        
        ```plaintext
        Name,Age,Occupation
        Alice,28,Engineer
        Bob,35,Developer
        Charlie,22,Student
        David,40,Manager
        ```
        
        Now, let's say you want to extract and display only the "Name" and "Occupation" columns. You can use the following AWK command:
        
        **Output**
        
    * ```plaintext
                  Name Occupation
                  Alice Engineer
                  Bob Developer
                  Charlie Student
                  David Manager
        ```
        
3. **Password File Analysis:**
    
    * **Scenario:** Extracting and displaying user information from the `/etc/passwd` file.
        
4. **Network Configuration Review:**
    
    * **Scenario:** Analyzing the output of the `ifconfig` command to display network interface details.
        
        For e.g. ifconfig commands give you this output.
        
        ```plaintext
        eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
                inet 192.168.0.2  netmask 255.255.255.0  broadcast 192.168.0.255
                inet6 fe80::a00:27ff:fe8a:e2ab  prefixlen 64  scopeid 0x20<link>
                ether 08:00:27:8a:e2:ab  txqueuelen 1000  (Ethernet)
                RX packets 1501  bytes 1666445 (1.6 MB)
                RX errors 0  dropped 0  overruns 0  frame 0
                TX packets 1275  bytes 96476 (96.4 KB)
                TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        
        lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
                inet 127.0.0.1  netmask 255.0.0.0
                inet6 ::1  prefixlen 128  scopeid 0x10<host>
                loop  txqueuelen 1000  (Local Loopback)
                RX packets 9  bytes 546 (546.0 B)
                RX errors 0  dropped 0  overruns 0  frame 0
                TX packets 9  bytes 546 (546.0 B)
                TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        ```
        
        By using awk command, you can extract and display network interface details.
        
        **Output**
        
        ```plaintext
        Interface: eth0 IP Address: 192.168.0.2
        Interface: lo IP Address: 127.0.0.1
        ```
        
5. **Custom Log Parsing:**
    

* **Scenario:** Parsing a custom log format to extract relevant information.
    
    Let's consider a custom log file named `custom.log` with entries like:
    
    ```plaintext
    2022-01-01T12:00:00+00:00 | User=John | Action=Login | Status=Success
    2022-01-01T12:15:00+00:00 | User=Alice | Action=Logout | Status=Success
    2022-01-01T12:30:00+00:00 | User=Bob | Action=Login | Status=Failure
    ```
    
    Assuming you want to extract and display the date, user, action, and status information, by using awk you can get the output like this
    
    **Output**
    
    ```plaintext
    Date: 2022-01-01T12:00:00+00:00 User: John Action: Login Status: Success
    Date: 2022-01-01T12:15:00+00:00 User: Alice Action: Logout Status: Success
    Date: 2022-01-01T12:30:00+00:00 User: Bob Action: Login Status: Failure
    ```
    

These examples demonstrate how AWK can be applied in various real-world scenarios for data extraction, processing, and analysis in a Linux or Unix environment. The flexibility and power of AWK make it a valuable tool for system administrators, developers, and data analysts.

There are numerous use cases where you can use AWK command efficiently without much writing. I believe now we are good to go for learning this thing.

---

## What is AWK?

Awk is a command line utility or program or scripting language or text processing utility that means you give it some text and it can grab certain columns, certain rows, certain fields from the text for you. You can tell it to go search for certain string, patterns in the text and even replace those string patterns with other strings. Its a really powerful program and that's why developers and engineer mostly use all the time.

<details data-node-type="hn-details-summary"><summary>What is utility?</summary><div data-type="detailsContent">A "utility" refers to a software tool or program designed to perform a specific task or set of tasks, often related to system management, data processing, or other essential functions. Utilities are typically command-line or graphical applications that assist users in performing various operations on a computer.</div></details>

> "As a developer working in big tech companies, i probably overused awk. I used awk everywhere especially in my shell scripting because I'm really comfortable with it and it's one of those programs that once you learn awk, you wonder why you didn't learn it sooner because it's such a powerful program. " by Vishnu Tiwari

AWK is particularly well-suited for text processing and is commonly used in Unix and Unix-like operating systems for tasks such as data extraction, reporting, and text pattern matching. Being a command-line utility, it allows users to efficiently perform these tasks by executing commands in a terminal environment.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704366123720/f743b267-d275-45d0-acbb-f99cf6f344f0.jpeg align="center")

## Basics - Syntax + Examples

Hey, let's head it to the basics and explore the awk command for printing the columns, to use input and output field separator.

### Syntax

```bash
awk options 'selection _criteria {action }' input-file > output-file
```

Examples

```bash
ps
```

**Output**

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704367630943/8bdd23ad-278e-45de-865a-9d9daacd770b.png align="left")

```bash
ps | awk '{print $1}'
```

<details data-node-type="hn-details-summary"><summary>ps command</summary><div data-type="detailsContent">The <code>ps</code> command in Linux is used to provide information about currently running processes on a system.</div></details>

**print $1** - print the first column , change col no by putting the no. after $. like $2 for second column.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704367975316/c664f893-aae2-4496-9de7-cc75e5f9439e.png align="center")

```bash
ps | awk '{print $0}'
```

**$0** - simply prints everything just like cat

<div data-node-type="callout">
<div data-node-type="callout-emoji">💡</div>
<div data-node-type="callout-text">{print} - also prints the all, just like {print $0}</div>
</div>

**Output**

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704368285379/7547afc8-9471-4638-b2f7-d7c8d24d88fb.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704368365155/d6b5c6ab-3ab7-40d1-b485-448ddcbefa82.png align="left")

So one of the files that people love to use awk on GNU/Linux system is the /etc/passwd file, that is a file that lists all the users on your Linux system

```bash
cat /etc/passwd
```

the output looks like this,

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704369155344/43d94e37-fa46-43ee-9c91-d7ec3d6c5369.png align="center")

there's not a lot of spaces to it, there are columns i mean it is separated into columns but the columns they are separated by colons here.

<div data-node-type="callout">
<div data-node-type="callout-emoji">💡</div>
<div data-node-type="callout-text">Note : AWK treats spaces as the column delineator</div>
</div>

so for printing all the users!

```bash
awk -F ":" '{print $1}' /etc/passwd
```

**\-F** : for providing field separator, by default awk uses spaces as field, it basically splits the column by ':'.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704370073074/23df3ba5-99e0-4623-8372-1a3858c44810.png align="center")

if you want to print multiple columns, add the column no.

```bash
awk -F ":" '{print $1 $6 $7}' /etc/passwd
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704370903191/2c147483-5ceb-43e6-a61e-57e083cdd62c.png align="center")

its not very readable because we didn't tell it to separate the columns with spaces or colons or anything. We told it, hey print 1,6 and 7 column.

```bash
awk -F ":" '{print $1 "\t" $6 "\t" $7}' /etc/passwd
```

<details data-node-type="hn-details-summary"><summary>\t</summary><div data-type="detailsContent">used for giving a proper tab space</div></details>

!\[\](https://cdn.hashnode.com/res/hashnode/image/upload/v1704533913460/70aba778-05c4-43f4-b8b7-858075ed178e.png align="left")

In the output, we can see its showing tab spaces after all column and it's much more readable that way.

Now other than specifying a field separator to search for and use you know to determine what the columns are, you can actually print out the field separator as well and you can tell it to change the field separator to a different character as part of the output. you can do all this by using this command.

```bash
awk 'BEGIN{FS=":"; OFS="-"} {print $1, $6, $7}' /etc/passwd
```

`BEGIN{FS=":"; OFS="-"}`: This part of the command is executed before processing any input lines. It sets the input field separator (FS) to a colon (`:`) and the output field separator (OFS) to a dash (`-`).

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704534721862/a662db71-d646-40a3-9147-65c57ed3ed62.png align="center")

## Intermediate - AWK for Daily + Official Use

**In this level, we going to upgrade a one level up from the basics and learn about patterns**

* if you want to get the last column, you can use $NF for printing the last column , for example , you want to print all the shells present in your machine
    

```bash
cat /etc/shells
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704542068518/effd06ae-c81d-43d2-8c38-eeb551a2e086.png align="center")

As you can see in the output, the first line is a comment, which is not a valid shell. Therefore, we only need the lines that start with a forward slash '/'. To achieve this, we can use a command like the one provided.

```bash
awk -F '/' '/^\// {print $NF}' /etc/shells
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704543347110/ecb39f9a-c4b0-4a86-9bec-e41387ac4f42.png align="center")

**//** - Anything inside these two forward slashes (//) will be used by AWK to search for patterns.

**^** - it is the anchor, used to Indicates the beginning of the line

**\\/** - uses a backslash to tell AWK that the next character, a forward slash, is not a closing slash.

But in the output, you can see duplicate entries, which don't look good. We can use '**uniq**' to specify that we don't need duplicates.

```bash
awk -F '/' '/^\// {print $NF}' /etc/shells | uniq
```

for sorting

```bash
awk -F '/' '/^\// {print $NF}' /etc/shells | uniq | sort
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704544912076/4537a35b-5747-4be2-8736-66561608ed32.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704545845640/b611be49-6a0d-4059-874f-a966d0c5a8a1.png align="center")

And let's run the 'df' command because it is another common command that provides nice columned information, and people often enjoy using AWK on its output

```bash
df
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704546383167/2a238d5a-8d85-4a40-a899-3994baa06184.png align="center")

for printing the only **tmpfs** system,

```bash
df | awk '/^tmpfs/'
```

You can also perform operations in the columns like addition, multiplication, division etc.

```bash
df | awk '/^tmpfs/ {print $1 "\t" $2 + $3}'
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704547455129/169e7df0-ae74-4d7a-aefa-4a45634d39ee.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704547932473/296180fe-37e2-42bf-a8d7-e0cd407e85ea.png align="center")

You can also perform operations based on certain conditions

```bash
cat /etc/shells
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704558049200/f8592e5a-9782-4dc9-aa00-f68381bd6dda.png align="left")

For printing a line less than 8 characters

```bash
awk 'length($0) < 8' /etc/shells
```

for printing shell less than 5 character

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704558915055/7c302ec8-a86f-4e2f-9292-25c24e913db1.png align="left")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704559034542/f2eb8612-d4f5-430a-87af-770f4ab98679.png align="center")

## Advance - AWK For Pro Developers

**In this level, we going to upgrade a one more level up from the intermediate and learn about if/else, loops etc.**

### if / else in awk

**Syntax**

```bash
awk '{
    if (condition) {
        print $1 ";
    } else {
        print $2;
    }
}' input_file
```

**Example**

```bash
ps -ef # For printing all the process in your system
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704646432987/681aaa2c-8db1-4ce8-95ba-f53c1d233917.png align="center")

now we can check if the process is kworker . for printing all the kworker process, we can do

<div data-node-type="callout">
<div data-node-type="callout-emoji">💡</div>
<div data-node-type="callout-text">Note: For checking patterns inside if/else use '~' instead of '=='.</div>
</div>

```bash
ps -ef | awk '{if ($NF~ /kworker/) print $0}'
```

Now we can also distinguish that which is a kworker process or which one is regular by using else.

```bash
ps -ef | awk '{if ($NF~ /kworker/){ print $NF "\t kworker process"} else {print $NF "\t regular process"}}'
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704647275721/fc469b1c-ed3a-447b-b63f-01bc78f40989.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704647312572/ac4f220c-9ef0-4531-b6a3-8412b5888df3.png align="center")

### For loop

Since, awk is an scripting language we can also use loops, In AWK, there are two main types of loops: the `for` loop and the `while` loop.

```bash
awk '{
    for (i = 1; i <= 5; i++) {
        print "Number:", i;
    }
}' input_file
```

If you use this syntax, you need to mandatorily pass a input file, it basically run the loop for each line

for just performing the operations without passing the input file, you can use BEGIN.

```bash
ifconfig | awk '/^[a-zA-Z]/{interface=$1; next} /inet addr:/{print "Interface:", interface, "IP:", $2}'
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704650276801/a5af6b98-1d6b-4fdc-86b8-a9a19fc4c06a.png align="center")

### while loop

In AWK, you can use a `while` loop to repeatedly execute a block of code as long as a certain condition is true. Here's a simple example of using a `while` loop in AWK

```bash
awk '{
    i = 1;
    while (i <= 5) {
        print "Number:", i;
        i++;
    }
}' input_file
```

In this example:

* `i = 1`: Initialization of the loop variable `i` to 1.
    
* `while (i <= 5)`: The condition for the loop to continue as long as `i` is less than or equal to 5.
    
* `print "Number:", i`: Code inside the loop that prints the current value of `i`.
    
* `i++`: Incrementing `i` by 1 in each iteration.
    

### else if

! In AWK, you can use `if` and `else if` statements to implement conditional logic. Here's an example:

```bash
awk '{
    if ($1 > 10) {
        print $1 " is greater than 10";
    } else if ($1 == 10) {
        print $1 " is equal to 10";
    } else {
        print $1 " is less than 10";
    }
}' input_file
```

In this example, the AWK script reads input from `input_file` (you can replace it with your actual file name), and for each line, it checks the value in the first column (`$1`). Depending on the value, it prints a different message.

Explanation:

* `if ($1 > 10)`: If the value in the first column is greater than 10, execute the corresponding block of code.
    
* `else if ($1 == 10)`: If the value is equal to 10, execute this block of code.
    
* `else`: If none of the above conditions are true, execute this block.
    

You can adjust the conditions and actions inside the blocks to suit your specific requirements.

## Some Common Useful Commands For Daily Use

1. **substr**
    
    The `substr` function in AWK is used to extract a portion of a string. Its basic syntax is:
    
    ```bash
    substr(string, start[, length])
    ```
    
    * `string`: The input string from which you want to extract a substring.
        
    * `start`: The position in the string where extraction begins. The position is 1-based.
        
    * `length` (optional): The number of characters to extract. If omitted, it extracts the substring from the start position to the end of the string.
        
    
    for example , you have a file and you want to use substr on it, the actual use is depend on the layout of the file , but for demo see this example
    
    ```bash
    awk '{print substr($0, 3)}' numbered.txt
    ```
    
    ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704652837605/adb3193c-c0ba-4635-a89b-0780de5f4a6d.png align="left")
    
2. **match, RSTART and RLENGTH**
    
    In AWK, the `match` function is used to search a string for a specified pattern and sets the values of `RSTART` and `RLENGTH` to the starting position and length of the matched substring, respectively. The basic syntax of `match` is as follows:
    
    ```bash
    match(string, regexp)
    ```
    
    * `string`: The input string where you want to search for the pattern.
        
    * `regexp`: The regular expression pattern to search for in the string.
        
    
    Here's an example:
    
    ```bash
    awk 'match($0,/o/) {print $0, "Has O character at index", RSTART, "with length", RLENGTH}' numbered.txt
    ```
    
    ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704654255200/746a47ef-e056-4c50-a228-89139afa4015.png align="center")
    
3. **NR**  
    In AWK, `NR` is a built-in variable that represents the current record (line) number being processed. It is automatically incremented by AWK as it reads each input line. `NR` is especially useful when you want to perform actions based on the line number.
    
    ```bash
    df | awk 'NR==6, NR==8 {print NR".", $0}'
    ```
    
    ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704654544897/9e6c5ead-7acc-4924-a138-2708420e5480.png align="center")
    
    **for checking the line count of any file!**
    
    ```bash
    df | awk 'END {print NR}'
    ```
    
    ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704654997405/2dbc832a-21bc-4779-a280-27070c318f9a.png align="center")
    
4. **For checking IP Addresses**  
    Using `ifconfig` with AWK can be useful for parsing and extracting specific information about network interfaces. `ifconfig` provides details about the network configuration on a system, and AWK can help filter and format this information according to your needs.
    
    ```bash
    ifconfig | awk  '/^[a-zA-Z]/ {print "Interface: " $1 } /inet/ {print "IP Addrress:" $2}'
    ```
    
    ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704655184642/51e88a05-f9ba-42bc-b56b-af76631a5997.png align="center")
    

## Conclusion

In conclusion, our exploration of AWK has provided a comprehensive understanding of this powerful text processing tool. From its fundamental syntax to advanced features, we've delved into the intricacies that make AWK a versatile and indispensable asset in the realm of data manipulation and analysis.

Throughout the article, we've uncovered how AWK can be applied in various real-world scenarios, demonstrating its utility in tasks ranging from simple text processing to more complex data transformations. The practical examples and use cases serve as a guide for both beginners and experienced users, showcasing the flexibility and efficiency of AWK in handling diverse text-based challenges.

As we bring this AWK journey to a close, I extend my gratitude for your dedication in navigating through the intricacies of this scripting language. The knowledge gained here lays the foundation for enhanced productivity and problem-solving capabilities. Whether you're a newcomer or a seasoned AWK enthusiast, may the skills acquired pave the way for seamless text processing endeavors.

![](https://media1.giphy.com/media/vxNCVEe0PI9A3YVJEX/giphy.gif align="center")

That's a wrap on this article, where we journeyed from AWK basics to advanced ninja moves and explored its everyday applications. Kudos for sticking with it till the end! Much love and catch you at the next article. Stay sharp and keep rocking that tech game! ✌️

Thank you again for your time and commitment. Here's to harnessing the full potential of AWK in your future endeavors. Until our paths cross again in the realm of knowledge exploration, stay curious and keep scripting!

![](https://i.pinimg.com/originals/0b/fa/6c/0bfa6c0be319f1af0b1f802a4f78842d.gif align="center")