This article aims to demystify these fundamental constructs by delving into their internal mechanisms, offering clarity on their roles and importance within Python's ecosystem. By dissecting the nuances between iterators, iterables, and generators, readers will gain a profound understanding of how these elements shape the flow and efficiency of Python code.

Let's first look at what is Iterable?

Iterable - In Python

An iterable means, as the name suggests, any object we can iterate over, but there's a little more we should know about this. Generally, we know what objects can be iterable. For example, a list or tuple can be iterated over through a loop. But how do we know what can be iterable or what can't? For this, we just need to check if the object has the __iter__ method or not. Let me show you with an example.

Let's create a basic list in Python and iterate over it using a for loop. Also, we'll print all the properties and methods using the dir() method..

my_list = [1, 2, 3, 4]
for i in my_list:
    print(i)

print(dir(i))

Output

1
2
3
4
['__add__', '__class__', '__class_getitem__', '__contains__', '__delattr__', '__delitem__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__geta
ttribute__', '__getitem__', '__getstate__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__',
 '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__sizeof__
', '__str__', '__subclasshook__', 'append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort']

[Process exited 0]

So, if you see the output and find a dunder method called __iter__, it means we can iterate over it; otherwise, no.

Now I guess, we are pretty much clear about iterables, Now let's move to Iterators.

Iterator - In Python

An iterator in Python is indeed iterable; it can be looped over, and it includes the dunder method __iter__ in its default properties. However, what sets an iterator apart from an iterable?

An iterator is an object with a state, allowing it to remember its position during iteration. While this terminology might seem dense, it becomes clearer with examples. Iterators have a state, enabling them to determine where they are in the iteration process. Moreover, iterators know how to retrieve their next value, accomplished through a dunder method called next.

Let's revisit our previous output. We observe that a list lacks a next method, and it also lacks any state information. Consequently, a list does not possess knowledge of its state or how to retrieve its next value, rendering it not an iterator.
Now, how can we create our iterator so that it can retain its state and retrieve the next value?

We achieve this by invoking the dunder method iter on our iterable, thereby transforming it into an iterator.

new = [1, 2, 3, 4]

i_new = iter(new) 
# OR
i_new = new.__iter__()

print(i_new)
print(dir(i_new))

Output

<list_iterator object at 0x100fc1c60>
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subcla
ss__', '__iter__', '__le__', '__length_hint__', '__lt__', '__ne__', '__new__', '__next__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__s
izeof__', '__str__', '__subclasshook__']

[Process exited 0]

Now, observing the properties, we notice the presence of both the dunder methods iter and next, confirming that it is indeed an iterator object. However, you may wonder why the iterator itself has an iter method, reminiscent of the case with a list. This occurrence arises because an iterator is also iterable. Therefore, running iter on an iterator simply returns the same object.

Let's delve into the next method with an example.

new = [1, 2, 3, 4]

i_new = iter(new)

print(i_new)

print(next(i_new))
print(next(i_new))
print(next(i_new))
print(next(i_new))
print(next(i_new))

Output

<list_iterator object at 0x1045b7fd0>
1
2
3
4
Traceback (most recent call last):
  File "/Users/vishnu.tiwari/Desktop/Python/Misc/iig.py", line 11, in <module>
    print(next(i_new))
          ^^^^^^^^^^^
StopIteration

[Process exited 1]

As mentioned, an iterator retains its state throughout the iteration process and knows its next value. In the previous example, every time we request the next value from our iterator object, it recalls its previous state and provides the subsequent value. Once it exhausts all available values, it raises the StopIteration exception. This exception indicates that the iterator has been depleted and has no more values to offer.

Now let's talk about the diamond topic which you won't find more articles about it which is the mechanism of a for loop in Python

Mechanism Of Foor Loop - Python

When we run a normal for loop, it knows how to handle the stop iteration exception and it doesn't show it to us. In the background a for loop is doing something like this, it's first getting an iterator of our original object and then it's getting the next values. until it hits a stop iteration exception.

Under the hood for loop itself uses a while loop and work like this.

# Normal For Loop 
for item in iterable:
    # do something with item
    pass

BTS

iterator = iter(iterable)
while True:
    try:
        item = next(iterator)
    except StopIteration:
        break
    # do something with item

Generator

Some of you may have used generators, they're extremely useful for creating easy to read iterator, they look a lot like normal functions but instead of retuning a result, they instead yield a value and when they yield a value it keeps that state until the generator is run again and yields the next value so generator are iterators as well but the dunder iter and next methods are created automatically.

When a function contains one or more yield statements, Python treats it as a generator function. When you call a generator function, it returns a generator object without executing the function's code immediately. Instead, the function's code runs in response to the iterator's __next__() method being called.

Here's a simplified explanation of how a generator function works under the hood:

• When you call a generator function, it returns a generator object.

• Each time you call the generator object's __next__() method (implicitly via a loop or explicitly), the generator function executes until it encounters a yield statement.

• When a yield statement is reached, the value specified with yield is returned, and the function's execution is paused. The generator object retains its state, including the local variables' values.

• When the generator object's __next__() method is called again, execution resumes from where it left off, continuing until the next yield statement or the end of the function is reached.

• When there are no more yield statements in the function or a return statement is encountered, a StopIteration exception is raised, indicating that the generator has exhausted its sequence.

Here's a conceptual example to illustrate how this works:

pythonCopy codedef my_generator():
    print("Start of generator")
    yield 1
    print("After first yield")
    yield 2
    print("After second yield")
    yield 3
    print("End of generator")

gen = my_generator()  # Create a generator object
print(next(gen))  # Output: 1
print(next(gen))  # Output: 2
print(next(gen))  # Output: 3
# print(next(gen))  # Raises StopIteration error

In this example, you can see how the generator function's code is executed up to the first yield statement when next(gen) is called for the first time. The function's execution is paused, and the value 1 is returned. Subsequent calls to next(gen) resume the function's execution from where it last yielded.

Summary

This article delves into the fundamental concepts of iteration, iterables, iterators, and generators in Python. It explains the differences between them, how to identify iterables, create iterators, and utilize generators. Additionally, it explores the mechanism of a for loop and provides insights into the workings of generators through examples.

Python is a popular programming language. It was created by Guido van Rossum, and released in 1991.....Hey F#ck Stop it!...

I know you are annoyed with this wikipedia shit, but trust me, things will make sense if you know why python was created.

Chapter 1 : Born of a warrior (Python)

The common programming languages between 1980 to 1990 were C, Pascal, Fortran, COBOL, BASIC, etc. During that time, the computer industry was growing rapidly. Many intelligent individuals understood the future potential of computers and recognised that softwares will play a crucial role in making them more intriguing. Consequently, some people changed their careers from different domains to software development. Most of them have a innovative ideas, but to implement their own idea, they first needed to learn a programming language, which was a complex task.

During that era, developers had to handle everything by themselves, from managing memory to dealing with common tasks like buffer overflow, memory leaks which were both commonplace and repetitive.

Guido van Rossum joined the chat...

During that era, Guido van Rossum was working at the Centrum Wiskunde & Informatica (CWI) in the Netherlands, a national research institute for mathematics and computer science.

Guido had been involved in various projects related to distributed systems and operating systems at CWI, and has gained valuable experience in software development.

During his time at CWI, Guido was dissatisfied with the existing programming languages available, finding them either too complex or lacking in certain areas. He had experience with languages like ABC and Modula-3, which influenced his thinking about language design.

Motivated by a desire to create a programming language that emphasized simplicity, readability, and productivity, Guido started working on a personal project to develop what would become Python. Drawing inspiration from his experiences with other languages and his desire to address their limitations, Guido set out to design a language that would be easy to learn and use, yet powerful enough to tackle a wide range of tasks.

Python was initially developed as a hobby project by Guido during his spare time at CWI. He worked on refining the language's syntax and semantics, carefully crafting features that would make it intuitive and expressive. Guido's goal was to create a language that would enable programmers to write clear, concise code that could be easily understood and maintained, without sacrificing flexibility or power.

As Python gained popularity within CWI and beyond, Guido decided to dedicate more time to its development, eventually leaving his job at CWI to focus on Python full-time.

Becoming a Bad Boy

After gaining popularity python started to roast their other friends like C, COBOL, FORTAN. it's not my words, he actually started bullying every programming language. Here are some of the examples.

C: Ah, C, the powerhouse of the era, loved for its speed and low-level control. But let's face it, memory management in you was like walking through a minefield. I (Python) comes in like a superhero with my automatic memory management, sparing developers from the headache of manual memory allocation and pesky segmentation faults.

Pascal: Bless Pascal's heart for its structured programming and strong typing, but its verbosity could put anyone to sleep. Python swoops in with its clean and concise syntax, making code elegant and readable without sacrificing functionality.

Fortran: Sure, Fortran was the go-to for numerical computing, but its syntax felt like a blast from the past even back then. Python's modern and flexible syntax feels like a breath of fresh air, attracting scientists and engineers with its ease of use and extensive libraries for numerical computing.

Since he become a bad boy, people started hating him for everything from being slow asf to being rude.

C(Queen) said : "what's with your performance? Sure, you're great for scripting and prototyping, but when it comes to heavy lifting, you're about as fast as a sloth on tranquilizers. Don't even get me started on your GIL (Global Interpreter Lock). Multithreading? More like multi-slowing."

Am just joking, please don't search for facts.. It is my made up controversy. But python being slow is really a thing 😂.

Chapter 2: Python Being Python (Memory Genius)

Let's now get into the serious stuff. For understanding how python works, we need to understand first , how python memory is managed because it directly impacts the performance and behaviour of Python programs.

Memory management in Python differs from languages like C or C++, where developers have explicit control over memory allocation and deallocation.

In Python, memory management is handled by the Python runtime using a combination of techniques such as automatic memory allocation, garbage collection, and reference counting.

Now let's start understanding

• Whenever a program or file is created and if you are using it! It takes a significant amount of memory in your RAM. Same goes for a python program in execution mode, that also takes some amount of memory in the RAM.

• The memory which is allocated to the python program in ram is further divided into two regions.

• Stack and Private Heap Space

• Object always gets created in Heap Space and the variable name gets allocated in stack.

• Let me explain you with an example. Suppose there is a file name called test.py which contains:

•       a = 10
  

• Now 10 is a int object and The variable a is created as a reference to this integer object. Now where the object 10 will be created? in the RAM, but where in RAM? the region allocated to the program in RAM? but which region? The Answer is PRIVATE HEAP SPACE.
  Each object also gonna have an address in the heap space.
  - Where does the name gets created?
  The name a gonna create in STACK and a has the address of its object. which means a is gonna point to the address of 10.

  And python automatically checks the type of your object, that's why it is a dynamically typed language.

• I created a diagram for better understanding.

  

Take out the trash : Garbage Collector

In Python, the garbage collector is responsible for reclaiming memory occupied by objects that are no longer in use, thus freeing up resources and preventing memory leaks. Python's garbage collector uses a technique called reference counting along with cyclic garbage collection to manage memory.

You won't understand like this let me tell you a story

Imagine Python's garbage collector as a diligent janitor named Gary. His job is to keep Python's memory clean and tidy, ensuring there's no garbage lying around.

Now, Gary's first tool in his arsenal is "reference counting." It's like keeping track of how many times someone mentions a particular item. So, whenever someone (or some variable) mentions an object, Gary scribbles a tally mark on his clipboard. When nobody mentions the object anymore, Gary checks his clipboard. If the tally reaches zero, he knows it's safe to throw that object into the garbage bin.

But wait, sometimes things get a bit tricky. Picture two objects, let's call them Bert and Ernie, holding hands in a circle and giggling like schoolkids. They're referencing each other, creating a loop that Gary's reference counting can't handle. It's like Bert says, "I'm holding Ernie's hand," and Ernie says, "I'm holding Bert's hand," and they just go around in circles forever.

Now, Gary scratches his head. He can't just rely on his tally marks to clean up this mess. So, he brings out his special tool: the "cyclic garbage collector." It's like a magical broom that can sweep away those circular references. With a flick of his wrist, Gary breaks the loop, and Bert and Ernie can finally let go of each other's hands and go their separate ways.

And that, my friend, is how Python's garbage collector, with the help of Gary the janitor, keeps Python's memory squeaky clean, one object at a time. Just don't let him catch you leaving your variables lying around unattended!Imagine Python's garbage collector as a diligent janitor named Gary. His job is to keep Python's memory clean and tidy, ensuring there's no garbage lying around.

Now, Gary's first tool in his arsenal is "reference counting." It's like keeping track of how many times someone mentions a particular item. So, whenever someone (or some variable) mentions an object, Gary scribbles a tally mark on his clipboard. When nobody mentions the object anymore, Gary checks his clipboard. If the tally reaches zero, he knows it's safe to throw that object into the garbage bin.

But wait, sometimes things get a bit tricky. Picture two objects, let's call them Bert and Ernie, holding hands in a circle and giggling like schoolkids. They're referencing each other, creating a loop that Gary's reference counting can't handle. It's like Bert says, "I'm holding Ernie's hand," and Ernie says, "I'm holding Bert's hand," and they just go around in circles forever.

Now, Gary scratches his head. He can't just rely on his tally marks to clean up this mess. So, he brings out his special tool: the "cyclic garbage collector." It's like a magical broom that can sweep away those circular references. With a flick of his wrist, Gary breaks the loop, and Bert and Ernie can finally let go of each other's hands and go their separate ways.

And that, my friend, is how Python's garbage collector, with the help of Gary the janitor, keeps Python's memory squeaky clean, one object at a time. Just don't let him catch you leaving your variables lying around unattended!

Chapter 3: The Matrix | Python

In "The Matrix," the protagonist Neo discovers that the world he perceives as reality is actually a simulated environment created by sentient machines. Similarly, in the world of Python programming, developers often work with abstracted concepts and tools that may seem straightforward on the surface but actually operate in a more complex manner behind the scenes.

In "The Matrix," the simulated reality is controlled and manipulated by the sentient machines, who use it to subdue and control humanity. Similarly, the Python interpreter serves as the gatekeeper to the simulated reality of Python code, interpreting and executing instructions according to its programming.

But How Does Python Interpreter Really Works?

First Step: Lexical Analysis and Parsing

When you make a script or add lines of code and execute the program. Python Interpreter reads the code line by line, just like the execution flow. However, there are some exceptions to this rule. For example, if a line of code contains a loop, function call, or other control flow structure, the interpreter may need to execute multiple lines of code before moving on to the next line.

Lexical Analysis: When you execute the python code. The first step is lexical analysis, also known as tokenization. During this process, the Python interpreter breaks the code into individual tokens such as keywords, identifiers, operators, and literals. These tokens form the basic building blocks of the Python language.

Parsing : Once the code has been tokenized, the Python interpreter parses it to create an abstract syntax tree (AST). The AST represents the hierarchical structure of the code, with nodes corresponding to different elements such as expressions, statements, and functions. The parser ensures that the code follows the syntactic rules of the Python language.

Second Step: ByteCode Generation

After parsing, the Python interpreter generates bytecode instructions based on the AST. Bytecode is a low-level, platform-independent representation of the Python code that the PVM(Python Virtual Machine) can execute. Each bytecode instruction corresponds to a specific operation or action, such as loading a value onto the stack, calling a function, or performing arithmetic operations.

In Python, the bytecode is stored in a .pyc file. In Python 3, the bytecode files are stored in a folder named __pycache__. This folder is automatically created when you try to import another file that you created:

Third Step: ByteCode Execution

The bytecode generated by the Python interpreter is executed by the Python Virtual Machine (PVM). The PVM is responsible for interpreting and executing the bytecode instructions generated from the Python source code. It provides a runtime environment that manages memory, handles exceptions, and interacts with the underlying operating system to execute Python programs.

In summary, the Python interpreter compiles Python source code into bytecode, and the Python Virtual Machine executes this bytecode to run the Python program. This separation of concerns between compilation and execution allows Python code to be platform-independent and easily portable across different operating systems and hardware architectures.

I made a diagram for better understanding.

Example : A python code working

Let's walk through the full working of a Python code example:

# Define a function to calculate the factorial of a number
def factorial(n):
    if n == 0:
        return 1
    else:
        return n * factorial(n - 1)

# Main program
if __name__ == "__main__":
    # Prompt the user for input
    num = int(input("Enter a number: "))

    # Calculate and display the factorial of the input number
    print("Factorial of", num, "is", factorial(num))

Now, let's break down the execution of this code step by step:

1. The Python interpreter reads the source code file line by line.

2. The interpreter encounters the def keyword, indicating the definition of a function named factorial. The function definition is stored in memory for later use.

3. The interpreter moves to the if __name__ == "__main__": block, which checks if the script is being run as the main program.

4. Since the script is indeed being run as the main program, the interpreter proceeds to execute the code inside the if block.

5. The input() function prompts the user to enter a number, which is then converted to an integer using int() and assigned to the variable num.

6. The factorial() function is called with the input number as an argument. This triggers a recursive chain of function calls to calculate the factorial of the input number.

7. Each recursive call to factorial() decrements the input number by 1 until it reaches 0, at which point the base case (if n == 0:) is triggered, and the function returns 1.

8. As the recursive calls unwind, each intermediate result is multiplied by the current number until the final factorial value is computed.

9. The calculated factorial value is then printed to the console using print().

10. The program execution finishes, and the Python interpreter exits.

During this process, the Python interpreter compiles the source code into bytecode, which is then executed by the Python Virtual Machine (PVM). The PVM manages memory, handles function calls, and performs other runtime tasks to execute the Python program efficiently.

Chapter 3: A Slow End

While Python has earned widespread acclaim for its simplicity, readability, and versatility, one criticism that often arises is its perceived lack of speed compared to lower-level languages like C or C++. In this chapter, we'll explore why Python may be slower in certain contexts and how developers can mitigate performance concerns. Let's dive in!

Understanding Python's Execution Model

Python's dynamic typing, automatic memory management, and high-level abstractions contribute to its ease of use and rapid development cycle. However, these features can also introduce overhead that impacts performance.

One factor that contributes to Python's runtime overhead is its interpreted nature. Unlike compiled languages, where code is translated directly into machine code before execution, Python code is first compiled into bytecode and then interpreted by the Python Virtual Machine (PVM). While this approach offers flexibility and platform independence, it can result in slower execution speeds compared to compiled languages.

Additionally, Python's Global Interpreter Lock (GIL) poses a limitation on multi-threaded performance. The GIL ensures that only one thread executes Python bytecode at a time, effectively preventing multi-core parallelism in CPU-bound tasks. While this simplifies memory management and concurrency control, it can lead to suboptimal performance in multi-threaded applications.

Example: Comparing Python and C in Matrix Multiplication

To illustrate the performance difference between Python and a compiled language like C, let's consider a simple example of matrix multiplication implemented in both languages.

Python Implementation:

import numpy as np

# Generate random matrices
size = 1000
matrix_a = np.random.rand(size, size)
matrix_b = np.random.rand(size, size)

# Perform matrix multiplication
result = np.dot(matrix_a, matrix_b)

C Implementation:

#include <stdio.h>
#include <stdlib.h>

#define SIZE 1000

void matrix_multiply(double matrix_a[SIZE][SIZE], double matrix_b[SIZE][SIZE], double result[SIZE][SIZE]) {
    for (int i = 0; i < SIZE; i++) {
        for (int j = 0; j < SIZE; j++) {
            result[i][j] = 0;
            for (int k = 0; k < SIZE; k++) {
                result[i][j] += matrix_a[i][k] * matrix_b[k][j];
            }
        }
    }
}

int main() {
    double matrix_a[SIZE][SIZE];
    double matrix_b[SIZE][SIZE];
    double result[SIZE][SIZE];

    // Initialize matrices with random values
    for (int i = 0; i < SIZE; i++) {
        for (int j = 0; j < SIZE; j++) {
            matrix_a[i][j] = (double) rand() / RAND_MAX;
            matrix_b[i][j] = (double) rand() / RAND_MAX;
        }
    }

    // Perform matrix multiplication
    matrix_multiply(matrix_a, matrix_b, result);

    return 0;
}

In this example, we generate two random matrices of size 1000x1000 and multiply them using both Python's NumPy library (which is implemented in C) and a C program. We then compare the execution times of both implementations.

Conclusion and Mitigation Strategies

While Python may not always match the raw speed of compiled languages like C, there are several strategies developers can employ to improve performance:

• Utilize libraries and extensions: Python offers extensive libraries and extensions, such as NumPy, Cython, and Numba, which provide optimized implementations for numerical and computationally intensive tasks.

• Profile and optimize critical code paths: Identify performance bottlenecks using profiling tools like cProfile or line_profiler, and optimize critical code paths using techniques such as algorithmic improvements, caching, or parallelization.

• Offload performance-critical tasks to compiled languages: Use tools like ctypes or Cython to interface with C/C++ code for performance-critical tasks while maintaining the high-level expressiveness of Python.

By understanding Python's execution model and employing optimization techniques, developers can strike a balance between productivity and performance, ensuring that Python remains a powerful tool for a wide range of applications.

With this chapter, we've explored the nuances of Python's performance characteristics and provided insights into mitigating performance concerns. Armed with this knowledge, developers can harness the full potential of Python while addressing performance requirements in their projects.

Why it is crucial?

Just imagine working on a feature for a product that is expected to be used by 3000-4000 users. For example, you write one piece of code that takes 15 seconds to complete a certain task. Afterward, you optimize the code, and now it performs the same task in 10 seconds. Initially, you might think it's just a 5-second difference, not much. However, if you calculate the impact, considering one user visiting that feature three times a day, resulting in 15 seconds saved daily, 7.5 minutes saved monthly, and 1.5 hours saved annually. If the product is used by 4000 users, this amounts to 6000 hours saved.

This means even a small improvement in your code could save a considerable amount of time for the customer. In the modern world, we all know that time is equivalent to real money.

But optimizing the code is boring asf!!!!

I know it looks boring but trust me, it won't be boring if you play it like a game. Think of each optimization as leveling up in your coding adventure. 🚀💻

Some intellectual says we are also saving energy by optimizing the code as optimized code requires fewer resources since it also contributes to energy efficiency. It aligns with the trend towards more sustainable computing practices.

Having grasped the significance of code performance and profiling, let's delve into the methods for measuring how well our code performs.

timeit Module - for small snippets of code

• Purpose:timeit is designed for measuring the execution time of small code snippets or functions.

• Usage: It's suitable for quick and simple measurements to get an idea of how long it takes to run a specific piece of code.

The timeit module in Python provides a simple way to measure the execution time of small bits of Python code. It is particularly useful for assessing the performance of specific code snippets or functions. The timeit module can be employed both in the command line and within scripts to obtain accurate timing information. It helps in evaluating the efficiency of different implementations and aids in making informed decisions regarding code optimization.

And I just want to stress that timeit should be used strictly for small snippets of code.

timeit runs the code multiple times and calculates the average time taken. This helps in getting more reliable and repeatable results, especially for short-running code where the timing might be affected by other processes running on the system.

Let's see how we can use timeit to check the code performance.
Here am importing three modules

import timeit # timeit module
import time
import random

I imported 3 modules, now let's take a simple example and check which is the fastest from two inbuilt random function

1. random.randint(a, b):

   This function is used to generate a random integer between the specified range [a, b].

2. random.random():

   It gives a random float between 0 and 1

Let's test which one is fast between these two.

import timeit
import random
import time

"""
There are two inbuilt function in random which is 
1. random.randint
2. random.random

1. it gives the random number between two numbers which user provides.
2. it gives random float between 0 and 1.
"""

print(random.randint(1,2))
print(random.random())


randint_checker = timeit.timeit(stmt='random.randint(0,5)',
                                setup='import random',
                                number=100_000)

random_checker = timeit.timeit(stmt='random.random()',
                               setup='import random',
                               number=100_000)

print(f'Time of randint function is {round(randint_checker,4)} '
      f'\nTime of random function is {round(random_checker,4)}')

print(f'We can clearly see {"random function" if random_checker < randint_checker else "RandInt"}'
      f' as high in performance.')

Let me explain the code for a bit.

1. The stmt parameter in timeit.timeit is the statement that will be executed and timed. In your case, it's the function calls random.randint(0,5) and random.random().

2. The setup parameter is used to set up the environment before running the timed code. In your case, it imports the random module.

3. The number parameter specifies the number of times the statement should be executed for each timing measurement. By default it sets to 1 million

4. The results are printed, and the code determines which function is faster based on the execution times.

Now let's get back to the real scenarios, eg: you wrote two functions to perform the same task and you want to find out which one is fast, In the example, we have two functions both used to generate the Fibonacci Series.

first 1

def fibonacci_series(n):
    fib_series = []
    a, b = 0, 1

    for _ in range(n):
        fib_series.append(a)
        a, b = b, a + b

    return fib_series

# Example: Print the first 10 numbers in the Fibonacci series
n = 10
result = fibonacci_series(n)
print(f"Fibonacci Series up to {n} terms: {result}")

Second 2

def fibonacci_generator(n):
    a, b = 0, 1
    count = 0

    while count < n:
        yield a
        a, b = b, a + b
        count += 1

# Example: Print the first 10 numbers in the Fibonacci series using the generator
n = 10
fibonacci_gen = fibonacci_generator(n)
result = list(fibonacci_gen)
print(f"Fibonacci Series up to {n} terms: {result}")

Let's check the performance

import timeit


def fibonacci_series(n):
    fib_series = []
    a, b = 0, 1

    for _ in range(n):
        fib_series.append(a)
        a, b = b, a + b

    return fib_series


# Example: Print the first 10 numbers in the Fibonacci series using the generator
def fibonacci_generator(n):
    a, b = 0, 1
    count = 0

    while count < n:
        yield a
        a, b = b, a + b
        count += 1


n = 10

checker_fs = timeit.timeit(stmt='fibonacci_series(n = n)',
                           globals=globals())
checker_fg = timeit.timeit(stmt='fibonacci_generator(n = n)',
                           globals=globals())

print(f'fibonacci_series: {round(checker_fs, 4)} '
      f'\nfibonacci_generator: {round(checker_fg, 4)}')
print(f'{"fibonacci series" if checker_fs<checker_fg else "fibonacci_generator"} '
      f'is faster')

if your code snippet relies on global variables and you want to include those global variables in the execution context, you can use the globals parameter of the timeit.timeit function

Output

timeit module also allows to repeat the test for specified times

repeat_fs = timeit.repeat(stmt='fibonacci_series(n = n)',
                          globals=globals(),
                          repeat=5)

repeat_fg = timeit.repeat(stmt='fibonacci_generator(n = n)',
                          globals=globals(),
                          repeat=5)


print(f'Fibonacci_series: {repeat_fs}')
print(f'Fibonacci_generator: {repeat_fg}')

Output

Cprofile - For the entire execution of a program

• Purpose:cProfile is used for profiling the entire execution of a program and provides detailed information about the time spent in each function call.

• Usage: It's suitable for identifying performance bottlenecks, understanding function call patterns, and getting a more in-depth analysis of where the program spends its time.

cProfile is a built-in module in Python that provides a set of functions for profiling the performance of a Python program. Profiling helps you understand how much time your program spends on different functions, which can be useful for identifying bottlenecks and optimizing your code.

Let's see this in practice.

We are going to make a fake website backend, so it's going to make API requests it's going to be able to refresh the page, and so on just, we can see where our program can be improved and we can get how long it took to execute certain parts of the code.

So first we go ahead and create an API call and we know API calls are very costly functions.

import cProfile
import time


def api_call():
    time.sleep(2)
    return None

Here, when we have some data, we want to process it, so we are going to create a function called process_data.

def process_data():
    for i in range(10 ** 7):
        pass

then sort the data and let's reload the data.

def sort_data():
    for i in range(10 ** 8):
        pass
    process_data()

def reload_page():
    process_data()
    sort_data()
    time.sleep(2)

let's execute in the main

def main():
    api_call()
    sort_data()
    reload_page()

The real benchmarking.

if __name__ == '__main__':
    cProfile.run('main()', sort='cumtime')

Output

Let me explain to you what's in the output.

1. General Information:

   • 13 function calls in 6.183 seconds: This line indicates that there were 13 function calls in the entire program, and the program took 6.183 seconds to execute.

2. Ordered by Cumulative Time:

   • Ordered by: cumulative time: The output is sorted by cumulative time, i.e., the total time spent in a function and its sub-functions.

3. Columns Explanation:

   • ncalls: The number of calls made to the function.

   • tottime: The total time spent in the function excluding time spent in sub-functions.

   • percall: The average time per call (excluding sub-functions).

   • cumtime: The cumulative time spent in the function and its sub-functions.

   • percall: The average cumulative time per call.

   • filename:lineno(function): Information about the function, including the filename, line number, and function name.

4. Specific Lines Explained:

   • {built-in method builtins.exec} and <string>:1(<module>): Overhead added by the Python interpreter for executing the code.

   • cprofile_module.py:28(main): Time spent in the main function (the entry point of your script).

   • {built-in method time.sleep}: Time spent in the time.sleep function calls.

   • cprofile_module.py:22(reload_page): Time spent in the reload_page function, including time spent in sort_data and process_data.

   • cprofile_module.py:16(sort_data): Time spent in the sort_data function, including time spent in process_data.

   • cprofile_module.py:5(api_call): Time spent in the api_call function.

   • cprofile_module.py:10(process_data): Time spent in the process_data function.

5. Profiler Disable Line:

   • {method 'disable' of '_lsprof.Profiler' objects}: Indicates that the profiler was successfully disabled.

CProfile is not suitable for the function with a shorter execution time but effective if you want to know other details like the number of function calls and inbuilt functions.

Let me show, what I mean by that.

import cProfile


def fibonacci_series(n):
    fib_series = []
    a, b = 0, 1

    for _ in range(n):
        fib_series.append(a)
        a, b = b, a + b

    return fib_series


def fibonacci_generator(n):
    a, b = 0, 1
    count = 0

    while count < n:
        yield a
        a, b = b, a + b
        count += 1


n = 10

# Profile the execution of fibonacci_series and fibonacci_generator
profiler = cProfile.Profile()
profiler.enable()
result_series = fibonacci_series(n)
result_generator = list(fibonacci_generator(n))
profiler.disable()
profiler.print_stats(sort='cumtime')

Output

You can see it's showing zero seconds, but in timeit , it was showing the accurate execution time.

If the cProfile output is showing 0 seconds for all functions, it might indicate that the execution time of your code is extremely short, possibly below the resolution of the timer used by cProfile.

Conclusion

Choosing BetweentimeitandcProfile:

• If you want a quick measurement of the execution time for a specific piece of code, use timeit.

• If you need detailed information about function calls and where time is spent in your entire program, use cProfile.

In practice, both tools can complement each other. You might use timeit for initial quick measurements and then use cProfile for more in-depth analysis if you identify potential performance issues.

By combining these tools, developers can adopt a two-tiered approach. Start with timeit for initial assessments, identifying potential areas of concern. Subsequently, use cProfile to delve deeper into the code, gaining insights into function-level performance and optimizing critical sections.

Ultimately, the judicious use of timeit and cProfile empowers developers to strike a balance between quick assessments and thorough profiling, leading to well-optimized and high-performance Python code.

What is CI/CD?

CI/CD stands for Continuous Integration and Continuous Deployment (or Continuous Delivery), and it represents a set of best practices, principles, and automated processes aimed at improving the software development and delivery lifecycle.

Yeah, but what is this and why should we use it?

Continuous Integration (CI): Imagine a team of developers working on a mobile app. Each developer is responsible for adding new features or fixing bugs. In a CI setup, every time a developer finishes their work on a particular feature or bug fix, they integrate their code changes into a shared code repository, let's say on GitHub. Automated tests are then triggered to ensure that the new code doesn't break the existing functionality. If any issues are found, the team is notified immediately, allowing them to fix the problems early in the development process.

Continuous Deployment (CD): Now, consider the same mobile app project, but this time with continuous deployment. Once the code changes pass all the automated tests in the CI phase, the updated app is automatically deployed to a staging environment or even directly to production. Users can start using the new features or bug fixes almost immediately. This rapid deployment ensures that the latest improvements are available to users as soon as they are deemed stable and functional.

Continuous Delivery (CD): In a continuous delivery scenario, the process is similar to continuous deployment, but with an additional manual step before deploying to the production environment. After passing automated tests in the CI phase, the updated app is deployed to a staging environment. A team member or a designated person then reviews the changes and decides when to manually trigger the deployment to the production environment. This adds a layer of human oversight before changes go live.

What is a pipeline?

A pipeline refers to a set of automated processes and steps through which software code progresses from development to deployment. This sequence of steps is known as a "pipeline" because it represents a flow of activities, typically organized in a linear or branching structure. The purpose of a pipeline is to automate and streamline the software delivery process, making it more efficient, consistent, and reliable.

Let's consider a simplified example of a continuous integration and deployment (CI/CD) pipeline for a web application using a hypothetical e-commerce project.

1. Source Code Repository:

   • Developers collaborate on the project and store the source code in a version control repository, such as Git on GitHub.

2. Continuous Integration (CI) Pipeline:

   • When a developer pushes changes to the repository, it triggers the CI pipeline.

   • The CI pipeline consists of the following stages and tasks:

     • Build Stage:

       • Compiles the source code into executable binaries.

     • Test Stage:

       • Runs automated tests to ensure that the new code changes haven't introduced regressions or errors.

     • Code Quality Stage:

       • Checks for coding standards, code complexity, and other code quality metrics.

     • Artifact Generation:

       • Creates deployable artifacts, such as a packaged web application.

3. Continuous Deployment (CD) Pipeline:

   • If all the tasks in the CI pipeline are successful, it triggers the CD pipeline.

   • The CD pipeline consists of the following stages and tasks:

     • Deploy to Staging:

       • Takes the artifacts from the CI pipeline and deploys them to a staging environment.

       • Automated tests are run in the staging environment to verify the application's behavior in a production-like setting.

     • Manual Approval:

       • A manual approval step where a team member reviews the changes in the staging environment and decides whether to proceed with deployment to production.

     • Deploy to Production:

       • If the manual approval is granted, the artifacts are deployed to the production environment.

       • The application is now live for end-users.

4. Rollback Mechanism:

   • The pipeline includes a rollback mechanism in case issues are discovered in the production environment after deployment.

   • If a problem occurs, the team can trigger a rollback to the previous version of the application.

5. Monitoring and Logging:

   • Throughout the pipeline, monitoring and logging tools are used to track the progress of each stage and capture any issues or anomalies.

   • Metrics related to application performance, errors, and resource utilization are monitored in both staging and production environments.

6. Automation Tool:

   • Jenkins, a popular CI/CD automation tool, is used to define and execute the pipeline stages and tasks.

In this example, the CI/CD pipeline automates the process from code changes to deployment, ensuring that the application is built, tested, and deployed consistently. The staging environment acts as a testing ground before changes are applied to the production environment, and the pipeline includes mechanisms for both manual approval and rollback to maintain control and reliability.

So Now we understand the basic let's get started with step by step on how to setup the ci/cd pipeline with Jenkins.

Let's Start

For the first step, you need to choose a machine; you can either use your own machine or any virtual server for hosting your application and building the pipeline. Since I want it to be real, I am going with an EC2 instance (an AWS service).

Step 1 : Setting up a EC2 instance

Setting up an ec2 instance is pretty straight forward.

1. Go to AWS console AWS Amazon and sign in with your account.

2. Go to EC2 and click on launch instance.

3. 

   Now give a name, select your machine type, create or select a key pair(for SSH login), rest you can select according to your project infrastructure or use default settings. (I used free tier ubuntu)

4. 

   Launch the instance and wait for some minutes to boot up the server.

5. 

   Great! Instance is successfully created, we completed the first step.

Step 2: Connect with the instance.

Connect with ec2 instance is also a very easy step. We going to connect with our server using ssh keys.

1. Just click on the instance id.

2. 

   Click on connect on the upper-right.

3. 

   Go to the SSH client, where there are clear instructions on how to connect through SSH to the instance.

4. 

5. Now after giving the permission to our key pair and running the command, we will be connected to our instance, like this.

6. 

7. Great! Step 2 is done.

Step 3: Installing Jenkins

For installing Jenkins on our ec2 instance, java is necessary because Jenkins is built using Java and runs as a Java application.

1. Updating and Upgrading the system.(will take some time Approx.: 2-3 min)

    sudo apt-get update -y
   

   Running sudo apt-get update on a fresh EC2 Linux instance is a best practice to ensure that your package manager has the latest information about available software packages.

    sudo apt-get upgrade -y
   

   sudo apt-get upgrade is used to install the latest versions of all packages currently installed on the system. It does not install new packages; instead, it upgrades the existing ones to their latest versions.

2. Java Installation

    sudo apt-get install openjdk-17-jre -y
   

   I am installing open-jdk version 17 as it is the latest. You may use different one according to when you are reading this article. For checking support policy for Jenkins Visit Support Policy Java Jenkins .

3. Jenkins Installation

    sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
      https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
   

    echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
      https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
      /etc/apt/sources.list.d/jenkins.list > /dev/null
   

    sudo apt-get update -y
   

    sudo apt-get install jenkins -y
   

   Run these 4 command one by one
   OR

   create a script called install_jenkins.sh

   Add the following content

    #!/bin/bash
   
    sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
      https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
   
    echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
      https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
      /etc/apt/sources.list.d/jenkins.list > /dev/null
   
    sudo apt-get update
    sudo apt-get install jenkins
   

   Make the script executable:

    chmod +x install_jenkins.sh
   

   Run the Script:

    ./install_jenkins.sh
   

   This will run all four commands sequentially.

   Visit Jenkins Installation Linux for more info.

4. Jenkins has been successfully installed.

Step 4: Django Project Setup

1. Create a virtual env for your project in your local machine or wherever you primarily storing the code like GitHub or Bitbucket etc.

    python3 -m venv env
   

   Activate it

    source env/bin/activate
    # OR
    env/Scripts/activate
   

2. Create a requirements.txt file

    django
    gunicorn
   

   Add all the dependencies in it which is required to run your project for eg: if you using API, add djangorestframework , pillow (if your models have image field or file field ) etc.

3. Install the requirements in your virtual environment (Optional: if you want to run your project locally)

    pip install -r requirements.txt
   

   check the installed package by

    pip freeze
   

4. Add .gitignore

    env
    .idea
    # Byte-compiled / optimized / DLL files
    __pycache__/
    *.py[cod]
    *$py.class
   

   Note : Don't forget to add .env file or any file which is storing any kind of DB config or API keys.

Step 5: Starting Jenkins

Our Jenkins is installed in the server, and let's start it.

• Starting the Jenkins server

    sudo systemctl daemon-reload
  

    sudo systemctl start jenkins
  

    sudo systemctl status jenkins
  

  

Accessing the Jenkins through browser

Jenkins uses port 8080 for web access. When you install Jenkins and start the Jenkins server, it will listen on port 8080 for incoming HTTP requests.

By default, Jenkins uses 8080

For accessing Jenkins on the browser, copy your machine Ip address, in our case we are using ec2 instance so we need to give the permission to expose the port (and if you are doing this on your local machine you can skip this process), so copy the address and go to this url.

http://your-server-ip:8080
or if using local machine
http://localhost:8080

In our case, the security is not setup for port 8080. let's set it up in aws console.

• Go to security tab.

• Add the rule for 8080 like this

Now try to access the URL.

Unlocking the Jenkins

 sudo cat /var/lib/jenkins/secrets/initialAdminPassword

copy the password and enter it.

And select Install Suggested Plugins

Looks like the installation is successful.

Step 6: Adding env setup file

Let's make the environment setup file to setup the virtual environment and logs.

Make a file called envsetup.sh

#!/bin/bash

# Check if virtualenv is installed
if command -v virtualenv &> /dev/null; then
    echo "virtualenv is already installed."
else
    echo "Installing virtualenv......"
    sudo apt install -y python3-virtualenv
fi


if [ -d "env" ]
then
    echo "Python virtual environment exists."
else
    echo "Creating a virtual environment"
    virtualenv env
fi

echo "The current directory"
echo $PWD
echo -e "\n\n\n"

echo "Activating the virtual environment"
source env/bin/activate
echo -e "\n\n\n"


# Check if pip3 is installed
echo "Checking for pip3 installation"
if command -v pip3 &> /dev/null; then
    echo "pip3 is already installed."
else
    echo "Installing pip3......."
    sudo apt install -y python3-pip
fi

echo -e "\n\n\n"
echo "Installing Requirements...."
pip3 install -r requirements.txt
echo "Requirements Installed."
echo -e "\n\n\n"

echo "Checking for logs"
if [ -d "logs" ]
then
    echo "Log folder exists."
else
    echo "Creating Logs"
    mkdir logs
    touch logs/error.log logs/access.log
fi

echo -e "\n\n\n"
echo "Giving Permission"
sudo chmod -R 777 logs
echo -e "\n\n\n"
echo "*********Script Ended************"

File Hierarchy

Let me explain what we did inside this file

1. Check for the existence of a Python virtual environment (env):

• Checks if the directory named "env" (presumably a Python virtual environment) exists.

• If it exists, prints a message stating that the virtual environment exists.

• If it doesn't exist, creates a Python virtual environment using python3 -m venv env.

1. Activate the Python virtual environment:

   • Activates the virtual environment using source env/bin/activate.

2. Install Python dependencies from requirements.txt:

   • Uses pip3 to install the dependencies listed in the requirements.txt file.

3. Set permissions for the "logs" folder:

   • Changes the permissions of the "logs" folder and its contents to allow read, write, and execute permissions for everyone (chmod -R 777 logs).

   • Note: Using chmod 777 provides full read, write, and execute permissions to everyone, which may have security implications

Feel free to change any of the code according to your convenience.

Step 7: Adding Jenkinsfile

Now create a file named "Jenkinsfile".

What is Jenkinsfile?

A Jenkinsfile is a text file that defines the steps or stages of a Jenkins Pipeline. A Jenkins Pipeline is a suite of plugins that supports implementing and integrating continuous delivery pipelines into Jenkins.

Let's create a pipeline, create a file named Jenkinsfile

pipeline{
    agent any
    stages {

        stage('Setup Python Virtual ENV for dependencies'){

      steps  {
            sh '''
            chmod +x envsetup.sh
            ./envsetup.sh
            '''}
        }
        stage('Setup Gunicorn Setup'){
            steps {
                sh '''
                chmod +x gunicorn.sh
                ./gunicorn.sh
                '''
            }
        }
        stage('setup NGINX'){
            steps {
                sh '''
                chmod +x nginx.sh
                ./nginx.sh
                '''
            }
        }
    }
}

Note: We will add gunicorn.sh and nginx.sh file later. Also I will explain why to use Nginx and Gunicorn.

Adding gunicorn setup file

Create a file name as "gunicorn.sh"

What is gunicorn?

Gunicorn (Green Unicorn) is a WSGI (Web Server Gateway Interface) server for running Python web applications.

Imagine you have a Python web application, like a website built with a framework such as Flask or Django. Now, when someone wants to visit your website, their browser sends a request to your server, asking for the webpage. Here's where Gunicorn comes into play:

• Gunicorn acts like a traffic cop for your website. It takes the incoming requests from users and directs them to the appropriate part of your Python web application.

• Gunicorn is good at handling many requests at the same time. It's like having multiple waiters at a restaurant. While one waiter takes an order, others can serve food to different tables, making the overall service faster.

• Production WSGI servers are designed to handle multiple requests concurrently, making them suitable for scaling in a production environment. Django's development server is single-threaded and not optimized for handling a large number of simultaneous requests.

It handles incoming requests, manages many requests at once, and ensures smooth updates or restarts. It's the waiter that takes orders (requests) and ensures the chef (your web app) serves the food (webpages) efficiently, even when the restaurant (your website) is busy.

#!/bin/bash

source env/bin/activate

cd /var/lib/jenkins/workspace/directory_name/project_name

python3 manage.py makemigrations
python3 manage.py migrate
python3 manage.py collectstatic -- no-input

echo "Migrations done"

cd /var/lib/jenkins/workspace/directory_name

sudo cp -rf gunicorn.socket /etc/systemd/system/
sudo cp -rf gunicorn.service /etc/systemd/system/

echo "$USER"
echo "$PWD"



sudo systemctl daemon-reload
sudo systemctl start gunicorn

echo "Gunicorn has started."

sudo systemctl enable gunicorn

echo "Gunicorn has been enabled."

sudo systemctl restart gunicorn


sudo systemctl status gunicorn

Now let's add the gunicorn.socket and gunicorn.service file.

gunicorn.socket

[Unit]
Description=gunicorn socket

[Socket]
ListenStream=/run/gunicorn.sock
# Our service won't need permissions for the socket, since it
# inherits the file descriptor by socket activation
# only the nginx daemon will need access to the socket
SocketUser=www-data
# Optionally restrict the socket permissions even more.
# SocketMode=600

[Install]
WantedBy=sockets.target

gunicorn.service

[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target

[Service]
User=root
Group=www-data
WorkingDirectory=/var/lib/jenkins/workspace/django-cicd/app
ExecStart=/var/lib/jenkins/workspace/django-cicd/env/bin/gunicorn 
           --workers 1 
           --log-level debug 
           --error-logfile /var/lib/jenkins/workspace/django-cicd/error.log 
           --bind unix:/run/gunicorn.sock app.wsgi:application

[Install]
WantedBy=multi-user.target


[Install]
WantedBy=multi-user.target

Change the paths according to your project structure. Don't worry i will tell how to know the path and all later. You can continue without updating the path.

Step 8: Reverse Proxy(Nginx) Setup

Now let's setup nginx as our reverse proxy but for those who don't know

What is Nginx and Why we are using?

Nginx is a popular open-source web server and reverse proxy server that is widely used for hosting websites and applications. It is known for its high performance, stability, and scalability.

When using Django in a production environment, it's common to deploy it behind a combination of Nginx and Gunicorn. Here's why this combination is popular:

1. Nginx as a Reverse Proxy: Nginx can serve as a reverse proxy, handling client requests and forwarding them to Gunicorn. This allows Nginx to handle tasks like SSL termination, static file serving, and load balancing. Nginx is particularly efficient at serving static files, so offloading this task from Gunicorn can improve overall performance.

2. Handling Static Files: Django is a powerful web framework, but it may not be as efficient as dedicated web servers like Nginx at serving static files (e.g., images, stylesheets, JavaScript). By using Nginx to serve static content, Gunicorn can focus on handling dynamic content and application logic.

Nginx is configured to act as a reverse proxy for your Django application served by Gunicorn. It handles static files, forwards dynamic requests to Gunicorn, and provides additional features like load balancing and security

Now let's back to work.

Create a file named "nginx.sh".

#!/bin/bash

sudo cp -rf app.conf /etc/nginx/sites-available/app
chmod 710 /var/lib/jenkins/workspace/django-cicd

sudo ln -s /etc/nginx/sites-available/app /etc/nginx/sites-enabled
sudo nginx -t

sudo systemctl start nginx
sudo systemctl enable nginx

echo "Nginx has been started"

sudo systemctl status nginx

Now create an app.conf file

server {
  listen 80;
  server_name IP;
  error_log /var/lib/jenkins/workspace/django-cicd/logs/error.log;
  access_log /var/lib/jenkins/workspace/django-cicd/logs/access.log;

  location = /favicon.ico { access_log off; log_not_found off; }

  location /static/ {
    root /var/lib/jenkins/workspace/django-cicd;
  }

  location / {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_pass http://unix:/run/gunicorn.sock;
    }
}

change the ip with the ec2 instance ip.

Installing NGINX

sudo apt-get install nginx

sudo systemctl start nginx

sudo systemctl enable nginx

sudo systemctl status nginx

Now if status is active it means, it is setup properly.

Step 9: Configuring The Pipeline

The Jenkins is already installed and unlocked, so let's get started with creating our first Admin User.

1. Access Jenkins through browser.

2. Fill the form

3. 

   Configure the URL(am using as it is.)

4. 

   Click on Start using Jenkins.

5. Now you will be on dashboard.

Now upload your project on any VCS like GitHub or Bitbucket. And now we are good to go with configuring our first pipeline for the project.

1. Go to the dashboard.

2. 

   Click on New Item.

3. Enter the name of your project and click on pipeline.

4. 

   Now In General section, give a meaningful description and use option Discard Old Builds.

5. 

   You can leave it everything as it is and go to Pipeline.

6. Give information like this.

7. 

   For credentials, create a personal access token and copy it.

   

   And Paste it like this in Jenkins.

8. 

   Provide the branch name and click on save.

   

   Later, i will tell on how to setup for auto build trigger on code push.

Step 10: Giving access to Jenkin User

We used a lot of sudo in our scripts. And our user is Jenkins and it doesn't have sudo privilege to run those commands also we are not providing any password.

so let's add Jenkins to our sudoers list.

1. Run this command in your server.

    sudo vi /etc/sudoers
   

2. Copy this line

    jenkins ALL=(ALL) NOPASSWD: ALL
   

3. Paste like this and save it.

   

Step 11: Refactoring the code

Now, this is a crucial step for efficiently building a pipeline. If you have followed all the steps with me, I am confident that you may encounter errors when starting the build.

Let's address and fix some common issues:

1. File Path of Your Project: Ensure that the file paths in your scripts and configuration files accurately reflect the structure of your project. Copy the paths from the console output if needed, and update them in your scripts.

2. Minor Bugs during Installation or Virtual Environment Activation: Examine the console output for any errors during the installation or activation of the virtual environment. If you come across minor bugs, investigate the error messages, search for solutions online, and consider making manual installations if necessary.

3. Remove the default ngnix conf : It's possible that your Nginx is still utilizing the default settings, leading to the display of the default Nginx page in the browser.

4. Giving Permission to static folder: Sometimes, Nginx may lack permission to use the static folder, leading to broken pages without styles or images. Ensure the static folder has proper permissions for Nginx to access and serve its contents."

By addressing these issues, you'll enhance the robustness of your pipeline and increase the chances of a successful build.

File path of your project

In our project, there are approximately seven script files that use commands such as python manage.py makemigrations or migrate, requiring the correct file path for your project.

In the bonus tip section, I will provide instructions on how to set this up for an enterprise-level application.

Now, run the build, examine the file path, and update it according to our project.

Now select the build no. and go to console output.

Copy the file path from the console output and examine it. You can navigate to this location to check for errors. Reference the path and update it in the scripts everywhere.

First, make changes in app.conf (two changes). Replace the file path with your project path.

Now in gunicorn.service file (2 changes)

in gunicorn.sh file (2 changes)

in ngnix.sh file (1 change)

After all these changes , your build should be successful.

Minor bugs during the installation or activating the virtual environment

If your build is not successful even after changing the file paths, examine the errors in the console output. Search the internet for solutions or consider installing the required tools manually.

If you encounter challenges activating the virtual environment, try doing it manually by navigating to the project location and creating a virtual environment from CLI.

Remove the default ngnix conf

1. Go to /etc/ngnix/sites-enabled/

2. Delete the default config.

    rm -rf default
   

3. Now go to /etc/ngnix/sites-available/

    rm -rf default
   

4. Great ! We successfully deleted the default config and ngnix will use your app config.

Giving Permission to static folder

The www-data user is commonly used by Nginx to run its worker processes. These worker processes handle incoming HTTP requests, and Nginx uses this user for security reasons. It's a good practice to have the Nginx worker processes run with a less privileged user like www-data to limit potential security risks.

sudo chown -R :www-data /var/

By following all these steps and addressing potential issues, your build should be successful. If you encounter any further challenges or errors during the process, carefully review the console output, search for solutions, and consider adapting the instructions based on the specific requirements and configurations of your project. Remember to thoroughly test your pipeline to ensure its reliability and efficiency.

🎉 Congratulations! 🎉We have successfully setup the project.

Step 12: Accessing The Project on Browser.

Now, we need to ensure that our project is accessible on the internet. To achieve this, let's navigate to the AWS console. We'll need to allow traffic to view our application through HTTP or HTTPS, which typically uses ports 80 and 443. Let's proceed with the necessary configurations.

Go to security

Click on Security Groups

Edit the inbound rules

Add rules like this.

Now try to access your app through the browser like this

If you encounter any issues, double-check the security group rules, the status of your Django application, and any potential firewall settings. Additionally, ensure that your AWS EC2 instance has the necessary IAM roles and permissions for internet access.

🎉 Congratulations on successfully setting up your pipeline with Jenkins, Nginx, Gunicorn, and other components! Building and configuring a continuous integration and deployment (CI/CD) pipeline can be a complex task, so completing the setup is a significant achievement. 🚀💻

Bonus Tips

I'd like to share some bonus tips that I believe are crucial for setting up a pipeline in a corporate or tech giant environment. Having worked with esteemed companies such as IBM, Papa John's, and eBay, I feel qualified to offer insights based on the advice I received from my experienced colleagues.

1. Auto Build Trigger On Code Push

   Automatic triggering ensures that builds are initiated immediately upon code push. This provides developers with rapid feedback on the impact of their changes, helping identify and address issues early in the development lifecycle.

   • Go to Configure

• Provide the GitHub Project URL

• Go to Build Trigger and Select GitHub hook trigger from GITScm Polling

• Go to your project on GitHub > Settings > Webhooks

• 

  Add webhook

• 

  Fill the form like this, provide the server-url

• 

  Click on Add webhook

• Great! We have successfully setup the Auto Build Trigger

1. Use S3 Bucket for static file serving

   Serving static files directly from S3 reduces the load on your Django server. Nginx can be configured to serve static files from S3, allowing the Django server to focus on processing dynamic requests.

   When deploying updates to your Django project, you don't need to transfer static files to multiple servers. The static files hosted on S3 are accessible from any server, simplifying the deployment process.
   When combined with Nginx, Amazon S3 provides a robust and scalable solution for handling static files and media in a Django project. It optimizes performance, simplifies deployment, and enhances the overall scalability of your application.

   Many corporations and large enterprises use Amazon S3 in conjunction with Django and Nginx, especially for handling static files and media storage in their web applications. Amazon S3 is a highly scalable, durable, and cost-effective object storage service provided by Amazon Web Services (AWS). Its features and benefits make it an attractive choice for various use cases, including those in corporate environments.

   I'm planning to craft a dedicated article on seamlessly integrating Amazon S3 with a Django project. Writing the details here might unintentionally extend this conversation.

2. Optimizing Jenkins Security: Preventing Builds on the Built-In Node
   Avoid executing any builds on the default built-in node by taking the following steps: Navigate to Manage Jenkins > Manage Nodes and Clouds, choose Built-In Node from the list, select Configure from the menu, set the number of executors to 0, and save the configuration. Ensure that you establish clouds or build agents for executing builds; otherwise, build initiation will not be possible.

   Why is this a crucial practice? By default, Jenkins is configured to execute builds on the built-in node for the sake of simplicity during initial setup. However, this approach is not recommended for the long term. Running builds on the built-in node grants them the same level of access to the controller file system as the Jenkins process. To mitigate security risks, it is highly recommended to abstain from running any builds on the built-in node and, instead, utilize agents (either statically configured or provided by clouds) to carry out builds.

3. Remove unnecessary permissions:
   Eliminate redundant permissions for the 'authenticated users' group by navigating to Manage Jenkins > Configure Global Security > Authorization > Authenticated users. It is crucial to follow this step because, in alignment with the preceding section, even when employing a Matrix-based authorization method, it is imperative to prevent the default 'authenticated users' group from possessing unnecessary permissions. Whenever feasible, revoke any excessive permissions assigned to this default group and allocate permissions specifically to the groups or users that you have defined."

4. Dockerize the Django Project

   Docker eliminates the "it works on my machine" problem by packaging your application and its dependencies into a single container. This reduces the likelihood of issues arising from differences in development and production environments.

   • Use a Docker image with Python and install necessary dependencies for your Django application. Run Gunicorn as the application server inside the Docker container to serve your Django app.

   • Set up a separate Docker container with Nginx as the reverse proxy. Configure Nginx to forward requests to the Gunicorn server. This separation allows Nginx to handle static file serving efficiently.

   • By dockerizing your Django project and integrating it with Nginx, Gunicorn, and Jenkins, you establish a reliable and portable infrastructure that fosters consistency, scalability, and efficient collaboration throughout the development and deployment lifecycle.

5. Add the test - Continuous Integration

   Every project has some test suite to test the application, the build should be successful, only if all the tests are passing, so you can run the test suite and add it as a stage in Jenkins.

    python3 manage.py test
   

   Create a File like testrunner.sh

    #!/bin/bash
   
    cd /var/lib/jenkins/workspace/django-cicd/
   
    source env/bin/activate
   
    cd your_project/
    python3 manage.py test
   

   Add it in the pipeline.

    pipeline {
        agent any
        stages {
            stage('Setup Python Virtual ENV for dependencies') {
                steps {
                    sh '''
                    chmod +x envsetup.sh
                    ./envsetup.sh
                    '''
                }
            }
            stage('Test Suite') {
                steps {
                    sh '''
                    chmod +x testrunner.sh
                    ./testrunner.sh
                    '''
                }
            }
            stage('Setup Gunicorn Setup') {
                steps {
                    sh '''
                    chmod +x gunicorn.sh
                    ./gunicorn.sh
                    '''
                }
            }
            stage('Setup NGINX') {
                steps {
                    sh '''
                    chmod +x nginx.sh
                    ./nginx.sh
                    '''
                }
            }
        }
    }
   

   This Jenkins pipeline script has successfully outlined a streamlined workflow for your Django project. We started by setting up a Python virtual environment for dependencies, ran a comprehensive test suite to ensure code integrity, and seamlessly integrated Gunicorn and NGINX for production-ready deployment. Embracing Continuous Integration (CI) principles, the script automates the testing phase, ensuring that code changes are validated consistently. Additionally, when combined with other stages, it lays the foundation for a powerful Continuous Deployment (CD) pipeline, automating the delivery process up to the production environment. By implementing this Jenkins pipeline, you're not just coding; you're orchestrating a robust and efficient development lifecycle.

Conclusion

In this guide, we've unlocked the power of Jenkins CI/CD for Django, seamlessly orchestrating builds, tests, and deployments on an EC2 instance. 🛠️ With Nginx as our reverse proxy and Gunicorn as the WSGI server, our Django projects now thrive in a secure and high-performance environment.

As you wrap up this journey, remember: Continuous improvement is the heartbeat of effective development. 🔄 Keep iterating, refining, and adapting your CI/CD pipeline to fuel the evolution of your Django projects.

Cheers to streamlined development, collaborative coding, and the endless possibilities that Jenkins unfolds! 🎉 Happy coding, and may your Django endeavors soar to new heights! 🚀

If you are a beginner or intermediate in using Linux, you must know some basic commands like cd, ls, sudo, rmdir etc. And I think these commands are ok but there is a powerful command which user might ignore or don't use due to its toughness or unawareness

When it comes to command-line text processing in Linux, AWK stands out as a versatile and powerful tool. In this article, we'll take a closer look at a basic AWK command and explore its capabilities in extracting specific information from text files.

Scenarios

Before you start learning any command or tool, it's important to understand how it can actually help you in real-life situations, especially in your daily tasks. If something seems fancy but doesn't have a practical use, it might not be worth learning.

Learning should be about gaining skills that make your everyday life easier and more efficient, tackling the challenges you face regularly. So, before diving in, consider whether what you're learning has a meaningful impact on your day-to-day activities.

So that's why am sharing some real use case or scenario where awk command could be really helpful.

1. Server Log Analysis:

   • Scenario: Analyzing Apache web server or any web server logs to extract information about the number of requests for each page.

     Assuming you have an Apache log file named access.log with entries like

       192.168.0.1 - - [01/Jan/2022:12:00:00 +0000] "GET /page1 HTTP/1.1" 200 1234
       192.168.0.2 - - [01/Jan/2022:12:01:00 +0000] "GET /page2 HTTP/1.1" 404 5678
       192.168.0.3 - - [01/Jan/2022:12:02:00 +0000] "GET /page1 HTTP/1.1" 200 7890
     

     You can use the awk command to extract and count the number of requests for each page

     Output

          2 /page1
          1 /page2
     

2. CSV File Manipulation:

   • Scenario: Consider you have a CSV file named data.csv with the following content.

       Name,Age,Occupation
       Alice,28,Engineer
       Bob,35,Developer
       Charlie,22,Student
       David,40,Manager
     

     Now, let's say you want to extract and display only the "Name" and "Occupation" columns. You can use the following AWK command:

     Output

   •             Name Occupation
                 Alice Engineer
                 Bob Developer
                 Charlie Student
                 David Manager
     

3. Password File Analysis:

   • Scenario: Extracting and displaying user information from the /etc/passwd file.

4. Network Configuration Review:

   • Scenario: Analyzing the output of the ifconfig command to display network interface details.

     For e.g. ifconfig commands give you this output.

       eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
               inet 192.168.0.2  netmask 255.255.255.0  broadcast 192.168.0.255
               inet6 fe80::a00:27ff:fe8a:e2ab  prefixlen 64  scopeid 0x20<link>
               ether 08:00:27:8a:e2:ab  txqueuelen 1000  (Ethernet)
               RX packets 1501  bytes 1666445 (1.6 MB)
               RX errors 0  dropped 0  overruns 0  frame 0
               TX packets 1275  bytes 96476 (96.4 KB)
               TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
     
       lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
               inet 127.0.0.1  netmask 255.0.0.0
               inet6 ::1  prefixlen 128  scopeid 0x10<host>
               loop  txqueuelen 1000  (Local Loopback)
               RX packets 9  bytes 546 (546.0 B)
               RX errors 0  dropped 0  overruns 0  frame 0
               TX packets 9  bytes 546 (546.0 B)
               TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
     

     By using awk command, you can extract and display network interface details.

     Output

       Interface: eth0 IP Address: 192.168.0.2
       Interface: lo IP Address: 127.0.0.1
     

5. Custom Log Parsing:

• Scenario: Parsing a custom log format to extract relevant information.

  Let's consider a custom log file named custom.log with entries like:

    2022-01-01T12:00:00+00:00 | User=John | Action=Login | Status=Success
    2022-01-01T12:15:00+00:00 | User=Alice | Action=Logout | Status=Success
    2022-01-01T12:30:00+00:00 | User=Bob | Action=Login | Status=Failure
  

  Assuming you want to extract and display the date, user, action, and status information, by using awk you can get the output like this

  Output

    Date: 2022-01-01T12:00:00+00:00 User: John Action: Login Status: Success
    Date: 2022-01-01T12:15:00+00:00 User: Alice Action: Logout Status: Success
    Date: 2022-01-01T12:30:00+00:00 User: Bob Action: Login Status: Failure
  

These examples demonstrate how AWK can be applied in various real-world scenarios for data extraction, processing, and analysis in a Linux or Unix environment. The flexibility and power of AWK make it a valuable tool for system administrators, developers, and data analysts.

There are numerous use cases where you can use AWK command efficiently without much writing. I believe now we are good to go for learning this thing.

What is AWK?

Awk is a command line utility or program or scripting language or text processing utility that means you give it some text and it can grab certain columns, certain rows, certain fields from the text for you. You can tell it to go search for certain string, patterns in the text and even replace those string patterns with other strings. Its a really powerful program and that's why developers and engineer mostly use all the time.

"As a developer working in big tech companies, i probably overused awk. I used awk everywhere especially in my shell scripting because I'm really comfortable with it and it's one of those programs that once you learn awk, you wonder why you didn't learn it sooner because it's such a powerful program. " by Vishnu Tiwari

AWK is particularly well-suited for text processing and is commonly used in Unix and Unix-like operating systems for tasks such as data extraction, reporting, and text pattern matching. Being a command-line utility, it allows users to efficiently perform these tasks by executing commands in a terminal environment.

Basics - Syntax + Examples

Hey, let's head it to the basics and explore the awk command for printing the columns, to use input and output field separator.

Syntax

awk options 'selection _criteria {action }' input-file > output-file

Examples

ps

Output

ps | awk '{print $1}'

print $1 - print the first column , change col no by putting the no. after $. like $2 for second column.

ps | awk '{print $0}'

$0 - simply prints everything just like cat

Output

So one of the files that people love to use awk on GNU/Linux system is the /etc/passwd file, that is a file that lists all the users on your Linux system

cat /etc/passwd

the output looks like this,

there's not a lot of spaces to it, there are columns i mean it is separated into columns but the columns they are separated by colons here.

so for printing all the users!

awk -F ":" '{print $1}' /etc/passwd

-F : for providing field separator, by default awk uses spaces as field, it basically splits the column by ':'.

if you want to print multiple columns, add the column no.

awk -F ":" '{print $1 $6 $7}' /etc/passwd

its not very readable because we didn't tell it to separate the columns with spaces or colons or anything. We told it, hey print 1,6 and 7 column.

awk -F ":" '{print $1 "\t" $6 "\t" $7}' /etc/passwd

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704533913460/70aba778-05c4-43f4-b8b7-858075ed178e.png align="left")

In the output, we can see its showing tab spaces after all column and it's much more readable that way.

Now other than specifying a field separator to search for and use you know to determine what the columns are, you can actually print out the field separator as well and you can tell it to change the field separator to a different character as part of the output. you can do all this by using this command.

awk 'BEGIN{FS=":"; OFS="-"} {print $1, $6, $7}' /etc/passwd

BEGIN{FS=":"; OFS="-"}: This part of the command is executed before processing any input lines. It sets the input field separator (FS) to a colon (:) and the output field separator (OFS) to a dash (-).

Intermediate - AWK for Daily + Official Use

In this level, we going to upgrade a one level up from the basics and learn about patterns

• if you want to get the last column, you can use $NF for printing the last column , for example , you want to print all the shells present in your machine

cat /etc/shells

As you can see in the output, the first line is a comment, which is not a valid shell. Therefore, we only need the lines that start with a forward slash '/'. To achieve this, we can use a command like the one provided.

awk -F '/' '/^\// {print $NF}' /etc/shells

// - Anything inside these two forward slashes (//) will be used by AWK to search for patterns.

^ - it is the anchor, used to Indicates the beginning of the line

\/ - uses a backslash to tell AWK that the next character, a forward slash, is not a closing slash.

But in the output, you can see duplicate entries, which don't look good. We can use 'uniq' to specify that we don't need duplicates.

awk -F '/' '/^\// {print $NF}' /etc/shells | uniq

for sorting

awk -F '/' '/^\// {print $NF}' /etc/shells | uniq | sort

And let's run the 'df' command because it is another common command that provides nice columned information, and people often enjoy using AWK on its output

df

for printing the only tmpfs system,

df | awk '/^tmpfs/'

You can also perform operations in the columns like addition, multiplication, division etc.

df | awk '/^tmpfs/ {print $1 "\t" $2 + $3}'

You can also perform operations based on certain conditions

cat /etc/shells

For printing a line less than 8 characters

awk 'length($0) < 8' /etc/shells

for printing shell less than 5 character

Advance - AWK For Pro Developers

In this level, we going to upgrade a one more level up from the intermediate and learn about if/else, loops etc.

if / else in awk

Syntax

awk '{
    if (condition) {
        print $1 ";
    } else {
        print $2;
    }
}' input_file

Example

ps -ef # For printing all the process in your system

now we can check if the process is kworker . for printing all the kworker process, we can do

ps -ef | awk '{if ($NF~ /kworker/) print $0}'

Now we can also distinguish that which is a kworker process or which one is regular by using else.

ps -ef | awk '{if ($NF~ /kworker/){ print $NF "\t kworker process"} else {print $NF "\t regular process"}}'

For loop

Since, awk is an scripting language we can also use loops, In AWK, there are two main types of loops: the for loop and the while loop.

awk '{
    for (i = 1; i <= 5; i++) {
        print "Number:", i;
    }
}' input_file

If you use this syntax, you need to mandatorily pass a input file, it basically run the loop for each line

for just performing the operations without passing the input file, you can use BEGIN.

ifconfig | awk '/^[a-zA-Z]/{interface=$1; next} /inet addr:/{print "Interface:", interface, "IP:", $2}'

while loop

In AWK, you can use a while loop to repeatedly execute a block of code as long as a certain condition is true. Here's a simple example of using a while loop in AWK

awk '{
    i = 1;
    while (i <= 5) {
        print "Number:", i;
        i++;
    }
}' input_file

In this example:

• i = 1: Initialization of the loop variable i to 1.

• while (i <= 5): The condition for the loop to continue as long as i is less than or equal to 5.

• print "Number:", i: Code inside the loop that prints the current value of i.

• i++: Incrementing i by 1 in each iteration.

else if

! In AWK, you can use if and else if statements to implement conditional logic. Here's an example:

awk '{
    if ($1 > 10) {
        print $1 " is greater than 10";
    } else if ($1 == 10) {
        print $1 " is equal to 10";
    } else {
        print $1 " is less than 10";
    }
}' input_file

In this example, the AWK script reads input from input_file (you can replace it with your actual file name), and for each line, it checks the value in the first column ($1). Depending on the value, it prints a different message.

Explanation:

• if ($1 > 10): If the value in the first column is greater than 10, execute the corresponding block of code.

• else if ($1 == 10): If the value is equal to 10, execute this block of code.

• else: If none of the above conditions are true, execute this block.

You can adjust the conditions and actions inside the blocks to suit your specific requirements.

Some Common Useful Commands For Daily Use

1. substr

   The substr function in AWK is used to extract a portion of a string. Its basic syntax is:

    substr(string, start[, length])
   

   • string: The input string from which you want to extract a substring.

   • start: The position in the string where extraction begins. The position is 1-based.

   • length (optional): The number of characters to extract. If omitted, it extracts the substring from the start position to the end of the string.

for example , you have a file and you want to use substr on it, the actual use is depend on the layout of the file , but for demo see this example

    awk '{print substr($0, 3)}' numbered.txt

1. match, RSTART and RLENGTH

   In AWK, the match function is used to search a string for a specified pattern and sets the values of RSTART and RLENGTH to the starting position and length of the matched substring, respectively. The basic syntax of match is as follows:

    match(string, regexp)
   

   • string: The input string where you want to search for the pattern.

   • regexp: The regular expression pattern to search for in the string.

Here's an example:

    awk 'match($0,/o/) {print $0, "Has O character at index", RSTART, "with length", RLENGTH}' numbered.txt

1. NR
   In AWK, NR is a built-in variable that represents the current record (line) number being processed. It is automatically incremented by AWK as it reads each input line. NR is especially useful when you want to perform actions based on the line number.

    df | awk 'NR==6, NR==8 {print NR".", $0}'
   

   

   for checking the line count of any file!

    df | awk 'END {print NR}'
   

   

2. For checking IP Addresses
   Using ifconfig with AWK can be useful for parsing and extracting specific information about network interfaces. ifconfig provides details about the network configuration on a system, and AWK can help filter and format this information according to your needs.

    ifconfig | awk  '/^[a-zA-Z]/ {print "Interface: " $1 } /inet/ {print "IP Addrress:" $2}'
   

   

Conclusion

In conclusion, our exploration of AWK has provided a comprehensive understanding of this powerful text processing tool. From its fundamental syntax to advanced features, we've delved into the intricacies that make AWK a versatile and indispensable asset in the realm of data manipulation and analysis.

Throughout the article, we've uncovered how AWK can be applied in various real-world scenarios, demonstrating its utility in tasks ranging from simple text processing to more complex data transformations. The practical examples and use cases serve as a guide for both beginners and experienced users, showcasing the flexibility and efficiency of AWK in handling diverse text-based challenges.

As we bring this AWK journey to a close, I extend my gratitude for your dedication in navigating through the intricacies of this scripting language. The knowledge gained here lays the foundation for enhanced productivity and problem-solving capabilities. Whether you're a newcomer or a seasoned AWK enthusiast, may the skills acquired pave the way for seamless text processing endeavors.

That's a wrap on this article, where we journeyed from AWK basics to advanced ninja moves and explored its everyday applications. Kudos for sticking with it till the end! Much love and catch you at the next article. Stay sharp and keep rocking that tech game! ✌️

Thank you again for your time and commitment. Here's to harnessing the full potential of AWK in your future endeavors. Until our paths cross again in the realm of knowledge exploration, stay curious and keep scripting!

The Django Data Migration Tutorial is a step-by-step guide to seamlessly transfer data between databases and deploy your Django application to different environments without losing critical data. This tutorial focuses on flexibility, allowing users to switch between different database backends such as SQLite, MySQL, or any other supported by Django.

This article provides a comprehensive tutorial on Django Data Migration, guiding you through the process of transferring data between databases and deploying Django applications without data loss. It emphasizes flexibility, enabling users to work with various database backends like SQLite and MySQL

Key-Features

• Database Switching: Easily switch between different database backends without data loss.

• Step-by-Step Guide: Follow a comprehensive guide to migrate data smoothly.

• Compatibility: Applicable to various databases supported by Django.

• Deployment Readiness: Ensure your application is ready for deployment in different environments.

Personal Experience

Throughout my tenure as a backend developer at IBM Corp, I encountered a situation where we populated our development database (PostgreSQL - version 14) with numerous records and users with diverse user types.

When it came to configuring the QA environment for both manual and automation testing, the testing teams expressed the need for a substantial amount of data to facilitate their testing and comprehension processes.

Recognizing the potential time constraints and inefficiencies associated with manual data input, I did some research and sought guidance from experienced colleagues, I successfully conducted a data migration from the development database to the test database, excluding specific tables. In this article, I will share the insights gained from this experience, providing a guide on how to seamlessly transfer data between databases.

Transferring Data to another DB in a Django Application

Helps In Switch to another Deploying Env. without Losing Data

Step 1: Generating a JSON file which includes your DB content

Imagine your app is a treasure chest, and your data is the treasure. To capture all that valuable info, there's a superhero command: python manage.py dumpdata > data.json. This command works like a camera, snapping a pic of your entire database and saving it in a file called "data.json." No need to worry about how complex your app is – this command is your data snapshot wizard.

Run this command on your terminal or cmd

python manage.py dumpdata > data.json

Note: For excluding any models use "--exclude"

python manage.py dumpdata --exclude auth.permission --exclude contenttypes > data.json

Step 2: Add your new database to the settings.

Add the new database to your settings.py file like this and run all your migrations for creating the table structure and DB schema.

Note: Don't hardcode the details of your db instead create a .env file(add this file to .gitignore)

Make the migrations now, use the default command for migration if u removed the old DB and named it 'default'.

Run the migrations

python manage.py migrate --database=new

Step 3: Convert the data.json to UTF-8 and Load your data into new db

Your data.json is of UTF-16LE and you need to convert it into UTF-8 and than load your data to new database.

You can change it online or while saving as select the file type to UTF-8.

 python manage.py loaddata data.json --database=new

Step 4: All data is succesfully transferred to new DB

This process is useful for creating backups, migrating data between databases, or sharing sample data with others working on the project. Keep in mind that the **dumpdata** and **loaddata** commands are part of Django's serialization framework, and they work with various serialization formats, not just JSON.

Conclusion

In conclusion, the process of migrating data from one database to another in Django emerges as a pivotal skill for developers seeking efficiency and seamlessness in their projects.

As we close this chapter, armed with the knowledge of how to navigate database migrations, developers can confidently ensure data continuity across various stages of their application's lifecycle. Here's to smooth migrations and the continued evolution of robust, data-driven Django applications!