I had already explored EchoAPI earlier, but while working on my recent project, I revisited it. And guess what? They’ve supercharged it with AI-powered features.

This update caught my attention immediately:

• ✅ One-click Docs

• ✅ Fake data generator

• ✅ Bulk parameter updates

• ✅ AI Convert

• ✅ AI Test-case generator

• ✅ AI-powered script & assertion generator

In this blog, I’ll take you through my full experience, step by step, so you’ll know exactly how these features can save you hours of manual testing.

Installing and Setting Up EchoAPI

Before we dive into the AI features, let’s start with the basics.

I downloaded and installed EchoAPI’s desktop app. The interface is clean and very intuitive, something between Postman and Insomnia, but with its own unique AI touch.

For this demo, I connected my FastAPI e-commerce backend. This app has typical endpoints like:

• POST /signup

• POST /login

• GET/cart

• POST /cart/add

• POST /order/place

• POST /payment

• GET/products

• GET/products/{product_id}

Perfect playground to test EchoAPI’s AI magic.

AI-Generated Test Cases

Let’s start with the AI Test-case Generator.

Normally, writing test cases for each endpoint is a tedious process:

• You have to decide scenarios.

• Write expected outcomes.

• Prepare data for each run.

• And then execute them manually.

But EchoAPI just makes it one-click.

Step 1: Pick an Endpoint

Inside EchoAPI, I selected an endpoint from my FastAPI project. Let’s say the signup endpoint.

Step 2: Go to the Cases Section

Here, EchoAPI shows 8 standard testing dimensions (like API Specification Compliance Tests, Invalid Input Handling Tests, HTTP Method Validation, etc.). I kept all of them selected.

Step 3: Generate

I clicked Generate, and instantly, the AI created:

• • Case Objectives (e.g., “Test signup with valid data”, “Test signup with missing email”)

    * Expected Outcomes (e.g., “Should return 200 OK”, “Should return 400 Bad Request”)

    * Test Data (realistic values automatically generated)

Step 4: Apply & Run

When I clicked Apply and Test, EchoAPI ran 13 cases with 35 different sets of data.

That’s a massive workload handled automatically. Imagine writing these manually; it could easily take a couple of hours per endpoint.

And here’s the best part: if something fails, EchoAPI shows why it exactly failed, with every detail in the response section.

Tip: You can re-run failed cases separately to debug faster.

AI-Powered Scripts & Assertions

Sometimes you need more than automated test cases; you want custom scripts and validation rules. EchoAPI covers that too.

AI Script Generator

Let’s say you want a test script for checking a complex payment flow. Instead of writing it yourself, you just ask the AI:

I typed “Generate a script to test a payment API with both valid and invalid card details.”

EchoAPI writes it out, and the script is ready to run.

You can edit these AI-generated scripts or chain them with your own logic if needed.

AI Assertions

Assertions are what validate your API’s response. Doing this manually is slow, especially when responses are large.

Example:

• You send a request.

• You get a JSON response like the following:

{ "status": "success", "order_id": "12345" }

Now, instead of manually writing like this:

assert response.status == "success"
assert "order_id" in response

You just click AI Assert in the response section, and EchoAPI generates these checks for you.

Even better, you can add or modify assertions in plain English:

“Assert that the response returns status 201 and user_id should not be empty.”

And it updates your tests. This makes it super friendly even for non-coders who just want to validate business logic.

Chaining Multiple APIs

Real-world apps rarely run on a single endpoint. You usually need a workflow: signup → login → add-to-cart → place-order → make-payment.

EchoAPI lets you chain APIs together so you can test them all in one flow.

Here’s how I did it:

Step 1: Create Folder & Subfolder

First, go to the “Tests” section. Here, I created a folder in EchoAPI for my test flows, then a subfolder for this specific checkout flow.

Step 2: Add Endpoints by Reference

I selected my endpoints into this folder by reference (not by copy). This way, if my endpoint changes, it stays automatically synchronized here.

Step 3: Configure Each Endpoint

• Signup Endpoint: Entered new user data and saved.

• 

• Login Endpoint: Used the same credentials. This gives me access_token.

• 

• • Captured the access token from the response and stored it as a variable using JSONPath expression in the Post-response section.

    • 

• Add-to-Cart Endpoint: Added a product. Passed the access token in the Auth header.

• 

• Place-Order Endpoint: Placed the order. Passed the access token in the Auth header. Captured the order ID and stored it as a variable in the Post-response section.

• 

• Make-Payment Endpoint: Used the captured order ID and passed the access token in Auth header to complete the payment.

• 

Step 4: Run the Flow

After choosing the preferred environment, I saved everything and ran the whole chain.

Result? EchoAPI executed the flow step by step, showing me where things passed or failed.

This is a game-changer; instead of manually juggling values between requests, everything flows automatically.

EchoAPI vs Postman & Thunder Client: AI Features Comparison

When it comes to developer tools, positioning matters, not just what features you have, but how they stand out and why someone should choose your solution over existing ones. Let’s break down how EchoAPI compares with Postman and Thunder Client.

Postman: AI Features & Strengths

Postman has long been the industry leader for API testing and collaboration, and over time, they’ve been adding AI capabilities. Some highlights include:

• Postbot (AI Assistant): Helps debug APIs, write tests, and analyze large responses or datasets with ease.

• AI Agent Builder: Lets you build agent-like workflows by combining AI models with APIs. You can design drag-and-drop visual workflows, use templates, and even tap into Postman’s growing network of APIs and MCP servers.

• Flows, MCP & AI Developer Tools: Postman supports experiments with AI models, evaluating multiple LLMs, generating prompts, and building test suites.

• Autocomplete, Suggestions & Docs Assistance: Useful for writing tests, analyzing large responses, and keeping documentation in sync.

Limitations:

• Test-case generation isn’t fully automatic: Postbot can help, but often requires manual tweaking.

• Workflows like chaining APIs, extracting variables, or setting up environments still require a lot of manual setup.

• Postman’s depth comes with complexity, meaning a steeper learning curve and heavier overhead for new users.

Thunder Client: AI Features & Strengths

Thunder Client takes a different approach: lightweight API testing inside VS Code. Its philosophy is simplicity and speed.

• Scriptless Testing & GUI Assertions: Instead of writing code, you can define tests via dropdowns and GUI selectors (e.g., status codes, response fields).

• Core Features: Collections, environments, request history, and local storage — all the basics you need for quick API testing.

• AI Integration (Early Stages): Thunder Client has announced AI-generated tests as “coming soon”, meaning their AI feature set is still developing.

• CI/CD & Git Sync: Useful for teams to collaborate, share collections, and integrate tests into workflows.

Limitations:

• AI support is limited and still maturing.

• Not ideal for complex workflows like chained requests or dynamic variable management.

• Lacks no-code flow builders or advanced automation beyond simple assertions.

EchoAPI’s Edge

Here’s where EchoAPI differentiates itself: AI isn’t just an add-on; it’s the engine of the testing workflow.

At a Glance

• Postman: Best for power users who want deep control, integrations, and community-driven workflows.

• Thunder Client: Great for developers who prefer simplicity, speed, and testing inside VS Code.

• EchoAPI: Combines the best of both worlds: advanced AI automation, full workflow chaining, variable management, and a friendly UI that works for both coders and non-coders.

Why This Comparison Matters

There are plenty of API tools out there. But the real value is in how much time you save, how many bugs you prevent, and how easily your team can scale test coverage.

• If you’re working on a big codebase, managing dozens of endpoints, or testing chained workflows, EchoAPI gives you automation that cuts down boilerplate and speeds up debugging.

• If you’re working on a small project, Thunder Client might be enough for its simplicity. If you want the biggest ecosystem and integrations, Postman is still a solid choice.

• But if you want AI-native testing that actually handles the heavy lifting, EchoAPI offers a strong alternative; one that’s already delivering mature, practical automation today.

Why This Matters

EchoAPI’s AI features literally compress hours of manual work into seconds.

• AI Test-case generator: No need to brainstorm test cases.

• AI Scripts: No need to code repetitive test scripts.

• AI Assertions: Validation without writing manual checks.

• Chained APIs: End-to-end workflow testing in one go.

For developers, it means faster testing and less frustration. For QA engineers, it means scaling test coverage without extra effort. For teams, it means fewer bugs slip into production.

Results Summary

After spending time with EchoAPI’s AI-powered tools, here’s my honest takeaway:

• The auto test-case generation is the highlight; it gives you complete coverage in seconds.

• The AI script & assertion generator saves tons of time when building complex validations.

• The multi-API chaining feels like real-world automation without the complexity of writing a giant test framework yourself.

Wrapping Up

EchoAPI with AI isn’t just another API client. It’s like having a QA assistant sitting beside you.

Instead of writing, organizing, and debugging everything manually, you just guide the AI and let it handle the heavy lifting.

That’s my full walkthrough of EchoAPI’s AI features. Have you tried AI-assisted testing yet?

That’s it for now…

Keep Coding✌✌

PyPy claims that it is almost a drop-in replacement for CPython and can beat it on speed and memory usage. It supports libraries like Django, NumPy, Scikit-learn, and Twisted, and comes with the stackless mode, providing micro-threads for massive concurrency.

In this article, we’ll see how to install and use PyPy in our system.

Installation

PyPy comes with Python3 and Python2. Click here to download the pre-built binary that is compatible with your system.

After downloading PyPy, you’ll get a compressed folder in your system. Now extract the content of the PyPy’s build folder into the desired path, and that’s it.

But how to use it because we didn’t add PyPy’s path to system variables like we used to do in the case of CPython installation?

How to Use PyPy?

It is recommended to use PyPy using a virtual environment. Let’s see how we can use PyPy from a virtual environment.

We’ll use virtualenv package to create a virtual environment. It is compatible with both CPython and PyPy. You’ll need to install this package (pip install virtualenv) to get started.

Open the command prompt or terminal and run the following command:

PS > virtualenv -p D:/SACHIN/pypy310/pypy.exe pypyenv

Let’s understand what the above command does:

• virtualenv: A command to create a virtual environment using virtualenv.

• -p: Specifies the path to the Python interpreter that will be used in the virtual environment. It can also be written --python.

• D:/SACHIN/pypy310/pypy.exe: Path to PyPy interpreter.

• pypyenv: Species the name of the virtual environment.

Run the command and PyPy will be ready to use.

You can check if you are using PyPy by running the following command:

PS > cd pypyenv
PS > Scripts/activate
(pypyenv) PS > cd ..
(pypyenv) PS > python --version
Python 3.10.14 (39dc8d3c85a7, Aug 27 2024, 14:33:33)
[PyPy 7.3.17 with MSC v.1929 64 bit (AMD64)]

You’ll notice that we used python command to check the version of PyPy installed on the system because in the virtual environment, the python command becomes a symlink (or equivalent) to the interpreter used when creating the virtual environment.

Installing Modules Using PyPy

The following command is used to install third-party modules using PyPy:

PS > pypy -m pip install module_name

You can directly install modules without writing pypy -m, if you are using PyPy as the main interpreter. If you are using PyPy within an isolated environment then activate the environment and use the above command to avoid any error.

Some third-party libraries such as Scipy create problems during the installation, so in that case, you can use conda with pypy3.9.

That’s all for now.

Keep Coding✌✌

In this article, we’ll discuss the functions that are as useful as any function in the os module but are rarely used.

os.path.commonpath()

When working with multiple files that share a common directory structure, you might want to find the longest shared path. os.path.commonpath() does just that. This can be helpful when organizing files or dealing with different paths across environments.

Here’s an example:

import os

paths = ['/user/data/project1/file1.txt', '/user/data/project2/file2.txt']
common_path = os.path.commonpath(paths)
print("Common Path:", common_path)

This code will give us the common path shared by these two paths.

Common Path: /user/data

You can see that os.path.commonpath() takes a list of path names, which might be impractical to manually write them down.

In that case, it is best to iterate over all of the directories, subdirectories, and file names and then look for the common path.

import os

def get_file_paths(directory, file_extension=None):
    # Collect all file paths in the directory (and subdirectories, if any)
    file_paths = []
    for root, dirs, files in os.walk(directory):
        for file in files:
            if file_extension is None or file.endswith(file_extension):
                file_paths.append(os.path.join(root, file))
    return file_paths


# Specify the root directory to start from
directory_path = 'D:/SACHIN/Pycharm/Flask-Tutorial'

# If you want to filter by file extension
file_paths = get_file_paths(directory_path, file_extension='.html')

# Find the common path among all files
if file_paths:
    common_path = os.path.commonpath(file_paths)
    print("Common Path:", common_path)
else:
    print("No files found in the specified directory.")

In this example, the function get_file_paths() traverses a directory from top to bottom and appends all the paths found in the file_paths list. This function optionally takes a file extension if we want to look out for specific files.

Now we can easily find the common path of any directory.

Common Path: D:\SACHIN\Pycharm\Flask-Tutorial\templates

os.scandir()

If you’re using os.listdir() to get the contents of a directory, consider using os.scandir() instead. It’s not only faster but also returns DirEntry objects, which provide useful information like file types, permissions, and whether the entry is a file or a directory.

Here’s an example:

import os

with os.scandir('D:/SACHIN/Pycharm/osfunctions') as entries:
    for entry in entries:
        print(f"{entry.name} : \n"
              f">>>> Is File: {entry.is_file()} \n"
              f">>>> Is Directory: {entry.is_dir()}")

In this example, we used os.scandir() and passed a directory and then we iterated over this directory and printed the info.

.idea : 
>>>> Is File: False 
>>>> Is Directory: True
main.py : 
>>>> Is File: True 
>>>> Is Directory: False
sample.py : 
>>>> Is File: True 
>>>> Is Directory: False

os.path.splitext()

Let’s say you’re working with files and need to check their extension, you can get help from os.path.splitext() function. It splits the file path into the root and extension, which can help you determine the file type.

import os

filename = 'report.csv'
root, ext = os.path.splitext(filename)
print(f"Root: {root} \n"
      f"Extension: {ext}")

Output

Root: report 
Extension: .csv

Look at some cases where paths can be weird, at that time how os.path.splitext() works.

import os

filename = ['.report', 'report', 'report.case.txt', 'report.csv.zip']
for idx, paths in enumerate(filename):
    root, ext = os.path.splitext(paths)
    print(f"{idx} - {paths}\n"
          f"Root: {root} | Extension: {ext}")

Output

0 - .report
Root: .report | Extension: 
1 - report
Root: report | Extension: 
2 - report.case.txt
Root: report.case | Extension: .txt
3 - report.csv.zip
Root: report.csv | Extension: .zip

os.makedirs()

There's already a frequently used function that allows us to create directories. But what about when you create nested directories?

Creating nested directories can be a hassle with os.mkdir() since it only makes one directory at a time. os.makedirs() allows you to create multiple nested directories in one go, and the exist_ok=True argument makes sure it doesn’t throw an error if the directory already exists.

import os

os.makedirs('project/data/files', exist_ok=True)
print("Nested directories created!")

When we run this program, it will create specified directories and sub-directories.

Nested directories created!

If we run the above program again, it won’t throw an error due to exist_ok=True.

os.replace()

Similar to os.rename(), os.replace() moves a file to a new location, but it safely overwrites any existing file at the destination. This is helpful for tasks where you’re updating or backing up files and want to ensure that old files are safely replaced.

import os

os.replace(src='main.py', dst='new_main.py')
print("File replaced successfully!")

In this code, main.py file will be renamed to new_main.py just as os.rename() function but this operation is like take it all or nothing. It means the file replacement happens in a single, indivisible step, so either the entire operation succeeds or nothing changes at all.

File replaced successfully!

os.urandom()

For cryptographic purposes, you need a secure source of random data. os.urandom() generates random bytes suitable for things like generating random IDs, tokens, or passwords. It’s more secure than the random module for sensitive data.

os.urandom() uses randomness generated by the operating system you are using from various resources to make bytes (data) unpredictable.

In Windows, it uses BCryptGenRandom() to generate random bytes.

import os

secure_token = os.urandom(16)  # 16 bytes of random data
print("Secure Token:", secure_token)
#Making it human-readable
print("Secure Token:", secure_token.hex())

Output

Secure Token: b'\x84\xd6\x1c\x1bKB\x7f\xcd\xf6\xb7\xc4D\x92z\xe3{'
Secure Token: 84d61c1b4b427fcdf6b7c444927ae37b

os.path.samefile()

The os.path.samefile() function in Python is used to check if two paths refer to the same file or directory on the filesystem. It’s particularly helpful in scenarios where multiple paths might point to the same physical file, such as when dealing with symbolic links, hard links, or different absolute and relative paths to the same location.

import os

is_same = os.path.samefile('/path/to/file1.txt', '/different/path/to/symlink_file1.txt')
print("Are they the same file?", is_same)

os.path.samefile() is designed to return True only if both paths reference the same file on disk, such as a file that’s hard-linked or symlinked to the same data on the filesystem.

os.path.relpath()

os.path.relpath() is a computation function that computes the relative path between two paths. This is particularly useful when building file paths dynamically or working with relative imports.

Consider the following example:

import os

# Target file path
target_path = "D:/SACHIN/Pycharm/osfunctions/project/engine/log.py"
# Starting point
start_path = "D:/SACHIN/Pycharm/osfunctions/project/interface/character/specific.py"

relative_path = os.path.relpath(target_path, start=start_path)
print(relative_path)

In this example, we have target_path which contains a path where we have to navigate and start_path contains a path from where we have to start calculating the relative path to target_path.

When we run this, we get the following output.

..\..\..\engine\log.py

This means we have to go up three directories and then down to engine/log.py.

os.fsync()

When we perform a file writing (file.write()) operation, the data isn’t saved to disk instantly instead the data is saved into the system’s buffer and if something unexpected happens before writing the data to the disk, the data gets lost.

os.fsync() forces the data to be written, ensuring data integrity. It’s especially useful in logging or when writing critical data that must not be lost.

import os

with open('data.txt', 'w') as f:
    f.write("gibberish!")
    os.fsync(f.fileno())  # Ensures data is written to disk

os.fsync(f.fileno()) is called to make sure the data is immediately written to the disk and not left in the buffer.

os.fsync() takes file descriptor that’s why we passed f.fileno() which is a unique integer assigned by the system to the file on which we are operating.

os.get_terminal_size()

If you’re creating CLI tools, formatting the output to fit the terminal width can make the output cleaner. os.get_terminal_size() gives you the current terminal width and height, making it easy to dynamically format content.

import os

size = os.get_terminal_size()
print(f"Terminal Width: {size.columns}, Terminal Height: {size.lines}")

When we run this code in the terminal, we get the size of the terminal on which we are running this script.

PS > py sample.py
Terminal Width: 158, Terminal Height: 12

Note: You may get an error when directly running the script on IDE where the program doesn’t have access to the terminal.

🏆Other articles you might be interested in if you liked this one

✅Streaming videos on the frontend in FastAPI.

✅How to fix circular imports in Python.

✅Template inheritance in Flask.

✅How to use type hints in Python?

✅How to find and delete mismatched columns from datasets in pandas?

✅How does the learning rate affect the ML and DL models?

That’s all for now.

Keep Coding✌✌.

In this article, we’ll see how to stream videos on frontend in FastAPI.

StreamingResponse

Stream Local Video

FastAPI provides a StreamingResponse class that is dedicated to streaming purposes. The StreamingResponse class takes a generator or iterator and streams the response.

Here’s a simple example of streaming local video to the browser.

# localvid.py
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

# Video path
vid_path = 'sample_video.mp4'

# Function to stream local video
def stream_local_video():
    with open(vid_path, 'rb') as vid_file:
        yield from vid_file

# Path to stream video
@app.get("/")
def video_stream():
    return StreamingResponse(stream_local_video(), media_type='video/mp4')

We imported the StreamingResponse class from the fastapi.responses module.

In the video_stream() path operation function, we returned the response using StreamingResponse. We passed the generator function stream_local_video() and media_type as an argument to the StreamingResponse class.

The generator function stream_local_video() reads the bytes of the video and yield from iterates over the bytes and each part iterated is then yielded.

The following command will run the server and stream the video on http://127.0.0.1:8000/.

> fastapi dev localvid.py

 ╭────────── FastAPI CLI - Development mode ───────────╮
 │                                                     │
 │  Serving at: http://127.0.0.1:8000                  │
 │                                                     │
 │  API docs: http://127.0.0.1:8000/docs               │
 │                                                     │
 │  Running in development mode, for production use:   │
 │                                                     │
 │  fastapi run                                        │
 │                                                     │
 ╰─────────────────────────────────────────────────────╯

Stream Online Video

# onlinevid.py
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import requests

app = FastAPI()

# Video URL
vid_url = 'https://cdn.pixabay.com/video/2023/07/28/173530-849610807_large.mp4'

# Function to stream online video
def stream_online_video(url):
    response = requests.get(url, stream=True)
    for portion in response.iter_content(chunk_size=1024*1024):
        yield portion

# Path to stream video
@app.get("/")
def video_stream():
    return StreamingResponse(stream_online_video(vid_url), media_type='video/mp4')

In this example, we used the requests library to fetch the content of the video from the URL and we set the stream=True (avoids reading the video at once into memory). Then we iterated and yielded the video content in chunks (1024 bytes at a time).

When we run the server, it will serve the video from the URL.

Instead of a hardcoded URL, we can specify the URL in the endpoint.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import requests

app = FastAPI()

# Function to stream online video
def stream_online_video(url):
    response = requests.get(url, stream=True)
    for portion in response.iter_content(chunk_size=1024*1024):
        yield portion

# Path to stream video
@app.get("/")
def video_stream(url):
    return StreamingResponse(stream_online_video(url), media_type='video/mp4')

In this example, we modified the path operation function video_stream() to accept a URL.

Now we can specify the URL of the video in the endpoint in the following format.

http://127.0.0.1:8000/?url=https://cdn.pixabay.com/video/2023/07/28/173530-849610807_large.mp4

We can also make the path operation function asynchronous using async.

FileResponse

Stream Local Video

The FileResponse class simply takes a file and streams the response.

from fastapi import FastAPI
from fastapi.responses import FileResponse

app = FastAPI()

# Video path
vid_path = 'video.mp4'

# Path to stream video
@app.get("/")
def video_stream():
    return FileResponse(vid_path, media_type='video/mp4')

In the above example, we simply passed a video file in the FileResponse class and returned the response.

We didn’t read and iterate over the bytes of the video to stream it like we did in the case of StreamingResponse.

The FileResponse class is ideal for relatively small or medium-sized files as it loads the file in the memory. Large files will consume more memory.

Stream Online Video

Streaming video from the URL using FileResponse isn’t the same as streaming the local video. As we know, the FileResponse class takes a file and streams it.

from fastapi import FastAPI
from fastapi.responses import FileResponse
import requests
import os

app = FastAPI()

# Video URL
vid_url = 'https://cdn.pixabay.com/video/2023/07/28/173530-849610807_large.mp4'
local_vid = 'sample_video.mp4'

# Function to save video from URL
def save_as_file(url, path):
    if not os.path.exists(path):
        response = requests.get(url)
        with open('sample_video.mp4', 'wb') as vid:
            vid.write(response.content)

save_as_file(vid_url, local_vid)

# Path to stream video
@app.get("/")
def video_stream():
    return FileResponse(local_vid, media_type='video/mp4')

In this example, instead of streaming the local video, we streamed the video from the URL using FileResponse.

This isn’t the best practice because we first saved the video from the URL in the local machine and then streamed the video.

The FileResponse class can be best utilized for local files, not web content.

That was all about streaming video directly to the browser but we don’t want to do that every time, instead, we want the video to be streamed on our website’s landing page.

Stream Video on Frontend

So, how to stream video on the frontend using FastAPI? We need to create a frontend using HTML and then stream the video.

First, create a directory named serve_video or whatever you want to name it, and create the following files and sub-directories.

serve_video/
   - templates/
      - display_video.html
   - app.py

The app.py file will contain our backend code and the display_video.html file will contain frontend code.

app.py

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse, HTMLResponse
from fastapi.templating import Jinja2Templates
import requests

app = FastAPI()
templates = Jinja2Templates(directory="templates")

vid_urls = [
    "https://cdn.pixabay.com/video/2023/07/28/173530-849610807_large.mp4",
    'https://cdn.pixabay.com/video/2024/03/31/206294_large.mp4',
    'https://cdn.pixabay.com/video/2023/10/11/184510-873463500_large.mp4',
    'https://cdn.pixabay.com/video/2023/06/17/167569-837244635_large.mp4'
]

# Stream the video from the URL
def stream_video(url):
    response = requests.get(url, stream=True)
    for portion in response.iter_content(chunk_size=1024*1024):
        yield portion

# Endpoint to render the HTML template with the video player
@app.get("/", response_class=HTMLResponse)
async def video_template(request: Request):
    return templates.TemplateResponse("display_video.html", {"request": request})

# Endpoint to stream the video
@app.get("/video/{video_id}")
async def video_stream(vid_id: int):
    if 0 <= vid_id < len(vid_urls):
        return StreamingResponse(stream_video(vid_urls[vid_id]), media_type="video/mp4")
    else:
        return HTMLResponse("Video not found", status_code=404)

In this example, we are streaming multiple videos on the frontend. We have a list of video URLs stored within vid_urls.

We have a stream_video() generator function that yields the video in smaller chunks.

We created an asynchronous path operation function video_template() that returns a response from HTML (@app.get("/", response_class=HTMLResponse)) file.

We are serving HTML from a template stored within the templates directory, so, we set up a Jinja2 template directory where HTML templates are stored (templates = Jinja2Templates(directory="templates")). This is where FastAPI will look for .html files.

Note: To render templates in FastAPI, you need to install the Jinja2 (pip install jinja2) library.

Then we returned the response using templates.TemplateResponse("display_video.html", {"request": request}). We passed our HTML template display_video.html and along with it, a dictionary containing Request object ({"request": request}). This Request object provides information about the incoming HTTP request, such as headers, cookies, and URLs, which can be accessed within the HTML template.

Next, we created an endpoint (@app.get("/video/{video_id}")) to stream individual videos based on the ID in the vid_ulrs. The video_stream(vid_id: int) path operation functions accept the index of the video in vid_urls.

Within the video_stream(), the if condition checks if the vid_id is within the range, if not, then raises an error otherwise the video is streamed based on the specified index.

display_video.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Stooooockzz</title>
</head>
<body>
    <h1>What we do?</h1>
    <p>We create awesome stock videos for free usage</p>
    <h3>Video Samples</h3>
    <div style="display: flex; justify-content: space-evenly;">
        <video width='20%' height='30%' style="border-radius: 10px;" controls autoplay>
            <source src="/video/0" type="video/mp4">
        </video>
        <video width='20%' height='30%' style="border-radius: 10px;" controls autoplay>
            <source src="/video/1" type="video/mp4">
        </video>
        <video width='20%' height='30%' style="border-radius: 10px;" controls autoplay>
            <source src="/video/2" type="video/mp4">
        </video>
        <video width='20%' height='30%' style="border-radius: 10px;" controls autoplay>
            <source src="/video/3" type="video/mp4">
        </video>
    </div>
</body>
</html>

This HTML file contains multiple <video> tags with the source pointing to the /video/{video_id} endpoint that streams the video based on ID.

Here, we passed <source src="/video/0" type="video/mp4"> within the <video> tag that represents streaming the first video from the vid_urls (in app.py) list. Similarly, we specified the same source but changed the ID within each <video> tag.

> fastapi dev app.py

Upon running the server using the above command, we get the following response on the frontend.

Conclusion

We’ve used StreaminResponse and FileResponse classes to stream local and web videos directly on the browser, and along with this we’ve also streamed multiple videos using StreamingResponse which was rendered using HTML files on the browser.

In this article, we’ve used URLs of the video residing on the internet but you can also fetch URLs from databases, local files, disks, etc.

That’s all for now.

Keep Coding✌✌.

Connect Database with Python

First, we need to install a MySQL database driver called PyMySQL that will help us bridge Python and MySQL.

pip install PyMySQL

Now, after installing this package, we can start the connection process by importing it into our Python script.

Create a Python file, let’s say db_connection.py, and write the following code.

import pymysql

# Establish MySQL connection with Python
sql_connection = pymysql.connect(
    host="localhost",
    user="root",
    password="xxxxxxxxx",
    database="consumer",
    cursorclass=pymysql.cursors.DictCursor
)

print("Connected to database.")

# Create cursor to interact with database
cursor = sql_connection.cursor()

# Create an entry in the database
cursor.execute(
      """INSERT INTO consumers (name, address, phone)
      values ('Max', '21 St. Parkway', 871231);"""
)

# Save the changes in the database
sql_connection.commit()

# Fetch data from the database
cursor.execute(
      """SELECT * FROM consumers;"""
)

data = cursor.fetchall()
print(data[0])

# Close the connection with database
sql_connection.close()

print("Operation successful.")

This code creates a connection to the MySQL database and then writes an entry in the database table.

First, the package is imported which will help us bridge Python and MySQL using Python code.

Next, we established a connection using pymysql.connect() and passed the required arguments:

• hostname: The location of the MySQL database. We have a MySQL server in the local system hence, we passed "localhost".

• user: Username of the MySQL server. In this case, we passed "root", which is a default name.

• password: Password of the MySQL server.

• database: The name of the database with which you want to connect. In this case, we are connecting to the "consumer" database on the server.

• cursorclass: This decides the format of the result returned by the cursor. In this case, the cursor will return results in a dictionary (DictCursor) format.

Then we created a cursor (sql_connection.cursor()) object that helps in executing SQL queries.

Using cursor.execute(), we executed an SQL query to insert an entry in the consumers table of the consumer database.

Then we saved the changes to the database using sql_connection.commit().

We then executed an SQL query to get all the data from the consumers table in the database.

Using cursor.fetchall(), we fetched all the entries from the table and printed the first result.

Finally, we closed the connection with the database using sql_connection.close().

Now, when we run this code, we’ll get the following result.

Connected to database.
{'name': 'Max', 'address': '21 St. Parkway', 'phone': 871231}
Operation successful.

That’s all for now.

Keep Coding✌✌.

Insert Multiple Entries in the Database

We’ll see how we can insert multiple entries in the two different databases SQLite and MySQL.

SQLite

Create a Python file and write the following code inside it.

import sqlite3

# Create and connect with database
connection = sqlite3.connect("cars.db")

# Create cursor to interact with database
cursor = connection.cursor()

# Create database table
cursor.execute(
    """CREATE TABLE IF NOT EXISTS car (brand text, model text)"""
)

entries = [
    ('Ferrari', '812 Superfast'),
    ('Porsche', 'Carrera GT'),
    ('BMW', 'i8'),
    ('BMW', 'X7'),
    ('Ferrari', '488 GTB')
]

# Insert entries in the database table
cursor.executemany(
    """INSERT INTO car (brand, model) VALUES (?, ?)""", entries
)

# Commit the changes
connection.commit()

# Fetch the result from the database table
result = cursor.execute(
    """SELECT * FROM car"""
)

# Print the entries in the table
data = result.fetchall()
for each in data:
    print(each)

# Close the database connection
connection.close()

This code imports the sqlite3 library that helps create and interact with the SQLite database.

We created and connected (sqlite3.connect("cars.db")) to the cars.db database and then created a cursor (connection.cursor()) to interact with the database.

Using the cursor, we executed an SQL query to create a table named car with two columns: brand and model in the database.

Then, we have a list of tuples containing the car information stored in the entries variable.

Now if you look closely, we used executemany() that will run for every parameter (entries). This single line is executed five times for five different entries.

We used question mark (?) in the SQL query as a placeholder for the values. In SQLite, question mark (?) is used as a placeholder but for MySQL or PostgreSQL, this will be different.

Next, the changes were saved using connection.commit().

Another SQL query was executed to select all of the data from the table, which was then retrieved using fetchall() and printed.

Finally, the database connection was closed using connection.close().

('Ferrari', '812 Superfast')
('Porsche', 'Carrera GT')
('BMW', 'i8')
('BMW', 'X7')
('Ferrari', '488 GTB')

MySQL

Create a Python file and write the following code.

import pymysql

# Establish MySQL connection with Python
sql_connection = pymysql.connect(
    host="localhost",
    user="root",
    password="xxxxxx",
    database='cars',
    cursorclass=pymysql.cursors.DictCursor
)

# Create cursor to interact with database
cursor = sql_connection.cursor()

# Create database table
cursor.execute(
    """CREATE TABLE IF NOT EXISTS car (brand text, model text)"""
)

# Entries to be made
entries = [
    ('Ferrari', '812 Superfast'),
    ('Porsche', 'Carrera GT'),
    ('BMW', 'i8'),
    ('BMW', 'X7'),
    ('Ferrari', '488 GTB')
]

# Insert entries in the database table
cursor.executemany(
    """INSERT INTO car (brand, model) VALUES (%s, %s)""", entries
)

# Save the changes in the database
sql_connection.commit()

# Fetch data from the database table
cursor.execute(
      """SELECT * FROM car"""
)

data = cursor.fetchall()
for each in data:
    print(each)

# Close the connection with database
sql_connection.close()

In this code, we’ve used the PyMySQL database driver to connect with the MySQL database in Python.

We performed all the preliminary operations such as connecting to MySQL server and creating a cursor object using PyMySQL.

Then, we created a table named car with two columns within the cars database by executing an SQL query.

Next, we stored a list of tuples containing relevant data for the table in the entries variable.

Then we used executemany() to execute a single query repeatedly for each entry. This operation will insert every entry in the table stored in the entries variable.

In MySQL, we use (%s) as a placeholder for a value.

Next, the changes were saved using connection.commit().

Then, we selected all rows in the database table car by executing an SQL query. The selected rows were then fetched using fetchall() and printed.

{'brand': 'Ferrari', 'model': '812 Superfast'}
{'brand': 'Porsche', 'model': 'Carrera GT'}
{'brand': 'BMW', 'model': 'i8'}
{'brand': 'BMW', 'model': 'X7'}
{'brand': 'Ferrari', 'model': '488 GTB'}

We got the results in a dictionary format because we set the cursor class to DictCursor.

That’s all for now.

Keep Coding✌✌.

Let's see how and when to use them.

DataFrame.ffill()

The DataFrame.ffill() method fills the missing or NaN values using the previous valid value in a column or row.

Suppose we have the following dataset that contains missing values in each column.

import pandas as pd
import numpy as np

df = pd.DataFrame(
    {
        "Max Temperature": [22.7, 24.8, np.nan, np.nan, 29.9, np.nan, 45],
        "Min Temperature": [np.nan, 20.1, 18.4, np.nan, np.nan, np.nan, 15.5],
        "Avg Temperature": [21.4, 23.5, np.nan, 20.3, np.nan, 19.8, 20.4]
    }
)
print(df)
--------------------
   Max Temperature  Min Temperature  Avg Temperature
0             22.7              NaN             21.4
1             24.8             20.1             23.5
2              NaN             18.4              NaN
3              NaN              NaN             20.3
4             29.9              NaN              NaN
5              NaN              NaN             19.8
6             45.0             15.5             20.4

Now, we can use the DataFrame.ffill() method to fill in the missing values.

ffill_df = df.ffill()
print(ffill_df)
--------------------
   Max Temperature  Min Temperature  Avg Temperature
0             22.7              NaN             21.4
1             24.8             20.1             23.5
2             24.8             18.4             23.5
3             24.8             18.4             20.3
4             29.9             18.4             20.3
5             29.9             18.4             19.8
6             45.0             15.5             20.4

We can see that missing values are filled with preceding values, such as 24.8 in the second and third rows of the Max Temperature column and 29.9 in the fifth row of the Max Temperature column.

In the same manner, all the missing values are filled for the other two (Min Temperature and Avg Temperature) columns.

If you observe that the first row of Min Temperature remains NaN, it's because there was no preceding value to fill.

Setting the Limit

We can also set the limit to forward fill the number of consecutive NaNs by specifying the limit parameter.

limit_ffill = df.ffill(limit=1)
print(limit_ffill)
--------------------
   Max Temperature  Min Temperature  Avg Temperature
0             22.7              NaN             21.4
1             24.8             20.1             23.5
2             24.8             18.4             23.5
3              NaN             18.4             20.3
4             29.9              NaN             20.3
5             29.9              NaN             19.8
6             45.0             15.5             20.4

After setting the limit=1, just one NaN value is filled in the columns Max Temperature and Min Temperature when consecutive NaNs were present.

Limit Area

In pandas v2.2.0, a new parameter is added called the limit_area which is by default set to None. It can be set to 'inside' and 'outside'.

It is used with the limit parameter and if set to

• None: The default behavior with no restrictions. NaNs are filled with the last valid value, subject to any limit specified.

• inside: Fills only those NaNs that are surrounded by valid values.

• outside: Fills only those NaNs that are not surrounded by valid values.

Using ffill() with limit=1 and limit_area='inside'

print("Original Dataset")
print(df)
print("*"*55)

in_ffill = df.ffill(limit=1,limit_area='inside')
print(in_ffill)
--------------------
Original Dataset
   Max Temperature  Min Temperature  Avg Temperature
0             22.7              NaN             21.4
1             24.8             20.1             23.5
2              NaN             18.4              NaN
3              NaN              NaN             20.3
4             29.9              NaN              NaN
5              NaN              NaN             19.8
6             45.0             15.5             20.4
*******************************************************
   Max Temperature  Min Temperature  Avg Temperature
0             22.7              NaN             21.4
1             24.8             20.1             23.5
2             24.8             18.4             23.5
3              NaN             18.4             20.3
4             29.9              NaN             20.3
5             29.9              NaN             19.8
6             45.0             15.5             20.4

We can see that NaN values surrounded by valid values are filled, and because the limit was set to 1, just one NaN was filled from consecutive NaNs.

Using ffill() with limit=1 and limit_area='outside'

print("Original Dataset")
print(df)
print("*"*55)

out_ffill = df.ffill(limit=1,limit_area='outside')
print(out_ffill)
--------------------
Original Dataset
   Max Temperature  Min Temperature  Avg Temperature
0             22.7              NaN             21.4
1             24.8             20.1             23.5
2              NaN             18.4              NaN
3              NaN              NaN             20.3
4             29.9              NaN              NaN
5              NaN              NaN             19.8
6             45.0             15.5             20.4
*******************************************************
   Max Temperature  Min Temperature  Avg Temperature
0             22.7              NaN             21.4
1             24.8             20.1             23.5
2              NaN             18.4              NaN
3              NaN              NaN             20.3
4             29.9              NaN              NaN
5              NaN              NaN             19.8
6             45.0             15.5             20.4

Here, we can see that dataset isn't affected at all, because all the NaN values are surrounded by valid values. Let's tweak the dataset and see how it affects it.

df = pd.DataFrame(
    {
        "Max Temperature": [22.7, 24.8, np.nan, np.nan, 29.9, 45, np.nan],
        "Min Temperature": [np.nan, 20.1, 18.4, np.nan, np.nan, np.nan, 15.5],
        "Avg Temperature": [21.4, 23.5, np.nan, 20.3, np.nan, 19.8, np.nan]
    }
)
print("Original Dataset")
print(df)
print("*"*55)
out_ffill = df.ffill(limit=1,limit_area='outside')
print(out_ffill)

If we run this, we'll get this output.

Original Dataset
   Max Temperature  Min Temperature  Avg Temperature
0             22.7              NaN             21.4
1             24.8             20.1             23.5
2              NaN             18.4              NaN
3              NaN              NaN             20.3
4             29.9              NaN              NaN
5             45.0              NaN             19.8
6              NaN             15.5              NaN
*******************************************************
   Max Temperature  Min Temperature  Avg Temperature
0             22.7              NaN             21.4
1             24.8             20.1             23.5
2              NaN             18.4              NaN
3              NaN              NaN             20.3
4             29.9              NaN              NaN
5             45.0              NaN             19.8
6             45.0             15.5             19.8

Notice that in our original dataset (df), we kept NaN values in the seventh row of Max Temperature and Avg Temperature.

When we used df.ffill(limit=1,limit_area='outside'), the newly added NaN values were filled because they were not surrounded by valid values.

Filling Missing Values Across the Axis

By specifying the axis parameter, we can control how the missing data should be filled in.

If the axis is set to 0 or 'index', the missing values will be filled down each column, moving vertically (from above) along the rows. This means the last valid value from above (in the same column) will be used to fill in the NaN values below it.

row_ffill = df.ffill(axis=0)
print(row_ffill)
--------------------
   Max Temperature  Min Temperature  Avg Temperature
0             22.7              NaN             21.4
1             24.8             20.1             23.5
2             24.8             18.4             23.5
3             24.8             18.4             20.3
4             29.9             18.4             20.3
5             29.9             18.4             19.8
6             45.0             15.5             20.4

Every NaN value is filled in each column moving along the row (vertically). So we can say that this is the default operation.

If the axis is set to 1 or 'columns', the missing values will be filled across each row, moving horizontally (from left to right) along the columns. This means the last valid value from the left (in the same row) will be used to fill in the NaN values to the right of it.

col_ffill = df.ffill(axis=1)
print(col_ffill)
--------------------
   Max Temperature  Min Temperature  Avg Temperature
0             22.7             22.7             21.4
1             24.8             20.1             23.5
2              NaN             18.4             18.4
3              NaN              NaN             20.3
4             29.9             29.9             29.9
5              NaN              NaN             19.8
6             45.0             15.5             20.4

We can see that the third row of Avg Temperature is filled with 18.4, the value present left of it, and in the same manner, the fifth row of Min Temperature and Avg Temperature is filled with the value 29.9.

DataFrame.bfill()

The DataFrame.bfill() method fills the missing or NaN values using the next valid value in a column or row.

import pandas as pd
import numpy as np

df = pd.DataFrame(
    {
        "Max Temperature": [22.7, 24.8, np.nan, np.nan, 29.9, np.nan, 45],
        "Min Temperature": [np.nan, 20.1, 18.4, np.nan, np.nan, np.nan, 15.5],
        "Avg Temperature": [21.4, 23.5, np.nan, 20.3, np.nan, 19.8, 20.4]
    }
)

print("Original Dataset")
print(df)
print("*"*55)
bfill_df = df.bfill()
print(bfill_df)

We have a dataset (df) and we are filling the NaNs using the bfill() (backward filling).

Original Dataset
   Max Temperature  Min Temperature  Avg Temperature
0             22.7              NaN             21.4
1             24.8             20.1             23.5
2              NaN             18.4              NaN
3              NaN              NaN             20.3
4             29.9              NaN              NaN
5              NaN              NaN             19.8
6             45.0             15.5             20.4
*******************************************************
   Max Temperature  Min Temperature  Avg Temperature
0             22.7             20.1             21.4
1             24.8             20.1             23.5
2             29.9             18.4             20.3
3             29.9             15.5             20.3
4             29.9             15.5             19.8
5             45.0             15.5             19.8
6             45.0             15.5             20.4

We can see that the NaN values are filled with the next values, such as 45.0 and 29.9 in the Max Temperature column. In the same manner, all the NaN values are filled.

Notice that in the first row of Min Temperature, the NaN value gets filled.

Setting the Limit

The DataFrame.bfill() method also has the limit parameter to limit the number of consecutive NaN values filling backward.

bfill_df = df.bfill(limit=1)
print(bfill_df)
--------------------
   Max Temperature  Min Temperature  Avg Temperature
0             22.7             20.1             21.4
1             24.8             20.1             23.5
2              NaN             18.4             20.3
3             29.9              NaN             20.3
4             29.9              NaN             19.8
5             45.0             15.5             19.8
6             45.0             15.5             20.4

In the above code, the limit parameter is set to 1 so only one NaN value was filled when consecutive NaNs were present.

Limit Area

The limit_area parameter in DataFrame.bfill() method is the same as the DataFrame.ffill()'s limit_area parameter.

It is used with the limit parameter and if set to

• None: The default behavior with no restrictions. NaNs are filled with the last valid value, subject to any limit specified.

• inside: Fills only those NaNs that are surrounded by valid values.

• outside: Fills only those NaNs that are not surrounded by valid values.

Using bfill() with limit=1 and limit_area='inside'

print("Original Dataset")
print(df)
print("*"*55)
bfill_df = df.bfill(limit=1, limit_area='inside')
print(bfill_df)
--------------------
Original Dataset
   Max Temperature  Min Temperature  Avg Temperature
0             22.7              NaN             21.4
1             24.8             20.1             23.5
2              NaN             18.4              NaN
3              NaN              NaN             20.3
4             29.9              NaN              NaN
5              NaN              NaN             19.8
6             45.0             15.5             20.4
*******************************************************
   Max Temperature  Min Temperature  Avg Temperature
0             22.7              NaN             21.4
1             24.8             20.1             23.5
2              NaN             18.4             20.3
3             29.9              NaN             20.3
4             29.9              NaN             19.8
5             45.0             15.5             19.8
6             45.0             15.5             20.4

We can see that the NaN surrounded by the valid values gets filled and since the limit was set to 1, just one NaN gets filled when consecutive NaNs were present.

Using bfill() with limit=1 and limit_area='outside'

print("Original Dataset")
print(df)
print("*"*55)
bfill_df = df.bfill(limit=1, limit_area='outside')
print(bfill_df)
--------------------
Original Dataset
   Max Temperature  Min Temperature  Avg Temperature
0             22.7              NaN             21.4
1             24.8             20.1             23.5
2              NaN             18.4              NaN
3              NaN              NaN             20.3
4             29.9              NaN              NaN
5              NaN              NaN             19.8
6             45.0             15.5             20.4
*******************************************************
   Max Temperature  Min Temperature  Avg Temperature
0             22.7             20.1             21.4
1             24.8             20.1             23.5
2              NaN             18.4              NaN
3              NaN              NaN             20.3
4             29.9              NaN              NaN
5              NaN              NaN             19.8
6             45.0             15.5             20.4

We can see that in the first row of Min Temperature column, the NaN value was filled because that NaN wasn't surrounded by the valid value.

Filling Missing Values Across the Axis

We can use axis parameter to fill in missing values along the column or row.

If the axis is set to 0 or 'index', the missing values will be filled down each column, moving vertically (from below) along the rows. This means the last valid value from below (in the same column) will be used to fill in the NaN values above it.

print("Original Dataset")
print(df)
print("*"*55)
bfill_df = df.bfill(axis=0)
print(bfill_df)
--------------------
Original Dataset
   Max Temperature  Min Temperature  Avg Temperature
0             22.7              NaN             21.4
1             24.8             20.1             23.5
2              NaN             18.4              NaN
3              NaN              NaN             20.3
4             29.9              NaN              NaN
5              NaN              NaN             19.8
6             45.0             15.5             20.4
*******************************************************
   Max Temperature  Min Temperature  Avg Temperature
0             22.7             20.1             21.4
1             24.8             20.1             23.5
2             29.9             18.4             20.3
3             29.9             15.5             20.3
4             29.9             15.5             19.8
5             45.0             15.5             19.8
6             45.0             15.5             20.4

It's like using the bfill() method without any parameters. The NaN values were filled vertically from below along the rows in each column.

If the axis is set to 1 or 'columns', the missing values will be filled across each row, moving horizontally (from right to left) along the columns. This means the last valid value from the right (in the same row) will be used to fill in the NaN values to the left of it.

print("Original Dataset")
print(df)
print("*"*55)
bfill_df = df.bfill(axis=1)
print(bfill_df)
--------------------
Original Dataset
   Max Temperature  Min Temperature  Avg Temperature
0             22.7              NaN             21.4
1             24.8             20.1             23.5
2              NaN             18.4              NaN
3              NaN              NaN             20.3
4             29.9              NaN              NaN
5              NaN              NaN             19.8
6             45.0             15.5             20.4
*******************************************************
   Max Temperature  Min Temperature  Avg Temperature
0             22.7             21.4             21.4
1             24.8             20.1             23.5
2             18.4             18.4              NaN
3             20.3             20.3             20.3
4             29.9              NaN              NaN
5             19.8             19.8             19.8
6             45.0             15.5             20.4

We can see that the third row of Max Temperature column gets filled by the value to its left (18.4), and the fourth and sixth rows of Max Temperature and Min Temperature get filled by the value to its left (20.3 and 19.8).

🏆Other articles you might be interested in if you liked this one

✅Merge, combine, and concatenate multiple datasets using pandas.

✅Find and delete duplicate rows from the dataset using pandas.

✅How to efficiently manage memory use when working with large datasets in pandas?

✅How to find and delete mismatched columns from datasets in pandas?

✅Upload and display images on the frontend using Flask.

✅How does the learning rate affect the ML and DL models?

That's all for now

Keep Coding✌✌

One of the major changes we get is making GIL (Global Interpreter Lock) optional in Python 3.13.

In this article, you’ll see how to disable GIL in Python 3.13 to make multi-threaded programs faster.

Disable GIL in Python 3.13

First, download the official Python 3.13 on your local machine. Then proceed to install the Python executable file.

Follow the steps just as you install any Python version, you will see a step in which you are asked to choose the advanced installation options.

Now select the checkbox appearing at the end, this will install the different executable Python build with the name python3.13t or python3.13t.exe.

This will allow you to optionally disable or enable GIL in the runtime. Now proceed to install the setup.

After completing the setup for Python 3.13, if you look at the file location of Python 3.13, you will see that you also got a different Python build called python3.13t.

If you remember that “free-threaded binaries” mode was marked “experimental”, this means that you don’t get GIL disabled by default.

But how do I disable GIL? Python organization provided a different CPython build which is also called “Free-threaded CPython”.

Free-threaded CPython will help you disable GIL. Let’s see how to do it.

Let’s say, you have a Python script and you want it to run with GIL disabled. You can use the following command.

python3.13t main.py

This CPython build comes with GIL disabled by default.

Practical

I have the following Python script (main.py) that runs a multi-threaded task and calculates the time taken to complete the execution.

import sys
import sysconfig
import math
import time
import threading

def compute_factorial(n):
    return math.factorial(n)

# Multi-threaded
def multi_threaded_compute(n):
    threads = []
    # Create 5 threads
    for num in n:
        thread = threading.Thread(target=compute_factorial, args=(num,))
        threads.append(thread)
        thread.start()

    # Wait for all threads to complete
    for thread in threads:
        thread.join()

    print("Factorial Computed.")

def main():
    # Checking Version
    print(f"Python version: {sys.version}")

    # GIL Status
    status = sysconfig.get_config_var("Py_GIL_DISABLED")
    if status is None:
        print("GIL cannot be disabled")
    if status == 0:
        print("GIL is active")
    if status == 1:
        print("GIL is disabled")

    numlist = [100000, 200000, 300000, 400000, 500000]

    # Multi-threaded Execution
    start = time.time()
    multi_threaded_compute(numlist)
    end = time.time() - start
    print(f"Time taken : {end:.2f} seconds")

if __name__ == "__main__":
    main()

Now, we’ll run this script with and without free-threaded CPython.

With GIL enable (without free-threaded CPython)

python main.py

With GIL disable (with free-threaded CPython)

python3.13t main.py

That’s all for now.

Keep Coding✌✌

What does "in-memory" mean? It means that it stores the data in temporary memory or RAM, in the form of key-value pairs. This makes it perfect for serving results instantly.

In this article, you’ll learn how to install Redis locally on Windows without any hassle.

Installation Process Begins

You probably don’t know that Redis isn’t officially available for Windows yet but there is a way that you can install Redis locally on Windows and get started with it.

Redis is available for MacOS, Linux but for Windows, they say:

Redis is not officially supported on Windows. However, you can install Redis on Windows for development.

So, how to install it then? Simple, actually not simple in my opinion but I will guide you step by step installing Redis on your Windows machine.

Enable WSL on Windows

To install Redis on Windows, you need to enable WSL (Windows Subsystem for Linux). It simply means that you need to install Linux on your Windows system.

To enable WSL, you must be running Windows 10 version 2004 and higher (Build 19041 and higher) or Windows 11.

You can enable the WSL2 (WSL version 2) by default with a single command.

Open Powershell in administrator mode and run the following command.

wsl --install

This command will enable the features necessary to run WSL and install the Ubuntu distribution of Linux.

You can change or install another distribution of Linux if you want to, just use the following command.

wsl --install -d <Distribution Name>

You can replace <Distribution Name> with the actual Linux distribution you want to install on the Windows machine. But, I would recommend keeping the default settings.

After all this installation process, restart your Windows machine once to see the changes.

If your Windows machine is a bit older and WSL2 is not supported then check out this guide from Microsoft itself to manually enable WSL on Windows.

Troubleshooting Error

There will be less chance that you will get errors while setting up WSL but in case you encounter any error such as “failed to register wsl distribution” or something like that then consider changing the WSL version.

Run the following command in Powershell in administrator mode to change the version of your WSL.

wsl --set-default-version 1

This command will change the default version which is WSL2 to WSL1. This will solve most of the problems.

Now, restart your Windows machine again to see the effect.

Set Up Linux

After enabling WSL and installing Linux distribution, you need to set up user info for Linux.

Open Ubuntu or whichever distribution you installed and give it a few minutes to install. After that, it will ask you to enter a username for Linux and set a password.

After setting up the Linux distribution, the next step is to install Redis.

Install Redis on Windows

Redis can be installed on Windows using the following commands

curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list

sudo apt-get update
sudo apt-get install redis

Copy the above commands all together and paste them into the Ubuntu terminal, this will start the process of installing Redis on Windows locally.

Start Redis

Now run the following command to start the Redis server.

sudo service redis-server start

Now to check if you’re connected to Redis, run the following command.

redis-cli

Redis server started on localhost and listening on port 6379 which is its default port.

That’s all for now.

Keep Coding✌✌

Starting from version 2.0 of Pandas, the Copy-on-Write (CoW) has taken effect but has not been fully implemented. Most of the optimizations that are possible through Copy-on-Write are supported.

Aim of Copy-on-Write

As the name suggests, the data will be copied when it is modified. What it means?

When a DataFrame or Series shares the same data as the original, it will initially share the same memory for the data rather than creating a copy. When the data of either the original or new DataFrame is modified, a new copy of the data is created for the DataFrame that is being modified.

This will efficiently save memory usage and improve performance when working with large datasets.

Enabling CoW in Pandas

It is not enabled by default, so we need to enable it using the copy_on_write configuration option in Pandas.

import pandas as pd

# Option1
pd.options.mode.copy_on_write = True
# Option2
pd.set_option("mode.copy_on_write" : True)

You can use any of the options to turn on CoW globally in your environment.

Note: CoW will be enabled by default in Pandas 3.0, so get used to it early on.

Impact of CoW in Pandas

The CoW will disallow updating the multiple pandas objects at the same time. Here's how it will happen.

import pandas as pd
df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
subset = df["A"]
subset.iloc[0] = 10
df

With CoW, the above snippet will not modify df rather it modifies only the data of subset.

# df
    A    B
0    1    4
1    2    5
2    3    6

# subset
    A
0    10
1    2
2    3

inplace Operations will Not Work

Similarly, the inplace operations will not work with CoW enabled, which directly modifies the original df.

df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
df["A"].replace(1, 5, inplace=True)
df
--------------------
    A    B
0    1    4
1    2    5
2    3    6

We can see that df has remained unchanged and additionally, we will see a ChainedAssignmentError warning.

The above operation can be performed in two different ways. One method is to avoid inplace, and another is to use inplace to directly modify the original df at the DataFrame level.

# Avoid inplace
df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
df["A"] = df["A"].replace(1, 5)
df
--------------------
    A    B
0    5    4
1    2    5
2    3    6

# Using inplace at DataFrame level
df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
df.replace({"A": {2: 34}}, inplace=True)
df
--------------------
    A    B
0    1    4
1    34    5
2    3    6

Chained Assignment Will Never Work

When we modify the DataFrame or Series using multiple indexing operations in a single line of code, this is what we call the chained assignment technique.

# CoW disabled
with pd.option_context("mode.copy_on_write", False):
    df = pd.DataFrame({"A": [1, 2, 3, 4], "B": [5, 6, 7, 8]})
    df["B"][df['A'] > 2] = 10
df

The above code snippet is trying to change column B from the original df where column A is greater than 2. It means the value at the 2nd and 3rd index in column B will be modified.

Since the CoW is disabled, this operation is allowed, and the original df will be modified.

    A    B
0    1    5
1    2    6
2    3    10
3    4    10

But, this will never work with CoW enabled in pandas.

# CoW enabled
df = pd.DataFrame({"A": [1, 2, 3, 4], "B": [5, 6, 7, 8]})
df["B"][df["A"] > 2] = 10
df
--------------------
    A    B
0    1    5
1    2    6
2    3    7
3    4    8

Instead, with copy-on-write, we can use .loc to modify the df using multiple indexing conditions.

# CoW enabled
df = pd.DataFrame({"A": [1, 2, 3, 4], "B": [5, 6, 7, 8]})
df.loc[(df["A"] == 1) | (df["A"] > 3), "B"] = 100
df

This will modify column B where column A is either 1 or greater than 3. The original df will look like the following.

    A    B
0    1    100
1    2    6
2    3    7
3    4    100

Read-only Arrays

When a Series or DataFrame is accessed as a NumPy array, that array will be read-only if the array shares the same data with the initial DataFrame or Series.

df = pd.DataFrame({"A": [1, 2, 3, 4], "B": ['5', '6', '7', '8']})
arr = df.to_numpy()
arr
--------------------
array([[1, '5'],
       [2, '6'],
       [3, '7'],
       [4, '8']], dtype=object)

In the above code snippet, arr will be a copy because df contains two different types of arrays (int and str). We can perform modifications on the arr.

arr[1, 0] = 10
arr
--------------------
array([[1, '5'],
       [10, '6'],
       [3, '7'],
       [4, '8']], dtype=object)

Take a look at this case.

df = pd.DataFrame({"A": [1, 2, 3, 4], "B": [5, 6, 7, 8]})
arr = df.to_numpy()
arr

The DataFrame df has only one NumPy array (array of same data types), so arr shares the data with df. This means arr will be read-only and cannot be modified inplace.

print(arr.flags.writeable)
arr[0,0] = 10
arr
--------------------
False
ValueError: assignment destination is read-only

Lazy Copy Mechanism

When two or more DataFrames share the same data, the copies will not be created immediately.

df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
df2 = df.reset_index(drop=True)

Both df and df2 shares the same reference in the memory as both share the same data. The copy mechanism will trigger only when any of the DataFrame is modified.

df2.iloc[0, 0] = 10
print(df2)
print(df)
--------------------
    A  B
0  10  4
1   2  5
2   3  6
   A  B
0  1  4
1  2  5
2  3  6

But this is not necessary, if we don't want initial df, we can simply reassign it to the same variable (df) and this process will create a new reference. This will avoid the copy-on-write process.

df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
print("Initial reference: ",id(df))
df = df.reset_index(drop=True)
print("New reference: ",id(df))
df.iloc[0, 0] = 10
print(df)
--------------------
Initial reference:  138400246865760
New reference:      138400246860336
    A  B
0  10  4
1   2  5
2   3  6

This same optimization (lazy copy mechanism) is added to the methods that don't require a copy of the original data.

DataFrame.rename()

df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
df.rename(columns={"A": "X", "B": "Y"})
--------------------
    X    Y
0    1    4
1    2    5
2    3    6

When CoW is enabled, this method returns the original DataFrame rather than creating an entire copy of the data, unlike the regular execution.

DataFrame.drop() for axis=1

Similarly, the same mechanism is implemented for DataFrame.drop() for axis=1 (axis='columns').

df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6], "C": [7, 8, 9]})
df.drop(["A"], axis=1)
--------------------
    B    C
0    4    7
1    5    8
2    6    9

Conclusion

Pandas will by default implement Copy-on-Write (CoW) in version 3.0. All these optimizations that are compliant with CoW will lead to efficient memory and resource management when working with large datasets.

This will reduce unpredictable or inconsistent behavior and greatly maximize performance.

🏆Other articles you might be interested in if you liked this one

✅Merge, combine, and concatenate multiple datasets using pandas.

✅Find and delete duplicate rows from the dataset using pandas.

✅Find and Delete Mismatched Columns From DataFrames Using pandas.

✅Create temporary files and directories using tempfile module in Python.

✅Upload and display images on the frontend using Flask.

✅How does the learning rate affect the ML and DL models?

That's all for now

Keep Coding✌✌

In this article, we'll see the methods provided by pandas to handle missing values in a dataset.

df.fillna()

The DataFrame.fillna() is used to fill in the missing values with the desired value. Let's see how we can use it.

data = {"A": [2, np.nan, 19, 34, np.nan],
        "B": [np.nan, 23, 12, 34, np.nan]}
df = pd.DataFrame(data)
df
--------------------
      A     B
0   2.0   NaN
1   NaN  23.0
2  19.0  12.0
3  34.0  34.0
4   NaN   NaN

Filling Arbitrary Value

filled_df = df.fillna(0)
print(filled_df)
--------------------
      A     B
0   2.0   0.0
1   0.0  23.0
2  19.0  12.0
3  34.0  34.0
4   0.0   0.0

We passed an arbitrary value (0) to fill those NaN values in the dataset df.

Fill Using a Dataset

We can also use a dataset to fill in the missing values.

df2 = pd.DataFrame({"A": [1,2,3,4,5], "B": [6,7,8,9,10]})
fill_using_df = df.fillna(df2)
print(fill_using_df)
--------------------
      A     B
0   2.0   6.0
1   2.0  23.0
2  19.0  12.0
3  34.0  34.0
4   5.0  10.0

When using the fillna() method with df2, the NaN values in the original DataFrame df are replaced by the corresponding values in df2. If a cell in df is NaN, the method will look for the corresponding value in df2 (at the same position) and use that value to fill in the NaN.

Filling Different Values in Each Column

If we want to fill in different values in each column, we can use the following approach.

values = {"A": 100, "B": 200}
diff_val = df.fillna(value=values)
print(diff_val)
--------------------
       A      B
0    2.0  200.0
1  100.0   23.0
2   19.0   12.0
3   34.0   34.0
4  100.0  200.0

The value dictionary holds values to fill NaN in columns A and B in the dataset. By using df.fillna(value=values), the NaN value in column A is filled with the value 100 and NaN value in column B is filled with the value 200.

df.interpolate()

The DataFrame.interpolate() method provides various interpolation techniques to fill in the missing values.

Instead of filling in hard-coded values, we can use an interpolation method to fill missing values that make the dataset even more expressive and real.

Filling Computed Value

df2 = df.interpolate()  # default: linear method and axis=0
print(df2)
--------------------
      A     B
0   2.0   NaN
1  10.5  23.0
2  19.0  12.0
3  34.0  34.0
4  34.0  34.0

When we use df.interpolate(), the default linear interpolation method is used that fills the NaN values equally spaced ignoring the index.

For example, in column A, 10.5 is filled which is equally spaced between the values 2.0 and 19.0 with the difference of 8.5.

But if we see the fourth row in both columns, they are filled with the same value (34.0) as above them because there were no values to compute in the fifth row.

Filling Nearest Values

data = {"A": [3, np.nan, 2, np.nan, 4],
        "B": [1, 4, np.nan, 2, 5]}
df = pd.DataFrame(data)
df3= df.interpolate(method='nearest')
print(df3)
--------------------
     A    B
0  3.0  1.0
1  3.0  4.0
2  2.0  4.0
3  2.0  2.0
4  4.0  5.0

When we use method='nearest', the NaN values are filled with the nearest valid values.

In this case, the second row in column A is filled with the value of 3.0. Why so? The nearest value is decided based on the index close to the NaN value index. The index 0 (3.0) is closest to the index 1. The same is applied to all the NaN values.

Filling Values Considering Index Values

data = {"A": [3, np.nan, 9, np.nan, 4],
        "B": [1, 10, np.nan, 20, 5]}
df = pd.DataFrame(data)
df4= df.interpolate(method='values')  # or method='index'
print(df4)
--------------------
     A     B
0  3.0   1.0
1  6.0  10.0
2  9.0  15.0
3  6.5  20.0
4  4.0   5.0

The NaN values are filled equally spaced considering the values of the index surrounding the NaN value index.

df.ffill() and df.bfill()

The DataFrame.ffill() method is used to fill the last valid value in the missing place whereas the DataFrame.bfill() method is used to fill the next valid value.

Forward Filling With ffill()

data = {"A": [2, np.nan, 19, 34, np.nan],
        "B": [np.nan, 23, 12, 34, np.nan]}
df = pd.DataFrame(data)

forward_fill = df.ffill()
print(forward_fill)
--------------------
      A     B
0   2.0   NaN
1   2.0  23.0
2  19.0  12.0
3  34.0  34.0
4  34.0  34.0

We can see that NaN values are filled with the preceding valid values, for instance, the second row of column A is filled with 2.0 which is the same value above it.

Backward Filling With bfill()

data = {"A": [2, np.nan, 19, 34, np.nan],
        "B": [np.nan, 23, 12, 34, np.nan]}
df = pd.DataFrame(data)

backward_fill = df.bfill()
print(backward_fill)
--------------------
      A     B
0   2.0  23.0
1  19.0  23.0
2  19.0  12.0
3  34.0  34.0
4   NaN   NaN

In this case, the NaN is filled with the next valid values, for instance, the first row of column B is filled with 23.0 which is the next value in the column.

We can also see that the fifth row of columns A and B remains unfilled (NaN) due to the absence of the next valid value in the dataset.

🏆Other articles you might be interested in if you liked this one

✅Pandas df.ffill() and df.bfill() to handle missing values.

✅Merge, combine, and concatenate multiple datasets using pandas.

✅Find and delete duplicate rows from the dataset using pandas.

✅How to efficiently manage memory use when working with large datasets in pandas?

✅How to find and delete mismatched columns from datasets in pandas?

✅How does the learning rate affect the ML and DL models?

That's all for now

Keep Coding✌✌

Data retrieval and manipulation are critical skills for both Data Scientists and Data Analysts, as they form the foundation for effective data analysis.

Below are the SQL topics and concepts, prioritized from highest to lowest, that a data scientist should be proficient in.

1. SQL Querying

A strong foundation in SQL is crucial for data scientists, as it forms the basis for building efficient and effective data analysis processes. Proficiency in basic SQL querying, along with the ability to write complex yet readable and manageable queries, is essential. If you're not familiar with constructing queries or using the right commands, you risk facing constant challenges in your data tasks.

2. SQL Joins

SQL joins are a crucial concept for data scientists. In real-world scenarios, you rarely work with data from a single table. Instead, databases contain data from multiple sources, fields, or departments - often spread across different tables. These datasets contribute to various aspects of business growth, and as a data scientist, you need to combine and analyze them effectively.

For instance, if your organization is in the book business, you might have separate tables for members, books, orders, and more. To provide comprehensive insights or create a case study for your organization, you must understand how to join these tables and extract meaningful, relevant data.

3. CTE (Common Table Expressions)

You will need CTE (Common Table Expressions) to simplify complicated queries such as nested/subqueries, multiple joins, or aggregations.

CTEs are used when you don’t want to complicate things during data retrieval. This will help you break down complex queries into smaller queries that are more readable and manageable and your queries will look more structured.

Recursive CTE

Another form of CTE, recursive CTE is useful when you need to explore or traverse relationships between data, particularly when dealing with hierarchical structures or recursive relationships.

For instance, if a company has data about their employees and wants to know which manager each employee reports to, you can use a recursive CTE to find the relationships between managers and their subordinates.

Another example could involve sales data where different products belong to multiple categories in a hierarchical structure. You can use recursive CTEs to explore how these product categories are related, for example, by traversing from top-level categories down to subcategories.

4. Window Functions

Window functions play a crucial role in SQL. They allow you to perform a wide range of tasks, from data partitioning to time-series analysis, without altering the original structure of the data.

With window functions, you can carry out complex calculations, comparative analysis, and analyze data across rows, helping to derive meaningful insights while preserving row-level details.

5. Conditionals

Conditional expressions are useful they enable dynamic and flexible querying, data transformation, and feature engineering.

These expressions allow you to perform calculations, filter data, or modify values based on specific conditions, which is vital for data analysis, reporting, and preparation of data for machine learning models.

Conditional expressions (like CASE or IF()) allow you to clean, standardize, and transform data based on certain criteria. This is essential when dealing with inconsistent or incomplete data.

6. Aggregate Functions

Aggregate functions are essential for performing mathematical computations and summarizing data across different categories or fields. They allow you to compute metrics like sums, averages, counts, and more, based on specific groups within the data.

7. PL/SQL

While PL/SQL is less commonly used in everyday data science tasks, it's important to know how to create functions, stored procedures, triggers, and views in SQL.

PL/SQL (Procedural Language/SQL) extends SQL with procedural programming features like loops, conditions, and exception handling, allowing for more complex operations similar to any other programming language.

It is particularly useful for complex data processing, automating data tasks, reusing code, and optimizing query performance, ultimately saving time and resources.

That’s all for now.

Keep Coding✌✌.

What is GIL? It is a mechanism used by the CPython interpreter to ensure that only one thread executes the Python bytecode at a time.

An Experimental Feature

Python 3.13 brings major new features compared to Python 3.12 and one of them is free-threaded mode, which disables the Global Interpreter Lock, allowing threads to run more concurrently.

This is an experimental feature and if you want to try it, you can download the beta version of Python 3.13 from here.

At the time of installation, check the option "free threaded binaries(experimental)" to get the feature in Python.

Making GIL Optional in Python 3.13

The GIL will be disabled when you configure the Python with the --disable-gil option which is nothing but a build configuration (free threading build) at the time of installation.

This will allow optionally enabling and disabling GIL using the environment variable PYTHON_GIL which can be set to 1 and 0 respectively.

It will also provide a command-line option -X gil which can also be set to 0 (disable) and 1 (enable).

# v3.13
# GIL disabled
python3 -X gil=0 sample.py

# GIL enabled
python3 -X gil=1 sample.py

We can also check if the current interpreter is configured with the free threading build (--disable-gil) by using the following code.

import sysconfig
print(sysconfig.get_config_var("Py_GIL_DISABLED"))

If we run this, we'll get either 0 which means GIL is enabled, or 1 which means GIL is disabled.

With this, we'll also get a function that can be used to check if GIL is disabled in the running process.

import sys
sys._is_gil_enabled()  # returns a boolean value

GIL Vs No GIL

Let's see how the performance of multi-threaded programs will be affected when GIL is enabled and disabled.

We have a simple Python program (gil.py) that computes the factorial of numbers and compares the execution time taken by single-threaded, multi-thread, and multi-process tasks. We'll run this Python program first with GIL and then without GIL.

import sys
import sysconfig
import math
import time
import threading
import multiprocessing

def compute_factorial(n):
    return math.factorial(n)

# Single-threaded
def single_threaded_compute(n):
    for num in n:
        compute_factorial(num)
    print("Single-threaded: Factorial Computed.")

# Multi-threaded
def multi_threaded_compute(n):
    threads = []
    # Create 5 threads
    for num in n:
        thread = threading.Thread(target=compute_factorial, args=(num,))
        threads.append(thread)
        thread.start()

    # Wait for all threads to complete
    for thread in threads:
        thread.join()

    print("Multi-threaded: Factorial Computed.")

# Multi-process
def multi_processing_compute(n):
    processes = []
    # Create a process for each number in the list
    for num in n:
        process = multiprocessing.Process(target=compute_factorial, args=(num,))
        processes.append(process)
        process.start()

    # Wait for all processes to complete
    for process in processes:
        process.join()

    print("Multi-process: Factorial Computed.")

def main():
    # Checking Version
    print(f"Python version: {sys.version}")

    # GIL Status
    status = sysconfig.get_config_var("Py_GIL_DISABLED")
    if status is None:
        print("GIL cannot be disabled")
    if status == 0:
        print("GIL is active")
    if status == 1:
        print("GIL is disabled")

    numlist = [100000, 200000, 300000, 400000, 500000]

    # Single-threaded Execution
    start = time.time()
    single_threaded_compute(numlist)
    end = time.time() - start
    print(f"Single-threaded time taken: {end:.2f} seconds")

    # Multi-threaded Execution
    start = time.time()
    multi_threaded_compute(numlist)
    end = time.time() - start
    print(f"Multi-threaded time taken : {end:.2f} seconds")

    # Multi-process Execution
    start = time.time()
    multi_processing_compute(numlist)
    end = time.time() - start
    print(f"Multi-process time taken  : {end:.2f} seconds")


if __name__ == "__main__":
    main()

Running gil.py with GIL

python gil.py
Python version: 3.12.2 (tags/v3.12.2:6abddd9, Feb  6 2024, 21:26:36) [MSC v.1937 64 bit (AMD64)]
GIL cannot be disabled
Single-threaded: Factorial Computed.
Single-threaded time taken: 9.04 seconds
Multi-threaded: Factorial Computed.
Multi-threaded time taken : 8.21 seconds
Multi-process: Factorial Computed.
Multi-process time taken  : 5.64 seconds

We have Python v3.12 in which there is no option to check the GIL status, so we got "GIL cannot be disabled".

The difference is not that much between the single-threaded and multi-threaded but we can see a pretty decent difference in the case of multi-process task.

Running gil.py without GIL

D:/SACHIN/Python13/python3.13t gil.py
Python version: 3.13.0b3 experimental free-threading build (tags/v3.13.0b3:7b41395, Jun 27 2024, 16:17:17) [MSC v.1940 64 bit (AMD64)]
GIL is disabled
Single-threaded: Factorial Computed.
Single-threaded time taken: 9.28 seconds
Multi-threaded: Factorial Computed.
Multi-threaded time taken : 4.86 seconds
Multi-process: Factorial Computed.
Multi-process time taken  : 6.14 seconds

This time we have a third beta version of Python 3.13 configured with free threading build and as we can see the GIL is disabled.

But the most important part is we can see a massive difference in the performance of multi-threaded task and on the other side, some degradation can be seen in the performance of multi-process and single-threaded task.

🏆Other articles you might be interested in if you liked this one

✅Create multi-threaded Python programs using a threading module.

✅Template inheritance in Flask.

✅Difference between exec and eval in Python.

✅Create temporary files and directories using tempfile module in Python.

✅Upload and display images on the frontend using Flask.

✅How does the learning rate affect the ML and DL models?

That's all for now

Keep Coding✌✌

The data you've gathered should be of high quality, so structure, construct, and clean it in such a way that it has the potential to produce a robust model.

In this article, you'll learn how to use pandas to find and remove columns from one dataset that don't match those in another.

Objective

When you have multiple datasets and want to merge them to augment the data points, some columns of the datasets do not match. You must find and remove them so that they do not have an adverse effect on the model while training.

What You'll Do

There are a few datasets that contain used car information such as price, name, distance driven, fuel type, and more. You will search the dataset for unmatched columns and then remove them.

The datasets used in this article are downloaded from Kaggle.

Dataset Overview

import pandas as pd

# Reading Car Dataset
df = pd.read_csv("car.csv")
print(f"Shape of the Datset: {df.shape}")

----------
Shape of the Dataset: (4340, 8)

The car.csv dataset has 4380 rows and 8 columns, and you will now find the columns that do not match the columns of the car.csv dataset and remove them.

The following are the names of the dataset's columns:

print("Column Names:")
for column in df.columns:
    print(column.capitalize())

----------
Column Names:
Name
Year
Selling_price
Km_driven
Fuel
Seller_type
Transmission
Owner

You now know which columns to keep in the second dataset and which to remove.

import pandas as pd

# Reading Car Dataset
df = pd.read_csv("used_car.csv")
print(f"Shape of the Datset: {df.shape}")

----------
Shape of the Dataset: (7906, 18)

As you can see the used_car.csv dataset has 7906 rows and 18 columns. This dataset contains more columns, but you must keep the columns consistent with the car.csv dataset and remove the remaining ten columns from the used_car.csv dataset.

Finding Mismatched Columns

In this section, you will identify the columns in the used_car.csv dataset that are not present in the car.csv dataset.

columns_to_delete = df2.columns.difference(df.columns)

The above code will find the columns that are not present in the car.csv dataset from the used_car.csv dataset using the difference() method on the dataset's columns attribute. The result will be stored inside the columns_to_delete variable.

print("Columns not Present in First Dataset:")
for counter, col in enumerate(columns_to_delete, start=1):
    print(f"{counter}- {col}")

The above code will list all the columns that are not present in the car.csv dataset. Run the above code and you'll get the following output.

Columns not Present in First Dataset:
1- City
2- Region
3- Sales_ID
4- State or Province
5- engine
6- max_power
7- mileage
8- seats
9- sold
10- torque

There are a total of 10 columns that are not present in the car.csv dataset. The next step is to delete or remove them from the used_car.csv dataset.

Deleting Mismatched Columns

In this section, you will delete the above-identified columns from the used_car.csv dataset and pandas provide a much simpler way to do this.

new_df2 = df2.drop(columns_to_delete, axis=1)

The above code will delete all those 10 columns from the used_car.csv dataset using the drop() function on df2 along the columns axis (axis=1). The new dataset is stored inside the new_df2 variable.

print("Columns Present in the Second Dataset:")
for column in new_df2.columns:
    print(column.capitalize())

The above code will print the column names that remain in the used_car.csv dataset.

Columns Present in the Second Dataset:
Name
Year
Selling_price
Km_driven
Fuel
Seller_type
Transmission
Owner

Now both datasets car.csv and used_car.csv contain the same columns. You can now merge both datasets along the row axis to augment the data.

Save new_df2 as the new dataset in CSV format using the pandas to use it for further purposes.

new_df2.to_csv('new_car_data.csv', index=False)

The above code will save new_df2 as new_car_data.csv in the current directory.

Conclusion

Data cleaning is a critical step in the data preprocessing pipeline, ensuring that your data is of high quality and suitable for building accurate and robust machine learning models.

Any irregularity in data can make a huge difference in the model's training. So you need to make a consistent dataset.

🏆Other articles you might be interested in if you liked this one

✅Merge, combine, and concatenate multiple datasets using pandas.

✅Find and delete duplicate rows from the dataset using pandas.

✅Create multi-threaded Python programs using a threading module.

✅Create temporary files and directories using tempfile module in Python.

✅Upload and display images on the frontend using Flask.

✅How does the learning rate affect the ML and DL models?

That's all for now

Keep Coding✌✌

Return Values

The exec() function doesn't return any value whereas the eval() function returns a value computed from the expression.

expression = "3 + 5"

result_eval = eval(expression)
print(result_eval)

result_exec = exec(expression)
print(result_exec)

When we run this code, we'll get 8 and None. This means that eval(expression) evaluated the result and stored it inside result_eval whereas exec(expression) returned nothing, so the value None gets stored into result_exec.

8
None

Execution

The exec() function is capable of executing multi-line and multi-statment code, it doesn't matter whether the code is simple, complex, has loops and conditions, classes, and functions.

On the other hand, the eval() function is restricted to executing the single-line code which might be a simple or complex expression.

expression = """
for x in range(5):
    print(x, end=" ")
"""

exec(expression)
eval(expression)

Look at this code, we have a multi-line code that prints numbers up to the given range. The exec(expression) will execute it and display the result but the eval(expression) will display the error.

0 1 2 3 4
SyntaxError: invalid syntax

However, if we convert the same expression into a single line as the following, the eval(expression) will not throw any error.

expression = "[print(x, end=' ') for x in range(5)]"
eval(expression)

--------------------
0 1 2 3 4

Summary

🏆Other articles you might be interested in if you liked this one

✅Execute complex code blocks from string input using exec() function.

✅Template inheritance in Flask.

✅Type hints in Python - Callable objects, Return values, and more?

✅Best Practices: Positional and Keyword Arguments in Python

✅Yield Keyword in Python with Examples?

✅Create a WebSocket Server and Client in Python.

That's all for now

Keep Coding✌✌

Python provides a module named logging to log messages. These messages can be written to various destinations, including files, for better monitoring and debugging of programs.

In this article, you'll learn:

• Logging a simple message using the logging module

• Severity level of the messages

• Configuring theloggingmodule to logINFOandDEBUGlevel messages and logging events to a file

• Logging the variable data

• Formatting the log messages using LogRecord attributes

• Advanced Configurations: Loggers, Handlers and Formatters

• Practical example

Logging Messages

Here's an example of logging a simple message on the console using the logging module in Python.

# Imported logging module
import logging

num = 0
# Logs a warning message
logging.warning("Entering Infinite Loop")
while True:
    if num % 2 == 0:
        print(num)
    num += 1

The code above runs endlessly, thus when it runs, a warning message will be logged to alert the user.

Here, logging.warning() has been used to log the warning message to the console. The warning here is the severity level of the message.

WARNING:root:Entering Infinite Loop
0
2
4
.
.
.

As you can see, the output begins by logging the warning message "Entering Infinite Loop" in the pattern WARNING:root:<message>, indicating the severity level and name assigned by the logger respectively.

Similar to this, the logging module offers several levels to show the events' severity.

# Imported logging module
import logging

logging.info("Used to log info message")
logging.debug("Used to log debug message")
logging.warning("Used to log warning message")
logging.error("Used to log error message")
logging.critical("Used to log critical message")

There are 5 different levels to log messages of different severity levels.

• INFO

• DEBUG

• WARNING

• ERROR

• CRITICAL

WARNING:root:Used to log warning message
ERROR:root:Used to log error message
CRITICAL:root:Used to log critical message

You can observe that the output is missing the first two messages. Why is it so? The messages with a severity level of WARNING and above are logged by the logging module.

You can log INFO and DEBUG messages by applying some configurations.

Configuring Logging Module

The logging module can be configured to unleash more power to let you do more stuff when logging events.

Basic Configuration

The logging module includes a function called basicConfig() that helps in configuring the logging system. Here's an example of configuring logging module to log INFO level message.

import logging

# Configuring logging
logging.basicConfig(level=logging.INFO)
logging.info("This info will be printed")

The logging module is configured in the code above to log a INFO message by passing the keyword argument level to the basicConfig() function and setting it to logging.INFO.

When you run the above code, the severity level INFO message will now be printed.

INFO:root:This info will be printed

Similarly, you can log DEBUG messages by setting the level argument to the value logging.DEBUG.

import logging

# Configuring logging to log DEBUG msg
logging.basicConfig(level=logging.DEBUG)
logging.debug("This is a DEBUG message")

--------------------
DEBUG:root:This is a DEBUG message

Logging Event to a File

The basicConfig() function accepts a keyword argument called filename to specify the file in which the event will be logged.

 import logging

# Logging to a file
logging.basicConfig(filename='debug.log', level=logging.DEBUG)
logging.debug("This is a DEBUG message")

The message will be logged into the debug.log file and it will be the same as the previous output but in the file instead of the console.

By default, the mode of the file is set to append (a) mode and you can change this by specifying the filemode parameter. Additionally, you can use the encoding parameter to specify the file's encoding.

import logging

# Logging to a file
logging.basicConfig(filename='debug_new.log',
                    filemode='w',
                    encoding='utf-8',
                    level=logging.DEBUG)
logging.debug("Debugging starts from here.")

In this example, the file is set to write (filemode='w') mode and encoding is set to 'utf-8'. This time the message will be logged into the debug_new.log file.

How to Log Variable Data?

Logging variable data is similar to formatting a string using the % operator. This helps when you want to include dynamic data in logged messages.

import logging

# Logging variable data
logging.error("%s: Not found - Status: %s", 404, "Fail")

In this code, two placeholders (%s) are present in the string that will help Python interpolate the string with the specified values i.e., 404 and 'Fail', and they will be placed in the same position as they appear.

When you run the code, you'll get the following output.

ERROR:root:404: Not found - Status: Fail

Formatting the Logged Messages

You now know how to log variable data; in this section, you'll utilize it to format the logged messages.

You can choose the format of your logged message to be displayed in the console or a file.

import logging

logging.basicConfig(format='%(levelname)s - %(message)s')
logging.warning("Things are getting worse.")

--------------------
WARNING - Things are getting worse.

As you can see, the message is logged with the formatting that the format argument set. The severity (levelname) is shown first, then the event's description (message).

You may be wondering now where the message and levelname came from. Well, these are the attributes specified by the LogRecord. Similarly, LogRecord has more attributes.

import logging

logging.basicConfig(
    format='%(levelname)s: %(message)s - %(asctime)s', 
    level=logging.DEBUG
)
logging.debug("Debugging started at")

Now, this time a new attribute asctime is added to the message that logs the date and time of the event. The output looks like the following.

DEBUG: Debugging started at - 2024-06-12 16:37:55,996

You can take more control of formatting the date and time of the event by passing the datefmt argument in the basicConfig().

import logging

logging.basicConfig(
    format='%(levelname)s: %(message)s - %(asctime)s',
    datefmt='%d/%m/%Y %I:%M:%S %p', 
    level=logging.DEBUG
)
logging.debug("Debugging started at")

--------------------
DEBUG: Debugging started at - 12/06/2024 05:55:38 PM

The date and time looks more readable now due to the string ('%d/%m/%Y %I:%M:%S %p') set by the datefmt argument. You can experiment with the characters within the string just like you would do in the time.strftime(). Here's an example.

import logging

logging.basicConfig(
    format='%(levelname)s: %(message)s - %(asctime)s',
    datefmt='%d %b %Y %a %I:%M:%S %p', 
    level=logging.DEBUG
)
logging.debug("Debugging started at")

--------------------
DEBUG: Debugging started at - 12 Jun 2024 Wed 06:08:20 PM

You can add some more attributes such as file name, module name, process ID, name of the logger, and much more.

So far you've learned to log messages at the basic level. In the next section, you'll learn to log messages at an advanced level such as configuring your logger, sending the log messages over multiple destinations, and much more.

Logging - Advanced Configurations

In this section, you'll use classes and functions to log messages. The commonly used classes in the logging module are as follows:

• Logger - Used to create logger objects so that logging can be performed by calling different methods on its objects.

• Handler - Used to send log messages to appropriate destinations such as into the file, email, HTTP server, etc.

• Formatter - Used to set the format of the log records in the final output.

Creating Logger Object

import logging

# Create a logger
logger = logging.getLogger("custom_logger")
logger.warning("Warning... Warning...")

In this example, a custom logger named custom_logger is created using logging.getLogger(). When you run this code, you'll get the following output.

Warning... Warning...

This is a simple message without the severity level and logger name even though the name of the logger was specified. This is because the custom logger can't be configured using basicConfig(). To format this output, you need to use Handlers and Formatters.

An important note:

It is a good practice to name your logger based on module like the following.

logger = logging.getLogger(__name__)

This will keep track of your module/package hierarchy. It means if you are using it in same module, the logger name will be __main__ and if you import this module into another module, then the logger name will the module_name, the name of the module in which it is defined.

The logger object doesn't log messages of severity level DEBUG and INFO. To log DEBUG or INFO level messages using the custom logger, you need to create a handler.

Creating Handler

Handler objects are responsible for logging records to the specified destinations. There are several handler types such as StreamHandler, FileHandler, HTTPHandler, SMTPHandler and more.

The addHandler() method is used to add one or more handlers to the logger object. You need to add the individual handler to log messages based on categories to different destinations.

After creating a logger object in the previous section, the next step is to create the handler.

# adv_logging.py

import logging

# Create a logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)  # Set the logger's level

# Creating Handlers
# Console handler
cs_handler = logging.StreamHandler()
cs_handler.setLevel(logging.INFO)

# File handler
fl_handler = logging.FileHandler("info.log")
fl_handler.setLevel(logging.ERROR)

In this code, some modifications have been done to the logger object such as the logger's name changed and the logger's level has been set to INFO.

Two handlers have been created cs_handler and fl_handler. The first handler (cs_handler) is responsible for sending messages in the console and its level is set to INFO. The second handler (fl_handler) sends the messages to the file name info.log which is set to ERROR level.

Creating Formatters

Formatters are responsible for applying the formatting of the developer's choice to the final output.

It is similar to the formatting you've learned while setting up the basic configurations of the messages using basicConfig(). But here, you need to use a Formatter class of the logging module.

Here's the extension of the previous code.

...

# Creating Formatters
# Formatter for console
cs_formatter = logging.Formatter(
    '%(asctime)s: %(name)s - %(levelname)s: %(message)s',
    datefmt='%d %b %Y %a %I:%M:%S %p'
)
# Setting formatter to handler
cs_handler.setFormatter(cs_formatter)

# Formatter for file
fl_formatter = logging.Formatter(
    '%(asctime)s: %(name)s - %(levelname)s: %(message)s',
    datefmt='%d %b %Y %a %I:%M:%S %p'
)
# Setting formatter to handler
fl_handler.setFormatter(fl_formatter)

The formatters cs_formatter and fl_formatter have been created. The formatting style of both formatters is identical, and setFormatter() has been used to assign each formatter to its corresponding handler.

To log messages with applied configurations, the final step is to add these handlers (cs_handler and fl_handler) to the logger.

...

# Adding handler to logger
logger.addHandler(cs_handler)
logger.addHandler(fl_handler)

Now add messages to be logged.

...

logger.info("Process started...")
logger.warning("Warning... Warning...")
logger.error("Unusual activity detected...")
logger.critical("System security compromised...")

Output

13 Jun 2024 Thu 03:55:24 PM: __main__ - INFO: Process started...
13 Jun 2024 Thu 03:55:24 PM: __main__ - WARNING: Warning... Warning...
13 Jun 2024 Thu 03:55:24 PM: __main__ - ERROR: Unusual activity detected...
13 Jun 2024 Thu 03:55:24 PM: __main__ - CRITICAL: System security compromised...

Here, the logging.info(), logging.warning(), logging.error(), logging.critical() passed the information to the handlers (cs_haldler and fl_handler).

Since the logger's level is set to INFO and cs_handler is a StreamHandler set to INFO level printed the messages with specified formatting of INFO level and above in the console.

The fl_handler is a FileHandler which is set to ERROR level will log the messages in the info.log file of the level ERROR and above.

13 Jun 2024 Thu 03:55:24 PM: __main__ - ERROR: Unusual activity detected...
13 Jun 2024 Thu 03:55:24 PM: __main__ - CRITICAL: System security compromised...

You can see that the name of the logger is logged as __main__ which means the source file is executed directly. If you import this module to another file and then run it, the name of the logger will change.

Create a new Python file, in this case, sample_log.py. Import the source file into it, and then run the file.

# sample_log.py
import adv_logging

When you run this file, your output will look like the following.

13 Jun 2024 Thu 03:58:12 PM: adv_logging - INFO: Process started...
13 Jun 2024 Thu 03:58:12 PM: adv_logging - WARNING: Warning... Warning...
13 Jun 2024 Thu 03:58:12 PM: adv_logging - ERROR: Unusual activity detected...
13 Jun 2024 Thu 03:58:12 PM: adv_logging - CRITICAL: System security compromised...

The info.log file will look like the following.

13 Jun 2024 Thu 03:55:24 PM: __main__ - ERROR: Unusual activity detected...
13 Jun 2024 Thu 03:55:24 PM: __main__ - CRITICAL: System security compromised...
13 Jun 2024 Thu 03:58:12 PM: adv_logging - ERROR: Unusual activity detected...
13 Jun 2024 Thu 03:58:12 PM: adv_logging - CRITICAL: System security compromised...

Practical Example

You've learned to create the logger and handlers with specified formatting. Now you can create a template for logging the messages and include it in your project code to log messages dynamically.

Here's a practical example of creating a logger with advanced configurations and then including it in the main script to log messages in a file.

First, create a Python file, in this case, log_temp.py. This file holds a template for logging a message.

# log_temp.py
import logging

# Create a logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

# File handler
fl_handler = logging.FileHandler("log_file.log")
fl_handler.setLevel(logging.INFO)

# Formatter for file
fl_formatter = logging.Formatter(
    '%(name)s - %(levelname)s: %(message)s'
)
# Setting formatter to handler
fl_handler.setFormatter(fl_formatter)

# Adding handler to logger
logger.addHandler(fl_handler)

# Function to log INFO level msg
def info_msg(msg):
    logger.info(msg)
# Function to log WARNING level msg
def warning_msg(msg):
    logger.warning(msg)
# Function to log ERROR level msg
def error_msg(msg):
    logger.error(msg)

This code is pretty much the same as you saw in the previous sections but this time three functions are added: info_msg(), warning_msg(), and error_msg().

These functions log INFO, WARNING, and ERROR level messages. The messages will be provided by the developer.

This template uses FileHandler to log messages in a file of level INFO and above.

Now, create another Python file, in this case, script.py, and dump the following code or write your own Python program.

# Imported template
import log_temp

try:
    num = 0
    # Logs a warning message
    log_temp.warning_msg("Entering Infinite Loop")
    while True:
        if num % 2 == 0:
            print(num)
        num += 1

except KeyboardInterrupt:
    # Logs an error message
    log_temp.error_msg("Program Interrupted")

# Logs an info
log_temp.info_msg("Program continued")
print("Exited infinite loop")

This code runs infinitely and when it runs, it logs a warning message and if the loop is interrupted, it logs an error message followed by the info that the program continued after exiting the loop.

When you run the script.py file, you'll get these details logged into the log_file.log file.

log_temp - WARNING: Entering Infinite Loop
log_temp - ERROR: Program Interrupted
log_temp - INFO: Program continued

When the code starts executing, a warning message is logged, and then when the program gets interrupted, an error message is logged followed by the info message that provides info that the program continued after interruption.

Conclusion

Keeping log records of the applications can help better supervise and troubleshoot them. Python provides a standard library named logging for recording log messages for your applications.

You can apply basic and advanced configurations to the logging module to unleash more power. You can log messages in the console as well as to various destinations like inside a file, email, HTTP connection, etc.

🏆Other articles you might be interested in if you liked this one

✅Type hints in Python - Callable objects, Return values, and more?

✅Best Practices: Positional and Keyword Arguments in Python

✅Yield Keyword in Python with Examples?

✅Create a WebSocket Server and Client in Python.

✅Create and Interact with MySQL Database in Python.

✅Understanding the use of global keyword in Python.

That's all for now

Keep Coding✌✌

Python has the "global" keyword, which allows users to modify global variables from within the local scope.

What is global Keyword?

Let's understand with an example.

# Example of "global" keyword

# Global variable
var = "declared as global"

def modify_var():
    # Function's scope
    global var
    # Modified the global variable "var"
    var = "modified global"
    print(var)

modify_var()
print(var)

In this example, the global variable var is defined first, followed by the function modify_var().

Inside the modify_var() function, the global var statement is used to reference the global variable var in the function's scope and then the variable var is modified and printed.

The function is then called. Upon running the code, you'll get the following result.

modified global
modified global

You can observe that the global variable var is modified.

In conclusion, the global keyword is used to reference and modify a global variable within the scope of a function.

Not Using global Keyword

What happens if you do not use the global keyword and attempt to modify a global variable within the local scope?

# Global variable
var = "declared as global"

def modify_var():
    # Function's scope
    # Trying to modify the global variable "var"
    var = "modified global"
    print(var)

modify_var()
print(var)

In this example, the global var statement is removed from the function and an attempt is made to modify the variable var.

modified global
declared as global

You can observe that the global variable var remains unchanged.

Making Local Variable a Global One

The global keyword can also be used to create a global variable in the local context.

def separate():
    text = "Geek Python"
    print(text.split(" "))

separate()
print(text)

In this example, the text variable is declared within the separate() function and has scope within the function only.

The text variable is being accessed outside of the function scope. This will result in an error.

['Geek', 'Python']
Traceback (most recent call last):
  ...
    print(text)
          ^^^^
NameError: name 'text' is not defined. Did you mean: 'next'?

When you add a global statement just above the text variable, it becomes the global variable, which may be accessed from anywhere in the code.

def separate():
    global text   # Added global stmt
    text = "Geek Python"
    print(text.split(" "))

separate()
print(text)

When you run the code, you'll get the value stored in the text variable.

Assigning Value to the Global Variable Within Local Scope

Consider the following example which attempts to change the global variable within the local scope without using the global statement.

# Global variable
initial = 2

def increment():
    for x in range(10):
        # Assigning value to the global var in local context
        initial += x

increment()
print(initial)

A global variable named initial is declared and initialized with the value 2.

The function named increment() increments the global variable initial. The function is then called and the value of the initial variable is printed.

When you run the code, you'll get the following result.

Traceback (most recent call last):
  ...
    increment()
  ...
    initial += x
    ^^^^^^^
UnboundLocalError: cannot access local variable 'initial' where it is not associated with a value

The error message indicates that the local variable "initial" cannot be accessed. Python interprets the function's initial variable as a new local variable, which is why the code failed to locate the initial variable within the function's scope and returned an error.

However, when you add a global statement (global initial) within the function, Python will know that it is now referencing the existing global variable within the local scope.

# Global variable
initial = 2

def increment():
    global initial
    for x in range(10):
        # Assigning value to the global var in local context
        initial += x

increment()
print(initial)

You won't get an error when you run the code as the global statement has been added to the function scope.

47

Conclusion

The global keyword can be used to reference and modify an existing global variable within a local scope. If the global variable does not already exist, the global keyword can be used to create it within the local scope of a function.

🏆Other articles you might be interested in if you liked this one

✅Yield Keyword in Python with Examples?

✅Best Practices: Positional and Keyword Arguments in Python

✅Python's __getitem__ method?

✅Create a WebSocket Server and Client in Python.

✅Create and Interact with MySQL Database in Python

✅Understanding the Different Uses of the Asterisk(*) in Python?

That's all for now

Keep Coding✌✌

Python introduced type hints or static typing with version 3.5, allowing developers to declare the data type of variables, parameters, etc.

What is Type Hint?

x: int = 5
y: str = "5"

In this example, the variable x expects an integer value while the variable y expects a string value.

This is called type hint or static typing where you specify the expected data type for a variable, parameter or return value of a function.

Python has a different way of declaring type hints for the variables, return values, collections, etc.

Consider the following example:

def circle(radius: float) -> str:
    area = 3.14 * radius ** 2
    return f"Area of circle: {area}"

In this example, the function circle accepts an argument radius which is expected to be a float value as indicated by type hint radius: float, and the return value of this function is expected to be a string, as indicated by the -> str hint.

Performing a Check

What if you pass the argument of a different type which is not expected? Consider the function below:

def date(dd: str, mm: str, yyyy: str) -> str:
    return f"Current Date: {dd}-{mm}-{yyyy}"

curr_date = date(22, 9, 2024)  # expected 'str' got 'int'
print(curr_date)

The function date() accepts three arguments and expects all of them to be a string value, however, integer values are supplied when the function is called.

What do you think? Will this program throw an error? Well, it looks like but Python or specifically interpreter completely ignores the type hints as this is not its purpose.

Current Date: 22-9-2024

The type of arguments was decided in the runtime that is why no error is thrown, however, you may see a warning in your IDE or code editor.

Annotating Multiple Return Types

In this section, you'll learn how to annotate multiple return types for a single value of alternative types and multiple values of different types.

Alternative Types for a Return Value

def cart(item: str) -> str | None:
    if item == "":
        return None
    return "Item added in the cart"

The function cart() accepts an argument item and returns None if it is not supplied, otherwise, it returns a string.

To represent multiple return types for a function, the (|) pipe can be used. It means either str or None. So, when you call the function, it indicates that the return value of the function can be a str or None.

You can also use typing.Union to accomplish the same task you've done using the (|).

from typing import Union

def cart(item: str) -> Union[None, str]:
    if item == "":
        return None
    return "Item added in the cart"

The Union[None, str] is equivalent to None | str. None | str is a shorthand and it is a recommended way.

Multiple Return Values of Different Types

def cart(item: str, quantity: int) -> dict[str, int] | None:
    if item == "" or quantity == 0:
        return None
    return {item: quantity}

The function cart() has been updated to accept an additional argument quantity and returns None if any of the arguments are left empty, otherwise, it returns a dictionary containing a key (str) and value (int).

If you look at the type hint for the return value, this time different types (str and int) are expected to return in a dictionary (dict).

You can do this in the following way.

from typing import Union, Dict

def cart(item: str, quantity: int) -> Union[Dict[str, int], None]:
    if item == "" or quantity == 0:
        return None
    return {item: quantity}

You can use Mapping instead of Dict to represent the dictionary.

from typing import Union, Mapping

def cart(item: str, quantity: int) -> Union[Mapping[str, int], None]:
    if item == "" or quantity == 0:
        return None
    return {item: quantity}

But all this seems a bit verbose so you can go for shorthand syntax.

Type Hinting Functions

from collections.abc import Callable

def apply_cart(
        func: Callable[[str, int], dict[str, int]],
        item: str,
        quantity: int
) -> dict[str, int]:
    return func(item, quantity)

def cart(item: str, quantity: int) -> dict[str, int] | None:
    if item == "" or quantity == 0:
        return None
    return {item: quantity}

The function apply_cart() accepts a callable object (which can be a function or any other callable object) and two arguments (item and quantity) and returns a dictionary containing the key and value.

The Callable type hint provides a list of arguments ([str, int]) that the callable object accepts. In this case, func() expects strings and integers. Callable's second parameter is the return value (dict[str, int]), which is a dictionary.

The second function, cart(), is identical to the previous example and returns a dictionary if everything is correct.

cart_item = apply_cart(cart, "Mouse", 2)
print(cart_item)
--------------------
{'Mouse': 2}

The apply_cart() function is invoked with the cart (a function) and two arguments ('Mouse', 2).

The apply_cart() function calls the cart() function with the given arguments and returns a result.

Here's an optimisation, if you have many arguments of different types, you can use ellipsis (...) rather than passing different input types.

The ellipsis literal (...) indicates that callable can accept any arbitrary list of arguments.

from collections.abc import Callable
from typing import TypeVar, Any

T = TypeVar("T")

def apply_cart(
        func: Callable[..., T],
        *args: Any
) -> T:
    return func(*args)

The apply_cart() function has been updated and now accepts an arbitrary list of arguments of any type ([..., T]) as well as variadic argument (*args) of any type (Any).

The first parameter in the Callable is an ellipsis (...), suggesting that any arbitrary parameter list is permitted.

The second parameter is a type variable (T = TypeVar("T")) that can work with any type. It indicates that the callable can accept any type and return an element of that type.

You can also use a parameter specification variable (ParamSpec) instead of an ellipsis to make callable objects accept any number of positional or keyword arguments.

from collections.abc import Callable
from typing import TypeVar, ParamSpec

P = ParamSpec("P")
T = TypeVar("T")

def apply_cart(
        func: Callable[P, T],
        *args: P.args
) -> T:
    return func(*args)

In this case, the first parameter of Callable is a parameter specification variable (P = ParamSpec("P")) indicating arbitrary list of arguments is acceptable. The second parameter (T) indicates that any type is acceptable.

The function apply_cart() also accepts a variadic argument (*args) of type P.args that represents a tuple of any number and type of positional arguments. To annotate **kwargs, P.kwargs must be used that represents the mapping of keyword parameters to their values.

Instead of using the ellipsis literal, you may now utilise ParamSpec and TypeVar to enable callable objects to accept any number of positional or keyword arguments of any type.

The code is further updated by adding **kwargs, which allows the function to accept keyword arguments of arbitrary length.

from collections.abc import Callable
from typing import TypeVar, ParamSpec

P = ParamSpec("P")
T = TypeVar("T")

def apply_cart(
        func: Callable[P, T],
        *args: P.args,
        **kwargs: P.kwargs
) -> T:
    return func(*args, **kwargs)


def cart(item: str, quantity: int) -> dict[str, int] | None:
    if item == "" or quantity == 0:
        return None
    return {item: quantity}

cart_item = apply_cart(cart, "Mouse", quantity=2)
print(cart_item)

Type Hinting Iterables

Consider the following function that takes a list of names and sorts them in ascending order.

def sort_student(names: list[str]) -> None:
    sorting = sorted(names)
    for name in sorting:
        print(name)

n = ["Gojo", "Yuta", "Yuji", "Megumi"]
sort_student(n)

The list[str] type is used to type hint the parameter names to indicate that it expects a list of strings.

But what if you want names to be a type tuple or set instead? You need to refactor your code. In this case, you need to change the type in one location, however, this might be hard in complex and big projects.

You can use the type Iterable to make the function accept any iterable object.

from collections.abc import Iterable

def sort_student(names: Iterable[str]) -> None:
    sorting = sorted(names)
    for name in sorting:
        print(name)

n1 = ("Gojo", "Yuta", "Yuji", "Megumi") # No error
n2 = ["Gojo", "Yuta", "Yuji", "Megumi"] # No error
n3 = {"Gojo", "Yuta", "Yuji", "Megumi"} # No error

Type Aliases for Better Readability

The complex type hints for callable objects like in the above case, arguments, and return values of the functions kind of feel cumbersome to write.

Why can't simplify the complex type by giving them an alias(name)? You can create an alias for the type in the same way you would create a variable.

type ReturnCart = dict[str, int] | None

def cart(item: str, quantity: int) -> ReturnCart:
    if item == "" or quantity == 0:
        return None
    return {item: quantity}

cart_item = apply_cart(cart, "Mouse", quantity=2)
print(cart_item)

In the above example, an alias ReturnCart is created and contains the type of return value (dict[str, int] | None).

The alias is defined with the soft keywordtype, which creates an instance of TypeAliasType. This keyword was added in Python 3.12. If you are using an older version of Python, you can choose an alternate method.

Type aliases can also be created like you would create a variable using a simple assignment.

ReturnCart = dict[str, int] | None

To clarify, you can mark it with TypeAlias to explicitly show this as a type alias, not a simple variable.

from typing import TypeAlias

ReturnCart: TypeAlias = dict[str, int] | None

The methods described above will assist you in simplifying complex types, and the key benefit is that you will only need to edit the type in one spot, eliminating the need for refactoring.

Another advantage of using type aliases is that they can be reused in other portions of your code, which improves code readability.

Type Checker Tools

Python entirely ignores type hints and determines the type of variables, function arguments, and return values at runtime.

Mypy, a popular third-party tool, can enforce type-checking in Python. Since it is a third-party tool, you must install it using the following command.

python -m pip install mypy

This will provide you access to the mypy command to type-check your program.

Static Type Checking Using Mypy

The function sort_student() takes an iterable object and sorts the student names in ascending order. Save this function to a file, such as student.py.

# student.py
from collections.abc import Iterable

def sort_student(names: Iterable[str]) -> None:
    sorting = sorted(names)
    for name in sorting:
        print(name)

n1 = ("Gojo", "Yuta", "Yuji", "Megumi")
sort_student(n1)

To type check your program using mypy, enter the command mypy followed by the file name you want to check in the terminal.

> mypy student.py 
Success: no issues found in 1 source file

The type checker identified no errors. One thing you may have noticed is that the code contains the function call (sort_student(n1)), which should have created the output, but there was none in the terminal except the message created by mypy.

This means that mypy does not execute your code, instead, it examines if the values match the expected type based on the type hints.

What if you made a mistake and the actual values do not match the expected type?

# student.py
def sort_student(names: list[str]) -> None:
    sorting = sorted(names)
    for name in sorting:
        print(name)

n1 = ("Gojo", "Yuta", "Yuji", "Megumi")
sort_student(n1)

This function expects a list of strings but the tuple was passed. If you now check your code, it will generate the following error.

> mypy student.py
student.py:8: error: Argument 1 to "sort_student" has incompatible type "tuple[str, str, str, str]"; expected "list[str]"  [arg-type]
Found 1 error in 1 file (checked 1 source file)

Mypy has identified the error on line number 8 saying that the argument passed to sort_student() has an incompatible type tuple[str, str, str, str], it is expected to be a list[str].

Resources

• Type cheat sheet

Conclusion

Type hints are optional in Python but they can be used to make the code more readable and using type hints can avoid possible bugs in your code.

In this article, you've learned:

• Implementing type hints

• Alternative types for single data and different types for multiple values using pipe (|) and Union

• Callable type for type hinting functions

• Ellipsis (...) and TypeVar in Callable type to make a callable object accept an arbitrary list of arguments of any type

• ParamSpec instead of an ellipsis with TypeVar in Callable type to make a good combination for a callable object accepting an arbitrary list of arguments of any type

• Iterable type for type hinting iterable objects

• Type aliases to simplify complex type

• Mypy for type checking in Python

There is much more you can do with type hints in Python.

🏆Other articles you might be interested in if you liked this one

✅Best Practices: Positional and Keyword Arguments in Python

✅Decorators in Python and How to Create a Custom Decorator?

✅Create a WebSocket Server and Client in Python.

✅Create and Interact with MySQL Database in Python

✅Understanding the Different Uses of the Asterisk(*) in Python?

✅Yield Keyword in Python with Examples?

That's all for now

Keep Coding✌✌