Perform High-Level File Operations In Python

Copying or moving files or folders manually from one directory to another directory could be a real pain. This can be automated using a Python module called shutil.

Shutil module provides some high-level operations on files and collection of files like copying, moving, or removing the files.

In other words, the shutil module helps in automating the task of file copying or moving from one directory to another directory. It comes under the Python standard library so it doesn't need to be installed externally.

In this article, we'll learn this module.

Copy files to another directory

shutil.copy() method is used to copy the content of the source file to the destination file or directory. While the process of copying also copies the file's permission mode but other metadata like the file’s creation and modification times, is not preserved.

The source must represent a file but the destination can be a file or a directory. If the destination is a directory then the file will be copied to the destination directory with a base filename from the source. The destination must be writable.

If the destination is a file and already exists then it will be replaced by the source file otherwise the new file will be created.

Syntax: shutil.copy(source, destination, *, follow_symlinks = True)

Parameter:

source: A string that shows the path of the source file.
destination: A string that shows the path of the destination file or directory.
follow_symlinks (optional): The default value of this parameter is True. If it is False and the source represents a symbolic link then the destination will be created as a symbolic link.

Copying source file to the destination

import shutil

# source and destination path
src_path = "hello.txt"
dst_path = "hello2.txt"

path = shutil.copy(src_path, dst_path)

# Printing the path of destination dir
print('Destination Path:', path)

Output

Destination Path: hello2.txt

If the destination is the directory

import os
import shutil

# Making a directory
os.mkdir('Project')

# Listing the files in it
print("Empty Folder: ", os.listdir('Project'))

# source and destination path
src_path = "hello.txt"
dst_path = "Project"

# Copying file to destination
path = shutil.copy(src_path, dst_path)

# Printing the name of copied filename and path of destination dir
print('File Copied:', os.listdir('Project'))
print('Destination Path:', path)

Output

Empty Folder:  []
File Copied: ['hello.txt']
Destination Path: Project\hello.txt

Copy metadata along with the file

shutil.copy2() method is used to copy the contents of the source file to the destination file or directory. It is the same as the shutil.copy() method but the only difference is that this method tries to preserve the metadata of the file.

Syntax: shutil.copy2(source, destination, *, follow_symlinks = True)

Parameter:

source: A string that shows the path of the source file.
destination: A string that shows the path of the destination file or directory.
follow_symlinks (optional): The default value of this parameter is True. If it is False and the source represents a symbolic link then it attempts to copy all metadata from the source symbolic link to the newly-created destination symbolic link. This functionality is platform dependent.

import os
import shutil

# Specifying src and dst paths
src_path = "Data\geek.txt"
dst_path = "NewData"

print("Before copying the metadata:")

# Listing the files inside src folder
src_folder = "Data"
print(os.listdir(src_folder))

# Printing the status of the files inside src path
print("Metadata:", os.stat(src_path), "\n")

# Copying content from src to dst
dest = shutil.copy2(src_path, dst_path)

print("After copying the metadata:")

# Listing the files inside dst path
print(os.listdir(dst_path))

# Printing the status of the files inside dst path
print("Metadata:", os.stat(dest), "\n")

# Printing the destination path of newly created files
print("Files copied to:", dest)

Output

Before copying the metadata:
['geek.txt', 'hello.py', 'python.txt']
Metadata: os.stat_result(st_mode=33206, st_ino=7036874417909405, st_dev=3836766283, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1662197106, st_mtime=1662196237, st_ctime=1662194099) 

After copying the metadata:
['geek.txt']
Metadata: os.stat_result(st_mode=33206, st_ino=20547673300020899, st_dev=3836766283, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1662197106, st_mtime=1662196237, st_ctime=1662204751) 

Files copied to: NewData\geek.txt

If the destination is a directory

import os
import shutil

path = "NewData"

print("Before copying the files:")

# Listing the files inside dst folder
print(os.listdir(path), "\n")

# Specifying src and dst paths
src_path = "Sample/hello.txt"
dst_path = "NewData"

# Copying content from src to dst
dest = shutil.copy2(src_path, dst_path)

print("After copying the metadata:")

# Listing the files inside dst folder
print(os.listdir(path))

# Printing the destination path of newly created files
print("Files copied to:", dest)

Output

Before copying the files:
[] 

After copying the files:
['hello.txt']
Files copied to: NewData\hello.txt

Copy content from one file to another file

shutil.copyfile() method is used to copy the content of the source file to the destination file without the metadata. The source and destination must represent a file and the destination file must be writable. If the destination already exists then it'll be replaced or a new file will be created.

Syntax: shutil.copyfile(source, destination, *, follow_symlinks = True)

Parameter:

source: A string that shows the path of the source file.
destination: A string that shows the path of the destination file or directory.
follow_symlinks (optional): The default value of this parameter is True. If False and source represent a symbolic link then a new symbolic link will be created instead of copying the file.

import shutil

# Specifying the src and dest file paths
src_path = "Data/python.txt"
dst_path = "NewData/geek.txt"

# Copying the src file to dst file
dest = shutil.copyfile(src_path, dst_path)

# Printing the path of newly created file
print("File Copied to:", dest)

Output

File Copied to: NewData/geek.txt

If the source and destination file path is the same

If it happens then a SameFIleError will be raised and we can handle the error by using shutil.SameFileError.

import shutil

# Specifying the dst file same as src file
src_path = "Data/python.txt"
dst_path = "Data/python.txt"

try:
    dest = shutil.copyfile(src_path, dst_path)
    print("File Copied to:", dest)

# Printing error 
except shutil.SameFileError:
    print("Source and Destination represent the same file.")

Output

Source and Destination represent the same file.

Copy the directory tree

shutil.copytree() method is used to replicate the complete directory tree. It recursively copies the entire directory tree rooted at the source directory to the destination directory. The destination directory must not already exist. The new directory will be created during the process.

Syntax: shutil.copytree(src, dst, symlinks = False, ignore = None, copy_function = copy2, ignore_dangling_symlinks = False)

Parameter:

src: A string that shows the path of the source directory.
dest: A string that shows the path of the destination.
symlinks (optional): This parameter accepts True or False, depending on which the metadata of the original links or linked links will be copied to the new tree.
ignore (optional): If ignore is given, it must be a callable that will receive as its arguments the directory being visited by copytree(), and a list of its contents, as returned by os.listdir().
copy_function (optional): The default value of this parameter is copy2, other copy function like copy() can be used for this parameter.
ignore_dangling_symlinks (optional): This parameter value when set to True is used to put a silence on the exception raised if the file pointed by the symlink doesn’t exist.

import shutil

# Specifying src directory
src_path = "Data"
# src dir tree will be copied to the destination
dst_path = "Project"

# Copy the entire src directory tree to the destination
dest = shutil.copytree(src_path, dst_path)
print("File Copied to:", dest)

Output

File Copied to: Project

Remove a directory tree

shutil.rmtree() method is used to delete the entire directory tree, the specified path must point to a directory (but not to a symbolic link to a directory).

Syntax: shutil.rmtree(path, ignore_errors=False, onerror=None)

Parameter:

path: A path-like object representing a file path. A path-like object is either a string or bytes object representing a path.
ignore_errors: If ignore_errors is true, errors resulting from failed removals will be ignored.
oneerror: If ignore_errors is false or omitted, such errors are handled by calling a handler specified by onerror.

Removing a top-level directory tree

import shutil

# Directory tree to be removed
dir_to_be_removed = "Sample"

# Removing a top-level dir
remove = shutil.rmtree(dir_to_be_removed)

print("Successfully removed.")

Output

Successfully removed.

Removing a directory tree under a directory

import shutil
import os

# Directory location
path = "Sample/NewDirectory"

# directory to be removed
dir_to_be_removed = "Python"

# joined the path
join_path = os.path.join(path, dir_to_be_removed)

# removing directory
shutil.rmtree(join_path)
print("Successfully removed.")

Output

Successfully removed.

Move files to another location

shutil.move() method is used to move the source file or a directory to the destination location. If the destination directory already exists then the source directory will move inside that directory.

Syntax: shutil.move(src, dst, copy_function=copy2)

Parameter:

src: A string that shows the path of the source file.

dst: A string that shows the path of the destination file or directory.

copy_function(optional): The default value of this parameter is copy2. We can use other copy functions also.

import shutil

# Specifying src and dst directories
src_path = "Project"
dst_path = "NewProject"

# Moving src dir to dst dir
dest = shutil.move(src_path, dst_path)

# Printing the destination location
print("Directory moved to:", dest)

Output

Directory moved to: NewProject\Project

If the destination doesn't exist

import shutil

# Specifying src directory
src_path = "New_dir"
# Non-existing directory
dst_path = "Moved_dir"

# Moving src dir to dst dir
dest = shutil.move(src_path, dst_path)

# Printing the destination location
print("Directory moved to:", dest)

Output

Directory moved to: Moved_dir

Finding executable files

shutil.which() method is used to find the path to an executable file specified to the cmd argument. If no cmd is specified then it will return None.

Syntax: shutil.which(cmd, mode = os.F_OK | os.X_OK, path = None)

Parameter:

cmd: File in string format.
mode: This parameter specifies the mode by which the method should execute.
os.F_OK tests the existence of the path
os.X_OK checks if the path can be executed
path: This parameter specifies the path to be used, if no path is specified then the results of os.environ() are used.

import shutil

# Executable file to be located
exec_file = "pip"

# Finding the location
location = shutil.which(exec_file)

# Printing the location of the executable file
print(location)

Output

D:\SACHIN\Python310\Scripts\pip.EXE

More cool methods

There are some other cool methods that the shutil module provides other than the above-mentioned methods.

Copy the stats of a file or a directory

shutil.copystat() method is used to copy the stats of a file or a directory like permission bits, last access time, last modification time, and flags from the source to the destination.

import shutil
import os
import time

src_path = "D:/SACHIN/Pycharm/Number-Converter"
dst_path = "D:/SACHIN/Pycharm/OpenCV/project1.py"

# Printing metadata of source dir
print("Source Data")
print("Modified at:", time.ctime(os.stat(src_path).st_mtime))
print("Accessed at:", time.ctime(os.stat(src_path).st_atime), "\n")

# Printing metadata of destination dir
print("Destination Data")
print("Modified at:", time.ctime(os.stat(dst_path).st_mtime))
print("Accessed at:", time.ctime(os.stat(dst_path).st_atime), "\n")

# Copying stats of src to dst
shutil.copystat(src_path, dst_path)

# Printing metadata after using shutil.copystat() method
print("Destination data after using shutil.stat()")
print("Modified at:", time.ctime(os.stat(dst_path).st_mtime))
print("Accessed at:", time.ctime(os.stat(dst_path).st_atime))

Output

Source Data
Modified at: Fri Jun 10 19:31:51 2022
Accessed at: Mon Sep  5 16:22:19 2022 

Destination Data
Modified at: Tue Jul 27 19:00:02 2021
Accessed at: Mon Sep  5 16:23:27 2022 

Destination data after using shutil.stat()
Modified at: Fri Jun 10 19:31:51 2022
Accessed at: Mon Sep  5 16:22:19 2022

Retrieve disk usage stats

shutil.disk_usage(path) method is used to get the statistics of disk usage of the specified path. The return value has attributes total, used, and free which are the amount(in bytes) of the total, used, and free space. A path may be a file or a directory.

import shutil

# Checking the stats
disk_size = shutil.disk_usage("NewSample")

# Printing the stats
print("The stats are:", disk_size)

Output

The stats are: usage(total=374277664768, used=33221120000, free=341056544768)

Archive a directory

shutil.make_archive() is used to create an archive file in a zip or tar format.

import shutil

# Path of directory to be archived
directory = "NewSample"

# Archiving the directory with zip format
archive = shutil.make_archive(directory, format="zip")

print(archive)

Output

NewSample.zip

Unpacking a directory

shutil.unpack_archive() is used to unpack the archive file. The method takes two arguments, the first is the filename which is the full path of the archive file and the second is the directory in which the files are extracted.

import shutil

# Path of directory to be unpacked
directory = "NewSample.zip"

# Unpacking the archive dir
shutil.unpack_archive(directory, "extracted_dir")

print(f"Successfully extracted the files from {directory}.")

Output

Successfully extracted the files from NewSample.zip.

For performing the high-level archiving operations on files in Python, See zipfile module.

Conclusion

We sure learned to handle high-level file operations using the shutil module. We've learned how to

Copy the file or a directory to another directory
Copy the metadata of the file
Permanently move the file or a directory to another location
Remove and copy the directory tree

Along with these operations, we've seen some of the cool methods which are provided by the shutil module.

Reference: https://docs.python.org/3.10/library/shutil.html

That's all for now

Keep Coding✌✌

Perform High-Level File Operations In Python - shutil Module

Comments

More from this blog

Power-up API Testing with EchoAPI's AI

How to Install PyPy and Use it?

10 Useful yet Rarely Used OS Functions in Python

How to Stream Video to Frontend in FastAPI

How to Fix Circular Imports in Python

Copy files to another directory

Copy metadata along with the file

Copy content from one file to another file

Copy the directory tree

Remove a directory tree

Move files to another location

Finding executable files

More cool methods

Copy the stats of a file or a directory

Retrieve disk usage stats

Archive a directory

Unpacking a directory

Conclusion

Command Palette

Comments

More from this blog

Copy files to another directory

Copy metadata along with the file

Copy content from one file to another file

Copy the directory tree

Remove a directory tree

Move files to another location

Finding executable files

More cool methods

Copy the stats of a file or a directory

Retrieve disk usage stats

Archive a directory

Unpacking a directory

Conclusion