So you think you know pytest? (WIP)

I recommend starting with understanding the Anatomy of the test. Then going through some of the flowcharts like this one and this one to understand what are the hooks being called for the pytest command (eg, how can we customize the Arrange phase of the test).

Project Set Up

The goal: We need to set up our project structure so that our folders are organized, and the test modules can easily import the packages, modules, and functions from other places.

Starting with a Simple Import

First, we need to understand how the import works. If you have a test module and a function module within the same module this will be easy, in the test module you can import the function module, and then Python will look for a module within the same module.

#Example folder structure
# - package_folder
#   - test.py
#   - function.py

# Then in test.py
import function  # this will import the module. 

More Complex Structure

But what if you organized your folder such that the test and function files are in different nested folders? In this case, a relative import, as explained in the official documents, might not work, or still work, but you lose track when too many relative import dots are used.

# Example folder structure
# - package_1
#   - function.py
# - tests
#   - test.py

One way around this is to install your package in editable mode. Some good reading references are in the official pytest doc, the Python packaging guide, and a good medium article.

To summarize, what you need to do are

Step 1: set up a project structure similar to below

# Assuming src layout
.
├── README.md
├── LICENSE
├── pyproject.toml
├── setup.py
├── setup.cfg
├── src/
│    └── package_1/
│       └── module_1.py
└── tests/
    ├── test_module_1.py

In your pyproject.toml, write the following

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

In your setup.cfg, write the following

# inside setup.cfg:

[metadata]
name = <your package name>
version = <version number, eg: 1.0>
author = <Firstname Lastname>
author_email = <your email>
description = <Your description of the project>
long_description = file: README.md
long_description_content_type = text/markdown
url = <url for your project>
classifiers =
    Programming Language :: Python :: 3
    Operating System :: OS Independent
license_files = LICENSE

[options]
packages = find:

[options.packages.find]
include = <package(s) you want to include>
exclude = <you can also add package(s) you want to exclude>

Then in your setup.py:

# inside setup.py :

from setuptools import setup
if __name__ == '__main__':
    setup()

Step 2 : Install your package in “editable” mode by running from the same directory (directory of your project folder)

pip install -e .

If you do it correctly, you will see a hidden folder with a name like <package_name>.egg_info in your project directory.

Now you can import the module in your test file directly. Example below

# inside test_module_1.py
from package_1 import module_1
# or alternative
import package_1.module_1

What if you have a flat layout instead of a src-layout?

Set up in previous section applicable to src-layout folder structure. You can copy the code, and modify a few entries without understanding what it does. But sometimes when you inherit a legacy project without a src folder you will have a project structure similar to the one below (known as a flat layout). How do you configure the setup.cfg file to correctly install the packages?

# A flat layout
.
├── README.md
├── LICENSE
├── pyproject.toml
├── setup.py
├── setup.cfg
├── package_1/
│    └── group_1/
│       └── module_1.py
├── package_2/
│    └── group_2/
│       └── module_2.py
└── tests/
    ├── test_module_1.py

First, a brief explanation of what setup.cfg does. This file provides the custom arguments when calling the setup() function, equivalent to the keywords accepted by the function (read here).

Next, we must understand how package discovery works (especially for namespace packages). I recommend reading through the following page. In summary, what you need to know:

1. If your root directory has more than one package, packages = find: for automatic discovery will not work for namespace packages.
2. One way to fix this is to add a __init__.py file in every folder (which turn your project into more than namespace packages)
3. The other way to fix this is to replace find: with find_namespace:, which will tell the setup tools to look for all namespace packages inside the root directory.
4. Another alternative is to specify the root directory with package_dir option. see below

# inside setup.cfg:

[metadata]
# Some meta data

# Option for namespace packages
[options]
packages = find_namespace:

# Alternative option
[options]
packages = find:

# set to this if want to install everything under root directory
package_dir =
    =.
# end of alternative option

# Then specify the includes or excludes package here
[options.packages.find]
where = .
exclude = tests

Other recommended topics

pytest_addoption(parser)

add custom command line option to the pytest . Reference

For example, if you want to execute the pytest based on different options specified in your custom option ‘cmdopt’, with cmd input like below

# Cmd line 
pytest <test_file>.py --cmdopt=opt1

# In .py file you will do 
def pytest_addoption(parser):
    parser.addoption(
        "--cmdopt", action="store", default="type1", help="my option: type1 or type2"
    )


@pytest.fixture
def cmdopt(request):
    return request.config.getoption("--cmdopt")

That will require pytext.fixture

pytext.fixture()

Reference , Good explanation & Good Video

Summary:

• fixtures are like functions that can be passed to a test case as arguments
• when it is passed, the fixture will be called and its return object can be used inside the test case
• some fixture can have scope of session, which means it will be called automatically when test session start.

Example

import pytest
@pytest.fixture()
def create_fix():
  return object

# This will called the create_fix function first
def test_something(create_fix):
  # Do something
  # Object returned by create_fix can be used

pytest_generate_test()

Good explanation: https://pytest-with-eric.com/introduction/pytest-generate-tests/

Example code:

import pytest

def func(input):
  # Do something & return the output

# Define the data for test case generation
test_data = [some datas]

# Define the pytest_generate_tests hook to generate test cases
def pytest_generate_tests(metafunc):
    if 'test_input' in metafunc.fixturenames:
        # Generate test cases based on the test_data list
        metafunc.parametrize('test_input,expected_output', test_data)

# Define the actual test function
def test_funct(test_input, expected_output):
    result = func(*test_input)
    assert result == expected_output

pytest.hookimpl()

TBD

Changing Log Level of pytest

Reference

# Add --log-cli-level=$LEVEL when running pytest command
# $LEVEL refer to level defined in logging module. 
# Example
pytest test.py --log-cli-level=INFO