Develop and Deploy CLI Tool on Python — with Poetry and Click

10 min readSep 20, 2024

Preface Scenario

You want to start building a new CLI tool with Python and deploying it on Linux or other Operating Systems.

There are some important requirements you need to consider when setting up the project:

You will normally want to develop your project in an isolated environment so that the packages you installed/updated for one project don't affect the packages you used for another project.
When collaborating with others you will need a package management tool so that your project can be easily “portable”. All packages/libraries you added to the project should be recorded in some file, and others can run a one-click command to install all dependencies.
In addition, it will be best if the package management tool can help to check, resolve, or avoid dependency conflicts. One example of dependency conflicts (or dependency hell): you need package A for your project and package A relies on a specific version of package B. At the same time, you use some features from another version of package B. The package management tool should either find a version of package B that is compatible with both your project and package A requirements or give you an early warning to resolve the dependency conflict before further action.
For deployment into the environment OS, you will need to set up some files so that your project can be installed using a standard tool (like pip).
For development purposes, you might also want to install your project in editable mode, so that any changes in your code, can straight away be tested out in your environment.

Available Tools and Selection Rationale

Understanding the requirements above will lead us to the most confusing and challenging question, especially for beginners — Which tool/combination of tools to use?

The confusion arises probably due to the ever-evolving nature of the software development world. Initially, tools are developed to address a specific need or scenario. When people use the tools and realize the deficiency of the old tools, while feeling the need to combine several tools into one for ease of use, new tools are then developed.

To my understanding, below is a list of tools available and what they can do about the requirements above.

PIP is the standard package manager included by default for Python 3.4 or later versions. It can

Install all dependencies specified in the requirement.txt file
Install (deploy) the project, also installing in editable mode with a pyproject.toml file.
It will warn you of dependency conflict (more on pip upgrade conflict here).

venv is included in the Python standard library for Python 3.3 or later versions to help create and manage a virtual environment for your project.

pipx uses pip and venv. You can read the comparison with other tools here. It can help you install your CLI projects directly or in editable mode.

Poetry is another popular package manager tool that can do the following:

Help to create and update the pyproject.toml file
Install dependencies in your project's virtual environment.
When installing new dependencies, stop when dependency conflicts arise.

My Selection Rationale

I will opt for Poetry over Pip for package manager due to

its ability to help update the pyproject.toml
its ability to detect dependency conflicts and force us to resolve them earlier
its ability to separate the main package dependencies into pyproject.toml file, and subpackage (packages relied on by the main package) into the poetry.lock file. Where as for pip all the main and sub-packages will be listed in the requirement.txt file.

For virtual environment management, Pipx is optional. I will probably use venv to create my virtual environment, and then use poetry to install any libraries. This will ensure if I run any commands accidentally without using the poetry command, it will still be within my virtual environment.

Getting Start

Prerequisites required

Install Python's latest version
Install pipx

# What worked for me 
python -m pip install --user pipx
python -m pipx ensurepath

# Then go to your environment variable on windows, check the path added and 
# update lowercase to uppercase if necessary
# restart your window.

Install poetry using pipx

# installation
pipx install poetry

Working with Poetry

Initializing a Project: Init vs New

If you already have a project folder, you can use the init method to generate the pyproject.toml file in the project directory. The poetry program will ask a few questions for your input during the generation.

If you want to start from scratch, you can use the poetry new <project-name> method to generate the entire new project folder for you.

# cd to your project directory, then start generating the pyproject.toml file
poetry init

# Alternative way to create new project
poetry new <project_name>

# Create new project in src layout
poetry new --src <project_name>

By default, poetry will generate a flat layout project structure for you, similar to the below

# A flat layout
.
├── README.md
├── LICENSE
├── pyproject.toml
├── poetry.lock
├── package_1/
│   └── __init__.py
└── tests/
    ├── __init__.py

You can generate a new project in the src layout such as below.

# Assuming src layout
.
├── README.md
├── LICENSE
├── pyproject.toml
├── poetry.lock
├── src/
│    └── package_1/
│       └── __init__.py
└── tests/
    ├── __init__.py

Choices of Virtual Environment

You have two choices:

Create your virtual environment using poetry. Then any poetry commands involved in changing the virtual environment, such as adding new libraries should automatically use the virtual environment created. The downside of this approach is any command not starting with poetry, will run in your background environment.
Create your virtual environment using venv, then run poetry inside it. Poetry should detect that you are working in your virtual environment, and modify libraries in the virtual environment you created without recreating a new one. To me, this is the safer way to avoid accidentally messing up with your background environment.

# Approach 1 - manage virtual environment using poetry
# Activate a Custom Virtual Environment with python 
# if your python is defined as python3, use python3
poetry env use python
# Messages will show
# Creating virtualenv <project_name>-<hash>-py3.12
# Using virtualenv: <path>

# You can check which env is being activated 
poetry env list

# Running any poetry command should automatically activate the associated
# virtual environment

# Add main dependencies package from pypi via 
poetry add <package_name>

# Remove dependencies package
poetry remove <package_name>

# Approach 2 - manage your virtual environment using venv - need venv
# installed globally
# navigate to your project directory and create a virtual environment
python -m venv <env_name>
# example
python -m venv myenv

# Activate venv for bash
source <env_name>/Scripts/activate
# example 
source myenv/Scripts/activate
# for window commands prompt
<env_name>\Scripts\activate

# Deactivate venv
deactivate

Install Your Project in Editable mode

Installing your project in editable mode means any changes you made to the codes in the modules installed will be immediately reflected in your environment package. The alternative is to install your project in the final state.

Once you have your virtual environment set up, there are two ways to install your project in editable mode. Both approaches will utilize your pyproject.toml file.

# Approach 1 - run pip install e . at your project directory
pip install e .

# Approach 2 - run poetry install should install your project in editable mode
poetry install

Understanding the pyproject.toml file

The pyproject.toml file is the specified file format of PEP 518 which contains the build system requirements for Python projects. It is equivalent to build.gradle / maven’s poml file for Java. There are some lengthy readings on StackOverflow and packaging Python tutorials here.

The essences you need to know:

with a later Python version, you can get rid of setup.cfg with pyproject.toml file.
when using poetry to generate the pyproject.toml file, some of the sections' text will be different than in a normal pyproject.toml file.
projects with flat layouts usually don't need to specify the modules to include. Projects with a src layout will need to specify the modules to include in the build.

For example, let's say you have a project with a src layout as follows:

.
├── README.md
├── LICENSE
├── pyproject.toml
├── poetry.lock
├── src/
│    └── calc/
|       ├── commands.py
|       ├── __main__.py    
│       └── __init__.py
└── tests/
    ├── __init__.py

A simple pyproject.toml file will look like this

# For project not maintained by poetry, this section text should be 
# something like [project]
[tool.poetry]
# name of your package, same name shown up when you install it 
name = "calc"
# version, description, authors, readme etc are less important 
version = "0.1.0"
description = ""
authors = ["sjchin88"]
readme = "README.md"
# This packages is important, in this example, it is telling the build tool
# to include the package calc from the src folder, which is the main folder
# for project with src layout
packages = [{include = "calc", from = "src"}]

# This section will be updated by poetry when you use 
# poetry add & poetry remove commands
[tool.poetry.dependencies]
python = "^3.12"
click = "^8.1.7"

# This specify the build system to use poetry
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

# For project not managed by poetry, this section will be called 
# [project.scripts], it actually tells the build system that
# there should be a command "calcx" pointed to the cli function inside the 
# __main__ module of the calc package. 
[tool.poetry.scripts]
calcx = "calc.__main__:cli"

Building Complex CLIs with Click

There are some good references out there, from a simple tutorial in realpython, to the official documentation guide. However, these references get elusive when you try to do things differently. For example, a more complex project usually places the handling logic to other packages/modules, separated from the package/module handling the command line interface functions. To illustrate, lets say we have a simplified src folder structure as below

.
├── README.md
├── LICENSE
├── pyproject.toml
├── poetry.lock
├── src/
│    └── calc/
|       ├── commands.py
|       ├── __main__.py    
│       └── __init__.py
│    └── calculator/
|       ├── power_mod.py   
│       └── __init__.py
└── tests/
    ├── __init__.py

In this project, the calc package will handle all CLI functions, the calculator package will have all the support logic required by the calc package. Let's say we have the following contents in the calc.__main__py, calc.commands.py and calculator.power_mod.py file:

# Content of calc.__main__py
import click

from . import commands

@click.group()
def cli():
    pass

# this line add the commands.power function to the command line group cli
cli.add_command(commands.power)

# Content of calc.commands.py
import click
# This line import the power_mod function from calculator module
from calculator import power_mod

# First click.command() wrap this function to be a cli function by click
# Second and third wrapper add the options to the command
# Note the arguments of the function will match the option name specified,
# click will try to find matching arg parsed and pass to the function
@click.command()
@click.option('--base', default=1, help='base of a number')
@click.option('--power', default=0, help='power for the number')
def power(base, power):
    """ Simple program that calculate base number raise to the poer
    """
    click.echo(power_mod.powerup(base, power))

# Content of calculator.power_mod.py. There is only one simple function. 
def powerup(base: int, power: int) -> int:
    return base**power

The last thing to do before we test this out is to ensure our pyproject.toml file includes both packages. This can be done by modifying the package line as shown below

[tool.poetry]
name = "calc"
version = "0.1.0"
description = ""
authors = ["sjchin88"]
readme = "README.md"
# This packages is important, in this example, it is telling the build tool
# to include the package calc and calculator from the src folder
packages = [
    {include = "calc", from="src"},
    {include = "calculator", from="src"},
]

[tool.poetry.dependencies]
python = "^3.12"
click = "^8.1.7"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.scripts]
calcx = "calcx.__main__:cli"

Now you can test your CLI using the following commands

# If you install using poetry install, you need to add prefix poetry run
# followed by main command calcx, subcommand power, 
# and the options --base and power
poetry run calcx power --base 2 --power 3
# Should return 8

# If you install using pip install -e . 
# you should be able to run the calcx command directly
calcx power --base 2 --power 3
# Should return 8

What if I lazy to include every package from the src

You can include the whole src package only. But if you do so, the import statement for all the packages must be updated accordingly. Example below

# pyproject.toml file 
[tool.poetry]
name = "calc"
version = "0.1.0"
description = ""
authors = ["sjchin88"]
readme = "README.md"
# Here we include src only
packages = [
    {include = "src"},
]

[tool.poetry.dependencies]
python = "^3.12"
click = "^8.1.7"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.scripts]
# Here the package reference need to start from src
calcx = "src.calcx.__main__:cli"

# Content of calc.commands.py
import click
# Here we import the src.calculator package
from src.calculator import power_mod

@click.command()
@click.option('--base', default=1, help='base of a number')
@click.option('--power', default=0, help='power for the number')
def power(base, power):
    """ Simple program that calculate base number raise to the poer
    """
    click.echo(power_mod.powerup(base, power))

What if I have too many Commands

In the example before, we use this line cli.add_command(commands.power)

to add our command to the main command group. If we have too many commands and don't want to have too many similar lines like

cli.add_command(command a)
cli.add_command(command b)
......
cli.add_command(command z)

We can utilize the command group. Reference and example below:

# Content of calc.__main__.py
import click

from commands import calc_cli

@click.group()
def cli():
    pass

# this line add the calc_cli group command to the command line group cli
cli.add_command(calc_cli)

# Content of calc.commands.py
import click
from calculator import power_mod

@click.group()
def calc_cli():
    pass

# First calc_cli.command() wrap this function to be a command under 
# calc_cli group
@calc_cli.command()
@click.option('--base', default=1, help='base of a number')
@click.option('--power', default=0, help='power for the number')
def power(base, power):
    """ Simple program that calculate base number raise to the poer
    """
    click.echo(power_mod.powerup(base, power))

Running the commands become

# With poetry, note the addition of sub command calc-cli 
poetry run calcx calc-cli power --base 2 --power 3
# Should return 8

# With Pip install
calcx calc-cli power --base 2 --power 3
# Should return 8

Ending

The reference code is available at this GitHub repo.