Fancy Command Line Interfaces · 125 Summer of Side Projects

Repo and Recording

You might have seen programmers everywhere using the command line for all sorts of daily tasks. It looks kind of unfriendly and complicated, right?

How about we make things a little prettier and easier for people to use?

print("Hello command line")

Hosted by: Harsh Deep

Motivation

Here’s an 18 minute audio recording made specially for this summer session by Donovan Keme, my mentor last summer who broadened my perspective on tech entirely. His open source Ruby work has almost 450 million downloads, and he’s very involved with the community.

This is a more somber and serious recording instead of the typical pep talk, but it’s important to consider in your journey after several months about learning about technology. At this point, it’s the right time to start thinking about what role technology plays in the world and what purpose you want to achieve with technology.

Recording link

Don’t worry if you don’t know what your purpose is right now. What this means can be different for everyone. Over the next few months, start to reflect and chart out your goals of what you want to do and the steps you want to take from there. Feel free to message any of the staff or create a forum post if you want to talk about it too.

Getting Ready

Learn a bit of git on the commmand line. You can use any intro tutorial, but this web browser one looks good.
Go over the basics of test driven development with pytest in this tutorial on our site. Code along everything just to get an idea of why we test and how to create your workflow around tests. While I don’t want all projects to have extensive tests (although it is a good habit), having even just 3-5 tests makes a huge difference.
Learn some basics of how a command line works. Here is a quick crash course with links to resources to learn even more. The command line is intimidating, but once you get used to it, you’ll wonder how you lived without it. It’s part of almost every programmer’s toolbox.

Workshop

Our goal is to make a fancy git add CLI with a nice interactive interface. We want to display a list of files that can be added to git and then have the user select which ones they want. Then we provide them an option to commit the files as well.

Note: I’ve done this on an Ubuntu 20.04 system. Your progress shouldn’t be drastically different, but you may need to adjust depending on how your OS does things. Feel free to post forum questions in case there are any problems. I used python 3.8.5.

Libraries

Create a file called requirements.txt and fill it with this. This is a common convention in Python projects for listing all the libraries required.

questionary
pyfiglet
plumbum
pylint
pytest

Once you’ve saved this file, run the following to install them (if you use python3 to execute commands, then use pip3):

$ pip install -r requirements.txt
[install output]

The -r flag tells pip to look at requirements.txt and install everything in it.

Let me explain how we’re using each library.

questionary give us fancy interactive menu interfaces.
pyfiglet provides ASCII art displays.
plumbum gives us a way of accepting input, displaying help information and calling existing system commands.
pylint is for style checking.
pytest is for simple testing.

Get Started

Create a file called fgit.py.

We’ll use cli from the plumbum library to create the base template for our command line application.

from plumbum import cli

class FancyGitAdd(cli.Application):
    def main(self):
        print("Welcome to Fancy Git Add!")

if __name__ == "__main__":
    FancyGitAdd()

Now when we run this, we get our starter message:

$ python fgit.py
Welcome to Fancy Git Add!

The reason we have a __name__ == "__main__" condition is because our program will be run in different ways. When running the program normally through Python, the condition will be true and the code inside will run, but in other cases it won’t.

Basically, without that, our tests will not work and give a screenful of nasty errors. We haven’t written any tests yet, but just to sanity check:

$ pytest fgit.py
[tests pass]

(If it says pytest is not in the path you can do python -m pytest fgit.py)

Which will tell us everything is good for now.

Provide Help

Almost every major command line program lets you ask for help using -h or --help. This is good because things can be quite confusing otherwise. Writing useful help messages for users is highly recommended.

Lets see what we get out of the box from plumbum.

$ python fgit.py --help
Usage:
    fgit.py [SWITCHES]

Meta-switches:
    -h, --help         Prints this help message and quits
    --help-all         Prints help messages of all sub-commands and quits
    -v, --version      Prints the program's version and quits

Not a bad start. Now we should give our program a version number. In the real world, remember to change the version number as you add new changes so that people know when to update.

from plumbum import cli

class FancyGitAdd(cli.Application):
    VERSION = "1.3"

    def main(self):
        print("Welcome to Fancy Git Add")

if __name__ == "__main__": 
    FancyGitAdd()  

This gets our version number display working nicely:

$ python fgit.py --version
fgit.py 1.3
$ python fgit.py -h
fgit.py 1.3

Usage:
    fgit.py [SWITCHES]

Meta-switches:
    -h, --help         Prints this help message and quits
    --help-all         Prints help messages of all sub-commands and quits
    -v, --version      Prints the program's version and quits

Note: One-character flags have a single dash - and are one letter long to be short. Longer flags (usually a word or phrase) have two dashes -.

Improve the display

The text output from print does look nice, but we want those fancy ASCII art displays. For this we’re going to use pyfiglet

from plumbum import cli 
from pyfiglet import Figlet

def print_banner(text):
    print(Figlet(font='slant').renderText(text))

class FancyGitAdd(cli.Application):
    VERSION = "1.3"

    def main(self):
        print_banner("Git Fancy add")

if __name__ == "__main__":
    FancyGitAdd()

We create a function print_banner(text) to print our ASCII banner on our screen based on the pyfiglet documentation.

In general, it’s good practice to extract out functions (like this one) so that we can create more layers of abstraction for readability or testing. The command line application itself doesn’t need to bother with the details of how we print a banner, just that it gets printed.

Collecting a List of Files

One of the most common commands in the command line is ls, which lists all files in the current folder.

I’m creating a folder called test_folder with two files: file1.txt and file2.txt. I’m also adding a file3.txt outside the folder. You can create whatever sample files you like.

If you want to do it the same way I am, use mkdir (create directory), cd (change directory), and touch (create file or update last edited).

$ mkdir test_folder
$ touch test_folder/file1.txt test_folder/file2.txt file3.txt

The end result looks something like this (generated via a command line program called tree):

$ tree
.
├── __pycache__
│   └── fgit.cpython-38-pytest-6.2.4.pyc
├── fgit.py
├── file3.txt
├── requirements.txt
└── test_folder
    ├── file1.txt
    └── file2.txt

Let’s try it out to get a list of 5 items I have (your system might vary, but adjust accordingly):

$ ls
__pycache__  fgit.py  file3.txt  requirements.txt  test_folder

Lets use this as a chance to show off the red-green-refactor workflow (more details in testing 101) with pytest.

Red

Now lets come up with a function get_files() which will return a List of the files in the given directory. We write a stub for that function and write a test test_get_files() to test it.

from pyfiglet import Figlet
from plumbum import cli

def print_banner(text):
    print(Figlet(font='slant').renderText(text))

def get_files():
    return [] # empty list for now

class FancyGitAdd(cli.Application):
    VERSION = "1.3"

    def main(self):
        print_banner("Git Fancy add")
        files = get_files()

if __name__ == "__main__":
    FancyGitAdd()

### TESTS

def test_get_files():
    files = get_files()
    assert len(files) == 5, "There should be enough files"

Now let’s run it with pytest:

$ pytest fgit.py
    def test_get_files():
        files = get_files()
>       assert len(files) == 5
E       assert 0 == 5
E        +  where 0 = len([])

fgit.py:39: AssertionError

This is what we expect. One we implement the feature, this test should become green.

Green

Using plumbum.cmd, we can import commands from the command line and use them within python. This import lets us import basically any existing system command so that we can use it in our program.

The first step of understanding how to process output of a command line program is to inspect it. Here, we’ll print it with the strings "start" and "end" to understand how the string processing is working.

from pyfiglet import Figlet
from plumbum import cli
from plumbum.cmd import ls

[...]

def get_files():
    ls_output = ls()
    print("start", ls_output, "end")
    return [] # empty list for now

[...]

def test_get_files():
    files = get_files()
    assert len(files) == 5, "There should be enough files"

And the output snippet we get is:

start __pycache__
fgit.py
file3.txt
requirements.txt
test_folder
 end

Looks like we have to split by line and take care of some whitespace around the start and end. We can use a combo of strip (strip is quite common to use when dealing with string output) and split. Our final function will be:

def get_files():
    ls_output = ls().strip()
    print("start", ls_output, "end")
    files = ls_output.split("\n")
    return files

[...]

def test_get_files():
    files = get_files()
    assert len(files) == 5, "There should be enough files"

Now if we run this with pytest, the color will be green.

$ pytest fgit.py
[test pass]

Refactor

One we have a working test for a feature, we can refactor (improve) our code with confidence. The test cases act as a safety net.

If we run our program:

$ python fgit.py
   _______ __     ____                                    __    __
  / ____(_) /_   / __/___ _____  _______  __   ____ _____/ /___/ /
 / / __/ / __/  / /_/ __ `/ __ \/ ___/ / / /  / __ `/ __  / __  /
/ /_/ / / /_   / __/ /_/ / / / / /__/ /_/ /  / /_/ / /_/ / /_/ /
\____/_/\__/  /_/  \__,_/_/ /_/\___/\__, /   \__,_/\__,_/\__,_/
                                   /____/

start __pycache__
fgit.py
file3.txt
requirements.txt
test_folder end

The debug output still appears. A refactor step would be removing that and checking if the tests still work.

Let’s update get_files():

def get_files():
    ls_output = ls().strip()
    files = ls_output.split("\n")
    return files

Running the tests should still result in green:

$ pytest fgit.py
[tests pass]

Our full code at this stage should be

from pyfiglet import Figlet
from plumbum import cli
from plumbum.cmd import ls

def print_banner(text):
    print(Figlet(font='slant').renderText(text))

def get_files():
    ls_output = ls().strip()
    files = ls_output.split("\n")
    return files

class FancyGitAdd(cli.Application):
    VERSION = "1.3"

    def main(self):
        print_banner("Git Fancy add")
        files = get_files()

if __name__ == "__main__":
    FancyGitAdd()

def test_get_files():
    files = get_files()
    assert len(files) == 5, "There should be enough files"

Note: This is a simple test for teaching reasons. In real life you should write more robust tests since this could easily break if I have one more or one less file. We want to account for all edge cases and potential changes.

Now let’s create a checkbox-like system that lets the user move up and down through the files. We’ll use questionary to ask. The format for checkbox questions is similar to this example from their documentation.

Here I will show another style of testing which I call “green-refactor”. Sometimes creating a failing test first is quite tedious, so I create the feature and then add a test for it. Over time, you’ll figure out your style of how to do things. The important thing is creating tests as you move along.

Green

First we create a new function for generating the questions in the right format:

def generate_question(files):
    return [{
        'type': 'checkbox',
        'name': 'files',
        'message': 'What would you like to add?',
        'choices': [{'name': file} for file in files],   
    }]

For generating our 'choices', we use Python List Comprehension, which is one of my favorite features from the language.

We add the import for prompt:

from questionary import prompt

In our main function, we slot this in and print to see if we got the right output.

def main(self):
    print_banner("Git Fancy add")
    files = get_files()
        
    question = generate_question(files)
    answers = prompt(question)
    print(answers['files'])

The output now shows the choices in an array.

? What would you like to add?  done (2 selections)
['file3.txt', 'requirements.txt']

Now let’s write a test for this. We can put it at the bottom like the other one.

def test_generate_question():
    files = ["best.rb", "good.kt", "small.py"]
    question = generate_question(files)
    assert len(question) == 1, "has to be one question"
    assert question[0]['type'] == 'checkbox', "has to allow multiple selections"
    assert len(question[0]['choices']) == len(files), "same number of choices as files"

If we run the test ($ pytest fgit.py), it should pass.

Refactor

Now that we have a test acting as a safety net, we can refactor with confidence. Remembering our difficulties from earlier, we decide to add a strip() to each menu option. With luck, this shouldn’t break everything.

def generate_questions(files):
    questions = [{
        'type': 'checkbox',
        'name': 'files',
        'message': 'What would you like to add?',
        'choices': [{'name': file.strip()} for file in files],   
    }]
    return questions

For generating our 'choices', we use Python List Comprehension, which is one of my favorite features from the language.

If we run pytest fgit.py, we’re still good.

At this point our full file should be:

from plumbum import cli 
from pyfiglet import Figlet
from plumbum.cmd import ls
from questionary import prompt

def print_banner(text: str):
    print(Figlet(font='slant').renderText(text))

def get_files():
    ls_output = ls().strip()
    files = ls_output.split("\n")
    return files

def generate_question(files):
    return [{
        'type': 'checkbox',
        'name': 'files',
        'message': 'What would you like to add?',
        'choices': [{'name': file.strip()} for file in files],   
    }]


class FancyGitAdd(cli.Application):
    VERSION = "1.3"
    def main(self):
        print_banner("Git Fancy add")
        files = get_files()
        
        question = generate_question(files)
        answers = prompt(question)
        print(answers['files'])

if __name__ == "__main__":
    FancyGitAdd()

### TESTS

def test_get_files():
    files = get_files()
    assert len(files) == 5, "There should be enough files"

def test_generate_question():
    files = ["best.rb", "good.kt", "small.py"]
    question = generate_question(files)
    assert len(question) == 1, "has to be one question"
    assert question[0]['type'] == 'checkbox', "has to allow multiple selections"
    assert len(question[0]['choices']) == len(files), "same number of choices as files"

Note: Fancier Syntax

This is based on inquirer.js dictionary syntax which is common in a lot of languages now. questionary also supports that syntax but it has it’s own syntax which you might find nicer but it’s also hard to test (in most cases that’s okay). If I had to use that I’d use:

[...]
files = get_files()
question = questionary.select("What would you like to add?", choices=files)
to_add = question.ask()

Check out the questionary docs for more question types and fancy things you can do.

Exercise

Filter the list of files to not have anything starting with _. Update your test case accordingly to 4 items since __pycache__ will not be included. See this guide on filter in python.

Calling Git Add

Now that we have the list of files to add, let’s actually pass them to git.

Similar to ls, we take advantage of plumbum.cmd.

from plumbum.cmd import ls, git

def main(self):
    print_banner("Git Fancy add")
    files = get_files()

    questions = generate_questions(files)
    answers = prompt(questions)
    git('add', answers['files'])

And if we run this and check the git status output, we can see that our program worked.

$ python3 fgit.py
   _______ __     ____                                    __    __
  / ____(_) /_   / __/___ _____  _______  __   ____ _____/ /___/ /
 / / __/ / __/  / /_/ __ `/ __ \/ ___/ / / /  / __ `/ __  / __  /
/ /_/ / / /_   / __/ /_/ / / / / /__/ /_/ /  / /_/ / /_/ / /_/ /
\____/_/\__/  /_/  \__,_/_/ /_/\___/\__, /   \__,_/\__,_/\__,_/
                                   /____/

? What would you like to add?  [fgit.py]
$ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        modified:   fgit.py

Commit Flag

Now we’re going to let people git commit as well. They can add -c/--commit to enable committing.

Normally I’d commit like this (ideally with a more helpful message):

$ git commit -m "add files"

Luckily for us, plumbum makes adding another flag relatively easy. If we use cli.Flag, the result will be attached to self and we can use it in our program.

class FancyGitAdd(cli.Application):
    VERSION = "1.3"
    commit = cli.Flag(['c', 'commit'], help="Commits the added files as well")

    def main(self):
        print_banner("Git Fancy add")
        files = get_files()

        questions = generate_questions(files)
        answers = prompt(questions)
        git('add', answers['files'])
        if self.commit:
            git('commit', '-m', 'updates')

And when we run it, we can verify with git log:

$ python fgit.py -c
   _______ __     ____                                    __    __
  / ____(_) /_   / __/___ _____  _______  __   ____ _____/ /___/ /
 / / __/ / __/  / /_/ __ `/ __ \/ ___/ / / /  / __ `/ __  / __  /
/ /_/ / / /_   / __/ /_/ / / / / /__/ /_/ /  / /_/ / /_/ / /_/ /
\____/_/\__/  /_/  \__,_/_/ /_/\___/\__, /   \__,_/\__,_/\__,_/
                                   /____/

? What would you like to add?  [fgit.py]
$ git log
commit ece48b181c8e88124dea8aa47001c6dc41786e64 (HEAD -> main)
Author: Harsh Deep <>
Date:   Mon Jun 7 06:44:12 2021 -0500

    updates

The help output will also show the input for the new option.

$ python fgit.py --help
fgit.py 1.3

Usage:
    fgit.py [SWITCHES]

Meta-switches:
    -h, --help         Prints this help message and quits
    --help-all         Prints help messages of all sub-commands and quits
    -v, --version      Prints the program's version and quits

Switches:
    -c, --commit       Commits the added files as well

Final Code

from plumbum import cli 
from pyfiglet import Figlet
from plumbum.cmd import ls, git
from questionary import prompt

def print_banner(text: str):
    print(Figlet(font='slant').renderText(text))

def get_files():
    ls_output = ls().strip()
    files = ls_output.split("\n")
    return files

def generate_question(files):
    return [{
        'type': 'checkbox',
        'name': 'files',
        'message': 'What would you like to add?',
        'choices': [{'name': file.strip()} for file in files],   
    }]

class FancyGitAdd(cli.Application):
    VERSION = "1.3"
    commit = cli.Flag(['c', 'commit'], help="Commits the added files as well")
    def main(self):
        print_banner("Git Fancy add")
        files = get_files()
        
        question = generate_question(files)
        answers = prompt(question)
        git('add', answers['files'])
        if self.commit:
            git('commit', '-m', 'updates')

if __name__ == "__main__":
    FancyGitAdd()

### TESTS

def test_get_files():
    files = get_files()
    assert len(files) == 5, "There should be enough files"

def test_generate_question():
    files = ["best.rb", "good.kt", "small.py"]
    question = generate_question(files)
    assert len(question) == 1, "has to be one question"
    assert question[0]['type'] == 'checkbox', "has to allow multiple selections"
    assert len(question[0]['choices']) == len(files), "same number of choices as files"

Running $pytest fgit.py should have all tests passing. (Don’t worry about warnings.)

Exercise

Add a optional command line flag (also called switch) called -p/--push to run git push at the end of the program.

Learn More About Fancy CLIs

Plumbum scripting - Learn more about the library used to interact with the actual system programs.
Building Beautiful Command Line Interfaces with Python - A good introduction to pretty command line libraries. Examples included!
Python Command Line Apps - Some theory on the aspects of command line apps, as well as library recommendations.

Learn More Python Concepts

How to use APIs in Python Beginner (basic requests and responses) and then do Intermediate (headers, authentication, pagination, rate limiting, cache, combining API requests). Recommended if you’re planning to use your CLI to wrap around an API
Map, Filter, Reduce and List Comprehension for doing more list based stuff in python
os module in python to interact with the operating system

Ethics

towards data science - The ethical automation toolkit - this article starts with a nuanced views towards existing criticism towards automation and looking towards a framework towards ethical automation.

Ideas

Feel free to come up with anything you want as long as it’s CLI related. Here are some ideas to help you get started, but feel free to come up with more. Don’t worry if it’s already been done or if someone else is doing it. The point is to learn and have fun. :)

Take some application you commonly use and try to make a command line version. For example, gratitude journals, weather updates or getting scores from a sports team of your choice.
Is there some commmand line application you found annoying or hard to use? How would you improve it?
Maybe the chatbot from last week could also be available in command line form. Could you spice it up with fancy menu selectors or ASCII displays?
Maybe you can use this to automate some workflow task you have by combining some steps together with plumbum. One of the greatest things about command lines is that you can use each program as a modular piece to compose very powerful utilities.

Requirements

Runs from your command line.
Has a version number.
Has a working -h/--help option.
Accepts at least one custom flag.
There should be at least two test cases.
Has to have a README where you explain how it works along with pictures/animations/videos.
Fix as many pylint errors as you can. Some errors are weird or might take a long time to fix, but the rest you definitely should.
Has to use an Open Source license via a LICENSE file.

Contributors: Harsh, Maaheen

« Chatbots Web Bots »

Motivation

Getting Ready

Workshop

Libraries

Get Started

Provide Help

Improve the display

Collecting a List of Files

Red

Green

Refactor

Fancy Menu Selectors

Green

Refactor

Note: Fancier Syntax

Exercise

Calling Git Add

Commit Flag

Final Code

Exercise

Learn More About Fancy CLIs

Learn More Python Concepts

Ethics

Ideas

Requirements