PHASE 17Advanced ~30 min· topic 8 of 10

Topic 17.8

Capstone: CLI, Config, Logging and Streaming

In one line

Turn the core library into a real command-line tool: argparse arguments, configuration from the environment with secrets handled safely, logging that helps you debug, clean error handling, streamed output, and retries with backoff.

Think of it like this

The front desk of a hotel. Guests do not walk into the kitchen. They talk to the front desk, which takes a clear request, checks it, passes it to the right department, and reports back in plain language, including 'sorry, that is not available' without showing the mess behind it. The command-line interface is the front desk of docqa: it parses what the user asked, checks it, calls the library, and turns problems into short, clear messages.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

CLI: Command-line interface: a program you run by typing its name and arguments in a terminal.
argv: The list of words typed after the program name.
Exit code: A number a program returns when it ends; 0 means success.
Environment variable: A named setting that the operating system gives to a program, such as an API key.
Log level: How important a log message is: DEBUG, INFO, WARNING, ERROR.
Backoff: Waiting longer before each retry.
Jitter: A small random extra wait, so many clients do not all retry at the same instant.
Transient error: A temporary problem that may disappear if you try again.

Step by step

01Design main(argv) so it can be tested

main takes argv and env as parameters instead of reading sys.argv and os.environ itself. The first runnable example calls it three times with fixed lists: a verbose run, a run with an invalid --top-k, and a run with no question at all. No terminal is needed, and each call returns an exit code you can assert on.

When argparse rejects the arguments it prints usage and the reason to standard error and raises SystemExit(2). In the example, the third call catches that so the program can continue; a real command line just exits.

02One Config object, built at the edge

Config.from_env(env, args) merges command-line options and environment variables into one frozen dataclass. Its describe() method prints the key as set or missing and never shows its value. This pattern prevents the most common secret leak: logging the whole config object while debugging.

03Logging for the person debugging at 2 a.m.

The example configures logging in main with logging.basicConfig(...), using force=True so repeated calls reconfigure it, and chooses DEBUG when -v is given and WARNING otherwise. In the output you can see the DEBUG line only in the verbose run, a WARNING for the missing key, and an ERROR for the invalid option. In a real tool log to standard error and keep standard output for the answer.

Log facts you will need later: how many documents and chunks were loaded, the top scores, the model name, token counts and latency. Never log the API key or whole prompts that contain private text unless you have decided that is acceptable.

04Retry transient errors, then give up clearly

The second runnable example defines TransientError, a model that fails twice and then succeeds, and with_retries, which doubles the delay each time. The sleep function is passed in (sleep=slow.append records the delays instead of waiting), so the demo and the tests run instantly. The last lines show what the caller sees when every attempt fails: an error log line and the exception.

Retry transient errors, then give up clearlydiagram

Rendering diagram…

05Wire it together

The final CLI function below parses arguments, builds the config, builds the retriever and the chosen client, prints the streamed answer, then prints the sources. It is shown as a non-runnable block because it needs the project's modules. Every piece of it was built and run on its own in this topic or the previous one.

src/docqa/cli.pywhole filepython

import logging
import sys

log = logging.getLogger("docqa")


def run(args, env) -> int:
    config = Config.from_env(env, args)
    docs = load_documents(Path(config.docs_dir))
    if not docs:
        log.error("no .txt files found in %s", config.docs_dir)
        print(f"No documents found in {config.docs_dir}", file=sys.stderr)
        return 1
    chunks = [c for d in docs for c in chunk_document(d)]
    retriever = Retriever(chunks, ToyEmbedder())
    hits = retriever.search(args.question, k=config.top_k)
    log.info("retrieved %d chunks, best score %.3f", len(hits), hits[0].score if hits else 0.0)

    client = make_client(config)                       # fake unless a key and model are set
    try:
        # Retry only until the first token arrives: after that, text is already on screen.
        first, rest = with_retries(lambda: open_stream(client, args.question, hits))
        print(first, end="", flush=True)
        for token in rest:
            print(token, end="", flush=True)
        print()
    except TransientError as error:
        print(f"The model service is busy, try again later ({error})", file=sys.stderr)
        return 1
    print("Sources:", ", ".join(sorted({h.chunk.doc for h in hits})))
    return 0


def open_stream(client, question, hits):
    """Start the stream and pull the first piece, so a failure at the start can be retried."""
    tokens = ask(client, question, hits)               # a generator: nothing runs until we ask for a piece
    return next(tokens, ""), tokens

Try it yourself

1
Break the arguments on purpose
In the first example, add a fourth call with ['hello', '--top-k', 'many'] and read what argparse says. Then wrap the call in try/except SystemExit and print the exit code, as the example does for the missing question.
2
Add jitter
In the second example, add random.seed(1) after the imports and call with_retries(..., jitter=0.1, sleep=slow.append). The delays are now slightly longer than 0.5 s and 1.0 s, and different on each retry, but still repeatable because of the seed. In production you would not seed it: the point of jitter is that every client picks a different wait.

Code & diagrams

argparse, Config and logging, called with fixed arguments New tab

Add an option such as '--docs other' and print it back, or change 'DOCQA_MODEL' in the env dictionary.

Expected output

DEBUG docqa: config: docs=docs top_k=2 model=fake api_key=missing
WARNING docqa: no API key set, using the fake model
asking 'fake': What is the refund policy?
exit code: 0
ERROR docqa: --top-k must be at least 1
exit code: 2
usage: docqa [-h] [--docs DOCS] [--top-k TOP_K] [-v] question
docqa: error: the following arguments are required: question
no question given -> SystemExit 2

Retries with exponential backoff against a flaky model New tab

Change `FlakyModel(failures=2)` to `failures=4` and read the error that reaches the caller. The delays are recorded, not slept.

Expected output

WARNING attempt 1 failed (429 too many requests), retrying in 0.5s
WARNING attempt 2 failed (429 too many requests), retrying in 1.0s
answer: Refunds take 5 days.
delays that would have been slept: [0.5, 1.0] | calls: 3
WARNING attempt 1 failed (429 too many requests), retrying in 0.5s
ERROR giving up after 2 attempts: 429 too many requests
raised to the caller: 429 too many requests

Using the tool from a terminalbash

Not run here: it needs the installed project. The usage text is what argparse prints when the question is missing.

$ docqa "What is the refund policy?" --docs ./docs --top-k 3
$ DOCQA_MODEL=MODEL_NAME DOCQA_API_KEY=... docqa "How long is shipping?" -v
$ docqa
usage: docqa [-h] [--docs DOCS] [--top-k TOP_K] [-v] question
docqa: error: the following arguments are required: question

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Pass a value of the wrong type to an option

Declare --top-k with type=int and pass --top-k many.

main.pywhole filepython

import argparse

parser = argparse.ArgumentParser(prog="docqa")
parser.add_argument("--top-k", type=int, default=3)
parser.parse_args(["--top-k", "many"])

terminal

$ python main.py

── what you'll see ──

usage: docqa [-h] [--top-k TOP_K]

docqa: error: argument --top-k: invalid int value: 'many'

Myth vs fact

Myth

print is fine for logging in small tools.

Fact

It works until you need levels, timestamps, files or silence in tests. logging costs three lines to set up and removes all of those problems.

Myth

Retry everything until it works.

Fact

Retrying a permanent error wastes time and money, and retrying without a limit or backoff can make an outage worse. Retry only transient errors, a limited number of times, with growing waits.

Myth

Environment variables are secure by themselves.

Fact

They keep secrets out of source code, but they can still leak through logs, crash reports or child processes. Never print them, and use a secrets manager in production.

When it breaks

The API key appears in a log file

What you see

Someone pasted the whole config object into a debug message. The key is now in log storage and in every copied support ticket, and must be revoked.

Fix & prevent

Give the config a describe() or __repr__ that hides secrets, rotate the exposed key immediately, and add a test that formats the config and asserts the key text is absent.

A retry loop hammers a rate-limited API

What you see

All instances retry at the same moments, the API keeps answering 429, and the job takes longer than if it had backed off.

Fix & prevent

Use exponential backoff with jitter, honour any retry-after hint from the response, cap attempts, and bound concurrency.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Libraries like Typer or Click build on the same ideas with less code, and argparse is in the standard library with no dependency. For a tool with subcommands (docqa index, docqa ask) use add_subparsers.
▸
Put timestamps and a request ID in production log lines, or emit JSON logs for a log platform. Keep the format configurable so tests can silence or capture it with caplog in pytest.
▸
Most SDKs include their own retry logic. Check before adding yours, so you do not multiply retries (3 attempts inside 3 attempts is 9 calls).

Remember this

1
**argparse builds the interface** (Topic 15.7). ArgumentParser turns a list of strings (normally sys.argv[1:]) into an object with typed attributes. A positional argument (question) is required; options (--docs, --top-k) have defaults and types; flags like -v use action="store_true". argparse writes --help, checks types (--top-k many is rejected), and exits with status 2 on bad usage. Design main(argv) to take the list as a parameter and return an exit code, so tests can call it with a fixed list and no real terminal.
2
Configuration and secrets (Topic 15.3). Settings come from three places, in this order of priority: command-line arguments, environment variables, then defaults. Put them into one frozen Config dataclass at the edge, and pass that around. Secrets (API keys) belong in environment variables or a secrets manager, never in code, in the repository or in logs. Log whether a key is set, never its value. Read configuration once, in main, so the rest of the code is easy to test with a hand-built Config.
3
**Logging is not print** (Topic 7.9). Use the logging module: create log = logging.getLogger("docqa") per module, log events at the right level (DEBUG for details, INFO for normal milestones, WARNING for something odd, ERROR for a failure), and configure the destination and format once in main. The answer goes to standard output (print), diagnostics go to the log (standard error by default), so users can pipe the answer elsewhere without log noise. Use lazy formatting, log.info("loaded %d chunks", n), so the string is only built if the level is enabled.
4
Streaming to the terminal. Printing each piece as it arrives (print(token, end="", flush=True)) makes the tool feel responsive, because the first words appear almost at once. Remember flush=True, otherwise the text may sit in a buffer (a waiting area in memory where Python holds output before showing it). Collect the pieces too so you can print the sources and save the full answer after the stream ends. If the user presses Ctrl-C, catch KeyboardInterrupt in main and exit cleanly.
5
Retries with backoff (Topics 15.2 and 16.4). Calls to a model API fail for temporary reasons: rate limits (HTTP 429), timeouts, brief server errors. Retry only those, wait longer each time (exponential backoff: 0.5 s, 1 s, 2 s...) and usually add random jitter so many clients do not retry in lockstep, and stop after a maximum number of attempts. Do not retry errors that will never succeed (a bad API key, an invalid request). Retry the whole call before any output has been shown; a half-printed streamed answer cannot be resumed safely.
6
Errors are for people. Catch the errors you expect (missing folder, no documents, no API key, retries exhausted) in main, log the details, print a one-line message, and return a non-zero exit code. Let unexpected errors raise with their traceback, because hiding them makes bugs impossible to find. Exit codes matter to scripts: 0 success, 2 bad usage (argparse's own), 1 for a general failure is a common convention.

Explain it without notes

Why should main accept argv as a parameter and return an exit code?

How do you handle secrets in a CLI tool?

When should you retry a failed API call, and how?

What is the difference between print and logging, and where should each go?

You wrap a streaming call in a retry loop, but the retry never fires. Why, and how do you retry a stream correctly?

Practice

Write a parser with a required positional name and an optional --times integer defaulting to 1, and call parse_args with a fixed list to show the result.

Write describe_config(env) that returns a string listing DOCQA_MODEL and whether DOCQA_API_KEY is set, without ever including the key value.

Write a function backoff_delays(attempts, base) returning the list of waits [base, base*2, base*4, ...] for the given number of retries.

Trade-offs

↔
**argparse versus Typer or Click.** argparse is built in and enough for most tools. Typer and Click give nicer syntax and automatic help from type hints, at the cost of a dependency.
↔
Stream versus wait for the full answer. Streaming feels faster and shows progress, but makes retries and post-processing harder (you cannot un-print). Stream the display, but decide how you handle errors halfway through.

Done when you can

Done when you can build an argparse CLI whose main takes a list and returns an exit code.
Done when you can load config from the environment into a frozen dataclass without leaking secrets.
Done when you can configure logging once and log at the right levels.
Done when you can write a retry loop with backoff that retries only transient errors.
Done when you can stream output with flush=True and still print sources afterwards.

Command Palette

Words you'll meet

Step by step

01Design main(argv) so it can be tested

02One Config object, built at the edge

03Logging for the person debugging at 2 a.m.

04Retry transient errors, then give up clearly

05Wire it together

Try it yourself

Break the arguments on purpose

Add jitter

Code & diagrams

Break it on purpose

Pass a value of the wrong type to an option

Myth vs fact

When it breaks

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can