Building an LLM CLI from Scratch in Python
Streaming, multi-provider support, terminal UX, and the engineering beneath an AI tool.

Streaming, multi-provider support, terminal UX, and the engineering beneath an AI tool.
Before reaching for an agent framework, I wanted to understand what actually happens underneath an LLM-powered developer tool. Instead of building another web wrapper, I built a terminal-first client designed around a developer’s daily workflow.
The goal wasn’t just to query models, but to tackle the system design problems that framework abstractions hide: asynchronous streaming, cross-vendor normalization, terminal state management, and persistence.
Architecture and Structure
A robust CLI requires decoupling the interactive terminal loop from upstream vendor churn. The application separates concerns into explicit boundaries:
chatbot/
├── config/ # Model registries, pricing tables, env parsing
├── engine/ # Session orchestrator & slash command dispatcher
├── models/ # Pydantic v2 schemas (Message, Session, TokenUsage)
├── persistence/ # JSON/Markdown exporter and auto-save
├── providers/ # Concrete adapters (OpenAI, Anthropic, Ollama)
├── ui/ # Rich terminal themes, spinners, markdown panels
└── main.py # Entry point & CLI flags
Using Pydantic v2 for the domain layer isolates the application from external schema drift. Internally, conversations exist as strongly typed Message objects. Serializing sessions or restoring context across terminal reboots is handled via native .model_dump_json() and .model_validate_json(), eliminating custom serialization glue.
The Provider Abstraction
Every LLM vendor handles streaming differently. OpenAI uses server-sent events, Anthropic implements custom streaming events, and Ollama serves newline-delimited JSON over plain HTTP.
If any vendor SDK touches the engine or UI, changing models becomes an untangling nightmare. To fix this, all adapters implement an asynchronous generator interface:
# chatbot/providers/base.py
from abc import ABC, abstractmethod
from typing import AsyncIterator
from chatbot.models.message import Message
from chatbot.models.token_usage import TokenUsage
class BaseLLMProvider(ABC):
@property
@abstractmethod
def name(self) -> str:
"""Provider name: 'openai', 'anthropic', or 'ollama'"""
...
@abstractmethod
async def stream_chat(
self,
messages: list[Message],
model: str,
system_prompt: str | None = None,
) -> AsyncIterator[tuple[str, TokenUsage | None]]:
"""Yields: (token_text, optional_token_usage)"""
...
The contract is strict: yield raw text chunks sequentially as they arrive, and pass back token usage metadata on the final frame. The chat engine never knows — or cares — which vendor generated the text.
Streaming Ollama via HTTP
For local inference, a heavy SDK is unnecessary. Standard httpx streaming against the /api/chat endpoint handles the payload:
# chatbot/providers/ollama_provider.py
import json
import httpx
from typing import AsyncIterator
from chatbot.models.message import Message
from chatbot.models.token_usage import TokenUsage
from chatbot.providers.base import BaseLLMProvider
class OllamaProvider(BaseLLMProvider):
def __init__(self, base_url: str = "http://localhost:11434"):
self.base_url = base_url.rstrip("/")
@property
def name(self) -> str:
return "ollama"
async def stream_chat(
self,
messages: list[Message],
model: str,
system_prompt: str | None = None,
) -> AsyncIterator[tuple[str, TokenUsage | None]]:
payload_messages = [{"role": m.role, "content": m.content} for m in messages]
if system_prompt:
payload_messages.insert(0, {"role": "system", "content": system_prompt})
payload = {
"model": model,
"messages": payload_messages,
"stream": True,
}
async with httpx.AsyncClient(timeout=60.0) as client:
async with client.stream("POST", f"{self.base_url}/api/chat", json=payload) as response:
response.raise_for_status()
async for line in response.aiter_lines():
if not line:
continue
data = json.loads(line)
chunk = data.get("message", {}).get("content", "")
if data.get("done", False):
usage = TokenUsage(
prompt_tokens=data.get("prompt_eval_count", 0),
completion_tokens=data.get("eval_count", 0),
)
yield chunk, usage
else:
yield chunk, None
Switching between remote APIs and local models requires only swapping the provider instance in the engine.
Terminal UX Challenges
1. Handling Time to First Token (TTFT)
Models introduce a 500ms–2000ms delay before returning the first token. Overlaying a standard terminal spinner causes artifacts: once streaming starts, incoming text clashes with spinner escape sequences, corrupting the terminal scrollback.
Using Rich’s Live display with transient=True isolates the spinner:
# chatbot/ui/terminal.py
import sys
from rich.console import Console
from rich.live import Live
from rich.spinner import Spinner
console = Console()
async def stream_response(token_stream):
first_token = True
collected = []
console.print("\n[bold cyan]Assistant:[/bold cyan]")
with Live(Spinner("dots", text="[dim]Thinking...[/dim]"), console=console, transient=True) as spinner:
async for token, usage in token_stream:
if first_token:
spinner.stop() # Clears the line completely
first_token = False
if token:
sys.stdout.write(token)
sys.stdout.flush()
collected.append(token)
print()
return "".join(collected), usage
When the first token arrives, spinner.stop() wipes the line before standard output streaming begins.
2. Multi-Line Code Input
Python’s standard input() interprets newlines as submissions, making multi-line code pastes fail immediately. Replacing it with prompt_toolkit provides dedicated multi-line entry, keybindings, and history tracking:
from prompt_toolkit import PromptSession
from prompt_toolkit.history import InMemoryHistory
class InputManager:
def __init__(self):
self.session = PromptSession(history=InMemoryHistory())
self.multiline = False
def prompt(self) -> str:
prompt_text = ">>> " if not self.multiline else "[multi] >>> (Esc+Enter to submit)\n"
return self.session.prompt(prompt_text, multiline=self.multiline)
Large blocks of code or migrations can be pasted, modified inline, and dispatched via Esc + Enter.
3. Usage and Cost Metrics
To track token usage across long contexts, the client maintains a pricing registry and evaluates costs per turn:
PRICING = {
"gpt-4o": {"input_per_m": 2.50, "output_per_m": 10.00},
"claude-3-5-sonnet-20241022": {"input_per_m": 3.00, "output_per_m": 15.00},
"llama3.2": {"input_per_m": 0.00, "output_per_m": 0.00},
}
def calculate_cost(model: str, prompt_tokens: int, completion_tokens: int) -> float:
rates = PRICING.get(model, {"input_per_m": 0.0, "output_per_m": 0.0})
input_cost = (prompt_tokens / 1_000_000) * rates["input_per_m"]
output_cost = (completion_tokens / 1_000_000) * rates["output_per_m"]
return input_cost + output_cost
Executing /tokens renders active and cumulative metrics:
In-Session Commands
To modify session state without exiting the REPL, commands are handled directly by the dispatcher:
**/provider <name>**: Switch providers (anthropic,openai,ollama).**/model <name>**: Change model checkpoints without dropping session memory.**/system <prompt>**: Update runtime system instructions.**/save <filename.md>**: Export conversation history to Markdown on disk.**/clear**: Reset the context buffer.
Key Takeaways
Building the client directly against vendor APIs exposed several engineering fundamentals:
- Streaming shifts application control flow: A streaming API cannot be treated as a regular request that arrives in small pieces. It alters error-handling boundaries, terminal I/O states, and token-accounting lifecycles.
- Abstractions earn their keep at vendor boundaries: A single interface only makes sense if it isolates genuine differences in protocol, authentication, and streaming schemas.
- The scaffolding matters more than the LLM call: The actual API invocation is trivial. The durable engineering work lies in state persistence, concurrency, and predictable terminal ergonomics.
Code & Setup
The complete code, setup instructions, and configuration files are open source on GitHub: llm-cli-chatbot
Discussion
Have a question, a correction, or a different approach? Sign in with GitHub to join the conversation.