LLM Flow

An AI toolkit for configuring, benchmarking, comparing, prompting, evaluating, fine-tuning, analyzing, and optimizing LLM workflows. Each command logs timestamped entries to local files with full export, search, and statistics support.

Commands

Core AI Operations

Utility Commands

How It Works

Every core command accepts free-text input. When called with arguments, LLM Flow:

1. Timestamps the entry (YYYY-MM-DD HH:MM)
2. Appends it to the command-specific log file (e.g. benchmark.log, cost.log)
3. Records the action in a central history.log
4. Reports the saved entry and running total

When called with no arguments, each command displays the 20 most recent entries from its log file.

Data Storage

All data is stored locally in plain-text log files:

~/.local/share/llm-flow/
├── configure.log     # Configuration changes
├── benchmark.log     # Benchmark results
├── compare.log       # Model comparisons
├── prompt.log        # Prompt templates & notes
├── evaluate.log      # Evaluation metrics
├── fine-tune.log     # Fine-tuning sessions
├── analyze.log       # Analysis observations
├── cost.log          # Cost tracking
├── usage.log         # API usage metrics
├── optimize.log      # Optimization attempts
├── test.log          # Test cases & results
├── report.log        # Report entries
├── history.log       # Central activity log
└── export.{json,csv,txt}  # Exported snapshots

Each log uses pipe-delimited format: timestamp|value.

Requirements

• Bash 4.0+ with set -euo pipefail
• Standard Unix utilities: wc, du, grep, tail, date, sed
• No external dependencies — pure bash

When to Use

1. Building AI agent workflows — log each step of your agent pipeline (configure → prompt → evaluate → optimize) with full traceability
2. Tracking LLM costs and usage — record per-request costs, token counts, and API usage to monitor spending across providers
3. Benchmarking and comparing models — log benchmark metrics side-by-side to make data-driven model selection decisions
4. Fine-tuning experiment tracking — capture hyperparameters, dataset details, and evaluation scores for every fine-tuning run
5. Generating compliance reports — export all logged activity to JSON/CSV for audits, SOC reviews, or stakeholder reporting

Examples

# Configure a new workflow
llm-flow configure "workflow: summarize → classify → respond, model=claude-3.5"

# Benchmark a model
llm-flow benchmark "claude-3.5-sonnet: 94% accuracy, 0.8s p50 latency, $0.003/req"

# Log a prompt template
llm-flow prompt "system: You are a helpful assistant. Always cite sources."

# Track API costs
llm-flow cost "March week 3: 890k tokens in, 210k tokens out, $12.40 total"

# Evaluate output quality
llm-flow evaluate "human eval score: 4.2/5.0 across 50 samples"

# Search across all logs
llm-flow search "claude"

# Export to CSV for analysis
llm-flow export csv

# Quick health check
llm-flow status

Configuration

Set the DATA_DIR variable in the script or modify the default path to change storage location. Default: ~/.local/share/llm-flow/

Powered by BytesAgain | bytesagain.com | hello@bytesagain.com