LLM Chain

An AI toolkit for configuring, benchmarking, comparing, prompting, evaluating, fine-tuning, analyzing, and optimizing LLM workflows. Each command logs timestamped entries to local files with full export, search, and statistics support.

Commands

Core AI Operations

Utility Commands

How It Works

Every core command accepts free-text input. When called with arguments, LLM Chain:

1. Timestamps the entry (YYYY-MM-DD HH:MM)
2. Appends it to the command-specific log file (e.g. benchmark.log, cost.log)
3. Records the action in a central history.log
4. Reports the saved entry and running total

When called with no arguments, each command displays the 20 most recent entries from its log file.

Data Storage

All data is stored locally in plain-text log files:

~/.local/share/llm-chain/
├── configure.log     # Configuration changes
├── benchmark.log     # Benchmark results
├── compare.log       # Model comparisons
├── prompt.log        # Prompt templates & notes
├── evaluate.log      # Evaluation metrics
├── fine-tune.log     # Fine-tuning sessions
├── analyze.log       # Analysis observations
├── cost.log          # Cost tracking
├── usage.log         # API usage metrics
├── optimize.log      # Optimization attempts
├── test.log          # Test cases & results
├── report.log        # Report entries
├── history.log       # Central activity log
└── export.{json,csv,txt}  # Exported snapshots

Each log uses pipe-delimited format: timestamp|value.

Requirements

• Bash 4.0+ with set -euo pipefail
• Standard Unix utilities: wc, du, grep, tail, date, sed
• No external dependencies — pure bash

When to Use

1. Tracking LLM experiments — log benchmark results, prompt variations, and evaluation scores as you iterate on model configurations
2. Cost monitoring — record token usage, API costs, and billing data to keep spending under control across multiple models
3. Comparing models side-by-side — use compare and benchmark to log performance differences between GPT-4, Claude, Gemini, etc.
4. Fine-tuning documentation — capture fine-tuning parameters, dataset info, and results for reproducibility
5. Generating operational reports — export all logged data to JSON/CSV for dashboards, audits, or stakeholder reviews

Examples

# Log a configuration change
llm-chain configure "switched to gpt-4o, temperature=0.7, max_tokens=2048"

# Record a benchmark result
llm-chain benchmark "gpt-4o MMLU=87.2% latency=1.3s cost=$0.012/req"

# Track a cost entry
llm-chain cost "2024-03-18: 142k tokens, $4.26 total (gpt-4o)"

# Compare two models
llm-chain compare "claude-3.5 vs gpt-4o: claude wins on reasoning, gpt wins on speed"

# Log a prompt engineering note
llm-chain prompt "added chain-of-thought prefix: 'Let me think step by step...'"

# Search all logs for a keyword
llm-chain search "gpt-4o"

# Export everything to JSON
llm-chain export json

# Check health and disk usage
llm-chain status

Configuration

Set the DATA_DIR variable in the script or modify the default path to change storage location. Default: ~/.local/share/llm-chain/

Powered by BytesAgain | bytesagain.com | hello@bytesagain.com