Skip to content

DATA_README.md

Latest commit

 

History

History
255 lines (203 loc) · 9.33 KB

DATA_README.md

File metadata and controls

255 lines (203 loc) · 9.33 KB

Consoul Static Data

This directory contains static data files used by Consoul for enhanced model discovery and rich descriptions.

Files

ollama_models.json

Purpose: Comprehensive database of Ollama library models with descriptions and metadata.

Generated by: scripts/scrape_ollama_library.py

Used by:

  • src/consoul/ai/ollama_library.py - Provides fallback when ollama.com is unreachable
  • src/consoul/sdk/services/model.py - Enriches Ollama model descriptions in LocalModelCard

Update frequency: Should be refreshed when new models are added to ollama.com (monthly recommended)

How It Works

1. Model Description Enrichment Flow

┌─────────────────────────────────────────────────────────────┐
│ User opens Enhanced Model Picker                           │
└─────────────────────────────┬───────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ ModelService.list_ollama_models(enrich_descriptions=True)  │
└─────────────────────────────┬───────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ fetch_library_models() attempts to get descriptions:       │
│                                                             │
│  1️⃣  Check ~/.consoul/cache/ollama_library_library.json   │
│      (24-hour cache)                                        │
│                                                             │
│  2️⃣  If cache expired/missing, scrape ollama.com          │
│      (with BeautifulSoup)                                   │
│                                                             │
│  3️⃣  If web scraping fails, use data/ollama_models.json   │
│      (static fallback - THIS FILE!)                         │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Descriptions mapped to local models by name                │
│ - "llama3.2:latest" → base name "llama3.2"                 │
│ - Look up description from library data                    │
│ - Return: "Meta's Llama 3.2 goes small with 1B and 3B..."  │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ LocalModelCard displays rich description + context size    │
└─────────────────────────────────────────────────────────────┘

2. Updating the Static Data

When to update:

  • New popular models released on ollama.com
  • Monthly maintenance refresh
  • Before major releases

How to update:

# Quick update (basic info, ~1 second)
poetry run python scripts/scrape_ollama_library.py \
    --basic \
    --pretty \
    --output data/ollama_models.json

# Full update (detailed info, ~3-5 minutes)
poetry run python scripts/scrape_ollama_library.py \
    --pretty \
    --output data/ollama_models.json \
    --delay 1.5

# Test update (first 10 models)
poetry run python scripts/scrape_ollama_library.py \
    --limit 10 \
    --basic \
    --pretty \
    --output data/ollama_models_test.json

Verification:

# Check file exists and has models
python -c "
import json
with open('data/ollama_models.json') as f:
    data = json.load(f)
    print(f'Total models: {len(data)}')
    print(f'First model: {data[0][\"name\"]} - {data[0][\"description\"][:60]}...')
"

# Test in Consoul
poetry run python -c "
from consoul.ai.ollama_library import load_static_models
models = load_static_models()
print(f'✅ Loaded {len(models)} models from static data')
"

3. File Format

The ollama_models.json file contains an array of model objects:

[
  {
    "name": "llama3.2",
    "full_name": "llama3.2",
    "description": "Meta's Llama 3.2 goes small with 1B and 3B models.",
    "url": "https://ollama.com/library/llama3.2",
    "model_url": "/library/llama3.2",
    "num_pulls": "Pulls",
    "num_tags": "Tags",
    "updated": "Updated",
    "license": "",
    "tags": [],
    "supports_vision": false,
    "supports_tools": false,
    "context_length": "",
    "family": "",
    "readme": ""
  }
]

Fields:

  • name (str): Model identifier (matches Ollama model name)
  • description (str): Most important - human-readable model description
  • url (str): Full URL to model page
  • model_url (str): Relative URL path
  • num_pulls, num_tags, updated: Metadata (may be literal strings in basic mode)
  • tags, license, context_length, family, readme: Additional metadata (populated in detailed mode)
  • supports_vision, supports_tools: Capability flags (populated in detailed mode)

Benefits

For Users

  • ✅ Rich descriptions even when offline or ollama.com is down
  • ✅ Faster startup - no web scraping delay on first run
  • ✅ Consistent UX - descriptions always available
  • ✅ Better model discovery - understand what each model does before downloading

For Developers

  • ✅ Fallback reliability - graceful degradation when web scraping fails
  • ✅ Version control - model descriptions tracked in git
  • ✅ CI/CD friendly - no external dependencies for model metadata
  • ✅ Testable - deterministic data for tests

Maintenance Checklist

  • Update data/ollama_models.json monthly
  • Verify file loads correctly after updates
  • Test enhanced model picker with new data
  • Commit updates with message: chore: update ollama models database
  • Document any new notable models in release notes

Integration Points

ollama_library.py

# Primary function that uses static data
from consoul.ai.ollama_library import fetch_library_models, load_static_models

# Automatically uses static fallback
models = fetch_library_models()  # Uses cache → web → static data

# Directly load static data
models = load_static_models()  # Always uses data/ollama_models.json

ModelService

from consoul.config import load_config
from consoul.sdk.services.model import ModelService

config = load_config()
service = ModelService.from_config(config)

# Enhanced descriptions automatically use static fallback
models = service.list_ollama_models(enrich_descriptions=True)

for model in models:
    print(f"{model.name}: {model.description}")
    # Output: "llama3.2:latest: Meta's Llama 3.2 goes small with 1B and 3B models. (1.9GB)"

EnhancedModelPicker

The LocalModelCard widget automatically displays enriched descriptions from the static data:

from consoul.tui.widgets.enhanced_model_picker import EnhancedModelPicker

# Model descriptions are automatically enriched
picker = EnhancedModelPicker(
    current_model="llama3.2",
    current_provider=Provider.OLLAMA,
    model_service=model_service,
)
# Shows: "Meta's Llama 3.2 goes small with 1B and 3B models. (1.9GB)"
# Instead of: "Local Ollama model (1.9GB)"

Troubleshooting

"No static models found"

# Check if file exists
ls -lh data/ollama_models.json

# Regenerate if missing
poetry run python scripts/scrape_ollama_library.py --basic --pretty --output data/ollama_models.json

"Static models loaded but descriptions empty"

# Verify JSON structure
head -50 data/ollama_models.json

# Check for "description" field
grep -o '"description"' data/ollama_models.json | wc -l
# Should equal number of models

"Descriptions not showing in TUI"

# Test description enrichment
from consoul.sdk.services.model import ModelService
service = ModelService.from_config()
models = service.list_ollama_models(enrich_descriptions=True)
print(models[0].description)
# Should show rich description, not "Local Ollama model"

See Also

  • scripts/README.md - Detailed scraper documentation
  • scripts/scrape_ollama_library.py - Data generation script
  • src/consoul/ai/ollama_library.py - Static data loader implementation
  • examples/sdk/enhanced_descriptions_example.py - Usage examples