Skip to content

log.md

Latest commit

 

History

History
279 lines (211 loc) · 7.04 KB

log.md

File metadata and controls

279 lines (211 loc) · 7.04 KB

gitsqlite Logging Guide

This document provides comprehensive information about gitsqlite's logging capabilities for monitoring performance and troubleshooting issues during clean and smudge operations.

Overview

gitsqlite provides structured JSON logging to help you:

  • Monitor performance of database conversions
  • Debug issues with clean/smudge operations
  • Track timing and file operations
  • Identify bottlenecks in large database processing

Enabling Logging

Basic Logging

Enable logging to create timestamped log files in the current directory:

# Enable logging (creates log files in current directory)
gitsqlite -log clean < database.db > output.sql
gitsqlite -log smudge < database.sql > database.db

Custom Log Directory

Direct logs to a specific directory for better organization:

# Create logs directory and log there
mkdir logs
gitsqlite -log-dir ./logs clean < database.db > output.sql
gitsqlite -log-dir ./logs smudge < database.sql > database.db

Log File Format

File Naming Convention

Log files are created with unique timestamped names to prevent conflicts:

  • Format: gitsqlite-{operation}-{timestamp}.log
  • Example: gitsqlite-clean-20250818T103045.123Z.log

Log Entry Structure

Each log entry is a JSON object with the following fields:

Field Description Example
time ISO 8601 timestamp "2025-08-18T10:30:45.123Z"
level Log level (INFO, WARN, ERROR) "INFO"
msg Human-readable message "Clean operation completed"
operation Current operation type "clean" or "smudge"
duration Human-readable duration (HH:MM:SS.mmm) "00:01:23.456"
input_size Input file size in bytes 1048576
output_lines Number of SQL lines generated 1234

Example Log Entries

Successful Clean Operation

{
  "time": "2025-08-18T10:30:45.123Z",
  "level": "INFO",
  "msg": "gitsqlite started",
  "operation": "clean",
  "input_file": "/tmp/gitsqlite_temp_db_12345.db"
}
{
  "time": "2025-08-18T10:30:45.200Z",
  "level": "INFO",
  "msg": "Copy operation completed",
  "operation": "clean",
  "duration": "00:00:00.077",
  "bytes_copied": 1048576
}
{
  "time": "2025-08-18T10:30:46.350Z",
  "level": "INFO",
  "msg": "SQLite dump completed",
  "operation": "clean",
  "duration": "00:00:01.150",
  "output_lines": 1234
}
{
  "time": "2025-08-18T10:30:46.355Z",
  "level": "INFO",
  "msg": "Clean operation completed",
  "operation": "clean",
  "duration": "00:00:01.232",
  "input_size": 1048576,
  "output_lines": 1234
}

Error Example

{
  "time": "2025-08-18T10:31:02.685Z",
  "level": "ERROR",
  "msg": "sqlite3 command failed",
  "operation": "clean",
  "error": "unable to open database file",
  "exit_code": 1
}

Log File Locations

Default Location

  • Current directory: Log files are created where you run the command
  • Naming: gitsqlite-clean.log or gitsqlite-smudge.log

Custom Location

  • Specified directory: Files are created in the directory specified by -log-dir
  • Directory creation: The directory will be created if it doesn't exist
  • Permissions: Ensure write access to the target directory

Git Filter Integration

Enabling Logging for Git Filters

For debugging Git filter issues, you can enable logging in your Git configuration:

# Enable logging for Git filters
git config filter.gitsqlite.clean "gitsqlite -log clean"
git config filter.gitsqlite.smudge "gitsqlite -log smudge"

# Or with custom log directory
git config filter.gitsqlite.clean "gitsqlite -log-dir /tmp/gitsqlite-logs clean"
git config filter.gitsqlite.smudge "gitsqlite -log-dir /tmp/gitsqlite-logs smudge"

Temporary Logging for Debugging

# Temporarily enable logging for Git filters
git config filter.gitsqlite.clean "gitsqlite -log clean"
git config filter.gitsqlite.smudge "gitsqlite -log smudge"

# Perform Git operations that trigger filters
git add your_database.db
git commit -m "test"

# Check log files for issues
ls gitsqlite-*.log

# Disable logging when done
git config filter.gitsqlite.clean "gitsqlite clean"
git config filter.gitsqlite.smudge "gitsqlite smudge"

Performance Monitoring

Large Database Operations

For large databases (>100MB), monitoring logs helps track progress:

# Enable logging and monitor in real-time
gitsqlite -log clean < large_database.db > output.sql &
tail -f gitsqlite-clean-*.log

Timing Analysis

Log entries include human-readable durations (HH:MM:SS.mmm format) for:

  • Overall operation time
  • File copy operations
  • SQLite dump/restore operations
  • Cleanup operations

Best Practices

When to Enable Logging

  • Development: Always enable logging during development and testing
  • Troubleshooting: Enable when experiencing issues with database conversions
  • Performance: Monitor large database operations
  • Production: Consider enabling for Git filter operations in critical repositories

Log Management

  • Organization: Use custom log directories to keep logs organized
  • Retention: Regularly clean up old log files to save disk space
  • Analysis: Use JSON parsing tools to analyze log data programmatically

Security Considerations

  • File Paths: Log files may contain temporary file paths
  • Database Names: Original database filenames are logged
  • Permissions: Ensure log directories have appropriate access controls

Troubleshooting with Logs

Common Log Patterns

SQLite Not Found:

{
  "level": "ERROR",
  "msg": "sqlite3 command not found",
  "operation": "clean"
}

Permission Issues:

{
  "level": "ERROR",
  "msg": "failed to create temporary file",
  "operation": "clean",
  "error": "permission denied"
}

Invalid Database:

{
  "level": "ERROR",
  "msg": "sqlite3 dump failed",
  "operation": "clean",
  "error": "file is not a database"
}

Log Analysis Tips

  1. Check timestamps to identify slow operations
  2. Look for ERROR level entries first
  3. Verify file sizes match expectations
  4. Compare durations across similar operations

Command Line Examples

# Basic logging
gitsqlite -log clean < database.db > output.sql

# Custom log directory
gitsqlite -log-dir ./debug-logs clean < database.db > output.sql

# Combine with other flags
gitsqlite -sqlite /usr/local/bin/sqlite3 -log clean < database.db > output.sql

# Check logs after operation
cat gitsqlite-clean-*.log | jq '.msg'

Integration with Development Workflow

Automated Testing

Include logging in your test scripts:

#!/bin/bash
# Test script with logging
mkdir -p test-logs
gitsqlite -log-dir ./test-logs clean < test.db > test.sql
gitsqlite -log-dir ./test-logs smudge < test.sql > restored.db

# Check for errors in logs
if grep -q '"level":"ERROR"' test-logs/*.log; then
    echo "Test failed - check logs in test-logs/"
    exit 1
fi

CI/CD Integration

Preserve logs as artifacts in your CI/CD pipeline for debugging failed builds.

For more information about gitsqlite, see the main README.md.