Skip to content

v1.5.2.md

Latest commit

 

History

History
74 lines (54 loc) · 2.65 KB

v1.5.2.md

File metadata and controls

74 lines (54 loc) · 2.65 KB
Raw

v1.5.2 - Bibliography RAG Cleanup & Thumbnail Upscaling 🧹🖼️

Released: Feb 4, 2026

Summary

Paused the Bibliography Storage feature (RAG 1.0) after research revealed fundamental limitations. Added automatic thumbnail upscaling for talk images to improve social sharing quality.

Changes

Bibliography RAG Cleanup

  • Removed RAG 1.0 ingestion pipeline (naive chunking approach deemed insufficient)
  • Cleaned up generated SQL files (~14MB of broken/oversized inserts)
  • Removed Drizzle schema entry for bibliography_chunks table
  • Preserved PDF collection and research documentation for future use

Thumbnail Upscaling

  • All new talk thumbnails automatically upscaled to 1280×720 (YouTube maxresdefault equivalent)
  • Converted to WebP format for better compression (25-30% smaller than JPEG)
  • Uses Sharp with Lanczos3 interpolation for high-quality upscaling
  • Graceful fallback to original image if processing fails
  • Improves social sharing appearance on Twitter/X, Facebook, LinkedIn

Files Changed

New Files

  • lib/utils/image-processing.ts - Sharp-based image upscaling and WebP conversion

Modified Files

  • lib/supabase/storage.ts - Integrated image processing into upload pipeline
  • lib/db/schema.ts - Removed bibliographyChunks table definition

Deleted Files

  • scripts/output/ - Generated SQL files (6.7MB+)
  • scripts/ingest-bibliography.ts - PDF ingestion script
  • lib/db/migrations/0011_bibliography_chunks.sql - Database migration
  • lib/db/queries/bibliography.ts - Query functions

Preserved Files

  • docs/pdfs/ - 10 Tarot PDF books (for future RAG 2.0)
  • devnotes/Improving Document Ingestion Strategy.md - Research for better approach

Background

RAG 1.0 approach using pdf-parse + naive 4000-char chunking was found to have:

  • Layout Blindness - loses visual hierarchy from PDFs
  • Semantic Fracture - severs context across chunks
  • Keyword-only matching - misses semantic similarity

Future RAG 2.0 will use:

  • Docling (IBM) for PDF → structured Markdown
  • Jina Embeddings v3 with Late Chunking
  • Hybrid Search (semantic + keyword via RRF)

Testing Checklist

Bibliography Cleanup

  • Verify deleted files no longer exist
  • Verify bibliographyChunks removed from schema.ts
  • Verify PDFs preserved in docs/pdfs/

Thumbnail Upscaling