readme

Author: zirkelc

README.md

@@ -58,101 +58,6 @@ Preserving a complete semantic unit like a section, paragraph, sentence, etc., i
 
 [Comparison of chunk size 200 with 1.5x overflow ratio: Chunkdown (left) / LangChain Markdown Splitter (right)](https://chunkdown.zirkelc.dev/?text=LSBbYGdlbmVyYXRlVGV4dGBdKC9kb2NzL2FpLXNkay1jb3JlL2dlbmVyYXRpbmctdGV4dCk6IEdlbmVyYXRlcyB0ZXh0IGFuZCBbdG9vbCBjYWxsc10oLi90b29scy1hbmQtdG9vbC1jYWxsaW5nKS4KICBUaGlzIGZ1bmN0aW9uIGlzIGlkZWFsIGZvciBub24taW50ZXJhY3RpdmUgdXNlIGNhc2VzIHN1Y2ggYXMgYXV0b21hdGlvbiB0YXNrcyB3aGVyZSB5b3UgbmVlZCB0byB3cml0ZSB0ZXh0IChlLmcuIGRyYWZ0aW5nIGVtYWlsIG9yIHN1bW1hcml6aW5nIHdlYiBwYWdlcykgYW5kIGZvciBhZ2VudHMgdGhhdCB1c2UgdG9vbHMuCi0gW2BzdHJlYW1UZXh0YF0oL2RvY3MvYWktc2RrLWNvcmUvZ2VuZXJhdGluZy10ZXh0KTogU3RyZWFtIHRleHQgYW5kIHRvb2wgY2FsbHMuCiAgWW91IGNhbiB1c2UgdGhlIGBzdHJlYW1UZXh0YCBmdW5jdGlvbiBmb3IgaW50ZXJhY3RpdmUgdXNlIGNhc2VzIHN1Y2ggYXMgW2NoYXQgYm90c10oL2RvY3MvYWktc2RrLXVpL2NoYXRib3QpIGFuZCBbY29udGVudCBzdHJlYW1pbmddKC9kb2NzL2FpLXNkay11aS9jb21wbGV0aW9uKS4KLSBbYGdlbmVyYXRlT2JqZWN0YF0oL2RvY3MvYWktc2RrLWNvcmUvZ2VuZXJhdGluZy1zdHJ1Y3R1cmVkLWRhdGEpOiBHZW5lcmF0ZXMgYSB0eXBlZCwgc3RydWN0dXJlZCBvYmplY3QgdGhhdCBtYXRjaGVzIGEgW1pvZF0oaHR0cHM6Ly96b2QuZGV2Lykgc2NoZW1hLgogIFlvdSBjYW4gdXNlIHRoaXMgZnVuY3Rpb24gdG8gZm9yY2UgdGhlIGxhbmd1YWdlIG1vZGVsIHRvIHJldHVybiBzdHJ1Y3R1cmVkIGRhdGEsIGUuZy4gZm9yIGluZm9ybWF0aW9uIGV4dHJhY3Rpb24sIHN5bnRoZXRpYyBkYXRhIGdlbmVyYXRpb24sIG9yIGNsYXNzaWZpY2F0aW9uIHRhc2tzLgotIFtgc3RyZWFtT2JqZWN0YF0oL2RvY3MvYWktc2RrLWNvcmUvZ2VuZXJhdGluZy1zdHJ1Y3R1cmVkLWRhdGEpOiBTdHJlYW0gYSBzdHJ1Y3R1cmVkIG9iamVjdCB0aGF0IG1hdGNoZXMgYSBab2Qgc2NoZW1hLgogIFlvdSBjYW4gdXNlIHRoaXMgZnVuY3Rpb24gdG8gW3N0cmVhbSBnZW5lcmF0ZWQgVUlzXSgvZG9jcy9haS1zZGstdWkvb2JqZWN0LWdlbmVyYXRpb24pLg%3D%3D&tab=aiSdk)
 
-## How It Works
-
-Chunkdown employs a sophisticated multi-layered approach that combines AST-based parsing with hierarchical processing to create semantically meaningful chunks while preserving markdown formatting.
-
-### Algorithm Overview
-
-Chunkdown uses a **hierarchical divide-and-conquer approach** that respects document structure:
-
-1. **Structure Recognition**: Parse markdown into a tree where headings organize their related content into logical sections
-2. **Smart Chunking**: Keep complete sections together when possible, intelligently merge related sections to maximize space utilization  
-3. **Graceful Degradation**: When sections are too large, progressively break them down using semantic boundaries (sentences, then phrases, then words)
-4. **Format Preservation**: Protect meaningful constructs like links and code from being split, maintaining markdown integrity throughout
-
-### Step-by-Step Process
-
-#### 1. AST Parsing and Hierarchical Transformation
-- Parse markdown using `mdast-util-from-markdown` with GitHub Flavored Markdown support
-- Transform flat AST into hierarchical sections using `createHierarchicalAST()`
-- **Key insight**: Headings become containers that hold their related content and nested subsections
-- Content size calculated from actual text content, not raw markdown characters
-
-#### 2. Top-Down Section Processing
-- **Size evaluation**: Check if entire sections fit within `maxAllowedSize` (chunkSize × maxOverflowRatio)
-- **Keep together**: Sections within limits are preserved as single chunks to maintain semantic coherence
-- **Break down intelligently**: Large sections are decomposed using multiple optimization strategies
-
-#### 3. Hierarchical Optimization Strategies
-
-**Parent-Child Merging**:
-- Attempt to merge parent section (heading + immediate content) with child sections
-- Find consecutive children that fit together with parent within size limits
-- Create merged sections while preserving hierarchical relationships
-
-**Sibling Section Merging**:
-- Group consecutive sibling sections at the same depth level
-- Create "orphaned sections" (no heading, depth 0) to combine related siblings
-- Maximize chunk utilization while respecting semantic boundaries
-
-**Content Grouping**:
-- Within sections, group content items to maximize chunk space utilization
-- Use `flushCurrentItems()` pattern to accumulate content until size limits are reached
-- Handle both regular sections (with headings) and orphaned sections (content-only)
-
-#### 4. Container-Specific Processing
-
-**Lists**: 
-- Process items individually while preserving list structure
-- Maintain correct numbering for ordered lists using `start` attribute
-- Group items when they fit within size limits
-
-**Tables**:
-- Keep table headers with data rows when possible 
-- Handle header-separator combinations for proper table formatting
-- Split by rows when table is too large
-
-**Blockquotes**:
-- Treat as containers with recursive item processing
-- Preserve quote formatting across chunks
-
-#### 5. Text-Level Fallback Mechanism
-
-When hierarchical processing cannot reduce content to acceptable sizes:
-
-**Protected Range Extraction**:
-- Re-parse content to extract position information for inline constructs
-- Protect links, images, inline code, emphasis, and other formatting from mid-construct splits
-- Create `ProtectedRange` objects that define no-split zones
-
-**Priority-Based Boundary Detection**:
-- Extract semantic boundaries with priority hierarchy (lower number = higher priority):
-  - Periods before newlines (priority 0)
-  - Periods before uppercase letters (priority 1) 
-  - Question/exclamation marks (priority 2)
-  - Safe periods (not abbreviations) (priority 3)
-  - Colons and semicolons (priority 4)
-  - Brackets and quotes (priority 5-8)
-  - Line breaks, commas, dashes (priority 9-12)
-  - Whitespace (priority 13, lowest)
-
-**Recursive Boundary Splitting**:
-- Try boundaries in priority order (highest priority first)
-- Find optimal split position using middle-point strategy
-- Recursively process both parts with remaining lower-priority boundaries
-- Ensure forward progress by creating smaller parts than original
-
-### Key Features in Action
-
-**Content-Based Sizing**: Uses `toString()` to extract plain text from AST nodes, ensuring consistent semantic density regardless of markdown complexity.
-
-**Controlled Overflow**: Allows chunks to exceed `chunkSize` up to `maxAllowedSize` to preserve complete semantic units like paragraphs or list items.
-
-**Semantic Preservation**: Multi-level protection system prevents breaking meaningful constructs, from document structure (sections) down to inline elements (links, code spans).
-
-**Structure Awareness**: Understands markdown document hierarchy and makes intelligent decisions about what content belongs together based on heading relationships.
-
 ## Usage
 
 > [!NOTE]