Skip to content

Commit 96f8665

Browse files
committed
caption
1 parent 6491118 commit 96f8665

1 file changed

Lines changed: 4 additions & 8 deletions

File tree

README.md

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -19,21 +19,17 @@ The **llama** ([/ˈlɑːmə/](https://en.wikipedia.org/wiki/Help:IPA/English "He
1919
#### Markdown as Hierarchical Tree
2020
A properly structured markdown document forms a hierarchical tree where headings define sections containing various nodes (paragraphs, lists, tables, etc.). We parse markdown into an Abstract Syntax Tree (AST) and transform it into a hierarchical structure where sections contain their related content. This enables intelligent chunking that keeps semantically related information together.
2121

22-
<figure>
2322
<img width="1266" height="542" alt="image" src="https://github.com/user-attachments/assets/0a49589c-fae2-4955-b042-5bee770f0344" />
24-
<sup><sub>Hierarchical Markdown Abstract Syntax Tree</sub></sup>
25-
<figcaption>Hierarchical Markdown Abstract Syntax Tree</figcaption>
26-
</figure>
23+
24+
<small>Hierarchical Markdown Abstract Syntax Tree</small>
2725

2826

2927
#### Words as Atomic Unit
3028
Words are the smallest meaningful unit of information for embedding purposes. While tokenizers may split words further, for practical RAG applications, breaking words mid-way creates meaningless chunks. Therefore, words are treated as indivisible atoms that cannot be split.
3129

32-
<figure>
33-
3430
<img width="1263" height="282" alt="image" src="https://github.com/user-attachments/assets/65d00092-7bf4-4a15-94a9-7c80078e59eb" />
35-
<figcaption>Comparison of chunks with size 1 of Chunkdown (left) / LangChain Markdown Splitter (right)</figcaption>
36-
</figure>
31+
32+
<small>Comparison of chunks with size 1 of Chunkdown (left) / LangChain Markdown Splitter (right)</small>
3733

3834
#### Never Break Semantics
3935
Semantic elements like links, images, inline code, and certain formatting elements should ideally always remain intact. Breaking a long link like `[example](https://example.com)` into `[exam` and `ple](https://example.com)` destroys meaning. The splitter preserves these constructs even if it means exceeding the target chunk size.

0 commit comments

Comments
 (0)