Q: How to spell check languages that do not have word breaks?

[lysari]

I was working on Khmer dictionary. Everything works find when checks word by word. But I have a problem when I test it as a paragraph all that goes wrong. How do I fix that?
words : ព័ត៌មាន, អំពី, សមាជិក, បក្ស - working
paragraph : ព័ត៌មានអំពីសមាជិកបក្ស - not working

[Jason3S]

@lysari,

That is exactly the challenge with languages like Khmer and Thai. See:

• Unicode Text Segmentation
• Thai line breaking: how to break Thai text effectively - Stack Overflow

It is possible to tell the spell checker to "compound" words, but that will only work with short sentences and will mark the entire sentence as wrong instead of just the incorrect word.

If this is a needed feature, consider funding its development.

[lysari]

@Jason3S Thank you for reply. If the feature is something others in the development community might be interested in, consider contributing to an existing open-source project, or working together with the community to develop the feature.

[heipiao233]

Also, Chinese, Japanese and Korean have no word breaks.

[JounQin]

@Jason3S I think we can use https://github.com/napi-rs/node-rs/tree/main/packages/jieba for Chinese at least?

[Jason3S]

@JounQin,

Thank you.

My plan is to use Intl.Segmenter for languages like Thai. I didn't realize that Intl.Segmenter was available until recently.

The challenge is to detect when it is needed and apply the right locale.

[JounQin]

@Jason3S It seems really promising:

const str = "吾輩は猫である。名前はたぬき。";
const segmenterJa = new Intl.Segmenter("ja-JP", { granularity: "word" });

const segments = segmenterJa.segment(str);
console.table(Array.from(segments));
// [{segment: '吾輩', index: 0, input: '吾輩は猫である。名前はたぬき。', isWordLike: true},
// etc.
// ]

The challenge is to detect when it is needed and apply the right locale.

See https://github.com/greyblake/whatlang-rs, and I'm going to make a @napi-rs wrapper for it at https://github.com/unrs/whatlang. Or https://github.com/pemistahl/lingua-rs with https://github.com/unrs/lingua.

[Jason3S]

@JounQin,

I'll take a look.

My initial plan was to define segmentation info in each dictionary that needs it.

• Define the locale to use with the segmenter (this is the locale of the dictionary).
• Define the Basic Multilingual Plane blocks to use.
• When checking a block of text, see if it is contained in the BMP block range, if so, segment it by words.

[JounQin]

Language detection is hard, BMP is not good enough for such task, for example, CJK shares same BMP AFAIK.