training on large data set is failing

[lihka1]

when i am trying to train on large data set text (around 500MB file) its failing with error

[info] loading text
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted

Any ideas please?

[bakwc]

Try to use machine with more amount of memory. Or may be a large swap file (can take a really long time).
Or you can purchase a "Pro" version - there is a memory optimization that reduce memory consumption while training.

[lihka1]

thanks.

Also one more question,

i trained on a smaller data.

when i tried to fix for a sample sentence like '"Otb.r Compreh�nsiYe Incone" model outputs "Otbr ComprehensIve Income"

is there any way to get Otb.r corrected as Other. I can see in the training data there are many bigrams, tigrams of other comprehensive income

[mirfan899]

@lihka1 it depends on the alphabet you used for training the model. Here you have . but in alphabet its not there so it will not correct this word.