Bangla Mojibake

[hbarnard]

I have a file that appears to contain the correct bangladeshi characters.

[[2025-02-17 18:31:44.43371] [137432] [trace] [qqzttcQY_zTh] Routing to a callback
$VAR1 = 'bn,বাথরুম কোথায়,Where\'s the bathroom?,বাথরুম লোকেশন দেওয়া নেই।,The bathroom is missing,Neutral
';
$VAR2 = 'bn,বাথরুম কোথায়,Where\'s the bathroom?,বাথরুম লোকেশন দেওয়া নেই।,The bathroom is missing,Positive
';
$VAR3 = 'bn,বাথরুম কোথায়,Where\'s the bathroom?,বাথরুম লোকেশন দেওয়া নেই।,The bathroom is missing,Negative
';
field is বাথরুম কোথায় 
field is বাথরুম কোথায় 
field is বাথরুম কোথায় ]

But whatever I try for rendering html, txt, removal of all HTML they come out as Mojibake, this was a plain text template:

bn	 
     বাথর�ম ��থায়
bn	 
     বাথর�ম ��থায়
bn	 
     বাথর�ম ��থায়

I've switched on utf8 everywhere and probably in some stupid places too. I also have a bangla font supplied from a CDN.
Hoping there's something I've missed here. Want to do this with a few other exotic languages too, mainly East Asian and maybe simplified Chinese.

TIA Hugh Barnard

[christopherraa]

There are so many factors at play here that you really need to provide a complete example that displays the behaviour you are experiencing. Using a Mojolicious::Lite application (single file) is perfect for this as it can be easily added as an attachment or even pasted into a code block. That way other people can both read the code and try it for themselves.

[hbarnard]

Thanks for being so quick! I'm ashamed of this application (!), it's a terrible mess at the moment. But it is the one I'm working on.
nifis.txt

[hbarnard]

Also here's the contents of sms.txt

bn,বাথরুম কোথায়,Where's the bathroom?,বাথরুম লোকেশন দেওয়া নেই।,The bathroom is missing,Neutral
bn,বাথরুম কোথায়,Where's the bathroom?,বাথরুম লোকেশন দেওয়া নেই।,The bathroom is missing,Positive
bn,বাথরুম কোথায়,Where's the bathroom?,বাথরুম লোকেশন দেওয়া নেই।,The bathroom is missing,Negative

[kraih]

Make sure everything is really UTF-8, every single file.

[guest20]

I think you want Encode::decode('utf-8' ...) instead of utf8::upgrade.

ideally you'd have your calls to encode/decode in the same place as the code that reads/writes the files, just so you can be sure the right transcoding is happening, and that it's happening every time.

Also, Mojo::File::slurp takes an encoding parameter which is handy for one-offs.

This is a very helpful youtube video about utf-8:
https://www.youtube.com/watch?v=TmTeXcEixEg

[s1037989]

That video was amazing. Thank you for sharing! And thank you @rjbs!

[rjbs]

Glad you liked it!

[guest20]

I like the way you show your working - very helpful for more "suborn" learners

[hbarnard]

Hi folks, Thank you all for your help. It looks as though this:
read_file('/var/www/html/sms.txt', { binmode => ':utf8' })
has worked its magic. I'll strip out some of the panic code now and see whether any of it was, in fact, necessary.

The application, when tidier is for: https://newham-citizen-science-fair.uk/ Newham has a big South Asian population. if you're in London on April 12th, registration for free tickets (we need to manage capacity) on the website.

[Grinnz]

See https://blogs.perl.org/users/leon_timmermans/2015/08/fileslurp-is-broken-and-wrong.html - consider File::Slurper read_text which would do that automatically, or with Mojo::File path($path)->slurp('UTF-8').

[hbarnard]

Thanks, didn't know that. Funnily enough (or not):

#my @r = $answer_file->slurp('UTF-8') ;
my @r = read_file('/var/www/html/sms.txt', { binmode => ':utf8' }) ;

seem to behave differently, File::Slurper (and this may be something else in my code) doesn't deliver the whole array to the template. I haven't really time to go into this at the moment, so I'm keeping read_file in this.

[guest20]

It's possible that one is giving you individual lines in a list and the other is giving you the whole file as single string. #contextMatters