(libSkShaper) Combined emoji will be drawn as multiple emoji

[Polyisoprene]

Describe the bug
a combined emoji like ✌🏻 will be drawn as two emoji

To Reproduce
Steps to reproduce the behavior:

text = "✌🏻"
surface = skia.Surface(50,50)
paint = skia.Paint(AntiAlias=True, Color=skia.Color(0,0,0,255))
typeface = skia.FontMgr().matchFamilyStyleCharacter( "Noto Sans CJK SC",skia.FontStyle().Normal,["zh", "en"], ord(text))
font = skia.Font(typeface,10)
blob = skia.TextBlob(text, font)
canvas.drawTextBlob(blob, 30,30, paint)

Expected behavior

single emoji appears on the image

Desktop (please complete the following information):

• OS: Ubuntu20.04
• Python: 3.10
• skia-python version: 87.5

Additional context
Add any other context about the problem here.

[HinTak]

This sounds a lot like this upstream issue - google/skia#129

[HinTak]

Changed my mind a bit. I think the correct answer involves a text shaper. Calling a harfbuzz binding of any sort is one answer, but using skshaper is perhaps preferred. Ie. We need to bind
https://api.skia.org/classSkShaper.html

[HinTak]

I tried the emoji modifier with my harfbuzz + freetype + skia code : https://github.com/HinTak/harfbuzz-python-demos/blob/master/hb-view-ot-svg-skia-GL.py and it works as desired, so I think using SkShaper is the correct way to go. My code is fairly harfbuzz and freetype centric, with skia only drawing bitmaps from fonts retrieved via freetype and harfbuzz. On Linux, skia.Font and skia.Fontmgr are essentially Freetype and Fontconfig correspondingly, so the freetype part of my code should be replaceable with indirect calls via skia. harfbuzz corresponds to SkShaper (i.e. SkShaper is implemented via harfbuzz and all SkShaper APIs eventually call harfbuzz on Linux), which isn't hooked up in skia-python yet.

If you are impatient, you can take my code and see what SkShaper calls correspond to those harfbuzz calls, and save me a bit of research time.

[HinTak]

This is fixed in b4331dc as part of #210 .

Mostly you need to do
blob = skia.TextBlob.MakeFromShapedText(text, font) instead of blob = skia.TextBlob(text, font). I thought of overloading the constructor, but it is a bit hard given the constructor and MakeShapedText takes different set of optional arguments, so overloading the two means some arguments get ignored in the other calls. Also the returned result have different bounds, so you need to read the blob.bounds() and adjust offsets. BTW, I found that you made two mistakes in your typeface call (the 2nd and the last arguments are wrong). Anyway, after fixing your mistakes, and make sure a relevant font is picked (check typeface - Noto Color Emoji works, and so does OpenMoji Black), it works.

@kyamagu also as a side-effect, MakeFromShapedText can handle Arabic and most of the middle East and right-to-left languages (Hebrew, Tibetan), and do the right thing for them.

[HinTak]

This is a RFC for the new MakeFromShapedText static method. @Polyisoprene @kyamagu

[HinTak]

You can cut and paste the Persian word for Harfbuzz, use an Arabic font, and have it rendered with MakeFromShapedText, and compare with online images, even if you don't read Arabic.

[Polyisoprene]

This is a RFC for the new MakeFromShapedText static method. @Polyisoprene @kyamagu

Thank you for spending so much time solving this problem, but this method seems to be ineffective。
I installed 'skia_python-120.0b4-cp310 cp310 win_amd64. whl',and i have fixed my incorrect code

import skia

text = "✌🏻"
surface = skia.Surface(350,350)
canvas = surface.getCanvas()
paint = skia.Paint(AntiAlias=True, Color=skia.Color(0,0,0,255))
typeface = skia.FontMgr().makeFromFile("F:\\Python\\test\\NotoColorEmoji.ttf")
font = skia.Font(typeface,30)
blob = skia.TextBlob.MakeFromShapedText(text, font)
canvas.drawTextBlob(blob, 30,30, paint)
img = skia.Image.fromarray(canvas.toarray(colorType=skia.ColorType.kRGBA_8888_ColorType), colorType=skia.ColorType.kRGBA_8888_ColorType)
img.save("1.png")

but I still haven't get the correct image

and after running the code, I get the following output

SkIcuLoader: datafile missing: E:\Python\icudtl.dat.
SkIcuLoader: datafile missing: F:\Envs\normal\lib\site-packages\icudtl.dat.

[HinTak]

@Polyisoprene you wrote you are on Linux:-(. I don't have icudtl in python locations, but I have 3 copies of that file on linux, under chrome/chromium and zoom/cef . The two copies under chrome/chromium are the same. All are over 10MB each. Pretty sure skia-python on Linux doesn't need it, but maybe skia-python on Windows does...

[HinTak]

@Polyisoprene If it wasn't clear, I am suggesting that maybe you copy that file from either Linux chrome or Windows chrome to Windows python where that location is. I think libshaper uses harbuzz on Linux but CoreText on Mac OS. The Window equivalent of Harfbuzz on Windows is uniscribe / usp AFAIK, but I haven't come across uniscribe reference in skia's source code yet, so I have no idea how libshaper works on Windows.

[Polyisoprene]

@Polyisoprene If it wasn't clear, I am suggesting that maybe you copy that file from either Linux chrome or Windows chrome to Windows python where that location is. I think libshaper uses harbuzz on Linux but CoreText on Mac OS. The Window equivalent of Harfbuzz on Windows is uniscribe / usp AFAIK, but I haven't come across uniscribe reference in skia's source code yet, so I have no idea how libshaper works on Windows.

Thanks for your reply. Anyway, at least it works on Linux

[HinTak]

I think it should work on mac os x too. If it doesn't work on Windows, that would be an upstream bug... I'd suggest try copying that file across to see if it works better.

It looks like most of that file is also shipped with nodejs on Linux as icudt73l.dat (your versioning might be different), and as part of libicudata.so.* . How did you build skia-python for Windows? I guess I mean did you do it the usual route (via scripts/Build_Windows.sh) or did you do something different? libicu is built by that script.

[Polyisoprene]

I think it should work on mac os x too. If it doesn't work on Windows, that would be an upstream bug... I'd suggest try copying that file across to see if it works better.

It looks like most of that file is also shipped with nodejs on Linux as icudt73l.dat (your versioning might be different), and as part of libicudata.so.* . How did you build skia-python for Windows? I guess I mean did you do it the usual route (via scripts/Build_Windows.sh) or did you do something different? libicu is built by that script.

Actually, I copied this file to the directory of Python according to the method you mentioned, but it seems not works better.And if copy icudt.dat to the taget directory, i will just get a blank PNG. I didn't build skia-python myself,but download it from Action.

do not copy

copy

[HinTak]

Argh, in that case, I think it is an upstream bug. There is probably a matching icudtl.dat built.

[HinTak]

I have added a test about it in my playground repo/ci : HinTak/skia-m1xx-python@45b25b7 so it should be interesting to know on which platforms it fails.

The test is a variant of this, except I explicitly test that a string with two emojis, without and with modifier should result in exactly two glyphs at exactly the correct glyph ids. Since I am including the font itself, the agreement should be exact on all platforms.

[HinTak]

CI seems to confirm your report: on Windows, two emojis one without modifier, one with, returns a count of 3 etc.

Curious enough, on Linux it only failed on python 12, still counts of two but the 2nd glyph id is not as expected, maybe a iterator lifetime issue. On mac it counts 2 also but the glyph ids are not as expected from python 3.7 .

So linux aarch64 all passes, x86_64 passed for cp37-11, but failed on the 2nd glyph id with cp312(this is curious as the first id test passes, so the glyph ids seem to change during testing, ie poorly written test...). Mac counts are correct but glyph id wrong for cp37. I am re-running the failed tests to see if the linux/mac failures is a iterator lifetime issue (ie just poorly written tests...).

But the Windows failure is an upstream one I think.

[HinTak]

I have refined the tests a bit. At the moment I think the failure with Mac and Windows are different - windows can load Noto Color Emoji but cannot shape; while Mac cannot load Noto Color Emoji, so ended up loading any (typeface is None, and Font getting initialied with None which means pick any font, and is successful). On Mac, it ends up picking a generic font, but can shape and so do two glyphs of the same (i.e. likely a not-color emoji).

When the current CI run finishes (still on-going), I 'll know when Mac actually loads instead of Noto Color Emoji.

The tests probably have some object lifetime issues and needs re-runs sometimes - or it might just be that CI needs about 5 hours to run, and changes should be in batches to push, or they will collide a bit.

Anyway, I think Linux works, Mac should if you use Apple Color Emoji instead of Noto, Windows issue is upstream.

[HinTak]

On Mac, it uses "Zapf Dingbats" when Noto fails to load. And glyph id 15 is about right, according to https://en.m.wikipedia.org/wiki/Zapf_Dingbats , the sign at position 13 (given notdef and null takes up two or three).

[HinTak]

Filed the windows issue upstream : https://issues.skia.org/issues/310510988

[HinTak]

In the tests I added, ( https://github.com/HinTak/skia-m1xx-python/pull/6/files ), initially I used NotoColorEmoji, which definitely works for Linux. Then I realised that CBDT (that particular type of colour fonts - there are 5, actually) definitely does not load on Apple, and load but not work correctly on windows, it is perhaps best to use one specific to the platform.

So here is the verdict: on Apple, the default fonts for emoji (or at least the hand sign) is Zapf Dingbats (not coloured), but you can explicitly ask for Apple Color Emoji, and both Zapf Dingbats and Apple Color Emoji works with skin color modifiers against the Apple text engine. On Windows, the Microsoft shipped colour Emoji font is Segoe UI Emoji. I actually got the exact version - 1.29 - on my local hard drive as github CI. Seguiemj 1.29 works with harfbuzz to shape, but not against libskshaper on windows. So it is definitely a bug in windows libskshaper - or lack of usage of microsoft uniscribe (don't know if it actually use harfbuzz on Windows - it shouldn't, as uniscribe is the equivalent).

So I am tempted to close this when m120 is out, and leave the windows issue for upstream.

[HinTak]

#210 includes the libshaper addition (which unfortunately does not work on windows, filed at the URL above) .

#210 also includes skia.FontMgr.New_Custom_Empty().makeFromFile(p, N) on Windows and Mac for loading fonts via Freetype. This does not solve the windows shaping problems, but it lets you load NotoColorEmoji on Apple (which Apple's own CoreText manager doesn't let you do) and also OT-SVG color fonts on non-linux, and some font formats Apple or Microsoft don't support.

If you really want this to work on Windows, bug the Google folks at https://issues.skia.org/issues/310510988

I looked at the skia code, it only interacts with uniscribe (window's harfbuzz equivalent) in exactly one place: in a routine called nonBmpCharToGlyph . It might be possible to do something on skia-python side to provide access to that... but I think libSkShaper needs to be fixed for windows, probably by copying come code internally from that.

Closing for now.

[HinTak]

Apparently the canonical place to get icudtl.dat is from ihttps://github.com/unicode-org/icu/releases , one of the data-bin-{l,b}.zip files. L/B for little / big endian, and it is versioned.

[HinTak]

I have figured out where to get a proper icudtl.dat, as well as where to copy it to:
#268

Don't know if it makes any difference, @Polyisoprene , but see if you care to try yourself?

[HinTak]

I think it should work on mac os x too. If it doesn't work on Windows, that would be an upstream bug... I'd suggest try copying that file across to see if it works better.
It looks like most of that file is also shipped with nodejs on Linux as icudt73l.dat (your versioning might be different), and as part of libicudata.so.* . How did you build skia-python for Windows? I guess I mean did you do it the usual route (via scripts/Build_Windows.sh) or did you do something different? libicu is built by that script.

Actually, I copied this file to the directory of Python according to the method you mentioned, but it seems not works better.And if copy icudt.dat to the taget directory, i will just get a blank PNG. I didn't build skia-python myself,but download it from Action.

do not copy

copy

I seems to have just reproduced this breakage - although copying a icudtl into some place allow skparagraph to load properly, it seems to break
MakeFromShapedText :

#270 (comment)

This is all Windows specific.

[HinTak]

Just to cover old ground - upto m125 you need v69, while from m126 onwards you need v74:
2b6ab54

This part of code is somewhat sensitive to the icudtl version if you have to copy one (on windows)

[HinTak]

Revisit the windows issue, now we know we needs to be specific about icudtl version.

[HinTak]

@Polyisoprene testing shows that version-matching icudtl.dat is the issue - on windows, you needs exactly v74 .

[HinTak]

@Polyisoprene #279 includes the fix for windows, and also a variant of your code above to test within CI on all platforms (counting how many glyphs we get, and the the expected number, not more), to make sure that it works. Give that a try on windows, or we'll merge it for the next release anyway.

[BalconyJH]

Thank you for the work in #279. It's working great on Windows now, and all combined emoji render correctly as expected. Appreciate the effort to make this run smoothly across platforms.

[HinTak]

#279 merged. M132 going out tomorrow.