Offline image

[jpolitz]

Progress on using node-canvas (https://www.npmjs.com/package/canvas) to make images work offline and in CLI mode (for, e.g., an autograder).

This boils down to:

• Finding uses of document.createElement("canvas") and polyfilling with nodeCanvas.createCanvas
• Finding places where canvas.style is used and working around it conditionally (offline canvases do not have styles; they aren't full DOM nodes)
• Making sure we only depend on how pyret-lang loads rather than CPO as well

This is surprisingly straightforward and many tests pass already, including ones that render to canvases via BaseImage.prototype.equals (e.g. cropping a rectangle to a square and checking equality).

Other TODOs include:

• Seeing if we can do image-url
• Making sure image-file is properly hooked up to fs and the right combo of browser/CLI libraries
• Probably adding some save-image function to actually view the images

[jpolitz]

(This was meant to be a draft, tests still fail)

[jpolitz]

The remaining issue is in text images, specifically with getTextDimensions in image-lib.js (https://github.com/brownplt/code.pyret.org/blob/horizon/src/web/js/trove/image-lib.js#L1411).

It has a comment next to it that there should be
https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/measureText,
but that was buggy at the time. The comment is at least 7 years old though, and
the referenced SO post has a comment on the top answer that MDN claims broad support for this:

https://stackoverflow.com/questions/1134586/how-can-you-find-the-height-of-text-on-an-html-canvas/9847841#9847841

That's next to look into.

[dbp]

The remaining issue is in text images, specifically with getTextDimensions in image-lib.js (https://github.com/brownplt/code.pyret.org/blob/horizon/src/web/js/trove/image-lib.js#L1411).

It has a comment next to it that there should be https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/measureText, but that was buggy at the time. The comment is at least 7 years old though, and the referenced SO post has a comment on the top answer that MDN claims broad support for this:

https://stackoverflow.com/questions/1134586/how-can-you-find-the-height-of-text-on-an-html-canvas/9847841#9847841

That's next to look into.

Even if the browsers are still unreliable (though it seems like maybe not now, as you said), it seems like node-canvas supports this, so could use it offline?

https://github.com/Automattic/node-canvas/wiki/Compatibility-Status#drawing-text

Either way, this is awesome! :)

[jpolitz]

Reporting back....

Annoyingly, the unsupported features (textBoundingBoxAscent and textBoundingBoxDescent) on node-canvas are the ones that exactly match the current behavior on CPO. Unfortunately for this PR, I think the current behavior is good behavior as I've revisited it. Need to think a little.

(The textBoundingBox fields are present here because it's in a browser, absent when run from the CLI with node-canvas)

import color as C
include from C: indigo end
indigo-text = text-font("Goodbye", 48, indigo, "Helvetica", "modern", "normal", "normal", false)
image-height(indigo-text)
image-baseline(indigo-text)

On CPO, this reports a height of 55 and a baseline of 44 (11 of the pixels are below the text baseline for the trailing y). This actually leaves padding around the words for this example when rendered; presumably because there are taller characters if accents, etc are included. This is the difference between the bounding box of the actual text vs the bounding box of what the font could need.

The supported fields of node-canvas, actualBoundingBoxAscent and actualBoundingBoxDescent change the height as the string changes, which is an annoying property at best. For example, if you change the string from Goodbye to Goodbe without the y which trails below, the height changes from 46 to 36.

I've also been getting some issues with the height calculated from actualBoundingBox not being quite big enough to render the text (maybe messing in .render would help, but still).

Conclusions:

• We could make the browser and CLI behave the same as one another, but differently from current CPO (by using actualBoundingBox instead of effectively fontBoundingBox)
• We could allow the browser and CLI to differ, keeping the browser behaving as-is (and maybe simplifying the implementation). Size calculations and pinhole overlays of text images might be different in an offline autograder than in-browser
• What CPO/the image library does is calculate the height needed for the font, which is arguably pretty good behavior.

[jpolitz]

This issue has a nice picture, duplicated here:

Automattic/node-canvas#2395

[jpolitz]

As a point of comparison, 2htdp/image appears to return the height-by-font.

> (image-height (text/font "Goodbye" 48 "indigo" "Helvetica" "modern" "normal" "normal" true))
48
> (image-height (text/font "Goodbe" 48 "indigo" "Helvetica" "modern" "normal" "normal" true))
48

[jpolitz]

Another note that using these calculations in image-lib avoids cutting off the top or bottom in a few small tests:

      this.height      = Math.ceil(metrics.actualBoundingBoxAscent) + Math.ceil(metrics.actualBoundingBoxDescent);
      this.alphaBaseline = Math.ceil(metrics.actualBoundingBoxAscent);

With Math.ceil:

Without:

[jpolitz]

OK, I thought about this some more. I think I'm pretty happy with the current pushed version for this PR (pending passing on CPO and some other details).

One of the things I said earlier isn't true – the heights in both 2htdp/image and in CPO-of-today can change depending on the contents of the string. The simplest thing is to use other alphabets (an analogous example shows a difference in the htdp image library)

# in CPO of today
hello1 = text("Hello", 48, "black")
hello2 = text("你好", 48, "black")

image-height(hello1) # 55
image-height(hello2) # 59

So as a general property, it's not the case that the height is fixed w.r.t. the font + size, so it's less weird to have things change height based on string contents.

Also, the baseline calculations are solid, and used well as the pinhole points. So, for example, monospace Good and good will overlay the ood pixel-perfectly (the screenshot below is on the new version using actualBoundingBox). This is the property that matters most for things looking right: if text changes in an animation or between otherwise-similar renderings, we don't want the text to feel like it “moved”.

The only cases of backwards-incompatibility this would introduce is in programs that care specifically about the pixel-height of text (which is already not quite consistent across browsers, OSes, and font availability). The position within existing overlays, etc, would look “good enough”.

[jpolitz]

I think this is close to ready to merge. It has the corresponding CPO changes in brownplt/code.pyret.org#585, which ends up having some related bits, because (via the VScode extension) the CPO code has to provide writeFile.