Verbatim line breaking issue #1798

Activity

1. Omikhleia commented on May 7, 2023

   

   Omikhleia

   on May 7, 2023

   Last edited by Omikhleia

   Member

   More actions

   Long lones of text (e.g. SHA512 hashes) in verbatim blocks run beyond the text area (and off the end of the page), unlike for text which contains spaces.

   If the question is not just theoretical (a SHA512 in print, really?), what is the expectation on where to break such text?
   An input filter could easily add breakpoints between each character (or every 2 characters, so as to respect hexadecimal?), but this seems somewhat heavy handed ^^
2. Omikhleia

   added

   questionAsk for advice or investigate solutionsAsk for advice or investigate solutions

   on May 7, 2023
3. hlandau commented on May 8, 2023

   

   hlandau

   on May 8, 2023

   Last edited by hlandau

   Author

   More actions

   I'm exploring the viability of sile and other solutions for unattended typesetting... Long lines of text are certainly possible in transcripts of terminal output and it can't be guaranteed there's any spaces in that text. This pipeline of mine would handle arbitrary documents in an automated fashion, so these cases would need to be handled sensibly. Suboptimal linebreaking is OK, so long as it doesn't run off the text area. Would it be possible?
4. Omikhleia commented on May 8, 2023

   

   Omikhleia

   on May 8, 2023

   Last edited by Omikhleia

   Member

   More actions

   Suboptimal linebreaking is OK, so long as it doesn't run off the text area. Would it be possible?

   Yes, that would be doable, and I can even think of several ways to achieve it :)

   • Since SILE's current "verbatim" environment is not really verbatim (it's somewhat ill-named, but so says the documentation, so it's not hidden) one can invoke commands there, and a command with an inputfilter could add breakpoints at "relevant" points. (Hence my question on the initial use case, because I would personally prefer not breaking hexadecimal numbers at random points, so I would split them 2 by 2 there... Still taking the risk of overlong lines, but what is the best solution for a given problem?).
   • It would also be possible in a more general way, for instance with a dedicated typesetter with a different line-breaking strategy (i.e. not pattern/language-based). That would even be easier than the previous point.

   But I'll be real honest, I am not sure how it really relates to "the viability" of a solution (would it be SILE or another). As I use to say to friends (whatever tool they are using), the ultimate aim is before all about getting typography done right in print-like quality, but please "Be Good Typographers"... Breaking lines at random points cannot be good typography, however one looks at it, it can only be a last-chance way of handling bad things with a weird fallback ultimately, IMHO.

   Also, as of "viability" of a typesetting software is concerned, is "verbatim" the right thing to pick and compare? (This is a serious question.)

   Nevertheless, I can however imagine and code such a thing -- One good point about SILE is that nearly everything is replaceable, so everything doable in Lua can make its way into it even in the most obscure place of it (i.e. it's not limited to a few hooks, literally everything is replaceable)... But as for the standard distribution, this shouldn't be the role of the "default" environments such as verbatim (or even autodoc:codeblock, just mentioning it because it's in your test script, though it's kind of a different beast), I think... A 3rd party package can provide a "terminal-transcript" environment, and I bet it would not be too hard to do.¹

   There are no dumb questions, and yours is certainly not². The question however still boils down to what would be the best typographic approach. Do you really think "random breaks" (likely unnoticed) are more satisfying than an overfull line (that one can notice and add a manual \break where seems best depending on real context?) I wouldn't be that sure it's a "viable" answer... It really depends on context. For instance, SILE does try to identify more breakpoints on long URLs formatting (via an inputfilter adding appropriate penalties at places where it seems breaking a URL is indeed acceptable).

   Footnotes

   1. I'd bet on less than 10 lines of code. Let's play, who's taking the bet, and what are the stakes? EDIT: I mean the command logic here. Obviously a package has some lines of "boilerplate" code, let's not consider these. ↩

   2. I would however question the need for keeping of 6 paragraphs of gibberrish "lorem ispsum" that do not relate to the point being raised, but heh :) 🤣 ↩
5. hlandau commented on May 12, 2023

   

   hlandau

   on May 12, 2023

   Author

   More actions

   There are no dumb questions, and yours is certainly not2. The question however still boils down to what would be the best typographic approach. Do you really think "random breaks" (likely unnoticed) are more satisfying than an overfull line (that one can notice and add a manual \break where seems best depending on real context?) I wouldn't be that sure it's a "viable" answer... It really depends on context. For instance, SILE does try to identify more breakpoints on long URLs formatting (via an inputfilter adding appropriate penalties at places where it seems breaking a URL is indeed acceptable).

   It's certainly true that an unattended typesetting flow isn't going to get perfect 'hand-tuned' output for everything one runs through it. But for my use case — which involves generating XHTML, PDF, etc. from a single source XML file — it's essential that if things aren't optimal, at least some basic parameters are met. So with the above example for example, not only do things overrun the text area, they actually run off the page and characters are lost. I will say I find that a very strange default, both in TeX and SILE. I can't really imagine the circumstances in which anyone might want it to handle things like this... ¹

   My basic ambition here is to get something that

   a) will perform adequately for essentially all input
   b) which I can hand-tune later to get something more optimal from a typesetting perspective

   tldr; I'm actually quite passionate about optimal typesetting... but when producing documentation one doesn't always have time to get there initially. Especially when one considers the alternatives are toolchains like FOP, which frankly produces far inferior output, one can't help but feel like this approach of TeX/SILE is making the perfect the enemy of the good... ²

   Admittedly the TeX (and descendant) typesetting world seems so set on the idea of painstaking, hand-tuned typesetting, maybe I'm just going against the grain trying to do this. Yet there's a surprising dearth of other open source PDF tooling that really generates good output. Though it seems like plenty of people have used TeX in automated ways... ³

   Thanks for the thoughtful response & for making SILE. ⁴

   Footnotes

   1. If there were a 'suboptimal emergency line break' function for text like this, it occurs to me it could still emit a warning like those for overfull hboxes so an attending typographer can be alerted to the issue. ↩

   2. A bit of background, if you find it interesting: I actually have an XML toolchain for producing various output formats and my vague plan is to have a "fork" system which allows manual input of different strings for different output types. So if my standard conversion scripts, or the schema, can't reasonably accommodate optimal output for both XHTML and TeX/SILE in some highly bespoke typesetting situation, you could do something like

      <fork>
        <choice type="tex">code to typeset something in the best way in TeX/SILE/etc.</choice>
        <choice type="xhtml">code to typeset something in the best way in XHTML</choice>
      </fork>
      

      The idea is to have decent output all the time and let document authors add and maintain optimisations for specific output formats in the circumstances needed. ↩

   3. Currently I'm sort of a refugee from ConTeXt, which has a nice way to accept XML input directly... theoretically this is ideal. What is less ideal is that many areas of it are wholly undocumented, and I have never seen a codebase more incomprehensible to me than ConTeXt's, meaning when it goes wrong I honestly find it beyond me to fix. If SILE's source itself is simply more comprehensible, that in itself would probably be a massive pro to me. Quite interesting how it's basically all Lua. ↩

   4. TIL GitHub markdown supports footnotes. Even though I'm such a ridiculous user of them on my own site, I had no idea... 🎉 ↩
6. sile-typesetter

   locked and limited conversation to collaborators on May 12, 2023
7. Omikhleia

   converted this issue into a discussion #1803 on May 12, 2023