Replace the Markdown parser and HTML renderer Misaka with Mistune #1040

[ritzmann]

Misaka has been abandoned several years ago. Misaka depends on Hoedown,
a Markdown parser implemented in C, which limits platform support.

Mistune provides a similar API and features, is a pure Python implementation and is actively
maintained.

The goal is to maintain backwards compatibility with the existing markdown options where
feasible and render HTML that closely resembles the implementation with Misaka.

Syntax differences

HTML elements

Misaka allows HTML expressions like <em>Hi</em> in Markdown. This expression
will be rendered to HTML as is: <em>Hi</em>. Mistune takes a stricter approach
and replaces any HTML element with rendered text like this: <em>Hi</em>.

Strong and emphasis

Misaka interprets asterisks like this:

1. **word**: Strong word
2. *word*: Emphasized word
3. **word1***word2*: Strong word1 and emphasized word2

Mistune is the same on the first two points but does not interpret asterisks inside words:

3. **word1***word2*: Renders as **word1***word2*

If you want to have the same rendered result with Mistune, use underscores for emphasis:

3. **word1**_word2_: Strong word1 and emphasized word2

Fenced code

In Misaka, fenced code can be marked by enclosing the code block in three backticks
``` or by putting four space characters in front of each code line. Mistune
only understands three backticks.

Plugins

Fenced-code

Mistune always renders fenced code. The behavior is not configurable and no
fenced-code plugin exists for Mistune.

Superscript

This plugin exists in both engines with the same name superscript. The
Markdown syntax is different however.

Misaka: ^(superscripted_text)

Mistune: ^superscripted_text^

[ritzmann]

TODOs:

• Reintroduce flags
• Log obsolete options/extensions
• Update/adapt failing tests
• Add more rendering tests
• Find where Mistune is rendering Markdown differently and tweak to stay close to Misaka
• Try to support options that are not supported out of the box such as highlight
• Look into support Python smartypants
• Update documentation

[jelmer]

Thanks! This looks reasonable to me overall. I think the main doubts are around automatically attempting to rewrite extensions.

[ritzmann]

Thanks! This looks reasonable to me overall. I think the main doubts are around automatically attempting to rewrite extensions.

In general, I see three options:

1. Forget about misaka extensions, don't try to be backwards compatible, document all differences and how to upgrade.
2. Try to be as backwards compatible as feasible. Map misaka extensions and flags to corresponding mistune settings and where mistune is incompatible, customize mistune to resemble misaka.
3. Find some middle-ground where we don't break existing configurations but acknowledge that mistune has a different feature set and renders elements differently.

Option 1 is likely to make people unhappy that don't read release notes carefully and that is probably a significant number of users. I was originally thinking to go for option 2 but it is likely to create additional maintenance overhead in the long run and makes it more difficult to use mistune features. I prefer option 3 now but I would be happy to hear arguments for the other options.

I will be traveling the next ten days and won't have time to do any substantial work until the second week of January.

[jelmer]

Thanks, I think that's

Thanks! This looks reasonable to me overall. I think the main doubts are around automatically attempting to rewrite extensions.

In general, I see three options:

1. Forget about misaka extensions, don't try to be backwards compatible, document all differences and how to upgrade.

2. Try to be as backwards compatible as feasible. Map misaka extensions and flags to corresponding mistune settings and where mistune is incompatible, customize mistune to resemble misaka.

3. Find some middle-ground where we don't break existing configurations but acknowledge that mistune has a different feature set and renders elements differently.

Option 1 is likely to make people unhappy that don't read release notes carefully and that is probably a significant number of users. I was originally thinking to go for option 2 but it is likely to create additional maintenance overhead in the long run and makes it more difficult to use mistune features. I prefer option 3 now but I would be happy to hear arguments for the other options.

Thanks, I think that's a good summary.

From my perspective, (2) isn't really an option. It creates more maintenance overhead (and confusion!) in the long term, and doesn't provide us with a way to ever get rid of that compatibility code. Even if we try, we won't completely match the behaviour of misaka so some people will experience breakage.

(1) I believe could be okay if we manage the transition well. For example, we could consider bumping the major version of isso to signify major changes.

I'm curious what you think (3) would look like in practice. I don't think you could prevent breaking existing installations without letting people select the backend (which from the project's perspective doesn't address #1040 even if it helps for *BSD users, and in fact makes maintenance harder in the long run).

[ritzmann]

@jelmer, I am currently looking into what it would take to make a clean break and I am wondering what to do about the server config:

[markup]
options = strikethrough, superscript, autolink, fenced-code
flags =
allowed-elements =
allowed-attributes =

The above has the issue that options really means Misaka extensions, not some generic set of markup options. The same goes for flags. Would it make sense to use a nested section in the configuration file that takes Mistune-specific settings like this:

[markup]
allowed-elements =
allowed-attributes =

[markup.mistune]
plugins = strikethrough, superscript, subscript
arguments = escape, hard_wrap

[jelmer]

That looks reasonable. It might make sense to support both mistune and misaka for a while (with sections named as you're suggesting), then warn people that misaka is going away and then eventually remove it.

[ritzmann]

Why not, something like this?

[markup]
renderer = misaka|mistune
allowed-elements =
allowed-attributes =

[markup.misaka]
options = strikethrough, superscript, autolink, fenced-code
flags =

[markup.mistune]
plugins = strikethrough, superscript, subscript
arguments = escape, hard_wrap

The renderer setting would be optional and default to mistune. If you have an old config with options/flags set in the [markup] section, those settings would be ignored and a warning logged.

[ritzmann]

DONE:

• Implemented code to separate Misaka from Mistune configuration.
• Instantiate Misaka or Mistune renderer depending on configuration.

TODO:

• Undo removal of Misaka pieces
• Add unit tests for Mistune rendering
• More logging
• Find where Mistune is rendering Markdown differently and document
• Update documentation

I wrote earlier:

Look into support Python smartypants

I am confused where that came from. Doesn't seem to make much sense or at least not as part of this change.

[jelmer]

Looks good

[ritzmann]

Code is practically done, all tests under tests are passing and unit tests for Mistune are an adapted copy of the unit tests for Misaka. Still more logging to add, more Mistune unit tests and more documentation.

[jelmer]

This looks great. Some leftover minor comments, but perhaps this is ready to be switched from Draft to ready for review?

There are three situations I think are important to support well are:

• existing config, no renderer specified - should stay working both warn the user that no renderer is specified and that misaka is used an deprecated
• config that explicitly specifies misaka: should warn that misaka is deprecated
• config that explicitly specifies mistune: should just work :-)

[ritzmann]

This looks great. Some leftover minor comments, but perhaps this is ready to be switched from Draft to ready for review?

I still need to resolve a few comments and need to check what Mistune plugins to enable by default to get closer to Misaka, and the documentation still needs a bit of fine-tuning. Also haven't actually run this for real yet. Would you prefer to get this merged and then create a second PR to finish the work?

[jelmer]

This looks great. Some leftover minor comments, but perhaps this is ready to be switched from Draft to ready for review?

I still need to resolve a few comments and need to check what Mistune plugins to enable by default to get closer to Misaka, and the documentation still needs a bit of fine-tuning. Also haven't actually run this for real yet. Would you prefer to get this merged and then create a second PR to finish the work?

We try to keep master in a working state so we can do a release any time, so I'd prefer to wait until this is properly ready to merge it. I do feel that it's beyond the state of a Draft at this point though, since you're just polishing up the last few things - but whether the Draft bit is set is honestly a minor point :)

[jelmer]

Hey @ritzmann , anything I can do to help get this to a mergeable state? It would be great to get it in.

[ritzmann]

No worries, I had hoped to get this wrapped up before my next holiday trip but too many other things to do. Now I am back from that trip, so hoping to drive this to a conclusion next week.

[ritzmann]

One minor item to be resolved still:
#1079 (comment)

And I still need to do more manual testing but otherwise this is ready from my point of view.

[ritzmann]

One minor item to be resolved still:
#1079 (comment)

Done