Extensions cannot be used with custom writers

[asmaier]

Explain the problem.
I created a custom writer in Lua for kramdoc based on pandoc markdown:

function Writer (doc, opts)
    local filter = {
      Math = function(elem)

        local math = elem

        if elem.mathtype == 'DisplayMath' then
            local delimited = '\n$$' .. elem.text ..'$$\n\n'
            math = pandoc.RawInline('markdown', delimited)
        end

        if elem.mathtype == 'InlineMath' then
            local delimited = '$$' .. elem.text ..'$$'
            math = pandoc.RawInline('markdown', delimited)
        end 
        
        return math
      end           
    }
    return pandoc.write(doc:walk(filter), 'markdown', opts)
  end

When using it like this it works:

pandoc -t kramdown.lua -o vector.md vector.tex

However when I try to activate or deactivate extensions by adding or substracting the extensions name to my custom writer (as described at https://pandoc.org/MANUAL.html#extensions) it doesn't work, e.g. when trying to deactivate the extension for smart quotes:

pandoc -t kramdown.lua-smart -o vector.md vector.tex
Error running Lua:
cannot open kramdown.lua-smart: No such file or directory

Obviously activating/deactivating extensions for custom writers doesn't work like for the inbuilt supported formats (I also tried the combinations kramdown-smart.lua or kramdown-smart). Is this a bug/missing feature or is there another way to activate/deactivate extensions when using custom writers?

Pandoc version?
Pandoc 2.18 on Mac OS 10.15.7

[tarleb]

That's an interesting problem. Let's start with the workaround first: an easy way would be to set a variable, e.g.:

pandoc -t kramdown.lua -V extensions=-smart ...

and then change the filter like that:

local exts = opts.variables.extensions or ''
return pandoc.write(doc:walk(filter), 'markdown' .. exts, opts)

---

I like the idea of specifying extensions for custom writers in the way you propose. There are technical details that must be answered, like

• Should we allow arbitrary extensions on custom readers, or just those that pandoc knows about?
• How would that information get passed to the Writer function?

We should probably do the same for readers.

[jgm]

How would that information get passed to the Writer function?

We already have an extensions field in PANDOC_WRITER_OPTIONS and PANDOC_READER_OPTIONS.

Should we allow arbitrary extensions on custom readers, or just those that pandoc knows about?

Since these options variables are derived from structured Haskell WriterOptions and ReaderOptions, where the extensions fields are lists of Extensions, we'd be limited to the ones that pandoc knows about, I think. At least in the most straightforward way to support this.

The remaining work would consist in recognizing that kramdown.lua-smart is a lua filter and parsing out the extensions.

[tarleb]

The problem I see with using the extensions field is that we don't know the default extensions that should be enabled. A call to getDefaultExtensions "kramdown.lua" would return the auto_identifiers extension, so kramdown.lua-smart would be the same as kramdown.lua. We could allow syntax like kramdown.lua:base_format+extensions, but that feels wrong. Or maybe we could allow writers to have a global field writer_extensions and use that?

writer_extensions = {
  smart = true,
  auto_identifiers = false,
}

function Writer (doc, opts)
  -- ...
end

[jgm]

Another possibility, I guess, is to just overwrite the extensions field in opts with whatever is provided on the command line. (Default would thus be no extensions.)
People could create whatever extensions they like, since this would all operate independently of pandoc's own extensions mechanism.

[tarleb]

I started to play around with this, and it leads to interesting problems. For example, hslua currently doesn't allow code like

(readCustom', extensions) <- runLua $ do
  runLuaFile
  extensions <- getExtensions
  reader <- makeReader
  return (reader, extensions)

-- later...

readCustom' opts sources -- this will fail, as the Lua state has already been closed.

I have an idea how to make it work, but it will require changes in hslua (relevant Q&A).

[asmaier]

Just a question from a users point of view: Wouldn't it be possible instead of attaching the extension name to the format name to simply add another commandline option allowing to activate or deactivate extensions, like

pandoc -t kramdown.lua -XX:+smart -XX:-citations -o vector.md vector.tex

This style is borrowed from the Java world, see e.g. https://picocli.info/#_negatable_options . Another way would be to use

pandoc -t kramdown.lua --smart --no-citations -o vector.md vector.tex  

A third way would simply be adding two new option --activate extension1,extension2 and --deactivate extension3,extension4,extension5 and then give a list of extensions like:

pandoc -t kramdown.lua --activate smart --deactivate citations -o vector.md vector.tex            

[tarleb]

While it would be possible, the problem of getting that info to the writer is orthogonal and would remain.

I've personally grown attached to the current way with which we specify extensions on the command line, but I can also see the advantages of your suggestion. You could raise the idea on the pandoc-discuss mailing list and see what others say.

[jgm]

@tarleb I wonder what you think of this possible change? (Note: a warning will be raised for one of the Lua modules; I wasn't quite sure how to handle it but you would be.)

diff --git a/src/Text/Pandoc/App.hs b/src/Text/Pandoc/App.hs
index 2bddcf1a3..7ba318bba 100644
--- a/src/Text/Pandoc/App.hs
+++ b/src/Text/Pandoc/App.hs
@@ -62,6 +62,7 @@ import Text.Collate.Lang (Lang (..), parseLang)
 import Text.Pandoc.Filter (Filter (JSONFilter, LuaFilter), Environment (..),
                            applyFilters)
 import Text.Pandoc.PDF (makePDF)
+import Text.Pandoc.Writers.Custom (writeCustom)
 import Text.Pandoc.SelfContained (makeSelfContained)
 import Text.Pandoc.Shared (eastAsianLineBreakFilter, stripEmptyParagraphs,
          headerShift, isURI, tabFilter, uriPathToPath, filterIpynbOutput,
@@ -336,6 +337,7 @@ convertWithOpts opts = do
 
     output <- case writer of
       ByteStringWriter f -> BinaryOutput <$> f writerOptions doc
+      CustomWriter fp -> TextOutput <$> writeCustom fp writerOptions doc
       TextWriter f -> case outputPdfProgram outputSettings of
         Just pdfProg -> do
                 res <- makePDF pdfProg (optPdfEngineOpts opts) f
diff --git a/src/Text/Pandoc/App/OutputSettings.hs b/src/Text/Pandoc/App/OutputSettings.hs
index 988f2def5..4eab1e236 100644
--- a/src/Text/Pandoc/App/OutputSettings.hs
+++ b/src/Text/Pandoc/App/OutputSettings.hs
@@ -39,7 +39,6 @@ import Text.Pandoc.App.FormatHeuristics (formatFromFilePaths)
 import Text.Pandoc.App.Opt (Opt (..))
 import Text.Pandoc.App.CommandLineOptions (engines, lookupHighlightStyle,
                                           setVariable)
-import Text.Pandoc.Writers.Custom (writeCustom)
 import qualified Text.Pandoc.UTF8 as UTF8
 
 readUtf8File :: PandocMonad m => FilePath -> m T.Text
@@ -103,19 +102,15 @@ optToOutputSettings opts = do
                  TextWriter w -> TextWriter $ \o d -> sandbox files (w o d)
                  ByteStringWriter w
                             -> ByteStringWriter $ \o d -> sandbox files (w o d)
+                 CustomWriter fp -> CustomWriter fp
 
 
   (writer, writerExts) <-
-            if ".lua" `T.isSuffixOf` format
-               then return (TextWriter
-                       (\o d -> writeCustom (T.unpack writerName) o d), mempty)
-               else if optSandbox opts
-                       then
-                         case runPure (getWriter writerName) of
-                           Left e -> throwError e
-                           Right (w, wexts) ->
-                                  return (makeSandboxed w, wexts)
-                       else getWriter (T.toLower writerName)
+    if optSandbox opts
+       then case runPure $ getWriter (T.toLower writerName) of
+              Left e -> throwError e
+              Right (w, exts) -> return (makeSandboxed w, exts)
+       else getWriter (T.toLower writerName)
 
   let standalone = optStandalone opts || not (isTextFormat format) || pdfOutput
 
diff --git a/src/Text/Pandoc/Extensions.hs b/src/Text/Pandoc/Extensions.hs
index 1579dfaeb..3e38f466e 100644
--- a/src/Text/Pandoc/Extensions.hs
+++ b/src/Text/Pandoc/Extensions.hs
@@ -609,13 +609,15 @@ getAllExtensions f = universalExtensions <> getAll f
 -- a set of extensions to enable, and a set of extensions to disable.
 parseFormatSpec :: T.Text
                 -> Either ParseError (T.Text, [Extension], [Extension])
-parseFormatSpec = parse formatSpec ""
-  where formatSpec = do
+parseFormatSpec t = parse formatSpec "" t
+  where isCustom = ".lua" `T.isInfixOf` t
+        formatSpec = do
           name <- formatName
           (extsToEnable, extsToDisable) <- foldl' (flip ($)) ([],[]) <$>
                                              many extMod
           return (T.pack name, reverse extsToEnable, reverse extsToDisable)
-        formatName = many1 $ noneOf "-+"
+        formatName | isCustom = (<> ".lua") <$> manyTill anyChar (try $ string ".lua")
+                   | otherwise = many1 $ noneOf "-+"
         extMod = do
           polarity <- oneOf "-+"
           name <- many $ noneOf "-+"
diff --git a/src/Text/Pandoc/Writers.hs b/src/Text/Pandoc/Writers.hs
index f4e1f9040..c44453194 100644
--- a/src/Text/Pandoc/Writers.hs
+++ b/src/Text/Pandoc/Writers.hs
@@ -125,6 +125,7 @@ import Text.Pandoc.Writers.ZimWiki
 
 data Writer m = TextWriter (WriterOptions -> Pandoc -> m Text)
               | ByteStringWriter (WriterOptions -> Pandoc -> m BL.ByteString)
+              | CustomWriter FilePath
 
 -- | Association list of formats and writers.
 writers :: PandocMonad m => [ (Text, Writer m) ]
@@ -199,20 +200,26 @@ getWriter s =
         Left e  -> throwError $ PandocAppError $
                     "Error parsing writer format " <> tshow s <> ": " <> tshow e
         Right (writerName, extsToEnable, extsToDisable) ->
-           case lookup writerName writers of
-                   Nothing  -> throwError $
-                                 PandocUnknownWriterError writerName
+           case lookup (T.toLower writerName) writers of
+                   Nothing  -- check for custom writer
+                     | ".lua" `T.isSuffixOf` writerName ->
+                       return
+                         (CustomWriter (T.unpack writerName),
+                          foldr disableExtension
+                            (foldr enableExtension mempty extsToEnable) extsToDisable)
+                     | otherwise -> throwError $ PandocUnknownWriterError writerName
                    Just  w  -> do
-                     let allExts = getAllExtensions writerName
+                     let writerName' = T.toLower writerName
+                     let allExts = getAllExtensions writerName'
                      let exts = foldr disableExtension
                            (foldr enableExtension
-                             (getDefaultExtensions writerName)
+                             (getDefaultExtensions writerName')
                                    extsToEnable) extsToDisable
                      mapM_ (\ext ->
                               unless (extensionEnabled ext allExts) $
                                 throwError $
                                    PandocUnsupportedExtensionError
-                                   (T.drop 4 $ T.pack $ show ext) writerName)
+                                   (T.drop 4 $ T.pack $ show ext) writerName')
                           (extsToEnable ++ extsToDisable)
                      return (w, exts)
 

If you like this idea, we could do the same for readers.

[jgm]

The changes to parseFormatSpec would allow us to use it in getDefaultTemplate, which would help solve #8137.