encode-json with prettyPrinting="true" does break

[dr0i]

From #159 (comment):

Using | encode-json(prettyPrinting="false") (not true) i ran the workflow on the same endpoint successfully! The result file is onyl 172 MB (it has been deleted since 1379edf ist deployed on test)

@dr0i @TobiasNx something went wrong during transformation

[dr0i]

The cause could reside in:

1. metafacture-core
2. metafacture-playground

but most probabyl in 2. along

metafacture-playground/src/clj/metafacture_playground/process.clj

Line 33 in a2ba292

(clj-str/replace #"\|(\s*|\n*)print(\s*|\n*);" output))]

.

[TobiasNx]

Tested it locally. It seems that prettyPrinting true does not create the breaking, but it has a bug since it adds tabs and linebreaks metafacture/metafacture-core#694 where they shouldnt

[Phu2]

Processing this Flux

"https://madoc.bib.uni-mannheim.de/cgi/oai2"
| open-oaipmh(metadataPrefix="marc21")
| decode-xml
| handle-generic-xml
| encode-json(prettyPrinting="true")
|write("stdout"); 

in PG results in a gigantic file >200GB, see eg. this comment by @dr0i #159 (comment)

I can reproduce this behavior. Some observations made:

• using encode-json(prettyPrinting="false") instead of true runs succesfully generating txt-file only 172MB
• processing records in 2 subsets also works. First subset using dateUntil="2020-01-01T00:00:00Z" and dateFrom="2020-01-01T00:00:00Z"for the second subset
• the gigantic result file contains line breaks and white spaces that accumulated more and more towards the end of the file as already noted Restrict file size of tmp files #159 (comment) and handle-generic-xml cant handle nested record tags of different namespaces in one xml metafacture-core#694
• PG writes approx. 20 GB to the result txt-file in 5 minutes, after 90 minutes 346 GB were written and still does

MADOC's OAI endpoint delivers approx. 52k records.