Enhancements: Regex Extraction, Advanced Flags, and CLI Improvements

[mitchcapper]

This code was AI assisted though I have reviewed and tested the code.

Overview

This PR introduces several enhancements to the extraction process and CLI functionality. Key changes include:

• CLI Improvements: Added support for custom output file extensions and fixed directory structure preservation when processing local files.
• Formatting Fixes: Improved markdown formatting for multiline code blocks (they require a newline to proceed them or they don't work in markdown).
• Advanced Flags: A new CLI arg for experimental or specific test behaviors, starting with a flag to treat all <pre> blocks as code.
• Regex-based Extraction: A new option to extract the main body content using a custom regular expression before standard processing.

Commits

cli: Make keep-dirs work for local file input

• Fixed determine_output_path in cli_utils.py to correctly calculate relative paths when using --keep-dirs with --input-dir. This allows recursive processing of local files while preserving directory structure. It also fixes the issue where --keep-dirs would cause the output-dir to be ignored and output files to appear next to the original.
• Added test_keep_dirs_and_extension to tests/cli_tests.py covering both directory preservation and custom extensions.

CLI: allow specifying the output extension

• Added --output-extension CLI argument to override the default extension derived from the output format. Personally I would assume markdown files defaulting to .md would be more standard but as that would be a pretty large breaking change I didn't do that.
• Updated cli_utils.py to respect the user-provided extension.

extraction: Added AdvancedFlags option, added ALL_PRE_BLOCKS_ARE_CODEBLOCKS flag

• Introduced AdvancedOptions enum in settings.py for managing experimental flags.
• Added --advanced-flags CLI argument.
• Implemented ALL_PRE_BLOCKS_ARE_CODEBLOCKS flag to force <pre> elements to be treated as code blocks.
• Added test_pre_conversion_flag to tests/unit_tests.py.
• The CLI --help is updated to reflect the new argument and any flags in the enum but I didn't add to the overall documentation.

extraction: Added regex option for pre-body extraction

The general idea here is to allow users to provide a regex that will be used to extract the main body of the document before any other extraction logic is applied. This is useful for documents with a consistent structure where a regex can reliably capture the desired content. I found while trafilatura largely worked great the webpages I was operating on had the body in a

and this caused some odd formatting. Using a regex to extract the body pre-processing allowed me to get a much cleaner result.

• Added --regex-file-for-body CLI argument to allow users to provide a regex pattern for initial content extraction. We read the regex from a file rather than the CLI due to the fact regex's often have characters that can be escape or interpreted when used. Coming from a file not only avoids the requirement to get escaping correct but is shell agnostic, and allows to copy and paste the regex into a test engine easily.

• The API option is just "regex" and takes the actual regex pattern.

• Updated Extractor class and bare_extraction in core.py to apply the regex if provided.

• Added test_regex_for_body to verify the functionality.

• Updated documentation to include the new option.

Testing

• New Tests:
  • test_regex_for_body: Verifies regex-based body extraction.
  • test_pre_conversion_flag: Checks the behavior of ALL_PRE_BLOCKS_ARE_CODEBLOCKS.
  • test_keep_dirs_and_extension: Validates --keep-dirs with local inputs and --output-extension.
• Existing Tests: Updated test_code_block expectations for better formatting.

Risks/Notes

• Regex Extraction: To make things work correctly I wrap the captured groups in as otherwise shorter context may not be recognized as valid HTML. Allowing multiple capture groups should allow fairly advanced capture patterns.

• Advanced Flags: The ALL_PRE_BLOCKS_ARE_CODEBLOCKS flag changes default behavior for <pre> tags; it should be used when the target content consistently uses <pre> for code. In theory <pre> is always preformatted text so fixed blocks would seem to make sense, but again defaults to off. While it could be just another command line it would seem better to have potential flags rather than a growing list of command line options for experimental or edge case behaviors. While there is the settings file, this seems more static unlikely to change options that this didn't seem to fit into.

[adbar]

Hi @P4X-ng, your PR looks interesting but for some reason the tests haven't run.

I suggest you edit your PR or add a commit like git commit --allow-empty -m "ci: retrigger" && git push to hopefully bypass this Github issue.