New ignore-filter option for tiddlywiki.files

[Mirodin]

I would like to propose an extension of tiddlywiki.files to add an ignore-filter ( or excludeFilter, ...) option which holds files that should not be included by the loader. Following a discussion over at Talk Mario suggested I should start a feature request. He did talk about an addition to plugin.info but I think that tiddlywiki.files is a better place since it deals with this stuff already and plugin.info should not be polluted with development requirements.

Right now the loader includes everything that is inside a directory by default which causes repository artifacts to be included as well. Current behaviour of tiddlywiki.files is that of a whitelist which forces one to manually list all files except those not to be included which makes the whole process quite error prone.

I am capable to create a pull request but would like to discuss syntax/features and if something like that would even be accepted beforehand to save unnecessary work for everybody.

[pmario]

The problem is, that tiddlywiki.files is in the wrong directory. It's plugin.info, which is in the same directory as your "alien" .something files. ... Or may be we generally ignore .files. ... But there are huge differences between linux and windows between those types of files. .. They are "normal" files and directories in windows

[Mirodin]

Maybe I am misunderstanding the documentation but to me it looks like whenever there is a tiddlywiki.files inside a directory the rules inside it will be used to decide what to load? So one could just put a tiddlywiki.files alongside plugin.info and that should solve the issue.

I just tried this the whitelist way that is currently possible with a test plugin and it seems to confirm my assumption. The problem with the current approach is that one has to extend the whitelist tiddlywiki.files file whenever there are additions to the plugin.

I envision this to be analogous to a .gitignore.

[pmario]

I just tried this the whitelist way that is currently possible with a test plugin and it seems to confirm my assumption. The problem with the current approach is that one has to extend the whitelist tiddlywiki.files file whenever there are additions to the plugin.

That's exactly the problem, why I suggested it to be in the directory. ... The initial intention of that config file was for external libraries, where we have to allow some files and that's basically it. External dependencies don't change often

[Mirodin]

I can see what you mean. But I still feel like plugin.info is for metadata about the plugin and not for build settings that are meaningless for users.

For the time being I fixed the problem for myself by creating a shell script that moves the files to /tmp, builds the plugin and moves them back. Feels hacky but at least it works.

[Jermolene]

Hi @Mirodin I think it makes sense to extend tiddlywiki.files with the ability to add file patterns to an ignore list, as well as the current accept list. As you note in the OP, the syntax/wording is critical to the usability of this feature.

[Mirodin]

Thank you @Jermolene for chiming in. I am willing to create a pull request with the proposed changes but as you pointed out naming is king because confusion is what breaks a feature. Do you have any preferences in that regard since I am by far not versed enough in TW internals to find a good name?

[Jermolene]

Apologies @Mirodin This new option only applies to the directories listed under directories, and not to the tiddlers entries. So, on the face of it, the most logical and consistent thing would be for each object in the directories object to have a separate, optional property called excludeRegExp whose value is a list of regexps (as strings). For example:

{
    "directories": [
        {
            "path": "../../../input/pdfs",
            "filesRegExp": "^.*\\.pdf$",
            "excludeRegExp": ["foo*.pdf"],
            "isTiddlerFile": false,
            "fields": {
                "title": {"source": "basename-uri-decoded"},
                "created": {"source": "created"},
                "modified": {"source": "modified"},
                "type": "application/pdf",
                "tags": ["$:/tags/AttachedFile"],
                "text": "",
                "_canonical_uri": {"source": "filename", "prefix": "pdfs/"}
            }
        }
    ]
}

(We would be to support filesRegExp and excludeRegExp having either a single string or an array of strings).

We already use a single regexp to filter out *.meta files. The guts of this change would appear to be to replace that regexp with the list of regexps (as strings) from the filesInfo structure. The code in $tw.loadTiddlersFromSpecification is already pretty hard to follow so we may want to consider some refactoring to simplify these changes.