In Browser KWIC Indexer

[DocOtak]

Topic for discussion

Since the KWIC indexer is broken and I wanted to learn some JavaScript DOM manipulation (have been doing react for a while), I've made a page that loads the standard name XML and turns it into a KWIC index, by default it will load the "current" standard name table, but the link to the standard name XML is changeable via a query parameter:

• Current: https://www.andrewbarna.org/cf_kwic/index.html
• V1 (example) but can literally be any standard name xml: https://www.andrewbarna.org/cf_kwic/index.html?tableURI=https://cfconventions.org/Data/cf-standard-names/1/src/cf-standard-name-table.xml
• V20: https://www.andrewbarna.org/cf_kwic/index.html?tableURI=https://cfconventions.org/Data/cf-standard-names/20/src/cf-standard-name-table.xml

We could remove all the static KWIC file from the website repo and just have links to this single file with the different name tables XMLs as their source. My napkin calculations suggest this would save 200MB from the website repo and we would have the added feature of having a kwic index for all the standard name versions.

This is a potential solution (or stop gap) for #296.

EDIT: Forgot to mention, I would add this file/indexer to the cf website and not be linking to andrewbarna.org, that is just for the demo. Source is here https://github.com/DocOtak/cf_kwic

[larsbarring]

Andrew -- this is really impressive! If we transfer this to the CF website, which is served by github pages, will this qualify as static webpages in a sense that is acceptable by github pages?

[DocOtak]

Yes it is "static" in the hosting sense. I'm already hosting it on github pages for this demo (just my own accounts version).

Google chrome is doing massive style recalculations for reasons I don't understand, so when you click on some keyword it can take 10+seconds to go to that section. What is most frustrating it that chrome is not doing this when the exact same file is loaded from my localhost or file system and the scrolls only take <50ms. As chrome (or its engine) is the most popular browser in the world, I suspect this is a blocker. The html version of the standard name table could get the same treatment as this KWIC index, then only the xml would need to be published on the CF website which would save a significant amount of data.

Aside, most recent KWIC index pages have 300k DOM nodes, which is 2 orders of magnitude more than what a tools like lighthouse recommends for performance reasons.

[DocOtak]

Ok even more frustrating, the performance issues are only on my site (or maybe relate to cloudflare). As an experiment I pulled the latest actually built kiwc index from the cf webite itself and put it on my test repo/server and it had performance issues just like the javascript version that I had made.

If folks are willing, I would like to try to put this on the cf website itself as a test.

CC @JonathanGregory

@cofinoa I wanted to make sure you saw this given your comment on the standard name v86 announcement

[JonathanGregory]

Dear Andrew @DocOtak

That's a clever idea, and only 150 lines of code! It is slow for me as well. I don't see a reason against trying it as a test from CF GitHub, to see if that makes it KWIKer. I suggest it belongs in the vocabularies repo, which is already exported to Pages.

Best wishes

Jonathan

[DocOtak]

@JonathanGregory just looking at the settings for the vocabularies repo, I don't think it is being exported to github pages yet. Is that something that is going to be turned on soon?

[JonathanGregory]

Sorry. I have switched it on. I think we will need it anyway.

[DocOtak]

Thanks @JonathanGregory I've opened a PR in the vocabularies repo. I've added a new feature: a link that will download the generated html for offline usage. Please have a look at the links on my original post to see how the downloads work.