Add support for rendering DataHarmonizer instances on GitHub pages

[jfy133]

Hello,

Thank you very much for DataHarmonizer, it is a really great tool - particularly that it doesn't need a standalone server to render.

I have two projects where I would like to host an instance of DataHarmonizer on our project website, which is hosted on GitHub pages.

Background

So far I've been able to make it work with two versions of DataHarmonizer, but with different levels of 'tweaks'.

1. Older version:
   • Repository: https://github.com/MIxS-MInAS/MInAS-DataHarmonizer
   • Rendered version: https://www.mixs-minas.org/MInAS-DataHarmonizer/
   • Forked from commit: b31a061
   • Note this has a few extra HTML customisations on the header bar, which is why it looks a little different from stock DataHarmonizer
2. Newer version:
   • Repository: https://github.com/MIxS-MInAS/miaard-DataHarmonizer
   • Rendered version: http://www.mixs-minas.org/miaard-DataHarmonizer/
   • Forked from commit: b31a061

The older one worked almost out of the box, I just had to:

1. Run yarn build:web
2. Copy web/dist/ to a docs/ directory at the root of the repository
3. Activate Github pages, and set on this page to Source: Deploy from Branch , then Branch master and directory docs/

However since commit b31a061 something appears to have changed in the main.js so that this no longer works out of the box in my fork from commit b31a061.

Seemingly this is something to do with how the generated scripts/main.js file in picks up the paths to the schema.json.

Investigation

After a lot of trial error, inspecting the GitHub Pages page with browser tools, and a little bit of AI help with Claude Sonnet - it pointed me to the fact that there is a function:

            : await (async function (e) {
                for (const n of Object.values(oe.menu))
                  if (n.folder === e) {
                    var t = await se(`/templates/${e}/schema.json`);
                    const r = { name: e, default: { schema: t }, locales: {} };
                    if (n.locales)
                      for (const [i, o] of n.locales.entries())
                        if (i > 0) {
                          let n = B(
                            t,
                            await se(`/templates/${e}/locales/${o}/schema.json`)
                          );
                          r.locales[o] = { schema: n };
                        }
                    return r;
                  }
                return null;
              })(e)),

Which hard codes the two locations of the schema.json to starting with a root of /templates/ . However when hosting this on GitHub pages directly, this fails because this page results in a 404, as it tries to find the file in the root of our domain, rather in the specific GitHub repository the website is coming from, i.e. in this case https://mix-minas.org/templates/miaard/schema.json rather than https://mixs-minas-org/miaard-Daharmonizer/templates/miaard/schema.json

Instead, if I manually edit the generated main.js to include the suffix of the GitHub repostiory to two paths, then the page will successfully render on GitHub page, i.e.

            : await (async function (e) {
                for (const n of Object.values(oe.menu))
                  if (n.folder === e) {
                    var t = await se(`/miaard-DataHarmonizer/templates/${e}/schema.json`);
                    const r = { name: e, default: { schema: t }, locales: {} };
                    if (n.locales)
                      for (const [i, o] of n.locales.entries())
                        if (i > 0) {
                          let n = B(
                            t,
                            await se(`/miaard-DataHarmonizer/templates/${e}/locales/${o}/schema.json`)
                          );
                          r.locales[o] = { schema: n };
                        }
                    return r;
                  }
                return null;
              })(e)),

Request

I was wondering if it would be possible to update the more recent main.js script to again support more dynamic root domains somehow.

I don't know what has changed exactly between the two commits, but given it was possible before in the older version of DataHarmonizer, I wonder if somehow this area of the file could be reverted - however I 0 javascript experience, so I am unable to myself inspect/work out what impact of these changes are nor how feasible it would be to do so.

[ddooley]

Thanks for your support and work on customizing to your aims! I'll have to take a closer look at this issue but am swamped until after the 16th December. What I can say quickly is going forward we are deprecating the whole ".../locales/${o}/schema.json" folder structure. It was one approach for multilingual support, but we've found another way to bundle language translations straight into the LinkML schema.yaml. All that newer work is represented in the dh2-1m-rework branch which we're aiming to beta and user test in January.
If it is possible, you are welcome to just comment out that whole code fetch of /locales/ ! (That would involve as well hiding the Language: " English / French" UI element too)

[ddooley]

BTW your MIxS-MInAS project implementation looks really interesting. I will send it on to our curation team!

Also, we do have a periodic DataHarmonizer development meeting if you want to attend sometime and bring issues etc. The next release, though it sticks to the existing DH LinkML specification compatibility quite closely, is filled with enhancements including ability to edit schemas from DH itself.

[jfy133]

Hi @ddooley ! Sorry for the delay in getting back to you (was running the workshop to fill the 'MIaaRD' schema with content, described in the OP)

No worries, for the time being! I can still manually 'fix' the path in the javascript for the time being. I think in the long run it would still be very nice to support as GitHub pages is generally an 'easy/accessible' way of people hosting small websites without too much fiddling.

Good to know about dropping of the locales, I will ignore that one for now and once merged into main will make sure to sync accordingly.

Thank you for the kind words about MIxS-MInAS! The small customisation with the header/description logo etc. was something else I wanted to make an issue about as it would also make the project more 'appealing' to people who want to stick some project branding etc on it, but it seemed minor for now ;). But could be nice to have e.g. specifying a logo and some text via a config file or the like.

Noted about the development meeting! Currently I'm only really using it as a 'reference' to have people poke around some of the schemas we are developing, but if we adopt it more heavily (and I find more issues, which generally pretty few so far) I'll come along :)

[jfy133]

Forgot to mention, if anyone else wants to try, to repair the copied /web/dist main.js file, you can run this command:

sed -i 's#/templates/${e}/schema#/miaard-DataHarmonizer/templates/${e}/schema#g' docs/scripts/main.js

It also appears to work on the dh2-1m-rework branch

[jfy133]

@ddooley do you have a rough timeline for the new branch to be merged/DH2 to be released?

I'm having some issues playing around with DataHarmonizer (i.e., I cannot remove the CanCOGeN Covid-19 directories, nor set display: false in menu.json, not move it to the end of menu.json).

A currently working verison: https://github.com/miaard/miaard-DataHarmonizer/tree/dh2-1m-rework with being hosted on GitHub actions here: https://miaard.github.io/miaard-DataHarmonizer/

But if ou run yarn dev locally and try to update menu.json to somehow not include the CanCoGen scehma, you'll get the error as seen in the screenshot below.

But this error happens on both the master and dh2-1m-rework branch..., but I don't know if this is because the code base of both branches are still unstable or not...

Do you have any suggestions/guidance?

[ddooley]

Note, to get a more precise javascript line error you can press option-command-J on a mac and go to the console errors report to see the precise javascript line.

Within the GitHub local folder environment, deleting the /web/templates/cancogen_covid19/folder/ should clear out its contents, but you will find that the GitHub commit system still keeps the folder itself in place - is that what you are speaking of? Removing the cancogen entry entirely in menu.json should be all you need to do. It won't work simply to set display:false since that only controls its appearance in menu system, and folder content will still be expected. I.e. remove this entirely:

"CanCOGeN_Covid-19": {
"folder": "canada_covid19",
"id": "https://example.com/CanCOGeN_Covid-19",
"version": "3.0.0",
"templates": {
"CanCOGeNCovid19": {
"name": "CanCOGeNCovid19",
"display": true
}
},
"locales": [
"en",
"fr"
]
},

[ddooley]

The dh2-1m-rework is in finishing stages + user experience review so I think its going to be at least another month. The parts that are subject to revision are related to the SchemaEditor application; loading DH templates and loading/ saving their data is not expected to change at all (except for a few tweaks to json data structure for 1-many table templates).

[jfy133]

@ddooley removingfrom the menu.json method is what I had done with earlier versions and had thought was the correct behaviour indeed!

Here is (a low quality) gif of what happens when I do this, you can see when I remove the entry but keeping the rest of the entries, I get the error below:

can't access property "split", template_path is null
loadFromMenu@webpack-internal:///../lib/Toolbar.js:1330:46
loadSelectedTemplate@webpack-internal:///../lib/Toolbar.js:1335:62
initialize@webpack-internal:///../lib/Toolbar.js:280:10
Toolbar@webpack-internal:///../lib/Toolbar.js:93:10
main/<@webpack-internal:///./index.js:83:5

My Firefox inspect mode (I'm on Linux) repots the following:

[ddooley]

This is quite a mystery. On dh2-1m-rework I tried what you did above and it seems to work - I remove the first CanCoGEN entry and it doesn't error out, instead it provides GISAID as default schema...

So the line that is erroring out is: const [schema_folder, template_name] = template_path.split('/');
and template_path comes from the pulldown menu: const template_path = this.$selectTemplate.val();
This suggests the select options for one of the json configurations doesn't exist in the JSON menu file. I see part of your json file but perhaps you could show here or send me the menu.json file?

[ddooley]

I'll check about folder name impact if that is somehow a connection , later today.

[jfy133]

This is quite a mystery. On dh2-1m-rework I tried what you did above and it seems to work - I remove the first CanCoGEN entry and it doesn't error out, instead it provides GISAID as default schema...

Ok that's very odd... that's the behaviour I have on the MIxS MInAS Dataharmonizer...

So the line that is erroring out is: const [schema_folder, template_name] = template_path.split('/'); and template_path comes from the pulldown menu: const template_path = this.$selectTemplate.val(); This suggests the select options for one of the json configurations doesn't exist in the JSON menu file. I see part of your json file but perhaps you could show here or send me the menu.json file?

https://github.com/miaard/miaard-DataHarmonizer/blob/dh2-1m-rework/web/templates/menu.json

And the templates: https://github.com/miaard/miaard-DataHarmonizer/tree/dh2-1m-rework/web/templates

[jfy133]

Thanks for looking into it @ddooley !

[ddooley]

I understand now. In your menu.js , the CanCoGEN schema/template was the only one that had display:true in one of its templates; deleting it deleted the last template from would have been a DH menu of templates, and so on load, there was no first (default) template to display! So set at least one other template to display: true ! In meantime I'll put an error warning about that situation into the code!
Thanks for finding a bug!