Tech Stack Considerations

[jankapunkt]

this is a direct continuation of #392

These are my first thoughts but we need to create a much more elaborate and detailed spec for this endeavor!

---

This is my answer to #392 (comment)

real html. strong css support. minimal js. no hydration.

Well, we are building a rich application with complex user interactions, which is why we chose VueJS, because with plain html + minimal js we will not get near any of the functionality we currently offer. So keeping the frontend in Vue is a no-brainer, especially because Vue makes everything right what is wrong with React. No hydration is also impossible, because that would mean - no lazy loading of data after the page has been loaded. Css is done via tailwind, which is also something I would not drop as it is the best way to support our custom component design.

TL;DR - for the frontend I would not change anything major.

what are alternatives? leveraging your deep MeteorJS knowledge? Going for an all python setup? Or going full leftfield with Ruby on Rails?

We have to consider what each language and framework offers and how well we can build a community around it. Also - we have to consider the greater package ecosystem (npm/pypi etc.) and also how well apps can be deployed with that stack.

• MeteorJS - perfect WebSocket integration and for collaborative integration; easy deployment; but it is bound to MongoDB and NodeJS, which both are very low in community-size within science and research software engineering
• Ruby / Rails - great opinionated and mature framework with good scalability and elegant code structures but also way too hard to build a community of scientists / RSE around it and we as team have not really experience / skills with it
• python - defacto standard in science and widely adopted plus a huge ecosystem in packages BUT web is not the first class citizen for it and developing web-apps is way harder to for web-first environments, such as php+laravel or NodeJS + whatever

Then we have other factors to consider:

• how well can we pack the stack in docker / docker compose?
• how well can we implement live collaboration? NodeJS has YJS, what alternatives exist in the other languages? We do not want to implement a full OT or CRDT server on our own...
• we also have to consider, how well we can package a local executable self-containted app from that stack to support full local offline work
• finally, we also have to consider the ease of deployment from users (labs, research groups, universities) perspective - the more complex this is, the less likely they will adopt OpenQDA for on-premise hosting
• there needs to be well designed and maintained packages for authentication (bcrypt, argon2), 2FA, passwordless, LDAP, SAML, OAuth etc. - we do not want to implement our own auth...
• async i/o and generally native async support are crucial as we will have many tasks that will communicate with external servers (plugins/services), which is async by nature. An example: Laravel has no native async support uses Queues that increase complexity by a lot and make it extremely annoying to manage in production. NodeJS has native async i/o and workers for heavy tasks, Python has async support and even threads (which are powerful but come with their own complexity), Ruby I don't know

and very good database performance for huge amounts of datapoints

• there we have to consider on the one hand, if we need multiple database, such as MySQL for general project data + a more specific one for critical huge amounts of data
• we also need to think about leveraging existing caching strategies etc.

Summary: the topic is very complex and need to create a list of factors for us but also users that matter when choosing a new stack

[kadewe]

Yes, based upon your writing I have thought about this in more detail. To really be as easy to deploy and to be integrated into data science tech stacks, I am currently toying around with the idea to replace the PHP/Laravel backend with a Python/Django (and maybe later a FastAPI for API-only access for specialised analysis pipelines) backend to get more people into co-developing and also in using all the data science python libraries (otherwise we would need to run php and python at least in the plugins as we already do).

I have created a proof of concept swap of the backend with python/django. While doing this I could solve some problems, such as communication between async processes and the GUI (especially important for transcription and importing right now), I flattened the project<->team relation, a fully functional refi project importer and exporter. The repo is private to the OpenQDA-org, but just contact me if you want to have a look at it.

The really important result from this prototype though is that it was rather trivial to keep all VUE UI logic and just replaced the PHP/Laravel backend. I really would like to work into this direction. Also for the analysis part, everything was easily ported, as we already have the VUE part doing the analysis (although i really want to think this through because there are more advanced analysis methods where it is easy to use python/pandas etc., so i think we have to further develop the analysis architecture a bit more - this is a topic i will spend more time on next month).

Therefore @helge-hb @jankapunkt we should discuss this as otherwise we have to do work on updating the php backend code.