Requirements & Vision Grid v2

[PhilippLgh]

Based on outstanding and old tickets this will be a collection of requirements and ideas for a potential Grid v2 release.
A lot of it won't be implemented due to a lack of resources but hope this can serve as some orientation of what features and software quality aspects would be desirable:

• cross platform e2e tests (As a developer, I want integration tests for Grid, so that I can release with confidence #92)
• allow to delete binaries / clean cache (As a user, I want to be able to delete binaries, so that I can preserve disk space and cleanup the version list. #100)
• support dapp developers / ganache (As a user, I want to run ganache, so that I can test my dapp without running a full node #189)
• be able to cancel started downloads (As a user, i want to be able to cancel a download, so I can save resources if I change minds. #192)
• start GUI without starting node / being able to connect to running node (As a user, I want to have information on how to connect with the running node #214, spike: connect to locally running client #450)
• provide easy way to spin up private networks (As a user, I want to start a Geth private network, so that I can use Grid in education workshops #223)
• have metadata about type of binary e.g. (service, wizard,..) (As a user, I want to see which binaries can be started as service / daemon and which not #295)
• provide easy access to version changelog (As a user, I want to see the version changelog for the client binaries #294)
• metadata about signature verification checks (As a user, I want to see which binaries passed (or didn't pass) signature verification  #292)
• aid onboarding: how to connect / use services (As a user, I'd love to know how to connect with the services I start, so I can take more advantage of what Grid has to offer. #253)
• sync indicator in nano (As a user, I want to check syncing status for my running clients on the tray window #372)
• improved logging with loglevels (Make better use of logging #371)
• improve look & feel + OS specific behaviour (improvements to nano window #331)
• multi client support + conflict resolution (ports) / rpc proxy concept (improvements to nano window #331, As a user, I want to be able to start a service multiple times #304)
• improved offline support (As a user, I want to use Grid while offline, so that I can enjoy additional security and usability #320)
• better support of "non-service" binaries (wizards, compilers, signers) (As a user, I want to execute actions through Grid UI e.g. from the terminal component #307)
• allow to "customize" ui / hide or prioritize clients (As a user, I want to hide plugins, so that I can see only those relevant to me #465)
• improved network visibility + protection (mainnet vs testnet) (As a user, I want to see which network I'm on more visibly, so that I can debug more confidently #448)
• decoupling from Electron (IPC) + transition to http (bug: navigating to another plugin while downloading a release loses track of the download #411)
• improved ui for network health / connectivity / peers (As a user, I want a notification (in nano?) that I've lost peers or my node has become unstable, so that I can maintain my dev workflow #496)
• improved cli support / better terminal UI e.g. based on node-pty (As a user I want to be able to see and make calls to supported client methods so that I don't have to use CLI especially if the node was not started by me (there is no CLI window) #36)
• better handling of network specific directories & default settings (Geth nitpicks #512)
• better handling of platform architectures (32 vs 64 bit etc) (Geth nitpicks #512)
• improved fs integration i.e. watching directories for changes (binaries deleted, created) (bug: deleting releases not acknowledged by UI and produces broken state #488)

[PhilippLgh]

Will use this issue also to outline some of the concepts and ideas we had for the future development

[PhilippLgh]

Workflows

What are workflows?

The idea of workflows was to move away from the "pick and configure a client" model towards goal oriented functionality. Instead of geth, besu, parity, ...

The UI would display:

• mainnet
• public testnet
• private testnet
• compile zk proofs
• make transaction
• ...

Workflows would have a config UI where they could be modified to e.g. use a certain client or set specific directories.

Related

#525

[PhilippLgh]

HTTP RPC Proxy

What is the HTTP RPC Proxy concept?

Currently, clients "compete" for ports. Let's say there are certain ports that are whitelisted / forwarded in the firewall that are the default ports. If a port is used by an application others cannot use the same port. This totally makes sense if there is only one application running but if we want to test multiple clients (which seems obvious for Grid) it causes problems. The idea was to have Grid listening on a default port like 8545.

Web applications could attempt to establish a connection with Grid and Grid would maintain a list of whitelisted apps. So if the user browses to a dapp the dapp could try to connect to Grid (this could also happen through metamask). Grid could then either start a client if necessary or forward to one of the running clients based on settings and uses case. clef for example could be used as signer while geth is used as RPC provider.

Screenshot

Related

#519

[PhilippLgh]

Grid Apps

What are Grid Apps?

Clients have very specific requirements for their UI and it does not make sense to create a one-size-fits-all GUI. Usually, it is also not enough to just configure and start a service but users want to interact with it.
The idea for grid apps was to use Grid as a platform to help projects provide web based desktop apps, do rapid prototyping and testing without all the cross platform deployment issues.

Grid would manage the binaries and permissions and inject an API into web apps similar to web3. Web apps could use this extra API to e.g. request the start of a binary with a certain config. A web app could for example request an IPFS connection. Grid would make sure that the binaries are available and wire the UI to the respective service.

It would then allow apps to be "installed" by creating shortcuts on the user's desktop to mimic stand-alone apps (similar to PWAs) while doing the hard work of providing the cross-platform runtime environment, update apps and orchestrate services.

Grid Apps would be offline-ready and don't need to be hosted. This is possible by signing them using ethpkg and packaging the built into a tar or asar. Grid would take care of caching and verifying these packages and putting them on GitHub releases or IPFS would be enough to have a "full" desktop app"

The result is some kind of marketplace for ethereum apps not to be confused with dapps.

The below wallet is one (early) example of such an app:

Screenshots

Related

https://medium.com/ethereum-grid/exploring-ethereum-with-geth-graphql-and-grid-6df38f2a86c

https://medium.com/ethereum-grid/ethereum-101-whats-a-blockchain-5a1e925963d7

[PhilippLgh]

Grid as a Service

Similar to the HTTP proxy concept Grid could also provide a local API to other applications that don't want to implement binary management. Remix is one example but a VSCode extension might have similar requirements.

The idea is to start Grid headless as a service. If the desktop app is started it will connect to the service otherwise it will just stay in the background until an app like Remix requests access to a binary like geth. In this case Grid would make sure that geth is running with a proper configuration and proxy the requests.

[PhilippLgh]

Grid Core

The idea behind grid-core was that a lot of Grid's functionality could be valuable outside of the scope of a desktop app: e.g. in CI, tests or on a server.

grid-core was started to separate the Electron based desktop app Grid and the binary management logic and make it available to other projects as direct dependency or via its own http API.

Grid core is is a secure execution environment that allows scripts to safely interact with binaries and even docker containers. It comes with many tools and helpers to configure binaries, spin up user interfaces and wire everything together into workflows that can be versioned and shared.

In this process, all code was migrated to typescript, docker support was added, the workflow concept got implemented, a CLI wrapper was implemented and an experimental HTTP API and HTTP based app server (no electron webview) were added.

Eventually, the project never reached a satisfying state because it ended up doing too many things while creating lots of extra complexity. At the same time, the team was reduced to ~15% capacity making a full re-integration almost impossible.

However, it was a good experiment to evaluate and test out different ideas. The best stable result of this refactoring is a reduced module https://github.com/PhilippLgh/ethereum-binaries that only focuses on the binary management aspects.