Skip to content

Latest commit

 

History

History
416 lines (333 loc) · 16.3 KB

development.md

File metadata and controls

416 lines (333 loc) · 16.3 KB

Development

To work on new features, bug fixes, tests and prototypes.

The project should be able to run locally without the need of a developement server, but requires manually fixing the "assets path" (CORS error if file:///.../internet4000/find/index.html is opened).

Similarely, an other server can be used than the one provided, ex: python3 -m http.server --directory . Node.js is currently only required for running the tests.

Documentation

  • the javascript file src/index.js contains all the code for I4kFind.
  • the javascript file src/ui/index.js contains all the code for I4kFind web-components, composing the graphical user interface

The web-components are only loaded after the find logic, if the user stays on the page (if there is no user query).

Find does not have any production dependency, uses vanilla javascript, html and css code, that runs only in the user's browser tab.

General

  • npm install to get the development dependencies (there are, and should be, no production dependencies)
  • npm start to run the local server (it should say at which URL the page is locally available)
  • There is no build system, to keep things small, simple and accessible; so the folder can be deployed as it is onto a web server, or after any change has been made (and we're satisfied with them).

All the Find logic code, is located in the folder /src/index.js. It does not, and should not import other files.

Other files are configuration files for the continous integration, testing, opensearch, hosting.

Notes the URL's root / are the base of the find instance, or of the project folder depending on the context.

NPM package

Find is available as a javascript es6 NPM module !npm i4k-find.

Install

In a new project use npm instal i4k-find.

import find from "https://cdn.jsdelivr.net/npm/i4k-find@latest";

const openWindow = false; ;; defaults to true
const userQuery = "&gh internet4000 find"
const queryResult = find.find(userQuery, openWindow)
console.info(queryResult)

Publish as a package

It uses a github workflow to run npm publish when a new tag is created, for example 0.0.10, which should match the value of the package.json.version (and not already but published with this version on npm).

Node.js CLI utility

To get a Find result from a shell, in node.js. Can run for example:

  • node ../com.github/internet4000/find/src/scripts/i4k-find.js "&gh i4k find" (if downloaded loacally, from an other folder)
  • npx i4k-find "my query in bash string", directly from npm
  • npm link i4k-find or npm link <path_to_local_find_index_js> (to test locally)
  • npm run find "hello" when developing in this repo (package.json.scripts.fin)
  • as a shell alias

It is possible to bind i4k-find to a "shell environment variable", to use Find from a shortcut, and pipe its output to other program (ex: browser open URL of the result), or pipe other program's input to it.

Running from the shell, should only output the "translated query"; and liberty to the user to "pipe" this output to an other unix utility (maybe open in a browser)

# feed "text string" to Find as arguments
echo hello | xargs npx i4k-find
npx i4k-find "&gh internet4000 find" | firefox
# the previous is currently not working @TODO

xargs is a unix utility to convert "stdin" to "arguments" in a shell. Find is trying to be "isomorphic" without build tool, getting to read stdin input in node.js requires to import readline

Note: it could be improved to allow calling more or its "methods" from the CLI, or be passed arugments (more than the default q query string), such as --json, to get a for detailed "parsing result answer" (also see i4k npm pkg media-url-parser).

From a queries.txt file with the content:

&gh internet4000 find
+space hello world
+data-json {"my_boolean": true}

The following could be used to generate a Find result for each line:

cat queries.txt | npx i4k-find
# will output
https://github.com/internet4000/find
https://goog.space/#input=hello%20world
data:application/json;charset=utf-8,%7B%22my_boolean%22%3A%20true%7D
# there is a newline charater at the end of the output (@TODO: rm)

Notes: Not sure about many things here:

  • new lines \n, insert or not? difference with ls output
  • how to best pipe out to other programs? pipe in to recieve from other programs, or user input npx i4k-find "hello" versus npx i4k-find then hello \n world \n +space test \n +m tokyo \n C-d / C-d
  • how to best read from files
  • how to best pass the result to the browser CLI command, to open a new tab with the result, or multiple tabs per "line result" ?

Testing

To run the tests, after having done npm install to get the initial dependencies, and npm run test to run the tests once.

See ava's documentation for usage.

Sync

Could try to auto sync user defined engines, by "submitting all changes" to a hidden update-password-form, and submit it.

Maybe that would triger the browser/password-manager to update the password value for this site.

URLs

The "base URL of the site" should of the format (window.location, with no hash or search param, though if only look for one search param value, q for user query):

These URLs can, or not, have a trailling slash /

There we can build these URLs for the user search query (if the param is customized to my-custom-param)

Open Search file generation

We're trying to (auto)generate an Open Search (Description) file, that is corect for each deployment context. That way a user can clone/fork the repo, setup the "one ENV var" (if we cannot get it in our scripts), and the deployment && XML file (and all config) would be correct.

The generation is done by the file src/scripts/opensearch-xml.js, which can be called with the command npm run opensearch, (and is run for each new deploy in the CI/CD recipes).

This should generate the correct opensearch.xml, from information that the scripts finds from the package.json file and "i4k-find" key. As reference, check the project's own.

export I4K_FIND_URL=https://example.org/i4k-find
npm run opensearch
# assets/opensearch.xml

Open Search Suggestions API

The objective with the "service-worker client-side API", is to allow Find to give feedback, directly in the URL bar, while typing.

Usage

When installed as the default search engine, Find could offer suggestions, since the browser is supposed to call the suggestions API endpoint defined in the =opensearch.xml= file.

If the "find deployment page of your default search engine", has been visited once, the web-worker.js is loaded, and can maybe execute code, even while the tab is closed (as it has been register).

The function of the web-worker, is it catch (middle man), all queries that are made (all change of characters in the input), and provide suggestions of search results for the current user input value (in the URL bar directly).

Find is setup to always get the data from the hash parameter, which is the part of the URL that is not sent to the server.

If we open the browser developper tools, and inspect the network requests made by a search query (from example in the UI search input). We can see that no data goes to the server in the Headers or in the Body of the request.

Note: we should probably make a test with wireshark and tcpdump, and look at the output of the queries we make (in the browser, via all Find methods), to see if any data is leaking out of the browser tab, into internet/the-networks. Probably also check with a node/python/.. script, to see what their server's can catch from our requests. Also if the web-worker is not (yet) registered.

References

What to try

We're currently only generatig suggestions for the avaiable default and user engines.

It would be nice to find a way to have a general, and specific methods (for each symbol, engine etc.), to seart getting suggestions directly from their content.

Maybe:

  • use their opensearch file, if they have one, and we can get data there (make a search query to their search/suggestion endpoint?)

open about:debugging (from the application dev tools tab, for accessing the service worker's console, debugging etc.)

Visiting the find URL

It is visited each time a search is made.

We try to only execute the user query part of the script, and not load any other file/assets if there is any, for speed.

So the UI part (web components) only loads if find does not need to redirect to the final page requested by the user.

Also, as a utility for "other internet protocols", Find tries to "polyfill" different protocols with a "proxy/gateway".

gopher://gopher.floodgap.com gemini://kennedy.gemi.dev spartan://spartan.mozz.us finger://happynetbox.com text://txt.textprotocol.org git://txt.textprotocol.org <URISchemeProtocol><:><//><ressource>

Where // is the default "engine used by the protocol, and the protocol symbol ID is the value of new URL(<schemeID:>//).protocol.

So to overvwritte the protocol proxy web app (protocol http(s) to protocol <schemeId>), we need to overvwritte its engine // (the default "web-to-protocol proxy").

Adding symbols

It is possible to edit the code and add new symbols.

For example, adding the ? symbol does not work, because the browser treat it as a special reserved character since it builds the search param part of a URI.

To test if a new symbol can work, try a new "omnibox search" (not just a Find input search), and see if the resulting "default search page", contains the exact query that was first inputed. If the resulting search includes all characters, then the first one (or <symbol>:) can be treated as a symbol since the browser does not catch it (find has the opportunity to do).

Questions

To be investigated.

symbols interpretation and combination

Some questions about symbols and their lexical interpretation, combination, and execution (build & open URL currently).

The pipe | symbol/operator? maybe to pipe the current url (or data) as input, to other Find symbols, in the same query.

"If first symbol is | pipe" or/and "if last symbol is | pipe";

# the user is on current URL: https://example.org
|https://example.org <FindQuery|<SomethingElse>>
|<stringData>|+space|+qrcode|+share-link
|<stringURL>|+|+qrcode|+share-link
|!sheet <spreadsheet-id>|&sheet-json

These could be saved as user/default defined "combinators" and function, to manipulate the data and URL they contain, and their app generates

## if a pipe `|` symbol existed
#add | note2space2qr |+space {}|+qrcode
## or if a #add-pipe "helper public user function" existed
#

Could that do somehing interesting?

Generally, a bit like shell or reverse polish notation or list based programming languages, could, from the URL, build things.

Using the "local suggestion api" (see web-worker) and maybe browser client side storage, there could be interesting applications.

i4k-find URI scheme

To represent a find query and what to expect; ref: https://en.wikipedia.org/wiki/List_of_URI_schemes

i4k-find://host[:port][;/#<param>=<query>]
;; https://host[:port][;/#<param>=<query>]
;; file://host[:port][;/#<param>=<query>]

with a # param, its value should not be sent by the client (accessing the URL), to the server (serving the document, pointed by this URL). But the client should use the hash param value, with the document (code/data) returned by the server

Could also imagine (if we trust the server):

i4k-find://host[:port]/<query>
i4k-find://host[:port]/<symbol>/<engine>/<query>
i4k-find://host[:port]/<symbol>/<engine>/<query>

Where <symbol> and <engine> are utf-8 text strings, without spaces, and query is also a string but can contain any characters.

In the case of find symbols and engines, where the engine URL, is a "non-standardly accesible URL" (ex: data:application/json;charset=utf-8,{}), we could also use a Find URI scheme, to return a i4k-find:// prefix. Find could therefore figure that it needs to be called again, to interpret a query again.

As an alternative, window.location is the equivalent if the current Find client needs to check if the user request should be re-analyzed by Find.

Links & refs

References, links and inspirations:

Explore as well:

Notes

Creating and using URI/URL, with a comprehensive syntax, allows to interact with the destination website/application, to feed it input parameters (and their values). This is a general pattern used in the Web and computing systems in general, for accessing web address and file system references.

The data contained in the URL can then be re-used in the web site as an application input value, and that works well with web-components (executed on the user client side).

https://example.org/my-app?my-attribute=true&my-data={"my-json": 2}
<my-component my-attribute="true" my-data='{"my-json": 2}'/>