Web UI search lists should not list page requisites by default

[douglasg14b]

Github for instance. There are many archived requests when you visit github.com. And while replaying enables one to replay these individually, I am unable to figure out how to replay all the requests for rendering the site?

There are many requests, as seen here, how can these be replayed "together"?

[douglasg14b]

Similarly, there are DOM snapshots. But they appear to be a manually triggerable thing, and not necessarily an automatic thing 🤔

Understandably knowing when to snapshot is rather difficult, but even coarse snapshots could be handy for going back and viewing full sites? What options are there there?

[douglasg14b]

(btw if you use email only and don't check the github thread itself, idk if email shows edits to the OP or comments)

---

I'm running on the assumption that automatic DOM snapshots are not yet a feature. This could be enabled through a combination of methods:

1. Wait for load (fires when the page & all dependant resources have fully loaded)
2. Observe network activity with PerformanceObserver for n seconds
3. Use a MutationObserver to ensure no DOM updates for n seconds
4. Fallback to waiting for a static n seconds in case of constant network activity & DOM mutations
   • Further Fallback to snapshotting if the page is being closed/navigated away from and a snapshot has not yet occurred
   • Further fallback to interaction, if no shapshot has been taken, and the user starts interacting with the content, that may be a safe time to take a snapshot

This isn't perfect, but it can be heuristically improved over time 🤔

[douglasg14b]

Followup, can I have my own userscript trigger the hoardy web extension snapshot? I could play around with this with user scripts for a while to figure it out if that's the case.

Hm, tried for a while and can't find a path for this. Userscript extensions don't really expose a the browser.runtime to send a message. I could modify hoardy-web to listen to a DOM event on every tag, but that's pretty gross, and totally abuseable by malicious pages 🤔 Plus then I'd have to package and add it to firefox as a temp extension, which is also kinda gross.

[douglasg14b]

Alright, I think I've essentially figured this out (Allowing a userscript to trigger a snapshot).

1. Add a content injection which is simply an event listener for a DOM event (ie trigger-snapshot)
2. In order to get the tabId the background script (core.js) needs to handle a getTabId command, and return the sender.tab.id for the request
3. The content.js script gets the tabId via a browser.runtime.sendMessage
4. The content.js script then send the snapshot message with the tabId
5. snapshot for that tab is triggered

Modified manifest-common.json

    "content_scripts": [
        {
            "matches": ["<all_urls>"],
            "js": ["inject/content.js"]
        }
    ],

content.js

document.addEventListener("snapshot-trigger", () => {
    console.log("Content script received snapshot-trigger event.");
    // Send a message to the background script
    browser.runtime.sendMessage(['getTabId']).then((tabId) => {
        // Once tabId is retrieved, trigger the snapshot action
        if (tabId) {
            browser.runtime.sendMessage(['snapshot', tabId]).catch((error) => {
                console.error("Error triggering snapshot:", error);
            });
        } else {
            console.error("Tab ID not found.");
        }
    }).catch((error) => {
        console.error("Error retrieving tab ID:", error);
    });
});

Modified core.js

function handleMessage(request, sender, sendResponse) {
    if (config.debugging)
        console.log("got message", request);


    let [cmd, arg1, arg2, arg3, arg4] = request;

    switch (cmd) {
    case "getTabId":
        getTabIdForContent(sender, sendResponse);
        break;
... 
}

function getTabIdForContent(sender, sendResponse) {
    if (sender.tab) {
        // Return the tabId from the sender
        sendResponse(sender.tab.id);
    } else {
        console.error("No tab information in sender.");
        sendResponse(null);
    }
    return true; // Keep the message channel open for async response
}

[oxij]

Unfortunately, GitHub does not send message updates as emails. Which is annoying, yes.

There are many requests, as seen here, how can these be replayed "together"?

I think you misunderstand my intentions here. I think, what you actually want is to filter out page requisites from that list on your screenshot. I suppose, the UI from your screenshot can be hidden under `/raw/*/*` and `/web/*/*` could instead only show "fetch root" pages (those that have at least one `reqres`/`WRR` with no `document_url` set, meaning they were loaded as a root document in a tab at least once). Would that do what you want?

DOM snapshots triggers via userscripts

They are not implemented because a proper implementation is a bit complex. I considered a solution similar to yours above, but it would inject that userscript into all pages, which has two major caveats: - content pages will now be able to tell they are running in a browser with `Hoardy-Web` enabled, which, for my purposes, absolutely must not ever happen; - this injection will consume additional memory for each tab, which is a slippery slope I rather not start slipping on. Instead, extensions can send messages to each other, so my plan was to add a public API to `Hoardy-Web` and then either figure out how to use that via userscript in `Tampermonkey` (if it can already do that) or patch it to support it. This way, script injection will only ever happen on pages that actually run userscripts, and pages that do not get userscripted will notice absolutely nothing.

[douglasg14b]

Code injection concerns

For automatic DOM snapshots yeah, those are valid concerns, but they are manageable problems to some degree, let me list them out:

1. Sites knowing hoardy-web is running
   • Likely a solvable problem, to some degree, by obfuscating the injected script. ofc signature based detection can detect it, but is that a threat model that's considerable in this situation?
2. Increasing page memory consumption
   • This can't be solved, though just like chemical toxicity, its all about moderation. What is the acceptable threshold for memory consumption? 500b, 1KB? 10KB? How many pages do users have open in a single browser instance?

---

Using Userscripts to call extensions:

Instead, extensions can send messages to each other, so my plan was to add a public API to Hoardy-Web and then either figure out how to use that via userscript in Tampermonkey (if it can already do that) or patch it to support it.

It does not, there is a PR open for this that is not received too well in Violentmonkey I believe. There seems to be resistance to adding GM_ capabilities. I went down this rabbit hole yesterday.

There are already forks of similar with the ability to send a message to an extension, however, these are:

1. Not necessarily safe and trusted
2. Not as accessible to the majority of users

Maintaining a fork would work ofc, but that doesn't necessarily make the API accessible to most users of this extension 🤔

[douglasg14b]

I suppose, the UI from your screenshot can be hidden under /raw/*/* and /web/*/* could instead only show "fetch root" pages (those that have at least one reqres/WRR with no document_url set, meaning they were loaded as a root document in a tab at least once).

Hm, I think that would be reasonable yeah.

Though don't break your back on that for my sake. I'm playing with the hoardy-web extension as a tool, and dumping the data through my own server which writes a detailed index to sqlite for easier index access/querying/aggregation.

My objective is to build my own UI in front of this index, and to keep playing with it.

HOWEVER. I actually have no idea how you actually are serving up pages, or why some DOM snapshots will show images, have css styling..etc and why some are essentially just the raw HTML with nothing else that was on page. Especially no idea how the mirroring bit works.

[oxij]

> I suppose, the UI from your screenshot can be hidden under `/raw/*/*` and `/web/*/*` could instead only show "fetch root" pages (those that have at least one `reqres`/`WRR` with no `document_url` set, meaning they were loaded as a root document in a tab at least once). Hm, I think that would be reasonable yeah. Though don't break your back on that for my sake.

Well, you do raise valid points with your Issues, so I'm not complaining.

I'm playing with the hoardy-web extension as a tool, and dumping the data through my own server which writes a detailed index to sqlite for easier index access/querying/aggregation.

Sounds like #6.

My objective is to build my own UI in front of this index, and to keep playing with it. HOWEVER. I actually have no idea how you actually are serving up pages, or why some DOM snapshots will show images, have css styling..etc and why some are essentially just the raw HTML with nothing else that was on page.

The bare pages are probably snapshots of pages taken without capturing other stuff from that page (like, with capture disabled, or stuff captured to and then discarded from limbo). Snapshot buttons do not re-fetch page resources.