JavaScript/TypeScript Semantics + Safety Concerns

[benStre]

This is the starting point of a (probably very long) discussion about how to integrate DATEX in a JS library with optimal developer experience while still guaranteeing safety (mostly regarding data privacy + read/write permissions, but also thread safety).

Some general paradigms/goals that we should always keep in mind imo:

1. We should try to expose as little DATEX concepts as possible directly to JS developers - from the view of the JS library, DATEX is only a tool that is used in the background for the most part
2. We should try to keep JavaScript/TypeScript behavior familiar and consistent - The introduction of fundamentally new concepts on a language level that break established developer expectations negatively impacts the developer experience.
3. We should try to keep the core DATEX JS library as compact as possible, providing only a "thin" wrapper around vanilla JavaScript + Web APIs. New paradigms and opinionated APIs should be avoided as far as possible.
4. TypeScript type constraints should not directly effect runtime behavior - this also breaks developer expectations, since TS is only viewed as a compile-time preprocessing tool - in theory, types can be erased completely without affecting runtime behavior. This does not mean that type definitions cannot be used to constrain DATEX types at runtime, e.g. only allowing specific properties for an object (enforced at runtime) or making values read-only. As long as the runtime behavior inferred from TS types reflects the behavior that the TS types dictate anyways, this is fine.

Before discussing on any further points, we should make sure that we can agree on the points above, or modify them until everyone is happy.

Now, let's first mention some shortcomings that make it hard to work safely with vanilla JS behavior in a DATEX application:

• JS does not really have a concept of mutability. TS has readonly modifiers, but the are very limited. Per default, every value is mutable. From a JS perspective, we don't need to think about mutability like in DATEX, but more about mutability inferred from permissions. A pointer from a remote endpoint can be readonly for the current endpoint, but it is still mutable from the perspective of DATEX, since it can be modified by the remote endpoint. This concept, combined with permission filters is something we definitely need to expose somehow to JS developers, violating paradigm 1.
• Primitive values are always shared by value, while all other values (objects) are always shared by reference. We cannot change this behavior. To support primitive and nested reference, we need to introduce a wrapper object. This is a necessary violation of paradigm 3.
• JS does not natively support sharing values between threads with mutexes or similar concepts. We probably need to introduce some kind of mutex wrapper object to allow for thread safe behavior if required - this violates paradigm 3 again.

Now, some other concepts that I would not implement in the DATEX JS library due to violation of our paradigms:

• Creating some "magic" behavior that automatically wraps variables for primitive values in DATEX pointers, making them reactive (like the legacy reactive labels in Svelte (https://svelte.dev/docs/svelte/legacy-reactive-assignments). This contradicts paradigm 2.
• Forcing users to work with raw DATEX types. This might be an option, but we should first of all support TS types to define DATEX types and interfaces (paradigm 3).
• Auto-cloning objects (JS references) when returning them from a remote function. E.g.

  // backend
  export function getUser(id: string): User {
    return users.get(id) // <- implicit clone
  }
  
  // frontend
  await getUser() == await getUser() // false

  This also contradicts paradigm 2 and leads to lots of inconsistent behavior when calling a function either locally or remote, since for normal, local function calls, the object is not cloned, but passed by reference. Nevertheless, this might be a security concern because "live" object references might be unintentionally leaked.
  Regardless of how we do this, I would also try to avoid to introduce some "magic" behavior implied from the TS return type (like only returning a reference when defining the return type as Ref<User>. This violates paradigm 4 and can get very confusing for developers (note: we currently have a similar concept with JUSIX, but only for reactive inputs in JSX contexts, which is a different scenario imo - but we also should think about that in the future))

@jonasstrehle @asbng @janiejestemja @TeeB3utel @Silauras @lennartstoelting @kha1dx
Feel free to comment if you have any remarks, suggestions etc.

[asbng]

I agree with the general paradigms 1–4. As pointed out by you, however, the limitations of JavaScript and TypeScript might force us to contradict these paradigms. I am wondering if it wouldn't be justifiable to require knowledge about the basic concepts of the DATEX protocol from the developer and reflect them on the JS side. Although not on the same OSI layer, DATEX could be compared to HTTP. In JavaScript, HTTP requires just as much preliminary knowledge from developers as DATEX would (e.g. METHODS, headers, even CORS). HTTP is far less well integrated into JavaScript than DATEX would be. I therefore think that we have a certain "budget of deviation" from paradigms 1 and 3, less so from 2 and 4. This tolerance should probably be limited slightly above what current web development frameworks employ (except for transpiler-heavy frameworks).

For now, I would like to only comment on a specific discussion point:

Auto-cloning objects (JS references) when returning them from a remote function.

We must indeed not violate paradigm 4 while still ensuring that data doesn't get inadvertently sent with inappropriate permissions. Without thinking about technical implications, I would indeed like to see a restrictive behavior as a default, especially if we are also aiming at more security-critical applications. However, I am not sure if you would actually have to clone the object for this to work. Would it be possible to have the getUser function return some kind of Derived Pointer from the original? This derived pointer would refer to the original object but mask / override the write permission. This derived pointer could then feature logic to check for equality.

Also, I am not sure whether paradigms 1 and 2 tend to clash. Looking at your "Auto-cloning objects" example, the code is very clean and does not impose any DATEX-specific code on the developer. However, this exported function does not behave at all like one would expect TypeScript imports to work. Even worse, it does perform this "magic" behind the scenes. One possible alternative:

@network function getUser(id: string): User {
    /* @network: this function can be called over the network and will return read-only values */
    return users.get(id); // <- read-only derived pointer
}

I am sure that we can agree on the rule that "explicit is better than implicit" while at the same time we aim to minimize code obstructions. I am unsure what to prefer when.

Note: When I'm mentioning that I am unsure about an approach, this does not mean that I am against it. It just means that I am seeking for more perspectives.

[benStre]

Thanks for your input --
Yeah I forget to mention this in the example - I definitely also think that we need something like a @network decorator (or @public or similar) that must be applied for functions and values to even be accessible from remote endpoints.
I would also be fine if we change the default permissions to read-only for the returned value of such a function, that would not be a problem, and we need to modify the type signature for remote function anyways to convert them from sync to async functions.

But the main discussion point here is not whether or not the pointer should be a writable or a "derived read-only" pointer per default, but whether it should be a static snapshot of the current pointer (what i called "cloning"), or a pointer reference that is synchronized per default and has the same object identity for each function call.

[asbng]

Ah, I see. I would opt for returning a pointer reference (not a snapshot) as this is much more consistent with JavaScript/TypeScript (paradigm 2). Do you have a scenario where, supposing the object itself and all referenced objects are read-only, there is a risk of exposing information unintentionally, even when you are aware of the "live" / synchronized nature of @network / @public / … functions?

[benStre]

Imo, this can only happen if you really don't understand what you are doing. You must just be aware that everything you return from a function is a "live" reference, like in normal JS.
As long as you only return objects that are created within the scope of the function and not referenced anywhere else, this is not a problem anyway. Even if you have a global variable that you reassign in a weird way inside a function and return it afterwards, later reassignments to the variable will not be synchronized, since we only synchronize the object that was assigned at the point of time that the function returns - theres no strong reactive binding to variables.
If you are using a Ref wrapper and reassign this value, it would of course potentially leak data, but this is a user fault imo:

const dataToReturn = $(null);

@public
export function getSecretDataForUser() {
   // .. do some stuff
   dataToReturn.val = secretData;
   return dataToReturn // this Ref will be kept synchronized with endpoints that previously called the function
}

The example is a bit constructed, maybe we can get some better examples where something like this might happen in actual code @jonasstrehle ?