What is the API story for this new codebase?

[DanielRosenwasser]

How will the API work? Will developers have a canonical JavaScript-based API for integrating with this new version of TypeScript?

---

While we are porting most of the existing TypeScript compiler and language service, that does not necessarily mean that all APIs will be ported over. Because of the challenges between language runtime interoperability, API consumers will typically not communicate within the same process. Instead, we expect our API to leverage a message-passing scheme, typically over an IPC layer. Because this sort of bridging is not "free", exposing all functionality will be impractical. We expect to have a more curated API that is informed by critical use-cases (e.g. linting, transforms, resolution behavior, language service embedding, etc.).

We knew that providing a solid API would be a big challenge, so as soon as we started exploring the TypeScript port, we investigated the possibilities here. Beyond how capable the API would be, we asked ourselves whether the sorts of use-cases would be constrained by the performance of IPC solutions. More specifically, even if we could come up with a set of APIs that did what users wanted, would the throughput of requests be a limiting factor? Would the new TypeScript be fast enough to offset the cost of serialization and data transfer? Would the chattiness of API usage overwhelm the wins of having a much faster compiler?

We've become increasingly confident and optimistic in answers around the IPC performance. @zkat has built a Node native module to use synchronous communication over standard I/O between external processes. Building on that, @andrewbranch has experimented with exposing an API server entrypoint to our compiler, along with a JavaScript client that "speaks" to the server over that communication layer. What we've found is fairly promising - while IPC overhead is not entirely negligible, it is small enough. We also can imagine opportunities to optimize, use other underlying IPC strategies, and provide batch-style APIs to minimize call overhead.

As our experiments solidify, we will post more concrete details on our plans, and what the API will look like.

[SLaks]

What are your IPC plans for WebAssembly in the browser?

[jakebailey]

I don't think we have a concrete plan (just prototypes), but we expect to be able to at least do the same IPC via mechanisms like WASI (what I do in https://jakebailey.dev/esbuild-playground, not to pump this side project). There's also the possibility to use plain messaging and only offer async in the browser, or some sort of async worker with SharedArrayBuffer based communication; the latter is already pretty well thought out for vscode.dev/github.dev, so there's theoretically some reusability there.

[rictic]

Have you considered WASM as a solution to providing a cross-language API?

I haven't used it for this purpose, but it seems like there could be a good match in requirements. WASM modules can pass typed data to one another, call one another's functions, etc.

[Meligy]

Actually, this has its own discussion here:
#514

[jakebailey]

People really need to provide their test code. I'm pretty sure the results are incorrect. Notably, threading needs to be disabled in WASM but we didn't do that in the repo yet.

[marcus-sa]

Wouldn't https://extism.org be ideal here?

[jakebailey]

Why is exitism ideal? Which problem are you specifically trying to solve? Plugins?

To be clear, I do not think it's the direction we want to take to allow embedding languages into tsc. It could be technically feasible, but we'd now be a "runtime" and have to go through a lot of compliance work.....

[marcus-sa]

@jakebailey it could allow us to hook into the compiler similarly to how its currently being done with the transformer API.

[dummdidumm]

How's the story for TypeScript plugin developers and people integrating with the language service? More specifically, will method patching still work? The official way to create a plugin is to patch methods. And across such use cases there are probably dozens of cases where people reach into TypeScript semi-private internals to patch other methods (so that things can be intercepted for example) - will that also still work? If not, how do you envisage plugin authors to migrate away from doing that?

[dummdidumm]

Yes that is probably my biggest concern - but since this all needs to be redesigned anyway, it sounds like you are also open to add proper hooks into these internals where it's needed. I think the Angular team, the Volar.js team and the Svelte team (which I'm a part of) are all doing lots of stuff with both the language service API and the plugin API, and are probably all very interested to talk details once the internals have stabilized a bit.

[jedwards1211]

@andrewbranch I'm bummed that it won't be possible to use ts-morph anymore to do codemods...people are mostly talking about more basic AST walking, but is type information going to be part of the API that operates over IPC?

[jakebailey]

We love ts-morph; it is explicitly an anti-goal to prevent ts-morph from working altogether. Type information is effectively the reason why we need to have an API at all 😄

[jedwards1211]

Okay whew!

[itsdouges]

WE'RE SO BACK

[saw]

I've used the tsc api for various codegen purposes, will you be exposing an API to walk/visit the AST tree as was available in the js tsc api?

[Stanzilla]

Will error reporting change? For a long time, TS errors have been a heated topic and there have been efforts by the community (https://github.com/yoavbls/pretty-ts-errors) to make it a little better. Will solutions like Pretty TS still work in the new world?

[lucsoft]

Are there plans about instead of IPC there would be something like FFI?

[jakebailey]

FFI with Go is problematic because Go assumes it's the only instance of Go in a process, but people can and do import multiple versions of TypeScript (for example, to test DefinitelyTyped); while we did have some success loading multiple instances in the same process, it's not something the Go team supports.

[jedwards1211]

Is there really no way to bootstrap go's global state once and then instantiate multiple things within that same global state from separate NAPI calls?

[jakebailey]

Loading multiple runtimes into the same process means they are all competing to register signal handlers, do preemption, manage certain process-specific limits, etc. It might work if all are built with the same exact version of Go, since things like signal handlers would still grab data at the right offsets into things like TLS, but it's definitely not a guaranteed.

See also: golang/go#65050 (comment)

[jedwards1211]

What I mean is, I'm surprised it's not possible to only load a single runtime on the first FFI call, and have subsequent FFI calls instantiate structs into that same runtime...

Which from skimming that thread, sounds like it is possible using plugins instead of c-shared libraries...but I guess there's still a problem if different versions of typescript-go are built for different versions of the Go runtime?

[jakebailey]

The Go runtime is entirely different between Go releases, and potentially Go patches. I don't even think it's guaranteed that it'll work if they happen to match.

Plugins are not an option; that's for loading Go code into Go code, not Go code into other things. That and Go plugins do not support Windows.

[puchm]

Not a question, but an observation: Right now, lots of tools which depend on types (TypeScript Eslint, Typia, etc) run their own Lexer, parser and type checker. It is not uncommon to have separate type checkers running in the ESLint process, in a development server and in the TypeScript language service. This leads to doing a lot of work twice or more and can also lead to inconsistencies if the tooling does some weird stuff.

With an IPC layer it could become very feasible to have all of these hook into one tsgo process and have one source of truth for type information, meaning that the development experience when using these kinds of tools could get a lot smoother. Maybe this is something that can be considered when designing the IPC layer - i.e. to allow having one tsgo "server" and multiple clients consuming it's API.

[thebanjomatic]

Is now maybe the time to also re-consider adopting semver from the API perspective?

I could usually care less about breaking changes from the language and type checking perspective, but I would love it if from the tooling perspective we could get some sort of guarantees of what versions of the typescript api are supported and compatible.

I will acknowledge that in the past this was a harder sell because of the nature of javascript and how the entire AST and compiler internals are essentially all public api. As a result, any change or internal refactoring could therefore be considered a breaking change, and thus nothing was. If the team is being more intentional about what is exposed across the API boundary, it can be more reasonable.

My ideal would be something like:
major: release contains breaking changes to the api / lsp
minor: release contains new language features, some of which may introduce new errors
patch: mainly bug fixes

Note: changes to the language that add new features may change the resulting AST that is returned from the API. This should not be considered a breaking change to the API, but maybe things like removing documented properties from existing nodes, might be?

That way, tooling providers can set their expectations of supported typescript releases based on the ^major to state which version of the API their code is expecting to hit, and applications that are concerned about not getting new build errors can just use ~major.minor to pin things at the language level.

[RyanCavanaugh]

Absolutely; this is going to be mandatory going forward since it won't be apparent to API consumers what version they're talking to. A better versioning story is one of the reasons we want to make a more intentional design here. That may or may not correspond to the "language version", but we're going to have some concept of an "API version" that probably revs on a different axis.

[Brooooooklyn]

Hey there, I'm developing an experimental Node.js binding here: https://github.com/Brooooooklyn/typescript-go-napi/tree/main/rust

The basic idea is to compile typescript-go into a static link library, and then wrap it into a JavaScript API in Rust using NAPI-RS.

I have now completed two basic APIs, among which transform is like this:

import { transform } from '@typescript-go/api'

transform('const a: number = 1;', '/absolute/path/a.ts')

// 'const a = 1;\n'

[jakebailey]

Do note #455 (reply in thread)

This works but is likely not going to be safe for a number of important users.

[jakebailey]

I do actually have the code to do NAPI directly with Go, it'd be awesome if we could use it 😞

[Brooooooklyn]

This works but is likely not going to be safe for a number of important users.

Perhaps my implementation can be provided to users who don't need to load multiple different ts-go instances 👀

[jedwards1211]

has built a Node native module to use synchronous communication over standard I/O between external processes

How future-proof is this module? node-fibers used to be the defacto way of doing blocking calls, but it was eventually abandoned after significant V8 changes. I saw that coming years before it happened, everything in the ecosystem has gone toward async architecture.

Webdriverio badly wanted to provide a sync API for writing tests, and relied on node-fibers for this, but they eventually had to give up.

[jedwards1211]

@andrewbranch @jakebailey thoughts on how futureproof libsyncrpc is?

I guess the fact that it's based upon NAPI is better than node-fibers, but it still seems iffy

[jakebailey]

I don't really understand the question. This surely isn't worse than anyone else using napi-rs, and we're only using it to shuffle bytes around without hitting the event loop, not even constructing any objects or anything. It's possible that this could be implemented without NAPI by using plain reads/writes to file descriptors in JS, though that hasn't been tested in the newer IPC implementation.

[jedwards1211]

Okay yeah I hadn't realized that if native code simply makes blocking calls that will block the source JS function call (and event loop). node-fibers had to do something complicated to enable function calls to block local function execution while allowing the event loop and other functions to continue executing, and that magic apparently depended on pretty specific V8 APIs. I mistakenly assumed that anything else that would make a JS function call block would be similar

[HerringtonDarkholme]

Adding my two cents here, if API is not possible for major languages (mainly JS and probably Rust for SWC/OXC), a static type info snapshot can also be helpful. (though it is not possible to do transformation)

Related issues #504

[joshring]

Would it be possible to use tsc-go as a library from Go and hook into it that way? As opposed to plugins and IPC etc?

[jakebailey]

#481

[pjeby]

The TypeDoc project uses TypeScript compiler APIs (including watch-related APIs) to generate documentation from source projects (via JSDoc as well as TypeScript type info) and to respond to changes in the code base and non-code documentation files. Will the same or substantially similar watch APIs be available from JS? (i.e. including the ability to add arbitrary watches as well as incremental-compile-on-change)

[andrewbranch]

The beginnings of the IPC API are now in a PR at #711.