fix snippet completion incorrect filtering

[vezwork]

Problem

Snippets were being filtered from completions because of a position miscalculation.

The itemFrom function was not correct, it was doing something clever that it is not quite right.

itemFrom should calculate how many characters into the document the start of a completion is. It did this by calculating the length of the completion and subtracting that from the text cursor's position. The way it was calculating the length of the completion was not correct. The completion gives its positions in the context of the cvContext.code, which is different from the positions in the actual document; For one thing, the line containing the language string (like {r}) is removed from cvContext.code. So, we can't use the positions the completion gives us directly. That is why itemFrom was cleverly not using the positions, but instead using the positions to calculate the length of the completion.

But, for example, if you are typing libr, a completion for the library snippet will say that its insertion character position is from 0 to 7. What's up with that? For whatever reason, since your cursor is at the end of a line with 4 characters, the completion pretends that it can replace 3 more non-existent characters in the rest of the line. If there were any characters in the line after libr, even just whitespace like libr , then the completion would instead say that its insertion character position is from 0 to 4.

This PR fixes the issue by snapping the ending position of a completion to the end of its line in cvContext.code.

After I fixed this issue, I was able to easily insert the library snippet in R code, and this uncovered a bug in snippets in codemirror autocomplete that I reported here codemirror/dev#1607.

[vezwork]

@marijnh fixed the codemirror/autocomplete issue and made a release of it. I spent a couple hours this morning on this PR's branch attempting to upgrade our codemirror/autocomplete version. I was not successful (facing Quarto monorepo problems that I don't understand).

I feel that this PR is blocked on getting this autocomplete fix in. If we are going to fix it so that snippet completions show up properly, snippets should work.

I will have to get help figuring out how to upgrade the package.

[vezwork]

Here's a description of the issues I was running into while upgrading: I used yarn upgrade-interactive --latest in the repo root to upgrade various codemirror packages to their latest versions: view, state, autocomplete and maybe another one or two. This changed packages/editor-codemirror/package.json to reflect the updated versions and updated the packages in packages/editor-codemirror/node_modules. The problem is that the root node_modules still contains old versions of these packages, and that causes type and build errors in files inside packages/editor-codemirror, such as completions.ts.

I tried deleting the root level yarn.lock and node_modules and doing a fresh yarn install, and this fixes the codemirror package issues, but seems to cause many other issues. The first issue I encountered going down this path was type issues with markdown-it-attrs. I commented code related to that issue out and ran into another issue. I am afraid.

I do not understand why the root node_modules still contains old versions of these packages. These packages are not referenced in any other places other than packages/editor-codemirror/package.json. Why are these packages installed in both the root node_modules and editor-codemirror node_modules? What is going on??

[marijnh]

Stop using yarn. It's a broken tool that will duplicate dependencies all over the place when you upgrade something. You can blow away your package lock on every update to work around that, or switch back to npm, which is a more reliable tool.

[cscheid]

I imagine part of what's happening here is that this is a https://turborepo.com/ monorepo, and yarn might be getting extra confused in the process?

(I would happily switch us all away to better tooling if the benefits are there, but I would really like to not to have to switch again in a year. So I think adding some friction here and thinking half a dozen times before pulling the trigger is worthwhile.)

[vezwork]

I hadn't considered that yarn has problems, I thought I was doing something wrong. Here's a yarn issue that seems to describe a related, if not the same, problem as the problem I was running into.

I am running an old yarn version. I suppose the smallest thing to try for now is to upgrade yarn (two major versions, I'm on 1.22.22 and that thread ends in a claim that this problem is fixed in 3.1.1) and see if that works and helps.

[vezwork]

I attempted to update yarn to the latest version. There were many many new warnings during the yarn install. Unfortunately, things break in a similar way to when I deleted the yarn.lock.

quarto-vscode-markdownit:dev: src/index.ts:17:34 - error TS7016: Could not find a declaration file for module 'markdown-it'. '/Users/posel/.yarn/berry/cache/markdown-it-npm-12.3.2-6c66b716e8-10c0.zip/node_modules/markdown-it/index.js' implicitly has an 'any' type.
quarto-vscode-markdownit:dev:   If the 'markdown-it' package actually exposes this module, consider sending a pull request to amend 'https://github.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/markdown-it'
quarto-vscode-markdownit:dev: 
quarto-vscode-markdownit:dev: 17 import type * as MarkdownIt from 'markdown-it';
quarto-vscode-markdownit:dev:                                     ~~~~~~~~~~~~~
quarto-vscode-markdownit:dev: 
quarto-vscode-markdownit:dev: src/index.ts:20:24 - error TS7016: Could not find a declaration file for module 'markdown-it-attrs'. '/Users/posel/Desktop/quarto3/.yarn/__virtual__/markdown-it-attrs-virtual-1de3767d45/3/.yarn/berry/cache/markdown-it-attrs-npm-4.1.6-7d96802127-10c0.zip/node_modules/markdown-it-attrs/index.js' implicitly has an 'any' type.
quarto-vscode-markdownit:dev:   Try `npm i --save-dev @types/markdown-it-attrs` if it exists or add a new declaration (.d.ts) file containing `declare module 'markdown-it-attrs';`
quarto-vscode-markdownit:dev: 
quarto-vscode-markdownit:dev: 20 import attrPlugin from "markdown-it-attrs";
quarto-vscode-markdownit:dev:                           ~~~~~~~~~~~~~~~~~~~
quarto-vscode-markdownit:dev: 
quarto-vscode-markdownit:dev: ../../packages/core/src/jupyter/options.ts:16:25 - error TS2307: Cannot find module 'js-yaml' or its corresponding type declarations.
quarto-vscode-markdownit:dev: 
quarto-vscode-markdownit:dev: 16 import * as jsYaml from "js-yaml";
quarto-vscode-markdownit:dev:                            ~~~~~~~~~
quarto-vscode-markdownit:dev: 
quarto-vscode-markdownit:dev: ../../packages/core/src/markdownit/gridtables/common/markdown-it/ParseTable.ts:6:21 - error TS2307: Cannot find module 'wcwidth' or its corresponding type declarations.
quarto-vscode-markdownit:dev: 
quarto-vscode-markdownit:dev: 6 import wcwidth from "wcwidth";
quarto-vscode-markdownit:dev:                       ~~~~~~~~~
quarto-vscode-markdownit:dev: 
quarto-vscode-markdownit:dev: ../../packages/core/src/markdownit/mermaid/index.ts:3:21 - error TS2307: Cannot find module 'mermaid' or its corresponding type declarations.
quarto-vscode-markdownit:dev: 
quarto-vscode-markdownit:dev: 3 import Mermaid from "mermaid";
quarto-vscode-markdownit:dev:                       ~~~~~~~~~
quarto-vscode-markdownit:dev: 
quarto-vscode-markdownit:dev: ../../packages/core/src/markdownit/yaml.ts:20:23 - error TS2307: Cannot find module 'js-yaml' or its corresponding type declarations.
quarto-vscode-markdownit:dev: 
quarto-vscode-markdownit:dev: 20 import * as yaml from "js-yaml";
quarto-vscode-markdownit:dev:                          ~~~~~~~~~
quarto-vscode-markdownit:dev: 
quarto-vscode-markdownit:dev: ../../packages/core/src/yaml.ts:16:25 - error TS2307: Cannot find module 'js-yaml' or its corresponding type declarations.
quarto-vscode-markdownit:dev: 
quarto-vscode-markdownit:dev: 16 import * as jsYaml from "js-yaml";
quarto-vscode-markdownit:dev:                            ~~~~~~~~~
quarto-vscode-markdownit:dev: 
quarto-vscode-markdownit:dev: 
quarto-vscode-markdownit:dev: Found 7 errors in 6 files.
quarto-vscode-markdownit:dev: 
quarto-vscode-markdownit:dev: Errors  Files
quarto-vscode-markdownit:dev:      2  src/index.ts:17
quarto-vscode-markdownit:dev:      1  ../../packages/core/src/jupyter/options.ts:16
quarto-vscode-markdownit:dev:      1  ../../packages/core/src/markdownit/gridtables/common/markdown-it/ParseTable.ts:6
quarto-vscode-markdownit:dev:      1  ../../packages/core/src/markdownit/mermaid/index.ts:3
quarto-vscode-markdownit:dev:      1  ../../packages/core/src/markdownit/yaml.ts:20
quarto-vscode-markdownit:dev:      1  ../../packages/core/src/yaml.ts:16
quarto-vscode-markdownit:dev: ERROR: command finished with error: command (/Users/posel/Desktop/quarto3/apps/vscode-markdownit) /private/var/folders/pv/pvncj5rd55s124hr58by6nqc0000gp/T/xfs-66d0cba1/yarn run dev exited (2)

After I hackily sidestepped these errors, I ran into another:

quarto:dev: Error: File 'tsconfig/base.json' not found.

There are red underlines in every file I looked at. Underlines on the return types of async functions. Underlines on so many import paths. Underlines.

[vezwork]

@cscheid It seems that we would have to embark on a proper migration of our monorepo to a new version of yarn to deal with these issues. It scares me though. I have little sense for how much work that would be and I can't think of a way to make incremental progress.

[vezwork]

Also, the yarn upgrade causes the number of changes in this PR to be wild. It should be really be in its own PR... Though, I'm hesitant to start such a PR until we decide we actually want to invest into a proper migration.

[vezwork]

I checked out the Positron repo to see how they are managing their dependencies and they are also using an old version of yarn...

An alternative to thing to look into rather than upgrading yarn: I heard there is something we can maybe do with the old version of yarn to do with "no hoisting" that could possibly help.