Adherence to XDG Base Specification

[Daeraxa]

Started from this Discord conversation

So I'm almost certain this question will come up eventually so thought it best to pre-empt it seeing as it has frequently come up on the Joplin project.

Basically it seems that Pulsar (by way of Atom) has its Config folder set to ~/.atom on Linux (and macOS).
This by itself isn't really an issue as such but there is mounting pressure for projects to correctly adhere to the XDG Base Directory Specification rather than forcing configuration files and application data to write to a specific location chosen by the developers.

Pulsar/Atom can already be made to adhere to it by exporting ATOM_HOME: export ATOM_HOME="$XDG_CONFIG_HOME"/atom (and the same for apm).

There are a few benefits to supporting the XDG spec:

1. People won't comment all the time and raise issues asking for it
2. It gives people more control over their home directories
3. It better supports some native features in the Appimage format
4. It better supports the sandboxing inherent to the Snap and Flatpak formats (see the Flathub script for modifications done to make it work correctly for Flatpak)

We are at a point, being at the very start of the project really, where we could look at implementing this with as little impact on users as possible.

There are potentially a few things we would need to look out for, such as any quirks or clashes with the portable mode.

One thing of note is that Electron by default already does seem to support $XDG_CONFIG_HOME as this is where it will write the Atom electron cache (~/.config/Atom) and it does change location depending on the location of your $XDG_CONFIG_HOME.

The other question is, what is actually the correct place. You will note that both the Arch wiki page on the topic recommends setting ATOM_HOME to $XDG_DATA_HOME rather than $XDG_CONFIG_HOME as does the Flatpak script above.
So either we would need consensus as to which is correct or we look at splitting the current .pulsar directory between them to support both data (packages in particular) and config files in different places.

Another alternative is that we put the ~/.pulsar inside the existing $XDG_CONFIG_HOME/Pulsar Electron cache folder - which is exactly what VSCode seems to do. Then we only have one folder to worry about and not two.

There are a few more resources if you want to have a look at other info or conversations on the topic.

• XDG Base Directory Specification
• DistroTube video on the topic
• Arch wiki page on the topic
• My own post to the Joplin forums on this exact same topic

[confused-Techie]

While, admittedly, I don't have much skin in this game. Of course I'd vote we use this standard in some way, since like you said allowing users better control over their systems should be a priority. But otherwise it likely will be much simpler to use a single directory. Because then we just worry about setting ATOM_HOME instead of having to create ATOM_CONFIG_HOME and ATOM_DATA_HOME and finding every single relevant place in the code to change it.

And personally I'd vote that to me, the config directory makes since. Since really no matter what that folder will contain a config. It won't always contain packages, unless the user installs them. So to me that seems like the right choice

[magenta404]

I think it is important to also part out anything that is considered cache into XDG_CACHE_HOME so the OS can clear it if needed and it will be excluded from backups. Also, some people version control XDG_CONFIG_HOME so parting out Pulsar between XDG_CONFIG_HOME and XDG_DATA_HOME is also valuable but not as important.

[Daeraxa]

I wonder if we could/should at look at implementing https://github.com/folder/xdg. It deals with all major OS paths, not just XDG on Linux.

[confused-Techie]

This looks really cool! I'd honestly be very on board with utilizing something like this, the only potential issue is taking a quick look at it, seems that it will list paths with directories and such that do not yet exist. Meaning we will have to handle that reality elsewhere in the code. But generally, this looks like it could be extremely useful.

A note I'd like to make though, is ideally we also allow these locations to be totally configurable. Plus we will still have to maintain some logic here, an easy example is the Pulsar exe looking nearby for a config folder when running in portable mode, but this seems like a great way to simplify our lives with using the right locations, while not having to do this research ourselves

[Daeraxa]

I found it via https://dotfiles-matter.click/ (to give credit). There are definitely a few things we would have to work out above my simplistic PoC PR which pretty much just changes the initial location of ATOM_HOME. Also no idea if the Windows stuff looks like it makes any sense or not, I still haven't fully got my head around what is correct or not in Windows land.

[faokryn]

Hello! I'm curious if there's been any more progress/discussion on making Pulsar XDG Base Directory Specification compliant, or at least exposing configuration to make it optional? I recently switched over from VSCode, and I found this discussion while looking for a way to clean the .pulsar directory from my home dir; if there's a better or more recent place for discussion, please let me know 🙂
For the time being, dropping export ATOM_HOME="$XDG_DATA_HOME/pulsar" in my shell's login profile seems to do the trick for at least cleaning up $HOME, but I'd love to be able to be more spec-compliant!

One thing to note on this topic:

The other question is, what is actually the correct place. You will note that both the Arch wiki page on the topic recommends setting ATOM_HOME to $XDG_DATA_HOME rather than $XDG_CONFIG_HOME as does the Flatpak script above.
So either we would need consensus as to which is correct or we look at splitting the current .pulsar directory between them to support both data (packages in particular) and config files in different places.

But otherwise it likely will be much simpler to use a single directory. Because then we just worry about setting ATOM_HOME instead of having to create ATOM_CONFIG_HOME and ATOM_DATA_HOME and finding every single relevant place in the code to change it.

If your goal is to be compliant with the XDG spec, then you must split the files into their relevant directories. To do otherwise would not be following the spec, which defines certain kinds of contents for each base dir, and as magenta404 pointed out, some people manage their XDG base dirs differently depending on the type of data expected to be there, such as inclusion or exclusion from backups/version control, placing them on different file systems (e.g. using a tmpfs mount for $XDG_CACHE_HOME), etc. If I had to guess, this is part of the reason that several sources recommend locating the monolith-folder in $XDG_DATA_HOME over $XDG_CONFIG_HOME: if you put it in the latter you're more likely to be version controlling binary files or excluding the larger files expected in $XDG_DATA_HOME from a file system you've configured for that purpose. On that note...

Another alternative is that we put the ~/.pulsar inside the existing $XDG_CONFIG_HOME/Pulsar Electron cache folder - which is exactly what VSCode seems to do. Then we only have one folder to worry about and not two.

Electron/Chromium's default of placing cache files in $XDG_CONFIG_HOME is one of the most common sources of XDG spec violation and there's even a utility for moving and symlinking Electron cache files. Ideally, an initiative to make Pulsar XDG compliant would include breaking the contents of the $XDG_CONFIG_HOME/Pulsar directory into the appropriate XDG base dirs (probably primarily $XDG_CACHE_HOME).

[Daeraxa]

I don't think much has been done since then unfortunately. I had been making a draft PR as a proof of concept (pulsar-edit/pulsar#390) but I'm not somebody who has the level of programming skill or knowledge of the overall app base to make this kind of change without support. The tiny thing I changed here works but adding or editing specs and tests, making sure we find any hard coded references or breaking existing packages etc. etc.

Back when this was written we were probably being more conservative with our the scope of our changes - the focus was on the big problems like supporting infrastructure (website, package backend etc.) and moving to modern and supported versions of node and electron so I think this could obtain more traction now if we were to push for it.

And if we were to do this it makes sense to do it properly and not just half-arse it by throwing the whole lot into one place - for example if we take what is in my .pulsar dir this is how we might actually need to split it (I emphasise "my" because I've had this for a few years and honestly there some stuff that I might have added or there might be relics of stuff that no longer exists...):

Item	$XDG_?_HOME
.apm	DATA
.node-gyp	DATA
blob-store	STATE
compile-cache	CACHE
crashdumps	STATE
packages	DATA
recovery	DATA
storage	STATE
.pulsar-socket-secret-dae-1.132.0-dev	CONFIG? STATE?
config.cson	CONFIG
github.cson	CONFIG
init.js	CONFIG
keymap.cson	CONFIG
nohup.out	STATE
session-store.db	DATA
session-store.db-shm	DATA
session-store.db-wal	DATA
snippets.cson	CONFIG
styles.less	CONFIG

That is just a very quick overview having just scanned through this quickly and seeing what each one feels like it should be classed as.

It does seem that there has at least been some development in this area in Electron which should make things easier to implement without hard coding the $XDG_CONFIG_HOME like I did in my PR. In the main-process we do have access to the app.getPath() and app.setPath() functions which have been somewhat extended in Electron versions since the one we were on when this topic was raised.

It seems the only ones I can find that we actively use are:

• home: app.getPath('home') -> ~ (used to set the "Atom Home")
• crashDumps: app.setPath('crashDumps', path.resolve(process.env.ATOM_HOME, 'crashdumps')) -> `~/.pulsar/crashdumps)
• userData: app.setPath('userData', electronUserDataPath) -> ~/.config/Pulsar

Newer electron versions also have sessionData which by default still links to userData which means as we resolve down the default chain we still end up with $XDG_CONFIG_HOME/Pulsar rather than recommending a suitable alternative location but it is at least something.

userData is the annoying one because we can only influence some of it due to Chromium/Electron. It states in the docs to specifically not use userData directly to avoid clashes:

It is recommended to store app-specific files within a subdirectory of userData (e.g., path.join(app.getPath('userData'), 'my-app-data')) rather than directly in userData itself, to avoid naming conflicts with Chromium's own subdirectories (such as Cache, GPUCache, and Local Storage).

Obviously the electron cache is the big blocker we don't have control over. Some of it should be STATE and some of it is CACHE but that isn't something so easy (or possible?) to solve downstream and whilst symlinking it does solve some issues it doesn't fix the "clutter" problem. Unfortunately it doesn't seem there is much appetite to fix it upstream by the maintainers and community members who have created PRs to solve this keep closing them out of frustration with the scope of requested changes. At the very least they added a stale-exempt label to the issue tracking all this which is now almost a decade old...

Ultimately I don't know the "right" answer here, what we would probably need to is to:

• Not alter any file or data locations for existing installations
• Implement a new "opt-in" setting for extended user testing (a pattern we often use - modern tree-sitter, legacy data store etc.)
  • Later on this would become "opt-out"
  • We would need to agree whether we ever deprecate the old paths entirely. We would still need a way of reading from and writing to the old config format but we could decide to stop creating it on new installs.

Now there is one weird exception to this, at least with the official packages, in that the AppImage can actually create its own folders when using portable mode. If you make a folder in the same root as the AppImage with the same name as the AppImage but plus .home then it will add a .pulsar as well as a .config/Pulsar which makes it act entirely differently as it sets $HOME to that root when launching it. You can also use .config instead and it will set $XDG_CONFIG_HOME which will create the .config/Pulsar folder but will use the existing system .pulsar folder. It isn't a proper answer as it is using functionality of a package format that Pulsar hasn't intentionally been modified to use.

I'm adding this to my ever growing list of things to look at, I've just come back from a long hiatus from FOSS in general.

[Daeraxa]

I don't think much has been done since then unfortunately. I had been making a draft PR as a proof of concept (pulsar-edit/pulsar#390) but I'm not somebody who has the level of programming skill or knowledge of the overall app base to make this kind of change without support. The tiny thing I changed here works but adding or editing specs and tests, making sure we find any hard coded references or breaking existing packages etc. etc.

Back when this was written we were probably being more conservative with our the scope of our changes - the focus was on the big problems like supporting infrastructure (website, package backend etc.) and moving to modern and supported versions of node and electron so I think this could obtain more traction now if we were to push for it.

And if we were to do this it makes sense to do it properly and not just half-arse it by throwing the whole lot into one place - for example if we take what is in my .pulsar dir this is how we might actually need to split it (I emphasise "my" because I've had this for a few years and honestly there some stuff that I might have added or there might be relics of stuff that no longer exists...):

Item	$XDG_?_HOME
.apm	DATA
.node-gyp	DATA
blob-store	STATE
compile-cache	CACHE
crashdumps	STATE
packages	DATA
recovery	DATA
storage	STATE
.pulsar-socket-secret-dae-1.132.0-dev	CONFIG? STATE?
config.cson	CONFIG
github.cson	CONFIG
init.js	CONFIG
keymap.cson	CONFIG
nohup.out	STATE
session-store.db	DATA
session-store.db-shm	DATA
session-store.db-wal	DATA
snippets.cson	CONFIG
styles.less	CONFIG

That is just a very quick overview having just scanned through this quickly and seeing what each one feels like it should be classed as.

It does seem that there has at least been some development in this area in Electron which should make things easier to implement without hard coding the $XDG_CONFIG_HOME like I did in my PR. In the main-process we do have access to the app.getPath() and app.setPath() functions which have been somewhat extended in Electron versions since the one we were on when this topic was raised.

It seems the only ones I can find that we actively use are:

• home: app.getPath('home') -> ~ (used to set the "Atom Home")
• crashDumps: app.setPath('crashDumps', path.resolve(process.env.ATOM_HOME, 'crashdumps')) -> `~/.pulsar/crashdumps)
• userData: app.setPath('userData', electronUserDataPath) -> ~/.config/Pulsar

Newer electron versions also have sessionData which by default still links to userData which means as we resolve down the default chain we still end up with $XDG_CONFIG_HOME/Pulsar rather than recommending a suitable alternative location but it is at least something.

userData is the annoying one because we can only influence some of it due to Chromium/Electron. It states in the docs to specifically not use userData directly to avoid clashes:

It is recommended to store app-specific files within a subdirectory of userData (e.g., path.join(app.getPath('userData'), 'my-app-data')) rather than directly in userData itself, to avoid naming conflicts with Chromium's own subdirectories (such as Cache, GPUCache, and Local Storage).

Obviously the electron cache is the big blocker we don't have control over. Some of it should be STATE and some of it is CACHE but that isn't something so easy (or possible?) to solve downstream and whilst symlinking it does solve some issues it doesn't fix the "clutter" problem. Unfortunately it doesn't seem there is much appetite to fix it upstream by the maintainers and community members who have created PRs to solve this keep closing them out of frustration with the scope of requested changes. At the very least they added a stale-exempt label to the issue tracking all this which is now almost a decade old...

Ultimately I don't know the "right" answer here, what we would probably need to is to:

• Not alter any file or data locations for existing installations
• Implement a new "opt-in" setting for extended user testing (a pattern we often use - modern tree-sitter, legacy data store etc.)
  • Later on this would become "opt-out"
  • We would need to agree whether we ever deprecate the old paths entirely. We would still need a way of reading from and writing to the old config format but we could decide to stop creating it on new installs.

Now there is one weird exception to this, at least with the official packages, in that the AppImage can actually create its own folders when using portable mode. If you make a folder in the same root as the AppImage with the same name as the AppImage but plus .home then it will add a .pulsar as well as a .config/Pulsar which makes it act entirely differently as it sets $HOME to that root when launching it. You can also use .config instead and it will set $XDG_CONFIG_HOME which will create the .config/Pulsar folder but will use the existing system .pulsar folder. It isn't a proper answer as it is using functionality of a package format that Pulsar hasn't intentionally been modified to use.

I'm adding this to my ever growing list of things to look at, I've just come back from a long hiatus from FOSS in general.

[faokryn]

Thanks for the reply!

I had been making a draft PR as a proof of concept (pulsar-edit/pulsar#390)

Nice! I took a look at this proof-of-concept and it at least gives a place to start. I might have to set up a dev version of Pulsar in a VM and take a pass at it myself 🙂

if we take what is in my .pulsar dir this is how we might actually need to split it ...

Looks like a reasonable list to start with to me! Only thing that jumps out to me is .pulsar-socket-secret-dae-1.132.0-dev, which, seeing "secret" in the filename, I'd want to investigate whether it's a permanent secret that needs to be persisted between sessions, in which case $XDG_STATE_HOME is probably the best place, or if it's a more ephemeral session token, in which case the best spot might actually be $XDG_RUNTIME_DIR, which I believe defaults to a tmpfs or ramfs file system mounted at /run/user/$(id -u) on most modern distros- get some security points by keeping secrets off of permanent storage 😉

Buuuut I'm definitely getting ahead of myself, and discussing exactly where each file should go can probably wait until after IDing the source of each of those files' paths, haha

Obviously the electron cache is the big blocker we don't have control over. Some of it should be STATE and some of it is CACHE but that isn't something so easy (or possible?) to solve downstream

True, but I probably should have clarified that when I said "an initiative to make Pulsar XDG compliant would include breaking the contents of the $XDG_CONFIG_HOME/Pulsar directory into the appropriate XDG base dirs" that I meant more keeping it in mind long term for the project. An initiative like this can certainly have more than one stable deliverable before it's "complete", and I wouldn't want the pursuit of perfection to impede progress. It doesn't need to be "fully compliant or nothing": "more compliant" or even "providing options for users to be more compliant" would be a win; I was happy just to find out about $ATOM_HOME from your post and get the files into any XDG dir 😂 I think starting by dealing with the files in $HOME/.pulsar/$ATOM_HOME would be awesome, and those in $XDG_CONFIG_HOME/Pulsar can wait (and hey, maybe by the time that's done there'll be some movement upstream from Electron/Chromium... one can hope 🤞)

Ultimately I don't know the "right" answer here, what we would probably need to is to:

• Not alter any file or data locations for existing installations
• Implement a new "opt-in" setting for extended user testing (a pattern we often use - modern tree-sitter, legacy data store etc.)
  • Later on this would become "opt-out"
  • We would need to agree whether we ever deprecate the old paths entirely. We would still need a way of reading from and writing to the old config format but we could decide to stop creating it on new installs.

That makes sense. I think eventually the path for each file in question should probably look something like

const LEGACY_PATH = `${process.env.HOME}/.pulsar`;
const HAS_LEGACY_PATH = await fs.promises.access(LEGACY_PATH).then(() => true).catch(() => false);
  
// ...

// determine location to save the foobar.cfg config file
const FOOBAR_PATH = `${process.env.PULSAR_FOOBAR_PATH || process.env.ATOM_HOME ||
  HAS_LEGACY_PATH ? LEGACY_PATH : process.env.XDG_CONFIG_HOME || `${process.env.HOME}/.config`}/foobar.cfg`;

so users can explicitly override the location with an environment variable (either unique to the file/file group, or the legacy $ATOM_HOME) if they choose, or else it will respect the existing legacy configs if they exist or try the appropriate XDG base dir if they don't, and finally fall back to the default XDG base dir location.

Anyway, I'll see if I can poke at this more, and I'll report back here if I make any progress. Thanks again for the info!

[savetheclocktower]

Looks like a reasonable list to start with to me! Only thing that jumps out to me is .pulsar-socket-secret-dae-1.132.0-dev, which, seeing "secret" in the filename, I'd want to investigate whether it's a permanent secret that needs to be persisted between sessions, in which case $XDG_STATE_HOME is probably the best place

It's this one. The filename is customized only in the username parameter (the dae in @Daeraxa’s case) and the application version — which means it's permanent enough to survive multiple application sessions.

---

The “easy” parts of this will involve

• finding all places that currently use process.env.ATOM_HOME to infer configuration locations;
• changing all those usages so that they are robust enough to allow for the right XDG path to be chosen for the given purpose, while
• ensuring that we have a solid heuristic for deciding whether the user wants Pulsar to follow the XDG Base Specification or would prefer the current structure.

I do not consider much of that easy — especially the last item! — but it's certainly easier than the other part: ensuring that community packages don't break.

Some packages store their own configuration and/or caches in specific files within ATOM_HOME. Any package that is actively maintained can easily be made to do the correct thing, but we have a long tail of community packages that still somehow still work even though they haven't been maintained in years.

If such a package does something naive like

const configFileLocation = path.join(process.env.ATOM_HOME, 'weird-package-config.cson');

then we’ll have to pick an ATOM_HOME value that minimizes annoyance.

I am aware that “following the XDG Base spec” is an all-or-nothing exercise, but since we must report a value for ATOM_HOME, we might as well pick the least worst option. For those packages that are actively being maintained, we'd want to encourage the correct patterns; one option would be to use env-paths internally and then expose an API to package authors that allows them to ask us the correct directory for a particular artifact.