Feature request: add support for CUSTOM_ID

[marczz]

When we comment with org-remark an org mode buffer, if we have used an ID in the section or a parent, if we use org-remark-use-org-id it is linked from the annotation. This is quite handy when we are working on the source buffer, when the subsection where the annotation resides move because line where added before it, or because when refactoring the source we move the subtree, all the marks of the region and the line is lost, but thanks to the org id; we can find where the annotation is coming from, and restore it.

But using systematic org id on my sections, is not in my habits, I use on nearly each section a CUSTOM_ID and use it as reference for my internal links. I find the source more readable, with a meaningful name. Only when I truly need an unique ID among all my org files, I use an ID, as example org-transclusion need it.

So my feature request would be to also add an option to link to the closer CUSTOM_ID, working like the org-id feature.

[nobiot]

Like this? See #98.

I am not sure if I like it yet -- For CUSTOM_ID, the file name needs to be added -- depending on the scenario, you may prefer absolute or relative name. This uses function org-remark-source-file-name.

[marczz]

Thank you for your quick answer, that I have examined.

First point that may influence the choice: I don't understand why you have chosen to put the org-id in the body of the annotation; for me it seems to be an information on the location of the annotation, in the same way as :org-remark-link and could be put in the property list. I don't see a valid reason to mix it with the body of annotation, and it let garbage when we delete the annotation, if we have not chosen the option to delete the body. As an example I made some annotations for a test, then I deleted them and now my annotation file look like that example-notes.txt.

For me it seems that to have the org-id as property, is natural since this is a metadata of the annotation, would be clear for the user, and avoid this problem of letting garbage on deletion.

Now for the CUSTOM_ID: I don't see the necessity to choose for all files between the :ID and :CUSTOM_ID, with org remark we try to annotate source text; which is a very impermanent and moving source, this is not the same for the org-noter package; which annotate mainly pdf, which have a certain stability. So our annotation will be unsettled when we work on the source file. And the more way we have to find what they were referencing, the better when the beg, end pair have moved. If their is a close ID very good, if there is a close CUSTOM_ID very good also, if not I will try to find out with the original text, and may be the line number.

So I would like to have everything in the property list, and not choose between the two. I don't even see why we have to ask explicitly to use an ID, in any case it does not harm.

I don't see why CUSTOM_ID being only unique in the source file is a problem for annotations, all the annotation of a specific source are relative to this source.

I must say that my knowledge of org-remark is very superficial, and probably some of my remarks are completely out of the track, I apologize for it.

[nobiot]

Thank you for the detailed comments and the sample file. You have raised a number of important points. Let me spend some time and come back with an extensive response.

In the meantime, can I ask you please to show me an example note file that has exactly what you would expect, especially about ID and CUSTOM_ID in the note property?

[marczz]

I attach a example.txt which is an example org source file, and its remark annotations example-notes.org.

I have put in the properties, supplementary properties for ID and CUSTOM_ID.

Then you will find an example of content added to example.txt, out of org-remark (by somebody else, or an other editor, or ....) the file example-stage2.txt.

Of course all the annotations are broken, the position, line number have changed, so what appear highlighted is only the previous position in the buffer.

Now suppose I want to recover my annotation, the only undeceiving info are the original text, the id and the custom id. These links are still working. Of course in this tiny example, only the original text would have been sufficient, but in a larger file not, and even this original text can have changed.

For the sake of clarity I also put below one of the (wished) header from my notes file:

** org leader map
:PROPERTIES:
:org-remark-id: 398baa75
:org-remark-label: nil
:org-remark-beg: 470
:org-remark-end: 484
:org-remark-link: [[file:example.txt::15]]
:org-remark-org-id: [[id:08205f3d-8f5d-44a5-ab8f-9d60bfa6d39d]]
:org-remark-custom-id: [[file:example.txt::#org-transclusion]]
:org-remark-original-text: org leader map
:END:

[nobiot]

A proposal to your feature request for ID and CUSTOM_ID properties

Now suppose I want to recover my annotation, the only undeceiving info are the original text, the id and the custom id. These links are still working. Of course in this tiny example, only the original text would have been sufficient, but in a larger file not, and even this original text can have changed.

For the sake of clarity I also put below one of the (wished) header from my notes file:

** org leader map
:PROPERTIES:
:org-remark-id: 398baa75
:org-remark-label: nil
:org-remark-beg: 470
:org-remark-end: 484
:org-remark-link: [[file:example.txt::15]]
:org-remark-org-id: [[id:08205f3d-8f5d-44a5-ab8f-9d60bfa6d39d]]
:org-remark-custom-id: [[file:example.txt::#org-transclusion]]
:org-remark-original-text: org leader map
:END:

Thank you for providing the example. For this, the following should get you what you have described so far.

(add-hook 'org-remark-highlight-other-props-functions
          #'my/org-remark-prop-org-id)

(add-hook 'org-remark-highlight-other-props-functions
          #'my/org-remark-prop-custom-id)

(defun my/org-remark-prop-org-id (highlight)
  "Add org-remark-org-id props."
  (and-let* ((org-mode-p (derived-mode-p 'org-mode))
             (point (overlay-start highlight))
             (org-id (org-entry-get point "ID" :inherit)))
    (list 'org-remark-org-id (concat "[[id:" org-id "]]"))))

(defun my/org-remark-prop-custom-id (highlight)
  "Add org-remark-custom-id props."
  (and-let* ((org-mode-p (derived-mode-p 'org-mode))
             (point (overlay-start highlight))
             (filename (org-remark-source-get-file-name
                        (org-remark-source-find-file-name)))
             (custom-id (org-entry-get point "CUSTOM_ID" :inherit)))
    (list 'org-remark-custom-id
          (concat "[[file:" filename "::#" custom-id "]]"))))

Org-remark has a facility with an abnormal hook (org-remark-highlight-other-props-functions) to let you add your own properties. The example functions above add an ID and CUSTOM_ID to the annotation entry in the notes buffer. The hooks enable them to be triggered automatically.

This abnormal hook is undocumented and not released at the moment (I wish to get round to it soon), but it's used by the org-remark-line.el so I think it will stay as is. Try it and let me know how you go.

Explanation of org-remark-highlight-adjust-position-after-load

Now for the CUSTOM_ID: I don't see the necessity to choose for all files between the :ID and :CUSTOM_ID, with org remark we try to annotate source text; which is a very impermanent and moving source, this is not the same for the org-noter package; which annotate mainly pdf, which have a certain stability. So our annotation will be unsettled when we work on the source file. And the more way we have to find what they were referencing, the better when the beg, end pair have moved. If their is a close ID very good, if there is a close CUSTOM_ID very good also, if not I will try to find out with the original text, and may be the line number.

Of course all the annotations are broken, the position, line number have changed, so what appear highlighted is only the previous position in the buffer.

Then you will find an example of content added to example.txt, out of org-remark (by somebody else, or an other editor, or ....) the file example-stage2.txt.

Here is how I interpret what you say in these sentences:

• Unlike PDF and org-noter, org-remark deals with texts that you still edit. There are a number of ways where the highlights are dislocated from their intended places as you edit the main text -- thus, breaking the correct association between the notes and main text.

• You often need to fix the dislocated highlights, but find it difficult to locate their original intended places.

• CUSTOM_ID and ID can help you locate the original place of broken highlights.

I share the same issue, and have been wanting to somehow fix it since the moment I released the package (it has been about 5 years now). It's been simply too hard a problem to tackle.

Two comments on this point of broken highlights:

1. [Future plan] I have a prototype of potential fix finally (!)

The code is not ready and it's still a prototype; located at [[https://git.sr.ht/~nobiot/chu]].

In this video, it shows an Org buffer (decorated with org-modern). You should see two headlines with titles "abandoned" and "abandonedly", with each having a highlight -- a yellow highlight in the first headline and a red in the second. Then I am doing org-meta-up/down to move the heading around. You see that the highlights follow with the headlines correctly.

Wayfarer_2025-02-04_192603.webm

Over the next months (hopefully within this year) I am planning to apply the method I use in this prototype to Org-remark. Not only does it support the structure edit of Org as you see in the video, but it also supports undo and yank within the same buffer and over to another. Its performance seems fine with a large text file as far as my current testing goes (I am testing with Moby Dick book, etc.).

Now, this is only a future plan and cannot solve the current issue of Org-remark.

2. Org-remark has a way to automatically adjust highlight positions

Org remark has a feature to automatically adjust highlight positions when the text file has changed from an external source. It's written in the user manual: URL is here or in the Info manual of Org-remark that comes with the package, there is a secion here:
(info "(org-remark) What is Automatic Adjustment of Highlight Positions?").

This adjustment only works if:

• The text contains the same text as the original, and
• The change of the location is within 2 paragraphs.

I have tested this feature with your example-stage2.txt. For this, you need to check 4 paragraphs, so you'd need to change the code like this:

org-remark-highlight-adjust-position-after-load

@@ -1331,8 +1331,8 @@ It searches for TEXT, which should be the original text of the highlight."
      (unless (org-remark-string= (buffer-substring-no-properties beg end) text)
        ;; Look at one paragraph ahead as it is possible that the
        ;; position has been displaced across a paragraph
-       (goto-char beg) (backward-paragraph 2) (setq paragraph-beg (point))
-       (goto-char beg) (forward-paragraph 2) (setq paragraph-end (point))
+       (goto-char beg) (backward-paragraph 4) (setq paragraph-beg (point))
+       (goto-char beg) (forward-paragraph 4) (setq paragraph-end (point))
        (goto-char paragraph-beg)
        ;; Search from the beginning of the previous paragraph to the end
        ;; of next paragraph relative to the begining of the highlight

The result looks like this:

The "(d)" that follows the highlight denotes that it's been dislocated and so adjusted. In this example, it appears that you've also changed the text from the original. This feature may help you re-place the dislocated highlights back to where you intend them to be.

Why ID is added to the body text of annotation?

I have run out of time for this. I will come back.

[marczz]

Thank you a lot @nobiot, I didn't expect a so detailed, enlightening, and useful answer.
Concerning the distinct parts of your answer:

id inside properties

The proposed code is perfect, I tested it with an org file having on sections a mix of ID and CUSTOM_ID, and it is very useful, you click on the property, and you go to the corresponding reference; as it is in the properties; it does not pollute your comments, and get deleted, if you choose to delete the annotation.

So the feature which I first mentioned is fully implemented, I let you choose if you consider that is is just an oddity of a peculiar user, or if can help other and incorporate it in next release.

Alas this work only for org-mode, in other modes we don't have this notion of ID, we could think at the notion of section which exist in all structured text formats: org, markdown, ReST; but I don't see a reliable way to use it.

Automatic adjustment of position.

I did not noticed this feature, It is surely useful in the case of EPUB, where changing the font size, can change the position of the annotation.

In adapting it for examples, like the one I gave, we are very limited by this 2 paragraphs, after or before, and we cannot change the code for each case.

The notion of paragraph may represent quite big motion or very little one depending of the buffer mode; compare what is a paragraph in code where it can be a full function; and in an enumeration list in structured text (org, markdown ...). And the usual case I encounter, is refactoring, you move a big lump of text from one place to another one, in code you move functions, in org-mode you move sections; they contain many paragraphs. Two or even four paragraphs is very limited

I understand that we cannot choose a too wide area for looking for the original text, as we may catch any unrelated text. A possibility would be to do it in an interactive way. "Do you want to move your annotation to this occurrence?". Then we could look at the whole buffer; beginning with the closer occurrence.

This problem of modified target of annotation is something I encounter with code; I want to annotate code in a cloned repository of some package; when I pull new changes, the files have changed, but if I commented on a stable element like a function name, my original remark is still present in the new file.

The chu package

It is a very nice idea, I looked at the repo and the chu-notes. I see that the code is quite compact. (albeit I did not yet tried to understand it!). It is a very promising package, which will improve the org-remark handling of moving targets.

Thank again for your answer, your reactivity, and for a set a great packages.