Skip to content

Conversation

@link2xt
Copy link
Collaborator

@link2xt link2xt commented May 28, 2025

Closes #6878
Based on #6867
This depends an a fix #6669, so we may want to postpone merging this until more users have it.

We do not try to delete resent messages
anymore. Previously resent messages
were distinguised by having duplicate Message-ID,
but future Date, but now we need to download
the message before we even see the Date.
We now move the message to the destination folder
but do not fetch it.

It may not be a good idea to delete
the duplicate in multi-device setups anyway,
because the device which has a message
may delete the duplicate of a message
the other device missed.

To avoid triggering IMAP busy move loop
described in the comments
we now only move the messages
from INBOX and Spam folders.

@link2xt
Copy link
Collaborator Author

link2xt commented May 28, 2025

Legacy Python test tests/test_1_online.py::test_resend_message got broken by this.
Resent message is not recognized as duplicate anymore because outer timestamp of just received message is compared to the timestamp of the fetched message:

core/src/imap.rs

Lines 603 to 610 in adcc8a9

let ts_sent = headers
.get_header_value(HeaderDef::Date)
.and_then(|v| mailparse::dateparse(&v).ok())
.unwrap_or_default();
let is_dup = is_dup_msg(is_chat_msg, ts_sent, ts_sent_old);
if is_dup {
info!(context, "Deleting duplicate message {message_id}.");
}

@link2xt link2xt force-pushed the link2xt/protect-date-header branch 4 times, most recently from caf6eff to 29d1d87 Compare May 28, 2025 07:51
@link2xt
Copy link
Collaborator Author

link2xt commented May 28, 2025

This has now broken the test added in c14f45a

@link2xt link2xt marked this pull request as draft May 28, 2025 09:37
@link2xt link2xt self-assigned this May 28, 2025
@link2xt link2xt force-pushed the link2xt/protect-date-header branch from 29d1d87 to e533c2c Compare May 28, 2025 09:51
@link2xt link2xt changed the base branch from main to link2xt/scan-folders-lock May 28, 2025 11:46
@link2xt link2xt force-pushed the link2xt/protect-date-header branch from e533c2c to 5663fff Compare May 28, 2025 11:48
@link2xt link2xt marked this pull request as ready for review May 28, 2025 11:49
@link2xt link2xt force-pushed the link2xt/protect-date-header branch 5 times, most recently from 68a4d43 to 154272e Compare May 28, 2025 15:46
@link2xt link2xt force-pushed the link2xt/scan-folders-lock branch from 7a65541 to 89df953 Compare May 28, 2025 15:46
Base automatically changed from link2xt/scan-folders-lock to main May 28, 2025 16:05
@link2xt link2xt force-pushed the link2xt/protect-date-header branch from 154272e to 10f3e09 Compare May 30, 2025 11:17
@link2xt link2xt mentioned this pull request May 30, 2025
@link2xt link2xt marked this pull request as draft May 30, 2025 19:17
@link2xt
Copy link
Collaborator Author

link2xt commented May 30, 2025

We will have to use something less suspicious than unix epoch, GMX is unhappy:
#6878 (comment)

@link2xt link2xt force-pushed the link2xt/protect-date-header branch 2 times, most recently from d06a61e to ebb7ce8 Compare June 22, 2025 14:50
@link2xt link2xt marked this pull request as ready for review June 22, 2025 14:50
@link2xt
Copy link
Collaborator Author

link2xt commented Jun 22, 2025

I have updated this PR to not send 1970 as the date. Should be good enough for review/merging now.

@link2xt link2xt requested review from Hocuri and iequidoo June 22, 2025 14:51
@link2xt
Copy link
Collaborator Author

link2xt commented Jun 22, 2025

So now the date looks like this Date: Sun, 15 Jun 2025 15:06:40 +0000.
You convert it and get:

date -d 'Sun, 15 Jun 2025 15:06:40 +0000' +%s
1750000000

@link2xt link2xt force-pushed the link2xt/protect-date-header branch from ebb7ce8 to 54dce24 Compare June 22, 2025 17:24
} else {
replace_msg_id = if rfc724_mid_orig == rfc724_mid {
None
} else if let Some((old_msg_id, old_ts_sent)) =
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part looks unrelated because looking at encrypted timestamp_sent is fine, and does this mean that duplicates will be never deleted, even if the message is deleted by the user?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When user manually deletes a message, all messages with the same rfc724_mid are deleted. See message::delete_msg_ex.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code was deleting duplicates on IMAP, w/o it duplicates will only be deleted locally

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently this code removal corresponds to this part of the commit message:

It may not be a good idea to delete
the duplicate in multi-device setups anyway,
because the device which has a message
may delete the duplicate of a message
the other device missed.

But even if don't delete the duplicate, there's no guarantee the other device didn't miss both messages. Leaving the duplicate is useful though if the user detected a missed message on the other device and asked to resend it. Anyway, if we don't want to IMAP-delete duplicates anymore, i'd add a comment here explaining the motivation / scenario in which keeping the duplicate is useful.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could add a comment, but also, it's fine as-is, because the code IMAP-deleting duplicates is removed now, and it's more important to explain why we do things than explain why we don't do things.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code in imap is being removed, but it was just an optimization removing duplicates before decryption and parsing. A short comment won't make harm.

But i think that it's better to IMAP-delete not the duplicate, but the existing message instead, this way we don't affect receiving re-sent messages by other devices.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But i think that it's better to IMAP-delete not the duplicate, but the existing message instead, this way we don't affect receiving re-sent messages by other devices.

If we change from previous behavior (removing last message) to removing the first message, then in multi-device setup with two different versions it's possible to end up with both copies deleted on the server.

@link2xt link2xt force-pushed the link2xt/protect-date-header branch from 54dce24 to 118c10d Compare June 26, 2025 18:52
@link2xt link2xt force-pushed the link2xt/protect-date-header branch from 118c10d to ce0a681 Compare August 15, 2025 02:29
@link2xt link2xt force-pushed the link2xt/protect-date-header branch from ce0a681 to 5a694aa Compare August 22, 2025 20:39
@link2xt link2xt force-pushed the link2xt/protect-date-header branch from 5a694aa to 9b675b5 Compare October 19, 2025 10:59
@link2xt link2xt marked this pull request as draft October 19, 2025 11:10
@link2xt link2xt force-pushed the link2xt/protect-date-header branch 3 times, most recently from c49a1b1 to 511f093 Compare October 19, 2025 18:03
@link2xt link2xt marked this pull request as ready for review October 19, 2025 18:21
@link2xt link2xt force-pushed the link2xt/protect-date-header branch 3 times, most recently from 69720bc to 3fcbe1d Compare October 19, 2025 20:00
@link2xt link2xt force-pushed the link2xt/protect-date-header branch from 3fcbe1d to a4dd140 Compare October 19, 2025 20:26
Copy link
Collaborator

@Hocuri Hocuri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

We do not try to delete resent messages
anymore. Previously resent messages
were distinguised by having duplicate Message-ID,
but future Date, but now we need to download
the message before we even see the Date.
We now move the message to the destination folder
but do not fetch it.

It may not be a good idea to delete
the duplicate in multi-device setups anyway,
because the device which has a message
may delete the duplicate of a message
the other device missed.

To avoid triggering IMAP busy move loop
described in the comments
we now only move the messages
from INBOX and Spam folders.
@Hocuri
Copy link
Collaborator

Hocuri commented Oct 23, 2025

@link2xt can we merge this, or is there a specific reason why you didn't merge it yet?

@link2xt link2xt merged commit 3eb1a7d into main Oct 23, 2025
29 checks passed
@link2xt link2xt deleted the link2xt/protect-date-header branch October 23, 2025 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Protect the Date header

3 participants