IMAP IDLE - unique id not found

[HelloSebastian]

Describe the bug
Hi @Webklex,

I am trying to use the IDLE function of IMAP. Folder provides the method idle(callback, ...). Unfortunately the method always gives me the error message "unique id not found".

I took a closer look at the method and found out that the value EXISTS is evaluated. If I interpret the specification of IMAP correctly, then EXISTS tells how many e-mails are in the respective folder. However, in the idle method, the number of emails (EXISTS) is used as UID to find out the new email. In my mailbox, the number is not equal to the UID and thus no email is found.

Folder also has a getStatus() method that returns the next UID by "uidnext". I tried to find my new email with this UID, but also without success.

Am I doing something wrong or did I not understand something correctly?

Used config
Please provide the used config, if you are not using the package default config.

I use the default configuration

Code to Reproduce
The troubling code section which produces the reported bug.

/** @var Folder $inbox */
$inbox = $client->getFolderByName("INBOX");

$inbox->idle(function (Message $message) {
    $text = sprintf("%s: %s", $message->getUid(), $message->getSubject());
    var_dump($text);
}, 1200, false);

Desktop / Server (please complete the following information):

• OS: [Mac OS Big Sur]
• PHP: [8.0.14]
• Version [3.1.0-alpha]
• Provider [Gmail]

Additional context

Here is my modified idle method:

while (true) {
    try {
        $line = $connection->nextLine();
                
        $connection->done();
                
        $status = $this->getStatus();
                
        echo $status["uidnext"];

        $this->client->openFolder($this->path, true);
        $message = $this->query()->getMessageByMsgn($status["uidnext"]);
        $message->setSequence($sequence);
        $callback($message);

        $event = $this->getEvent("message", "new");
        $event::dispatch($message);
                
        $connection->idle();
    }catch (Exceptions\RuntimeException $e) {
        // ...
   }
}

Maybe the method should also check if IDLE is available from the IMAP server. I have added this check:

if (!in_array("IDLE", $connection->getCapabilities())) {
    throw new Exceptions\RuntimeException("IMAP server does not support IDLE");
}

[Webklex]

Hi @HelloSebastian ,
many thanks for your suggestions. As you can see above I've already implemented one of them :)

However I have a few questions:

1. $connection->done(); isn't that interrupting the "listener" and could result in missing mails if they come in while fetching the last message? I'm aware that it is currently doing the exact same thing, which made me question it.. :) I would like to know what you think about this.
2. uidnext isn't always available (?). I'm not sure under which circumstances but the problem can be seen within Message::move - so it won't work for all?
3. Without the position check if (($pos = strpos($line, "EXISTS")) !== false) { the callback could be called multiple times?
4. EXISTS should be the message number? Unless I'm mixing something up - I thought the message number is basically an self updating index, which correlates with the number of messages within a given folder / inbox. Query::getMessageByMsgn isn't using / shouldn't use an UID but instead a message number.

I guess it should be a mixture of both?

I hope to hear your thoughts and thanks again for your time and support!

Best regards,

[HelloSebastian]

Hi @Webklex,

sorry for leaving the issue still open! I can not explain why it did not work. I tried it again a few days later and it worked.

1. you are right about that. If a message is received between done and idle, it will not be seen.
2. at the time I opened the issue, I didn't quite understand what UID, message number and sequence number were. So I thought you had to retrieve the message using the UID.
3. Probably, yes. As I said, my solution was based on the fact that I did not yet know the different number of a message.
4. According to the RFC 3501, EXISTS tells you the number of messages. However, since the newest message has the sequence number of the number of messages, we can use the EXISTS to determine the newest message. (this is my understanding)

I find the different numbers of a message very confusing.

[HelloSebastian]

As I understand it, a message can have three "numbers" assigned to it.

1. UID: Assigned by the server and unique per mailbox. The UID is an incrementing number that is assigned to each new message of the mailbox. It doesn't matter where the message comes from; i.e. whether it was delivered via SMPT or moved from another mailbox. If you delete or move a message, all other messages keep their UID.
2. Message ID: Assigned by the client in the header. Does not have to be a "number", but rather a hash with mostly "@[provider.com]" (e.g. gmail [unique hash]@gmail.com). The message ID should always be unique for all messages in the world. Technically, however, this is not mandatory. The server also accepts messages with the same Message ID. I couldn't find out if the "<", ">" characters belong to the message ID or are just for parsing the header. (see Inconsistent parsing of message_id and in_reply_to #197)
3. Sequence number: Assigned and managed by the server. The sequence number serves as a kind of index for the position of a message within a mailbox. That means that with deletion or shifting of messages the following ones get a new sequence number assigned.

[Webklex]

Thanks, looks like we think alike :)

Regarding 2. Message ID: I have the same problem, I don't know how to handle the <>. Removing them would be nice but could also break something somewhere else. I'm not sure why they are there... It could be possible that the syntax is equal to "from" or "to" (aka "Some Name < email / message ID>") but I haven't seen something like this in the wild and can't find something about it inside the RFC. If used as part of a query the <> are meaningless (at least if you are using dovecot).. I'm kind of lost and can't decide..

[HelloSebastian]

Hi @Webklex,

I found something about the message id and the angle brackets. See my answer: #197 (comment)