[BUG] OTA Demo is constantly downloading and activating the same job

[fjuliofontes]

OTA Demo seems to be always downloading the same job.

[AniruddhaKanhere]

Hello @fjuliofontes, sorry to hear that you are having trouble with the example.

Can you please check whether the job is marked as complete on the IoT core? The demo is supposed to send a completion notification to the IoT core.

[fjuliofontes]

Hello @AniruddhaKanhere ,
Thank you for replying. No, the job is with "PENDING". I can get you a full text file with logs.
By the way, I'm running the master version.

[ActoryOu]

Hi @fjuliofontes,
While creating OTA Jobs, there are two Job run types. Which one are you using?

• Your job will complete after deploying to the devices and groups that you chose (snapshot)
• Your job will continue to deploy to any devices added to the groups that you chose (continuous)

Thank you.

[ActoryOu]

Besides, could you elaborate your scenario a little bit?
Have you completed the OTA? Or is it failing at some point so the OTA job is not completed yet?

Thank you.

[fjuliofontes]

Hi @ActoryOu ,
Thank you for the ongoing support.
I created a snapshot, but let me get back to you tomorrow. I will be again working on this and I will try to capture all the logs to help.

Thank you

[AniruddhaKanhere]

@fjuliofontes would you please enable all logs? I want to see why doesn't the job send a notification to IoT core. If it completed successfully and if the device rebooted, then the firmware should send an update to the MQTT broker informing the cloud of completion.

You can enable MQTT logs here by defining all the following macros:

#define   CONFIG_CORE_MQTT_LOG_ERROR
#define   CONFIG_CORE_MQTT_LOG_WARN
#define   CONFIG_CORE_MQTT_LOG_INFO
#define   CONFIG_CORE_MQTT_LOG_DEBUG

[fjuliofontes]

Hi @AniruddhaKanhere,

I collected the logs before seeing your previous message. Can you check if this is enough? As you can see it continues to download the same job. The only way to stop is by cancelling it on AWS.

ota_log.txt

[AniruddhaKanhere]

@fjuliofontes, thank you for the logs! I am looking at them and will get back to you.

One question though - did you generate a new image and upload that to the console? Sometimes if I find a bug and fix it and run the OTA demo, I forget to also update the image in the cloud which is to be downloaded - that causes a bad image to be downloaded.

Thanks

[GillesHaverbeke]

Hi! It seems I'm having the same issue. After completion & reboot, the PUBLISH packet to set the job to Succeeded is not accepted due to a "VersionMismatch":

I (4078) ota_over_mqtt_demo: OTA Completed successfully!

I (4158) coreMQTT: De-serialized incoming PUBLISH packet: DeserializerResult=MQTTSuccess.
I (4158) coreMQTT: State record updated. New state=MQTTPublishDone.
I (4168) ota_over_mqtt: Received update response: $aws/things/744DBDB372DC/jobs/AFR_OTA-DEV_241111_UPDATE_3/update/rejected{"timestamp":1731330895,"executionState":{"status":"IN_PROGRESS","statusDetails":{"self_test":"ready","updatedBy":"0x00000018"},"versionNumber":15},"code":"VersionMismatch","message":"Expected version 2 but found version 15"}0888,"versionNumber":15,"executionNumber":1,"jobDocument":{"afr_ota":{"protocols":["MQTT"],"streamname":"AFR_OTA-9fd67c0d-dddd-49d5-b1ca-8e757d2d975e","files":[{"filepath":"/","filesize":1275600,"fileid":0,"certfile":"/","sig-sha256-ecdsa":"MEEZCCBReQy8V97LtR+WqUFM85zasb6kWoOFZPe6rToYC0mEgIgIWs4pDhRxXnHjw1jyezCSa8y8c44+c6dMHaB62I7/Jg="}]}}}}.
I (4228) coreMQTT: Ack packet deserialized with result: MQTTSuccess.
I (4238) coreMQTT: State record updated. New state=MQTTPublishDone.
I (4238) mqtt_communication: Publish 3: SUCCESS

After this, the code is again picking up the same Job and keeps updating endlessly.

I think the cause is in this code:

        size_t messageBufferLength = Jobs_UpdateMsg( Succeeded,
                                                     "2",
                                                     1U,
                                                     messageBuffer,
                                                     UPDATE_JOB_MSG_LENGTH );

The hardcoded "2" is mismatching the version in AWS, which seems to be 15.

Maybe a fix is to describe the job document version before updating to retrieve the correct version?

[AniruddhaKanhere]

@GillesHaverbeke, yes you are absolutely correct. If the job version reported is not correct, the IoT core will reject the update.
I will raise a PR soon as time permits to fix this issue! Thanks a lot for taking the time to help debug this issue.

[GuGu927]

@AniruddhaKanhere Hi. Version mismatch problem is fixed?

[GuGu927]

@AniruddhaKanhere Hi. Version mismatch problem is fixed?

I solve the problem with this way.

1. Extract verionNumber when job received.
2. Save verionNumber to NVS when OtaAgentEventRequestFileBlock(when only currentBlockOffset is 0, "Starting the Download.")
3. After reboot, load verionNumber from NVS and put it in here
   size_t messageBufferLength = Jobs_UpdateMsg(Succeeded, versionStr, versionStrLength, messageBuffer, UPDATE_JOB_MSG_LENGTH);

[RomanKhudobei]

I figured it is related to this option in sdkconfig. Once I turned it on, OTA update finished successfully without attempting the same job. Might give anybody clues what's the issue

[fjuliofontes]

I did the following:

Added two new fields on AfrOtaJobDocumentFields_t (job_parser.h):

typedef struct
{
    /** @brief Code Signing Signature */
    const char * signature;

    /** @brief Length of signature */
    size_t signatureLen;

    /** @brief File path to store OTA Update on device */
    const char * filepath;

    /** @brief Length of filepath */
    size_t filepathLen;

    /** @brief Path to Code Signing Certificate on Device */
    const char * certfile;

    /** @brief Length of certfile */
    size_t certfileLen;

    /** @brief Authentication Scheme for HTTP URL ( null for MQTT ) */
    const char * authScheme;

    /** @brief Length of authScheme */
    size_t authSchemeLen;

    /** @brief MQTT Stream or HTTP URL */
    const char * imageRef;

    /** @brief Length of imageRef */
    size_t imageRefLen;

    /** @brief File ID */
    uint32_t fileId;

    /** @brief Size of the OTA Update */
    uint32_t fileSize;

    /** @brief File Type */
    uint32_t fileType;

    /** @brief Job OTA Version */
    const char * jobVersion;

    /** @brief Length of Job OTA Version */
    size_t jobVersionLen;
    
} AfrOtaJobDocumentFields_t;

Created this function on jobs.c:

size_t Jobs_GetJobVersion( const char * message,
    size_t messageLength,
    const char ** jobVersion )
{
    size_t jobVersionLength = 0U;
    JSONStatus_t jsonResult = JSONNotFound;

    jsonResult = JSON_Validate( message, messageLength );

    if( jsonResult == JSONSuccess )
    {
    jsonResult = JSON_SearchConst( message,
                        messageLength,
                        "execution.versionNumber",
                        CONST_STRLEN( "execution.versionNumber" ),
                        jobVersion,
                        &jobVersionLength,
                        NULL );
}

return jobVersionLength;
}

Added on jobs.h:

/**
 * @brief Retrieves the job version from a given message (if applicable)
 *
 * @param message [In] A JSON formatted message
 * @param messageLength [In] The length of the message
 * @param jobVersion [Out] The job version
 * @return size_t The job ID length
 *
 * <b>Example</b>
 * @code{c}
 *
 * // The following example shows the usage of the Jobs_GetJobVersion API to
 * // extract the jobVersion and its length from a message, if present.
 *
 * const char * message;    // A JSON formatted message from the IoT core
 * size_t messageLength;    // Length of the JSON formatted message
 * const char * jobVersion;      // variable to hold jobVersion
 * size_t jobVersionLength = 0U; // Holds length of the jobVersion
 *
 * jobVersionLength = Jobs_GetJobVersion( message,
 *                              messageLength,
 *                              &jobVersion);
 *
 * if ( jobVersion > 0 )
 * {
 *     // Job Version was successfully extracted from the message
 *     // Store the Job version so that it can be used as needed by other functions.
 *     // For example, as an input to Jobs_Update.
 * }
 * @endcode
 */
size_t Jobs_GetJobVersion( const char * message,
    size_t messageLength,
    const char ** jobVersion );
/* @[declare_jobs_getjobversion] */

And then on the handler modified the jobDocumentParser:

static bool jobDocumentParser( char * message,
                               size_t messageLength,
                               AfrOtaJobDocumentFields_t * jobFields )
{
    const char * jobDoc;
    size_t jobDocLength = 0U;
    int8_t fileIndex = 0;
    const char ** jobVersionPtr = (const char **)&jobFields->jobVersion;

    /*
     * AWS IoT Jobs library:
     * Extracting the job version from the received OTA job document.
     */
    jobFields->jobVersionLen = Jobs_GetJobVersion( message, messageLength, jobVersionPtr );
    if (jobFields->jobVersionLen <= 0)
    {
        ESP_LOGE(TAG,"Failed to parse job version from: %.*s", messageLength, message);
    }
    else {
        ESP_LOGI(TAG,"Received Job version: %.*s", jobFields->jobVersionLen, jobFields->jobVersion);
    }

And finaly the sendSuccessMessage:

static bool sendSuccessMessage( void )
{
    char topicBuffer[ TOPIC_BUFFER_SIZE + 1 ] = { 0 };
    size_t topicBufferLength = 0U;
    char messageBuffer[ UPDATE_JOB_MSG_LENGTH ] = { 0 };
    bool result = true;
    JobsStatus_t jobStatusResult;

    /*
     * AWS IoT Jobs library:
     * Creating the MQTT topic to update the status of OTA job.
     */
    jobStatusResult = Jobs_Update( topicBuffer,
                                   TOPIC_BUFFER_SIZE,
                                   board_uuid,
                                   strlen( board_uuid ),
                                   globalJobId,
                                   ( uint16_t ) strnlen( globalJobId, sizeof( globalJobId ) ),
                                   &topicBufferLength );

    if( jobStatusResult == JobsSuccess )
    {
        /*
         * AWS IoT Jobs library:
         * Creating the message which contains the status of OTA job.
         * It will be published on the topic created in the previous step.
         */
        size_t messageBufferLength = Jobs_UpdateMsg( Succeeded,
                                                     jobFields.jobVersion,
                                                     jobFields.jobVersionLen,
                                                     messageBuffer,
                                                     UPDATE_JOB_MSG_LENGTH );

[gamename]

I think I'm running into this same bug. @fjuliofontes, did your changes above fix the issue for you?