MQTT Support

[ayushsingh01042003]

Hello everyone, as part of GSoC'25, I will be working on bringing the MQTT client and broker support to ERDDAP.
For the past few weeks I, @ChrisJohnNOAA and @srstsavage have been working on ways we will be building this and have a short summary doc that goes over the design for this.

Please have a look at it here.

We encourage any and all feedback!
Any opinions? improvements? suggestions? thoughts?

Please go ahead and drop them here. Thanks!

[abkfenris]

With MQTT being able to notify other systems about dataset changes, it would be really useful to be able to notify ERDDAP via MQTT to changes in the underlying data for non MQTT datasets as well.

Currently our data processing system needs to have write access to the ERDDAP flags directory in order to drop flags in there when there are updates. While we make this work, I think if we could notify ERDDAP about changes via a MQTT topic we could have stronger security boundaries (ERDDAP would only need to read where the dataset is written, instead of the data pipelines also needing write access to ERDDAPs directory).

[abkfenris]

I've looked at the HTTP method, but from my exploring it looks like while you can set your own <flagKeyKey> the process to generate an individual dataset flagKey isn't deterministic, which has us back to either manually parsing logs/emails or giving our pipelines access to programmatically parse them.

I was thinking with MQTT we could manage the permissions for the flag topic rather than on a per-dataset level.

[srstsavage]

The original suggestion to add MQTT dataset update support is a good one and I don't want to distract too much from that, but also wanted to share some code extracted (and thus verging on pseudocode) from an internal library that we use to update ERDDAP datasets via HTTP flagKeys:

import hashlib

def erddap_flag_key(erddap_base_url, flag_key_key, dataset_id):
    phrase = f'https://{erddap_base_url}/erddap_{dataset_id}_{flag_key_key}'
    return hashlib.sha256(str.encode(phrase)).hexdigest()

def update_erddap_dataset(base_url, flag_key_key, dataset_id):
    # build dataset update url
    key = erddap_flag_key(base_url, flag_key_key, dataset_id)
    url = f'http://{erddap_url}/erddap/setDatasetFlag.txt?datasetID={dataset_id}&flagKey={key}'
    response = util.requests_get(url)

[srstsavage]

Honestly not sure if the flag key generation is documented anywhere (I don't see it), but we referenced the internal ERDDAP method when working that up:

https://github.com/ERDDAP/erddap/blob/main/WEB-INF/classes/gov/noaa/pfel/erddap/dataset/EDD.java#L3273-L3293

  /**
   * This returns the flagKey (a String of digits) for the datasetID. The result if different if
   * EDStatic.preferredUrl starts with https.
   *
   * @param datasetID
   * @return the flagKey (a String of digits)
   */
  public static String flagKey(String tDatasetID) {
    if (EDStatic.preferredErddapUrl.startsWith("https"))
      // new system
      return String2.passwordDigest(
          "SHA-256", // makes it harder for hacker to deduce flagKeyKey from a set of flagKeys
          EDStatic.preferredErddapUrl + "_" + tDatasetID + "_" + EDStatic.config.flagKeyKey);

    // original system. Best to not change, because existing subscriptions use it.
    return Math2.reduceHashCode(
        EDStatic.erddapUrl.hashCode()
            ^ // always use non-https url
            tDatasetID.hashCode()
            ^ EDStatic.config.flagKeyKey.hashCode());
  }

[rmendels]

Flagkeys are also in the daily email if you look at those.

[abkfenris]

Oh, I had seen random used in conjunction with the key, so I thought it was saving and reusing non-deterministic values, and not something that could be derived from the flagKeyKey and dataset_id.