Add support for mnemonics in hsm_secret #8400

sangbida · 2025-07-08T00:42:19Z

Important

25.09 FREEZE July 28TH: Non-bugfix PRs not ready by this date will wait for 25.12.

RC1 is scheduled on August 11th

The final release is scheduled for September 1st.

Checklist

Before submitting the PR, ensure the following tasks are completed. If an item is not applicable to your PR, please mark it as checked:

The changelog has been updated in the relevant commit(s) according to the guidelines.
Tests have been added or modified to reflect the changes.
Documentation has been reviewed and updated as needed.
Related issues have been listed and linked, including any that this PR closes.

rustyrussell · 2025-07-08T00:44:55Z

common/hsm_secret.c

+#include <ccan/tal/str/str.h>
+#include <ccan/tal/tal.h>
+#include <common/errcode.h>
+#include <common/utils.h>
+#include <common/hsm_secret.h>
+#include <errno.h>
+#include <termios.h>
+#include <unistd.h>
+#include <ccan/crypto/sha256/sha256.h>  
+#include <ccan/mem/mem.h>               
+#include <sodium.h>                     
+#include <wally_bip39.h>                
+#include <sys/stat.h>                   
+


These need to be in alphabetical order!

rustyrussell · 2025-07-08T00:45:48Z

common/hsm_secret.c

+	case HSM_SECRET_INVALID:
+		return false;
+	}
+	return false;


rustyrussell · 2025-07-08T00:46:54Z

common/hsm_secret.c

+		return HSM_SECRET_ENCRYPTED;
+
+	/* Check if it starts with our type bytes (mnemonic formats) */
+	if (len > 32) {


if (len < 32)
return HSM_SECRET_INVALID;

rustyrussell · 2025-07-08T00:48:31Z

common/hsm_secret.c

+static void hash_passphrase(const char *passphrase, u8 hash[PASSPHRASE_HASH_LEN])
+{
+	struct sha256 sha;
+	sha256(&sha, passphrase, strlen(passphrase));
+	memcpy(hash, sha.u.u8, PASSPHRASE_HASH_LEN);
+}


We have a sha256 type for a reason, let's use it, not convert to u8[]

rustyrussell · 2025-07-08T00:50:59Z

common/hsm_secret.c

+	/* First 32 bytes are the stored passphrase hash */
+	const u8 *stored_hash = hsm_secret;
+	u8 computed_hash[32];
+
+	hash_passphrase(passphrase, computed_hash);
+	return memcmp(stored_hash, computed_hash, 32) == 0;
+}
+


struct sha256 actual, expected;

/* First 32 bytes are the stored passphrase hash */
memcpy(&expected, hsm_secret, sizeof(struct sha256));
sha256(&actual, passphrase, strlen(passphrase));

return sha256_eq(&expected, &actual);

rustyrussell · 2025-07-08T01:43:55Z

tools/hsmtool.c

+	if (strlen(passphrase) == 0) {
+		passphrase = NULL;


Mildly prefer if (streq(passphrase, "")) ?

rustyrussell · 2025-07-08T01:44:18Z

tools/hsmtool.c

-	if (bip39_mnemonic_to_seed(mnemonic, passphrase, bip32_seed, sizeof(bip32_seed), &bip32_seed_len) != WALLY_OK)
-		errx(ERROR_LIBWALLY, "Unable to derive BIP32 seed from BIP39 mnemonic");
-
+	/* Write to file using your new mnemonic format */
 	int fd = open(hsm_secret_path, O_CREAT|O_EXCL|O_WRONLY, 0400);
 	if (fd < 0) {
 		errx(ERROR_USAGE, "Unable to create hsm_secret file");


err() not errx()

rustyrussell · 2025-07-08T01:44:51Z

tools/hsmtool.c

+		sha256(&sha, passphrase, strlen(passphrase));
+
+		if (!write_all(fd, sha.u.u8, PASSPHRASE_HASH_LEN))
+			errx(ERROR_USAGE, "Error writing passphrase hash to hsm_secret file");


rustyrussell · 2025-07-08T01:50:12Z

tools/hsmtool.c

+	if (passphrase) {
+		/* Write passphrase hash (32 bytes) + mnemonic for protected format */
+		struct sha256 sha;
+		sha256(&sha, passphrase, strlen(passphrase));


This should be hash of the final seed. This is a bit trickier: we need hsm_secret to expose that "calculate seed from mnemonic and passphrase" function (or opencode it with libwally?)

rustyrussell · 2025-07-08T01:50:39Z

tools/hsmtool.c

-	u8 bip32_seed[BIP39_SEED_LEN_512];
-	size_t bip32_seed_len;
-
-	if (bip39_mnemonic_to_seed(mnemonic, passphrase, bip32_seed, sizeof(bip32_seed), &bip32_seed_len) != WALLY_OK)
-		errx(ERROR_LIBWALLY, "Unable to derive BIP32 seed from BIP39 mnemonic");


We still need this for below.,

This is a newer API to replace hsm_encryption.c and hsm_encryption.c, this tidies up the API to be used and also cleans things up to support our new formts. Our hsm_secret formats now include: - Legacy 32-byte plain format - Legacy 73-byte encrypted format - New mnemonic format without passphrase (32 zero bytes + mnemonic) - New mnemonic format with passphrase (32-byte hash + mnemonic) This commit includes support to detect the format based on the file size and content structure. The hsm will store mnemonics in the hsm_secret file as: `passphraseHash`mnemonic`

Changelog-Added: `hsmtool` now supports hsm_secret files using a 12-word mnemonic.

In preparation for BIP-39, we need to hand the passphrase (if any) to HSMD. So we extend the hsmd wire protocol to allow that.

Changelog-Changed: hsmd: New nodes will now be created with a BIP-39 12-word phrase as their root secret.

Update the exposesecret plugin to work with the new unified HSM secret format that supports BIP39 mnemonics.

BIP86 derivation requires the full 64-byte seed that comes from the BIP39 mnemonic. The first 32 bytes goes towards to master seed material and the nest 32 bytes go towards the chaincode, so we need the entire 64 bytes for deterministic derivations. I've kept the old secret struct in for now for backwards compatibility and also added some accessors which will eventually die in this branch's git multiverse but that's a spoiler, they're on the ride for the next few commits at least to help us migrate to this length aware API throughout the rest of the code without making a lot of breaking changes.

hsmd: plumb length-aware secret into hsmd_init; keep 32B mirror BIP86 (from BIP39) wants the full 64-byte BIP32 seed. This commit plumbs a variable-length (32/64B) secret into hsmd and uses the accessors from the previous commit. We keep the old 32B hsm_secret mirror and, for now, only use the first 32 bytes so legacy paths keep working. Spoiler: HKDFs will keep using the 32B seed; only wallet address derivation will switch to the full 64B in a follow-up.

Here's some *foreshadowing* for what's to come. Here's what we're aiming for with our derivation flow: Derivation split (hardened vs unhardened) ======================================== ┌───────────────┐ │ HSM │ (secrets live here) │ │ │ BIP39 → seed (64B) │ ↓ │ m/86'/0'/0' ← derive hardened base (private) │ ↓ (neuter) │ BIP86 base xpub ← public-only + chain code │ ↓ │ [send once over wire] └───────────────┘ │ ▼ ┌───────────────────────┐ │ lightningd / wallet │ │ │ │ local (unhardened) derivations: │ /0/i → external │ /1/i → change │ │ │ P2TR(BIP86) from pubkey_i │ (optionally: CHECK with HSM) └───────────────────────┘ We want to do part of the derivation inside hsmd and then send this base "pubkey" over the wire so our wallet can do the remaining derivation based on the address type and index. This lays the foundation for the base key wire message.

Add HSM_PERM_DERIVE_BIP86_KEY to gate future requests that derive under m/86'/0'/0' (BIP86 base). No behaviour change yet—handlers are not enforcing this permission in this commit.

BIP86 wants the full 64-byte BIP32 seed (from BIP39). This wires up BIP86 support so the HSM derives the hardened base m/86'/0'/0' inside the box, and exposes helpers: • derive_bip86_base_key() // m/86'/0'/0' • bip86_key(index) // m/86'/0'/0'/0/index Spoiler: derive_bip86_base_key() and bip86_key() now live in libhsmd.c as they will later be used to check the derived wallet address against hsmd's derivation, this is just to sanity check that we haven't had an accidental bit flip while we have generated this address.

RIP to this commit there's a good chance a lot of this code doesn't even make this into the final PR. Pour one out for the fallen lines of code. This commit is doing the rest of the derivation. There was a significant overlap between the bip32_pubkey derivation and the bip86_pubkey derivation so that has been refactored in one place.

Add the UTXO_P2TR_BIP86 in preparation to add BIP86 wallet functions such as newaddr, listaddr etc. We also add a new index in the database for BIP86 as this is using a completely different derivation path and hsm_secret.

We should now be able to get BIP86 Taproot addresses through lightning-cli! For now we're just adding taproot addresses.

In the case where we receive a taproot utxo we want to be able to tell if it was derived using a BIP32 seed or a BIP86 seed. Considering we will only be supporting BI86 type wallet addresses for mnemonics we can check if the out secret is 64 bytes long and if it is we can use our BIP86 for the withdrawal.

This commit will be rebased for its crimes. This commit is updating hsmtool and exposesecrets to use the new pattern for storing the secret, which is the secret_data and secret_len, to support both 64 byte and 32 byte seeds. However, it also introduces some anti patterns to handle memleak because we provide a NULL context to tal because want the secret_data and to hang around for longer. We also get rid of the accessors we introduced at the start in this commit.

rustyrussell · 2025-08-27T02:51:09Z

common/hsm_secret.c

+}
+
+/* Helper function to validate a mnemonic string */
+static bool validate_mnemonic(const char *mnemonic, enum hsm_secret_error *err)


Nit: this would be better returning the error? Or maybe just clearer.

rustyrussell · 2025-08-27T02:52:30Z

common/hsm_secret.c

+	enum hsm_secret_type type = detect_hsm_secret_type(hsm_secret, len);
+
+	switch (type) {


Maybe skip the tmp var?

rustyrussell · 2025-08-27T02:54:47Z

common/hsm_secret.c

+		return HSM_SECRET_ENCRYPTED;
+
+	/* Check if it starts with our type bytes (mnemonic formats) */
+	if (memeqzero(hsm_secret, 32))


Implied assumption: len >= 32.

/* We would have exited above if we didn't have 32 bytes! */ BUILD_ASSERT(HSM_SECRET_PLAIN_SIZE > 32);

Maybe sizeof(struct sha256)?

rustyrussell · 2025-08-27T02:57:25Z

common/hsm_secret.c

+					       size_t len, 
+					       enum hsm_secret_error *err)
+{
+	struct hsm_secret *hsms = tal(ctx, struct hsm_secret);


assert len == sizeof(hsms->secret) just before memcpy?

rustyrussell · 2025-08-27T02:58:01Z

common/hsm_secret.c

+	*err = HSM_SECRET_OK;
+	return hsms;
+}


This err is unused!

rustyrussell · 2025-08-27T03:15:43Z

tools/hsmtool.c

+	tal_resize(&contents, tal_bytelen(contents) - 1);
+	enum hsm_secret_type type = detect_hsm_secret_type(contents, tal_bytelen(contents));


Again, you don't need to resize, you can just hand tal_bytelen(contents)-1 to detect_hsm_secret_type...

rustyrussell · 2025-08-27T03:17:14Z

tools/hsmtool.c

+	return 0;
+}
+
+static int getemergencyrecover(const char *emer_rec_path)


Ditto. print_emergencyrecover, returns void,

rustyrussell · 2025-08-27T03:18:53Z

tools/hsmtool.c

-};
-
-static bool check_lang(const char *abbr)
+static int generate_hsm(const char *hsm_secret_path)


can be void, does this ever return anything but 1?

rustyrussell · 2025-08-27T03:19:27Z

tools/hsmtool.c

@@ -651,53 +513,58 @@ static int dumponchaindescriptors(const char *hsm_secret_path,
 	return 0;


here too...

rustyrussell · 2025-08-27T03:20:42Z

tests/test_plugin.py

@@ -4141,6 +4141,7 @@ def test_important_plugin_shutdown(node_factory):
    l1.rpc.plugin_start(os.path.join(os.getcwd(), 'plugins/pay'))


+@pytest.mark.skip(reason="Temporarily disabled expose secrets test")
 @unittest.skipIf(VALGRIND, "It does not play well with prompt and key derivation.")


Changelog-Removed: hsmtool support for mnemonics in non-english languages removed.

rustyrussell

Excellent work! Minor tweaks and some reshuffling...

rustyrussell · 2025-08-27T03:30:20Z

hsmd/hsmd.c

+	status_debug("HSM: Starting create_hsm with passphrase=%s", passphrase ? "provided" : "none");
+
+	/* Initialize wally tal context for libwally operations */
+
+	status_debug("HSM: Initialized wally tal context");


too chatty!

rustyrussell · 2025-08-27T03:33:48Z

hsmd/hsmd.c

+	hsm_secret_len = PASSPHRASE_HASH_LEN + strlen(mnemonic);
+	hsm_secret_data = tal_arr(tmpctx, u8, hsm_secret_len);
+
+	/* Copy seed hash first */
+	memcpy(hsm_secret_data, &seed_hash, PASSPHRASE_HASH_LEN);
+	/* Copy mnemonic after seed hash */
+	memcpy(hsm_secret_data + PASSPHRASE_HASH_LEN, mnemonic, strlen(mnemonic));
+	status_debug("HSM: Created hsm_secret data structure");


This is v efficient. However, it's slightly safer to use an extend pattern, and in fact we have towire() which does this:

hsm_secret_data = tal_arr(tmpctx, u8, 0); towire_sha256(&hsm_secret_data, &seed_hash); towire(&hsm_secret_data, mnemonic, strlen(mnemonic))

rustyrussell · 2025-08-27T03:35:15Z

hsmd/hsmd.c

-			exit(1);
-		}
+	/* Remove the NUL terminator that grab_file adds */
+	tal_resize(&hsm_secret_contents, tal_bytelen(hsm_secret_contents) - 1);


Still unsure about this tal_resize() pattern, but it's not wrong.

rustyrussell · 2025-08-27T03:37:11Z

hsmd/hsmd.c

+	 * the TLV, and left the old "hsm_encryption_key" field in place (and lightningd
+	 * never sets that anymore), and we use the TLV instead. */
+	if (tlvs->hsm_passphrase)
+		hsm_passphrase = (const char *)tlvs->hsm_passphrase;


Do you need this cast?

rustyrussell · 2025-08-27T03:38:06Z

hsmd/hsmd.c


 	/* Define the minimum common max version for the hsmd one */
 	hsmd_mutual_version = maxversion < our_maxversion ? maxversion : our_maxversion;
-	return req_reply(conn, c, hsmd_init(hsm_secret, hsmd_mutual_version,
+
+	/* This was tallocated off NULL, and memleak complains if we don't free it */


Yeah, really, it's because not freeing it is wrong! Don't complain about your tools!

rustyrussell · 2025-08-27T04:43:59Z

hsmd/libhsmd.c

+/* BIP86 key derivation functions moved from hsmd.c */
+void derive_bip86_base_key(struct ext_key *bip86_base)
+{
+	/* Check if we have the full BIP32 seed available */


Move back to reduce diff!

rustyrussell · 2025-08-27T04:44:06Z

hsmd/libhsmd.h

+/* Forward declaration for secretstuff access */
+u8 *get_secretstuff_bip32_seed(void);
+


rustyrussell · 2025-08-27T04:45:08Z

doc/schemas/newaddr.json

-          "It specifies the type of address wanted; currently *bech32* (e.g. `tb1qu9j4lg5f9rgjyfhvfd905vw46eg39czmktxqgg` on bitcoin testnet or `bc1qwqdg6squsna38e46795at95yu9atm8azzmyvckulcc7kytlcckxswvvzej` on bitcoin mainnet), or *p2tr* taproot addresses. The special value *all* generates all known address types for the same underlying key."
+          "It specifies the type of address wanted; currently *bech32* (e.g. `tb1qu9j4lg5f9rgjyfhvfd905vw46eg39czmktxqgg` on bitcoin testnet or `bc1qwqdg6squsna38e46795at95yu9atm8azzmyvckulcc7kytlcckxswvvzej` on bitcoin mainnet), *p2tr* taproot addresses, or *bip86* for BIP86-derived taproot addresses. The special value *all* generates all known address types for the same underlying key."


Obv drop API change

rustyrussell · 2025-08-27T04:46:59Z

tests/test_wallet.py

+@unittest.skipIf(TEST_NETWORK != 'regtest', "BIP86 tests are regtest-specific")
+def test_bip86_deterministic_addresses(node_factory):


We can remove this with more precise checks above...

rustyrussell · 2025-08-27T04:48:05Z

wallet/test/run-wallet.c

 }
+
 u8 *towire_hsmd_derive_bip86_key(const tal_t *ctx UNNEEDED, u32 index UNNEEDED, bool is_change UNNEEDED)


rustyrussell requested changes Jul 8, 2025

View reviewed changes

sangbida force-pushed the bip39-hsm branch 2 times, most recently from 5de3e77 to 1ae8dc0 Compare July 17, 2025 05:15

rustyrussell force-pushed the bip39-hsm branch from 38e191b to 877e5a1 Compare July 23, 2025 06:50

sangbida force-pushed the bip39-hsm branch from 877e5a1 to 68cb21c Compare July 24, 2025 05:58

sangbida added 4 commits August 3, 2025 21:45

tools: update hsmtool to use the new hsm_secret API.

b0ddad6

Changelog-Added: `hsmtool` now supports hsm_secret files using a 12-word mnemonic.

hsmd: take the passphrase raw, not the derived secret.

45467c1

In preparation for BIP-39, we need to hand the passphrase (if any) to HSMD. So we extend the hsmd wire protocol to allow that.

hsmd: use the new mnemonic-compatible hsm_secret routines.

f6030aa

Changelog-Changed: hsmd: New nodes will now be created with a BIP-39 12-word phrase as their root secret.

exposesecret: Add support for mnemonic-based HSM secrets

aaea608

Update the exposesecret plugin to work with the new unified HSM secret format that supports BIP39 mnemonics.

sangbida force-pushed the bip39-hsm branch from 68cb21c to aaea608 Compare August 3, 2025 12:47

sangbida force-pushed the bip39-hsm branch 2 times, most recently from 67a6b9b to e8c0864 Compare August 25, 2025 11:42

sangbida added 11 commits August 25, 2025 22:30

hsmd: reserve permission bit for BIP86 derivation

65746a3

Add HSM_PERM_DERIVE_BIP86_KEY to gate future requests that derive under m/86'/0'/0' (BIP86 base). No behaviour change yet—handlers are not enforcing this permission in this commit.

wallet: scaffold BIP86 addrtype

b91395d

Add the UTXO_P2TR_BIP86 in preparation to add BIP86 wallet functions such as newaddr, listaddr etc. We also add a new index in the database for BIP86 as this is using a completely different derivation path and hsm_secret.

walletrpc: add addresstype "bip86"; make newaddr+listaddresses use it

141a9b0

We should now be able to get BIP86 Taproot addresses through lightning-cli! For now we're just adding taproot addresses.

tests: add BIP86 support

3e25fd9

sangbida force-pushed the bip39-hsm branch from e8c0864 to 3e25fd9 Compare August 27, 2025 00:31

rustyrussell reviewed Aug 27, 2025

View reviewed changes

		enum hsm_secret_type type = detect_hsm_secret_type(hsm_secret, len);

		switch (type) {

		tal_resize(&contents, tal_bytelen(contents) - 1);
		enum hsm_secret_type type = detect_hsm_secret_type(contents, tal_bytelen(contents));

		@@ -651,53 +513,58 @@ static int dumponchaindescriptors(const char *hsm_secret_path,
		return 0;

		/* Forward declaration for secretstuff access */
		u8 *get_secretstuff_bip32_seed(void);

		"It specifies the type of address wanted; currently bech32 (e.g. `tb1qu9j4lg5f9rgjyfhvfd905vw46eg39czmktxqgg` on bitcoin testnet or `bc1qwqdg6squsna38e46795at95yu9atm8azzmyvckulcc7kytlcckxswvvzej` on bitcoin mainnet), or p2tr taproot addresses. The special value all generates all known address types for the same underlying key."
		"It specifies the type of address wanted; currently bech32 (e.g. `tb1qu9j4lg5f9rgjyfhvfd905vw46eg39czmktxqgg` on bitcoin testnet or `bc1qwqdg6squsna38e46795at95yu9atm8azzmyvckulcc7kytlcckxswvvzej` on bitcoin mainnet), p2tr taproot addresses, or bip86 for BIP86-derived taproot addresses. The special value all generates all known address types for the same underlying key."

		@unittest.skipIf(TEST_NETWORK != 'regtest', "BIP86 tests are regtest-specific")
		def test_bip86_deterministic_addresses(node_factory):

		}

		u8 towire_hsmd_derive_bip86_key(const tal_t ctx UNNEEDED, u32 index UNNEEDED, bool is_change UNNEEDED)

Add support for mnemonics in hsm_secret #8400

Are you sure you want to change the base?

Add support for mnemonics in hsm_secret #8400

Uh oh!

Conversation

sangbida commented Jul 8, 2025

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rustyrussell left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!