Regression: Token renewal fails consistently with messsage "Could not validate signature" starting with 1.5.374.158 and above

[barnstee]

Type of issue

• Bug
• Enhancement
• Compliance
• Question
• Help wanted

Current Behavior

When operating UA servers and clients with issuer-signed certs (GDS functionality), token renewal fails consistently and the session between clients and servers are not reviewed, causing a communication exception with "cloud not verify security" in the servers. This is a regression as this used to work fine with 1.5.374.126. Please roll back any certificate-related changes in releases > 1.5.374.126 until this issue can be investigated.

Expected Behavior

Token renews works and the connection isn't interrupted.

Steps To Reproduce

Run the Station and MES clients/servers that ship with the "production line simulation" from the following Microsoft Learn article: https://learn.microsoft.com/en-us/azure/iot/tutorial-iot-industrial-solution-architecture#production-line-simulation This can be installed as part of the referenced tutorial on any Windows PC by following the steps mentioned: https://learn.microsoft.com/en-us/azure/iot/tutorial-iot-industrial-solution-architecture#install-the-production-line-simulation-and-cloud-services. Upgrade the UA stack used in the "Station" to the latest UA NuGets and run the simulation.

Environment

- OS:Windows 10 with K3s and Dockerized UA servers and clients
- Environment: Visual Studio 2022
- Runtime: .NetCore
- Nuget Version: > 1.5.374.126
- Component:Opc.Ua.Core
- Server: Reference Server
- Client: Reference Client

Anything else?

Happy to help reproduce it, just reach out to me.

[romanett]

@barnstee So you use the UpdateCertifiacte method of the Server to assign a new Application Certificate?

Do you also use the GDS Server to sign the new Application Certificate or even generate it with NewKeyPairRequest?

Which kind of certificate are you using RSA or ECC?

Do you see any errors during the certificate update in the server?
What klind of issue is reported by the certificate validator while trying to renew the token?

[barnstee]

So you use the UpdateCertifiacte method of the Server to assign a new Application Certificate?

Correct

Do you also use the GDS Server to sign the new Application Certificate or even generate it with NewKeyPairRequest?

No, I generate the signing cert using the CertificateFactory, see https://github.com/barnstee/UA-CloudPublisher/blob/6d4feb2a714b28e1662c26875a4aab615d0efd42/UAApplication.cs#L78

Which kind of certificate are you using RSA or ECC?

RSA.

Do you see any errors during the certificate update in the server?

No. The code is here, BTW: https://github.com/barnstee/UA-CloudPublisher/blob/6d4feb2a714b28e1662c26875a4aab615d0efd42/UAClient.cs#L1064

What kind of issue is reported by the certificate validator while trying to renew the token?

Just "Could not verify security" from the Secure Channel.

[romanett]

@barnstee I needed to modify your project to provide certificate.RawData to the UpdateCertificate Method to make it pass, but then the update succeeded. I will no let this setup run for some time.

However the seems to be an issue with the session reconnect implementation in your application:

[barnstee]

Yes, I think the session reconnect has changed recently and it is possible that my reconnect logic is out of date, but to be clear, the issue is with the Station code found here: https://github.com/digitaltwinconsortium/ManufacturingOntologies/tree/main/Tools/FactorySimulation/Station. the UA Cloud Publisher just pushes certs to the Station. The Station can be wither a server or a client. The communication between the client and the server fails after 50 minutes (during token renewal).

[romanett]

@barnstee I will setup with the other project.

I could observe this error in the Server every~50 minutes in the server just letting it run with the CloudPublisher Connected is this the issue you are referring to?

[12:59:12 INF] ChannelId 1633: New Token created. CreatedAt=11:59:12.887-107199078. Lifetime=3600000.
[12:59:12 INF] SECURE CHANNEL RENEWED [.NET Standard ServerChannel UA-TCP 1.5.376.12] [ID=1633]
[12:59:12 INF] ChannelId 1633: Renewed Token #2 set. CreatedAt=11:59:12.887-107199078. Lifetime=3600000.
[12:59:12 ERR] Channel1633: Could not validate signature.
ChannelId=1633, TokenId=1, MessageType=MSGF, Length=112
ExpectedSignature=38584EE699BC226A9967FE15DEBCF444948BE0D9F29F2FE4CCE63C04376B40A8
ActualSignature=7A9DF4F796251DD48149D15FE88742D69A5C486CDBF6B9D444E233FA7A8A005F

[12:59:12 ERR] ChannelId 1633: Could not verify signature on message.
[12:59:13 ERR] TCPSERVERCHANNEL ForceChannelFault Socket=021DCFBC, ChannelId=1633, TokenId=1, Reason=BadSecurityChecksFailed 'Could not verify security on incoming request.'

[barnstee]

Yes.

[romanett]

@barnstee also occurs with self signed certificate & reference client, depending on set SecurityTokenLifetime, after which certificate renewal starts.

I think it is a regression from #2846

[barnstee]

Yes, looks that way. Do you not have regression tests that cover at least 1 day of operation before a release? That would be critically important, if you ask me...