Add pulsar relay mode

[mvdbeek]

No description provided.

[jmchilton]

Is calling it a proxy a like common pattern or something I'm missing. I read through the whole PR and I think it is very exciting but I don't understand the messaging. It is a different message communication schema than AMQP but it is a similar idea right? Why is this a proxy but we wouldn't call AMQP a proxy? I would call it like reverse long polling or something instead but I may just be not seeing it through your eyes.

[mvdbeek]

yeah, it's not a exactly a proxy .. messages pass unmodified between pulsar and galaxy and yes, the messages are the same as what we use with amqp modality. pulsar-proxy can also do websockets, and we should probably switch transparently between these when websockets are an option. which direction is reverse in the long polling scheme ?

Do you like one of these ?

• relay
• broker
• HTTP message queue
• message proxy

long polling is just the modality I've implemented for now, I'd like to keep that a detail and call both the server (what I call pulsar-proxy) and the procedure in the docs the same thing.

[mvdbeek]

Bit of a chicken and egg, but the integration test in galaxyproject/galaxy#21118 passes locally.

[jmchilton]

Very exciting! Thank you so much for pushing in this direction - how did Claude do with Pulsar - any lessons learned? Did you make any CLAUDE.md files or anything to help it out?

[nsoranzo]

Should these be:

1. Galaxy -> proxy
2. proxy -> Pulsar
3. Pulsar -> proxy
4. Proxy -> Galaxy

?

[mvdbeek]

yep, and I thought I changed the language to relay 🤔

[mvdbeek]

I'm rarely using claude.md, I started with this prompt:

Make a plan for an additional communication mode with galaxy via pulsar-proxy. pulsar-proxy is deployed at
0.0.0.0:9000, you can log in with username: admin, password admin1234. The api is documented at
http://0.0.0.0:9000/openapi.json. You can and should still directly communicate with galaxy where possible,
but galaxy can't reach pulsar, so the pulsar code running with galaxy should post messages to pulsar-proxy,
and the pulsar side should read messages either via a websocket connection or long polling.

and then reviewed the plan and let it do its thing. The next prompt was

Is proxy the best description for this new messaging mode ?

😅

[natefoo]

Does this retry all message types, or just status updates?