[question] Trying to get last price stream from Okex #78

nagualcode · 2020-04-05T18:26:57Z

Hello, I like the idea to use websocat to get last price streams from cryto exchanges, from the command line.
With Binacne exchange, it is as easy as:
websocat "wss://fstream.binance.com/stream?streams=$pair@markPrice"
But am I having a hard time from Okex exchange.
According to the docs at:
https://www.okex.com/docs/en/#spot-singleness
The URL is:
wss://real.okex.com:8443/ws/v3
I have tried many URLs and all I can get are scramble chinese characters.
Also, it seem to be DEFALTE compressed.
Wonder if would be possible to get the BTC-USD last price ticker, just like it is possible with the Binance example above.

vi · 2020-04-05T23:57:20Z

Websocat may not be the best tool for this, unless a special deflate: overlay is added to it.

Currently it is cumbersome to extract meaningful reply from the server.

$ { printf "\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00"; echo 'QQQ' | websocat --no-line -t - wss://real.okex.com:8443/ws/v3; } | zcat

{"event":"error","message":"Unrecognized request: QQQ\n","errorCode":30039}
gzip: stdin: unexpected end of file

Websocat outputs binary WebSocket messages to stdout as is, without any separators and prefixes. This makes it able to only reliably receive one compressed message.

Are compressed WebSocket messages a popular thing? Are they also used in other places? I'm not sure about adding site-specific things to Websocat.

Maybe I'll add a slow "filterer" overlay that would be able to launch a process for each message that would transform it in arbitrary way.

vi · 2020-04-06T10:55:26Z

Another more universal idea: implement a --base64 option to encode each binary WebSocket message as base64 while preserving text messages. This should be rather universal and would allow to uncompress the messages with the next program in pipeline.

Partially addresses #78, #47. Closes #20.

vi · 2020-04-06T20:02:46Z

Implemented support of base64-encoding binary WebSocket messages.

target/debug/websocat wss://real.okex.com:8443/ws/v3 --base64 | xargs -n1 -- sh -c '{ echo H4sIAAAAAAAAAA== | base64 -d; echo $0 | base64 -d; } | gunzip 2> /dev/null; echo'

qwerwqer
{"event":"error","message":"Unrecognized request: qwerwqer\n","errorCode":30039}
{"op":"subscribe","args":["Lol"]}
{"event":"error","message":"Channel Lol doesn't exist","errorCode":30040}

xargs part should obviously be rewritten to something more optimal if you want to use it for real.

nagualcode · 2020-04-08T17:33:59Z

Nice!
Is this example of yours, where do I enter this msg:
{"op": "subscribe", "args":["swap/ticker:BTC-USD-SWAP"]}

target/debug/websocat wss://real.okex.com:8443/ws/v3 --base64 | xargs -n1 -- sh -c '{ echo '{"op": "subscribe", "args":["swap/ticker:BTC-USD-SWAP"]}' | base64 -d; echo $0 | base64 -d; } | gunzip 2> /dev/null; echo'
?

vi · 2020-04-08T21:53:21Z

To stdin:

$ target/debug/websocat wss://real.okex.com:8443/ws/v3 --base64 | xargs -n1 -- sh -c '{ echo H4sIAAAAAAAAAA== | base64 -d;
echo $0 | base64 -d; } | gunzip 2> /dev/null; echo'
{"op": "subscribe", "args":["swap/ticker:BTC-USD-SWAP"]}
{"event":"subscribe","channel":"swap/ticker:BTC-USD-SWAP"}
{"table":"swap/ticker","data":[{"last":"7355.3","open_24h":"7134.8","best_bid":"7355.2","high_24h":"7432.3","low_24h":"7133.7","volume_24h":"4536696","volume_token_24h":"62179.1352","best_ask":"7355.3","open_interest":"850830","instrument_id":"BTC-USD-SWAP","timestamp":"2020-04-08T21:26:46.608Z","best_bid_size":"762","best_ask_size":"4","last_qty":"2"}]}
{"table":"swap/ticker","data":[{"last":"7356.1","open_24h":"7134.8","best_bid":"7356.1","high_24h":"7432.3","low_24h":"7133.7","volume_24h":"4536718","volume_token_24h":"62179.4341","best_ask":"7356.2","open_interest":"850826","instrument_id":"BTC-USD-SWAP","timestamp":"2020-04-08T21:26:53.163Z","best_bid_size":"522","best_ask_size":"1","last_qty":"19"}]}
{"table":"swap/ticker","data":[{"last":"7356.1","open_24h":"7134.8","best_bid":"7356.1","high_24h":"7432.3","low_24h":"7133.7","volume_24h":"4536718","volume_token_24h":"62179.4341","best_ask":"7356.2","open_interest":"850826","instrument_id":"BTC-USD-SWAP","timestamp":"2020-04-08T21:26:53.168Z","best_bid_size":"522","best_ask_size":"1","last_qty":"3"}]}
{"table":"swap/ticker","data":[{"last":"7356.2","open_24h":"7134.8","best_bid":"7356.1","high_24h":"7432.3","low_24h":"7133.7","volume_24h":"4536721","volume_token_24h":"62179.4746","best_ask":"7356.2","open_interest":"850826","instrument_id":"BTC-USD-SWAP","timestamp":"2020-04-08T21:26:57.477Z","best_bid_size":"522","best_ask_size":"1","last_qty":"1"}]}
{"table":"swap/ticker","data":[{"last":"7356.2","open_24h":"7134.8","best_bid":"7356.1","high_24h":"7432.3","low_24h":"7133.7","volume_24h":"4536721","volume_token_24h":"62179.4746","best_ask":"7356.2","open_interest":"850804","instrument_id":"BTC-USD-SWAP","timestamp":"2020-04-08T21:26:57.481Z","best_bid_size":"810","best_ask_size":"3","last_qty":"1"}]}
{"table":"swap/ticker","data":[{"last":"7356.4","open_24h":"7134.8","best_bid":"7356.1","high_24h":"7432.3","low_24h":"7133.7","volume_24h":"4536722","volume_token_24h":"62179.4881","best_ask":"7356.2","open_interest":"850804","instrument_id":"BTC-USD-SWAP","timestamp":"2020-04-08T21:26:59.692Z","best_bid_size":"810","best_ask_size":"3","last_qty":"1"}]}
^C

Here is also optimized Perl-based version of the decompressor:

$ echo '{"op": "subscribe", "args":["swap/ticker:BTC-USD-SWAP"]}' | target/debug/websocat -n wss://real.okex.com:8443/ws/v3
 --base64 |  perl -wne 'use strict; use MIME::Base64; use IO::Uncompress::RawInflate qw(rawinflate);  my $b = decode_base64($_); open B, "<", \$b; my $
o=""; open C, ">", \$o; rawinflate(*B,*C); print "$o\n"'
{"event":"subscribe","channel":"swap/ticker:BTC-USD-SWAP"}
{"table":"swap/ticker","data":[{"last":"7330","open_24h":"7201","best_bid":"7331.9","high_24h":"7432.3","low_24h":"7135","volume_24h":"4524856","volume_token_24h":"61997.7287","best_ask":"7332","open_interest":"837833","instrument_id":"BTC-USD-SWAP","timestamp":"2020-04-08T21:54:05.005Z","best_bid_size":"40","best_ask_size":"1","last_qty":"0"}]}
{"table":"swap/ticker","data":[{"last":"7332","open_24h":"7201","best_bid":"7331.9","high_24h":"7432.3","low_24h":"7135","volume_24h":"4524887","volume_token_24h":"61998.1512","best_ask":"7332","open_interest":"837833","instrument_id":"BTC-USD-SWAP","timestamp":"2020-04-08T21:54:19.690Z","best_bid_size":"161","best_ask_size":"31","last_qty":"10"}]}
^C

aldanor · 2022-06-11T23:42:15Z

Would it be possible to add an optional deflate layer? (since there's many gzipped websocket servers these days, like okex, huobi etc)

vi · 2022-06-12T02:06:13Z

Note that Websocket protocol itself provides permessage-deflate compression (not supported by Websocat v1 or v3 unfortunately).
Why do those servers compress data within WebSocket messages instead of using compression protocol extension?
Do all of them use gzip compression (and not, for example, lzma or brotli or zstd or lz4), so that just one additional overlay enables access to multiple services?

aldanor · 2022-06-15T16:14:01Z

@vi I'm not sure what's the answer to the "why" question, but it seems to be growing more and more popular for whatever reason. And yes, pretty much all of them use gzip and not any other compression method - so having a simple gunzip: layer would cover all of those cases and would be immensely useful.

(if deflate layer was implemented, it would also allow you to combine it with timestamp: layer, IIUC, so you could have a stream of deflated and timestamped messages)

aldanor · 2022-06-15T16:18:00Z

A possible answer to "why" could be that not all libraries and clients (only a selected subset) will support permessage-deflate. Meanwhile, you can always use any websocket library and just unpack things manually using whatever tooling you want to use.

vi · 2022-06-15T16:30:45Z

Is there a public service supplying such "gzipped" WebSocket messages to test implementations?

wss://real.okex.com:8443/ws/v3 no longer works.

aldanor · 2022-06-15T18:10:19Z

This should work:

(echo -e '{"op":"subscribe","args":["spot/ticker:ETH-USDT"]}' && cat) \
  | websocat wss://real.okcoin.com:8443/ws/v3 --base64 \
  | xargs -n1 -- sh -c '{ echo H4sIAAAAAAAAAA== | base64 -d; echo $0 | base64 -d; } | gunzip 2> /dev/null; echo'

This should also work:

(echo -e '{"sub":"market.btcusdt.bbo","id":"1"}' && cat) \
  | websocat --base64 wss://api.huobi.pro/ws \
  | xargs -n1 -- sh -c '{ echo $0 | base64 -d; } | gunzip; echo'

aldanor · 2022-06-15T18:15:45Z

By the way, having to send a single "subscribe" message (or a few of them) is also extremely common, wonder if it's something that could be integrated in so you wouldn't have to (echo -e foo && cat) | websocat? (which is extremely unobvious if you're not aware of sending EOF and the stream closing)

E.g. websocat --send-text 'foo' --send-text 'bar' to simply have it send a few messages on connection.

vi · 2022-06-15T18:19:49Z

Specifying init messages on command line, to send them to WebSocket before reading further messages from stdin (unless -u) does indeed look like a reasonable feature.

vi · 2022-09-24T09:42:01Z

Implemented --preamble (-p) / --preamble-reverse (-P) options, which work like --send-text above.

vi · 2022-09-24T10:36:11Z

Implemented decompression as well.

$ websocat -nU --max-messages-rev=3  wss://real.okcoin.com:8443/ws/v3 --uncompress-deflate   -p '{"op":"subscribe","args":["spot/ticker:ETH-USDT"]}'
{"event":"subscribe","channel":"spot/ticker:ETH-USDT"}
{"table":"spot/ticker","data":[{"last":"1328.45","open_24h":"1305.39","best_bid":"1326.07","high_24h":"1339.53","low_24h":"1263.63","open_utc0":"1328.22","open_utc8":"1288.69","base_volume_24h":"1048.726528","quote_volume_24h":"1369725.083642","best_ask":"1327.39","instrument_id":"ETH-USDT","timestamp":"2022-09-24T10:32:03.130Z","best_bid_size":"0.75","best_ask_size":"0.018834","last_qty":"2.634"}]}
{"table":"spot/ticker","data":[{"last":"1328.45","open_24h":"1305.39","best_bid":"1325.03","high_24h":"1339.53","low_24h":"1263.63","open_utc0":"1328.22","open_utc8":"1288.69","base_volume_24h":"1048.726528","quote_volume_24h":"1369725.083642","best_ask":"1325.87","instrument_id":"ETH-USDT","timestamp":"2022-09-24T10:33:03.074Z","best_bid_size":"0.6281","best_ask_size":"0.018857","last_qty":"2.634"}]}

$ websocat -nU --max-messages-rev=3  wss://api.huobi.pro/ws  --uncompress-gzip -p '{"sub":"market.btcusdt.bbo","id":"1"}'
{"id":"1","status":"ok","subbed":"market.btcusdt.bbo","ts":1664015732450}
{"ping":1664015732924}
{"ch":"market.btcusdt.bbo","ts":1664015733639,"tick":{"seqId":159767128361,"ask":19035.2,"askSize":0.11906853906499539,"bid":19035.19,"bidSize":0.38414,"quoteTime":1664015733638,"symbol":"btcusdt"}}

aldanor · 2022-09-27T16:40:08Z

@vi Looks awesome, thanks!

vi added a commit that referenced this issue Apr 6, 2020

--binary-prefix --text-prefix --base64 --base64-text

fd548b2

Partially addresses #78, #47. Closes #20.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[question] Trying to get last price stream from Okex #78

[question] Trying to get last price stream from Okex #78

nagualcode commented Apr 5, 2020

vi commented Apr 5, 2020 •

edited

Loading

vi commented Apr 6, 2020 •

edited

Loading

vi commented Apr 6, 2020 •

edited

Loading

nagualcode commented Apr 8, 2020

vi commented Apr 8, 2020 •

edited

Loading

aldanor commented Jun 11, 2022

vi commented Jun 12, 2022 •

edited

Loading

aldanor commented Jun 15, 2022 •

edited

Loading

aldanor commented Jun 15, 2022

vi commented Jun 15, 2022

aldanor commented Jun 15, 2022

aldanor commented Jun 15, 2022

vi commented Jun 15, 2022

vi commented Sep 24, 2022

vi commented Sep 24, 2022

aldanor commented Sep 27, 2022

[question] Trying to get last price stream from Okex #78

[question] Trying to get last price stream from Okex #78

Comments

nagualcode commented Apr 5, 2020

vi commented Apr 5, 2020 • edited Loading

vi commented Apr 6, 2020 • edited Loading

vi commented Apr 6, 2020 • edited Loading

nagualcode commented Apr 8, 2020

vi commented Apr 8, 2020 • edited Loading

aldanor commented Jun 11, 2022

vi commented Jun 12, 2022 • edited Loading

aldanor commented Jun 15, 2022 • edited Loading

aldanor commented Jun 15, 2022

vi commented Jun 15, 2022

aldanor commented Jun 15, 2022

aldanor commented Jun 15, 2022

vi commented Jun 15, 2022

vi commented Sep 24, 2022

vi commented Sep 24, 2022

aldanor commented Sep 27, 2022

vi commented Apr 5, 2020 •

edited

Loading

vi commented Apr 6, 2020 •

edited

Loading

vi commented Apr 6, 2020 •

edited

Loading

vi commented Apr 8, 2020 •

edited

Loading

vi commented Jun 12, 2022 •

edited

Loading

aldanor commented Jun 15, 2022 •

edited

Loading