ws2_32: HACK: Hold a BattlEye ack for The Crew Motorfest.

[PhialsBasement]

The Crew Motorfest (appid 2698940) resends its BattlEye report every 7s until the server kicks it at 210s, whenever it sees the ack of the 94 byte control record sent before the report ahead of fragment 0. Leave that ack queued in the socket until fragment 0 has been sent.

ValveSoftware/Proton#9119

@gofman reworked this after digging further, turns out the missing fragments were never what got you kicked. The client resends the whole report every 7s if it sees the ack of the 94 byte control record before fragment 0 goes out, and the server drops it after 31 resends. On Windows the ack lands after fragment 0 because BE builds the report faster, which is why you saw it send once and stop. On Proton BEClient spends ~500ms on the main thread building it (perf, Wine is 3% of that), so the ack always lands first. So the hack is now just leave that one ack queued until fragment 0 has been sent. 56 lines, no RTT or timing involved, appid gated, once per session, and inert if any packet size changes. Should fulfill the criteria since its fairly maintainable.

[gofman]

Well, this is not a matter of bargaining the number of lines. TBH we don't have any criteria for hack acceptance, if anything, we try to avoid adding Proton and game specific hacks whenever possible and work on proper fixes itself in the right place so that improves Proton and makes it more compatible or performant for any games instead of making it worse by slapping some ad-hoc tweaks which happen to help. The hack still fiddles with ad-hoc timing based on ad-hoc conditions which are based are gory specifics of internal game and BE functioning.

It perhaps would be very interesting if you could elaborate a bit more / in a stricter way on those conditions, I can't say I am following exactly what is the failure condition, how do you identify the type of package and what do you think is expected result with quicker or slower data send request. Maybe it will be useful for future reference. Or what if mentioned 500ms, depending on some details, is something actually improvable in Proton (the time actually involves some Wine calls) or maybe on BE side. In that case that would maybe open the possibility for an improvement outside of any game specific hacks.

[gofman]

WRT the details, note that on Windows the initial pattern, without artificial packet delays, is the same as I see on Proton here, it is interesting, if you know that, how exactly things go different on Proton, in the presence of packet delays when things do look different vs Windows in terms of resends / failure.

[PhialsBasement]

how do you identify the type of package

By payload size on port 4000. 1038 in is the BE query, 94 out is a control record sent 10 to 16 ms after it, 62 in with byte 2 = 0xdb is its ack, 1118 out are the 20 report fragments on a 100 ms tick, the first one 486 to 524 ms after the record.

what is the failure condition ... what do you think is expected result with quicker or slower data send request

The client seeing that ack before it sends fragment 0. It then resends the whole report every 6.9 s until the server terminates at +209.5 s. On Proton at 220 ms RTT the ack lands 280 ms before fragment 0 every time. If the ack reaches the client within about 80 ms after fragment 0 (the hook, or tc delaying the record 330 ms) it sends the report twice and stops. 206 ms after fragment 0 is too late and the loop returns. The same six fragments are dropped either way.

what if mentioned 500ms, depending on some details, is something actually improvable in Proton

The 500 ms is BE building the report. perf over that window shows the main thread busy 410 ms with 716 of 842 samples in BEClient_x64.dll and 19 syscalls total, so no Wine call is on that path. ((unless you have contacts over at BE))

how exactly things go different on Proton, in the presence of packet delays

Your Windows client sent once and stopped, so its ack arrived after fragment 0, meaning the Windows BE client builds the report faster than the added RTT. The gap from the outgoing 94 byte record to the first outgoing 1118 in your capture would confirm that.

[PhialsBasement]

No bargaining intended on the line count btw, I hated the 319 line version more than you did. Mostly I wanted to get to the bottom of it, and I finally finished a summit in one sitting yesterday, so the week wasn't wasted either way.

[gofman]

FWIW I looked a bit at this mentioned ACK of received size 62 and I don't see it affecting just anything in BattleEye conversation per se. I don't think that ACK is relayed to BE anyhow. The actual BE server reply for successfully receive sequence has a different length and only arrives (at least under Proton) after all the data packets were successfully sent. BE will retry sending everything each 5 sec in the absence of BE server reply (which is only received, as far as I can tell, once all the parts were delivered). From what I am seeing I am guessing that under repro conditions (which I still do not have outside of artificially delaying packets) it just tweaks send / ack timing in a way that breaks unfortunate pattern with game buffering vs big latency and allows the missing parts to be resent eventually on consequent tries (i. e., in principal similar to what original patch was doing just injecting the delays differently).

[PhialsBasement]

I am guessing that under repro conditions it just tweaks send / ack timing in a way that breaks unfortunate pattern with game buffering vs big latency and allows the missing parts to be resent eventually on consequent tries

Yeah you called it, I pulled the AES key out of the running process and decrypted the relay stream and read the part indices off the record headers and confirmed that all 20 parts arrive across two rounds, so the backend (BE) stops retrying. Holding the ACK occupies one send slot and changes the round 0 drops, allowing round 1 to fill the gaps. The 330 ms delay worked for the same reason. My earlier conclusion was based solely on packet timing. This is a workaround for the game's two-slot send buffer.

With this patch it sends the following
On round 0: 0 2 4 5 7 8 10 11 13 14 16 17 19
On round 1: 0 1 3 4 6 7 9 10 12 13 15 16 18 19

[PhialsBasement]

I've amended the commit msg to reflect that

[gofman]

For the reasons discussed earlier such sort of a hack is still a no go (and if that is not enough, delaying some game interaction traffic has the potential of breaking things for reasonable latency connections, even if not right now possibly after slight changes in game).

We will probably try to find a way to relay the problem to game developers (as it looks clear that the game is loosing packets, how that happens and how that leads to the issue). No idea if that is going to be a priority or ETA on that, sorry.

[PhialsBasement]

Understood, closing this. Thanks for taking it to the developers.

Side question, would you be alright with me linking a prebuilt Proton with this patch from my repo in the #9119 thread, marked as a temporary workaround for high latency players until the game is fixed? Same build as the current bleeding-edge with only this commit on top. If you'd rather keep the thread to the tc rule I'll leave it at that.

[gofman]

Proton is opensource, making downstream builds does not require my or anyone else's permission :) The only restrictions are from open source LGPL license, that is, you are supposed to provide the source code for your modifications (having source modification in the repo with the build is one of the ways to do it).

[PhialsBasement]

Ha, yeah I know the licence part, the source is right here on the branch. What I meant was throwing the build up as a GitHub release on my repo and linking that in the #9119 thread. It's your tracker so I figured I'd ask before parking a random link to my binary in it.

[gofman]

Moderating issue tracker is absolutely not my territory but AFAIK various downstream builds are frequently mentioned and are not so far banned from that.