Network desynchronisation

[Falcon]

Hi,
 I've encountered 3 times a critical desynchronisation during an Online session in the last 5 months or so.
I think it has to be on my side, as it happened to my PC and not to anyone else during those sessions. The desynchronisation is severe and it practically creates a parallel session (sometimes, a few things synchronize over time, but after quite a long time - for example map markers, new vehicle wrecks, my tank suddenly getting hit - because it was moving in the host's session into the enemy units while in mine it was out of any engagement, behind a hill in cover, etc...).

I've reinstalled the SB after the 2nd desync. occasion. It was when the latest update came out. I uninstalled it, searched and deleted all remaining files, even cleaned my registry (via CCleaner, so not really that perfectly) and installed a fresh new version of SB. It did help for ca 2 months but it is back again and I don't know what to do next to prevent it.

My internet connection is an average one, I have 42 Mbps download and 10 Mbps upload speed. I don't observe this in any other software. During the desynchronisation, I'm running a TeamSpeak as well (and I never noticed a "lag" in TS during the desync.).


There is a one curiosity, which I can't logically explain, but every time it happened I was in a Leopard 2A4.

Did anyone encountered this problem? Any tips? Otherwise, I think I will start having a holy water next to me when driving the tank again ...

Thank you

[Ssnake]

If a network game experiences an interruption in the connection between Host and Client, Steel Beasts switches into single player mode to allow you to continue playing. Of course, from that moment on it's two different games. If you are on Teamspeak and the voice communication is not interrupted, and if the enemy is scripted and it's some sort of a cooperative session, I can see how the "seamless disconnect" (which we thought was a nice feature at a time (ca. 2000) when internet connections were still less common, low bandwith, and generally unreliable) would go unnoticed, and could then create the impression that it's a serious bug when the sessions are drifting apart and you suddenly feel like you were in the Twilight Zone.

[Lumituisku]

Umm... Is it something like this? 

 

Unit you own is in point A. You route and observe it to move (on map) to point B. 

After say 5 minutes you want to tell it to go somewhere but can't find it because it's in point  A... In state that suggest that it never moved from there. and has no route to point B that you had created for it? 

 

Something like this happened to me yesterday multiple times with one particular MT-LB troop carrier and i just cannot understand it.

 

I had lots of units but this one just kept being one or two steps back for no obvious reason. Also i was a host too and it was MP session. 

 

---

 

Back to what Falcon says. I think i have read similar report from someone in forum. And it too was Leo 2A4. Perhaps yours? 

 

Also my friend with whom i played opfor on kanium game two weeks back complained something similar to what i just described. But with twist that sone units were heading of the route at times without obvious reason and constantly.

We both had our hands full so i cave that little though back then but now it seems that it is similar to what i experienced. 

[Falcon]

On 5/20/2023 at 9:32 AM, Ssnake said:

If a network game experiences an interruption in the connection between Host and Client, Steel Beasts switches into single player mode to allow you to continue playing. Of course, from that moment on it's two different games. If you are on Teamspeak and the voice communication is not interrupted, and if the enemy is scripted and it's some sort of a cooperative session, I can see how the "seamless disconnect" (which we thought was a nice feature at a time (ca. 2000) when internet connections were still less common, low bandwith, and generally unreliable) would go unnoticed, and could then create the impression that it's a serious bug when the sessions are drifting apart and you suddenly feel like you were in the Twilight Zone.

Hello Ssnake,
 thank you for your answer and for your confirmation that it happens due to an interruption in network connection.

Nevertheless, if the Steel Beasts switches into SP mode, I therefore experienced many game-breaking issues where some informations were still flowing (from host to client direction only) and affecting my "new SP" session. From my understanding of your reply, this should not be happening as I should be continuing the scenario in my own "parallel SP" session, thus being fully isolated from anything happening in the (ex-)host's session.

I observed these concrete informations being transmitted into my "new SP" session from the host's one:
- tank gunfire sounds
- sharing map markers
- newly sustained vehicle damage
- not sure in this one, but I think the "newly created" vehicle wrecks were appearing in my session as well
- (ex-)host ending his session also ended mine

All these informations "reached" my session with some significant time delay (a few minutes). I had no severe internet connection issue as I was on the TeamSpeak all the time, communicating with others.

My keyboard input to control the vehicle (movement, controlling GPS, switching roles in tank) was also significantly delayed and appeard to be "acknowledged" in chunks - nothing happening and then all of sudden all the control inputs happening at once - so I was perhaps sending my "control input" to (ex-)host and then it travelled back to my session?

Also, a few vehicles in my "new SP" session, which were previously under other player's control, were continuing in their motion, driving in a straight line on the surface of terrain, not interacting with any object on the map ("ghosting through them"). Their "physics" were very odd, as they were sometimes even sliding sideways (while still moving in the same direction). The movement was linear and not changing during the remaining time of session (30 minutes or so).
Similar issue was happening to my tank in host's session.


Perhaps a few ideas, if you want to repair/further develop this function:
- give unit control to the person in "new SP" session of the units from all players from host's session (and vice versa), as in most time, it is impossible to finish the scenario with only a partial strenght / single tank
- the game should notify the person that he is no longer in host's session, as then the desynced player can immediately tell other players that he is out of game


I hope this is of any help.
Thank you for your time reading this.

Have a nice day

Edited May 21, 2023 by Falcon

[Falcon]

Hello Lumi,
 thank you for input here.

 

On 5/20/2023 at 10:09 AM, Lumituisku said:

Umm... Is it something like this? 

 

Unit you own is in point A. You route and observe it to move (on map) to point B. 

After say 5 minutes you want to tell it to go somewhere but can't find it because it's in point  A... In state that suggest that it never moved from there. and has no route to point B that you had created for it? 

 

Something like this happened to me yesterday multiple times with one particular MT-LB troop carrier and i just cannot understand it.

 

I had lots of units but this one just kept being one or two steps back for no obvious reason. Also i was a host too and it was MP session. 

 

---

I can't confirm loosing the path of "move order". But I observe a moments where my tank "teleports" to position where it was in in a few minutes back, and then teleporting it back to it's current position. It is happening after I desynchronize.

It is interesting that a same thing happens even when the host occurs "Network Overload" issue (green text on top left end of screen). But in these cases, it is "resynchronized" in a second or two.

It appears to me like it is trying to "resynchronize", thus the teleportation.


And just to clarify, I was a client, not a host. In all 3 of these instances.


 

 

On 5/20/2023 at 10:09 AM, Lumituisku said:

---

 

Back to what Falcon says. I think i have read similar report from someone in forum. And it too was Leo 2A4. Perhaps yours? 

 

Also my friend with whom i played opfor on kanium game two weeks back complained something similar to what i just described. But with twist that sone units were heading of the route at times without obvious reason and constantly.

We both had our hands full so i cave that little though back then but now it seems that it is similar to what i experienced. 

I'm not sure to which exact post are you reffering to, but I've been talking about this issue previously in forum. It was a discussion about one of the newly released SB version (I think 1 version back from the current one?).


Also a question - that Kanium session 2 weeks back, was it the Estonian one, where i was BluFor CO? There might be a connection.


Anyway, thank you for your input and time as well.

Edited May 21, 2023 by Falcon

[Falcon]

After a discussion with Lumi, I am also pointing out that I have Kaspersky Free Antivirus.

I had 1 problem with my antivirus in the past regarding SB - it marked SB installer .exe as a Troyan virus. But, since it is a known issue with Avast, after researching it being a false positive, I've decided to ignore this message. Obviously, nothing bad happened, and a deep scan after installation (and deleting installer file) showed nothing.

Edited May 21, 2023 by Falcon

[Ssnake]

Yeah, CodeMeter-protected applications are partially encrypted, which can trigger relative simplistic heuristics. A lot of malicious software is encrypted to protect against AV scans ... so the logical fallcy is "if it's encrypted, it must be a virus"...

[Ssnake]

I agree that the described symptoms suggest that there was NO disconnect between Host and Client.

Having a look at the debugLog.txt of that session (search for "ERROR") might yield a clue (you should make a backup soon after such a session, and if you have several of them we can start looking for identical event chains in them). Steel Beasts keeps the last 12 copies in the logs folder, but of course at some point the oldest ones get overwritten.

 

If the whole issue always happens in conjunction with "network overload", well, there isn't anything that we can do except recommending that you run sessions with fewer clients. Once that network packages are lost, all kinds of "funny stuff" may happen.

[Lumituisku]

In my case I found this as the event happened yesterday ...  also session where MTLB anomaly happened was just with me and friend.  So downsizing multiplayer session is hardly possible.  

 

That error seems to be same error happening several times over couple of days.  Interesting.  

 

Also I am not 100 % sure that my case is similar. I witnessed problem on map...  not on 3D world, I was way too busy to jump on vehicle in 3D world. Even when my case sounds similar I have some doupts of it. 

Edited May 21, 2023 by Lumituisku

[Falcon]

On 5/21/2023 at 5:29 PM, Ssnake said:

I agree that the described symptoms suggest that there was NO disconnect between Host and Client.

Having a look at the debugLog.txt of that session (search for "ERROR") might yield a clue (you should make a backup soon after such a session, and if you have several of them we can start looking for identical event chains in them). Steel Beasts keeps the last 12 copies in the logs folder, but of course at some point the oldest ones get overwritten.

 

If the whole issue always happens in conjunction with "network overload", well, there isn't anything that we can do except recommending that you run sessions with fewer clients. Once that network packages are lost, all kinds of "funny stuff" may happen.

I'm sending the Debug logfile.

It is quite a large file, but it consists of 1 Error and 3 Warning message types repeating itself during the session (which started at approx. 20:25, desynchronized at approx. 22:15 and ended at approx. 23:00, given time is a gross estimation). Here are the examples of such messages:

[20:26:56,372] CNetwork ERROR: [LOCAL] Ping time too large (2971 ms)!

[20:47:18,310] CGame WARN : CTimeKeeper::UpdateTime(): forcing monotonic mission time, host reported time offset of -59 capped to -25

[21:14:18,880] CGame WARN : CTimeKeeper::UpdateTime(): offset mission time by 714 to 1627122

[22:16:45,824] CNetwork WARN : Message of type MYMSGID_BUILDINGMOD (58) was sent 121 times in the last 5000 ms [FPS=53.92 (65.06 - 41.57)]!

Last warning message appears a few times after multiple Error messages with gradually increasing latency time (signalizing the moment of "desynchronisation"?).
Similar Warning message appears later a few times as well:

[22:26:48,618] CNetwork WARN : Message of type MYMSGID_BUILDINGMOD (58) was sent 121 times in the last 5000 ms [FPS=37.74 (39.62 - 35.72)]!




note1:
Since I am posting this file online, I've replaced the host's IP address with ***DELETED_BY_FALCON***, which was logged at the end of the logfile.

note2:
I've checked my last 12 debug logfiles for this exact error message and it appears only when joining to one exact host only. Other Online Session's logfiles don't contain it. I didn't check for Warning messages though. If needed, I can provide these as well.

note3:
I was the only person noticing this issue of desynchronisation. Everyone else was able to finish the scenario without any problem.



@Lumituisku
I didn't have error you've mentioned in my logfile. Perhaps it is not related 🤷‍♂️
 

 

Edited May 26, 2023 by Falcon

[mrivers]

20 hours ago, Lumituisku said:

In my case I found this as the event happened yesterday ...  also session where MTLB anomaly happened was just with me and friend.  So downsizing multiplayer session is hardly possible.  

 

That error seems to be same error happening several times over couple of days.  Interesting.  

 

Also I am not 100 % sure that my case is similar. I witnessed problem on map...  not on 3D world, I was way too busy to jump on vehicle in 3D world. Even when my case sounds similar I have some doupts of it. 

This one is harmless.  I bit of framework code is overzealous in reporting it cannot find a 3d resource it didn't ask for.   (Component missing blankframe '<noname>').  If similar reports exist where a name is supplied, these we would want to investigate, however even these would not harm a SP or MP game.

[mrivers]

Quote

It is quite a large file, but it consists of 1 Error and 3 Warning message types repeating itself during the session (which started at approx. 20:25, desynchronized at approx. 22:15 and ended at approx. 23:00, given time is a gross estimation). Here are the examples of such messages:

[20:26:56,372] CNetwork ERROR: [LOCAL] Ping time too large (2971 ms)!

[20:47:18,310] CGame WARN : CTimeKeeper::UpdateTime(): forcing monotonic mission time, host reported time offset of -59 capped to -25

[21:14:18,880] CGame WARN : CTimeKeeper::UpdateTime(): offset mission time by 714 to 1627122

[22:16:45,824] CNetwork WARN : Message of type MYMSGID_BUILDINGMOD (58) was sent 121 times in the last 5000 ms [FPS=53.92 (65.06 - 41.57)]!

...

Quote

note2:
I've checked my last 12 debug logfiles for this exact error message and it appears only when joining to one exact host only. Other Online Session's logfiles don't contain it. I didn't check for Warning messages though. If needed, I can provide these as well.

note3:
I was the only person noticing this issue of desynchronisation. Everyone else was able to finish the scenario without any problem.

 

This gives the impression that there is some connectivity issue between you and this specific host, and perhaps geographic locations might explain why other players aren't seeing an issue.   You mentioned TeamSpeak continued to work, but I expect that is relayed through a different host than SB (which is a direct connection).

 

It's interesting that the ping time increase to 13 minutes, and yet the connection isn't dropped (which should happen at 30s).

 

It's not really possible to tell if the MYMSGID_BUILDINGMOD warnings are related, a cause of, or side-effect.  Generally, the warnings would indicate that a lot of buildings are having their status changed if there isn't a bug.

 

Can you provide the scenario so we can further investigate it?

Edited May 22, 2023 by mrivers

[Falcon]

On 5/22/2023 at 5:37 PM, mrivers said:

This gives the impression that there is some connectivity issue between you and this specific host, and perhaps geographic locations might explain why other players aren't seeing an issue.   You mentioned TeamSpeak continued to work, but I expect that is relayed through a different host than SB (which is a direct connection).

 

It's interesting that the ping time increase to 13 minutes, and yet the connection isn't dropped (which should happen at 30s).

 

It's not really possible to tell if the MYMSGID_BUILDINGMOD warnings are related, a cause of, or side-effect.  Generally, the warnings would indicate that a lot of buildings are having their status changed if there isn't a bug.

 

Can you provide the scenario so we can further investigate it?

Thank you for looking into it.

I've attached the scenario file to this message. It is from my .../My Scenarios/downloads/ folder, so it should be the exact played scenario version.
- a question: Should I delete the file from this forum discussion when this issue is solved? I was told that it is better to do so in order to not use the forum "disk space".


I will ask the host if the TS server is running on the same PC (or some local one) as the SB sessions.
However, when I'm in the lobby of these sessions, my shown ping time is approx. 10 - 20 ms, which makes sense since we are from neighbouring countries in Europe.


It is perhaps interesting to note that I can join and play even when the host (different one) is across the world (with ping time approx. 300 - 350 ms). Only with the visual "vehicles teleportation" happening from time to time, which is expectable with this ping and is always "automatically re-synchronised" after max. 2 seconds. I have a debug file for one of these sessions as well. It has only a few (6) Errors "Ping time too large" with maximum ping time 2 s. Then there are a lot of warnings about "forcing monotonic mission time" (same as in the problematic session).
 

 

Edited May 26, 2023 by Falcon

[Falcon]

1 hour ago, Falcon said:

I will ask the host if the TS server is running on the same PC (or some local one) as the SB sessions.

The TS server and SB server "applications" are running on the same machine


In the next session (Friday), I will try to disable my Kaspersky Antivirus. Maybe it is the cause.

[Gibsonm]

1 hour ago, Falcon said:

The TS server and SB server "applications" are running on the same machine

 

Are you sure?

 

The IP addresses are different.

[Falcon]

4 minutes ago, Gibsonm said:

 

Are you sure?

 

The IP addresses are different.

I've asked @Abraxas

If I understand the answer correctly, they are on the same machine.

[Gibsonm]

2 minutes ago, Falcon said:

I've asked @Abraxas

If I understand the answer correctly, they are on the same machine.

 

Perhaps they are running VLANs or something but:

 

The TS server is on: 62.104.XXX.XXX

 

While the session was hosted on:  93.205.XXX.XXX or similar.

 

[mrivers]

On 5/23/2023 at 1:43 PM, Falcon said:

Thank you for looking into it.

I've attached the scenario file to this message. It is from my .../My Scenarios/downloads/ folder, so it should be the exact played scenario version.
- a question: Should I delete the file from this forum discussion when this issue is solved? I was told that it is better to do so in order to not use the forum "disk space".


I will ask the host if the TS server is running on the same PC (or some local one) as the SB sessions.
However, when I'm in the lobby of these sessions, my shown ping time is approx. 10 - 20 ms, which makes sense since we are from neighbouring countries in Europe.


It is perhaps interesting to note that I can join and play even when the host (different one) is across the world (with ping time approx. 300 - 350 ms). Only with the visual "vehicles teleportation" happening from time to time, which is expectable with this ping and is always "automatically re-synchronised" after max. 2 seconds. I have a debug file for one of these sessions as well. It has only a few (6) Errors "Ping time too large" with maximum ping time 2 s. Then there are a lot of warnings about "forcing monotonic mission time" (same as in the problematic session).
 

Blauer Phoenix Leo2.sce 1.36 MB · 1 download

Thank you for providing the scenario, I'll look into it next week (unfortunately no time this week), and perhaps it will yield some clues.  I've downloaded the scenario, so you may delete at your discretion.

 

I have one more request, sorry I didn't think to ask earlier.  Can you provide an AAR of a problem session, this might help nail down what's going on with the MYMSGID_BUILDINGMOD warnings.

 

[Gibsonm]

25 minutes ago, mrivers said:

Thank you for providing the scenario, I'll look into it next week (unfortunately no time this week), and perhaps it will yield some clues.  I've downloaded the scenario, so you may delete at your discretion.

 

I have one more request, sorry I didn't think to ask earlier.  Can you provide an AAR of a problem session, this might help nail down what's going on with the MYMSGID_BUILDINGMOD warnings.

 

 

Unsure if it helps but here is a "normal" AAR (I was in Falcon's Platoon, but not impacted) for comparison:

 

https://u.pcloud.link/publink/show?code=XZpve6VZbfsiCT2HcLjqGtnX5yCqLmB7Y4skhttps://u.pcloud.link/publink/show?code=XZpve6VZbfsiCT2HcLjqGtnX5yCqLmB7Y4sk

 

Here is my Debug Log from the session (no "MYMSGID_BUILDINGMOD" warnings, that I could see, for me):

 

DebugLog_5.txt

 

I was connecting to Germany from Australia. Falcon was also connecting to Germany but from Czech.

 

Edited May 26, 2023 by Gibsonm

[Falcon]

22 minutes ago, mrivers said:

Thank you for providing the scenario, I'll look into it next week (unfortunately no time this week), and perhaps it will yield some clues.  I've downloaded the scenario, so you may delete at your discretion.

 

I have one more request, sorry I didn't think to ask earlier.  Can you provide an AAR of a problem session, this might help nail down what's going on with the MYMSGID_BUILDINGMOD warnings.

 

Unfortunatelly, I didn't save my AAR.

Today, we are having another session (and with Leo2A4 as well). I will report how it went (and save AAR, just for a good measure).

[Sean]

If kaspersky is acting as a proxy, and "reading" all your web traffic, that could be causing this.  

[Falcon]

@mrivers
 

We've played 90 minutes long scenario with Leopard 2A4 (from circa 20:30 to 22:00)
I'm sending you the link to the RAR file, which contains my DebugLog, my AAR, @Gibsonm AAR (we were in same platoon) and the scenario itself.

https://ufile.io/36iqc9lv

What I've observed:
- in-game desync behaviour in the last circa 10 seconds (delayed seat-switch control inputs), only on my side

- DebugLog shows high-ping errors (8-12s) in the last circa 4 seconds

- no MYMSGID_BUILDINGMOD, but a single warning of MYMSGID_ASSIGNEMPLACEMENTS at 20:57:44,798

- a lot of warnings about null IDs (perhaps scenario related?)

- my and Gibsonm's AAR looks same up until the very last second:
At 1:30:01, Gibsonm's AAR reports me destroying vehicle (5/021). My AAR doesn't show this hit and at it's end (1:30:04) the vehicle is alive. I remember that my gunner was shooting at something at the very end of scenario. At that moment, I was already having the delayed control input issues. This corresponds with this error message in the log, which reappears even when looking at AAR:
[22:04:14,556] CActorListArray WARN : Tried to retrieve combatant with NULL ID!


 

Before starting SB (and joining the session), I've disabled my Kaspersky's protection and quitted the application. However, Kaspersky Secure Connection application was still running on the background (I've noticed that after the session). It is a VPN connecting tool, which I am not using - I've double-checked that.



So, it is replicable, possibly connected to Leo2A4 (I want to test this next session, since it happened twice in a row at +- the same time) and Kaspersky Free Antivirus might not be the cause, although that VPN application running on the background could have do something - I can test it being turned off as well. However, to test it properly, I think I will have to uninstall it. But that is the step which I'm really not happy about and I would consider it only as the last resort. Kaspersky serves me and all my friends using it very well and I don't trust "naked" Windows to defend itself on nowadays internet.


note:
That last second vehicle destruction looks weird to me. I understand that at the end of online session, things may go haywire, so it might not be important at all. But why Gibsonm observed destruction and I didn't, when it was my tank destroying it? That suggests to me, that although my reported high-ping, information from my PC reached host. But perhaps the return ("destruction/hit confirmation"?) didn't reach my PC back?


Anyway, looking forward for the answer.

Edited May 26, 2023 by Falcon

[Abraxas]

I have the following statement regarding the connection:

 If I use my computer with the usual license as a server with up to 8 participants, there are no problems.

But when I use the license for more participants, problems have been occurring since version 4.379:

- it takes a very long time for the connection to be established;

- sometimes the scenarios are not fully uploaded (mission plans, paths, waypoints, triggers, conditions etc.)

- during the process strange things occur like loss of control or abnormal behavior of vehicles up to loss of connections.

Edited May 30, 2023 by Abraxas

[mrivers]

On 5/26/2023 at 6:47 PM, Falcon said:

@mrivers
 

We've played 90 minutes long scenario with Leopard 2A4 (from circa 20:30 to 22:00)
I'm sending you the link to the RAR file, which contains my DebugLog, my AAR, @Gibsonm AAR (we were in same platoon) and the scenario itself.

https://ufile.io/36iqc9lv

 

In the attached scenario, 4 gray units have invalid IDs.  I'm trying to determine how this could come about.

[mrivers]

21 hours ago, mrivers said:

In the attached scenario, 4 gray units have invalid IDs.  I'm trying to determine how this could come about.

 

For ROTER-HERBST_PzKp_ohne.sce, as mentioned, there are 4 gray units with invalid call signs.  The invalid callsigns are responsible for these messages:

 

Quote

[21:58:21,537] CNetwork ERROR:     The previous actor type: <xyz>
[21:58:21,715] CActorListArray WARN : Tried to retrieve combatant with NULL ID!
[21:58:21,715] CNetwork ERROR: Error receiving state block data. Can't find Zivil combatant with ID <xyz> 

 

There's a bug in the Mission Editor's 'Set unit name' menu that allows invalid call signs to be set, when combined with what appears to be incomplete 'Unit call signs' settings.

 

At this point, I don't know if there's a connection to the ping times logged.