Arcen Idea Tracker (Bugs and Features)

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryDate SubmittedLast Update
0002460AI War[All Projects] Bug - OtherJan 16, 2011 10:05 amMar 22, 2011 11:13 pm
ReporterMithror 
Assigned Tox4000Bughunter 
Severitymajor 
StatusresolvedResolutionfixed 
Product Version4.068 
Fixed in Version5.006 
Summary0002460: Unable to play multiplayer game.
DescriptionNo issues in 4.065, but after having upgraded to 4.068, we are now unable to play a game for longer than 5 minutes (which is a guestimate).

Included are two save games at a time of desync (Waiting for Players).

We've tried changing hosts, removing the router, using hamachi, disabling firewalls, etc...
TagsNo tags attached.
Internal Weight
Attached Files? file icon desync issue - Mithror.sav [^] (190,835 bytes) Jan 16, 2011 10:05 am
? file icon desync.sav [^] (190,879 bytes) Jan 16, 2011 10:06 am
? file icon desync_example2_host.sav [^] (223,583 bytes) Jan 17, 2011 4:17 pm
? file icon desync_example2.sav [^] (223,579 bytes) Jan 17, 2011 4:17 pm
txt file icon DesyncErrors_example2.txt [^] (515 bytes) Jan 17, 2011 4:19 pm [Show Content]
? file icon MoonKing_desync_01.sav [^] (244,875 bytes) Jan 17, 2011 4:55 pm
? file icon MoonKing_desync_02.sav [^] (244,919 bytes) Jan 17, 2011 4:55 pm
txt file icon MoonKing_DesyncErrors.txt [^] (6,482 bytes) Jan 17, 2011 5:18 pm [Show Content]
? file icon Desync_KingFox_01.sav [^] (160,279 bytes) Jan 24, 2011 2:36 pm
txt file icon Desync_KingFox_01.txt [^] (7,532 bytes) Jan 24, 2011 2:37 pm [Show Content]

- Relationships
related to 0003077resolvedx4000Bughunter Desync 
related to 0002480closedkeith.lamothe Multiplayer Connection Issues 
related to 0002546resolvedx4000Bughunter Scout Exporing -> Desynch 

-  Notes
(0008701)
x4000Bughunter (administrator)
Jan 16, 2011 11:32 am

Did it actually say desync in the stats, players screen? Also, was there anything in particular that you guys were doing at the time of the event? Sometimes that helps us track the desync down if that's what it was for sure.
(0008709)
Mithror (reporter)
Jan 16, 2011 11:53 am

Yes, it did say desync in the stats. I was host (mainly) and the descyn message was next to the other player (Kittens).

I don't think we were doing anything out of the ordinary. I believe we were ok as long as ships didn't fight, though now I think about it, I don't think the first wave caused this. The issue usually seems to occur when we move ships to another planet (which was an AI planet because we only had our home planets).
(0008710)
TechSY730 (reporter)
Jan 16, 2011 11:57 am

Mithror said: "I believe we were ok as long as ships didn't fight"

Well there is your problem. You clearly need to be making a passivist play-through. :D
(0008711)
Mithror (reporter)
Jan 16, 2011 11:59 am

I tried looking for the diplomacy option ("We cool, bro?"), but there wasn't one. Perhaps I should enter on issue on this :p
(0008712)
x4000Bughunter (administrator)
Jan 16, 2011 12:07 pm

Awesome. Thanks for the added details, I will look into this tonight. I didnt figure that anything you were doing was unusual, but a desync is usually (but not always) tied to some sort of player activity, unusual or not. Once in alpha it was literally if one player looked at a forcefield while ships spawned under it during a fight, and the other player did not. In this case, it sounds like it might be related to either combat or else going through wormholes. Could be related to the recent collision changes, or the forcefield ones. Wish me luck on it tonight -- these can be a devil of a time to find (see previous example)!
(0008713)
Mithror (reporter)
Jan 16, 2011 12:12 pm

I don't think it's the actual act of going through wormholes, but I can't be sure. I think it occurs after being on an other (enemy?) planet for some time (or performing on action there).
(0008728)
Mithror (reporter)
Jan 16, 2011 1:13 pm

I don't know if this helps or if it matters, but I did change the disabledaitypes.txt to disable entrenched homeworlder, tank and zenith descendant. (Coz we wanted new AI's)
(0008729)
x4000Bughunter (administrator)
Jan 16, 2011 1:14 pm

Shouldn't matter -- but thanks!
(0008732)
lyravega (reporter)
Jan 16, 2011 1:27 pm

We experienced something might be related to this. My connection was bad, so when I hosted, we were able to play for the first few minutes, then it says "waiting for players" but we were able to chat in-game. When we reload, use player managament, etc... we were able to play for another minute, then it happens again. Any ideas?
(0008734)
x4000Bughunter (administrator)
Jan 16, 2011 1:34 pm

It's the same thing, I expect. Any details on what was going on when your stuff was desyncing? What were you and the other player doing, was there fighting going on, etc?
(0008739)
kiwiattack (reporter)
Jan 16, 2011 2:59 pm

We had this exact same problem, and I reported it in another section. We lost about 2 hours trying to fix it ourselves, but clearly its a bug in the new patch.

We used himachi, and the most we got to was about 8 minutes in. It didn't seem to be connected to moving through AI ports, but it always happened when we had ships in enemy territory.
(0008749)
lyravega (reporter)
Jan 16, 2011 3:31 pm

When we're or AI are doing nothing, it was going up to 10+ minutes. But when a wave arrives, it is more of an instant.

And, this happened when we tried to load up a game. We can start a new game, play it without any problems, but after a load most of the time, we cannot do much due to desync issues, other than chat in game
(0008755)
TechSY730 (reporter)
Jan 16, 2011 4:09 pm

You mentioned that desyncs get almost the same priority as crashes? Does this mean there will be an emergency patch when this fixed?
(0008765)
Mithror (reporter)
Jan 16, 2011 5:17 pm

Just played 3 hours or so without an issue...
(0008766)
x4000Bughunter (administrator)
Jan 16, 2011 5:20 pm

In multiplayer?
(0008767)
Mithror (reporter)
Jan 16, 2011 5:38 pm

Indeed. Nothing really changed, except... my friend did reboot his modem because he was having high ping after downloading League of Legends. Which apparently was due to the download utility still seeding, so I'm not sure it is related. Anything's possible when it comes to network issues though :)
(0008769)
keith.lamothe (administrator)
Jan 16, 2011 5:46 pm

Ok, just want to make sure I got the right saves and properly understand:

I loaded the attached "desync issue - Mithror.sav" in my 4.068 copy (just a steam copy updated to the latest public version) and it's 1 minute and 28 seconds into the game and the only human units on AI planets are scouts. Only the green human player even has any ships capable of fighting (to wit, 89 mkI fighters next to their command station).

Then I loaded the attached "desync.sav" and it's identical as far as I can tell, which is what I would expect.

Just wanted to make sure that this is actually a point in time at which you got a desync, as I was getting the impression that it was related to moving ships onto enemy planets and/or combat.

Not to say that minor factions might be duking it out somewhere, just asking.
(0008770)
keith.lamothe (administrator)
Jan 16, 2011 5:50 pm

Hmm, so it may be that it only shows up at high pings... but that makes a distinct lack of sense because the simulation just sits on its rear and waits for the desired packets to come.

Corrupted packets would kill things in a hurry, I think, but you'd have to have a really awful connection for that to happen, I figure.
(0008771)
Mithror (reporter)
Jan 16, 2011 5:54 pm
edited on: Jan 16, 2011 5:55 pm

Yes, they're saves from the last time we got the desync issue. I believe it's also the first save of the new game we started (because we thought it might solve the issue). If there's no combat going on, then it might not be related to combat at all. Or it could be desync for another reason (like a modem fault, though other mediums like Mumble did not suffer from it). I just remember from previous desyncs that it usually occurs after we've started attacking another planet. I couldn't really remember if this game left a likewise impression.

If we ever have the issue again, we'll certainly upload new saves for you to analyze :)

edit:
High ping meant 300 in this case.

(0008772)
keith.lamothe (administrator)
Jan 16, 2011 5:56 pm

Thanks :)

And I don't think I've _ever_ seen an actual desync from a networking fault. I mean, anything's possible if you actually manage to get packets so messed up that they slip through the TCP/IP/etc error-checking, but I've never seen that actually happen in AI War, and certainly not in any kind of regular fashion.
(0008773)
x4000Bughunter (administrator)
Jan 16, 2011 5:57 pm

lyravega -- if it happens again, can you look in the Players tab of the Stats window (if you're the host), and see if it says Desync next to any of the player names? Or, if not, get the host to do it? It's possible that you're actually just running into network issues that need to be solved a different way, but I assumed it was a desync since we had two, possibly three, reports all at the same time.

We've been through the code more, however, and based on Mithor's and kiwiattack's further information this is seeming less and less like a typical desync event.

Just because we haven't found anything and just because it's not always reproducible doesn't mean it's not a desync, though -- by contrast, that's something that often happens with them, which is why they are so hard to track down.

Any added info folks can provide would be good. There's a general desync procedure posted here, which I realized was out of date since AI War 4.0, and so I've now updated as of today: http://www.arcengames.com/forums/index.php/topic,53 [^]
(0008774)
Mithror (reporter)
Jan 16, 2011 5:57 pm

Yeah, I figured as much, but every detail matters in this case, I guess :)
(0008776)
lyravega (reporter)
Jan 16, 2011 6:05 pm

x4000 - will do that.
(0008777)
x4000Bughunter (administrator)
Jan 16, 2011 6:06 pm

Ah, new comments while I was typing that last. A few questions:

Keith, did you compare the two save files with a diffing tool? Were they literally identical? Actually, know what -- we need to put in that Dump functionality in order to make use of any saves! That was in 3.x, and was never ported over to 4.0, I recall recently noticing. The idea is that when the F3 debug info is on, you go into the Load Game dialog and there is a dump button (this currently works). Then you select a relevant save and click dump, and it dumps the savegame to disk in a completely noncompressed format. Then we look for differences. Without that, there's nothing useful about getting savegames from folks except to see if we can duplicate the issue by running it between two of our own computers.

As far as corruption of packets causing a desync, I think that is highly unlikely. Most likely if corruption were to happen, it would simply cause a game crash, or at least errors to be logged. Mithror, can you and the other players look in your RuntimeData folder and see if there are any new error messages from the relevant time period? If so, then that could easily cause this sort of thing. If not, I think that probably network conditions are not causing this if it is actually a desync every time (it's possible that if you just saw waiting for players again and assumed it was a desync, it was actually just a network connection lost the second time, I expect).

Sometimes we run into a first-run-after-a-savegame-upgrade-to-new-version-only desync that goes away after the host uses manage players and then everyone reconnects, or after the host saves the game and then everyone reconnects. It's something we also like to fix, but it points us in a completely different direction than normal for a desync. If that was the case, then you'd keep seeing a desync if you just loaded the old save repeatedly, but other than that it wouldn't be desyncs after that first time.

Does anyone have any added info on what the last version of the game that seemed to work for them in multiplayer was? That you played on for a substantial amount of time? We'd been assuming that if there is a desync it was introduced since 4.065, based on comments above, and so we've restricted our code analysis to just that and above. But that might be completely incorrect if Mithror just happened to get lucky and not run into a desync in that version, for instance.
(0008778)
lyravega (reporter)
Jan 16, 2011 6:14 pm
edited on: Jan 16, 2011 6:16 pm

Not corruption, but what about choke/loss? Because our issue was happening when there was something major going on. Is there a way to see if there is loss/choke? F3 reflects that?

The weird thing is, we are still able to chat, I mean the players are still connected but somehow, game is not continuing, and I'm assuming that is related to a connection issue. Choke, loss, whatever, but after that, I believe game stops trying to resync, but it shows "Waiting for players". Or it shows "Waiting", but doesn't try to resync, leading to same result. When we force a resync (manage players, change colors, etc...), it resumes for a while, then goes "Waiting" again.

(0008779)
x4000Bughunter (administrator)
Jan 16, 2011 6:41 pm

There is a way to see packet loss, on... I think it's Ctrl+Shift+F3. That shows a different network stats view. It's VERY possible that it could get into a resend cycle if the MTU is too high for one of the ISPs/routers/etc, but we've set it low enough that really shouldn't be an issue. It could also be an issue that the Network Sync Size is too high.

Thing is, if that's the case, it won't say desync. That simply isn't possible, because a desync happens when all the data arrives correctly, and then something different still happens anyway. If you get packet loss or an indefinite resend cycle, it will just say Waiting For Players and no Desync in the status column of the Stats, Players column.

In terms of the chat still working, that is common in the case of a desync or packet loss. The main channel of info has hosed up, but that runs on a slightly different one that still is able to function.

Please read that desync tech support link I posted above, it has all the info on how to tell if this is a desync or a dropped connection or what, you have to check in the Stats, Players tab on the host to really know.
(0008786)
x4000Bughunter (administrator)
Jan 16, 2011 8:54 pm

Well, the "good" news is that Mithror, yours is definitely a desync. We're looking into it, but it's a very rare case apparently. It's not seeming to be Random Number Generator related, but that's always hard to be sure of.
(0008787)
keith.lamothe (administrator)
Jan 16, 2011 8:55 pm

One question that we're hoping isn't important:
Which OS is each player on?
(0008788)
x4000Bughunter (administrator)
Jan 16, 2011 9:17 pm

Okay -- there is definitely a desync evident in the two saves that were posted so far. However, we've hit a brick wall on finding them with the information we have now. We've collected a number of curious bits of information, but we can't duplicate the issue and from hunting through the code in the likely places, everything we've found looks great.

So: it's up to players at this point, we really need more info in order to be able to find this one. A bug we can't reproduce is often a bug we can't fix, which is the hardest thing about desyncs. On the bright side, we have seemingly eliminated a number of possible causes with our investigation so far, so that makes things a little more focused. But at this stage it's like having narrowed it down from 500 haystacks to 300; we're still not finding the single needle that's in one of them anytime soon without more clues.

Wow, that's actually a really apt analogy. ;)
(0008801)
keith.lamothe (administrator)
Jan 16, 2011 11:12 pm

Mithror, just checking, but did you use auto-explore on any of those 9 scouts that ended their careers on those AI planets?
(0008802)
x4000Bughunter (administrator)
Jan 16, 2011 11:15 pm

Ooh! Something like that could really explain why it's not seen by everybody. Good thought.
(0008804)
TechSY730 (reporter)
Jan 16, 2011 11:19 pm

Nice brainstorming. Assuming it is that, how come all of these "psuedo multi stepped orders" (like the auto-whatever orders) have been such a pain to nail down bug free?
(0008847)
Mithror (reporter)
Jan 17, 2011 3:26 am

Sorry to dissapoint, but I've never used auto-explore in this game. (I wouldn't even know how atm) :(

I know how hard it can be to track down this type of bugs, so I'll start to pay extra attention to what we're doing! Perhaps it's related to one of the settings in CTRLS? I know I always start off with Auto-FRD mode for both military and engineers. I also set scouts to not have evade behavior on friendly planets and recently I have set engineers to not assist allies as well. In previous games where it desync'd we've both enabled auto-kite behavior, but in the last game I think (pretty sure) we moved away from that.
(0008848)
Mithror (reporter)
Jan 17, 2011 3:35 am

As to the OS:
I'm on XP (sp 2 or 3, will check when home).
Don't know which OS Kittens is on, but I'll ask him tonight! It's definitely windows.
(0008849)
Mithror (reporter)
Jan 17, 2011 3:47 am

Ok, scratch the last statement. I just remembered he got a Mac from work, so there is a possibility (a small one imo) that he was playing on his Mac. I'll get back to you tonight!
(0008900)
Mithror (reporter)
Jan 17, 2011 1:15 pm

Ok, update:

me (host): xp sp3 (32 bit)
Kittens: vista sp2 (64 bit)
(0008903)
x4000Bughunter (administrator)
Jan 17, 2011 1:22 pm

Ok -- the OS really should have no chance of mattering, then. Even with OSX it should have been okay, but we have less testing data with that in multiplayer with windows, so it would have been a question mark.

Thanks!
(0008917)
Kordy (reporter)
Jan 17, 2011 3:33 pm

For lyravega's (and mine, we were playing those desyncing games) OS', we both use win7, and he was the host.

Lately, we've encountered a LOT of ISP problems so we thought it was that, but desyncs happened on the first ever wave with around 150-200 ai ships attacking on one of our tries; something was choking (and dying) when the AI was doing anything heavy.

We'll try to reproduce and save if possible.
(0008919)
x4000Bughunter (administrator)
Jan 17, 2011 3:46 pm

Thanks -- it will also be more clear now in-game if it's a real desync or what it is. The host will have a new log that will also be useful for us in debugging, etc, too.
(0008931)
Mithror (reporter)
Jan 17, 2011 4:15 pm

Just happened again. Uploading saves. It just said: Kittens: desync error or something like that. Where can I find more logging?
(0008934)
Mithror (reporter)
Jan 17, 2011 4:19 pm

Found the log file (I think) and added it. Hope it helps!
(0008936)
x4000Bughunter (administrator)
Jan 17, 2011 4:24 pm

Okay -- thanks! Wow, it's nice having the log file. That very easily tells me two things:

1. Player 9 (the second AI player) killed on of player 1's ships, but only on host machine. Additionally, this did not have any effect on the number of random numbers generated, so it had to have been something non-RNG-related.

Any clues about what player 1 (I think that would be Kittens) was doing? I'm going to take a look into the save files now to see what else I can uncover, but this is a good start!
(0008940)
Mithror (reporter)
Jan 17, 2011 4:29 pm

Ok, so I am player 0?

He thinks he was either moving his beam frigates or ordering his missile frigates to attack (maybe vampire claws, which are immune). Or at least moving or shooting SOMETHING. Though that still keeps it very open =)
(0008943)
keith.lamothe (administrator)
Jan 17, 2011 4:31 pm

I did change the beam frigate mechanic recently, so that might have something to do with it. But I think that would have caused a desync by killing an AI ship on one machine and not the other. Unless there were AI cutlasses involved, if cutlasses can't do more damage than they have health (I forget).
(0008945)
x4000Bughunter (administrator)
Jan 17, 2011 4:33 pm

I'll look and see what I can see. Meanwhile, can you burn through the remaining balance stuff that's outstanding? It's my turn on the desync trail, I think. ;)
(0008957)
Mithror (reporter)
Jan 17, 2011 4:45 pm

(Bear in mind that there were no beam frigates on the other saves, I believe?)
(0008962)
Moonshine Fox (reporter)
Jan 17, 2011 4:56 pm

Uploaded two saves to this issue. This will desync within 2 minutes of playing.

MoonKing_desync_01.sav is the actual savegame prior to it desyncing. Usually our desyncs have something to do with the southern planet "Cold Sack". Everytime it desyncs, King was either watching that planet, or doing something there.

MoonKing_desync_02.sav is after the desync.

We were able to reproduce a desync about four times, but couldn't nail WHAT caused the desync, only that Cold Sack was usually involved.
(0008963)
x4000Bughunter (administrator)
Jan 17, 2011 5:02 pm

Wicked! That sounds super promising, too. Do you have the DesyncErrors.txt file also?
(0008964)
x4000Bughunter (administrator)
Jan 17, 2011 5:18 pm

Interesting, there are a lot of heavy beam guardians coming into Cold Sack pretty soon after the load. I haven't managed to get it to desync yet, but that is really interesting.
(0008965)
Moonshine Fox (reporter)
Jan 17, 2011 5:18 pm

Uploaded the error file. Sorry, forgot about it earlier.
(0008966)
x4000Bughunter (administrator)
Jan 17, 2011 5:24 pm

Moonshine Fox -- do you and king get a desync if he just does nothing, and just watches the Cold Sack planet? I do not.

Is there anything in particular that he is doing that I can do to duplicate the issue? This seems like it is so close to letting us find and fix this, but I can't duplicate it with your save at the moment.
(0008967)
Moonshine Fox (reporter)
Jan 17, 2011 5:27 pm

It seemed pretty random. It usually involved him trying to build defenses on Cold Sack. I only sent my blob of ships (on my homeworld) towards our northern front. (Can't remember the planet name right now). That usually didn't have an impact though.
(0008968)
x4000Bughunter (administrator)
Jan 17, 2011 5:29 pm

He was building turrets? Or other stuff also?

Interestingly, your desyncerrrors.txt file notes that the errors were all over the place. In one, AI 1 lost a ship. In another AI 2 did. In another, AI 1 has a higher score but no ships are missing yet. And so on.

I'll try spamming in some random turrets and see if I can get it to happen.
(0008969)
kingisaaclinksr (reporter)
Jan 17, 2011 5:39 pm

I also built a shield-gen on Commando and redirected my Ship Dock on my homeplanet to Cold Sack before actually looking at Cold Sack.
(0008970)
Moonshine Fox (reporter)
Jan 17, 2011 5:42 pm

I might also add that we never had these errors before. We could play for hours on end without desyncs. But we upgraded to 4.069 today, and now...well, you see :D
(0008971)
x4000Bughunter (administrator)
Jan 17, 2011 6:05 pm

Yes, I know something is recently different, I just don't know what. I still can't duplicate this, though I tried building the shield-gen and redirecting the space dock. :/
(0008972)
Moonshine Fox (reporter)
Jan 17, 2011 6:12 pm

:(

Too bad you can't find it. We regularly had desyncs during that game and no matter WHAT we did, we got desynced within minutes.
(0008974)
keith.lamothe (administrator)
Jan 17, 2011 6:15 pm

Moonshine, which OS's are you guys on?
(0008975)
x4000Bughunter (administrator)
Jan 17, 2011 6:20 pm

I wonder if there is something different about modulus between 32 and 64 bit machines. Surely not.
(0008976)
keith.lamothe (administrator)
Jan 17, 2011 6:22 pm

Goodness, that would be awful. But weren't we using modulus for other things before?
(0008977)
kingisaaclinksr (reporter)
Jan 17, 2011 6:23 pm

Windows 7 64 Bit.
(0008978)
x4000Bughunter (administrator)
Jan 17, 2011 6:25 pm

I think so. I'm going through the code changes, bit by bit, since 4.063.
(0008979)
keith.lamothe (administrator)
Jan 17, 2011 6:26 pm

Well, we could try adding the bool to Game.Instance.Options for always using the old random, and have it be toggled (for everyone, obviously) by a cmd chat line.
(0008980)
TechSY730 (reporter)
Jan 17, 2011 6:28 pm

Does determining whether the zenith beam hit extra targets use floating point operations? If so, are there suble differences between 32 and 64 bit archetecturrs with rounding? That could make a difference when determining whether the beam hit extra target in near miss cases. Just a wild idea.
(0008981)
keith.lamothe (administrator)
Jan 17, 2011 6:29 pm

We don't use float anywhere in the simulation, for that exact reason.
(0008982)
x4000Bughunter (administrator)
Jan 17, 2011 6:31 pm

We do use modulus all over the place, Keith. I'd forgotten quite how much. Surely this isn't related to the new randoms.
(0008983)
TechSY730 (reporter)
Jan 17, 2011 6:34 pm

Oh that's right. You use the C#'s precision floating point library that guarantees system independent behavior for your non-integer needs.
(0008984)
keith.lamothe (administrator)
Jan 17, 2011 6:40 pm

No, C#s float does no such thing ;) We use a fixed point Int64-based structure for non-integer stuff.
(0008985)
x4000Bughunter (administrator)
Jan 17, 2011 6:41 pm

If you're curious, here's a slightly older version of what we use: http://stackoverflow.com/questions/605124/fixed-point-math-in-c [^]
(0008987)
TechSY730 (reporter)
Jan 17, 2011 6:51 pm

Not the float primitive, but special fixed precision float math class. Looks like your already doing an offshoot of that though.
(0008988)
x4000Bughunter (administrator)
Jan 17, 2011 6:52 pm

That's still using floating point math, it's still not something that's okay to use.
(0008989)
keith.lamothe (administrator)
Jan 17, 2011 6:55 pm

And using something based on Int64 lets us avoid any genuinely in-software math. A specialized deterministic floating point library might be good for a more precision-demanding case, but for our case we more need the speed of having math happen in the ALU. And there's only two primitives that can do that: integral, or floating-point.
(0008990)
x4000Bughunter (administrator)
Jan 17, 2011 7:07 pm

Okay, I found one definite desync when looking through the changes in ForegroundObject:

* Fixed a desync that would occur whenever the harvesters were reset in a multiplayer game.

However, not to get anyone's hopes up, because I'm pretty positive that's not the desync that is affecting folks in this thread.
(0008993)
Moonshine Fox (reporter)
Jan 17, 2011 7:45 pm

Just an FYI here, I am also on Win7 64 bit. Heading to bed now, but hopefully me anand King can get some more testing done tomorrow.
(0008997)
x4000Bughunter (administrator)
Jan 17, 2011 8:18 pm

Ok -- thanks! If you find a way to reproduce the issue with that save you posted, that would be ideal. I really think that I can nail this if that happens, but at the moment I'm limited to going through thousands of lines of changed code by hand, hoping to notice an error.
(0009028)
x4000Bughunter (administrator)
Jan 18, 2011 1:11 am

Well, so here's the final score: after hours of searching through the code, looking at recent changes, and just searching through likely candidates (stuff that shouldn't be accessed in the sim or vice-versa, but sometimes accidentally is), I've found three obscure desyncs.

So obscure, in fact, that I don't think anyone has encountered them yet -- I don't think these are the desyncs that folks here have been running into, based on the savegames and desync logs submitted. But I could be wrong! I sure hope I am.

There was also a fourth potential desync that may or may not have actually been a desync, but that was an unclear bit of code that couldn't easily be deciphered (bad boolean comparison by mistake). That's been cleaned up, and so if that fourth desync was actually a desync, it's not fixed.

That one, if it really was a desync, was more likely to have been a culprit, and it's a recently added one, too. I don't think that's the one that King and Moonshine Fox have been running into, but it might be what was affecting some other folks.

Normally we go months without having any desyncs whatsoever, so to find three (or possibly four) in one night of searching is unheard-of. But when the codebase changes so rapidly and so majorly as it has since even just 4.021, that's somewhat inevitable, I guess.

My best estimate is that there is one desync remaining, and that that's the one that King and Moonshine Fox are running into, and which also may be affecting other folks (but I'm not sure). I've literally been poring over code for the last three hours looking for this, and for another two hours in total tonight, and didn't find it. I don't think I will, therefore, without some really strong leads from savegames. So fingers crossed folks find a reproducible case, if there is one!
(0009056)
Kittens (reporter)
Jan 18, 2011 5:35 am

Kittens here, from the game with Mithror. Just had a thought: Did our desyncs always happen when melee ships were involved? I don't think any were present in the game we managed to play for hours without issue. There have been changes to a bunch of melee ships in the last patch, iirc.
(0009077)
x4000Bughunter (administrator)
Jan 18, 2011 12:05 pm

Hey Kittens -- very interesting thought. I just spent a bit looking through the code once more at the beam stuff and the melee stuff, and no dice. Well, nothing that looks broken in 4.070, anyway. It's possible that this was fixed in 4.070, actually, I guess.
(0009116)
x4000Bughunter (administrator)
Jan 18, 2011 6:17 pm

Has anyone had any issues with 4.071? There were some players in the forums that had been having issues, but then stopped having them after the last round of upgrades. Maybe the 3 (possibly 4) desyncs that I fixed actually were responsible for the desyncs folks were really seeing. There's always hoping...
(0009122)
Mithror (reporter)
Jan 18, 2011 6:28 pm

We've played about 2 hours today without having a single descyn issue. So it's definitely hopeful ;)
(0009123)
x4000Bughunter (administrator)
Jan 18, 2011 6:30 pm

Wow! That's not definitive, but as you say, hopeful. Thanks for the update! :)
(0009124)
kingisaaclinksr (reporter)
Jan 18, 2011 6:40 pm

Fox and I never met up today so hopefully tomorrow we can tell you more. Real life and all that fun stuff. :)

King
(0009125)
x4000Bughunter (administrator)
Jan 18, 2011 6:40 pm

No worries -- understood! Thanks also for the update. :)
(0009213)
x4000Bughunter (administrator)
Jan 18, 2011 11:44 pm

Per issue 0002546, it looks like this issue might simply be based on players running the game in a second session after having run a different game in that same run of the application (load save1 on the host, host plays solo, then host loads save2 and plays multiplayer, or some variant thereof).

Can anyone in this thread comment if that seems to match what you were doing? It doesn't matter if it was the host or the client who was playing a different save. And given that there were three other desyncs that were fixed, you might have been running into one of those instead of the specific bug mentioned here, so not all responses have to line up exactly.

But this is at least the first promising line of inquiry we've had in a while, so that's something!
(0009221)
Mithror (reporter)
Jan 19, 2011 3:42 am

It could be a combination of several? I know we started multiple games in one sessions trying to find a nice seed and they could have desync'd in accordance to what you mention. But... I'm also confident that, on some occasions, we both restarted our clients to see if this helped and noticed that it didn't.
(0009223)
lyravega (reporter)
Jan 19, 2011 6:29 am

One of the desync issues we had was, I wanted to show what I've done to my friend. I loaded up a solo save and opened another player slot, and my friend joined - however we weren't able to play. Then we created another game, which we had no problems or what so ever
(0009239)
x4000Bughunter (administrator)
Jan 19, 2011 10:56 am

Ok -- thanks for the added details. The issue with having a friend join a former solo game should be fixed as of 4.069 or 4.070, at least, anyway.
(0009275)
kingisaaclinksr (reporter)
Jan 19, 2011 4:26 pm

Fox and I did a quick run today (wuss was tired ;) and we could not replicate the desync. Either it was thanks to you guys or my internet decided to play nice. Hard to say at this point so for now we'll consider the issue not a problem.

King
(0009276)
x4000Bughunter (administrator)
Jan 19, 2011 4:28 pm

Okay -- thanks a lot for the info! We're going to give it a bit more time, but we also had a desync that we fixed in 4.072 (excellent catch by Keith!), so hopefully that was the last of them. Assuming there are no more reports, I imagine we'll go for a release on Monday, or thereabouts. Even trying to release this Friday seems like it would be rushing it, at the moment, given how nebulous something like desyncs can be.
(0009277)
kingisaaclinksr (reporter)
Jan 19, 2011 4:31 pm

Yup. Again, it could entirely be my internet at this point because of how random the desyncs are and our inability to replicate it (except for that one game). Centurylink has been....unkind to us and refuses to fix our DSL issues and blames us instead. (/me ends rant before it begins)

But its hard to say. For now, it looks fixed. Should be able to play a few more games before Monday though. :)

King
(0009278)
x4000Bughunter (administrator)
Jan 19, 2011 4:35 pm

Well, actually, an internet connection shouldn't ever be able to cause a desync -- a disconnection, sure. But a desync is an actual corruption of a game that is still connected properly.

In terms of having difficulty replicating, that's often par for the course with desyncs. At this stage, what we think might have been the case was that if either you or Fox had been playing AI War already before you loaded up a multiplayer game, and then got into your multiplayer game without restarting the program of AI War first, that this might have caused the desync. And thus if you both started fresh this time, you wouldn't have run into the issue.

Of course, that's just one of four (possibly five) confirmed desyncs that were in place a few versions back, which are all now fixed. So hopefully, as of 4.072, that's the last of them. Fingers crossed! :)
(0009285)
Kordy (reporter)
Jan 19, 2011 7:06 pm

Hmm, lyravega and I were checking out saves before our desync-happy game, on both sides. However, after the first few desync's we closed the game and restarted from the scratch, but still had those desyncs.

lyravega's busy with finals atm so I'm unable to test it thoroughly.
(0009286)
keith.lamothe (administrator)
Jan 19, 2011 7:09 pm

Kordy, were those desyncs-after-fresh-start in 4.071? And have you had _any_ in 4.072?
(0009287)
x4000Bughunter (administrator)
Jan 19, 2011 7:11 pm

Ok -- well, there was definitely more than one desync present. My hope is just that we fixed all of them. People's experiences have definitely varied because of the multiple desyncs.
(0009354)
Kordy (reporter)
Jan 20, 2011 12:18 pm

Keith; no, not really. We stopped at .68 or so because of the problems and lyravega's classes, and I don't have anyone else to play with.

I suspect our ISP's are also guilty, somehow. Funny thing was, we were able to use text messages in all cases, and they were instant. Ah well, hope you guys nailed it.
(0009637)
Moonshine Fox (reporter)
Jan 24, 2011 2:36 pm

Version 4.073: Desync after I enter "Cold Lift" and move about halfway towards the command station (was intending to gate raid it).

Uploaded "Desync_KingFox_01.sav" and "Desync_KingFox_01.txt".
(0009645)
x4000Bughunter (administrator)
Jan 24, 2011 4:23 pm

Gah, nuts. While it is fresh in your mind, however:

1. Were you able to reproduce the issue?

2. Had either you or King loaded another savegame in the same session that desynced?

3. Is that save from before the desync? If so, that's helpful for us in reproducing it, if you're able to do so.

4. If that save is from after, can you please make sure you follow the desync reporting procedures and give us a save from both you and King when that happens? It's 100% useless to us without both saves, because we have nothing to run a differential against.
(0009647)
kingisaaclinksr (reporter)
Jan 24, 2011 4:45 pm

1. We didn't try to reproduce the issue, our bad.
2. I'm not sure what you mean by this. But if I'm hearing you right, I had been playing another game before we started that. (Solo)
3. I'm pretty sure that's during the desync. You'd have to ask Fox on that.
4. We'll be sure to do this.

King
(0009648)
x4000Bughunter (administrator)
Jan 24, 2011 4:47 pm

Hi King,

Thanks a lot. You're exactly right on item 2. I suspect that may be what led to the desync, and that if you "start fresh" before you both play, you won't be able to duplicate. Item 2 shouldn't be an issue, but at this point I think it is.
(0009649)
kingisaaclinksr (reporter)
Jan 24, 2011 4:49 pm

We did boot up the game I was playing (Before Fox got online) after I fully restarted AIW and had no desyncs. Will test tomorrow if we get the chance and see if that truly is the issue.

King
(0009650)
x4000Bughunter (administrator)
Jan 24, 2011 4:49 pm

Okay -- thanks a lot!
(0009651)
Moonshine Fox (reporter)
Jan 24, 2011 6:02 pm

We'll definately check that. And yeah, mine was freshly started, but King had been playing earlier.
(0009657)
TechSY730 (reporter)
Jan 24, 2011 7:10 pm

Dang it. I didn't get to post this in time. But whatever.

Keith and Chris's reaction when they first saw there is still a desync in 4.073:
http://www.youtube.com/watch?v=WWaLxFIVX1s [^]
(0009658)
kingisaaclinksr (reporter)
Jan 24, 2011 7:13 pm

@Tech You may have posted late, but we allllllllll heard that, from wherever they live. It was rather loud. :P
(0009659)
x4000Bughunter (administrator)
Jan 24, 2011 7:15 pm

Pretty. Darn. Much. ;)
(0009660)
kingisaaclinksr (reporter)
Jan 24, 2011 7:17 pm
edited on: Jan 24, 2011 7:18 pm

xD On a side note Chris, besides the constant spam of Marauders and desyncs, its starting to look finished :)

(0009661)
x4000Bughunter (administrator)
Jan 24, 2011 7:23 pm

That part, at least, is good. :)
(0011296)
x4000Bughunter (administrator)
Mar 22, 2011 11:13 pm

Optimistically, I think that 0003077 was also the cause of this.

Issue Community Support
This issue is already marked as resolved.
If you feel that is not the case, please reopen it and explain why.
Supporters: No one explicitly supports this issue yet.
Opponents: No one explicitly opposes this issue yet.

- Issue History
Date Modified Username Field Change
Jan 16, 2011 10:05 am Mithror New Issue
Jan 16, 2011 10:05 am Mithror File Added: desync issue - Mithror.sav
Jan 16, 2011 10:06 am Mithror File Added: desync.sav
Jan 16, 2011 11:32 am x4000Bughunter Note Added: 0008701
Jan 16, 2011 11:33 am x4000Bughunter Assigned To => x4000Bughunter
Jan 16, 2011 11:33 am x4000Bughunter Status new => acknowledged
Jan 16, 2011 11:53 am Mithror Note Added: 0008709
Jan 16, 2011 11:57 am TechSY730 Note Added: 0008710
Jan 16, 2011 11:59 am Mithror Note Added: 0008711
Jan 16, 2011 12:07 pm x4000Bughunter Note Added: 0008712
Jan 16, 2011 12:12 pm Mithror Note Added: 0008713
Jan 16, 2011 1:13 pm Mithror Note Added: 0008728
Jan 16, 2011 1:14 pm x4000Bughunter Note Added: 0008729
Jan 16, 2011 1:27 pm lyravega Note Added: 0008732
Jan 16, 2011 1:34 pm x4000Bughunter Note Added: 0008734
Jan 16, 2011 2:59 pm kiwiattack Note Added: 0008739
Jan 16, 2011 3:31 pm lyravega Note Added: 0008749
Jan 16, 2011 4:09 pm TechSY730 Note Added: 0008755
Jan 16, 2011 5:17 pm Mithror Note Added: 0008765
Jan 16, 2011 5:20 pm x4000Bughunter Note Added: 0008766
Jan 16, 2011 5:38 pm Mithror Note Added: 0008767
Jan 16, 2011 5:46 pm keith.lamothe Note Added: 0008769
Jan 16, 2011 5:50 pm keith.lamothe Note Added: 0008770
Jan 16, 2011 5:54 pm Mithror Note Added: 0008771
Jan 16, 2011 5:55 pm Mithror Note Edited: 0008771 View Revisions
Jan 16, 2011 5:56 pm keith.lamothe Note Added: 0008772
Jan 16, 2011 5:57 pm x4000Bughunter Note Added: 0008773
Jan 16, 2011 5:57 pm Mithror Note Added: 0008774
Jan 16, 2011 6:05 pm lyravega Note Added: 0008776
Jan 16, 2011 6:06 pm x4000Bughunter Note Added: 0008777
Jan 16, 2011 6:14 pm lyravega Note Added: 0008778
Jan 16, 2011 6:16 pm lyravega Note Edited: 0008778 View Revisions
Jan 16, 2011 6:41 pm x4000Bughunter Note Added: 0008779
Jan 16, 2011 8:54 pm x4000Bughunter Note Added: 0008786
Jan 16, 2011 8:55 pm keith.lamothe Note Added: 0008787
Jan 16, 2011 9:17 pm x4000Bughunter Note Added: 0008788
Jan 16, 2011 9:17 pm x4000Bughunter Status acknowledged => feedback
Jan 16, 2011 9:18 pm x4000Bughunter Relationship added related to 0002480
Jan 16, 2011 11:12 pm keith.lamothe Note Added: 0008801
Jan 16, 2011 11:15 pm x4000Bughunter Note Added: 0008802
Jan 16, 2011 11:19 pm TechSY730 Note Added: 0008804
Jan 17, 2011 3:26 am Mithror Note Added: 0008847
Jan 17, 2011 3:26 am Mithror Status feedback => assigned
Jan 17, 2011 3:35 am Mithror Note Added: 0008848
Jan 17, 2011 3:47 am Mithror Note Added: 0008849
Jan 17, 2011 9:37 am keith.lamothe Status assigned => feedback
Jan 17, 2011 1:15 pm Mithror Note Added: 0008900
Jan 17, 2011 1:15 pm Mithror Status feedback => assigned
Jan 17, 2011 1:22 pm x4000Bughunter Note Added: 0008903
Jan 17, 2011 3:33 pm Kordy Note Added: 0008917
Jan 17, 2011 3:46 pm x4000Bughunter Note Added: 0008919
Jan 17, 2011 4:15 pm Mithror Note Added: 0008931
Jan 17, 2011 4:17 pm Mithror File Added: desync_example2_host.sav
Jan 17, 2011 4:17 pm Mithror File Added: desync_example2.sav
Jan 17, 2011 4:19 pm Mithror File Added: DesyncErrors_example2.txt
Jan 17, 2011 4:19 pm Mithror Note Added: 0008934
Jan 17, 2011 4:24 pm x4000Bughunter Note Added: 0008936
Jan 17, 2011 4:29 pm Mithror Note Added: 0008940
Jan 17, 2011 4:31 pm keith.lamothe Note Added: 0008943
Jan 17, 2011 4:33 pm x4000Bughunter Note Added: 0008945
Jan 17, 2011 4:45 pm Mithror Note Added: 0008957
Jan 17, 2011 4:55 pm Moonshine Fox File Added: MoonKing_desync_01.sav
Jan 17, 2011 4:55 pm Moonshine Fox File Added: MoonKing_desync_02.sav
Jan 17, 2011 4:56 pm Moonshine Fox Note Added: 0008962
Jan 17, 2011 5:02 pm x4000Bughunter Note Added: 0008963
Jan 17, 2011 5:18 pm x4000Bughunter Note Added: 0008964
Jan 17, 2011 5:18 pm Moonshine Fox File Added: MoonKing_DesyncErrors.txt
Jan 17, 2011 5:18 pm Moonshine Fox Note Added: 0008965
Jan 17, 2011 5:24 pm x4000Bughunter Note Added: 0008966
Jan 17, 2011 5:27 pm Moonshine Fox Note Added: 0008967
Jan 17, 2011 5:29 pm x4000Bughunter Note Added: 0008968
Jan 17, 2011 5:39 pm kingisaaclinksr Note Added: 0008969
Jan 17, 2011 5:42 pm Moonshine Fox Note Added: 0008970
Jan 17, 2011 6:05 pm x4000Bughunter Note Added: 0008971
Jan 17, 2011 6:12 pm Moonshine Fox Note Added: 0008972
Jan 17, 2011 6:15 pm keith.lamothe Note Added: 0008974
Jan 17, 2011 6:20 pm x4000Bughunter Note Added: 0008975
Jan 17, 2011 6:22 pm keith.lamothe Note Added: 0008976
Jan 17, 2011 6:23 pm kingisaaclinksr Note Added: 0008977
Jan 17, 2011 6:25 pm x4000Bughunter Note Added: 0008978
Jan 17, 2011 6:26 pm keith.lamothe Note Added: 0008979
Jan 17, 2011 6:28 pm TechSY730 Note Added: 0008980
Jan 17, 2011 6:29 pm keith.lamothe Note Added: 0008981
Jan 17, 2011 6:31 pm x4000Bughunter Note Added: 0008982
Jan 17, 2011 6:34 pm TechSY730 Note Added: 0008983
Jan 17, 2011 6:40 pm keith.lamothe Note Added: 0008984
Jan 17, 2011 6:41 pm x4000Bughunter Note Added: 0008985
Jan 17, 2011 6:51 pm TechSY730 Note Added: 0008987
Jan 17, 2011 6:52 pm x4000Bughunter Note Added: 0008988
Jan 17, 2011 6:55 pm keith.lamothe Note Added: 0008989
Jan 17, 2011 7:07 pm x4000Bughunter Note Added: 0008990
Jan 17, 2011 7:45 pm Moonshine Fox Note Added: 0008993
Jan 17, 2011 8:18 pm x4000Bughunter Note Added: 0008997
Jan 18, 2011 1:11 am x4000Bughunter Note Added: 0009028
Jan 18, 2011 1:11 am x4000Bughunter Status assigned => feedback
Jan 18, 2011 5:35 am Kittens Note Added: 0009056
Jan 18, 2011 12:05 pm x4000Bughunter Note Added: 0009077
Jan 18, 2011 6:17 pm x4000Bughunter Note Added: 0009116
Jan 18, 2011 6:28 pm Mithror Note Added: 0009122
Jan 18, 2011 6:28 pm Mithror Status feedback => assigned
Jan 18, 2011 6:30 pm x4000Bughunter Note Added: 0009123
Jan 18, 2011 6:40 pm kingisaaclinksr Note Added: 0009124
Jan 18, 2011 6:40 pm x4000Bughunter Note Added: 0009125
Jan 18, 2011 11:42 pm x4000Bughunter Relationship added related to 0002546
Jan 18, 2011 11:44 pm x4000Bughunter Note Added: 0009213
Jan 19, 2011 3:42 am Mithror Note Added: 0009221
Jan 19, 2011 6:29 am lyravega Note Added: 0009223
Jan 19, 2011 10:56 am x4000Bughunter Note Added: 0009239
Jan 19, 2011 4:26 pm kingisaaclinksr Note Added: 0009275
Jan 19, 2011 4:28 pm x4000Bughunter Note Added: 0009276
Jan 19, 2011 4:31 pm kingisaaclinksr Note Added: 0009277
Jan 19, 2011 4:35 pm x4000Bughunter Note Added: 0009278
Jan 19, 2011 7:06 pm Kordy Note Added: 0009285
Jan 19, 2011 7:09 pm keith.lamothe Note Added: 0009286
Jan 19, 2011 7:11 pm x4000Bughunter Note Added: 0009287
Jan 20, 2011 12:18 pm Kordy Note Added: 0009354
Jan 24, 2011 2:36 pm Moonshine Fox Note Added: 0009637
Jan 24, 2011 2:36 pm Moonshine Fox File Added: Desync_KingFox_01.sav
Jan 24, 2011 2:37 pm Moonshine Fox File Added: Desync_KingFox_01.txt
Jan 24, 2011 4:23 pm x4000Bughunter Note Added: 0009645
Jan 24, 2011 4:45 pm kingisaaclinksr Note Added: 0009647
Jan 24, 2011 4:47 pm x4000Bughunter Note Added: 0009648
Jan 24, 2011 4:49 pm kingisaaclinksr Note Added: 0009649
Jan 24, 2011 4:49 pm x4000Bughunter Note Added: 0009650
Jan 24, 2011 6:02 pm Moonshine Fox Note Added: 0009651
Jan 24, 2011 7:10 pm TechSY730 Note Added: 0009657
Jan 24, 2011 7:13 pm kingisaaclinksr Note Added: 0009658
Jan 24, 2011 7:15 pm x4000Bughunter Note Added: 0009659
Jan 24, 2011 7:17 pm kingisaaclinksr Note Added: 0009660
Jan 24, 2011 7:18 pm kingisaaclinksr Note Edited: 0009660 View Revisions
Jan 24, 2011 7:23 pm x4000Bughunter Note Added: 0009661
Mar 22, 2011 11:13 pm x4000Bughunter Note Added: 0011296
Mar 22, 2011 11:13 pm x4000Bughunter Status assigned => resolved
Mar 22, 2011 11:13 pm x4000Bughunter Fixed in Version => 5.006
Mar 22, 2011 11:13 pm x4000Bughunter Resolution open => fixed
Mar 22, 2011 11:13 pm x4000Bughunter Relationship added related to 0003077


Copyright © 2000 - 2017 MantisBT Team
Powered by Mantis Bugtracker