View Issue Details

IDProjectCategoryLast Update
0007188Valley 1Crash/ExceptionApr 20, 2012 12:27 pm
ReporterToll Assigned ToChris_McElligottPark  
Severitycrash 
Status resolvedResolutionfixed 
Product Version0.970 (RC1) 
Fixed in Version0.971 (RC2) 
Summary0007188: Managed to crash the game
DescriptionSorry :( Fortunately (or perhaps unfortunately) it's perfectly reproducible though.

Step 1: Enter a region (not sure if this step is necessary, but that's what I did both of the times I crashed it).
Step 2: Think "Oh, right, that's what I forgot to do".
Step 3: Pause the game and alt-tab out.
Step 4: Return half an hour later and try to enter the game again.
TagsNo tags attached.
Internal WeightNew

Activities

Toll

Apr 20, 2012 8:33 am

reporter  

Chris_McElligottPark

Apr 20, 2012 9:18 am

administrator   ~0022476

Do you have to actually leave it for particularly long, or is it literally a half hour?

From what I see in your logs, this is more or less what I know:

1. You didn't run out of RAM or anything like that.

2. The actual game logic doesn't look to be at fault.

3. The new render logic doesn't look to be at fault.

4. I notice that among the DLLs it says are active, there is a security scanner in there.

5. The crash itself is really nonspecific and _seems_ to originate in unity's underlying engine itself rather than the managed runtime that hosts our code.

6. Well, okay, actually the crash says that it tried to write to a piece of memory that it couldn't. Which is usually a sign of being out of RAM, except you weren't; and otherwise is just a sign of unity's black box engine... basically killing itself.


From that I'm having to conclude for the moment that this is probably actually the security software starting to scan when you're idle on your machine for a certain amount of time -- a lot of them are set to automatically do that. And then during a scan, it looks like it might be doing something that temporarily locks some memory that unity wanted to use, causing unity to crash.

That would actually be a bug with unity (I think) rather than the security software. But it's hard to be sure, and there's still a remote possibility that this is actually a bug in our code somehow -- that we were causing runaway memory allocations during alt-tab out or something. I think the error signatures would look a bit different if that were the case, though. But I can't be sure, because all I have are the unity error signatures (since this originates in their runtime) rather than our actual managed error signatures (which actually gets to code that Keith and I have access to).

SO... this is good data to have, but I don't _think_ it's a generalized bug in the game. I would be interested if simply changing your security software to not scan while you are idle for only 30 minutes would fix this, or if disabling the security software temporarily simply as a test would do the same. I'm also going to get Josh looking at this and seeing if he can duplicate it, just to be sure.

Thanks!

Toll

Apr 20, 2012 9:25 am

reporter   ~0022477

I didn't leave my computer idle though; I just alt-tabbed out of AVWW. So I don't think it's a case of F-Secure starting to scan my system. I'll try to disable it completely though and see if that makes a difference.

Chris_McElligottPark

Apr 20, 2012 9:26 am

administrator   ~0022478

Okay, that's important clarification -- thanks. Will try this as well and see what I can see!

Toll

Apr 20, 2012 9:27 am

reporter   ~0022479

Oh, and regarding the length of time: I don't really know, to be honest. Half an hour did the trick both times for me though, but I don't know if ten or twenty minutes works too.

Chris_McElligottPark

Apr 20, 2012 9:27 am

administrator   ~0022480

Three questions:

1. What is the shortest time you've been able to leave it and have that happen?

2. Were you running fullscreen or windowed?

3. Did you literally alt-tab out, or use some other keys (windows key, etc)? And what did you specifically do to re-enter the game? (Sounds like alt-tab for sure there).

Chris_McElligottPark

Apr 20, 2012 9:27 am

administrator   ~0022481

Fair enough on the time, thanks.

TechSY730

Apr 20, 2012 9:31 am

reporter   ~0022482

x4000 said: "just a sign of unity's black box engine... basically killing itself."

...again

Chris_McElligottPark

Apr 20, 2012 9:31 am

administrator   ~0022483

So, Josh was able to repro it by tabbing in and out a lot.

This may just be a glitch in unity, thought; looking into it more:

http://www.interstellarmarines.com/forum/threads/id/352/

tigersfan

Apr 20, 2012 9:36 am

reporter   ~0022484

Well, the thing is, it's new. I'm not using tabbing out anymore than I was before. It might be related to the graphics changes you did recently.

I used to be able to have the game open in the background for hours. So long, in fact, that I used to forget about it.

I had a crash last night that I thought was related to a video I was streaming, but now that I had it again today, I'm pretty sure it was AVWW last night too.

All I have to do is tab out of the game and let it sit. I don't know if it takes a whole 30 minutes, but, it takes more than 5 or 10, to be sure. Could it be a VRAM issue?

Chris_McElligottPark

Apr 20, 2012 9:40 am

administrator   ~0022485

It could be related to the graphics changes I did recently. The thing with the graphics changes is that it hands off a bunch of data to unity for unity to do whatever the heck it wants to with it. On the old way of doing the graphics, it wasn't a lot more directly making calls to the GPU.

So if unity has some sort of issue in their general render stack related to this, then that could be related. Given you say you saw this last night, Josh, I don't think this is related to the steam wrapper dll that was added. That was one other suspicion of mine.

This may not be fixable without upgrading to a newer version of unity (we're still using version 3.3, and they are now on version 3.5). However, version 3.4 was buggy as heck and I didn't want to introduce new unknown issues right before release by moving to 3.5. At this point I sure as _heck_ don't want to risk further instability by doing that. So... we may just be stuck, but I'm seeing if I can google up anything on other unity devs running into this.

tigersfan

Apr 20, 2012 9:43 am

reporter   ~0022486

I'm about 96.783% sure that it's not related to the Steam wrapper.

Chris_McElligottPark

Apr 20, 2012 9:47 am

administrator   ~0022487

Same. A few things:

1. Some folks were thinking this was related to an ATI driver bug, but Toll is running nVidia I see from his log.

2. I still can't duplicate this, leaving it for 10+ minutes at a go. What I can duplicate is it hanging for quite a long while as unity gets its RAM back if it's been idle too long. It may be that when it tries to do this on your machines, it's coming up with not enough RAM or something.

3. I'm not running any antivirus. Apparently there's a good history of AV products causing various issues with various versions of unity, including alt-tab crashes (but those are mainly with the web player or unity editor): http://forum.unity3d.com/threads/78979-Anti-virus-problems/page3

tigersfan

Apr 20, 2012 9:51 am

reporter   ~0022488

I get the long hang too, but, I'm not sure how related to the crash that is. When it crashes I don't have to be trying to tab back into it. I'm just doing something else and I get a box that pops up that says "AVWW.exe has stopped running blah blah blah".

Also, no AV at all? You are a brave soul. I'm running Microsoft Security Essentials.

Chris_McElligottPark

Apr 20, 2012 9:54 am

administrator   ~0022489

Okay, that's good info that it just happens in the background randomly. What else do you have running? Running any videos or anything else that might use hardware acceleration of the GPU?

And yes, I don't use antivirus because I use gmail and I don't download strange files. I have AV installed and do scans if I'm concerned over something, but I've not had a virus in the last 8-9 years on my personal machine. Having had viruses many times in the past at work and at home, I've kind of learned my lessons on what to and what not to do.

Not that I recommend that path to folks, but I'm a huge hypocrite on that. ;)

Minotaar

Apr 20, 2012 10:00 am

reporter   ~0022490

I also had a crash today in the same situation; alt-tabbed out, leave for a prolonged time (around half an hour), crash, pop-up box, etc. No antivirus running here as well.
Would attach the crash log if I could find it anywhere.

Chris_McElligottPark

Apr 20, 2012 10:00 am

administrator   ~0022491

Okay, I'm coming up with nothing. There are lots of crash reports that are similar on unity 3.1, 3.3, 3.4, and 3.5. Most in the editor, then in the web player to a lesser extent. I guess now that we're using their more mainstream render path we're running into that now whereas we were not before.

I'm not sure this is solvable, leastways not prior to 1.0 without causing more problems than we fix. :/

On the bright side, the game would have already saved your world and stuff, so it's not like it loses progress if you do this. On the downside... blah this looks bad when this sort of happens.

tigersfan

Apr 20, 2012 10:00 am

reporter   ~0022492

Well, last night I was watching the Tigers game, which streams through their own wrapper over Flash. Today I think I had hulu.com streaming, which I think is Silverlight, though I'm not sure about that. (yes, I stream a lot. I'm sure my ISP hates me.)

Let me try this without streaming anything and see if it's still an issue.

Chris_McElligottPark

Apr 20, 2012 10:01 am

administrator   ~0022493

Minotaar: were there any logs in your runtimedata folder? And then in your AVWW_Data folder is there an output.log file with any extra data? The latter gets overwritten each time you start the game, so it would be lost if you've already played more.

Minotaar

Apr 20, 2012 10:03 am

reporter  

output_log.txt (1,243,779 bytes)

Minotaar

Apr 20, 2012 10:04 am

reporter   ~0022494

Uploaded the output log, didn't run the game since the crash yet. Nothing new in RuntimeData.

Chris_McElligottPark

Apr 20, 2012 10:07 am

administrator   ~0022495

Minotaar: Thanks, no details were in there, though.

I just managed to get the same thing to happen to me. I'm not sure if I wasn't waiting long enough or if it was that I just fired up a youtube video. Going to test this more.

Toll

Apr 20, 2012 10:14 am

reporter   ~0022496

So; I tested again, no AV running (I unloaded F-Secure), and disabled my internet-connection as well (yes, I'm slightly paranoid). Since Chris mentioned RAM, I had a look at task manager as well; during the first minutes being alt-tabbed, nothing happened, but after that it started going up at a rate of about 700kB/s, and it finally dies at about 1.8GB RAM.

Toll

Apr 20, 2012 10:15 am

reporter   ~0022497

I did get a different error message this time though, and no error log was generated. The error message was:

"Runtime Error!

Program:


This application has requested the Runtime to terminate in an unusual way.
Please contact the application's support team for more information."

Chris_McElligottPark

Apr 20, 2012 10:22 am

administrator   ~0022498

I can also confirm the RAM increase that you're seeing -- it's the same for me. I'm going to look and see if I can track this down, but at least now:

a) The error logged actually does make sense.

b) There's a clear symptom of the issue that is easily tracked well before it crashes.


Also, I will note that I've already confirmed that this is not managed memory that is increasing. So there's something going on in the actual render call stack that is causing this, or else possibly something we're doing with texture loading if it's causing textures to get invalidated and reloaded.

Given that, I actually -- knock on wood -- have a few ideas on how I might be able to solve this!

tigersfan

Apr 20, 2012 10:25 am

reporter   ~0022499

Yep, I'm getting the same thing. Good luck squashing this one!

Chris_McElligottPark

Apr 20, 2012 11:06 am

administrator   ~0022500

Well, the "good" news is that I can verify what is causing this: calling Graphics.DrawMesh while the game does not have focus seems to do it. Or something in that call stack leading up to it; still hunting back.

But basically, we're back to the problem of not being able to detect when the unity editor has focus or not. I'm going to see what I can do, sigh, but this gets into pretty hacky territory to work around their bugs.

Chris_McElligottPark

Apr 20, 2012 11:16 am

administrator   ~0022502

AHAHAHAHAHAHHAHAHAHAHAA! Giddy glee!

After... two plus years of trying to figure out a way to detect window focus in unity, we FINALLY have a way to do it. And it solves this problem.

It's a hack, but it's a reliable one because it's one that we were unintentionally relying on prior to the new rendering model; will explain more later.

And yes, we've bugged unity about this as far back as 2010, but they never have done anything about the focus detection fix. This sort of thing doesn't work: http://forum.unity3d.com/threads/87860-Lost-Graphics-Device

But I found something that did, and I'll share for other unity devs to know as well. What a huge relief. KNOCK. ON. WOOD.

Will have a build out in 20-30 minutes for you guys to bang on, after I finish de-instrumenting the code and making my own final tests on this one.

Toll

Apr 20, 2012 11:18 am

reporter   ~0022503

Woo! Sounds awesome. So, about all those ideas that were denied because there was no way to detect focus... *reopens a hundred issues*

tigersfan

Apr 20, 2012 11:18 am

reporter   ~0022504

What?!??! Really?!?!? Does this mean that eventually we could make AVWW pause if I click on my second monitor? (I know that's out for 1.0)

tigersfan

Apr 20, 2012 11:20 am

reporter   ~0022506

BTW, there is a good bit of wood in this apartment, my knuckles are sore now. :)

Chris_McElligottPark

Apr 20, 2012 11:21 am

administrator   ~0022507

"So, about all those ideas that were denied because there was no way to detect focus... *reopens a hundred issues*"

"Does this mean that eventually we could make AVWW pause if I click on my second monitor?"

That is EXACTLY what this means. :D And why I'm soooo happy. This will be a huge boost for both AI War AND AVWW.

Chris_McElligottPark

Apr 20, 2012 11:25 am

administrator   ~0022508

Although, one clarification: technically I can't detect if the application has focus _literally_. If you're running the application in windowed mode and click to something else, for instance, I can't detect that to my knowledge with this trick.

But if you're running fullscreen and click out of the application, I can definitely detect that now -- at least on windows. Haven't tested on OSX yet.

TechSY730

Apr 20, 2012 11:29 am

reporter   ~0022509

Wait, does this mean those blasted "key is still held down when I switch out and then switch back in" errors in AI war can finally be fixed?
Similar thing to the planet view still scrolling when minimized?

HOORAY!!!
This will help the first impressions of lots of new AI war players, and many long time players will be grateful as well.

Chris_McElligottPark

Apr 20, 2012 11:36 am

administrator   ~0022511

"Wait, does this mean those blasted "key is still held down when I switch out and then switch back in" errors in AI war can finally be fixed?"

MAYBE. Definitely the scrolling when-minimized thing, and MAYBE the key-held-down thing. However, it won't help with the key-held-down thing with the Steam overlay, unfortunately, so that one will persist no matter what until unity actually addresses the core bugs.

But this lets us work around at least a pretty good swath of the issues!

TechSY730

Apr 20, 2012 11:38 am

reporter   ~0022512

Awesome. So who wants to be in charge of reopening the dozens of issues in all three games that could be fixed (or at least partially fixed) with this? Of course, they will need to be linked to this issue.

Chris_McElligottPark

Apr 20, 2012 11:48 am

administrator   ~0022514

Thanks again, folks!

The explanation text:

A crash bug was discovered this morning, but had apparently been in the game since the new graphics rendering system updates a few days back. This was a particularly tricky crash bug, as you can see if you read the history of the linked issue. However, it has now been fixed -- and as a side bonus, it led us to the holy grail of things we've been wanting to be able to do in unity: detect when the @#$#$ window doesn't have focus. This has been a bug in unity since pretty much forever, and we've told them about it as far back as 2010, but nothing has changed on it. Frustratingly.

A minor digression: not knowing if the window has focus has all sorts of bad consequences for us. It means that if the application loses focus, we can't pause it. It means that if the application doesn't have focus, it still takes keyboard input. And, in the case of this bug, it means that we keep calling Graphics.DrawMesh and unity keeps queuing up more and more unmanaged memory that it never gives back until the application is closed or dies of its own accord. On average it would cause the application to leak about 700kb of memory per second while you were alt-tabbed out of a fullscreen copy of the game, and thus you'd get a crash after some amount of time (usually when the total RAM usage of the process hit about 1.8GB for whatever reason).

As a side effect of this issue with unity, it made alt-tabbing incredibly slow. Apparently all those Graphics.DrawMesh calls were being queued up until you came back (or something to that effect), so then it would take forever to load when you came back, even if it didn't crash. Now that we are able to detect when the game doesn't have focus (in fullscreen mode only -- none of this, including the original crash, applies to windowed mode), we simply stop doing any drawing while it doesn't have focus. That makes it alt-tab back in really snappily, and prevents the memory leak.

So how are we detecting the loss of focus? Well, a hack -- but a reliable one, based on the fact that this bug only came in once we started using Graphics.DrawMesh. Normally unity 3D keeps running its Update calls on its camera even when the game has lost focus (well, there is a way to disable that -- but then network connections get killed when you alt-tab out, so we can't use that sort of thing). So far so good, and this is what we expect. We just didn't have any way to know if we should be calling draw calls (because the window has a valid device state attached to it) or not during that update method. Normally unity doesn't expose the device-lost state to the game-logic level, because apparently they think we don't need to know.

What I kind of stumbled into, after two years of off-and-on intensive searching for a workaround for this, was the following: OnPostRender does not get called when the device has been lost. That's where our Graphics.DrawMeshNow calls go, and why we haven't had this sort of crash before we switched to also using Graphics.DrawMesh (which was where all that performance boost came from a couple of versions back). Knowing that OnPostRender doesn't happen when the device is lost is the key to the whole solution: if there have been too many Update calls since the last OnPostRender call, you know your device has been lost. Normally if there have been more than 1 Update calls since the last OnPostRender call, you can probably infer that it's been lost. But I'm using 20 as the cutoff point just to be on the safe side, and only cutting of the render calls after that point is reached. Once you alt-tab back in it immediately starts doing OnPostRender again and then the Update calls to Graphics.DrawMesh resume, and everything is happy.

Toll

Apr 20, 2012 11:52 am

reporter   ~0022515

... Wow. For a hack, that's a real neat one, I must say!

TechSY730

Apr 20, 2012 11:54 am

reporter   ~0022516

x4000 said: "it led us to the holy grail of things we've been wanting to be able to do in unity: detect when the @#$#$ window doesn't have focus. This has been a bug in unity since pretty much forever, and we've told them about it as far back as 2010, but nothing has changed on it. Frustratingly."

I can see you have been just as frustrated over this as many new and long time players are.

The root cause may not be fixed, but nice work finding a way to detect a "side effect" of the window being minimized! :)

Chris_McElligottPark

Apr 20, 2012 11:54 am

administrator   ~0022517

I thought so, too!


Okay, the new version just went live, though I've not put up the blog posts yet -- if you guys don't mind all retesting just to make sure it is indeed fixed for all of you, that would be enormously appreciated!

Chris_McElligottPark

Apr 20, 2012 11:55 am

administrator   ~0022518

"I can see you have been just as frustrated over this as many new and long time players are."

Sure. I'm frustrated by getting bug reports I can't fix, on the one hand. And on the other hand, those bugs affect me as much as the next person -- I also run dual monitors, etc, so it's annoying on a personal level also.

Hyfrydle

Apr 20, 2012 11:56 am

reporter   ~0022519

Ah this explains the black screen when I was alt tabbing back in on the new version. Gald this issue is also fixed.

Minotaar

Apr 20, 2012 12:13 pm

reporter   ~0022523

Alt-tabbing is back to being almost-instant. Yay! I missed that. Will leave it running for a while.

tigersfan

Apr 20, 2012 12:18 pm

reporter   ~0022524

That's weird, did you guys have it go from using the custom cursor to using the OS cursor when you updated? Or was that just me?

Minotaar

Apr 20, 2012 12:21 pm

reporter   ~0022526

Yeah, I also had my settings flip to the mouse cursor. It did remember after I changed it and restarted the game, though, so no big deal.

Toll

Apr 20, 2012 12:23 pm

reporter   ~0022527

So; updated, started, paused, alt-tabbed out. Wait 20 minutes, and basically no change in RAM usage (about 60kB at most). Success!

tigersfan

Apr 20, 2012 12:25 pm

reporter   ~0022528

Yeah, it's working for me too. My RAM usage actually dropped while I was tabbed out. It only did this once, so I'm assuming the GC cycle hit. :)

Chris_McElligottPark

Apr 20, 2012 12:27 pm

administrator   ~0022529

Sweeet! :)

Very strange on the OS cursor, though.

Issue History

Date Modified Username Field Change
Apr 20, 2012 8:33 am Toll New Issue
Apr 20, 2012 8:33 am Toll File Added: Crash_2012-04-20_134452.zip
Apr 20, 2012 9:18 am Chris_McElligottPark Note Added: 0022476
Apr 20, 2012 9:18 am Chris_McElligottPark Assigned To => Chris_McElligottPark
Apr 20, 2012 9:18 am Chris_McElligottPark Status new => feedback
Apr 20, 2012 9:25 am Toll Note Added: 0022477
Apr 20, 2012 9:25 am Toll Status feedback => assigned
Apr 20, 2012 9:26 am Chris_McElligottPark Note Added: 0022478
Apr 20, 2012 9:27 am Toll Note Added: 0022479
Apr 20, 2012 9:27 am Chris_McElligottPark Note Added: 0022480
Apr 20, 2012 9:27 am Chris_McElligottPark Note Added: 0022481
Apr 20, 2012 9:31 am TechSY730 Note Added: 0022482
Apr 20, 2012 9:31 am Chris_McElligottPark Note Added: 0022483
Apr 20, 2012 9:36 am tigersfan Note Added: 0022484
Apr 20, 2012 9:40 am Chris_McElligottPark Note Added: 0022485
Apr 20, 2012 9:43 am tigersfan Note Added: 0022486
Apr 20, 2012 9:47 am Chris_McElligottPark Note Added: 0022487
Apr 20, 2012 9:51 am tigersfan Note Added: 0022488
Apr 20, 2012 9:54 am Chris_McElligottPark Note Added: 0022489
Apr 20, 2012 10:00 am Minotaar Note Added: 0022490
Apr 20, 2012 10:00 am Chris_McElligottPark Note Added: 0022491
Apr 20, 2012 10:00 am tigersfan Note Added: 0022492
Apr 20, 2012 10:01 am Chris_McElligottPark Note Added: 0022493
Apr 20, 2012 10:03 am Minotaar File Added: output_log.txt
Apr 20, 2012 10:04 am Minotaar Note Added: 0022494
Apr 20, 2012 10:07 am Chris_McElligottPark Note Added: 0022495
Apr 20, 2012 10:14 am Toll Note Added: 0022496
Apr 20, 2012 10:15 am Toll Note Added: 0022497
Apr 20, 2012 10:22 am Chris_McElligottPark Note Added: 0022498
Apr 20, 2012 10:25 am tigersfan Note Added: 0022499
Apr 20, 2012 11:06 am Chris_McElligottPark Note Added: 0022500
Apr 20, 2012 11:16 am Chris_McElligottPark Note Added: 0022502
Apr 20, 2012 11:18 am Toll Note Added: 0022503
Apr 20, 2012 11:18 am tigersfan Note Added: 0022504
Apr 20, 2012 11:20 am tigersfan Note Added: 0022506
Apr 20, 2012 11:21 am Chris_McElligottPark Note Added: 0022507
Apr 20, 2012 11:25 am Chris_McElligottPark Note Added: 0022508
Apr 20, 2012 11:29 am TechSY730 Note Added: 0022509
Apr 20, 2012 11:36 am Chris_McElligottPark Note Added: 0022511
Apr 20, 2012 11:38 am TechSY730 Note Added: 0022512
Apr 20, 2012 11:48 am Chris_McElligottPark Internal Weight => New
Apr 20, 2012 11:48 am Chris_McElligottPark Note Added: 0022514
Apr 20, 2012 11:48 am Chris_McElligottPark Status assigned => resolved
Apr 20, 2012 11:48 am Chris_McElligottPark Fixed in Version => 0.971 (RC2)
Apr 20, 2012 11:48 am Chris_McElligottPark Resolution open => fixed
Apr 20, 2012 11:52 am Toll Note Added: 0022515
Apr 20, 2012 11:54 am TechSY730 Note Added: 0022516
Apr 20, 2012 11:54 am Chris_McElligottPark Note Added: 0022517
Apr 20, 2012 11:55 am Chris_McElligottPark Note Added: 0022518
Apr 20, 2012 11:56 am Hyfrydle Note Added: 0022519
Apr 20, 2012 12:13 pm Minotaar Note Added: 0022523
Apr 20, 2012 12:18 pm tigersfan Note Added: 0022524
Apr 20, 2012 12:21 pm Minotaar Note Added: 0022526
Apr 20, 2012 12:23 pm Toll Note Added: 0022527
Apr 20, 2012 12:25 pm tigersfan Note Added: 0022528
Apr 20, 2012 12:27 pm Chris_McElligottPark Note Added: 0022529
Apr 14, 2014 9:28 am Chris_McElligottPark Category Bug - Crash or Exception => Crash/Exception