F@H 7.6.13 + RTX 2080 Ti 456.55 Windows Crash

[VxJasonxV]

Hey folks,

In the last week or two I've started having a really weird Windows crash, sometimes when I'm using the machine, always over night when my session is locked with no foreground apps, F@H being allowed to run full tilt. This same behavior has happened during VR (Star Wars: Squadrons), but in that case it is a consistently repeatable crash and is also a full system restart, as opposed to what happens when the computer is idling, less F@H, is running overnight.

When actively using my computer I generally set Folding to "On Idle" and then turn that option off whenever I'm done. Even if I don't, it kicks into high gear when I lock my computer of course. Over the last week or so when I come back to my computer it's asleep, and when I turn it on and put in my pin I'm logged in. Nothing had been running since some point overnight, not my ever-present processes (Steam, GoG), not F@H, nothing.

Thinking Blue Screens were happening I turned 'Automatically restart' in Windows settings off, but that's not the case, I've never seen one. Windows boots fast nowadays via UEFI and all that, but I *never* see that screen, I don't think my system is rebooting. So it seems to me like Windows is crashing pretty badly, so bad that it ends my session, but isn't a Blue Screen, it isn't entirely fatal. My Windows process knowledge is lacking, but in Linux terms I would call this a "Window Server" or "Desktop Environment" crash.

When these crashes happen the display goes black, and even the power light on my system goes out. It's as if it is forced into some sort of sleep state, the power down behavior is particularly weird considering that I believe Windows never actually goes away. But this is more than a sleep / hibernate due to the nature of my account getting logged out.

In the event viewer there are some errors under 'Application' that I raise my eye at. Fahcore is listed in all of them, and strangely Rocket League is listed too. I just finished playing it but the crash has never kicked me out mid-RL. As previously stated it has happened mid-Star Wars: Squadrons, but consistently and I don't believe the behavior to be related.

I see a cluster of 5 of these errors pretty often in the "Application" logs:

Code: Select all

Fault bucket , type 0
Event Name: BEX64
Response: Not available
Cab Id: 0

Problem signature:
P1: FahCore_a7.exe
P2: 0.0.0.0
P3: 5dde687c
P4: FahCore_a7.exe
P5: 0.0.0.0
P6: 5dde687c
P7: 0000000000f161e7
P8: c0000409
P9: 0000000000000005
P10: 

Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER8FEF.tmp.WERInternalMetadata.xml

These files may be available here:
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_FahCore_a7.exe_d3a07ecd28195878f6fcb1b80c5a41937fbf981_b9fa5d94_9125c930-d9b7-4f61-ba88-e96a417bb390

Analysis symbol: 
Rechecking for solution: 0
Report Id: bdf19f7c-7879-4090-98ca-78e8e25d6143
Report Status: 4196
Hashed bucket: 
Cab Guid: 0

There are also very similar entries that reference FahCore_22.exe / OpenMMCUDA.dll. I understand these to be the CPU optimized and GPU optimized process runners. I think Rocket League's presence may be due to gifyourgame, or just the fact that I recently played the game around the time of the crash? Sometimes there are hours between playing RL and the crash, though.

I'm at a bit of a loss on what else to look at. What I'm going to do tonight is quit F@H and see if I still get the 'logged out' behavior overnight, or just a normal sleep, wake, unlock behavior. If F@H is catalyzing these crashes / bad behavior, any suggestions of what to do?

[VxJasonxV]

I looked up the F@H logs, and they seem to point to a Video Card specific crash too, but I'm not gleaning any knowledge I didn't already have, or at least strongly assume;

Code: Select all

06:15:19:WU01:FS00:0xa8:Completed 140000 out of 500000 steps (28%)
06:15:38:WU00:FS01:0x22:Completed 110000 out of 1000000 steps (11%)
06:16:09:WU00:FS01:0x22:Completed 120000 out of 1000000 steps (12%)
06:16:40:WU00:FS01:0x22:Completed 130000 out of 1000000 steps (13%)
06:17:00:WU01:FS00:0xa8:Completed 145000 out of 500000 steps (29%)
06:17:07:WU00:FS01:0x22:An exception occurred at step 138818: Error invoking kernel: CUDA_ERROR_UNKNOWN (999)
06:17:07:WU00:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
06:17:07:WU00:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
06:17:13:WARNING:WU00:FS01:FahCore returned an unknown error code which probably indicates that it crashed
06:17:13:WARNING:WU00:FS01:FahCore returned: UNKNOWN_ENUM (-1073740791 = 0xc0000409)
06:17:14:WU00:FS01:Starting
06:17:14:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\vxjas\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 8116 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0

Those timestamps overlap with the Windows Application logs I was looking at, which were timestamped 00:17 local time, MDT currently GMT-6, these GMT logs being 06:17 lines up. That "CUDA_ERROR_UNKNOWN" seems relevant to me, but I don't know what to make of it nor do about it. Something something uninstall, restart, clean install nVidia drivers? Pray for a > 456.55 soon?

[PantherX]

Assuming that you're having issues even with only Star Wars: Squadrons playing, it seems to be hardware related. Could be with your PSU, RAM, GPU overclocked, etc. While we can help you in some aspects, given that the system is generally unstable, maybe you can get better support from your favorite hardware site?

[ajm]

It comes to mind that there have been weird behaviors observed on Windows with FAH and GOG, something linked to GOG using .dll similar to what FAH is using, or in the same directories, or something.

viewtopic.php?f=108&t=34665

[VxJasonxV]

That behavior has specifically to do with uninstalling or upgrading, and I was the one that actually documented it on this forum: viewtopic.php?f=108&t=34484&p=328197#p328197

[VxJasonxV]

I regret bringing up Star Wars: Squadrons, even if I did caveat it. But sure enough, days later, Squadrons had been patched and no longer crashes in the same manner, nor at all.

I’ve also turned down Folding to medium, and light, and it is still behaving in the same way. Crashing some level of Windows and kicking me out to an un-logged in state. This also means the computer actually goes to sleep since it is left running but locked, crash and logout, recovers to the login screen, but then sleeps after however long I have set.

F@H is the only remaining thing that is behaving in this way, and I’m reasonably convinced that it’s not a hardware specific or otherwise physical issue.

(I also don’t have a favorite hardware site to seek support on, but that’s my own problem.)

[PantherX]

I would suggest that you use GPU-Z to log what's happening to your GPU. I think that when you lock your system, the client continues to fold without issues. However, it might encounter an issue when a new WU is downloaded and starts to fold. We would also need to see the client's log file to figure out what's happening.

Alternatively, if you don't lock your system but turn off your monitors, what happens?

[VxJasonxV]

There is never a situation where it folds without issues.

If I turn Folding on I will get a sudden crash + logout while I’m actively using my computer.

If I lock my computer and Folding was in any state (active or “on idle”) a crash will occur sometime between the lock and when I return to my computer.

I’ve even turned Folding power down to Low and it will crash. I’m thinking of disabling the GPU folders and seeing if it still happens, seeing as the CPU folders were in those logs too.

I’ve never heard of GPU-Z, I’ll check it out, thank you for the insight.

Was the log in my second comment not sufficient?

[ajm]

Maybe you have a software that launches automatically or periodically and somehow causes the crashes when the machine is folding?
Apart from the logs suggested by PantherX, you can try https://www.resplendence.com/whocrashed - maybe it will give you a clue.
And is your Windows OK? Have you tried https://support.microsoft.com/en-us/hel ... ted-system ?

[PantherX]

...Was the log in my second comment not sufficient?

The log was missing the first 100 lines which describes your system and configuration as seen by the client. While we can see the error message, we don't know what your system configuration is as per the client. Thus, we would need it.

In addition to what ajm mentioned, here are some Nirsoft tools that I have used in the past and heard great things about:
https://www.nirsoft.net/utils/what_is_hang.html
https://www.nirsoft.net/utils/app_crash_view.html
https://www.nirsoft.net/utils/blue_screen_view.html

[VxJasonxV]

For posterity, I updated to 456.71 at some point in the last week, naturally the problem is still occurring. I don't know enough about the nVidia version scheme but I'm pretty sure any 456.* version won't have substantial changes, and if it is driver related I'm waiting on >= 457. Given the only note in the changelog is "compatibility with Call of Duty something or other, now featuring NVIDIA Reflex technology", wowie.

[PantherX]

Have you considered some older WHQL Drivers like 452.06 to see if you get it to fold or not? In other words, what was the last driver version that you had which allowed you to fold without issues?

[VxJasonxV]

I had a theory that it is strange that Folding would be causing this suddenly. Given that Rocket League is showing up in those crash logs I pulled out from Event Viewer, I’m going to close Gif Your Game and see if my computer survives.

[VxJasonxV]

GYG being closed didn't change anything, Rocket League still showed up in the core crash logs which I still don't understand at all. It is absolutely not running when I lock my computer. A handful of background processes are, Steam, GOG Galaxy, Stream Deck, Dropbox, OneDrive, GYG, and then all the non-iconed things like Windows builtins, Microsoft Store whatevers, every thing nVidia puts in place, etc. And of course, F@H.

afaik the last crash-free version was whatever came before .55, but I never paid attention until I started needing to pay attention to it. I have a vague memory of some weird nVidia driver installation problem many versions ago, I did some searching and was pointed to Display Driver Uninstaller, so I've used that and just reinstalled Game Ready 456.71. If the crashes persist I'll use the Studio version, if the crashes STILL persist I'll do <456. If the crashes still STILL persist, I'll … cry? I guess?

[ajm]

Another area of investigation would be the software and/or hardware that can interfere with the video system, or even the audio if you are using the digital output of the Graphics card.
A soundbar or something connected to a HDMI output of the Graphics card? A software used to tweak a "gaming" monitor? Maybe some utility that was updated recently and got stuck, or that should have been updated and was not.
Have you tried to replace the monitor cable? Or to use another port?