CUDA_ERROR_LAUNCH_FAILED

[Tuna_Ertemalp]

Just a bit of additional data: Another host running only one 1080Ti had been stopping with FahControl_22.exe crash dialogs, but without any CUDA errors in the logs. The logs would simply say something along the lines of "Not responding. Must have crashed. Too many errors. Failing." I switched that machine to disable CUDA at the same time as this quad GPU machine, and they are both happily churning without any crash since then. All my other hosts, one with 3x 1080Ti, one with 2x 3090 FTW3 Ultra, and 5 single GPU hosts, are still doing fine with CUDA enabled.

Tuna

[bruce]

The Development Team do harvest failure reports. I didn't inspect your logs (yet) but when a failed WU is reported, that report does include the log you posted plus other data of interest to the developer. Those reports generally lead to future FAHCore revisions that fix such problems. It's pretty rare that the developer will need to deliver a specialized debug version of the FAHCore to you or to harvest a failed WU you're holding.

[Tuna_Ertemalp]

The Development Team do harvest failure reports. I didn't inspect your logs (yet) but when a failed WU is reported, that report does include the log you posted plus other data of interest to the developer.

Ah! So, I am reading this as: When WUs fail on people's hosts, logs+info get uploaded to the server along with the failure report, automatically, without people like me doing anything.

That is awesome!

Tuna

PS: Still, if I can be of any further help, you know where to find me.

[bruce]

That also enables the developers to detect frequent/infrequent failures globally and concentrate on the important issues. Making a report here (and including the PRCG numbers) does allow a specific problem to be hilighted.

It also enables us to help individuals with problems that they can fix manually when we've seen it before or can guess what's actually wrong.

[bruce]

First check whether the background process FAHClient.exe is running.

Post your FAH log per the instuctions in my first post's sig block.

Assuming windows, you may or may not have enabled the optional feature "telnet"

[Tuna_Ertemalp]

That also enables the developers to detect frequent/infrequent failures globally and concentrate on the important issues. Making a report here (and including the PRCG numbers) does allow a specific problem to be hilighted.

It also enables us to help individuals with problems that they can fix manually when we've seen it before or can guess what's actually wrong.

Happy to hear that. Good engineering.

I hope this problem finds a solution before (if ever) it gets more widespread.

Tuna

[bruce]

Ah! So, I am reading this as: When WUs fail on people's hosts, logs+info get uploaded to the server along with the failure report, automatically, without people like me doing anything.

Whenever possible, yes.

Error reports are separated from completed WUs (which go directly to the researcher). Clean error reports (aka, logs) are also uploaded and generally discarded, so there are three distinct paths.

[Four actually. The statistical report with your name/team/PRCG/start-time/finish-time (etc?) go to the stats server.]

[Tuna_Ertemalp]

First check whether the background process FAHClient.exe is running.

When the crashes happen, FAHClient.exe is still running since there are other FahCore_xx.exe running without the crash (for the CPU and the non-crashed GPUs), and presumably this FAHClient.exe is still managing them. And, before you ask: I do make sure that I have only one FAHClient.exe running. That is how I discovered that 7.6.16->7.6.20 update resulted in two running during boot, and my report resulted in 7.6.21.

Post your FAH log per the instuctions in my first post's sig block.

I think I did, a few times in this thread, and actually even created a OneDrive repository for them and provided the link below. Your signature How to provide enough info to get helpful support points to a list of items and 6 - How To Post A Log File (Windows) talks about posting the logs.txt. Initially I had my verbosity at 5, but then I scaled back to 3, and those logs are either here or uploaded & shared on OneDrive. Am I missing something else due to old age?

Assuming windows, you may or may not have enabled the optional feature "telnet"

Yes, Windows 10. The TELNET CLIENT probably isn't enabled on my hosts uniformly, unless I needed to use it explicitely to access my router from that host etc. On this particular host with 4x GPUs, it is not. Is that relevant? On all my hosts, I can do things like "FAHClient --send-pause" or "FAHClient --send-unpause", without enabling the telnet client.

[bruce]

Am I missing something else due to old age?.

Nope. If one of us is missing things due to old age, it's probably me.

telnet is useful to to validate communications between FAHClient and FAHControl, but that doesn't relate to your issues.

Does the windows even log give indications that windows detects errors at the same time these crashes of a FAHCore happen?

[Tuna_Ertemalp]

Does the windows even log give indications that windows detects errors at the same time these crashes of a FAHCore happen?

Hmmm... Didn't check. Doubt it. If that's really important to you, I'd gladly reenable CUDA, and report back the next time this happens.

I'll also look at the event log around the one single GPU crash I had reported somewhere here on GPU #3. Only if I can get the UTC/PSD conversion right...

[bruce]

I'll also look at the event log around the one single GPU crash I had reported somewhere here on GPU #3. Only if I can get the UTC/PSD conversion right...

Hehehe. My brain is still a hour off.

PST = (UTC -8)
UTC 19:00 there is 11 am here in CA.

[Tuna_Ertemalp]

I'll also look at the event log around the one single GPU crash I had reported somewhere here on GPU #3. Only if I can get the UTC/PSD conversion right...

Hmmmm... For that one GPU #3 crash that happened while trying the GPUs one at a time, the CUDA_ERROR_LAUNCH_FAILED was reported in the FAH log at 2020-10-28T05:56:11Z, which was Oct 27th 10:56:11pm in Seattle. And, the Administrative Events filtered for errors/warnings has two events for one second later, and nothing relevant or close-in-time before...

First a warning about the driver ("Display driver nvlddmkm stopped responding and has successfully recovered."):

Code: Select all

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Display" /> 
  <EventID Qualifiers="0">4101</EventID> 
  <Level>3</Level> 
  <Task>0</Task> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2020-10-28T05:56:12.244990900Z" /> 
  <EventRecordID>157675</EventRecordID> 
  <Channel>System</Channel> 
  <Computer>QUADZILLA2</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data>nvlddmkm</Data> 
  <Data /> 
  </EventData>
  </Event>

Then the error about fahcore_22.exe crashing:

Code: Select all

Faulting application name: FahCore_22.exe, version: 0.0.0.0, time stamp: 0x5f656e96
Faulting module name: OpenMMCUDA.dll, version: 0.0.0.0, time stamp: 0x5f6518bf
Exception code: 0xc0000409
Fault offset: 0x00000000000e7fa0
Faulting process id: 0xcd8
Faulting application start time: 0x01d6acee8778021a
Faulting application path: C:\Users\Master\AppData\Roaming\FAHClient\cores\cores.foldingathome.org\win\64bit\22-0.0.13\Core_22.fah\FahCore_22.exe
Faulting module path: C:\Users\Master\AppData\Roaming\FAHClient\cores\cores.foldingathome.org\win\64bit\22-0.0.13\Core_22.fah\OpenMMCUDA.dll
Report Id: a5da76b9-7300-428e-aacf-08c567e69f14
Faulting package full name: 
Faulting package-relative application ID: 

Code: Select all

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Application Error" /> 
  <EventID Qualifiers="0">1000</EventID> 
  <Level>2</Level> 
  <Task>100</Task> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2020-10-28T05:56:12.714698500Z" /> 
  <EventRecordID>89408</EventRecordID> 
  <Channel>Application</Channel> 
  <Computer>QUADZILLA2</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data>FahCore_22.exe</Data> 
  <Data>0.0.0.0</Data> 
  <Data>5f656e96</Data> 
  <Data>OpenMMCUDA.dll</Data> 
  <Data>0.0.0.0</Data> 
  <Data>5f6518bf</Data> 
  <Data>c0000409</Data> 
  <Data>00000000000e7fa0</Data> 
  <Data>cd8</Data> 
  <Data>01d6acee8778021a</Data> 
  <Data>C:\Users\Master\AppData\Roaming\FAHClient\cores\cores.foldingathome.org\win\64bit\22-0.0.13\Core_22.fah\FahCore_22.exe</Data> 
  <Data>C:\Users\Master\AppData\Roaming\FAHClient\cores\cores.foldingathome.org\win\64bit\22-0.0.13\Core_22.fah\OpenMMCUDA.dll</Data> 
  <Data>a5da76b9-7300-428e-aacf-08c567e69f14</Data> 
  <Data /> 
  <Data /> 
  </EventData>
  </Event>

So, seems like

• something in FAHcore_22.exe ended making the Nvidia driver take enough time for some operation such that Win10 thought the driver became unresponsive and it needed to be shutdown/restarted,
• which crashed the FAHcore22_exe resulting in the UI crash dialog and froze the execution of that fahcore_22.exe,
• which made fahclient.exe report CUDA_ERROR_LAUNCH_FAILED in the log,
• and the driver recovered to continue serving the host's video/compute needs.
• Eventually I hit OK on the crash dialog that unforze that fahcore_22.exe, and things continued on that WU, possibly until this happened one too many times to the same WU, resulting in a failure upload.

At least that is my theory. Could be completely wrong.

All of my hosts always run the latest NVidia driver, so right now that is 457.09, but at the time it was 456.71. Since that is the same driver running on all of my hosts, some single GPU, some multi GPU, a lot of them with 1080Ti, I would be very doubtful that this is an NVidia driver issue. Of course, it could be exposing a driver bug only under QuadGPU config, even if FAH is using just a single GPU, but the probabilities are getting slimmer...

[Tuna_Ertemalp]

Hehehe. PST = (UTC -8)
UTC 19:00 there is 11 am here.

You'd think, but nope. Right now it is PST=(UTC-8), but USA only just changed from summer to winter time, thus from PSD to PST. So, last week, it was PSD, and PSD=(UTC-7).

Gets me every time...

[bruce]

Your theory is a good one. Development is aware of an occasional Windows Timeout event and working to develop a suitable fix. In the past, some people have tweaked the timeout setting in the registry successfully but unless you're really confident about editing the registry, I don't recommend it. A permanent fix is strongly preferred.

[Tuna_Ertemalp]

unless you're really confident about editing the registry

Oh, I surely am. 17-50yo range was being a developer.

The question is: what's been observed as a high enough value in which regkey?

I never noted the numbers that people actually found to work, so (like most developers know) it'll be trial-and-error because MS guessed at how long something should be able to inhibit a screen update before the user objected
.