Project 16435 and RX Vega 56/64

[Crawdaddy79]

I hope I'm doing this right.

So since I've started folding I've been trying to find a pattern to my crashes, and by golly I think I've found it! It's something to do with this project and my hardware.

This is my Excel spreadsheet tally for the GPU WUs that I've done since last week.




Log from crash to startup:

Code: Select all

18:56:16:WU01:FS01:0x22:Completed 950000 out of 5000000 steps (19%)
18:56:44:WU00:FS00:0xa7:Completed 370000 out of 500000 steps (74%)
18:57:29:WU00:FS00:0xa7:Completed 375000 out of 500000 steps (75%)
18:58:15:WU00:FS00:0xa7:Completed 380000 out of 500000 steps (76%)
18:59:01:WU00:FS00:0xa7:Completed 385000 out of 500000 steps (77%)
18:59:24:WU01:FS01:0x22:Completed 1000000 out of 5000000 steps (20%)
18:59:47:WU00:FS00:0xa7:Completed 390000 out of 500000 steps (78%)
19:00:32:WU00:FS00:0xa7:Completed 395000 out of 500000 steps (79%)
19:01:18:WU00:FS00:0xa7:Completed 400000 out of 500000 steps (80%)
19:02:04:WU00:FS00:0xa7:Completed 405000 out of 500000 steps (81%)

/* CRASH - THREAD_STUCK_DEVICE_DRIVER BSOD */
/*Restart of FAH client below. */

*********************** Log Started 2020-05-06T19:12:55Z ***********************
19:12:56:WU01:FS01:Starting
19:12:56:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\crawd\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 16292 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
19:12:56:WU01:FS01:Started FahCore on PID 9284
19:12:56:WU01:FS01:Core PID:15856
19:12:56:WU01:FS01:FahCore 0x22 started
19:12:56:WU01:FS01:0x22:*********************** Log Started 2020-05-06T19:12:56Z ***********************
19:12:56:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
19:12:56:WU01:FS01:0x22:       Type: 0x22
19:12:56:WU01:FS01:0x22:       Core: Core22
19:12:56:WU01:FS01:0x22:    Website: https://foldingathome.org/
19:12:56:WU01:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
19:12:56:WU01:FS01:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
19:12:56:WU01:FS01:0x22:             <rafal.wiewiora@choderalab.org>
19:12:56:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 9284 -checkpoint 15
19:12:56:WU01:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
19:12:56:WU01:FS01:0x22:     Config: <none>
19:12:56:WU01:FS01:0x22:************************************ Build *************************************
19:12:56:WU01:FS01:0x22:    Version: 0.0.2
19:12:56:WU01:FS01:0x22:       Date: Dec 6 2019
19:12:56:WU01:FS01:0x22:       Time: 21:30:31
19:12:56:WU01:FS01:0x22: Repository: Git
19:12:56:WU01:FS01:0x22:   Revision: abeb39247cc72df5af0f63723edafadb23d5dfbe
19:12:56:WU01:FS01:0x22:     Branch: HEAD
19:12:56:WU01:FS01:0x22:   Compiler: Visual C++ 2008
19:12:56:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
19:12:56:WU01:FS01:0x22:   Platform: win32 10
19:12:56:WU01:FS01:0x22:       Bits: 64
19:12:56:WU01:FS01:0x22:       Mode: Release
19:12:56:WU01:FS01:0x22:************************************ System ************************************
19:12:56:WU01:FS01:0x22:        CPU: AMD Ryzen 7 2700X Eight-Core Processor
19:12:56:WU01:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
19:12:56:WU01:FS01:0x22:       CPUs: 16
19:12:56:WU01:FS01:0x22:     Memory: 31.95GiB
19:12:56:WU01:FS01:0x22:Free Memory: 24.00GiB
19:12:56:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
19:12:56:WU01:FS01:0x22: OS Version: 6.2
19:12:56:WU01:FS01:0x22:Has Battery: false
19:12:56:WU01:FS01:0x22: On Battery: false
19:12:56:WU01:FS01:0x22: UTC Offset: -4
19:12:56:WU01:FS01:0x22:        PID: 15856
19:12:56:WU01:FS01:0x22:        CWD: C:\Users\crawd\AppData\Roaming\FAHClient\work
19:12:56:WU01:FS01:0x22:         OS: Windows 10 Home
19:12:56:WU01:FS01:0x22:    OS Arch: AMD64
19:12:56:WU01:FS01:0x22:********************************************************************************
19:12:56:WU01:FS01:0x22:Project: 16435 (Run 1875, Clone 4, Gen 9)
19:12:56:WU01:FS01:0x22:Unit: 0x0000000e03854c135e9a4ef823fac093
19:12:56:WU01:FS01:0x22:Digital signatures verified
19:12:56:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
19:12:56:WU01:FS01:0x22:Version 0.0.2
19:12:56:WU01:FS01:0x22:  Found a checkpoint file
19:13:08:WU01:FS01:0x22:ERROR:Guru Meditation #0.3153f69697f955 (7.7) '01/01/stepsDone'
19:13:08:WU01:FS01:0x22:WARNING:Unexpected exit() call
19:13:08:WU01:FS01:0x22:WARNING:Unexpected exit from science code
19:13:08:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
19:13:08:WU01:FS01:0x22:Saving result file checkpointState.xml
19:13:09:WU01:FS01:0x22:Saving result file checkpt.crc
19:13:09:WU01:FS01:0x22:Saving result file positions.xtc
19:13:09:WU01:FS01:0x22:Saving result file science.log
19:13:09:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
19:13:09:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
19:13:09:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:16435 run:1875 clone:4 gen:9 core:0x22 unit:0x0000000e03854c135e9a4ef823fac093
19:13:09:WU01:FS01:Uploading 38.14MiB to 3.133.76.19

I have the 47 MB zip file of the work folder for this crash. Would that even be helpful?

[muziqaz]

This project demands not only GPU, but also want's few threads of the CPU, and if you are loading your CPU as well, it might impact the stability of this WU. There is another person shown who completed this particular WU successfully.
Projects also runs very poorly on AMD GCN based GPUs (which Vega is of)

[Crawdaddy79]

In that case, should I expect stability to increase if I increase the CPU priority of the GPU core (FahCore_a22, I think)?

[muziqaz]

No, no, there is no need to play around with priorities. You might end up with very unresponsive system.
If similar issues continue then you can start worrying about it, but if it was one off, don't worry

[BobWilliams757]

I've run 3 completed with my very modest Vega 11 onboard the 2400G in my system, with no errors.

It does seem to report in a strange way, with my estimated PPD jumping from base to about twice base points. And of the three that have run, all three of them ran beyond the timeout time by a few minutes with around the clock folding. BUT the estimated points in the log were still about twice the base points, as if it were a two day timeout rather than one.

I did notice a little CPU use. On the first run of that WU I thought I had something small running in the background. But for my system the CPU use seems to kind of vary up slightly, then down. It also seems the WU uses more system memory as well. In my case the onboard graphics and shared memory might be the cause of that part of things.


It is the only WU my modest system didn't reach the timeout with plenty of time to spare, but then it's giving points that are just on the slightly low side in PPD. Either way, the science is getting done.

When the first one was running, it kept showing I wouldn't meet the timeout, and PPD estimate was base. But when I did the math, unless a mega GPU picked it up right at the timeout time, I figured mine would still make the results available the soonest.

[Crawdaddy79]

My PPD per project does seem to vary, but within the projects it does not. Right now, Project 14416 is my favorite - my GPU processes @ 1.7M PPD. Project 14538, just 500k PPD

For 16435, it's pretty much in line with everything else, right around 1.0M PPD.

No, no, there is no need to play around with priorities. You might end up with very unresponsive system.
If similar issues continue then you can start worrying about it, but if it was one off, don't worry

My chart is a little hard to understand (EDIT: Actually it's impossible to understand because I never explained it before this post), but look back at it. The right column - anything that did not reach 100% is a crash, combined with a send-back of BAD_WORK_UNIT. (It's definitely not a one-off)

That said, the FahCore_22.exe does not chew up the CPU like the FahCore_a7.exe does. I'm leaving a7 at low priority, but am already experimenting with a22 at Above Normal. Seems to be working out so far - I just have to remember to set it with each new WU. As of now, I have a 3/3 success rate including finishing a Project 16435 WU.

[muziqaz]

If you are experiencing instabilities with multiple projects, please consider stopping folding, and making sure your hardware is stable. Dial down GPU OC (even if it is factory overclock), blowing out all the dust from heatsinks and fans, checking if your system memory is 100% stable, and same with CPU. Unstable hardware is only slowing down the project.

There is only few GPU projects which require full CPU thread. The rest only need couple of % of the CPU thread occasionally. There is no need to play around with priorities, as that will affect your system more than required. Setting priority to certain processes will not give you stability

[Crawdaddy79]

Luckily, I'm only experiencing instability with Project 16435 - so there is a definite issue with the way my hardware interprets what the WU is trying to do.

As for my system, I've underclocked/undervolted this thing as low (and everything in between) as it would go and it still crashed. I've maxed out my GPU fan at 70C and it still crashed. I've significantly increased the the airflow in my case and it still crashed. I overclocked it and saw a temporary increase in stability, but it still crashed.

The crashes are not heat related, not clock related, not power usage related. The only thing left is "bad hardware" but I don't think that fits either. I started the spreadsheet last week with the intent of finding a pattern because there didn't seem to be one, and the pattern I've found is this project #.

Raising the CPU priority of the GPU core has no impact to the general performance of my system. If doing this doesn't increase stability, I'll simply chalk it up to the growing list of things that don't work. At some point I will take your advice and consider not folding, but it will be when I've given up.

EDIT: To anyone who happens upon this thread with the same issue: raising CPU priority did not correct the issue, as muzicaz explained would be the case.

[PantherX]

Different Projects can have varying amount of stress on the GPU. In this case, your GPU isn't able to fold Project 16435 but can fold others. What happens if you set the frequencies and voltage to AMD stock values?

[Joe_H]

Also, one downside of undervolting is that while it does decrease heat, if taken too far it also reduces the stability of calculations. So you may need to try a slightly higher voltage and see if things work then.

[Crawdaddy79]

@ PantherX - all of the above examples are at default settings. No utilities other than Radeon Software and prior to recording every WU on the spreadsheet, I stopped using AMD's Power Saver preset because it didn't prove to be any more stable (and it was annoying having to re-enable it every time after a crash).

That said, I'm using the Power Saver preset now, because I don't think I've tried it since I installed these 20.4.2 drivers that supposedly fix folding black screen crashes.

@ Joe_H - The preset keeps the clock speed around 1400 Mhz and power draw seems stable at 166W. Does this seem like it would be out of whack?

[Joe_H]

Someone with this card should be able to comment on those settings. Does the Power Saver preset keep the clock from entering boost if load and temperature would otherwise allow?

[Crawdaddy79]

It does. Boost clock for my card is spec'd to 1574 Mhz, though I do suspect some MSI trickery. Under load with the preset, it touches 1475 but stabilizes at 1400 - 1450. The lower clock speeds add about 0.5% to my TPF in general, so it's not a big deal to use. Just waiting on some 16435 WUs now.


======================================EDIT BELOW===========================================
I figure I'll edit this post with new information rather than spam this thread over and over every time I contradict myself. When I find conclusive information, I'll make a new post and bump the thread. Without further ado:

EDIT: The above information is inaccurate. Watching my clock for three WUs in a row, it never broke 1500 Mhz. I got a 16435 WU, and it's ramping up to 1550 Mhz under load, and jumping to over 1650 Mhz when the load drops.

Code: Select all

GPU UTIL	GPU SCLK	GPU MCLK	GPU TEMP	GPU PWR	GPU FAN	GPU VRAM UTIL	CPU UTIL	RAM UTIL
68	1547	945	75	158	2416	583	82.8	10.44
80	1557	945	74	166	2433	595	84.92	10.44
71	1551	945	75	164	2435	595	82.52	10.44
84	1549	945	75	164	2466	606	85.02	10.42
80	1553	945	75	164	2474	606	85.99	10.38
85	1545	945	76	164	2476	606	82.16	10.39
81	1544	945	75	164	2499	597	84.03	10.37
73	1550	945	76	157	2498	597	82.46	10.37
82	1554	945	75	164	2516	597	84.01	10.39
86	1554	945	75	166	2521	597	85.79	10.36
79	1551	945	76	165	2533	597	84.78	10.38
0	1674	945	73	43	2499	597	88.47	10.5
84	1585	945	73	164	2454	597	90.96	10.46
77	1553	945	74	163	2411	597	83.32	10.43
79	1546	945	74	166	2392	597	84.94	10.44
75	1555	945	74	164	2398	597	83.89	10.39
81	1551	945	75	164	2400	581	82.9	10.4
70	1556	945	75	160	2433	562	88.04	10.42
77	1558	945	75	164	2448	562	84.99	10.41
75	1547	945	75	162	2453	562	82.92	10.42
82	1543	945	76	164	2486	562	81.57	10.43
58	1553	945	75	159	2495	562	87.25	10.43
79	1550	945	76	164	2499	562	88.95	10.42
67	1551	945	76	161	2520	559	88.52	10.43
42	1559	945	76	158	2511	560	91.4	10.4
80	1549	945	75	162	2534	561	91.93	10.48
67	1553	945	76	161	2516	558	88.96	10.49
67	1556	945	76	163	2531	558	89.2	10.47
2	1674	945	73	46	2501	558	91.59	10.58
75	1571	945	73	165	2455	558	89.01	10.49
83	1567	945	74	165	2411	558	91.05	10.43
81	1565	945	74	164	2397	600	90.86	10.45
79	1577	945	75	165	2396	600	90.37	10.47

Not totally sure why this is - but I read in a reddit thread (because finding what AMD says about the preset only turned up "Power Saver is only available on Vega 56/64 GPUs") that the Power Saver/Balanced/Turbo presets only change the power limit, and the clock speed can go wherever it wants as long as there's enough power available - theoretically if there's less power, the lower the limit for the clock speed. This is fun.

EDIT 2: In my testing, I have found strong evidence that this project saves a checkpoint at every percent. GPU WUs generally do this every 5%.

I've decided to discard Power Saver as a viable option, and instead manually tuned the clock down, knowing that would be the next step if a crash were to occur during Power Saver.

This is what my numbers look like. TPF is increased by 33%, but if it doesn't crash, it's an overall win.

Code: Select all

GPU UTIL	GPU SCLK	GPU MCLK	GPU TEMP	GPU PWR	GPU FAN	GPU VRAM UTIL	CPU UTIL	RAM UTIL
76	1383	945	68	133	2282	157	81.51	8.21
79	1482	945	69	131	2228	157	79.11	8.21
81	1466	945	69	138	2305	157	79.84	8.21
74	1455	945	70	127	2391	157	78.78	8.21
82	1475	945	70	145	2393	157	78.58	8.2
84	1468	945	70	145	2395	157	78.83	8.2
80	1479	945	70	137	2404	157	79.02	8.2
83	1477	945	70	136	2366	157	79.02	8.2
84	1465	945	70	137	2363	157	78.63	8.2
77	1471	945	70	137	2361	157	78.33	8.2
83	1462	945	70	137	2362	157	77.95	8.2
85	1482	945	70	140	2361	157	78.97	8.2
68	1476	945	70	134	2363	157	78.63	8.2
80	1467	945	70	136	2361	157	78.64	8.2
82	1478	945	70	138	2362	157	78.26	8.2
0	332	945	69	14	2365	157	81.36	8.24
64	262	945	68	71	2286	157	81.8	8.22
79	1439	945	69	132	2202	157	78.56	8.22
80	1472	945	69	142	2186	157	79.03	8.22
82	1468	945	70	140	2266	157	80	8.22
78	1473	945	70	139	2357	157	79.09	8.22
82	1471	945	70	142	2406	157	79.39	8.21
73	1468	945	70	130	2406	157	80.2	8.21
75	1478	945	70	137	2385	157	79.23	8.21
78	1471	945	70	145	2362	157	78.83	8.21
70	1480	945	71	140	2361	157	78.52	8.21
74	1472	945	70	131	2362	157	78.64	8.2

[PantherX]

...In my testing, I have found strong evidence that this project saves a checkpoint at every percent. GPU WUs generally do this every 5%...

Please note that GPUs write a checkpoint every 1% to 5% and that depends on the researcher as checkpoint allow sanity checks to be performed. If you're interested confirming the checkpoint for a WU, then navigate to the correct Work folder (which maps to the WU) and open up the science.log to view what the value is.
Path on my system: C:\Users\PantherX-H\AppData\Roaming\FAHClient\work\01\01
Value in science.log: Checkpoint write frequency: 20000 (1%)

[Crawdaddy79]

Interesting - for this WU it was actually 0.2%!

Code: Select all

[ Entering Main ]
  Reading core settings...
  Total number of steps: 5000000
  XTC write frequency: 10000
  Checkpoint write frequency: 10000 (0.2%)
  Number of frames per WU: 500

System crashed on a 16435 WU overnight, but starting the core this morning it looks like it recovered. Manual tune: Left everything else to automatic, but set the max clock speeds [State 5]: 1475 Mhz [State 6]: 1515 Mhz, [State 7]: 1555 Mhz (24 Mhz below spec). I'll drop each by 15 Mhz and see where this goes.

Log before crash (16435 being crunched by : 0x22)

Code: Select all

10:09:57:WU03:FS01:0x22:Completed 1550000 out of 5000000 steps (31%)
10:11:00:WU02:FS00:0xa7:Completed 97500 out of 250000 steps (39%)
10:13:36:WU03:FS01:0x22:Completed 1600000 out of 5000000 steps (32%)
10:14:00:WU02:FS00:0xa7:Completed 100000 out of 250000 steps (40%)
10:16:56:WU02:FS00:0xa7:Completed 102500 out of 250000 steps (41%)
10:19:47:WU02:FS00:0xa7:Completed 105000 out of 250000 steps (42%)
10:22:38:WU02:FS00:0xa7:Completed 107500 out of 250000 steps (43%)
10:25:29:WU02:FS00:0xa7:Completed 110000 out of 250000 steps (44%)
10:28:19:WU02:FS00:0xa7:Completed 112500 out of 250000 steps (45%)
10:31:11:WU02:FS00:0xa7:Completed 115000 out of 250000 steps (46%)
10:34:01:WU02:FS00:0xa7:Completed 117500 out of 250000 steps (47%)
10:36:52:WU02:FS00:0xa7:Completed 120000 out of 250000 steps (48%)
10:39:42:WU02:FS00:0xa7:Completed 122500 out of 250000 steps (49%)
10:42:33:WU02:FS00:0xa7:Completed 125000 out of 250000 steps (50%)
10:45:25:WU02:FS00:0xa7:Completed 127500 out of 250000 steps (51%)
10:48:15:WU02:FS00:0xa7:Completed 130000 out of 250000 steps (52%)
10:51:06:WU02:FS00:0xa7:Completed 132500 out of 250000 steps (53%)
10:53:56:WU02:FS00:0xa7:Completed 135000 out of 250000 steps (54%)
10:56:47:WU02:FS00:0xa7:Completed 137500 out of 250000 steps (55%)
10:59:38:WU02:FS00:0xa7:Completed 140000 out of 250000 steps (56%)
11:02:29:WU02:FS00:0xa7:Completed 142500 out of 250000 steps (57%)
11:05:19:WU02:FS00:0xa7:Completed 145000 out of 250000 steps (58%)
11:08:10:WU02:FS00:0xa7:Completed 147500 out of 250000 steps (59%)
11:11:00:WU02:FS00:0xa7:Completed 150000 out of 250000 steps (60%)
11:13:50:WU02:FS00:0xa7:Completed 152500 out of 250000 steps (61%)
11:16:42:WU02:FS00:0xa7:Completed 155000 out of 250000 steps (62%)
11:19:33:WU02:FS00:0xa7:Completed 157500 out of 250000 steps (63%)
11:22:23:WU02:FS00:0xa7:Completed 160000 out of 250000 steps (64%)
11:25:13:WU02:FS00:0xa7:Completed 162500 out of 250000 steps (65%)
11:28:03:WU02:FS00:0xa7:Completed 165000 out of 250000 steps (66%)