GPU CORE22 0.0.2 coming to FAH - p11737-9 feedback thread

[Zangetsu]

I've started the ADV flag on my other pc and ran into this problem today:

CPU:7820X
GPU:1070TI
16GB RAM

the Core22 downloaded but didn't start, it stayed on 0% for 40 min.
log=

Code: Select all

09:16:32:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:11737 run:0 clone:1910 gen:108 core:0x22 unit:0x000000728ca304f15dfbe6dad92609de
09:18:09:WU01:FS01:0x21:Completed 1000000 out of 1000000 steps (100%)
09:18:17:WU01:FS01:0x21:Saving result file logfile_01.txt
09:18:17:WU01:FS01:0x21:Saving result file checkpointState.xml
09:18:17:WU01:FS01:0x21:Saving result file checkpt.crc
09:18:17:WU01:FS01:0x21:Saving result file log.txt
09:18:17:WU01:FS01:0x21:Saving result file positions.xtc
09:18:18:WU01:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
09:18:19:WU01:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
09:18:19:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14282 run:0 clone:31 gen:107 core:0x21 unit:0x0000007480fccb0a5d9e11642a6bf408
09:18:19:WU01:FS01:Uploading 69.71MiB to 128.252.203.10
09:18:19:WU01:FS01:Connecting to 128.252.203.10:8080
09:18:19:WU00:FS01:Starting
09:18:19:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Zangetsu\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 705 -lifeline 12772 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
09:18:19:WU00:FS01:Started FahCore on PID 17848
09:18:19:WU00:FS01:Core PID:14292
09:18:19:WU00:FS01:FahCore 0x22 started
09:18:25:WU01:FS01:Upload 7.26%
09:18:31:WU01:FS01:Upload 17.03%
09:18:37:WU01:FS01:Upload 25.64%
09:18:43:WU01:FS01:Upload 32.27%
09:18:49:WU01:FS01:Upload 41.33%
09:18:55:WU01:FS01:Upload 51.19%
09:19:01:WU01:FS01:Upload 60.96%
09:19:07:WU01:FS01:Upload 70.20%
09:19:13:WU01:FS01:Upload 78.53%
09:19:19:WU01:FS01:Upload 87.95%
09:19:25:WU01:FS01:Upload 97.00%
09:19:27:WU01:FS01:Upload complete
09:19:27:WU01:FS01:Server responded WORK_ACK (400)
09:19:27:WU01:FS01:Final credit estimate, 81806.00 points
09:19:27:WU01:FS01:Cleaning up

that's where the log ends.
FAH advanced control show a 1h 45min TPF and 1293 PPD
The core didn't start, so after a reboot didn't fix it i looked into more possibility's.
I've seen a 70% system interrupts (Task manager) spike a few times , thought it might be the driver freaking but it wasn't that.
It was the "AV Comodo Internet Security Premium"that blocked it with the auto containment.
After unblocking it, the Core22 worked and showed 783k PPD/ 1min02 TPF.
There might be more AV's blocking core22, it was blocked on the setting (in the AV auto-containment tab) that it was "unrecognized".

[foldy]

There might be more AV's blocking core22, it was blocked on the setting (in the AV auto-containment tab) that it was "unrecognized".

I tried on google virustotal and there is no hit.
https://www.virustotal.com/gui/file/6d6 ... /detection

[bruce]

I've started the ADV flag on my other pc and ran into this problem today:

CPU:7820X
GPU:1070TI
16GB RAM

the Core22 downloaded but didn't start, it stayed on 0% for 40 min.

The log you posted does not show a download of Core_22 nor does it show a message containing "0%"
Maybe that's because of the information you selected to post or maybe it does indicate a problem.

There should be two steps. First, the message "Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:11737 run:0 clone:1910 gen:108 core:0x22" indicates that a WU from project 11737 did download. Assuming]/u] that was the first WU that needed Core_22 it should have been followed by messages indicating that the Core_22 software was then downloaded and I don't see those messages.

I'm going to assume that's your problem.

Unfortunately many firewall/AV suites that you might be running will block a download like that and it's not uncommon to have messages from your firewall disabled so you won't see a message saying (in essence) "Your firewall has protected you from the unauthorized download of a new piece of software by FAHClient." Personally, I"d go turn on messages like that one from my firewall, but that's up to you. In any case, you need to create an exception in your firewall configuration saying (in essence) "It's okay for FAHClient to download software" Then when you restart your system, FAHClient will restart and will try again and you'll either get a message saying the download is still blocked or you'll get messages saying the download succeeded and FAH will begin processing that WU.

The downloading of a new FAHCore is a rare event and it probably won't happen again until your firewall has been updated (turning off messages again?) and FAHClient has been updated (perhaps requiring the configuration of a new exception), and you've forgotten about this discussion.

[Zangetsu]

the Core22 downloaded but didn't start, it stayed on 0% for 40 min.

The log you posted does not show a download of Core_22 nor does it show a message containing "0%"
Maybe that's because of the information you selected to post or maybe it does indicate a problem.

There should be two steps. First, the message "Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:11737 run:0 clone:1910 gen:108 core:0x22" indicates that a WU from project 11737 did download. Assuming]/u] that was the first WU that needed Core_22 it should have been followed by messages indicating that the Core_22 software was then downloaded and I don't see those messages.

I'm going to assume that's your problem.

Unfortunately many firewall/AV suites that you might be running will block a download like that and it's not uncommon to have messages from your firewall disabled so you won't see a message saying (in essence) "Your firewall has protected you from the unauthorized download of a new piece of software by FAHClient." Personally, I"d go turn on messages like that one from my firewall, but that's up to you. In any case, you need to create an exception in your firewall configuration saying (in essence) "It's okay for FAHClient to download software" Then when you restart your system, FAHClient will restart and will try again and you'll either get a message saying the download is still blocked or you'll get messages saying the download succeeded and FAH will begin processing that WU.

The downloading of a new FAHCore is a rare event and it probably won't happen again until your firewall has been updated (turning off messages again?) and FAHClient has been updated (perhaps requiring the configuration of a new exception), and you've forgotten about this discussion.

Checked the AV log :
No stopping on the download of FAH client, it would seem weird that it would block a file on the firewall component.
No firewall log entry was found to indicate a stopping of the download.
I did find a log entry that it was "Run Virtually", and it did so about 4 times before i found about about it. (AV log)
After checking in with the saved log files of folding@home all 4 of them failed (Core22) due to being "Run Virtually"
FAHclient wasn't blocked from downloading software, the AV grabbed the Core22.exe and isolated it.
It's true that i don't get a message from an isolation event , and i'm looking into turning that on.
Here is the download of Core22 entry not being blocked by the firewall.
Being "Run Virtually" isn't something the client can detect i think.

Anyway fixed it , and it's running happy now (also for Core22 )

Code: Select all

01:37:27:WU00:FS01:0x21:Completed 990000 out of 1000000 steps (99%)
01:37:28:WU01:FS01:Connecting to 65.254.110.245:8080
01:37:28:WU01:FS01:Assigned to work server 140.163.4.241
01:37:28:WU01:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GP104 [GeForce GTX 1070 Ti] 8186 from 140.163.4.241
01:37:28:WU01:FS01:Connecting to 140.163.4.241:8080
01:37:29:WU01:FS01:Downloading 11.66MiB
01:37:36:WU01:FS01:Download 79.85%
01:37:39:WU01:FS01:Download complete
01:37:39:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:11737 run:0 clone:1735 gen:118 core:0x22 unit:0x000000778ca304f15dfbe6d80ee19f51
01:37:39:WU01:FS01:Downloading core from http://cores.foldingathome.org/v7/win/64bit/Core_22.fah
01:37:39:WU01:FS01:Connecting to cores.foldingathome.org:80
01:37:39:WU01:FS01:FahCore 22: Downloading 4.04MiB
01:37:40:WU01:FS01:FahCore 22: Download complete
01:37:40:WU01:FS01:Valid core signature
01:37:40:WU01:FS01:Unpacked 13.49MiB to cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe
01:38:57:WU00:FS01:0x21:Completed 1000000 out of 1000000 steps (100%)
01:39:05:WU00:FS01:0x21:Saving result file logfile_01.txt
01:39:05:WU00:FS01:0x21:Saving result file checkpointState.xml
01:39:05:WU00:FS01:0x21:Saving result file checkpt.crc
01:39:05:WU00:FS01:0x21:Saving result file log.txt
01:39:06:WU00:FS01:0x21:Saving result file positions.xtc
01:39:06:WU00:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
01:39:06:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
01:39:06:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14262 run:0 clone:17 gen:115 core:0x21 unit:0x0000008580fccb0a5daa0008d3dde443
01:39:06:WU00:FS01:Uploading 59.70MiB to 128.252.203.10
01:39:06:WU00:FS01:Connecting to 128.252.203.10:8080
01:39:07:WU01:FS01:Starting
01:39:07:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Zangetsu\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 705 -lifeline 12772 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
01:39:07:WU01:FS01:Started FahCore on PID 15336
01:39:07:WU01:FS01:Core PID:3136
01:39:07:WU01:FS01:FahCore 0x22 started
01:39:12:WU00:FS01:Upload 5.03%
01:39:18:WU00:FS01:Upload 14.03%
01:39:24:WU00:FS01:Upload 25.44%
01:39:30:WU00:FS01:Upload 36.96%
01:39:36:WU00:FS01:Upload 48.37%
01:39:42:WU00:FS01:Upload 59.78%
01:39:48:WU00:FS01:Upload 71.29%
01:39:54:WU00:FS01:Upload 82.71%
01:40:00:WU00:FS01:Upload 94.22%
01:40:03:WU00:FS01:Upload complete
01:40:03:WU00:FS01:Server responded WORK_ACK (400)
01:40:03:WU00:FS01:Final credit estimate, 73126.00 points
01:40:03:WU00:FS01:Cleaning up
03:21:16:WU01:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
03:21:16:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11737 run:0 clone:1735 gen:118 core:0x22 unit:0x000000778ca304f15dfbe6d80ee19f51
03:21:16:WARNING:WU01:FS01:Produced no results, failing
03:21:16:WU01:FS01:Connecting to 140.163.4.241:8080
03:21:16:WU01:FS01:Server responded WORK_ACK (400)
03:21:16:WU01:FS01:Final credit estimate, 93.00 points
03:21:16:WU01:FS01:Cleaning up
03:21:16:WU00:FS01:Connecting to 65.254.110.245:8080
03:21:17:WU00:FS01:Assigned to work server 128.252.203.10
03:21:17:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070 Ti] 8186 from 128.252.203.10
03:21:17:WU00:FS01:Connecting to 128.252.203.10:8080
03:21:18:WU00:FS01:Downloading 73.66MiB
03:21:24:WU00:FS01:Download 47.51%
03:21:29:WU00:FS01:Download complete
03:21:29:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:14282 run:0 clone:21 gen:112 core:0x21 unit:0x0000008080fccb0a5d9e1163e4d8c42b
03:21:29:WU00:FS01:Starting
03:21:29:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Zangetsu\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/nvidia/Core_21.fah/FahCore_21.exe -dir 00 -suffix 01 -version 705 -lifeline 12772 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
03:21:29:WU00:FS01:Started FahCore on PID 7916
03:21:29:WU00:FS01:Core PID:7360
03:21:29:WU00:FS01:FahCore 0x21 started
03:21:29:WU00:FS01:0x21:*********************** Log Started 2020-01-06T03:21:29Z ***********************
03:21:29:WU00:FS01:0x21:Project: 14282 (Run 0, Clone 21, Gen 112)
03:21:29:WU00:FS01:0x21:Unit: 0x0000008080fccb0a5d9e1163e4d8c42b
03:21:29:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
03:21:29:WU00:FS01:0x21:Machine: 1
03:21:29:WU00:FS01:0x21:Reading tar file core.xml
03:21:29:WU00:FS01:0x21:Reading tar file integrator.xml
03:21:29:WU00:FS01:0x21:Reading tar file state.xml
03:21:30:WU00:FS01:0x21:Reading tar file system.xml
03:21:30:WU00:FS01:0x21:Digital signatures verified
03:21:30:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
03:21:30:WU00:FS01:0x21:Version 0.0.20
03:21:53:WU00:FS01:0x21:Completed 0 out of 1000000 steps (0%)
03:21:53:WU00:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
03:23:37:WU00:FS01:0x21:Completed 10000 out of 1000000 steps (1%)

[Prettz]

When I get this project it immediately fails a gives a popup dialog saying something about Entry Point Not Found with openCL. I didn't take down the message before the client started on a new WU.
I'm on a Geforce 1060 6GB on Windows 7, and my video drivers are fairly old.

Here's the log:

Code: Select all

14:18:44:******************************* System ********************************
14:18:44:            CPU: Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz
14:18:44:         CPU ID: GenuineIntel Family 6 Model 58 Stepping 9
14:18:44:           CPUs: 8
14:18:44:         Memory: 15.82GiB
14:18:44:    Free Memory: 13.72GiB
14:18:44:        Threads: WINDOWS_THREADS
14:18:44:     OS Version: 6.1
14:18:44:    Has Battery: false
14:18:44:     On Battery: false
14:18:44:     UTC Offset: -5
14:18:44:            PID: 5988
14:18:44:             OS: Windows 7 Home Premium
14:18:44:        OS Arch: AMD64
14:18:44:           GPUs: 1
14:18:44:          GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:7 GP106 [GeForce GTX 1060 6GB] 4372
14:18:44:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:6.1 Driver:10.0
14:18:44:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:411.70
14:18:44:OpenCL Device 2: Platform:1 Device:1 Bus:NA Slot:NA Compute:1.2 Driver:10.18
14:18:44:  Win32 Service: false
14:18:44:***********************************************************************
...
22:31:07:WU01:FS01:Starting
22:31:07:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\...\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 705 -lifeline 5988 -checkpoint 8 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
22:31:07:WU01:FS01:Started FahCore on PID 10304
22:31:07:WU01:FS01:Core PID:8848
22:31:07:WU01:FS01:FahCore 0x22 started
22:31:13:WU00:FS01:Upload 19.21%
22:31:18:WARNING:WU01:FS01:FahCore returned an unknown error code which probably indicates that it crashed
22:31:18:WARNING:WU01:FS01:FahCore returned: UNKNOWN_ENUM (-1073741511 = 0xc0000139)

It gives this same error and shows the popup dialog every time it tries this WU.

[MeeLee]

Is it still necessary to set the beta flag for core 22?
I haven't done this, and am seeing 3+M PPD on my 2080Ti.
I presume core 22 works without the beta flag?

[Joe_H]

Is it still necessary to set the beta flag for core 22?
I haven't done this, and am seeing 3+M PPD on my 2080Ti.
I presume core 22 works without the beta flag?

Do you have any AMD 5700 XT GPU's? If so, then yes you need to keep the beta flag, otherwise no.

[bruce]

For Navi, support for FAHCore_22 will be extended from beta to advanced and eventually to full FAH, but the progress will be gradual. It's good to catch any problems while something new is exposed to a smaller segment of the FAH community than to expose it quickly and then discover a problem. (Of course, problems are never "expected" but it's still good to progress cautiously.)

[rafwiewiora]

Is it still necessary to set the beta flag for core 22?

You set flag to advanced? It's working right now because 11737 is in advanced -- I might turn it off anytime without notice so don't blame me if you suddenly stop getting work, my advice is to run beta flag only as I will ensure there's always beta work for core22 until we roll out full F@h projects.

[Prettz]

I got this project again and the popup error is "The procedure entry point clReleaseDevice could not be located in the dynamic link library OpenCL.dll." No idea if that's helpful beyond what was in my log.

[dfgirl12]

If you update to the latest Nvidia driver from the Nvidia website, do you still get the same error?
Old GPU drivers, and GPU overheating/unstable overclocking issues seemed to be the cause of most of the issues the last time Core22 went to advanced.

[MeeLee]

Is it still necessary to set the beta flag for core 22?

You set flag to advanced? It's working right now because 11737 is in advanced -- I might turn it off anytime without notice so don't blame me if you suddenly stop getting work, my advice is to run beta flag only as I will ensure there's always beta work for core22 until we roll out full F@h projects.

No flag was set to advanced. Just the regular client install.

I guess I got either some badly calibrated WUs, or your beta WUs went public for a while.
It's the only way I can explain +1M PPD rise from 12/26 to 01/04, as only a few GPUs got those fast WUs during this time.

[MeeLee]

FAHCore22:
project 11737
3 X Vega64 on PCIe 1X gen 2 with Risers v007
celeron G4400.
no over/underclock.
1 GPU got the project.
Gives 568k PPD with TPF of 1min 29.

core21 on overall projects (disregarding outliners) gave 600k PPD, so on risers(or on low end CPU) it gives a slight bottleneck.
It runs stable thou

*EDIT the PPD were wrong, now corrected

Just letting you know,
The cables you power the risers or GPUs with, may only provide power over not all the pins.
Especially Sata to 6pin converters often only provide power over 2 out of 3 (yellow wires) pins.
It took me a while to figure out that a lower PPD was the cause of low quality 6pin to sata converter cables.

[Prettz]

Updating the driver fixed it. There must have been an updated version of openCL at some point.
Also getting something like 20% higher PPD on this project than the usual.

[toTOW]

Thank you for confirming that we need to update minimum driver requirement for NV ... we'll try to identify what's exactly needed and post an update.