GPU points are not noticed

[Folder_0815]

Added here config and log for WU 14446

Please note: my local time is 2hours higher compared to UTC
For me 19:29:20 UTC from the log is already 21:29:20 local time MEST

Code: Select all

19:29:20:************************* Folding@home Client *************************
19:29:20:        Website: https://foldingathome.org/
19:29:20:      Copyright: (c) 2009-2018 foldingathome.org
19:29:20:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:29:20:           Args: --open-web-control
19:29:20:         Config: C:\Users\eriwa\AppData\Roaming\FAHClient\config.xml
19:29:20:******************************** Build ********************************
19:29:20:        Version: 7.5.1
19:29:20:           Date: May 11 2018
19:29:20:           Time: 13:06:32
19:29:20:     Repository: Git
19:29:20:       Revision: 4705bf53c635f88b8fe85af7675557e15d491ff0
19:29:20:         Branch: master
19:29:20:       Compiler: Visual C++ 2008
19:29:20:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
19:29:20:       Platform: win32 10
19:29:20:           Bits: 32
19:29:20:           Mode: Release
19:29:20:******************************* System ********************************
19:29:20:            CPU: Intel(R) Core(TM) i7-3820QM CPU @ 2.70GHz
19:29:20:         CPU ID: GenuineIntel Family 6 Model 58 Stepping 9
19:29:20:           CPUs: 8
19:29:20:         Memory: 15.58GiB
19:29:20:    Free Memory: 13.57GiB
19:29:20:        Threads: WINDOWS_THREADS
19:29:20:     OS Version: 6.2
19:29:20:    Has Battery: true
19:29:20:     On Battery: false
19:29:20:     UTC Offset: 2
19:29:20:            PID: 1320
19:29:20:            CWD: C:\Users\eriwa\AppData\Roaming\FAHClient
19:29:20:             OS: Windows 10 Enterprise
19:29:20:        OS Arch: AMD64
19:29:20:           GPUs: 1
19:29:20:          GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:3 GK107 [Quadro 2100M]
19:29:20:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:3.0 Driver:10.0
19:29:20:OpenCL Device 1: Platform:0 Device:1 Bus:NA Slot:NA Compute:1.2 Driver:10.18
19:29:20:OpenCL Device 2: Platform:1 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:412.40
19:29:20:  Win32 Service: false
19:29:20:***********************************************************************

log:

Code: Select all

05:41:52:WU02:FS01:Connecting to 65.254.110.245:8080
05:41:52:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
05:41:52:WU02:FS01:Connecting to 18.218.241.186:80
05:41:53:WU02:FS01:Assigned to work server 3.133.76.19
05:41:53:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GK107 [Quadro 2100M] from 3.133.76.19
05:41:53:WU02:FS01:Connecting to 3.133.76.19:8080
05:43:17:WU02:FS01:Downloading 20.50MiB
05:43:23:WU02:FS01:Download 39.02%
05:43:29:WU02:FS01:Download 65.24%
05:43:35:WU02:FS01:Download 80.17%
05:43:41:WU02:FS01:Download 94.81%
05:43:43:WU02:FS01:Download complete
05:43:43:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:14446 run:0 clone:574 gen:18 core:0x22 unit:0x0000001e03854c135ea7b89040baf94a
05:43:43:WU02:FS01:Starting
05:43:43:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\eriwa\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 705 -lifeline 1320 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 1 -opencl-device 0 -cuda-device 0 -gpu 0
05:43:43:WU02:FS01:Started FahCore on PID 11284
05:43:43:WU02:FS01:Core PID:9844
05:43:43:WU02:FS01:FahCore 0x22 started
05:43:44:WU02:FS01:0x22:*********************** Log Started 2020-05-25T05:43:43Z ***********************
05:43:44:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
05:43:44:WU02:FS01:0x22:       Type: 0x22
05:43:44:WU02:FS01:0x22:       Core: Core22
05:43:44:WU02:FS01:0x22:    Website: https://foldingathome.org/
05:43:44:WU02:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
05:43:44:WU02:FS01:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
05:43:44:WU02:FS01:0x22:             <rafal.wiewiora@choderalab.org>
05:43:44:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 705 -lifeline 11284 -checkpoint 15
05:43:44:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 1 -opencl-device 0 -cuda-device
05:43:44:WU02:FS01:0x22:             0 -gpu 0
05:43:44:WU02:FS01:0x22:     Config: <none>
05:43:44:WU02:FS01:0x22:************************************ Build *************************************
05:43:44:WU02:FS01:0x22:    Version: 0.0.5
05:43:44:WU02:FS01:0x22:       Date: Apr 22 2020
05:43:44:WU02:FS01:0x22:       Time: 04:42:59
05:43:44:WU02:FS01:0x22: Repository: Git
05:43:44:WU02:FS01:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
05:43:44:WU02:FS01:0x22:     Branch: HEAD
05:43:44:WU02:FS01:0x22:   Compiler: Visual C++ 2008
05:43:44:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
05:43:44:WU02:FS01:0x22:   Platform: win32 10
05:43:44:WU02:FS01:0x22:       Bits: 64
05:43:44:WU02:FS01:0x22:       Mode: Release
05:43:44:WU02:FS01:0x22:************************************ System ************************************
05:43:44:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i7-3820QM CPU @ 2.70GHz
05:43:44:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 58 Stepping 9
05:43:44:WU02:FS01:0x22:       CPUs: 8
05:43:44:WU02:FS01:0x22:     Memory: 15.58GiB
05:43:44:WU02:FS01:0x22:Free Memory: 12.08GiB
05:43:44:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
05:43:44:WU02:FS01:0x22: OS Version: 6.2
05:43:44:WU02:FS01:0x22:Has Battery: true
05:43:44:WU02:FS01:0x22: On Battery: false
05:43:44:WU02:FS01:0x22: UTC Offset: 2
05:43:44:WU02:FS01:0x22:        PID: 9844
05:43:44:WU02:FS01:0x22:        CWD: C:\Users\eriwa\AppData\Roaming\FAHClient\work
05:43:44:WU02:FS01:0x22:         OS: Windows 10 Pro
05:43:44:WU02:FS01:0x22:    OS Arch: AMD64
05:43:44:WU02:FS01:0x22:********************************************************************************
05:43:44:WU02:FS01:0x22:Project: 14446 (Run 0, Clone 574, Gen 18)
05:43:44:WU02:FS01:0x22:Unit: 0x0000001e03854c135ea7b89040baf94a
05:43:44:WU02:FS01:0x22:Reading tar file core.xml
05:43:44:WU02:FS01:0x22:Reading tar file integrator.xml
05:43:44:WU02:FS01:0x22:Reading tar file state.xml
05:43:46:WU02:FS01:0x22:Reading tar file system.xml
05:43:48:WU02:FS01:0x22:Digital signatures verified
05:43:48:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
05:43:48:WU02:FS01:0x22:Version 0.0.5
05:44:26:WU02:FS01:0x22:Completed 0 out of 2000000 steps (0%)
05:44:26:WU02:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
07:05:59:WU02:FS01:0x22:Completed 20000 out of 2000000 steps (1%)
08:27:28:WU02:FS01:0x22:Completed 40000 out of 2000000 steps (2%)
09:49:01:WU02:FS01:0x22:Completed 60000 out of 2000000 steps (3%)
11:10:33:WU02:FS01:0x22:Completed 80000 out of 2000000 steps (4%)

[Folder_0815]

@ crawdaddy79
I am surprised as well
My GPU is that slow ((
so I shouldn't receive such WU at all.

I have analyzed the problem and found some other issue:

If I stop the process, it makes a checkpoint, to be able to continue from that step after restart.
Such a checkpoint is stored every 4%, so if I stop i.e. at 7%, it will restore from 4%, the time from 4% to 7% is lost.
For fast WUs it is not really a big deal.

In my case, if the WU needs 5 days, 4% is about 3.5 hours , if I stop every 3 hours, I never come to continue.
Thats what happend in the WU described in the OP.
If anybody is interested, I can post the complete log for a successfully comkpleted WU on the GPU, even with these gaps.

Maybe checkpoints should not be calculated by % of progess, but by time.

/br

[Folder_0815]

@crawdaddy79

Maybe something bad is slowing your system down? Do you have latest drivers, not using any intense screensavers, etc?

Nothing else running in the background, no screensavers etc
Taskmanager says 100% GPU for task FAHcore_22.exe
I'll countercheck the driver, in a post above I've posted the configuration, could anybody check that please, is there a problem with it?

What's happening there is the client doesn't know how long it will take to finish until it finishes the first percent or so. It defaults to two days (though most WUs I get default to one day).

That's what I observed as well, after the first percent it recalculates the time and changes the default to real values.
Usually the time to complete is factor 2 or 3 higher than the ETA.
But if the default for the max time is 5 days in the beginning, and after recalculation it is 4.7 days, while the ETA is increased from 2 days to 5 days, there is definitely something strange

I do think it keeps a historical database somewhere on our systems and pulls up information from previous WUs that match new ones (same project #, core type, etc.), and will use that to give you a relatively accurate ETA at 0.0% (this could be in my imagination though).

If this would be the case, I don't have influence on it, and it should be checked by a developer.
/br

[Folder_0815]

Hi all,
this the the corrupted WU:
WU 14446 (0, 574, 18) ETA 5.24 days 69100Pts
My computer has 4.57 days to complete this work unit.

It doesn't make sense to continue this WU.
How can I stop it, but continue the CPU- WU "16803 (19, 611, 37) 6 hours 53 mins 10167Pts"?
Anybody can help?
/br

[Folder_0815]

Update:
I have checked the GPU driver with Lenovo, there is no newer version available.

[JimboPalmer]

https://www.techpowerup.com/gpu-specs/q ... 000m.c1424
https://www.techpowerup.com/gpu-specs/q ... 100m.c2429

Tech power up claims the GK107 is the Quadro k2000M, and the GK106 is the Quadro K2100M, but they are similar chips that meet the minimum specs to start folding. both are over 7 years old, so won't be fast.

19:29:20:OpenCL Device 2: Platform:1 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:412.40

The latest driver is 418 also called 426.50

https://www.nvidia.com/Download/driverR ... 7253/en-us

"After Release 418, the NVIDIA Professional Drivers for Windows will no longer support Quadro notebook products based on the Kepler GPU architecture." - https://us.download.nvidia.com/Windows/ ... -notes.pdf

So this is not only the latest driver, it is the last driver.

[Folder_0815]

@ JimboPalmer

thanks a lot for all your research, I updated the driver for the K2000M with the 426.50.
The funny thing is that I made the test only via LenovoServiceBridge which said it's the latest and last one (the 412.40).
Question is of course, if this makes a big difference in the performance.
No idea how to verify that.
Basically, they say in the update notes that some security issues were fixed.

At first I have to get rid of the current WU.

/br

[JimboPalmer]

Folder_0815

I would guess fewer bugs, not more performance.

I hope it is fast enough to finish!

[PantherX]

...My GPU is that slow ((
so I shouldn't receive such WU at all...

Unfortunately, the current system only detects the GPU architecture, not it's model. It treats all Pascal GPUs as the same. The system was designed over 10 years ago and at that time, it worked well enough. Fast forward a decade, that is no longer enough and there are open issues to resolve this: https://github.com/FoldingAtHome/fah-issues/issues/1504

[Folder_0815]

@PantherX
@JimboPalmer "fewer bugs" that's a good one


thanks for the background information.

.... which is a little bit frustrating, because if I see how many people woldwide are currently supporting the FAH project, how much percentage of their donated processor power and electrical energy is wasted because of insufficent management of WUs.

Let us say, if a (old) computer runs 24/7 without any other running processes in the foreground, timing should be allright. But as soon as I have to interrupt the core processes occasionally, timing can become critical on slow machines.
To my GPU only small WUs should be assigned.

3 out of 14 GPU WUs were dumped since I started supporting FAH on April, 17
Here is an extract from the logs, if anybody is interested.
Also I can post more details for specific information, if required .

Code: Select all

*********************** Log Started 2020-04-15T00:05:22Z ***********************
01:28:00:WU01:FS01:0x22:Project: 11759 (Run 0, Clone 14126, Gen 37)
01:28:00:WU01:FS01:0x22:Unit: 0x0000004080fccb0a5e6f07faf851dec2
16:24:10:WU01:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
16:24:10:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11759 run:0 clone:14126 gen:37 core:0x22 unit:0x0000004080fccb0a5e6f07faf851dec2
2020-04-17T16:28:29Z ***********************

05:14:05:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
05:14:05:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11759 run:0 clone:9019 gen:29 core:0x22 unit:0x0000003d80fccb0a5e6eaf98519c57c3
2020-04-22T05:14:05Z ***********************

05:14:06:WU01:FS01:0x22:*********************** Log Started 2020-04-22T05:14:05Z ***********************
05:14:06:WU01:FS01:0x22:Project: 11752 (Run 0, Clone 288, Gen 23)
05:14:06:WU01:FS01:0x22:Unit: 0x000000328ca304e75e6a8059d6a618cf
22:21:42:WU01:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
22:21:42:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11752 run:0 clone:288 gen:23 core:0x22 unit:0x000000328ca304e75e6a8059d6a618cf

22:21:43:WU00:FS01:0x22:*********************** Log Started 2020-04-25T22:21:42Z ***********************
22:21:43:WU00:FS01:0x22:Project: 11747 (Run 0, Clone 4767, Gen 8)
22:21:43:WU00:FS01:0x22:Unit: 0x000000128ca304e75e6a7fdd670fbc70
17:52:40:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
17:52:40:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11747 run:0 clone:4767 gen:8 core:0x22 unit:0x000000128ca304e75e6a7fdd670fbc70

17:52:41:WU02:FS01:0x22:*********************** Log Started 2020-04-28T17:52:40Z ***********************
17:52:41:WU02:FS01:0x22:Project: 16433 (Run 546, Clone 3, Gen 4)
17:52:41:WU02:FS01:0x22:Unit: 0x0000000403854c135e9a4efd7346a221
16:31:49:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
16:31:49:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:16433 run:546 clone:3 gen:4 core:0x22 unit:0x0000000403854c135e9a4efd7346a221

16:34:35:WU00:FS01:0x22:*********************** Log Started 2020-04-30T16:34:34Z ***********************
16:34:35:WU00:FS01:0x22:Project: 11746 (Run 0, Clone 7272, Gen 25)
16:34:35:WU00:FS01:0x22:Unit: 0x000000258ca304f15e6bc33ba0a1b582
23:04:50:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
23:04:50:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11746 run:0 clone:7272 gen:25 core:0x22 unit:0x000000258ca304f15e6bc33ba0a1b582

18:01:16:WU00:FS01:0x22:*********************** Log Started 2020-05-04T18:01:15Z ***********************
18:01:16:WU00:FS01:0x22:Project: 14538 (Run 0, Clone 1186, Gen 113)
18:01:16:WU00:FS01:0x22:Unit: 0x0000009e0d5262775e7b9cdc49e6337d
03:33:17:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
03:33:17:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14538 run:0 clone:1186 gen:113 core:0x22 unit:0x0000009e0d5262775e7b9cdc49e6337d

03:33:17:WU01:FS01:0x22:*********************** Log Started 2020-05-06T03:33:17Z ***********************
03:33:17:WU01:FS01:0x22:Project: 14436 (Run 1203, Clone 4, Gen 26)
03:33:17:WU01:FS01:0x22:Unit: 0x0000002103854c135e9a790faecfd1f8
03:13:11:WARNING:WU01:FS01:Past final deadline 2020-05-11T03:13:10Z, dumping
############################################################################

03:14:18:WU02:FS01:0x22:*********************** Log Started 2020-05-11T03:14:17Z ***********************
03:14:18:WU02:FS01:0x22:Project: 11751 (Run 0, Clone 9923, Gen 11)
03:14:18:WU02:FS01:0x22:Unit: 0x000000148ca304e75e6bbd849e993bcd
21:22:13:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
21:22:13:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11751 run:0 clone:9923 gen:11 core:0x22 unit:0x000000148ca304e75e6bbd849e993bcd

21:22:14:WU01:FS01:0x22:*********************** Log Started 2020-05-12T21:22:13Z ***********************
21:22:14:WU01:FS01:0x22:Project: 11746 (Run 0, Clone 2273, Gen 48)
21:22:14:WU01:FS01:0x22:Unit: 0x0000004d8ca304f15e67feb8587d57fc
20:08:05:WU01:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
20:08:05:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11746 run:0 clone:2273 gen:48 core:0x22 unit:0x0000004d8ca304f15e67feb8587d57fc

20:08:06:WU00:FS01:0x22:*********************** Log Started 2020-05-15T20:08:06Z ***********************
20:08:06:WU00:FS01:0x22:Project: 14437 (Run 906, Clone 3, Gen 30)
20:08:06:WU00:FS01:0x22:Unit: 0x0000002b03854c135e9d22623431ce90
19:32:17:WARNING:WU00:FS01:Past final deadline 2020-05-20T19:32:16Z, dumping
############################################################################

20:52:51:WU01:FS01:0x22:*********************** Log Started 2020-05-20T20:52:51Z ***********************
20:52:51:WU01:FS01:0x22:Project: 14251 (Run 71, Clone 3, Gen 25)
20:52:51:WU01:FS01:0x22:Unit: 0x0000002acedfaa925eab032302785d08
...... sucessfully completed

currently runnning WU14446 it will not reach the time limit

but probably all the dumping is already covered by the change request on github ..

/br

[PantherX]

My suggestion would be that you stop folding on the GPU slot at least temporarily. In the meantime you can fold on all your CPU threads which would still contribute scientifically as there are lots of CPU WUs. Keep an eye out to see if there's any update on the GPU aspect or not. While it isn't the situation you expected, it is a middle ground so let's see what happens in the future