Declining performance - any advice?

[Nick200]

Hi

Grateful for any advice people have to offer. The issue is that the overall FAH output of my PCs seems to have fallen considerably despite having upgraded some of the hardware - and followed collective advice as to the best hardware mix and software configurations (thanks to Napoleon].

My systems used to bring in around 600K PPD but now generate only around 350 - 400K PPD total (if I am lucky). The number of WUs completed seems comparable with previous performance. My FAH ID is montague-cripps.

So, my info is as follows:

All my rigs operate under windows 8.1 Pro with media centre, fully patched, running FAH 7.4.4.

I have configured the BIOS where possible to use the iGPU so that the graphics cards are wholly dedicated to folding.

The individual details including FAH logs are as follows:

Rig 1
Intel driver 327.23
Intel Core i7 4790K @ 4.00GHz
3072MB NVIDIA GeForce GTX 780 (ASUStek Computer Inc)
3072MB NVIDIA GeForce GTX 780
FAH PPD estimate 115751

Code: Select all

*********************** Log Started 2015-02-25T18:31:47Z ***********************
18:31:47:************************* Folding@home Client *************************
18:31:47:      Website: http://folding.stanford.edu/
18:31:47:    Copyright: (c) 2009-2014 Stanford University
18:31:47:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:31:47:         Args: 
18:31:47:       Config: C:/Users/Nick/AppData/Roaming/FAHClient/config.xml
18:31:47:******************************** Build ********************************
18:31:47:      Version: 7.4.4
18:31:47:         Date: Mar 4 2014
18:31:47:         Time: 20:26:54
18:31:47:      SVN Rev: 4130
18:31:47:       Branch: fah/trunk/client
18:31:47:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
18:31:47:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
18:31:47:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
18:31:47:     Platform: win32 XP
18:31:47:         Bits: 32
18:31:47:         Mode: Release
18:31:47:******************************* System ********************************
18:31:47:          CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
18:31:47:       CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
18:31:47:         CPUs: 8
18:31:47:       Memory: 7.86GiB
18:31:47:  Free Memory: 3.34GiB
18:31:47:      Threads: WINDOWS_THREADS
18:31:47:   OS Version: 6.2
18:31:47:  Has Battery: false
18:31:47:   On Battery: false
18:31:47:   UTC Offset: 13
18:31:47:          PID: 8512
18:31:47:          CWD: C:/Users/Nick/AppData/Roaming/FAHClient
18:31:47:           OS: Windows 8.1 Pro with Media Center
18:31:47:      OS Arch: AMD64
18:31:47:         GPUs: 2
18:31:47:        GPU 0: NVIDIA:3 GK110 [GeForce GTX 780]
18:31:47:        GPU 1: NVIDIA:3 GK110 [GeForce GTX 780]
18:31:47:         CUDA: 3.5
18:31:47:  CUDA Driver: 5050
18:31:47:Win32 Service: false
18:31:47:***********************************************************************
18:31:47:<config>
18:31:47:  <!-- Network -->
18:31:47:  <proxy v=':8080'/>
18:31:47:
18:31:47:  <!-- Slot Control -->
18:31:47:  <power v='full'/>
18:31:47:
18:31:47:  <!-- User Information -->
18:31:47:  <passkey v='********************************'/>
18:31:47:  <team v='142900'/>
18:31:47:  <user v='Montague-Cripps'/>
18:31:47:
18:31:47:  <!-- Folding Slots -->
18:31:47:  <slot id='0' type='CPU'>
18:31:47:    <cpus v='6'/>
18:31:47:  </slot>
18:31:47:  <slot id='1' type='GPU'/>
18:31:47:  <slot id='2' type='GPU'/>
18:31:47:</config>
18:31:47:Trying to access database...
18:31:47:Successfully acquired database lock

Rig 2:
Intel driver 347.52
Intel Core i7 4770K @ 3.50GHz
4095MB NVIDIA GeForce GTX 980 (Gigabyte)
FAH PPD estimate 286,537

Code: Select all

*********************** Log Started 2015-02-26T07:02:04Z ***********************
07:02:04:************************* Folding@home Client *************************
07:02:04:      Website: http://folding.stanford.edu/
07:02:04:    Copyright: (c) 2009-2014 Stanford University
07:02:04:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
07:02:04:         Args: 
07:02:04:       Config: C:/Users/Nick/AppData/Roaming/FAHClient/config.xml
07:02:04:******************************** Build ********************************
07:02:04:      Version: 7.4.4
07:02:04:         Date: Mar 4 2014
07:02:04:         Time: 20:26:54
07:02:04:      SVN Rev: 4130
07:02:04:       Branch: fah/trunk/client
07:02:04:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
07:02:04:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
07:02:04:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
07:02:04:     Platform: win32 XP
07:02:04:         Bits: 32
07:02:04:         Mode: Release
07:02:04:******************************* System ********************************
07:02:04:          CPU: Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz
07:02:04:       CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
07:02:04:         CPUs: 8
07:02:04:       Memory: 15.89GiB
07:02:04:  Free Memory: 14.42GiB
07:02:04:      Threads: WINDOWS_THREADS
07:02:04:   OS Version: 6.2
07:02:04:  Has Battery: false
07:02:04:   On Battery: false
07:02:04:   UTC Offset: 13
07:02:04:          PID: 6060
07:02:04:          CWD: C:/Users/Nick/AppData/Roaming/FAHClient
07:02:04:           OS: Windows 8.1 Pro with Media Center
07:02:04:      OS Arch: AMD64
07:02:04:         GPUs: 1
07:02:04:        GPU 0: NVIDIA:5 GM204 [GeForce GTX 980]
07:02:04:         CUDA: 5.2
07:02:04:  CUDA Driver: 7000
07:02:04:Win32 Service: false
07:02:04:***********************************************************************
07:02:04:<config>
07:02:04:  <!-- Network -->
07:02:04:  <proxy v=':8080'/>
07:02:04:
07:02:04:  <!-- Slot Control -->
07:02:04:  <power v='FULL'/>
07:02:04:
07:02:04:  <!-- User Information -->
07:02:04:  <passkey v='********************************'/>
07:02:04:  <team v='142900'/>
07:02:04:  <user v='Montague-Cripps'/>
07:02:04:
07:02:04:  <!-- Folding Slots -->
07:02:04:  <slot id='0' type='CPU'>
07:02:04:    <client-type v='advanced'/>
07:02:04:  </slot>
07:02:04:  <slot id='2' type='GPU'>
07:02:04:    <client-type v='advanced'/>
07:02:04:  </slot>
07:02:04:</config>
07:02:04:Trying to access database...
07:02:04:Successfully acquired database lock
07:02:04:Enabled folding slot 00: READY cpu:7
07:02:04:Enabled folding slot 02: READY gpu:0:GM204 [GeForce GTX 980]

Rig 3:
Nvidia driver 327.23
Intel Core i7 2600 @ 3.40GHz
2048MB NVIDIA GeForce GTX 770 (NVIDIA)
FAH PPD estimate 22068

Code: Select all

*********************** Log Started 2015-02-28T02:20:23Z ***********************
02:20:23:************************* Folding@home Client *************************
02:20:23:      Website: http://folding.stanford.edu/
02:20:23:    Copyright: (c) 2009-2014 Stanford University
02:20:23:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
02:20:23:         Args: 
02:20:23:       Config: C:/ProgramData/FAHClient/config.xml
02:20:23:******************************** Build ********************************
02:20:23:      Version: 7.4.4
02:20:23:         Date: Mar 4 2014
02:20:23:         Time: 20:26:54
02:20:23:      SVN Rev: 4130
02:20:23:       Branch: fah/trunk/client
02:20:23:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
02:20:23:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
02:20:23:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
02:20:23:     Platform: win32 XP
02:20:23:         Bits: 32
02:20:23:         Mode: Release
02:20:23:******************************* System ********************************
02:20:23:          CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
02:20:23:       CPU ID: GenuineIntel Family 6 Model 42 Stepping 7
02:20:23:         CPUs: 8
02:20:23:       Memory: 7.91GiB
02:20:23:  Free Memory: 4.38GiB
02:20:23:      Threads: WINDOWS_THREADS
02:20:23:   OS Version: 6.2
02:20:23:  Has Battery: false
02:20:23:   On Battery: false
02:20:23:   UTC Offset: 13
02:20:23:          PID: 16768
02:20:23:          CWD: C:/ProgramData/FAHClient
02:20:23:           OS: Windows 8.1 Pro with Media Center
02:20:23:      OS Arch: AMD64
02:20:23:         GPUs: 1
02:20:23:        GPU 0: NVIDIA:3 GK104 [GeForce GTX 770]
02:20:23:         CUDA: 3.0
02:20:23:  CUDA Driver: 5050
02:20:23:Win32 Service: false
02:20:23:***********************************************************************
02:20:23:<config>
02:20:23:  <!-- Network -->
02:20:23:  <proxy v=':8080'/>
02:20:23:
02:20:23:  <!-- Slot Control -->
02:20:23:  <power v='full'/>
02:20:23:
02:20:23:  <!-- User Information -->
02:20:23:  <passkey v='********************************'/>
02:20:23:  <team v='142900'/>
02:20:23:  <user v='Montague-Cripps'/>
02:20:23:
02:20:23:  <!-- Folding Slots -->
02:20:23:  <slot id='0' type='GPU'>
02:20:23:    <client-type v='advanced'/>
02:20:23:  </slot>
02:20:23:  <slot id='1' type='CPU'>
02:20:23:    <client-type v='advanced'/>
02:20:23:  </slot>
02:20:23:</config>
02:20:23:Trying to access database...
02:20:23:Successfully acquired database lock
02:20:23:Enabled folding slot 00: READY gpu:0:GK104 [GeForce GTX 770]
02:20:23:Enabled folding slot 01: READY cpu:7
02:20:23:WU02:FS00:Starting
02:20:23:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 02 -suffix 01 -version 704 -lifeline 16768 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
02:20:23:WU02:FS00:Started FahCore on PID 17340
02:20:23:WU02:FS00:Core PID:24984
02:20:23:WU02:FS00:FahCore 0x18 started
02:20:23:WU00:FS01:Starting
02:20:23:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 704 -lifeline 16768 -checkpoint 15 -np 7
02:20:23:WU00:FS01:Started FahCore on PID 5572
02:20:23:WU00:FS01:Core PID:7572
02:20:23:WU00:FS01:FahCore 0xa4 started
02:20:24:WU00:FS01:0xa4:
02:20:24:WU00:FS01:0xa4:*------------------------------*
02:20:24:WU00:FS01:0xa4:Folding@Home Gromacs GB Core
02:20:24:WU00:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
02:20:24:WU00:FS01:0xa4:
02:20:24:WU00:FS01:0xa4:Preparing to commence simulation
02:20:24:WU00:FS01:0xa4:- Ensuring status. Please wait.
02:20:24:WU02:FS00:0x18:*********************** Log Started 2015-02-28T02:20:24Z ***********************
02:20:24:WU02:FS00:0x18:Project: 9114 (Run 26, Clone 1, Gen 7)
02:20:24:WU02:FS00:0x18:Unit: 0x0000000c0a3b1e78546a5a072618edda
02:20:24:WU02:FS00:0x18:CPU: 0x00000000000000000000000000000000
02:20:24:WU02:FS00:0x18:Machine: 0
02:20:24:WU02:FS00:0x18:Digital signatures verified
02:20:24:WU02:FS00:0x18:Folding@home GPU core18
02:20:24:WU02:FS00:0x18:Version 0.0.3
02:20:24:WU02:FS00:0x18:  Found a checkpoint file
02:20:33:WU00:FS01:0xa4:- Looking at optimizations...
02:20:33:WU00:FS01:0xa4:- Working with standard loops on this execution.
02:20:33:WU00:FS01:0xa4:- Previous termination of core was improper.
02:20:33:WU00:FS01:0xa4:- Files status OK
02:20:33:WU00:FS01:0xa4:- Expanded 923066 -> 1534204 (decompressed 166.2 percent)
02:20:33:WU00:FS01:0xa4:Called DecompressByteArray: compressed_data_size=923066 data_size=1534204, decompressed_data_size=1534204 diff=0
02:20:33:WU00:FS01:0xa4:- Digital signature verified
02:20:33:WU00:FS01:0xa4:
02:20:33:WU00:FS01:0xa4:Project: 9014 (Run 252, Clone 1, Gen 136)
02:20:33:WU00:FS01:0xa4:
02:20:33:WU00:FS01:0xa4:Entering M.D.
02:20:39:WU00:FS01:0xa4:Using Gromacs checkpoints
02:20:39:WU00:FS01:0xa4:Mapping NT from 7 to 7 
02:20:39:WU00:FS01:0xa4:Resuming from checkpoint
02:20:39:WU00:FS01:0xa4:Verified 00/wudata_01.log
02:20:39:WU00:FS01:0xa4:Verified 00/wudata_01.trr
02:20:39:WU00:FS01:0xa4:Verified 00/wudata_01.xtc
02:20:39:WU00:FS01:0xa4:Verified 00/wudata_01.edr
02:20:39:WU00:FS01:0xa4:Completed 79115 out of 250000 steps  (31%)
02:20:54:WU02:FS00:0x18:Completed 1700000 out of 2500000 steps (68%)
02:20:54:WU02:FS00:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
02:21:23:WU00:FS01:0xa4:Completed 80000 out of 250000 steps  (32%)

Rig 4:
Nvidia driver 340.52
Intel Core 2 Quad Q6600 @ 2.40GHz
1023MB NVIDIA GeForce GTX 750 (Gigabyte)
FAH PPD estimate 3982

Code: Select all

*********************** Log Started 2015-02-21T21:31:36Z ***********************
21:31:36:************************* Folding@home Client *************************
21:31:36:      Website: http://folding.stanford.edu/
21:31:36:    Copyright: (c) 2009-2014 Stanford University
21:31:36:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
21:31:36:         Args: 
21:31:36:       Config: C:/ProgramData/FAHClient/config.xml
21:31:36:******************************** Build ********************************
21:31:36:      Version: 7.4.4
21:31:36:         Date: Mar 4 2014
21:31:36:         Time: 20:26:54
21:31:36:      SVN Rev: 4130
21:31:36:       Branch: fah/trunk/client
21:31:36:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
21:31:36:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
21:31:36:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
21:31:36:     Platform: win32 XP
21:31:36:         Bits: 32
21:31:36:         Mode: Release
21:31:36:******************************* System ********************************
21:31:36:          CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
21:31:36:       CPU ID: GenuineIntel Family 6 Model 15 Stepping 11
21:31:36:         CPUs: 4
21:31:36:       Memory: 4.00GiB
21:31:36:  Free Memory: 1.70GiB
21:31:36:      Threads: WINDOWS_THREADS
21:31:36:   OS Version: 6.2
21:31:36:  Has Battery: false
21:31:36:   On Battery: false
21:31:36:   UTC Offset: 13
21:31:36:          PID: 4184
21:31:36:          CWD: C:/ProgramData/FAHClient
21:31:36:           OS: Windows 8.1 Pro with Media Center
21:31:36:      OS Arch: AMD64
21:31:36:         GPUs: 1
21:31:36:        GPU 0: NVIDIA:4 GM107 [GeForce GTX 750]
21:31:36:         CUDA: 5.0
21:31:36:  CUDA Driver: 6050
21:31:36:Win32 Service: false
21:31:36:***********************************************************************
21:31:36:<config>
21:31:36:  <!-- Network -->
21:31:36:  <proxy v=':8080'/>
21:31:36:
21:31:36:  <!-- Slot Control -->
21:31:36:  <power v='full'/>
21:31:36:
21:31:36:  <!-- User Information -->
21:31:36:  <passkey v='********************************'/>
21:31:36:  <team v='142900'/>
21:31:36:  <user v='Montague-Cripps'/>
21:31:36:
21:31:36:  <!-- Folding Slots -->
21:31:36:  <slot id='0' type='CPU'>
21:31:36:    <client-type v='advanced'/>
21:31:36:  </slot>
21:31:36:  <slot id='1' type='GPU'/>
21:31:36:</config>
21:31:36:Trying to access database...
21:31:36:Successfully acquired database lock

So is there anything I can do to get back to the levels I used to be able to generate? Is waiting for a "fixed" Nvidia driver the only hope, however much that seems to be pie in the sky? Is it down to the mix of cores, WUs and points schemes at the moment - will that change or is it just the new normal?

I am not keen to invest any more in hardware if all that results in is declining performance and less valued science...

Views?

[bruce]

There's hope. A potentially "fixed" version of the nVidia driver. It may be a bit before it's rolled out, but progress has been made.

You didn't mention the power saving features in Windows. Please confirm that all your CPUs are configured to avoid sleeping (unless you have a reason to choose otherwise).

From the configurations that you've shown, I see a total of 9 slots. WUs have been returned from more slots than that, but my data counts some older configurations that may have completed WUs and then gone off-line within the last 90 days due to the reinstall of one or more clients.

18:31:47: CPUs: 8 [Using 6]
18:31:47: GPU 0: NVIDIA:3 GK110 [GeForce GTX 780]
18:31:47: GPU 1: NVIDIA:3 GK110 [GeForce GTX 780]

07:02:04: CPUs: 8 [Using 7]
07:02:04: GPU 0: NVIDIA:5 GM204 [GeForce GTX 980]

02:20:23: CPUs: 8 [Using 7]
02:20:23: GPU 0: NVIDIA:3 GK104 [GeForce GTX 770]

21:31:36: CPUs: 4 [possibly using 3 ]
21:31:36: GPU 0: NVIDIA:4 GM107 [GeForce GTX 750]

Please identify a recent WU completed by each slot. Everything seems to be getting QRB except I'm seeing some strange errors from one or more of your GPUs. Please identify a recent WU that was completed successively by each GPU.

What types of changes have you made to your settings within that time period?

[Nick200]

Kia ora Bruce

Thanks for getting back to me.

Other than swapping some hardware around and consequential driver changes, which was completed in early Feb, I have not made any changes. The only other recent change and outage was due to blowing a GTX 780 which was RTMed and replaced. Unfortunately that took out the SLI rig (rig 1) as I had to install some new fans in to stop it overheating as it's summer down here. I am now up and folding with rig 1 - and with much better temperatures now.

I do not enable any power-saving features in Windows. I have switched off any sleep functions as I want the machines to run FAH unattended 24/7.

Yes, 9 slots at present.

• Rig 1: 3 slots: 2 x GPU and 1 CPU: recent WUs: WU03:FS00:0xa4:Project: 9007 (Run 1233, Clone 2, Gen 127); WU00:FS00:0xa4: Project: 9009 (Run 308, Clone 2, Gen 314), WU01:FS01: 0x17:Project: 13000 (Run 1906, Clone 0, Gen 41) and WU03:FS02:0x18:Project: 10488 (Run 3, Clone 4, Gen 15)
  
  Rig 2: 2 slots (1 x GPU, 1 x CPU): recent WUs: WU02:FS00:0xa4:Project: 9014 (Run 206, Clone 8, Gen 51); WU01:FS02:0x17:Project: 9411 (Run 1174, Clone 0, Gen 1)
  
  Rig 3: 2 slots (1 x GPU, 1 x CPU): recent WUs: WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:9114 run:26 clone:1 gen:7 core:0x18; WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:9014 run:252 clone:1 gen:136 core:0xa4 unit:0x000000a5664f2de453e558bd61673e61
  
  Rig 4: 2 slots (1 x GPU, 1 x CPU): recent WUs: WU04:FS00:0xa4:Project: 9016 (Run 285, Clone 7, Gen 87) and the GPU seems to be having trouble with some Bad Units at the moment:

Code: Select all

*********************** Log Started 2015-02-28T03:23:10Z ***********************
03:23:19:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:10478 run:1 clone:86 gen:28 core:0x18 unit:0x0000002d538b3dba548f6f28becd800f
03:23:19:WU00:FS01:Uploading 8.75MiB to 140.163.4.234
03:23:19:WU00:FS01:Connecting to 140.163.4.234:8080
03:23:20:WU02:FS01:Connecting to 171.67.108.200:80
03:23:26:WU00:FS01:Upload 0.71%
03:23:26:WU02:FS01:Assigned to work server 140.163.4.233
03:23:27:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GM107 [GeForce GTX 750] from 140.163.4.233
03:23:27:WU02:FS01:Connecting to 140.163.4.233:8080
03:23:28:WU02:FS01:Downloading 4.93MiB
03:23:32:WU00:FS01:Upload 6.43%
03:23:34:WU02:FS01:Download 10.15%
03:23:38:WU00:FS01:Upload 15.72%
03:23:40:WU02:FS01:Download 16.49%
03:23:45:WU00:FS01:Upload 25.01%
03:23:47:WU02:FS01:Download 19.03%
03:23:51:WU00:FS01:Upload 32.87%
03:23:53:WU02:FS01:Download 21.57%
03:23:57:WU00:FS01:Upload 43.59%
03:23:59:WU02:FS01:Download 30.45%
03:24:03:WU00:FS01:Upload 48.59%
03:24:06:WU02:FS01:Download 39.33%
03:24:09:WU00:FS01:Upload 55.73%
03:24:12:WU02:FS01:Download 46.94%
03:24:15:WU00:FS01:Upload 64.31%
03:24:18:WU02:FS01:Download 54.55%
03:24:21:WU00:FS01:Upload 82.17%
03:24:25:WU02:FS01:Download 59.63%
03:24:27:WU00:FS01:Upload 87.17%
03:24:31:WU02:FS01:Download 73.58%
03:24:33:WU00:FS01:Upload 92.89%
03:24:37:WU02:FS01:Download 81.19%
03:24:43:WU02:FS01:Download 91.34%
03:24:44:WU00:FS01:Upload complete
03:24:44:WU00:FS01:Server responded WORK_ACK (400)
03:24:44:WU00:FS01:Final credit estimate, 20197.00 points
03:24:44:WU00:FS01:Cleaning up
03:24:48:WU02:FS01:Download complete
03:24:48:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:10469 run:0 clone:276 gen:20 core:0x17 unit:0x00000028538b3db9538f3f9c1a131f2f
03:24:48:WU02:FS01:Starting
03:24:48:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe -dir 02 -suffix 01 -version 704 -lifeline 1180 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
03:24:48:WU02:FS01:Started FahCore on PID 6944
03:24:48:WU02:FS01:Core PID:6964
03:24:48:WU02:FS01:FahCore 0x17 started
03:24:50:WU02:FS01:0x17:*********************** Log Started 2015-02-28T03:24:50Z ***********************
03:24:50:WU02:FS01:0x17:Project: 10469 (Run 0, Clone 276, Gen 20)
03:24:50:WU02:FS01:0x17:Unit: 0x00000028538b3db9538f3f9c1a131f2f
03:24:50:WU02:FS01:0x17:CPU: 0x00000000000000000000000000000000
03:24:50:WU02:FS01:0x17:Machine: 1
03:24:50:WU02:FS01:0x17:Reading tar file state.xml
03:24:51:WU02:FS01:0x17:Reading tar file system.xml
03:24:52:WU02:FS01:0x17:Reading tar file integrator.xml
03:24:52:WU02:FS01:0x17:Reading tar file core.xml
03:24:52:WU02:FS01:0x17:Digital signatures verified
03:24:52:WU02:FS01:0x17:Folding@home GPU core17
03:24:52:WU02:FS01:0x17:Version 0.0.52
03:31:03:WU02:FS01:0x17:ERROR:exception: Force RMSE error of 453.878 with threshold of 5
03:31:03:WU02:FS01:0x17:Saving result file logfile_01.txt
03:31:03:WU02:FS01:0x17:Saving result file log.txt
03:31:03:WU02:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
03:31:04:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
03:31:04:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:10469 run:0 clone:276 gen:20 core:0x17 unit:0x00000028538b3db9538f3f9c1a131f2f
03:31:04:WU02:FS01:Uploading 2.19KiB to 140.163.4.233
03:31:04:WU02:FS01:Connecting to 140.163.4.233:8080
03:31:05:WU00:FS01:Connecting to 171.67.108.200:80
03:31:05:WU02:FS01:Upload complete
03:31:05:WU02:FS01:Server responded WORK_ACK (400)
03:31:05:WU02:FS01:Cleaning up
03:31:09:WU00:FS01:Assigned to work server 140.163.4.233
03:31:09:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GM107 [GeForce GTX 750] from 140.163.4.233
03:31:09:WU00:FS01:Connecting to 140.163.4.233:8080
03:31:10:WU00:FS01:Downloading 4.30MiB
03:31:17:WU00:FS01:Download 90.07%
03:31:17:WU00:FS01:Download complete
03:31:17:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:10468 run:0 clone:400 gen:28 core:0x17 unit:0x00000048538b3db9538cb4a2a8d50ea0
03:31:18:WU00:FS01:Starting
03:31:18:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe -dir 00 -suffix 01 -version 704 -lifeline 1180 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
03:31:18:WU00:FS01:Started FahCore on PID 4484
03:31:18:WU00:FS01:Core PID:3204
03:31:18:WU00:FS01:FahCore 0x17 started
03:31:24:WU00:FS01:0x17:*********************** Log Started 2015-02-28T03:31:23Z ***********************
03:31:24:WU00:FS01:0x17:Project: 10468 (Run 0, Clone 400, Gen 28)
03:31:24:WU00:FS01:0x17:Unit: 0x00000048538b3db9538cb4a2a8d50ea0
03:31:24:WU00:FS01:0x17:CPU: 0x00000000000000000000000000000000
03:31:24:WU00:FS01:0x17:Machine: 1
03:31:24:WU00:FS01:0x17:Reading tar file state.xml
03:31:25:WU00:FS01:0x17:Reading tar file system.xml
03:31:25:WU00:FS01:0x17:Reading tar file integrator.xml
03:31:25:WU00:FS01:0x17:Reading tar file core.xml
03:31:25:WU00:FS01:0x17:Digital signatures verified
03:31:25:WU00:FS01:0x17:Folding@home GPU core17
03:31:25:WU00:FS01:0x17:Version 0.0.52
03:35:57:WU00:FS01:0x17:ERROR:exception: Force RMSE error of 413.597 with threshold of 5
03:35:57:WU00:FS01:0x17:Saving result file logfile_01.txt
03:35:57:WU00:FS01:0x17:Saving result file log.txt
03:35:57:WU00:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
03:35:58:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
03:35:58:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:10468 run:0 clone:400 gen:28 core:0x17 unit:0x00000048538b3db9538cb4a2a8d50ea0
03:35:58:WU00:FS01:Uploading 2.19KiB to 140.163.4.233
03:35:58:WU00:FS01:Connecting to 140.163.4.233:8080
03:35:58:WU01:FS01:Connecting to 171.67.108.200:80
03:35:58:WU00:FS01:Upload complete
03:35:58:WU00:FS01:Server responded WORK_ACK (400)
03:35:59:WU00:FS01:Cleaning up
03:35:59:WU01:FS01:Assigned to work server 140.163.4.233
03:35:59:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GM107 [GeForce GTX 750] from 140.163.4.233
03:35:59:WU01:FS01:Connecting to 140.163.4.233:8080
03:36:00:WU01:FS01:Downloading 4.30MiB
03:36:06:WU01:FS01:Download 59.56%
03:36:08:WU01:FS01:Download complete
03:36:08:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:10468 run:0 clone:139 gen:30 core:0x17 unit:0x00000045538b3db9538cae15c761f55c
03:36:09:WU01:FS01:Starting
03:36:09:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe -dir 01 -suffix 01 -version 704 -lifeline 1180 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
03:36:09:WU01:FS01:Started FahCore on PID 6056
03:36:09:WU01:FS01:Core PID:1544
03:36:09:WU01:FS01:FahCore 0x17 started
03:36:15:WU01:FS01:0x17:*********************** Log Started 2015-02-28T03:36:14Z ***********************
03:36:15:WU01:FS01:0x17:Project: 10468 (Run 0, Clone 139, Gen 30)
03:36:15:WU01:FS01:0x17:Unit: 0x00000045538b3db9538cae15c761f55c
03:36:15:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
03:36:15:WU01:FS01:0x17:Machine: 1
03:36:15:WU01:FS01:0x17:Reading tar file state.xml
03:36:15:WU01:FS01:0x17:Reading tar file system.xml
03:36:16:WU01:FS01:0x17:Reading tar file integrator.xml
03:36:16:WU01:FS01:0x17:Reading tar file core.xml
03:36:16:WU01:FS01:0x17:Digital signatures verified
03:36:16:WU01:FS01:0x17:Folding@home GPU core17
03:36:16:WU01:FS01:0x17:Version 0.0.52
03:40:47:WU01:FS01:0x17:ERROR:exception: Force RMSE error of 411.68 with threshold of 5
03:40:47:WU01:FS01:0x17:Saving result file logfile_01.txt
03:40:47:WU01:FS01:0x17:Saving result file log.txt
03:40:47:WU01:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
03:40:48:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
03:40:48:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:10468 run:0 clone:139 gen:30 core:0x17 unit:0x00000045538b3db9538cae15c761f55c
03:40:48:WU01:FS01:Uploading 2.19KiB to 140.163.4.233
03:40:48:WU01:FS01:Connecting to 140.163.4.233:8080
03:40:48:WU00:FS01:Connecting to 171.67.108.200:80
03:40:49:WU01:FS01:Upload complete
03:40:49:WU01:FS01:Server responded WORK_ACK (400)
03:40:49:WU01:FS01:Cleaning up
03:40:50:WU00:FS01:Assigned to work server 140.163.4.233
03:40:50:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GM107 [GeForce GTX 750] from 140.163.4.233
03:40:50:WU00:FS01:Connecting to 140.163.4.233:8080
03:40:51:WU00:FS01:Downloading 4.31MiB
03:40:57:WU00:FS01:Download 91.33%
03:40:57:WU00:FS01:Download complete
03:40:57:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:10468 run:0 clone:241 gen:28 core:0x17 unit:0x00000053538b3db9538cb0a541baf841
03:40:57:WU00:FS01:Starting
03:40:57:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe -dir 00 -suffix 01 -version 704 -lifeline 1180 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
03:40:57:WU00:FS01:Started FahCore on PID 1284
03:40:57:WU00:FS01:Core PID:6592
03:40:57:WU00:FS01:FahCore 0x17 started
03:41:03:WU00:FS01:0x17:*********************** Log Started 2015-02-28T03:41:03Z ***********************
03:41:03:WU00:FS01:0x17:Project: 10468 (Run 0, Clone 241, Gen 28)
03:41:03:WU00:FS01:0x17:Unit: 0x00000053538b3db9538cb0a541baf841
03:41:03:WU00:FS01:0x17:CPU: 0x00000000000000000000000000000000
03:41:03:WU00:FS01:0x17:Machine: 1
03:41:03:WU00:FS01:0x17:Reading tar file state.xml
03:41:04:WU00:FS01:0x17:Reading tar file system.xml
03:41:04:WU00:FS01:0x17:Reading tar file integrator.xml
03:41:04:WU00:FS01:0x17:Reading tar file core.xml
03:41:04:WU00:FS01:0x17:Digital signatures verified
03:41:04:WU00:FS01:0x17:Folding@home GPU core17
03:41:04:WU00:FS01:0x17:Version 0.0.52
03:45:33:WU00:FS01:0x17:ERROR:exception: Force RMSE error of 412.386 with threshold of 5
03:45:33:WU00:FS01:0x17:Saving result file logfile_01.txt
03:45:33:WU00:FS01:0x17:Saving result file log.txt
03:45:33:WU00:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
03:45:33:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
03:45:33:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:10468 run:0 clone:241 gen:28 core:0x17 unit:0x00000053538b3db9538cb0a541baf841
03:45:33:WU00:FS01:Uploading 2.19KiB to 140.163.4.233
03:45:33:WU00:FS01:Connecting to 140.163.4.233:8080
03:45:34:WU01:FS01:Connecting to 171.67.108.200:80
03:45:34:WU00:FS01:Upload complete
03:45:34:WU00:FS01:Server responded WORK_ACK (400)
03:45:34:WU00:FS01:Cleaning up
03:45:35:WU01:FS01:Assigned to work server 140.163.4.233
03:45:35:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GM107 [GeForce GTX 750] from 140.163.4.233
03:45:35:WU01:FS01:Connecting to 140.163.4.233:8080
03:45:37:WU01:FS01:Downloading 2.29MiB
03:45:40:WU01:FS01:Download complete
03:45:40:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:10466 run:0 clone:323 gen:33 core:0x17 unit:0x00000065538b3db95382305428cd110d
03:45:40:WU01:FS01:Starting
03:45:40:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe -dir 01 -suffix 01 -version 704 -lifeline 1180 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
03:45:41:WU01:FS01:Started FahCore on PID 3932
03:45:41:WU01:FS01:Core PID:2516
03:45:41:WU01:FS01:FahCore 0x17 started
03:45:47:WU01:FS01:0x17:*********************** Log Started 2015-02-28T03:45:47Z ***********************
03:45:47:WU01:FS01:0x17:Project: 10466 (Run 0, Clone 323, Gen 33)
03:45:47:WU01:FS01:0x17:Unit: 0x00000065538b3db95382305428cd110d
03:45:47:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
03:45:47:WU01:FS01:0x17:Machine: 1
03:45:47:WU01:FS01:0x17:Reading tar file state.xml
03:45:47:WU01:FS01:0x17:Reading tar file system.xml
03:45:48:WU01:FS01:0x17:Reading tar file integrator.xml
03:45:48:WU01:FS01:0x17:Reading tar file core.xml
03:45:48:WU01:FS01:0x17:Digital signatures verified
03:45:48:WU01:FS01:0x17:Folding@home GPU core17
03:45:48:WU01:FS01:0x17:Version 0.0.52
03:47:38:WU01:FS01:0x17:Completed 0 out of 5000000 steps (0%)
03:47:38:WU01:FS01:0x17:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
03:55:38:WU01:FS01:0x17:Completed 50000 out of 5000000 steps (1%)
04:03:22:WU01:FS01:0x17:Completed 100000 out of 5000000 steps (2%)
04:11:23:WU01:FS01:0x17:Completed 150000 out of 5000000 steps (3%)
04:19:07:WU01:FS01:0x17:Completed 200000 out of 5000000 steps (4%)
04:26:52:WU01:FS01:0x17:Completed 250000 out of 5000000 steps (5%)
04:34:52:WU01:FS01:0x17:Completed 300000 out of 5000000 steps (6%)
04:42:35:WU01:FS01:0x17:Completed 350000 out of 5000000 steps (7%)
04:50:35:WU01:FS01:0x17:Completed 400000 out of 5000000 steps (8%)
04:58:18:WU01:FS01:0x17:Completed 450000 out of 5000000 steps (9%)
05:06:01:WU01:FS01:0x17:Completed 500000 out of 5000000 steps (10%)
05:14:02:WU01:FS01:0x17:Completed 550000 out of 5000000 steps (11%)
05:21:45:WU01:FS01:0x17:Completed 600000 out of 5000000 steps (12%)
05:29:44:WU01:FS01:0x17:Completed 650000 out of 5000000 steps (13%)
05:37:27:WU01:FS01:0x17:Completed 700000 out of 5000000 steps (14%)
05:45:10:WU01:FS01:0x17:Completed 750000 out of 5000000 steps (15%)
05:53:10:WU01:FS01:0x17:Completed 800000 out of 5000000 steps (16%)
06:00:53:WU01:FS01:0x17:Completed 850000 out of 5000000 steps (17%)
06:08:54:WU01:FS01:0x17:Completed 900000 out of 5000000 steps (18%)
06:16:37:WU01:FS01:0x17:Completed 950000 out of 5000000 steps (19%)
06:24:20:WU01:FS01:0x17:Completed 1000000 out of 5000000 steps (20%)
06:32:20:WU01:FS01:0x17:Completed 1050000 out of 5000000 steps (21%)
06:40:03:WU01:FS01:0x17:Completed 1100000 out of 5000000 steps (22%)
06:48:05:WU01:FS01:0x17:Completed 1150000 out of 5000000 steps (23%)

I tried posting the full logs from all 4 rigs but as they have been on constantly for weeks the post exceeded the character limit. So I deleted everything but the headers. Let me know if you want the full log details for any of the rigs.

Other slots may occasionally produce WUs but they are laptop based, and I don't monitor them.

I think you might also have seen some WUs from a failed experiment of mine: I tried installing an old GT9880 and a GT250 in spare PCIe-16 slots on top of what was in the main slots - but I could not get FAH 7.4.4 and the FAH 6.2 GPU2 console to co-exist. There were too many driver clashes, crahses and confusions. So that stopped and with it any extra WUs.

As for the future, I will wait for the promised Nvidia driver that sorts out all these performance issues. I run 347.52 for the GTX 980 and it seems fine. Then again, I cannot tell whether other drivers would have performed better as that's the only driver I have tried for that bit of kit. It is odd that its release went by without any postings on this forum at all.

I will also try the NACL client on other PCs that I access - but only when there is a version that will work over my work network. And I will also try the android version once it is available for more than just Sony handsets.

Naku noa, na

Nick

[rwh202]

The 750ti will need a driver update to avoid the errors - 347 and up seem to be the requirements.
The lower PPD could just be the mix of WUs being seen - core-18 on Maxwell gets roughly half the ppd at the moment, but the latest drivers and beta core seems to fix that.
Hope that helps

[7im]

Yep. The mix of work units assigned is always fluctuating as old projects finish and new projects start. Long term PPD will average out but will never be constant.

[bruce]

Full logs can exceed the posting limits but there is often useful information if you just trim it down to something approaching the limit. For example, I could have picked out recent GPU WUs without having to ask you for them. (No big deal, though.)

As far as mVodoa divers are concerned, they have divided their downloads into two choices: Legacy GPUs and up-to-date GPUs. (I'm not sure if the drivers for your GT9880 and GT250 would have coexisted happily with drivers for one of your GTX 7x0 or GTX 980 but so far I have not needed to know that. Even if they do, it would require two downloads.)

In any case, rwh202 is right about a strong recommendation to update the drivers for all GMxxx (Maxwell) hardware (Rig 4 in particular). That may be the sole cause of your reduced throughput since the Force RMSE error / BAD_WORK_UNIT / SEND error:FAULTY errors would have been eliminated. At the present time, updating the drivers for the GKxxx GPUs is optional.

[Nick200]

Thanks for all the advice

On long-run performance, I agree that it should level out - but my weekly stats at http://folding.extremeoverclocking.com/ ... =&u=661730 are all over the place, as can be seen at:

As for updating the drivers for rig 4, FAH reports the GTX 750 as a GM107. It's not a Ti card and I thought that, in that case, the Maxwell drivers would halve the card's performance? Happy for that to be wrong ....

I baulk at trying to install two Nvidia drivers on one machine - is that even possible? Certainly the legacy and new graphics hardware did not seem to want to coexist under one driver, regardless of version. Problems were compounded by the need for two versions of FAH as well.

I couldn't find online an earlier version than 327

Again, thanks

Nick

[bruce]

The gridpoint on 01.04 and the next grid point are below the green band but the gridpoint on 01.18 is above the green band. That simply indicates that some of the results from the first two weeks were delayed and appeared in the third week. 7im already said that things average out over time. I'd start at 10.12 (probably when your last system came on-line) and draw a horizontal line to the right. If that were the trend line, you'd have trouble arguing that it shows declining performance.

The GM107 is a Maxwell. Drivers for GMxxx require CUDA 7 which comes with the new drivers to avoid aborted WUs. I have not isolated the GMxxx portion of my performance data so I can't comment on a possible loss of Maxwell performance (other than aborted WUs, which are easy to spot). Perhaps you can fold the first half of a WU with one set of drivers, replace them with the other set of drivers, and finish the WU to give us a bit of data.

[Nick200]

Hi Bruce

Sure. So, I replaced the 340.52 driver for the GTX 750 with the 347.52 version, 45% through a Core 17 WU for Project 9201.

(I had previously tried to downgrade to the 327 version but that installer decided that there was no compatible hardware and gave up.)

At the start (i.e. at 45.00% complete), FAH showed 6.04 mins per TPF, with 5 hours 33 minutes to complete the WU at an estimated total credit of 19,470 on a base credit of 8000, with estimated PPD of 46,214.

Just a few minutes before completion. FAH showed 6.06 mins per TPF, with 0 hours 2 minutes to complete the WU at an estimated total credit of 19,139 on a base credit of 8000, with estimated PPD of 45,882. On completion FAH reported 19,087 credits.

So it looks as though upgrading the driver had no significant adverse or positive effect, although there's a slight drop.

I will stick with it until the fixed nvidia driver arrives.

Nick

[gwildperson]

I will stick with it until the fixed nvidia driver arrives.

Nick

In my reading of the general sentiment around the forum, 347.52 IS the fixed driver.