Work units too big for GPU

[taken31]

Hello, I have just started working with F@H today. I downloaded the software, setup a new user with a pass key and the whole 9-yards. But when I started the actual program, everything is at 0% On the web client, everything is at 0% on the "advanced control" client on the computer, everything is at 0%. I don't think it is doing anything. Here is my logs:

Code: Select all

*********************** Log Started 2014-10-14T01:27:23Z ***********************
01:27:23:************************* Folding@home Client *************************
01:27:23:      Website: http://folding.stanford.edu/
01:27:23:    Copyright: (c) 2009-2014 Stanford University
01:27:23:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
01:27:23:         Args: --open-web-control
01:27:23:       Config: <none>
01:27:23:******************************** Build ********************************
01:27:23:      Version: 7.4.4
01:27:23:         Date: Mar 4 2014
01:27:23:         Time: 20:26:54
01:27:23:      SVN Rev: 4130
01:27:23:       Branch: fah/trunk/client
01:27:23:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
01:27:23:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
01:27:23:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
01:27:23:     Platform: win32 XP
01:27:23:         Bits: 32
01:27:23:         Mode: Release
01:27:23:******************************* System ********************************
01:27:23:          CPU: AMD FX(tm)-6300 Six-Core Processor
01:27:23:       CPU ID: AuthenticAMD Family 21 Model 2 Stepping 0
01:27:23:         CPUs: 6
01:27:23:       Memory: 7.97GiB
01:27:23:  Free Memory: 5.49GiB
01:27:23:      Threads: WINDOWS_THREADS
01:27:23:   OS Version: 6.1
01:27:23:  Has Battery: false
01:27:23:   On Battery: false
01:27:23:   UTC Offset: -7
01:27:23:          PID: 4644
01:27:23:          CWD: C:/Users/Slippy-Fish/AppData/Roaming/FAHClient
01:27:23:           OS: Windows 7 Ultimate
01:27:23:      OS Arch: AMD64
01:27:23:         GPUs: 2
01:27:23:        GPU 0: NVIDIA:1 G92 [GeForce 8800 GT]
01:27:23:        GPU 1: NVIDIA:3 GK107 [GeForce GT 630]
01:27:23:         CUDA: 1.1
01:27:23:  CUDA Driver: 6050
01:27:23:Win32 Service: false
01:27:23:***********************************************************************
01:27:23:<config>
01:27:23:  <!-- Folding Slots -->
01:27:23:</config>
01:27:23:Connecting to assign-GPU.stanford.edu:80
01:27:26:Updated GPUs.txt
01:27:26:Read GPUs.txt
01:27:26:Trying to access database...
01:27:28:Successfully acquired database lock
01:27:28:Enabled folding slot 00: PAUSED cpu:3 (not configured)
01:27:28:Enabled folding slot 01: PAUSED gpu:0:G92 [GeForce 8800 GT] (not configured)
01:27:28:Enabled folding slot 02: PAUSED gpu:1:GK107 [GeForce GT 630] (not configured)
01:27:39:11:127.0.0.1:New Web connection
01:28:04:Set client configured
01:28:05:WU00:FS00:Connecting to 171.67.108.200:8080
01:28:05:WU01:FS01:Connecting to 171.67.108.200:8080
01:28:05:WU02:FS02:Connecting to 171.67.108.200:8080
01:28:10:WU01:FS01:Connecting to 171.67.108.201:80
01:28:10:WU00:FS00:Connecting to 171.67.108.200:8080
01:28:10:WU02:FS02:Connecting to 171.67.108.201:80
01:28:11:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
01:28:11:WU01:FS01:Connecting to 171.67.108.204:80
01:28:11:WU00:FS00:Assigned to work server 128.252.203.2
01:28:11:WU00:FS00:Requesting new work unit for slot 00: READY cpu:3 from 128.252.203.2
01:28:11:WU00:FS00:Connecting to 128.252.203.2:8080
01:28:12:WU02:FS02:Assigned to work server 140.163.4.231
01:28:12:WU02:FS02:Requesting new work unit for slot 02: READY gpu:1:GK107 [GeForce GT 630] from 140.163.4.231
01:28:12:WU02:FS02:Connecting to 140.163.4.231:8080
01:28:12:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.204:80': Empty work server assignment
01:28:12:ERROR:WU01:FS01:Exception: Could not get an assignment
01:28:12:WU01:FS01:Connecting to 171.67.108.201:80
01:28:12:WU02:FS02:Downloading 4.84MiB
01:28:13:WU00:FS00:Downloading 1.17MiB
01:28:13:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
01:28:13:WU01:FS01:Connecting to 171.67.108.204:80
01:28:15:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.204:80': Empty work server assignment
01:28:15:ERROR:WU01:FS01:Exception: Could not get an assignment
01:28:18:WU02:FS02:Download 6.46%
01:28:20:WU00:FS00:Download 16.06%
01:28:24:WU02:FS02:Download 11.63%
01:28:29:Saving configuration to config.xml
01:28:29:<config>
01:28:29:  <!-- Slot Control -->
01:28:29:  <power v='FULL'/>
01:28:29:
01:28:29:  <!-- Folding Slots -->
01:28:29:  <slot id='0' type='CPU'/>
01:28:29:  <slot id='1' type='GPU'/>
01:28:29:  <slot id='2' type='GPU'/>
01:28:29:</config>
01:29:12:WU01:FS01:Connecting to 171.67.108.201:80
01:29:13:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
01:29:13:WU01:FS01:Connecting to 171.67.108.204:80
01:29:14:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.204:80': Empty work server assignment
01:29:14:ERROR:WU01:FS01:Exception: Could not get an assignment
01:30:49:WU01:FS01:Connecting to 171.67.108.201:80
01:30:50:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
01:30:50:WU01:FS01:Connecting to 171.67.108.204:80
01:30:52:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.204:80': Empty work server assignment
01:30:52:ERROR:WU01:FS01:Exception: Could not get an assignment
01:33:26:WU01:FS01:Connecting to 171.67.108.201:80
01:33:27:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
01:33:27:WU01:FS01:Connecting to 171.67.108.204:80
01:33:29:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.204:80': Empty work server assignment
01:33:29:ERROR:WU01:FS01:Exception: Could not get an assignment
01:37:41:WU01:FS01:Connecting to 171.67.108.201:80
01:37:41:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
01:37:41:WU01:FS01:Connecting to 171.67.108.204:80
01:37:42:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.204:80': Empty work server assignment
01:37:42:ERROR:WU01:FS01:Exception: Could not get an assignment
01:44:32:WU01:FS01:Connecting to 171.67.108.201:80
01:44:33:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
01:44:33:WU01:FS01:Connecting to 171.67.108.204:80
01:44:34:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.204:80': Empty work server assignment
01:44:34:ERROR:WU01:FS01:Exception: Could not get an assignment
01:49:50:Saving configuration to config.xml
01:49:50:<config>
01:49:50:  <!-- Slot Control -->
01:49:50:  <power v='FULL'/>
01:49:50:
01:49:50:  <!-- User Information -->
01:49:50:  <passkey v='********************************'/>
01:49:50:  <team v='227202'/>
01:49:50:  <user v='nathan'/>
01:49:50:
01:49:50:  <!-- Folding Slots -->
01:49:50:  <slot id='0' type='CPU'/>
01:49:50:  <slot id='1' type='GPU'/>
01:49:50:  <slot id='2' type='GPU'/>
01:49:50:</config>
01:53:10:44:127.0.0.1:New Web connection
01:55:38:WU01:FS01:Connecting to 171.67.108.201:80
01:55:39:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
01:55:39:WU01:FS01:Connecting to 171.67.108.204:80
01:55:40:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.204:80': Empty work server assignment
01:55:40:ERROR:WU01:FS01:Exception: Could not get an assignment
01:59:12:Saving configuration to config.xml
01:59:12:<config>
01:59:12:  <!-- Network -->
01:59:12:  <proxy v=':8080'/>
01:59:12:
01:59:12:  <!-- Slot Control -->
01:59:12:  <power v='FULL'/>
01:59:12:
01:59:12:  <!-- User Information -->
01:59:12:  <passkey v='********************************'/>
01:59:12:  <team v='227202'/>
01:59:12:  <user v='nathan'/>
01:59:12:
01:59:12:  <!-- Folding Slots -->
01:59:12:  <slot id='0' type='CPU'/>
01:59:12:  <slot id='1' type='GPU'/>
01:59:12:  <slot id='2' type='GPU'/>
01:59:12:</config>
02:00:00:Saving configuration to config.xml
02:00:00:<config>
02:00:00:  <!-- Network -->
02:00:00:  <proxy v=':8080'/>
02:00:00:
02:00:00:  <!-- Slot Control -->
02:00:00:  <power v='FULL'/>
02:00:00:
02:00:00:  <!-- User Information -->
02:00:00:  <passkey v='********************************'/>
02:00:00:  <team v='227202'/>
02:00:00:  <user v='nathan'/>
02:00:00:
02:00:00:  <!-- Folding Slots -->
02:00:00:  <slot id='0' type='CPU'/>
02:00:00:  <slot id='1' type='GPU'/>
02:00:00:  <slot id='2' type='GPU'/>
02:00:00:</config>
02:13:42:WU01:FS01:Connecting to 171.67.108.201:80
02:13:43:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
02:13:43:WU01:FS01:Connecting to 171.67.108.204:80
02:13:44:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.204:80': Empty work server assignment
02:13:44:ERROR:WU01:FS01:Exception: Could not get an assignment
02:42:36:WU01:FS01:Connecting to 171.67.108.201:80
02:42:37:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
02:42:37:WU01:FS01:Connecting to 171.67.108.204:80
02:42:37:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.204:80': Empty work server assignment
02:42:37:ERROR:WU01:FS01:Exception: Could not get an assignment

(Not sure how to put my logs in that little text box thing, so sorry about the pages of log if it doesn't shrink it automatically)
Any help would be appreciated!
~Nate

[Joe_H]

From the section of log, no you are not yet doing any folding. It looks like the download of a WU for your CPU slot was interrupted, and the client has not detected the interruption and retried the download. This is a known bug for the client. It will probably take restarting the FAHClient process to restart that download.

Next, the two GPU slots are getting "empty work server" messages. For the slot associated with the GeForce 8800 card that is to be expected. The server for projects that used to be assigned to that generation of GPU went down back in early August, and was not able to be repaired. Those projects had also been announced to be end of life last year. At this time it does not appear that any new assignments will be coming for that generation GPU, you may as well pause that folding slot.

For the slot with the GT630 there may be a temporary shortage of WU's for the settings you are using. I don't recall if it is capable of meeting deadlines for work available, other owners of that GPU may comment on that.

[SJA]

I 'm attempting to fold on an ATI 7570 GPU that is left on 24x7. I'm consistently downloading WUs that fold for 10+ days and then time out. Calculated completion time for a typical unit is about 280 hours, timeout is about 245 hrs, expiration is about 275. Base credit per unit is 17k. These units get 90+% complete, often 97+% and then expire. This is a frustrating waste of resources for all concerned. This has been going on for about a year, prior to that units all seemed to complete before time out, but were smaller units. Any thoughts?

[jrweiss]

What is the GPU load (per GPUz or Catalyst) when Folding? Are you Folding a CPU slot as well? If so, do you reserve 1 core for the GPU slot?

[bruce]

GPUs come in an extremely wide range of capabilities. The Pande Group has discussed the need for a method of differentiating between a low-throughput iBPU from a more powerful dGPU and for a method to provide different projects based on the capabilities of your GPU. That problem has not been solved and I have no idea if/when it might be.

At the present time, you have to choose between folding on your GPU or not folding on your GPU. On a similar system, I decided I could upgrade my GFU and ignore the iGPU (or use it just for video if your BIOS provides concurrent support for both GPUs).

[MileN]

Today I installed the FAH client. My GPU folding slot is stuck at 0.0%. I have a NVIDIA GeForce GT 555M GPU. The driver is GeForce 344.11.

Here is the log when I add the GPU and start to fold it:

Code: Select all

19:27:40:Adding folding slot 01: READY gpu:0:GF108 [GeForce GT 555M]
19:27:40:Removing old file 'configs/config-20141014-181623.xml'
19:27:40:Saving configuration to config.xml
19:27:40:<config>
19:27:40:  <!-- Network -->
19:27:40:  <proxy v=':8080'/>
19:27:40:
19:27:40:  <!-- Slot Control -->
19:27:40:  <power v='FULL'/>
19:27:40:
19:27:40:  <!-- User Information -->
19:27:40:  <user v='Mile'/>
19:27:40:
19:27:40:  <!-- Folding Slots -->
19:27:40:  <slot id='0' type='CPU'/>
19:27:40:  <slot id='1' type='GPU'>
19:27:40:    <gpu-index v='0'/>
19:27:40:  </slot>
19:27:40:</config>
19:27:40:FS02:Shutting core down
19:27:40:WU02:FS02:0x17:WARNING:Console control signal 1 on PID 7412
19:27:40:WU02:FS02:0x17:Exiting, please wait. . .
19:27:40:WU01:FS01:Connecting to 171.67.108.201:80
19:27:42:WU01:FS01:Assigned to work server 140.163.4.231
19:27:42:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GF108 [GeForce GT 555M] from 140.163.4.231
19:27:42:WU01:FS01:Connecting to 140.163.4.231:8080
19:27:43:WU01:FS01:Downloading 4.83MiB
19:27:49:WU01:FS01:Download 18.10%
19:27:55:WU01:FS01:Download 32.32%
19:28:02:WU01:FS01:Download 40.08%
19:28:08:WU01:FS01:Download 47.83%
19:28:14:WU01:FS01:Download 56.88%
19:28:20:WU01:FS01:Download 71.10%
19:28:26:WU01:FS01:Download 91.79%
19:28:27:Removing old file 'configs/config-20141014-181715.xml'
19:28:27:Saving configuration to config.xml
19:28:27:<config>
19:28:27:  <!-- Network -->
19:28:27:  <proxy v=':8080'/>
19:28:27:
19:28:27:  <!-- Slot Control -->
19:28:27:  <power v='FULL'/>
19:28:27:
19:28:27:  <!-- User Information -->
19:28:27:  <user v='Mile'/>
19:28:27:
19:28:27:  <!-- Folding Slots -->
19:28:27:  <slot id='0' type='CPU'/>
19:28:27:  <slot id='1' type='GPU'>
19:28:27:    <gpu-index v='0'/>
19:28:27:  </slot>
19:28:27:</config>
19:28:30:WU01:FS01:Download complete
19:28:30:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13001 run:365 clone:9 gen:27 core:0x17 unit:0x00000032538b3db75328b760869cb6c2
19:28:30:WU01:FS01:Starting
19:28:30:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Mile/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe -dir 01 -suffix 01 -version 704 -lifeline 5272 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
19:28:30:WU01:FS01:Started FahCore on PID 6876
19:28:30:WU01:FS01:Core PID:1008
19:28:30:WU01:FS01:FahCore 0x17 started
19:28:31:WU01:FS01:0x17:*********************** Log Started 2014-10-14T19:28:31Z ***********************
19:28:31:WU01:FS01:0x17:Project: 13001 (Run 365, Clone 9, Gen 27)
19:28:31:WU01:FS01:0x17:Unit: 0x00000032538b3db75328b760869cb6c2
19:28:31:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
19:28:31:WU01:FS01:0x17:Machine: 1
19:28:31:WU01:FS01:0x17:Reading tar file state.xml
19:28:37:WU01:FS01:0x17:Reading tar file system.xml
19:28:40:WU01:FS01:0x17:Reading tar file integrator.xml
19:28:40:WU01:FS01:0x17:Reading tar file core.xml
19:28:40:WU01:FS01:0x17:Digital signatures verified
19:28:40:WU01:FS01:0x17:Folding@home GPU core17
19:28:40:WU01:FS01:0x17:Version 0.0.52
19:28:41:WARNING:FS02:Killing WU02
19:28:41:WU02:FS02:FahCore returned: INTERRUPTED (102 = 0x66)
19:28:41:WARNING:WU02:FS02:Slot ID 2 no longer exists, migrating to FS01

Any suggestions on how to fix it?

[bruce]

I've merged three topics. Although you may have an ATI GPU or a NVidia GPU, the problem is the same -- no work suitable for low-throughput hardware.

[SJA]

What is the GPU load (per GPUz or Catalyst) when Folding? Are you Folding a CPU slot as well? If so, do you reserve 1 core for the GPU slot?

Catalyst reports mostly 99% GPU utilization. This GPU can be overclocked but isn't. There is also a SMP client running on this machine, and GPU performance is unaffected by whether the SMP client is running or paused. This is a discrete graphics card and an upgrade from the discrete card that was in this machine when it was new. It far exceeds the needs for this machine's intended purpose. I'm using an older and supported version of Catalyst because of FAH.

Is a GPU reporting 1800 pts a day a low throughput piece of hardware? There are many uniprocessor clients out there still chugging along at 300 pts a day.

The GPU WU running when I wrote the original note terminated at about 92% complete, after running for about 10 days and 5 hours, and a new unit downloaded. SMP WUs vary widely in size but average about 1 day each running time and rarely run more than 3 days. SMP run times vary greatly with other system usage. That new GPU WU, listed as 17k base credit, has been running for many hours at 99% GPU utilization and still shows 0% complete. The log reports a complete download but 0 steps complete. Meanwhile, SMP units of a wide variety of sizes, from 300 pts to 5k+ pts seem to cycle through uneventfully. There is no apparent change in SMP performance with GPU running except for a huge difference in fan noise and small rise in system temp.

There seems to be multiple issues.

One is that the available GPU work units are too large to complete either within the time window provided, or within a reasonable time period.

A second issue is the No Progress issue, which is an intermittent issue encountered with some WUs and not others. The only way to recover from the No Progress issue is to uninstall the GPU client, clean up the system, and then re-install the GPU client to get another work unit. That is a poor solution as has been said previously.

Reading other postings, it appears to be a conscious decision by the project to utilize only very high performance bleeding edge gaming type GPUs. However, the No Progress issue seems to be across the board regardless of GPU performance.

[bruce]

I can't address all of your issues right now, but I'll start with a couple.

This is a discrete graphics card and an upgrade from the discrete card that was in this machine when it was new. It far exceeds the needs for this machine's intended purpose. I'm using an older and supported version of Catalyst because of FAH.

Is a GPU reporting 1800 pts a day a low throughput piece of hardware? There are many uniprocessor clients out there still chugging along at 300 pts a day.

Well, 1800 is a lot more than 300 so it's all in your perspective.

Moreover, the original SSE benchmark was originally set at 110 PPD for a Pentium 4 uniprocessor and it has never been officially changed. I've been told that people earned 1 point for a WU in the early days of FAH. Points-inflation is not encouraged, but it happens anyway in spite of steps to control it.

First of all, please be sure you understand both Deadlines and the QRB (Quick Return Bonus). If a WU is returned after the Preferred Deadline, QRB is zero and you get only baseline points.

It's possible that a GPU can be a low throughput piece of hardware. I don't see a report of which GPU you have, but the only GPU mentioned by any of those who posted in this topic is a GT 630 which has a throughput only slightly better than many modern CPUs, especially if the slot is configured to pause when active.

Reading other postings, it appears to be a conscious decision by the project to utilize only very high performance bleeding edge gaming type GPUs. However, the No Progress issue seems to be across the board regardless of GPU performance.

This forum contains posts from a variety of donors expressing their own opinions. The Pande Group is not responsible for those opinions, the donors, themselves, are. Please point me to statements made by the Pande Group that indicate that they've made that conscious decision.

[bollix47]

viewtopic.php?p=269984#p269984

... ATI 7570 GPU ...

[bruce]

See the announcement: viewtopic.php?f=24&t=14714&p=273584#p273584

I don't have any way to test this new feature, so reports are encouraged.

[debs3759]

It's a shame that they favour NVidia. My HD 7770 keeps getting wu that it can't complete in time, I would love to be able to only download shorter work for it.

[bruce]

I understand your concern, but the non-bonus projects have all been announced to be end-of-life and the projects for ATI have been finished. Since both Core_15 and Core_16 are EOLed, there will be no new projects for either one.