Fermi Cards were assigned non-Fermi WUs

ahpla · Post by **ahpla** » Mon Nov 19, 2012 8:53 am

Long time cruncher here. Woke up this morning to find a core 11 unit had been picked up but my GPU usage was at 0% and work unit progress was 0% and had been like this for around an hour and a half with no progress.

I tried deleting the work folder so a new unit would be picked up; this was core 11 again and it exhibited the exact same behaviour. I tried deleting the core executable; upon resuming the work unit and redownloading the core, it was still the same. When I try to pause computation, the FahCore_11.exe process does not end unless I choose to do so forcibly from the task manager.

Windows 7 Home Premium 64-bit
f@h 7.2.9
MSI GTX 460 Cyclone (768MB)
nVidia driver version 306.97
Only crunching GPU, no SMP.

Code: Select all

08:39:40:Saving configuration to config.xml
08:39:40:<config>
08:39:40:  <!-- Folding Slot Configuration -->
08:39:40:  <cause-pref v='CANCER'/>
08:39:40:  <gpu v='true'/>
08:39:40:  <smp v='false'/>
08:39:40:
08:39:40:  <!-- Logging -->
08:39:40:  <verbosity v='4'/>
08:39:40:
08:39:40:  <!-- Network -->
08:39:40:  <proxy v=':8080'/>
08:39:40:
08:39:40:  <!-- User Information -->
08:39:40:  <passkey v='********************************'/>
08:39:40:  <team v='758'/>
08:39:40:  <user v='alpha'/>
08:39:40:
08:39:40:  <!-- Folding Slots -->
08:39:40:  <slot id='0' type='GPU'/>
08:39:40:</config>
08:40:02:FS00:Paused
08:40:03:FS00:Shutting core down
08:40:11:WU00:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
08:40:30:FS00:Unpaused
08:40:30:WU00:FS00:Starting
08:40:30:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_11.fah/FahCore_11.exe -dir 00 -suffix 01 -version 702 -lifeline 3876 -checkpoint 15 -gpu 0
08:40:30:WU00:FS00:Started FahCore on PID 3288
08:40:30:WU00:FS00:Core PID:3884
08:40:30:WU00:FS00:FahCore 0x11 started
08:40:31:WARNING:WU00:FS00:FahCore returned: MISSING_WORK_FILES (116 = 0x74)
08:40:31:WARNING:WU00:FS00:Fatal error, dumping
08:40:31:WU00:FS00:Sending unit results: id:00 state:SEND error:DUMPED project:5765 run:10 clone:353 gen:13 core:0x11 unit:0x7c5f434b50a9ea5b000d0161000a1685
08:40:31:WARNING:WU00:FS00:Work server too old for dump report
08:40:31:WU00:FS00:Cleaning up
08:40:31:WU00:FS00:Connecting to assign-GPU.stanford.edu:80
08:40:32:WU00:FS00:News: Welcome to Folding@Home
08:40:32:WU00:FS00:Assigned to work server 171.67.108.11
08:40:32:WU00:FS00:Requesting new work unit for slot 00: READY gpu:0:"GF104 [GeForce GTX 460]" from 171.67.108.11
08:40:32:WU00:FS00:Connecting to 171.67.108.11:8080
08:40:33:WU00:FS00:Downloading 46.11KiB
08:40:34:WU00:FS00:Download complete
08:40:34:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:5768 run:4 clone:142 gen:4 core:0x11 unit:0x46284c5450a9f0810004008e00041688
08:40:34:WU00:FS00:Starting
08:40:34:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_11.fah/FahCore_11.exe -dir 00 -suffix 01 -version 702 -lifeline 3876 -checkpoint 15 -gpu 0
08:40:34:WU00:FS00:Started FahCore on PID 1880
08:40:34:WU00:FS00:Core PID:4852
08:40:34:WU00:FS00:FahCore 0x11 started
08:40:34:WU00:FS00:Downloading project 5768 description
08:40:34:WU00:FS00:Connecting to fah-web.stanford.edu:80
08:40:34:WU00:FS00:0x11:
08:40:34:WU00:FS00:0x11:*------------------------------*
08:40:34:WU00:FS00:0x11:Folding@Home GPU Core
08:40:34:WU00:FS00:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
08:40:34:WU00:FS00:0x11:
08:40:34:WU00:FS00:0x11:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
08:40:34:WU00:FS00:0x11:Build host: amoeba
08:40:34:WU00:FS00:0x11:Board Type: Nvidia
08:40:34:WU00:FS00:0x11:Core      : 
08:40:34:WU00:FS00:0x11:Preparing to commence simulation
08:40:34:WU00:FS00:0x11:- Looking at optimizations...
08:40:34:WU00:FS00:0x11:DeleteFrameFiles: successfully deleted file=00/wudata_01.ckp
08:40:34:WU00:FS00:0x11:- Created dyn
08:40:34:WU00:FS00:0x11:- Files status OK
08:40:34:WU00:FS00:0x11:- Expanded 46707 -> 252912 (decompressed 541.4 percent)
08:40:34:WU00:FS00:0x11:Called DecompressByteArray: compressed_data_size=46707 data_size=252912, decompressed_data_size=252912 diff=0
08:40:34:WU00:FS00:0x11:- Digital signature verified
08:40:34:WU00:FS00:0x11:
08:40:34:WU00:FS00:0x11:Project: 5768 (Run 4, Clone 142, Gen 4)
08:40:34:WU00:FS00:0x11:
08:40:34:WU00:FS00:0x11:Assembly optimizations on if available.
08:40:34:WU00:FS00:0x11:Entering M.D.
08:40:35:WU00:FS00:Project 5768 description downloaded successfully
08:40:40:WU00:FS00:0x11:Tpr hash 00/wudata_01.tpr:  2689796529 1594937108 597917264 1492161446 2845505097
08:40:40:WU00:FS00:0x11:
08:40:40:WU00:FS00:0x11:Calling fah_main args: 14 usage=100
08:40:40:WU00:FS00:0x11:
08:41:24:FS00:Paused
08:41:24:FS00:Shutting core down
08:42:10:WU00:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
08:42:20:FS00:Unpaused
08:42:21:WU00:FS00:Downloading core from http://www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_11.fah
08:42:21:WU00:FS00:Connecting to www.stanford.edu:80
08:42:21:WU00:FS00:FahCore 11: Downloading 648.82KiB
08:42:25:WU00:FS00:FahCore 11: Download complete
08:42:25:WU00:FS00:Valid core signature
08:42:25:WU00:FS00:Unpacked 1.82MiB to cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_11.fah/FahCore_11.exe
08:42:25:WU00:FS00:Starting
08:42:25:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_11.fah/FahCore_11.exe -dir 00 -suffix 01 -version 702 -lifeline 3876 -checkpoint 15 -gpu 0
08:42:25:WU00:FS00:Started FahCore on PID 936
08:42:25:WU00:FS00:Core PID:4908
08:42:25:WU00:FS00:FahCore 0x11 started
08:42:26:WU00:FS00:0x11:
08:42:26:WU00:FS00:0x11:*------------------------------*
08:42:26:WU00:FS00:0x11:Folding@Home GPU Core
08:42:26:WU00:FS00:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
08:42:26:WU00:FS00:0x11:
08:42:26:WU00:FS00:0x11:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
08:42:26:WU00:FS00:0x11:Build host: amoeba
08:42:26:WU00:FS00:0x11:Board Type: Nvidia
08:42:26:WU00:FS00:0x11:Core      : 
08:42:26:WU00:FS00:0x11:Preparing to commence simulation
08:42:26:WU00:FS00:0x11:- Ensuring status. Please wait.
08:42:35:WU00:FS00:0x11:- Looking at optimizations...
08:42:35:WU00:FS00:0x11:- Working with standard loops on this execution.
08:42:35:WU00:FS00:0x11:- Previous termination of core was improper.
08:42:35:WU00:FS00:0x11:- Files status OK
08:42:35:WU00:FS00:0x11:- Expanded 46707 -> 252912 (decompressed 541.4 percent)
08:42:35:WU00:FS00:0x11:Called DecompressByteArray: compressed_data_size=46707 data_size=252912, decompressed_data_size=252912 diff=0
08:42:35:WU00:FS00:0x11:- Digital signature verified
08:42:35:WU00:FS00:0x11:
08:42:35:WU00:FS00:0x11:Project: 5768 (Run 4, Clone 142, Gen 4)
08:42:35:WU00:FS00:0x11:
08:42:35:WU00:FS00:0x11:Entering M.D.
08:42:41:WU00:FS00:0x11:Tpr hash 00/wudata_01.tpr:  2689796529 1594937108 597917264 1492161446 2845505097
08:42:41:WU00:FS00:0x11:
08:42:41:WU00:FS00:0x11:Calling fah_main args: 14 usage=100
08:42:41:WU00:FS00:0x11:

I usually pick up core 15 work units and it works flawlessly.

bollix47 · Post by **bollix47** » Mon Nov 19, 2012 9:41 am

Because some servers were down I suspect Fermi cards were being assigned non-Fermi work units. This should never happen!

The servers are back up so deleting the work folder should work now.

rpmouton · Post by **rpmouton** » Mon Nov 19, 2012 10:40 am

Thanks for the heads up guys, I was stuck as well but hadn't realized it yet..

Ripper36 · Post by **Ripper36** » Mon Nov 19, 2012 11:44 am

I have just had problems with these units completely stalling in the same way on 3 different GPUs on 2 different PCs. The logs aren't very informative - just nothing for two hours. I've had to dump them.

They were units running core 11, so thanks for the information - very reassuring!

thebluebumblebee · Post by **thebluebumblebee** » Mon Nov 19, 2012 4:50 pm

How can a Fermi card get assigned a GPU2 WU? And how do I get rid of it?

Here's the log file:

Code: Select all

07:26:56:WU00:FS00:Connecting to assign-GPU.stanford.edu:80
07:26:56:WU00:FS00:News: Welcome to Folding@Home
07:26:56:WU00:FS00:Assigned to work server 171.67.108.11
07:26:56:WU00:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:"GF114 [GeForce GTX 560 Ti]" from 171.67.108.11
07:26:56:WU00:FS00:Connecting to 171.67.108.11:8080
07:26:56:WU00:FS00:Downloading 44.83KiB
07:26:57:WU00:FS00:Download complete
07:26:57:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:OK project:5771 run:3 clone:104 gen:2844 core:0x11 unit:0x586bb32f50a9df440b1c00680003168b
07:26:57:WU00:FS00:Downloading core from http://www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_11.fah
07:26:57:WU00:FS00:Connecting to www.stanford.edu:80
07:26:57:WU00:FS00:FahCore 11: Downloading 648.82KiB
07:27:01:WU00:FS00:FahCore 11: Download complete
07:27:01:WU00:FS00:Valid core signature
07:27:01:WU00:FS00:Unpacked 1.82MiB to cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_11.fah/FahCore_11.exe
07:27:01:WU00:FS00:Downloading project 5771 description
07:27:01:WU00:FS00:Connecting to fah-web.stanford.edu:80
07:27:01:WU00:FS00:Project 5771 description downloaded successfully
07:28:53:WU00:FS00:Starting
07:28:53:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Christopher/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_11.fah/FahCore_11.exe -dir 00 -suffix 01 -version 701 -lifeline 4760 -checkpoint 15 -gpu 0
07:28:53:WU00:FS00:Started FahCore on PID 1848
07:28:53:WU00:FS00:Core PID:900
07:28:53:WU00:FS00:FahCore 0x11 started
07:28:54:WU00:FS00:0x11:
07:28:54:WU00:FS00:0x11:*------------------------------*
07:28:54:WU00:FS00:0x11:Folding@Home GPU Core
07:28:54:WU00:FS00:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
07:28:54:WU00:FS00:0x11:
07:28:54:WU00:FS00:0x11:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
07:28:54:WU00:FS00:0x11:Build host: amoeba
07:28:54:WU00:FS00:0x11:Board Type: Nvidia
07:28:54:WU00:FS00:0x11:Core      : 
07:28:54:WU00:FS00:0x11:Preparing to commence simulation
07:28:54:WU00:FS00:0x11:- Looking at optimizations...
07:28:54:WU00:FS00:0x11:DeleteFrameFiles: successfully deleted file=00/wudata_01.ckp
07:28:54:WU00:FS00:0x11:- Created dyn
07:28:54:WU00:FS00:0x11:- Files status OK
07:28:54:WU00:FS00:0x11:- Expanded 45395 -> 251112 (decompressed 553.1 percent)
07:28:54:WU00:FS00:0x11:Called DecompressByteArray: compressed_data_size=45395 data_size=251112, decompressed_data_size=251112 diff=0
07:28:54:WU00:FS00:0x11:- Digital signature verified
07:28:54:WU00:FS00:0x11:
07:28:54:WU00:FS00:0x11:Project: 5771 (Run 3, Clone 104, Gen 2844)
07:28:54:WU00:FS00:0x11:
07:28:54:WU00:FS00:0x11:Assembly optimizations on if available.
07:28:54:WU00:FS00:0x11:Entering M.D.
07:29:00:WU00:FS00:0x11:Tpr hash 00/wudata_01.tpr:  2905459331 508545711 315661601 1414638683 1287387629
07:29:00:WU00:FS00:0x11:
07:29:00:WU00:FS00:0x11:Calling fah_main args: 14 usage=100
07:29:00:WU00:FS00:0x11:

Post by **bruce** » Mon Nov 19, 2012 5:04 pm

At the time that Core15 and core16 were being developed, the Fermi platform was able to process WUs from Core11, though there was an assignment preference for core 15/16 so not many folks got WUs for 11. I'm not sure what changed except that there's a lot going on with servers right now and if there's nothing in the GPU3 category, the AS either has to assign you something from the GPU2 category or tell you it can't assign you any work.

It's Monday morning and there are likely several folks at Stanford that are trying to fix downed servers and check on WU availability. Give it a few more hours and things will probably be back to normal availability. I notice that VSP12 was down again last night but some things were corrected at about 1:00 AM Stanford time. Other issues still remain, though.

thebluebumblebee · Post by **thebluebumblebee** » Mon Nov 19, 2012 5:18 pm

Bruce-Thanks!

(too much partying over Stanford beating Oregon?

)

Added the log file. Hope it helps.

thebluebumblebee · Post by **thebluebumblebee** » Mon Nov 19, 2012 5:50 pm

Deleted WU and then got:

"Work server too old for dump report"

ahpla · Post by **ahpla** » Mon Nov 19, 2012 5:55 pm

bollix47 wrote:Because some servers were down I suspect Fermi cards were being assigned non-Fermi work units. This should never happen!

The servers are back up so deleting the work folder should work now.

Yeah, just tried to pick up a new work unit and it was for core 15 this time.

Hopefully someone will get around to looking at the non-Fermi work units being assigned to Fermi cards so that this kind of downtime can be avoided in future

Thanks.

klasseng · Post by **klasseng** » Mon Nov 19, 2012 8:53 pm

I've got a Windows 7 PC with 4 GTS 450 that's been folding 24/7 for the past 7 weeks without a hiccup.

All of a sudden I get two WU's
5768 (2, 234, 1056)
5772 (9, 121, 8)
that are not being processed:
Progress 0.00%
ETA: Unkown
Base Credit Unkown
Esitmated Credit Unkown
Estimated PPD Unkown
Estimated TPF Unknown

GPU-Z says the cards have 0 GPU load.

How do I purge these WU's and let it get some fresh ones?

Sailer · Post by **Sailer** » Mon Nov 19, 2012 10:23 pm

I've been getting the same thing on numerous of my computers. 5765 (8, 109, 1901) is not running on this particular computer. On three other computers I resorted to uninstalling the whole folding program, including DATA files, and reinstalling the program. Unfortunately, that's only a temporary fix because as soon as problem WU comes up again, they will die. I'm wondering if there is a problem with the entire 57xx series of WUs.

Post by **Joe_H** » Mon Nov 19, 2012 11:13 pm

There are a couple other topics covering this problem, see viewtopic.php?f=18&t=23033. At one time core 11 WU's would process on Fermi cards, now many don't. To remove these from your work queue, delete the work folder corresponding to the WU after pausing F@H. When you restart your client should pick up a new WU now that the servers are back up.

klasseng · Post by **klasseng** » Tue Nov 20, 2012 12:59 am

@ Joe_H:

So I guess my question should have been:

In a stock F@H home installation on Windows 7, where is the work folder so I can delete it?

bollix47 · Post by **bollix47** » Tue Nov 20, 2012 1:07 am

See this post: viewtopic.php?p=229420#p229420

klasseng · Post by **klasseng** » Tue Nov 20, 2012 1:16 am

@ bollix47

Thanks!

peace,
Grant

Folding Forum

Fermi Cards were assigned non-Fermi WUs

Fermi Cards were assigned non-Fermi WUs

Re: Core 11 at 0% usage, 0% progress

Re: Core 11 at 0% usage, 0% progress

Similar problems WUs 5765, 5768, 5771

GTX 560 Ti assigned a Project: 5771 (Run 3, Clone 104, Gen 2

Re: GTX 560 Ti assigned a Project: 5771 (Run 3, Clone 104, G

Re: GTX 560 Ti assigned a Project: 5771 (Run 3, Clone 104, G

Re: GTX 560 Ti assigned a Project: 5771 (Run 3, Clone 104, G

Re: Core 11 at 0% usage, 0% progress

Unkown Unknown Unknown

Re: Unkown Unknown Unknown

Re: Unkown Unknown Unknown

Re: Unkown Unknown Unknown

Re: Unkown Unknown Unknown

Re: Unkown Unknown Unknown