Failed WU 5766 5768 5770 5771

nickjd · Post by **nickjd** » Sat Dec 17, 2011 8:54 am

I'm getting a number of failed work units. Ok, so yes I am running this on a laptop with both SMP and GPU with the new v7 console and I understand heat is an issue but this is a thinkpad with quadro graphics so I expect it to be as stable as a decent desktop. Also, I never once had problems with heat or a failed WU with my old video drivers (I monitor everything closely). I recently upgraded to 285.62 and I found that to cause problems with failed WU so it was recommended to me to use the Lenovo supplied drivers (seems to be version 268.24 which is probably fine considering the hardware is a couple years old now).

I find it hard to believe that my hardware has suddenly failed right after updating my video drivers. These are all I could find in the log file:

FAILED project:5766 run:5 clone:252 gen:1343 core:0x11 unit:0x00969fad4eebffae053f00fc00051686
FAILED project:5768 run:5 clone:44 gen:951 core:0x11 unit:0x17becd2f4eebffa303b7002c00051688
FAILED project:5770 run:2 clone:352 gen:1862 core:0x11 unit:0x4cd8eb834eebff97074601600002168a
FAILED project:5770 run:4 clone:260 gen:1905 core:0x11 unit:0x7e03c2524eebff8b077101040004168a
FAILED project:5771 run:10 clone:158 gen:296 core:0x11 unit:0x626fd2874eebfe0e0128009e000a168b

I realize there must be a way of checking myself if others have been able to complete these work units. All I know of for looking at work unit information is through here:
http://fah-web.stanford.edu/psummary.html
but that of course doesn't tell me much information but I DID notice that they are all the same projects and the same group. So how can I look to see if someone else has finished any in these projects?

And I would just delete the work folder so it will just download another WU, but v7 uses the same folder for both SMP and GPU and I don't want to delete the wrong file (Since SMP is working fine and I don't want to lose any points

)
I would appreciate any recommendations. I've been folding on my computers since late 2005 nearly 24/7 and haven't had much trouble until now

nickjd · Post by **nickjd** » Sat Dec 17, 2011 4:15 pm

I'm also getting no credit for the units that don't fail. In FAHControl, none of them show up as having any credit value.

Post by **PantherX** » Sat Dec 17, 2011 5:34 pm

Could you please post the log between the Code Tags (in Full Editor)?

One of the GPU WUs shows this in my log:
11:06:20:Unit 01: 100.00%
11:06:20:Unit 01: Upload complete
11:06:20:Server responded WORK_ACK (400)
11:06:20:Cleaning up Unit 01

The SMP usually shows this:
15:15:15:Unit 00: Upload complete
15:15:15:Server responded WORK_ACK (400)
15:15:15:Final credit estimate, 2246.00 points
15:15:15:Cleaning up Unit 00

Regarding the video drivers, it could be that the new drivers are more efficient thus it has pushed your GPU from the edge of stability to instability region.

project:5766 run:5 clone:252 gen:1343 and project:5770 run:2 clone:352 gen:1862 have been completed successfully by other donors but the others don't have any data in the WU Database so I have marked it for a follow-up.

nickjd · Post by **nickjd** » Sat Dec 17, 2011 5:55 pm

Here's some of the log:

Code: Select all

02:34:15:Unit 01:
02:34:15:Unit 01:*------------------------------*
02:34:15:Unit 01:Folding@Home GPU Core
02:34:15:Unit 01:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
02:34:15:Unit 01:
02:34:15:Unit 01:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
02:34:15:Unit 01:Build host: amoeba
02:34:15:Unit 01:Board Type: Nvidia
02:34:15:Unit 01:Core      : 
02:34:15:Unit 01:Preparing to commence simulation
02:34:15:Unit 01:- Looking at optimizations...
02:34:15:Unit 01:DeleteFrameFiles: successfully deleted file=01/wudata_01.ckp
02:34:15:Unit 01:- Created dyn
02:34:15:Unit 01:- Files status OK
02:34:15:Unit 01:- Expanded 46681 -> 252912 (decompressed 541.7 percent)
02:34:15:Unit 01:Called DecompressByteArray: compressed_data_size=46681 data_size=252912, decompressed_data_size=252912 diff=0
02:34:15:Unit 01:- Digital signature verified
02:34:15:Unit 01:
02:34:15:Unit 01:Project: 5768 (Run 5, Clone 44, Gen 951)
02:34:15:Unit 01:
02:34:15:Unit 01:Assembly optimizations on if available.
02:34:15:Unit 01:Entering M.D.
02:34:21:Unit 01:Tpr hash 01/wudata_01.tpr:  865149534 2799151471 3023772244 21019454 1964823866
02:34:21:Unit 01:
02:34:21:Unit 01:Calling fah_main args: 14 usage=80
02:34:21:Unit 01:
02:34:21:Unit 01:Working on Protein
02:34:22:Unit 01:mdrun_gpu returned 
02:34:22:Unit 01:Self-test failure
02:34:22:Unit 01:
02:34:22:Unit 01:Folding@home Core Shutdown: UNSTABLE_MACHINE
02:34:22:FahCore, running Unit 01, returned: UNSTABLE_MACHINE (122 = 0x7a)
02:34:22:Sending unit results: id:01 state:SEND error:FAILED project:5768 run:5 clone:44 gen:951 core:0x11 unit:0x17becd2f4eebffa303b7002c00051688
02:34:22:WARNING: Work server too old for fail report, dumping
02:34:22:Cleaning up Unit 01
02:34:22:Connecting to assign-GPU.stanford.edu:80
02:34:24:News: Welcome to Folding@Home
02:34:24:Assigned to work server 171.67.108.11
02:34:24:Requesting new work unit for slot 00: READY gpu:0:"GT216 [Quadro FX 880M]" from 171.67.108.11
02:34:24:Connecting to 171.67.108.11:8080
02:34:25:Slot 00: Downloading 46.10KiB
02:34:26:Slot 00: Download complete
02:34:26:Received Unit: id:01 state:DOWNLOAD error:OK project:5766 run:5 clone:252 gen:1343 core:0x11 unit:0x00969fad4eebffae053f00fc00051686
02:34:26:Starting Unit 01
02:34:26:Running core: "C:/Users/Nick/FAH/Foldinghome-gpu NEW/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe" -dir 01 -suffix 01 -lifeline 4476 -version 701 -checkpoint 15 -gpu 0
02:34:26:Started core on PID 9780
02:34:26:FahCore 0x11 started
02:34:26:Unit 01:
02:34:26:Unit 01:*------------------------------*
02:34:26:Unit 01:Folding@Home GPU Core
02:34:26:Unit 01:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
02:34:26:Unit 01:
02:34:26:Unit 01:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
02:34:26:Unit 01:Build host: amoeba
02:34:26:Unit 01:Board Type: Nvidia
02:34:26:Unit 01:Core      : 
02:34:26:Unit 01:Preparing to commence simulation
02:34:26:Unit 01:- Looking at optimizations...
02:34:26:Unit 01:DeleteFrameFiles: successfully deleted file=01/wudata_01.ckp
02:34:26:Unit 01:- Created dyn
02:34:26:Unit 01:- Files status OK
02:34:27:Unit 01:- Expanded 46696 -> 252912 (decompressed 541.6 percent)
02:34:27:Unit 01:Called DecompressByteArray: compressed_data_size=46696 data_size=252912, decompressed_data_size=252912 diff=0
02:34:27:Unit 01:- Digital signature verified
02:34:27:Unit 01:
02:34:27:Unit 01:Project: 5766 (Run 5, Clone 252, Gen 1343)
02:34:27:Unit 01:
02:34:27:Unit 01:Assembly optimizations on if available.
02:34:27:Unit 01:Entering M.D.
02:34:32:Unit 01:Tpr hash 01/wudata_01.tpr:  852260094 2587545001 2837822654 1821463590 4253230139
02:34:32:Unit 01:
02:34:32:Unit 01:Calling fah_main args: 14 usage=80
02:34:32:Unit 01:
02:34:33:Unit 01:Working on Protein
02:34:34:Unit 01:Client config unavailable.
02:34:34:Unit 01:Starting GUI Server
02:34:34:Unit 01:mdrun_gpu returned 
02:34:34:Unit 01:NANs detected on GPU
02:34:34:Unit 01:
02:34:34:Unit 01:Folding@home Core Shutdown: UNSTABLE_MACHINE
02:34:34:FahCore, running Unit 01, returned: UNSTABLE_MACHINE (122 = 0x7a)
02:34:34:Sending unit results: id:01 state:SEND error:FAILED project:5766 run:5 clone:252 gen:1343 core:0x11 unit:0x00969fad4eebffae053f00fc00051686
02:34:34:WARNING: Work server too old for fail report, dumping
02:34:34:Cleaning up Unit 01
02:50:02:Downloading project 5770 description
02:50:02:Connecting to fah-web.stanford.edu:80
02:50:40:Unit 00:Completed 600000 out of 1500000 steps  (40%)
03:15:47:Unit 00:Completed 615000 out of 1500000 steps  (41%)
03:40:48:Unit 00:Completed 630000 out of 1500000 steps  (42%)

I keep getting these self test failures. I'm not sure what "NANs detected on GPU" means. Is this the information that you require? I think that the failures are all like this. I have a GPU monitor and it never tries to load anything to video RAM or use the GPU at all. As a side note, I noticed that the GPU % use limitation in the FAHControl settings does not work. It recommends 80% so I set it there and it always still uses 99% of my GPU.

Do you know why these WUs have no credit applied to them?

codysluder · Post by **codysluder** » Sat Dec 17, 2011 9:12 pm

nickjd wrote:Do you know why these WUs have no credit applied to them?

You only get credit if you complete the calculations correctly.

Self test failures means you have a hardware problem. NANs detected on GPU also indicates your GPU is failing to compute results correctly.

UNSTABLE_MACHINE also indicates a GPU hardware failure.

Get your GPU repaired/replaced or get drivers that work correctly. (The OS considers a driver failure as a hardware failure.)

nickjd · Post by **nickjd** » Sat Dec 17, 2011 9:26 pm

yeah I've had the mainboard replaced once already and it was identical before with the non Lenovo drivers. I know that the points are only given if they are completed but in the console it doesn't display the points. It says unknown, unlike for SMP. I never had a problem with the drivers from about 6 months ago but I can't find the old drivers online anymore. I think I had a special cuda developer's driver package. sorry for any typos as I'm typing on my phone.

codysluder · Post by **codysluder** » Sat Dec 17, 2011 9:35 pm

nickjd wrote: in the console it doesn't display the points. It says unknown, unlike for SMP.

The V7 client can only display points when the server provides that information to it. This is a function of the code that is running on the server. Some servers still need to have their code upgraded to a newer version. This issue will disappear when the server code is upgraded. Notice the other message indicating the server code is "old"

Code: Select all

WARNING: Work server too old for fail report, dumping

The server still awards the expected credits for completed WUs. (When we ran V6, we always had to look it up on the Project Summary or install a 3rd party monitoring program that did that for us.)

WiSK · Post by **WiSK** » Sun Dec 18, 2011 2:49 am

I also has this error message UNSTABLE_MACHINE with projects 5766 and 5768. See discussion viewtopic.php?f=50&t=20174. Solution for me was using older console client.

nickjd · Post by **nickjd** » Sun Dec 18, 2011 9:37 am

Interesting, but I thought that they still use the same core so it should be functioning the same.

nickjd · Post by **nickjd** » Wed Dec 28, 2011 8:49 pm

Well, the solution for me seems to be using an older client as well. v7 seems to be giving me only WUs in the 5700's (such as the ones in the title of this thread). None of those seem to work on my computer as they fail immediately. idk if it's my computer, the v7 installation, driver or what, but the GPU3 6.41 client works absolutely fine.

I did notice that all of the 5700's projects listed at the following page are from the same contact, vvoelz: http://fah-web.stanford.edu/psummary.html
I don't know if that means anything, but it seems like for whatever reason his work units don't agree with my hardware. 6.41 is giving me other work units that seem to be OK at the moment at least (meaning all I can say is that it hasn't failed yet lol).

-Nick

Post by **bruce** » Wed Dec 28, 2011 11:58 pm

nickjd wrote:Well, the solution for me seems to be using an older client as well. v7 seems to be giving me only WUs in the 5700's (such as the ones in the title of this thread). None of those seem to work on my computer as they fail immediately. idk if it's my computer, the v7 installation, driver or what, but the GPU3 6.41 client works absolutely fine.

I did notice that all of the 5700's projects listed at the following page are from the same contact, vvoelz: http://fah-web.stanford.edu/psummary.html
I don't know if that means anything, but it seems like for whatever reason his work units don't agree with my hardware. 6.41 is giving me other work units that seem to be OK at the moment at least (meaning all I can say is that it hasn't failed yet lol).

-Nick

Did I miss it, or did you forget to tell us which GPU you have? A V7 installation does have access to some projects that V6.41 can't access, and if those projects don't run on your hardware or your drivers, we do need to identify the problem and either correct something in the WUs or somehow exclude them from being assigned to hardware/drivers like yours. How about posting the first section of the V7 log so we can see what you have and describe anything that might not be identified there.

This is something that should have been picked up when the projects were beta tested but maybe nobody had hardware/drivers like yours.

nickjd · Post by **nickjd** » Thu Dec 29, 2011 10:20 am

I have an quadro fx 880m on a thinkpad w510. Sorry, I don't think I stated that. I have the lenovo supplied drivers (268.24) and they have the same problems as the direct from nvidia drivers (whatever the newest were, I forget... 284 or something).

I don't think that my laptop is very common. I don't think I even know anyone who bought one of these workstations lol.

I hope this is the right information that you requested. It has a few failed work units and for whatever reason, some download errors.

Code: Select all

*********************** Log Started 2011-12-28T19:53:00 ************************
19:53:00:************************* Folding@home Client *************************
19:53:00:      Website: http://folding.stanford.edu/
19:53:00:    Copyright: (c) 2009-2011 Stanford University
19:53:00:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:53:00:         Args: --lifeline 5384 --command-port=36330
19:53:00:       Config: C:/Users/Nick/FAH/Foldinghome-gpu NEW/config.xml
19:53:00:******************************** Build ********************************
19:53:00:      Version: 7.1.38
19:53:00:         Date: Oct 6 2011
19:53:00:         Time: 19:57:04
19:53:00:      SVN Rev: 3080
19:53:00:       Branch: fah/trunk/client
19:53:00:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
19:53:00:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
19:53:00:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT
19:53:00:     Platform: win32 XP
19:53:00:         Bits: 32
19:53:00:         Mode: Release
19:53:00:******************************* System ********************************
19:53:00:          CPU: Intel(R) Core(TM) i7 CPU Q 720 @ 1.60GHz
19:53:00:       CPU ID: GenuineIntel Family 6 Model 30 Stepping 5
19:53:00:         CPUs: 8
19:53:00:       Memory: 11.93GiB
19:53:00:  Free Memory: 7.93GiB
19:53:00:      Threads: WINDOWS_THREADS
19:53:00:   On Battery: false
19:53:00:   UTC offset: 1
19:53:00:          PID: 1548
19:53:00:          CWD: C:/Users/Nick/FAH/Foldinghome-gpu NEW
19:53:00:           OS: Windows 7 Professional
19:53:00:      OS Arch: AMD64
19:53:00:         GPUs: 1
19:53:00:        GPU 0: NVIDIA:1 GT216 [Quadro FX 880M]
19:53:00:         CUDA: 1.2
19:53:00:  CUDA Driver: 3020
19:53:00:Win32 Service: false
19:53:00:***********************************************************************
19:53:00:<config>
19:53:00:  <!-- Folding Slot Configuration -->
19:53:00:  <gpu v='true'/>
19:53:00:  <gpu-usage v='80'/>
19:53:00:
19:53:00:  <!-- Network -->
19:53:00:  <proxy v=':8080'/>
19:53:00:
19:53:00:  <!-- User Information -->
19:53:00:  <passkey v='********************************'/>
19:53:00:  <team v='50511'/>
19:53:00:  <user v='nick_dittes'/>
19:53:00:
19:53:00:  <!-- Folding Slots -->
19:53:00:  <slot id='0' type='GPU'/>
19:53:00:  <slot id='1' type='SMP'/>
19:53:00:</config>
19:53:00:Trying to access database...
19:53:00:Successfully acquired database lock
19:53:00:Enabled folding slot 00: READY gpu:0:"GT216 [Quadro FX 880M]"
19:53:00:Enabled folding slot 01: READY smp:8
19:53:00:Starting Unit 00
19:53:00:Running core: "C:/Users/Nick/FAH/Foldinghome-gpu NEW/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe" -dir 00 -suffix 01 -lifeline 1548 -version 701 -checkpoint 15 -np 8
19:53:00:Started core on PID 8768
19:53:00:FahCore 0xa4 started
19:53:00:Connecting to assign-GPU.stanford.edu:80
19:53:01:Unit 00:
19:53:01:Unit 00:*------------------------------*
19:53:01:Unit 00:Folding@Home Gromacs GB Core
19:53:01:Unit 00:Version 2.27 (Dec. 15, 2010)
19:53:01:Unit 00:
19:53:01:Unit 00:Preparing to commence simulation
19:53:01:Unit 00:- Ensuring status. Please wait.
19:53:03:Server connection id=1 on 0.0.0.0:36330 from 127.0.0.1
19:53:05:News: Welcome to Folding@Home
19:53:05:Assigned to work server 171.67.108.11
19:53:05:Requesting new work unit for slot 00: READY gpu:0:"GT216 [Quadro FX 880M]" from 171.67.108.11
19:53:05:Connecting to 171.67.108.11:8080
19:53:09:Slot 00: Downloading 46.00KiB
19:53:10:Unit 00:- Looking at optimizations...
19:53:10:Unit 00:- Working with standard loops on this execution.
19:53:10:Unit 00:- Previous termination of core was improper.
19:53:10:Unit 00:- Going to use standard loops.
19:53:10:Unit 00:- Files status OK
19:53:11:Unit 00:- Expanded 2053848 -> 5365960 (decompressed 261.2 percent)
19:53:11:Unit 00:Called DecompressByteArray: compressed_data_size=2053848 data_size=5365960, decompressed_data_size=5365960 diff=0
19:53:11:Unit 00:- Digital signature verified
19:53:11:Unit 00:
19:53:11:Unit 00:Project: 7808 (Run 3, Clone 23, Gen 30)
19:53:11:Unit 00:
19:53:11:Unit 00:Entering M.D.
19:53:15:Slot 00: 86.96%
19:53:15:Slot 00: Download complete
19:53:15:Received Unit: id:01 state:DOWNLOAD error:OK project:5765 run:3 clone:36 gen:522 core:0x11 unit:0x4752f8744efb73a2020a002400031685
19:53:15:Starting Unit 01
19:53:15:Running core: "C:/Users/Nick/FAH/Foldinghome-gpu NEW/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe" -dir 01 -suffix 01 -lifeline 1548 -version 701 -checkpoint 15 -gpu 0
19:53:15:Started core on PID 8920
19:53:15:FahCore 0x11 started
19:53:15:Unit 01:
19:53:15:Unit 01:*------------------------------*
19:53:15:Unit 01:Folding@Home GPU Core
19:53:15:Unit 01:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
19:53:15:Unit 01:
19:53:15:Unit 01:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
19:53:15:Unit 01:Build host: amoeba
19:53:15:Unit 01:Board Type: Nvidia
19:53:15:Unit 01:Core      : 
19:53:15:Unit 01:Preparing to commence simulation
19:53:15:Unit 01:- Looking at optimizations...
19:53:15:Unit 01:DeleteFrameFiles: successfully deleted file=01/wudata_01.ckp
19:53:15:Unit 01:- Created dyn
19:53:15:Unit 01:- Files status OK
19:53:15:Unit 01:- Expanded 46588 -> 252912 (decompressed 542.8 percent)
19:53:15:Unit 01:Called DecompressByteArray: compressed_data_size=46588 data_size=252912, decompressed_data_size=252912 diff=0
19:53:15:Unit 01:- Digital signature verified
19:53:16:Unit 01:
19:53:16:Unit 01:Project: 5765 (Run 3, Clone 36, Gen 522)
19:53:16:Unit 01:
19:53:16:Unit 01:Assembly optimizations on if available.
19:53:16:Unit 01:Entering M.D.
19:53:17:Unit 00:Using Gromacs checkpoints
19:53:17:Unit 00:Mapping NT from 8 to 8 
19:53:19:Unit 00:Resuming from checkpoint
19:53:19:Unit 00:Verified 00/wudata_01.log
19:53:19:Unit 00:Verified 00/wudata_01.trr
19:53:19:Unit 00:Verified 00/wudata_01.xtc
19:53:19:Unit 00:Verified 00/wudata_01.edr
19:53:20:Unit 00:Completed 245440 out of 1500000 steps  (16%)
19:53:21:Unit 01:Tpr hash 01/wudata_01.tpr:  2028863636 140372942 3747285140 2699888196 95518832
19:53:21:Unit 01:
19:53:21:Unit 01:Calling fah_main args: 14 usage=80
19:53:21:Unit 01:
19:53:22:Unit 01:Working on Protein
19:53:23:Unit 01:mdrun_gpu returned 
19:53:23:Unit 01:Self-test failure
19:53:23:Unit 01:
19:53:23:Unit 01:Folding@home Core Shutdown: UNSTABLE_MACHINE
19:53:24:FahCore, running Unit 01, returned: UNSTABLE_MACHINE (122 = 0x7a)
19:53:24:Sending unit results: id:01 state:SEND error:FAILED project:5765 run:3 clone:36 gen:522 core:0x11 unit:0x4752f8744efb73a2020a002400031685
19:53:24:WARNING: Work server too old for fail report, dumping
19:53:24:Cleaning up Unit 01
19:53:24:Connecting to assign-GPU.stanford.edu:80
19:53:25:News: Welcome to Folding@Home
19:53:25:Assigned to work server 171.67.108.11
19:53:25:Requesting new work unit for slot 00: READY gpu:0:"GT216 [Quadro FX 880M]" from 171.67.108.11
19:53:25:Connecting to 171.67.108.11:8080
19:53:26:Slot 00: Downloading 44.87KiB
19:53:29:Slot 00: Download complete
19:53:29:Received Unit: id:01 state:DOWNLOAD error:OK project:5769 run:7 clone:44 gen:370 core:0x11 unit:0x2a60a2834efb73b50172002c00071689
19:53:29:Starting Unit 01
19:53:29:Running core: "C:/Users/Nick/FAH/Foldinghome-gpu NEW/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe" -dir 01 -suffix 01 -lifeline 1548 -version 701 -checkpoint 15 -gpu 0
19:53:29:Started core on PID 7000
19:53:29:FahCore 0x11 started
19:53:29:Unit 01:
19:53:29:Unit 01:*------------------------------*
19:53:29:Unit 01:Folding@Home GPU Core
19:53:29:Unit 01:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
19:53:29:Unit 01:
19:53:29:Unit 01:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
19:53:29:Unit 01:Build host: amoeba
19:53:29:Unit 01:Board Type: Nvidia
19:53:29:Unit 01:Core      : 
19:53:29:Unit 01:Preparing to commence simulation
19:53:29:Unit 01:- Looking at optimizations...
19:53:29:Unit 01:DeleteFrameFiles: successfully deleted file=01/wudata_01.ckp
19:53:29:Unit 01:- Created dyn
19:53:29:Unit 01:- Files status OK
19:53:29:Unit 01:- Expanded 45430 -> 251112 (decompressed 552.7 percent)
19:53:29:Unit 01:Called DecompressByteArray: compressed_data_size=45430 data_size=251112, decompressed_data_size=251112 diff=0
19:53:29:Unit 01:- Digital signature verified
19:53:30:Unit 01:
19:53:30:Unit 01:Project: 5769 (Run 7, Clone 44, Gen 370)
19:53:30:Unit 01:
19:53:30:Unit 01:Assembly optimizations on if available.
19:53:30:Unit 01:Entering M.D.
19:53:35:Unit 01:Tpr hash 01/wudata_01.tpr:  729641215 3050449559 3616641009 804156768 2046378886
19:53:35:Unit 01:
19:53:35:Unit 01:Calling fah_main args: 14 usage=80
19:53:35:Unit 01:
19:53:36:Unit 01:Working on Protein
19:53:38:Unit 01:Client config unavailable.
19:53:38:Unit 01:Starting GUI Server
19:53:38:Unit 01:mdrun_gpu returned 
19:53:38:Unit 01:NANs detected on GPU
19:53:38:Unit 01:
19:53:38:Unit 01:Folding@home Core Shutdown: UNSTABLE_MACHINE
19:53:39:FahCore, running Unit 01, returned: UNSTABLE_MACHINE (122 = 0x7a)
19:53:39:Sending unit results: id:01 state:SEND error:FAILED project:5769 run:7 clone:44 gen:370 core:0x11 unit:0x2a60a2834efb73b50172002c00071689
19:53:39:WARNING: Work server too old for fail report, dumping
19:53:39:Cleaning up Unit 01
19:53:39:Connecting to assign-GPU.stanford.edu:80
19:53:41:News: Welcome to Folding@Home
19:53:41:Assigned to work server 171.67.108.11
19:53:41:Requesting new work unit for slot 00: READY gpu:0:"GT216 [Quadro FX 880M]" from 171.67.108.11
19:53:41:Connecting to 171.67.108.11:8080
19:53:43:Slot 00: Downloading 46.10KiB
19:53:47:Slot 00: Download complete
19:53:47:Received Unit: id:01 state:DOWNLOAD error:OK project:5767 run:8 clone:31 gen:885 core:0x11 unit:0x5f1b70344efb73c60375001f00081687
19:53:47:Starting Unit 01
19:53:47:Running core: "C:/Users/Nick/FAH/Foldinghome-gpu NEW/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe" -dir 01 -suffix 01 -lifeline 1548 -version 701 -checkpoint 15 -gpu 0
19:53:47:Started core on PID 9064
19:53:47:FahCore 0x11 started
19:53:47:Unit 01:
19:53:47:Unit 01:*------------------------------*
19:53:47:Unit 01:Folding@Home GPU Core
19:53:47:Unit 01:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
19:53:47:Unit 01:
19:53:47:Unit 01:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
19:53:47:Unit 01:Build host: amoeba
19:53:47:Unit 01:Board Type: Nvidia
19:53:47:Unit 01:Core      : 
19:53:47:Unit 01:Preparing to commence simulation
19:53:47:Unit 01:- Looking at optimizations...
19:53:47:Unit 01:DeleteFrameFiles: successfully deleted file=01/wudata_01.ckp
19:53:47:Unit 01:- Created dyn
19:53:47:Unit 01:- Files status OK
19:53:47:Unit 01:- Expanded 46696 -> 252912 (decompressed 541.6 percent)
19:53:47:Unit 01:Called DecompressByteArray: compressed_data_size=46696 data_size=252912, decompressed_data_size=252912 diff=0
19:53:47:Unit 01:- Digital signature verified
19:53:47:Unit 01:
19:53:47:Unit 01:Project: 5767 (Run 8, Clone 31, Gen 885)
19:53:47:Unit 01:
19:53:47:Unit 01:Assembly optimizations on if available.
19:53:47:Unit 01:Entering M.D.
19:53:53:Unit 01:Tpr hash 01/wudata_01.tpr:  4068530083 1591654400 1417444788 164357565 3507681498
19:53:53:Unit 01:
19:53:53:Unit 01:Calling fah_main args: 14 usage=80
19:53:53:Unit 01:
19:53:54:Unit 01:Working on Protein
19:53:56:Unit 01:Client config unavailable.
19:53:56:Unit 01:Starting GUI Server
19:53:56:Unit 01:mdrun_gpu returned 
19:53:56:Unit 01:NANs detected on GPU
19:53:56:Unit 01:
19:53:56:Unit 01:Folding@home Core Shutdown: UNSTABLE_MACHINE
19:53:56:FahCore, running Unit 01, returned: UNSTABLE_MACHINE (122 = 0x7a)
19:53:56:Sending unit results: id:01 state:SEND error:FAILED project:5767 run:8 clone:31 gen:885 core:0x11 unit:0x5f1b70344efb73c60375001f00081687
19:53:56:WARNING: Work server too old for fail report, dumping
19:53:56:Cleaning up Unit 01
19:53:56:Connecting to assign-GPU.stanford.edu:80
19:54:00:News: Welcome to Folding@Home
19:54:00:Assigned to work server 171.67.108.11
19:54:00:Requesting new work unit for slot 00: READY gpu:0:"GT216 [Quadro FX 880M]" from 171.67.108.11
19:54:00:Connecting to 171.67.108.11:8080
19:54:02:Slot 00: Downloading 44.89KiB
19:54:05:Slot 00: Download complete
19:54:05:Received Unit: id:01 state:DOWNLOAD error:OK project:5769 run:0 clone:111 gen:128 core:0x11 unit:0x5e997de54efb73d90080006f00001689
19:54:05:Starting Unit 01
19:54:05:Running core: "C:/Users/Nick/FAH/Foldinghome-gpu NEW/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe" -dir 01 -suffix 01 -lifeline 1548 -version 701 -checkpoint 15 -gpu 0
19:54:05:Started core on PID 8440
19:54:05:FahCore 0x11 started
19:54:05:Unit 01:
19:54:05:Unit 01:*------------------------------*
19:54:05:Unit 01:Folding@Home GPU Core
19:54:05:Unit 01:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
19:54:05:Unit 01:
19:54:05:Unit 01:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
19:54:05:Unit 01:Build host: amoeba
19:54:05:Unit 01:Board Type: Nvidia
19:54:05:Unit 01:Core      : 
19:54:05:Unit 01:Preparing to commence simulation
19:54:05:Unit 01:- Looking at optimizations...
19:54:05:Unit 01:DeleteFrameFiles: successfully deleted file=01/wudata_01.ckp
19:54:05:Unit 01:- Created dyn
19:54:05:Unit 01:- Files status OK
19:54:05:Unit 01:- Expanded 45459 -> 251112 (decompressed 552.3 percent)
19:54:05:Unit 01:Called DecompressByteArray: compressed_data_size=45459 data_size=251112, decompressed_data_size=251112 diff=0
19:54:05:Unit 01:- Digital signature verified
19:54:05:Unit 01:
19:54:05:Unit 01:Project: 5769 (Run 0, Clone 111, Gen 128)
19:54:05:Unit 01:
19:54:05:Unit 01:Assembly optimizations on if available.
19:54:05:Unit 01:Entering M.D.
19:54:11:Unit 01:Tpr hash 01/wudata_01.tpr:  2635616140 1162523147 2590863232 4137979304 2063232660
19:54:11:Unit 01:
19:54:11:Unit 01:Calling fah_main args: 14 usage=80
19:54:11:Unit 01:
19:54:12:Unit 01:Working on Protein
19:54:14:Unit 01:Client config unavailable.
19:54:14:Unit 01:Starting GUI Server
19:54:14:Unit 01:mdrun_gpu returned 
19:54:14:Unit 01:NANs detected on GPU
19:54:14:Unit 01:
19:54:14:Unit 01:Folding@home Core Shutdown: UNSTABLE_MACHINE
19:54:14:FahCore, running Unit 01, returned: UNSTABLE_MACHINE (122 = 0x7a)
19:54:14:Sending unit results: id:01 state:SEND error:FAILED project:5769 run:0 clone:111 gen:128 core:0x11 unit:0x5e997de54efb73d90080006f00001689
19:54:14:WARNING: Work server too old for fail report, dumping
19:54:14:Cleaning up Unit 01
19:54:14:Connecting to assign-GPU.stanford.edu:80
19:54:16:News: Welcome to Folding@Home
19:54:16:Assigned to work server 171.67.108.11
19:54:16:Requesting new work unit for slot 00: READY gpu:0:"GT216 [Quadro FX 880M]" from 171.67.108.11
19:54:16:Connecting to 171.67.108.11:8080
19:54:17:Slot 00: Downloading 46.12KiB
19:54:19:Slot 00: Download complete
19:54:19:Received Unit: id:01 state:DOWNLOAD error:OK project:5765 run:4 clone:138 gen:291 core:0x11 unit:0x028ac99d4efb73e80123008a00041685
19:54:19:Starting Unit 01
19:54:19:Running core: "C:/Users/Nick/FAH/Foldinghome-gpu NEW/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe" -dir 01 -suffix 01 -lifeline 1548 -version 701 -checkpoint 15 -gpu 0
19:54:19:Started core on PID 1728
19:54:19:FahCore 0x11 started
19:54:19:Unit 01:
19:54:19:Unit 01:*------------------------------*
19:54:19:Unit 01:Folding@Home GPU Core
19:54:19:Unit 01:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
19:54:19:Unit 01:
19:54:19:Unit 01:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
19:54:20:Unit 01:Build host: amoeba
19:54:20:Unit 01:Board Type: Nvidia
19:54:20:Unit 01:Core      : 
19:54:20:Unit 01:Preparing to commence simulation
19:54:20:Unit 01:- Looking at optimizations...
19:54:20:Unit 01:DeleteFrameFiles: successfully deleted file=01/wudata_01.ckp
19:54:20:Unit 01:- Created dyn
19:54:20:Unit 01:- Files status OK
19:54:20:Unit 01:- Expanded 46714 -> 252912 (decompressed 541.4 percent)
19:54:20:Unit 01:Called DecompressByteArray: compressed_data_size=46714 data_size=252912, decompressed_data_size=252912 diff=0
19:54:20:Unit 01:- Digital signature verified
19:54:20:Unit 01:
19:54:20:Unit 01:Project: 5765 (Run 4, Clone 138, Gen 291)
19:54:20:Unit 01:
19:54:20:Unit 01:Assembly optimizations on if available.
19:54:20:Unit 01:Entering M.D.
19:54:25:Unit 01:Tpr hash 01/wudata_01.tpr:  4208680099 2113533140 4050919395 1494874042 150073715
19:54:25:Unit 01:
19:54:25:Unit 01:Calling fah_main args: 14 usage=80
19:54:25:Unit 01:
19:54:26:Unit 01:Working on Protein
19:54:28:Unit 01:Client config unavailable.
19:54:28:Unit 01:Starting GUI Server
19:54:28:FahCore, running Unit 01, returned: UNSTABLE_MACHINE (122 = 0x7a)
19:54:28:Sending unit results: id:01 state:SEND error:FAILED project:5765 run:4 clone:138 gen:291 core:0x11 unit:0x028ac99d4efb73e80123008a00041685
19:54:29:WARNING: Work server too old for fail report, dumping
19:54:29:Cleaning up Unit 01
19:56:50:Slot 00 paused
19:56:52:Slot 00 unpaused
19:56:53:Connecting to assign-GPU.stanford.edu:80
19:56:55:News: Welcome to Folding@Home
19:56:55:Assigned to work server 171.67.108.11
19:56:55:Requesting new work unit for slot 00: READY gpu:0:"GT216 [Quadro FX 880M]" from 171.67.108.11
19:56:55:Connecting to 171.67.108.11:8080
19:56:58:Slot 00: Downloading 44.84KiB
19:57:01:Slot 00: Download complete
19:57:01:Received Unit: id:01 state:DOWNLOAD error:OK project:5771 run:7 clone:121 gen:817 core:0x11 unit:0x4f9e9e524efb7488033100790007168b
19:57:02:Starting Unit 01
19:57:02:Running core: "C:/Users/Nick/FAH/Foldinghome-gpu NEW/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe" -dir 01 -suffix 01 -lifeline 1548 -version 701 -checkpoint 15 -gpu 0
19:57:02:Started core on PID 1096
19:57:02:FahCore 0x11 started
19:57:02:Unit 01:
19:57:02:Unit 01:*------------------------------*
19:57:02:Unit 01:Folding@Home GPU Core
19:57:02:Unit 01:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
19:57:02:Unit 01:
19:57:02:Unit 01:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
19:57:02:Unit 01:Build host: amoeba
19:57:02:Unit 01:Board Type: Nvidia
19:57:02:Unit 01:Core      : 
19:57:02:Unit 01:Preparing to commence simulation
19:57:02:Unit 01:- Looking at optimizations...
19:57:02:Unit 01:DeleteFrameFiles: successfully deleted file=01/wudata_01.ckp
19:57:02:Unit 01:- Created dyn
19:57:02:Unit 01:- Files status OK
19:57:02:Unit 01:- Expanded 45409 -> 251112 (decompressed 553.0 percent)
19:57:02:Unit 01:Called DecompressByteArray: compressed_data_size=45409 data_size=251112, decompressed_data_size=251112 diff=0
19:57:02:Unit 01:- Digital signature verified
19:57:02:Unit 01:
19:57:02:Unit 01:Project: 5771 (Run 7, Clone 121, Gen 817)
19:57:02:Unit 01:
19:57:02:Unit 01:Assembly optimizations on if available.
19:57:02:Unit 01:Entering M.D.
19:57:08:Unit 01:Tpr hash 01/wudata_01.tpr:  2836170464 2721175299 3574220446 3581411764 979836516
19:57:08:Unit 01:
19:57:08:Unit 01:Calling fah_main args: 14 usage=80
19:57:08:Unit 01:
19:57:09:Unit 01:Working on Protein
19:57:11:Unit 01:Client config unavailable.
19:57:11:Unit 01:Starting GUI Server
19:57:11:Unit 01:mdrun_gpu returned 
19:57:11:Unit 01:NANs detected on GPU
19:57:11:Unit 01:
19:57:11:Unit 01:Folding@home Core Shutdown: UNSTABLE_MACHINE
19:57:11:FahCore, running Unit 01, returned: UNSTABLE_MACHINE (122 = 0x7a)
19:57:11:Sending unit results: id:01 state:SEND error:FAILED project:5771 run:7 clone:121 gen:817 core:0x11 unit:0x4f9e9e524efb7488033100790007168b
19:57:11:WARNING: Work server too old for fail report, dumping
19:57:11:Cleaning up Unit 01
19:57:11:Connecting to assign-GPU.stanford.edu:80
19:57:12:News: Welcome to Folding@Home
19:57:12:Assigned to work server 171.67.108.11
19:57:12:Requesting new work unit for slot 00: READY gpu:0:"GT216 [Quadro FX 880M]" from 171.67.108.11
19:57:12:Connecting to 171.67.108.11:8080
19:57:14:Slot 00: Downloading 44.84KiB
19:57:17:Slot 00: Download complete
19:57:17:Received Unit: id:01 state:DOWNLOAD error:OK project:5770 run:3 clone:18 gen:332 core:0x11 unit:0x1cccea184efb7499014c00120003168a
19:57:17:Starting Unit 01
19:57:17:Running core: "C:/Users/Nick/FAH/Foldinghome-gpu NEW/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe" -dir 01 -suffix 01 -lifeline 1548 -version 701 -checkpoint 15 -gpu 0
19:57:17:Started core on PID 8912
19:57:17:FahCore 0x11 started
19:57:17:Unit 01:
19:57:17:Unit 01:*------------------------------*
19:57:17:Unit 01:Folding@Home GPU Core
19:57:17:Unit 01:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
19:57:17:Unit 01:
19:57:17:Unit 01:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
19:57:17:Unit 01:Build host: amoeba
19:57:17:Unit 01:Board Type: Nvidia
19:57:17:Unit 01:Core      : 
19:57:17:Unit 01:Preparing to commence simulation
19:57:17:Unit 01:- Looking at optimizations...
19:57:17:Unit 01:DeleteFrameFiles: successfully deleted file=01/wudata_01.ckp
19:57:17:Unit 01:- Created dyn
19:57:17:Unit 01:- Files status OK
19:57:17:Unit 01:- Expanded 45405 -> 251112 (decompressed 553.0 percent)
19:57:17:Unit 01:Called DecompressByteArray: compressed_data_size=45405 data_size=251112, decompressed_data_size=251112 diff=0
19:57:17:Unit 01:- Digital signature verified
19:57:17:Unit 01:
19:57:17:Unit 01:Project: 5770 (Run 3, Clone 18, Gen 332)
19:57:17:Unit 01:
19:57:17:Unit 01:Assembly optimizations on if available.
19:57:17:Unit 01:Entering M.D.
19:57:23:Unit 01:Tpr hash 01/wudata_01.tpr:  2533223749 1905490652 2991865763 1121488614 4049463648
19:57:23:Unit 01:
19:57:23:Unit 01:Calling fah_main args: 14 usage=80
19:57:23:Unit 01:
19:57:24:Unit 01:Working on Protein
19:57:26:Unit 01:Client config unavailable.
19:57:26:Unit 01:Starting GUI Server
19:57:26:Unit 01:mdrun_gpu returned 
19:57:26:Unit 01:NANs detected on GPU
19:57:26:Unit 01:
19:57:26:Unit 01:Folding@home Core Shutdown: UNSTABLE_MACHINE
19:57:26:FahCore, running Unit 01, returned: UNSTABLE_MACHINE (122 = 0x7a)
19:57:26:Sending unit results: id:01 state:SEND error:FAILED project:5770 run:3 clone:18 gen:332 core:0x11 unit:0x1cccea184efb7499014c00120003168a
19:57:26:WARNING: Work server too old for fail report, dumping
19:57:26:Cleaning up Unit 01
19:57:26:Connecting to assign-GPU.stanford.edu:80
19:57:28:News: Welcome to Folding@Home
19:57:28:Assigned to work server 171.67.108.11
19:57:28:Requesting new work unit for slot 00: READY gpu:0:"GT216 [Quadro FX 880M]" from 171.67.108.11
19:57:28:Connecting to 171.67.108.11:8080
19:57:32:Slot 00: Downloading 46.10KiB
19:57:36:Slot 00: Download complete
19:57:36:Received Unit: id:01 state:DOWNLOAD error:OK project:5768 run:9 clone:178 gen:439 core:0x11 unit:0x6733cfea4efb74ab01b700b200091688
19:57:36:Starting Unit 01
19:57:36:Running core: "C:/Users/Nick/FAH/Foldinghome-gpu NEW/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe" -dir 01 -suffix 01 -lifeline 1548 -version 701 -checkpoint 15 -gpu 0
19:57:36:Started core on PID 6156
19:57:36:FahCore 0x11 started
19:57:37:Unit 01:

Post by **bruce** » Thu Dec 29, 2011 6:52 pm

Project: 5765 (Run 3, Clone 36, Gen 522) UNSTABLE_MACHINE (122 = 0x7a) no data
Project: 5769 (Run 7, Clone 44, Gen 370) UNSTABLE_MACHINE (122 = 0x7a) no data
Project: 5767 (Run 8, Clone 31, Gen 885) UNSTABLE_MACHINE (122 = 0x7a) Successfully completed by someone else.
Project: 5769 (Run 0, Clone 111, Gen 128) UNSTABLE_MACHINE (122 = 0x7a) Successfully completed by someone else.
Project: 5765 (Run 4, Clone 138, Gen 291) UNSTABLE_MACHINE (122 = 0x7a) Successfully completed by someone else.
Project: 5771 (Run 7, Clone 121, Gen 817) UNSTABLE_MACHINE (122 = 0x7a) no data
Project: 5770 (Run 3, Clone 18, Gen 332) UNSTABLE_MACHINE (122 = 0x7a) no data

The relative uniqueness of your hardware is something to consider ... but maybe you should also consider that you might have an Unstable-Machine.

I had trouble with my GPU recently and testing it with MemtestG80 / MemtestCL turned up some flaky memory.

nickjd · Post by **nickjd** » Thu Dec 29, 2011 8:05 pm

This is my second mainboard (the first was replaced by the idiots by the German Lenovo support center even though it wasn't a problem... it was a faulty wwan card...). I had similar problems with gpu fah in the past as well. I find it hard to believe I have two identically faulty mainboards on more than one installation of windows where it worked using other gpu clients. I mean it seems like it fails doing nothing before it even tries to load the WU. Plus I think my previous comment still applies here that 6.41 seems to be functional with other projects. What is your opinion now knowing this is my second mainboard? The GPU is soldered on of course so I know its all new. Once I'm home and not on my cell phone I'll try memtest like you said

nickjd · Post by **nickjd** » Thu Dec 29, 2011 10:09 pm

I tried both programs and they didn't find any errors. Any other suggestions?

Folding Forum

Failed WU 5766 5768 5770 5771

Failed WU 5766 5768 5770 5771

Re: Failed WU 5766 5768 5770 5771

Re: Failed WU 5766 5768 5770 5771

Re: Failed WU 5766 5768 5770 5771

Re: Failed WU 5766 5768 5770 5771

Re: Failed WU 5766 5768 5770 5771

Re: Failed WU 5766 5768 5770 5771

Re: Failed WU 5766 5768 5770 5771

Re: Failed WU 5766 5768 5770 5771

Re: Failed WU 5766 5768 5770 5771

Re: Failed WU 5766 5768 5770 5771

Re: Failed WU 5766 5768 5770 5771

Re: Failed WU 5766 5768 5770 5771

Re: Failed WU 5766 5768 5770 5771

Re: Failed WU 5766 5768 5770 5771