Project: 7810 (Run 0, Clone 66, Gen 402)
Posted: Mon Nov 11, 2013 4:06 am
One Faulty 7810, also after 72% as folding_hoomer before ...
Latest NV driver 331.20
Before and after that WU the GPU was working ok; bad WU ?
Latest NV driver 331.20
Code: Select all
*********************** Log Started 2013-11-08T04:30:28Z ***********************
04:30:28:************************* Folding@home Client *************************
04:30:28: Website: http://folding.stanford.edu/
04:30:28: Copyright: (c) 2009-2013 Stanford University
04:30:28: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
04:30:28: Args: --child --lifeline 1113 /etc/fahclient/config.xml --run-as
04:30:28: fahclient --pid-file=/var/run/fahclient.pid --daemon
04:30:28: Config: /etc/fahclient/config.xml
04:30:28:******************************** Build ********************************
04:30:28: Version: 7.3.6
04:30:28: Date: Feb 18 2013
04:30:28: Time: 07:24:08
04:30:28: SVN Rev: 3923
04:30:28: Branch: fah/trunk/client
04:30:28: Compiler: GNU 4.4.7
04:30:28: Options: -std=gnu++98 -O3 -funroll-loops -mfpmath=sse -ffast-math
04:30:28: -fno-unsafe-math-optimizations -msse2
04:30:28: Platform: linux2 3.2.0-1-amd64
04:30:28: Bits: 64
04:30:28: Mode: Release
04:30:28:******************************* System ********************************
04:30:28: CPU: Intel(R) Core(TM) i7-2600S CPU @ 2.80GHz
04:30:28: CPU ID: GenuineIntel Family 6 Model 42 Stepping 7
04:30:28: CPUs: 8
04:30:28: Memory: 7.74GiB
04:30:28:Free Memory: 7.38GiB
04:30:28: Threads: POSIX_THREADS
04:30:28:Has Battery: false
04:30:28: On Battery: false
04:30:28: UTC offset: 9
04:30:28: PID: 1243
04:30:28: CWD: /var/lib/fahclient
04:30:28: OS: Linux 3.8.0-32-generic x86_64
04:30:28: OS Arch: AMD64
04:30:28: GPUs: 3
04:30:28: GPU 0: NVIDIA:3 GK110 [GeForce GTX 780]
04:30:28: GPU 1: NVIDIA:3 GK110 [GeForce GTX 780]
04:30:28: GPU 2: NVIDIA:3 GK104 [GeForce GTX 660 Ti]
04:30:28: CUDA: 3.5
04:30:28:CUDA Driver: 6000
04:30:28:***********************************************************************
04:30:28:<config>
04:30:28: <!-- Folding Slot Configuration -->
04:30:28: <gpu v='true'/>
04:30:28: <power v='full'/>
04:30:28:
04:30:28: <!-- HTTP Server -->
04:30:28:
04:30:28: <!-- Logging -->
04:30:28: <log-rotate-max v='1024'/>
04:30:28:
04:30:28: <!-- Network -->
04:30:28: <proxy v=':8080'/>
04:30:28:
04:30:28: <!-- User Information -->
04:30:28: <team v='3446'/>
04:30:28: <user v='ChristianFAH'/>
04:30:28:
04:30:28: <!-- Folding Slots -->
04:30:28: <slot id='0' type='CPU'>
04:30:28: <client-type v='beta'/>
04:30:28: <cpus v='4'/>
04:30:28: <next-unit-percentage v='99'/>
04:30:28: <pause-on-start v='true'/>
04:30:28: </slot>
04:30:28: <slot id='1' type='GPU'>
04:30:28: <client-type v='beta'/>
04:30:28: <pause-on-start v='true'/>
04:30:28: </slot>
04:30:28: <slot id='2' type='GPU'>
04:30:28: <client-type v='beta'/>
04:30:28: <pause-on-start v='true'/>
04:30:28: </slot>
04:30:28: <slot id='3' type='GPU'>
04:30:28: <client-type v='beta'/>
04:30:28: <pause-on-start v='true'/>
04:30:28: </slot>
04:30:28:</config>
04:30:28:Switching to user fahclient
04:30:28:Trying to access database...
04:30:28:Successfully acquired database lock
04:30:28:Enabled folding slot 00: PAUSED cpu:4 (paused)
04:30:28:Enabled folding slot 01: PAUSED gpu:0:GK110 [GeForce GTX 780] (paused)
04:30:28:Enabled folding slot 02: PAUSED gpu:1:GK110 [GeForce GTX 780] (paused)
04:30:28:Enabled folding slot 03: PAUSED gpu:2:GK104 [GeForce GTX 660 Ti] (paused)
04:36:01:FS01:Unpaused
...
18:44:02:WU01:FS02:0x17:Saving result file log.txt
18:44:02:WU01:FS02:0x17:Saving result file positions.xtc
18:44:03:WU01:FS02:0x17:Folding@home Core Shutdown: FINISHED_UNIT
18:44:03:WU01:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
18:44:03:WU01:FS02:Sending unit results: id:01 state:SEND error:NO_ERROR project:7810 run:0 clone:694 gen:292 core:0x17 unit:0x0000013e0a3b1e8651d34d8137690c39
18:44:03:WU01:FS02:Uploading 5.74MiB to 171.64.65.98
18:44:03:WU01:FS02:Connecting to 171.64.65.98:8080
18:44:06:WU00:FS02:Download 29.89%
18:44:12:WU00:FS02:Download 56.78%
18:44:12:WU01:FS02:Upload complete
18:44:12:WU01:FS02:Server responded WORK_ACK (400)
18:44:12:WU01:FS02:Final credit estimate, 16751.00 points
18:44:12:WU01:FS02:Cleaning up
18:44:18:WU00:FS02:Download 92.65%
18:44:18:WU00:FS02:Download complete
18:44:18:WU00:FS02:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:7810 run:0 clone:66 gen:402 core:0x17 unit:0x000001b50a3b1e8651d34661a08b8f39
18:44:18:WU00:FS02:Starting
18:44:18:WU00:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/www.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/beta/Core_17.fah/FahCore_17 -dir 00 -suffix 01 -version 703 -lifeline 1243 -checkpoint 15 -gpu 1 -gpu-vendor nvidia
18:44:18:WU00:FS02:Started FahCore on PID 7316
18:44:18:WU00:FS02:Core PID:7320
18:44:18:WU00:FS02:FahCore 0x17 started
18:44:19:WU00:FS02:0x17:*********************** Log Started 2013-11-10T18:44:18Z ***********************
18:44:19:WU00:FS02:0x17:Project: 7810 (Run 0, Clone 66, Gen 402)
18:44:19:WU00:FS02:0x17:Unit: 0x000001b50a3b1e8651d34661a08b8f39
18:44:19:WU00:FS02:0x17:CPU: 0x00000000000000000000000000000000
18:44:19:WU00:FS02:0x17:Machine: 2
18:44:19:WU00:FS02:0x17:Reading tar file state.xml
18:44:19:WU00:FS02:0x17:Reading tar file system.xml
18:44:19:WU00:FS02:0x17:Reading tar file integrator.xml
18:44:19:WU00:FS02:0x17:Reading tar file core.xml
18:44:19:WU00:FS02:0x17:Digital signatures verified
18:44:38:WU00:FS02:0x17:Completed 0 out of 2000000 steps (0%)
18:46:12:WU00:FS02:0x17:Completed 20000 out of 2000000 steps (1%)
18:47:43:WU00:FS02:0x17:Completed 40000 out of 2000000 steps (2%)
18:48:22:WU02:FS01:0x17:Completed 880000 out of 2000000 steps (44%)
18:49:17:WU00:FS02:0x17:Completed 60000 out of 2000000 steps (3%)
18:49:56:WU02:FS01:0x17:Completed 900000 out of 2000000 steps (45%)
18:50:48:WU00:FS02:0x17:Completed 80000 out of 2000000 steps (4%)
18:51:32:WU02:FS01:0x17:Completed 920000 out of 2000000 steps (46%)
18:52:20:WU00:FS02:0x17:Completed 100000 out of 2000000 steps (5%)
18:53:06:WU02:FS01:0x17:Completed 940000 out of 2000000 steps (47%)
18:53:54:WU00:FS02:0x17:Completed 120000 out of 2000000 steps (6%)
18:54:43:WU02:FS01:0x17:Completed 960000 out of 2000000 steps (48%)
18:55:25:WU00:FS02:0x17:Completed 140000 out of 2000000 steps (7%)
18:56:17:WU02:FS01:0x17:Completed 980000 out of 2000000 steps (49%)
18:56:59:WU00:FS02:0x17:Completed 160000 out of 2000000 steps (8%)
...
19:32:24:WU00:FS02:0x17:Completed 620000 out of 2000000 steps (31%)
19:32:38:WU02:FS01:0x17:Completed 1440000 out of 2000000 steps (72%)
19:33:56:WU00:FS02:0x17:Completed 640000 out of 2000000 steps (32%)
19:34:15:WU02:FS01:0x17:Completed 1460000 out of 2000000 steps (73%)
19:34:49:WU00:FS02:0x17:Bad State detected... attempting to resume from last good checkpoint
19:35:48:WU02:FS01:0x17:Completed 1480000 out of 2000000 steps (74%)
19:36:20:WU00:FS02:0x17:Completed 620000 out of 2000000 steps (31%)
19:37:22:WU02:FS01:0x17:Completed 1500000 out of 2000000 steps (75%)
19:37:52:WU00:FS02:0x17:Completed 640000 out of 2000000 steps (32%)
19:38:45:WU00:FS02:0x17:Bad State detected... attempting to resume from last good checkpoint
19:38:59:WU02:FS01:0x17:Completed 1520000 out of 2000000 steps (76%)
19:40:17:WU00:FS02:0x17:Completed 620000 out of 2000000 steps (31%)
19:40:32:WU02:FS01:0x17:Completed 1540000 out of 2000000 steps (77%)
19:41:48:WU00:FS02:0x17:Completed 640000 out of 2000000 steps (32%)
19:42:09:WU02:FS01:0x17:Completed 1560000 out of 2000000 steps (78%)
19:42:42:WU00:FS02:0x17:Bad State detected... attempting to resume from last good checkpoint
19:42:42:WU00:FS02:0x17:Max number of retries reached. Aborting.
19:42:42:WU00:FS02:0x17:ERROR:exception: Max Retries Reached
19:42:42:WU00:FS02:0x17:Saving result file logfile_01.txt
19:42:42:WU00:FS02:0x17:Saving result file badStateCheckpoint_416496834
19:42:42:WU00:FS02:0x17:Saving result file badStateCheckpoint_798946464
19:42:43:WU00:FS02:0x17:Saving result file badStateCheckpoint_8351929
19:42:43:WU00:FS02:0x17:Saving result file badStateForceGroup0_416496834Core.xml
19:42:44:WU00:FS02:0x17:Saving result file badStateForceGroup0_416496834Ref.xml
19:42:45:WU00:FS02:0x17:Saving result file badStateForceGroup0_798946464Core.xml
19:42:46:WU00:FS02:0x17:Saving result file badStateForceGroup0_798946464Ref.xml
19:42:47:WU00:FS02:0x17:Saving result file badStateForceGroup0_8351929Core.xml
19:42:48:WU00:FS02:0x17:Saving result file badStateForceGroup0_8351929Ref.xml
19:42:48:WU00:FS02:0x17:Saving result file badStateForceGroup1_416496834Core.xml
19:42:49:WU00:FS02:0x17:Saving result file badStateForceGroup1_416496834Ref.xml
19:42:50:WU00:FS02:0x17:Saving result file badStateForceGroup1_798946464Core.xml
19:42:51:WU00:FS02:0x17:Saving result file badStateForceGroup1_798946464Ref.xml
19:42:52:WU00:FS02:0x17:Saving result file badStateForceGroup1_8351929Core.xml
19:42:52:WU00:FS02:0x17:Saving result file badStateForceGroup1_8351929Ref.xml
19:42:53:WU00:FS02:0x17:Saving result file badStateForceGroup2_416496834Core.xml
19:42:54:WU00:FS02:0x17:Saving result file badStateForceGroup2_416496834Ref.xml
19:42:55:WU00:FS02:0x17:Saving result file badStateForceGroup2_798946464Core.xml
19:42:55:WU00:FS02:0x17:Saving result file badStateForceGroup2_798946464Ref.xml
19:42:56:WU00:FS02:0x17:Saving result file badStateForceGroup2_8351929Core.xml
19:42:57:WU00:FS02:0x17:Saving result file badStateForceGroup2_8351929Ref.xml
19:42:58:WU00:FS02:0x17:Saving result file log.txt
19:42:58:WU00:FS02:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
[93m19:42:58:WARNING:WU00:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)[0m
19:42:58:WU00:FS02:Sending unit results: id:00 state:SEND error:FAULTY project:7810 run:0 clone:66 gen:402 core:0x17 unit:0x000001b50a3b1e8651d34661a08b8f39
19:42:58:WU00:FS02:Uploading 37.33MiB to 171.64.65.98
19:42:58:WU00:FS02:Connecting to 171.64.65.98:8080
19:42:58:WU01:FS02:Connecting to assign-GPU.stanford.edu:80
19:42:59:WU01:FS02:News: Welcome to Folding@Home
19:42:59:WU01:FS02:Assigned to work server 171.64.65.98
19:42:59:WU01:FS02:Requesting new work unit for slot 02: READY gpu:1:GK110 [GeForce GTX 780] from 171.64.65.98
19:42:59:WU01:FS02:Connecting to 171.64.65.98:8080
19:42:59:WU01:FS02:Downloading 2.07MiB
19:43:04:WU00:FS02:Upload 17.75%
19:43:05:WU01:FS02:Download 24.10%
19:43:10:WU00:FS02:Upload 34.99%
19:43:11:WU01:FS02:Download 60.25%
19:43:16:WU00:FS02:Upload 50.73%
19:43:17:WU01:FS02:Download 84.36%
19:43:21:WU01:FS02:Download complete
19:43:21:WU01:FS02:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:7810 run:0 clone:465 gen:423 core:0x17 unit:0x000001c70a3b1e8651d34ae9e3c92149
19:43:21:WU01:FS02:Starting
19:43:21:WU01:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/www.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/beta/Core_17.fah/FahCore_17 -dir 01 -suffix 01 -version 703 -lifeline 1243 -checkpoint 15 -gpu 1 -gpu-vendor nvidia
19:43:21:WU01:FS02:Started FahCore on PID 10881
19:43:21:WU01:FS02:Core PID:10885
19:43:21:WU01:FS02:FahCore 0x17 started
19:43:22:WU01:FS02:0x17:*********************** Log Started 2013-11-10T19:43:21Z ***********************
19:43:22:WU01:FS02:0x17:Project: 7810 (Run 0, Clone 465, Gen 423)
19:43:22:WU01:FS02:0x17:Unit: 0x000001c70a3b1e8651d34ae9e3c92149
19:43:22:WU01:FS02:0x17:CPU: 0x00000000000000000000000000000000
19:43:22:WU01:FS02:0x17:Machine: 2
19:43:22:WU01:FS02:0x17:Reading tar file state.xml
19:43:22:WU01:FS02:0x17:Reading tar file system.xml
19:43:22:WU01:FS02:0x17:Reading tar file integrator.xml
19:43:22:WU01:FS02:0x17:Reading tar file core.xml
19:43:22:WU01:FS02:0x17:Digital signatures verified
19:43:22:WU00:FS02:Upload 61.28%
19:43:28:WU00:FS02:Upload 71.83%
19:43:34:WU00:FS02:Upload 83.55%
19:43:40:WU00:FS02:Upload 95.10%
19:43:41:WU01:FS02:0x17:Completed 0 out of 2000000 steps (0%)
19:43:42:WU02:FS01:0x17:Completed 1580000 out of 2000000 steps (79%)
19:43:43:WU00:FS02:Upload complete
19:43:43:WU00:FS02:Server responded WORK_ACK (400)
19:43:43:WU00:FS02:Cleaning up
19:45:14:WU01:FS02:0x17:Completed 20000 out of 2000000 steps (1%)
