Constant WU Failures on Both GPUs

If you think it might be a driver problem, see viewforum.php?f=79

Moderators: Site Moderators, FAHC Science Team

Post Reply
Nirvana
Posts: 21
Joined: Tue Aug 19, 2014 4:20 pm

Constant WU Failures on Both GPUs

Post by Nirvana »

Hey guys,

I've got a Dell T3500 with a Xeon CPU (working fine) and 2 GPUs. One is a Nvidia FX 580, the other is the Nvidia CUDA K600. Neither GPU is able to fold, I keep getting bad WU or download failures. My settings are all set to default / Auto, all drivers are up to date, I don't have a network proxy or VPN, etc. I have tried folding with just the one GPU, it still did not work.

Please let me know if you need more information to help diagnose this issue. :D


Log:

Code: Select all

16:01:59:WARNING:WU02:FS02:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
16:02:00:WARNING:WU02:FS02:Failed to get assignment from '171.64.65.160:80': Empty work server assignment
16:02:00:ERROR:WU02:FS02:Exception: Could not get an assignment
16:02:00:WARNING:WU02:FS02:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
16:02:02:WARNING:WU02:FS02:Failed to get assignment from '171.64.65.160:80': Empty work server assignment
16:02:02:ERROR:WU02:FS02:Exception: Could not get an assignment
16:02:45:WU01:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
16:02:45:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:03:00:WARNING:WU02:FS02:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
16:03:07:WARNING:WU02:FS02:Failed to get assignment from '171.64.65.160:80': Empty work server assignment
16:03:07:ERROR:WU02:FS02:Exception: Could not get an assignment
16:03:22:WU01:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
16:03:23:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:04:00:WU01:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
16:04:01:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:04:36:WU01:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
16:04:37:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:04:37:WARNING:WU02:FS02:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
16:04:38:WARNING:WU02:FS02:Failed to get assignment from '171.64.65.160:80': Empty work server assignment
16:04:38:ERROR:WU02:FS02:Exception: Could not get an assignment
16:05:14:WU01:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
16:05:14:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:05:53:WU01:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
16:05:54:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:06:31:WU03:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
16:06:31:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:07:09:WU01:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
16:07:09:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:07:15:WARNING:WU02:FS02:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
16:07:15:WARNING:WU02:FS02:Failed to get assignment from '171.64.65.160:80': Empty work server assignment
16:07:15:ERROR:WU02:FS02:Exception: Could not get an assignment
16:07:45:WU03:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
16:07:46:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:08:21:WU01:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
16:08:22:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:11:29:WARNING:WU02:FS02:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
16:11:30:WARNING:WU02:FS02:Failed to get assignment from '171.64.65.160:80': Empty work server assignment
16:11:30:ERROR:WU02:FS02:Exception: Could not get an assignment
16:18:20:WARNING:WU02:FS02:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
16:18:21:WARNING:WU02:FS02:Failed to get assignment from '171.64.65.160:80': Empty work server assignment
16:18:21:ERROR:WU02:FS02:Exception: Could not get an assignment

Thanks!
Image
Top
Joe_H
Site Admin
Posts: 7937
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Constant WU Failures on Both GPUs

Post by Joe_H »

Useful additional information would be posting the beginning section of your log that shows system information and client configuration. Also, what version of the video drivers are you using and do you have the OpenCL support installed?

As for the failure to get assignments, the server which supplied Core_11 WU's suitable for processing on your FX 580 went down a few weeks ago. It probably will not be returning as they are trying to recover any usable data from it and Core_11 had been announced as EOL about a year ago. A topic that covers the last outages of the servers is located here - viewtopic.php?f=18&t=26481. Some pre-Fermi GPU's are suitable for assigning Core_15 projects, I don't know if the FX 580 is in that group.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Top
Nirvana
Posts: 21
Joined: Tue Aug 19, 2014 4:20 pm

Re: Constant WU Failures on Both GPUs

Post by Nirvana »

Here is the config:

Code: Select all

*********************** Log Started 2014-08-19T17:26:14Z ***********************
17:26:14:************************* Folding@home Client *************************
17:26:14:      Website: http://folding.stanford.edu/
17:26:14:    Copyright: (c) 2009-2014 Stanford University
17:26:14:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
17:26:14:         Args: 
17:26:14:       Config: C:/Users/ZZeppo/AppData/Roaming/FAHClient/config.xml
17:26:14:******************************** Build ********************************
17:26:14:      Version: 7.4.4
17:26:14:         Date: Mar 4 2014
17:26:14:         Time: 20:26:54
17:26:14:      SVN Rev: 4130
17:26:14:       Branch: fah/trunk/client
17:26:14:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
17:26:14:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
17:26:14:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
17:26:14:     Platform: win32 XP
17:26:14:         Bits: 32
17:26:14:         Mode: Release
17:26:14:******************************* System ********************************
17:26:14:          CPU: Intel(R) Xeon(R) CPU W3570 @ 3.20GHz
17:26:14:       CPU ID: GenuineIntel Family 6 Model 26 Stepping 5
17:26:14:         CPUs: 8
17:26:14:       Memory: 14.00GiB
17:26:14:  Free Memory: 12.09GiB
17:26:14:      Threads: WINDOWS_THREADS
17:26:14:   OS Version: 6.2
17:26:14:  Has Battery: false
17:26:14:   On Battery: false
17:26:14:   UTC Offset: -5
17:26:14:          PID: 4732
17:26:14:          CWD: C:/Users/ZZeppo/AppData/Roaming/FAHClient
17:26:14:           OS: Windows 8.1 Enterprise
17:26:14:      OS Arch: AMD64
17:26:14:         GPUs: 2
17:26:14:        GPU 0: NVIDIA:1 G96 [Quadro FX 580]
17:26:14:        GPU 1: NVIDIA:3 GK107 [Quadro K600]
17:26:14:         CUDA: 3.0
17:26:14:  CUDA Driver: 6000
17:26:14:Win32 Service: false
17:26:14:***********************************************************************
17:26:14:<config>
17:26:14:  <service-description v='Folding@home Client'/>
17:26:14:  <service-restart v='true'/>
17:26:14:  <service-restart-delay v='5000'/>
17:26:14:
17:26:14:  <!-- Client Control -->
17:26:14:  <client-threads v='6'/>
17:26:14:  <cycle-rate v='4'/>
17:26:14:  <cycles v='-1'/>
17:26:14:  <data-directory v='.'/>
17:26:14:  <disable-sleep-when-active v='true'/>
17:26:14:  <exec-directory v='C:\Program Files (x86)\FAHClient'/>
17:26:14:  <exit-when-done v='false'/>
17:26:14:  <fold-anon v='false'/>
17:26:14:  <open-web-control v='false'/>
17:26:14:
17:26:14:  <!-- Configuration -->
17:26:14:  <config-rotate v='true'/>
17:26:14:  <config-rotate-dir v='configs'/>
17:26:14:  <config-rotate-max v='16'/>
17:26:14:
17:26:14:  <!-- Debugging -->
17:26:14:  <assignment-servers>
17:26:14:    assign3.stanford.edu:8080 assign4.stanford.edu:80
17:26:14:  </assignment-servers>
17:26:14:  <auth-as v='true'/>
17:26:14:  <capture-directory v='capture'/>
17:26:14:  <capture-on-error v='false'/>
17:26:14:  <capture-packets v='false'/>
17:26:14:  <capture-requests v='false'/>
17:26:14:  <capture-responses v='false'/>
17:26:14:  <capture-sockets v='false'/>
17:26:14:  <core-exec v='FahCore_$type'/>
17:26:14:  <core-wrapper-exec v='FAHCoreWrapper'/>
17:26:14:  <debug-sockets v='false'/>
17:26:14:  <exception-locations v='true'/>
17:26:14:  <gpu-assignment-servers>
17:26:14:    assign-GPU.stanford.edu:80 assign-GPU2.stanford.edu:80
17:26:14:  </gpu-assignment-servers>
17:26:14:  <stack-traces v='false'/>
17:26:14:
17:26:14:  <!-- Error Handling -->
17:26:14:  <max-slot-errors v='10'/>
17:26:14:  <max-unit-errors v='5'/>
17:26:14:
17:26:14:  <!-- Folding Core -->
17:26:14:  <checkpoint v='10'/>
17:26:14:  <core-dir v='cores'/>
17:26:14:  <core-priority v='idle'/>
17:26:14:  <cpu-affinity v='false'/>
17:26:14:  <cpu-usage v='100'/>
17:26:14:  <gpu-usage v='100'/>
17:26:14:  <no-assembly v='false'/>
17:26:14:
17:26:14:  <!-- Folding Slot Configuration -->
17:26:14:  <cause v='ANY'/>
17:26:14:  <client-subtype v='STDCLI'/>
17:26:14:  <client-type v='normal'/>
17:26:14:  <cpu-species v='X86_PENTIUM_II'/>
17:26:14:  <cpu-type v='AMD64'/>
17:26:14:  <cpus v='-1'/>
17:26:14:  <gpu v='true'/>
17:26:14:  <max-packet-size v='normal'/>
17:26:14:  <os-species v='UNKNOWN'/>
17:26:14:  <os-type v='WIN32'/>
17:26:14:  <project-key v='0'/>
17:26:14:  <smp v='true'/>
17:26:14:
17:26:14:  <!-- GUI -->
17:26:14:  <gui-enabled v='true'/>
17:26:14:
17:26:14:  <!-- HTTP Server -->
17:26:14:  <allow v='127.0.0.1'/>
17:26:14:  <connection-timeout v='60'/>
17:26:14:  <deny v='0/0'/>
17:26:14:  <http-addresses v='0:7396'/>
17:26:14:  <https-addresses v=''/>
17:26:14:  <max-connect-time v='900'/>
17:26:14:  <max-connections v='800'/>
17:26:14:  <max-request-length v='52428800'/>
17:26:14:  <min-connect-time v='300'/>
17:26:14:  <threads v='8'/>
17:26:14:
17:26:14:  <!-- Logging -->
17:26:14:  <log v='log.txt'/>
17:26:14:  <log-color v='false'/>
17:26:14:  <log-crlf v='true'/>
17:26:14:  <log-date v='false'/>
17:26:14:  <log-date-periodically v='21600'/>
17:26:14:  <log-debug v='true'/>
17:26:14:  <log-domain v='false'/>
17:26:14:  <log-header v='true'/>
17:26:14:  <log-level v='true'/>
17:26:14:  <log-no-info-header v='true'/>
17:26:14:  <log-redirect v='false'/>
17:26:14:  <log-rotate v='true'/>
17:26:14:  <log-rotate-dir v='logs'/>
17:26:14:  <log-rotate-max v='16'/>
17:26:14:  <log-short-level v='false'/>
17:26:14:  <log-simple-domains v='true'/>
17:26:14:  <log-thread-id v='false'/>
17:26:14:  <log-thread-prefix v='true'/>
17:26:14:  <log-time v='true'/>
17:26:14:  <log-to-screen v='true'/>
17:26:14:  <log-truncate v='false'/>
17:26:14:  <verbosity v='4'/>
17:26:14:
17:26:14:  <!-- Network -->
17:26:14:  <proxy v=':8080'/>
17:26:14:  <proxy-enable v='false'/>
17:26:14:  <proxy-pass v=''/>
17:26:14:  <proxy-user v=''/>
17:26:14:
17:26:14:  <!-- Process Control -->
17:26:14:  <child v='false'/>
17:26:14:  <daemon v='false'/>
17:26:14:  <pid v='false'/>
17:26:14:  <pid-file v='Folding@home Client.pid'/>
17:26:14:  <respawn v='false'/>
17:26:14:  <service v='false'/>
17:26:14:
17:26:14:  <!-- Remote Command Server -->
17:26:14:  <command-address v='0.0.0.0'/>
17:26:14:  <command-allow-no-pass v='127.0.0.1'/>
17:26:14:  <command-deny-no-pass v='0/0'/>
17:26:14:  <command-enable v='true'/>
17:26:14:  <command-port v='36330'/>
17:26:14:
17:26:14:  <!-- Slot Control -->
17:26:14:  <idle v='false'/>
17:26:14:  <max-shutdown-wait v='60'/>
17:26:14:  <pause-on-battery v='true'/>
17:26:14:  <pause-on-start v='false'/>
17:26:14:  <paused v='false'/>
17:26:14:  <power v='FULL'/>
17:26:14:
17:26:14:  <!-- User Information -->
17:26:14:  <machine-id v='0'/>
17:26:14:  <passkey v='********************************'/>
17:26:14:  <team v='111065'/>
17:26:14:  <user v='Zepppster'/>
17:26:14:
17:26:14:  <!-- Web Server -->
17:26:14:  <web-allow v='127.0.0.1'/>
17:26:14:  <web-deny v='0/0'/>
17:26:14:  <web-enable v='true'/>
17:26:14:
17:26:14:  <!-- Web Server Sessions -->
17:26:14:  <session-cookie v='sid'/>
17:26:14:  <session-lifetime v='86400'/>
17:26:14:  <session-timeout v='3600'/>
17:26:14:
17:26:14:  <!-- Work Unit Control -->
17:26:14:  <dump-after-deadline v='true'/>
17:26:14:  <max-queue v='16'/>
17:26:14:  <max-units v='0'/>
17:26:14:  <next-unit-percentage v='99'/>
17:26:14:  <stall-detection-enabled v='false'/>
17:26:14:  <stall-percent v='5'/>
17:26:14:  <stall-timeout v='1800'/>
17:26:14:
17:26:14:  <!-- Folding Slots -->
17:26:14:  <slot id='0' type='CPU'/>
17:26:14:  <slot id='1' type='GPU'>
17:26:14:    <gpu-index v='1'/>
17:26:14:  </slot>
17:26:14:  <slot id='2' type='GPU'>
17:26:14:    <gpu-index v='0'/>
17:26:14:  </slot>
17:26:14:</config>
17:26:14:Trying to access database...
17:26:14:Successfully acquired database lock
17:26:14:Enabled folding slot 00: READY cpu:6
17:26:14:Enabled folding slot 01: READY gpu:1:GK107 [Quadro K600]
17:26:14:Enabled folding slot 02: READY gpu:0:G96 [Quadro FX 580]
17:26:14:WU00:FS00:Starting
17:26:14:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/ZZeppo/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a3.fah/FahCore_a3.exe -dir 00 -suffix 01 -version 704 -lifeline 4732 -checkpoint 10 -np 6
17:26:14:WU00:FS00:Started FahCore on PID 2500
17:26:14:WU00:FS00:Core PID:608
17:26:14:WU00:FS00:FahCore 0xa3 started
17:26:14:WU01:FS01:Connecting to 171.67.108.201:80
17:26:14:WU02:FS02:Connecting to 171.67.108.201:80
17:26:14:1:127.0.0.1 GET /api/updates?_=1408464629508&sid=4d51b98bc78e7b701fe6f80f5b075853
17:26:15:WU00:FS00:0xa3:
17:26:15:WU00:FS00:0xa3:*------------------------------*
17:26:15:WU00:FS00:0xa3:Folding@Home Gromacs SMP Core
17:26:15:WU00:FS00:0xa3:Version 2.27 (Dec. 15, 2010)
17:26:15:WU00:FS00:0xa3:
17:26:15:WU00:FS00:0xa3:Preparing to commence simulation
17:26:15:WU00:FS00:0xa3:- Looking at optimizations...
17:26:15:WU00:FS00:0xa3:- Files status OK
17:26:15:WU00:FS00:0xa3:- Expanded 3806047 -> 4136808 (decompressed 108.6 percent)
17:26:15:WU00:FS00:0xa3:Called DecompressByteArray: compressed_data_size=3806047 data_size=4136808, decompressed_data_size=4136808 diff=0
17:26:15:WU00:FS00:0xa3:- Digital signature verified
17:26:15:WU00:FS00:0xa3:
17:26:15:WU00:FS00:0xa3:Project: 6098 (Run 0, Clone 48, Gen 492)
17:26:15:WU00:FS00:0xa3:
17:26:15:WU00:FS00:0xa3:Assembly optimizations on if available.
17:26:15:WU00:FS00:0xa3:Entering M.D.
17:26:15:WU01:FS01:Assigned to work server 140.163.4.233
17:26:15:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:GK107 [Quadro K600] from 140.163.4.233
17:26:15:WU01:FS01:Connecting to 140.163.4.233:8080
17:26:15:WARNING:WU02:FS02:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
17:26:15:WU02:FS02:Connecting to 171.64.65.160:80
17:26:15:2:127.0.0.1 GET /?nocache=0.18053506407886744
17:26:15:WU01:FS01:Downloading 4.25MiB
17:26:15:WARNING:WU02:FS02:Failed to get assignment from '171.64.65.160:80': Empty work server assignment
17:26:15:ERROR:WU02:FS02:Exception: Could not get an assignment
17:26:15:WU02:FS02:Connecting to 171.67.108.201:80
17:26:15:WARNING:WU02:FS02:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
17:26:15:WU02:FS02:Connecting to 171.64.65.160:80
17:26:15:3:127.0.0.1 GET /css/normalize.css
17:26:15:4:127.0.0.1 GET /css/main.css
17:26:15:5:127.0.0.1 GET /images/mail.png
17:26:15:6:127.0.0.1 GET /images/facebook.png
17:26:15:7:127.0.0.1 GET /images/twitter.png
17:26:15:WU01:FS01:Download complete
17:26:15:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:10467 run:0 clone:94 gen:28 core:0x17 unit:0x0000002d538b3db9538bbc4b9626b20b
17:26:16:WU01:FS01:Starting
17:26:16:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/ZZeppo/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe -dir 01 -suffix 01 -version 704 -lifeline 4732 -checkpoint 10 -gpu 1 -gpu-vendor nvidia
17:26:16:WU01:FS01:Started FahCore on PID 4556
17:26:16:WU01:FS01:Core PID:1316
17:26:16:WU01:FS01:FahCore 0x17 started
17:26:16:8:127.0.0.1 GET /js/jquery-1.10.2.min.js
17:26:16:9:127.0.0.1 GET /js/libs/jquery.selectbox-0.2.js
17:26:16:WARNING:WU02:FS02:Failed to get assignment from '171.64.65.160:80': Empty work server assignment
17:26:16:ERROR:WU02:FS02:Exception: Could not get an assignment
17:26:16:10:127.0.0.1 GET /js/intercom.min.js
17:26:16:11:127.0.0.1 GET /js/main.js
17:26:16:11:127.0.0.1:New Web connection
17:26:16:12:127.0.0.1 GET /images/report-bug.png
17:26:16:13:127.0.0.1 GET /images/template/bg.jpg
17:26:16:14:127.0.0.1 GET /images/template/logo_folding_home.png
17:26:16:15:127.0.0.1 GET /images/template/input_radio.png
17:26:16:16:127.0.0.1 GET /images/template/button_bg.png
17:26:16:WU01:FS01:0x17:*********************** Log Started 2014-08-19T17:26:16Z ***********************
17:26:16:WU01:FS01:0x17:Project: 10467 (Run 0, Clone 94, Gen 28)
17:26:16:WU01:FS01:0x17:Unit: 0x0000002d538b3db9538bbc4b9626b20b
17:26:16:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
17:26:16:WU01:FS01:0x17:Machine: 1
17:26:16:WU01:FS01:0x17:Reading tar file state.xml
17:26:16:17:127.0.0.1 GET /api/updates/set?_=1408469176286&sid=26347e3b2d34e94cde21d2f3fb34399a&update_id=0&update_path=/api/basic&update_rate=1
17:26:16:18:127.0.0.1 GET /api/updates/set?_=1408469176287&sid=26347e3b2d34e94cde21d2f3fb34399a&update_id=1&update_path=/api/slots&update_rate=1
17:26:17:WU01:FS01:0x17:Reading tar file system.xml
17:26:17:19:127.0.0.1 GET /api/configured?_=1408469176288&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:17:20:127.0.0.1 GET /images/template/select-icons.png
17:26:17:21:127.0.0.1 GET /images/template/ui-bg-slider.png
17:26:17:22:127.0.0.1 GET /images/template/ui-icon-slider.png
17:26:17:WU01:FS01:0x17:Reading tar file integrator.xml
17:26:17:WU01:FS01:0x17:Reading tar file core.xml
17:26:17:WU01:FS01:0x17:Digital signatures verified
17:26:17:WU01:FS01:0x17:Folding@home GPU core17
17:26:17:WU01:FS01:0x17:Version 0.0.52
17:26:18:23:127.0.0.1 GET /api/updates?_=1408469176289&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:18:24:127.0.0.1 GET /images/template/ui-progress-bg.png
17:26:18:23:127.0.0.1 GET /images/template/anim_arrow.gif
17:26:18:26:127.0.0.1 GET /api/updates?_=1408469176293&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:19:27:127.0.0.1 GET /api/updates?_=1408469176294&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:20:27:127.0.0.1 GET /api/updates?_=1408469176295&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:21:WU00:FS00:0xa3:Using Gromacs checkpoints
17:26:21:WU00:FS00:0xa3:Mapping NT from 6 to 6 
17:26:21:WU00:FS00:0xa3:Resuming from checkpoint
17:26:21:WU00:FS00:0xa3:Verified 00/wudata_01.log
17:26:22:WU00:FS00:0xa3:Verified 00/wudata_01.trr
17:26:22:WU00:FS00:0xa3:Verified 00/wudata_01.edr
17:26:22:WU00:FS00:0xa3:Completed 299585 out of 500000 steps  (59%)
17:26:22:26:127.0.0.1 GET /api/updates?_=1408469176296&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:24:26:127.0.0.1 GET /api/updates?_=1408469176297&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:26:26:127.0.0.1 GET /api/updates?_=1408469176298&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:27:26:127.0.0.1 GET /api/updates?_=1408469176299&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:29:27:127.0.0.1 GET /api/updates?_=1408469176300&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:30:26:127.0.0.1 GET /api/updates?_=1408469176301&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:32:27:127.0.0.1 GET /api/updates?_=1408469176302&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:35:26:127.0.0.1 GET /api/updates?_=1408469176303&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:37:27:127.0.0.1 GET /api/updates?_=1408469176304&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:40:26:127.0.0.1 GET /api/updates?_=1408469176305&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:42:27:127.0.0.1 GET /api/updates?_=1408469176306&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:44:26:127.0.0.1 GET /api/updates?_=1408469176307&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:45:26:127.0.0.1 GET /api/updates?_=1408469176308&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:46:26:127.0.0.1 GET /api/updates?_=1408469176309&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:48:26:127.0.0.1 GET /api/updates?_=1408469176310&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:49:26:127.0.0.1 GET /api/updates?_=1408469176311&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:50:26:127.0.0.1 GET /api/updates?_=1408469176312&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:52:26:127.0.0.1 GET /api/updates?_=1408469176313&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:53:26:127.0.0.1 GET /api/updates?_=1408469176314&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:54:26:127.0.0.1 GET /api/updates?_=1408469176315&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:56:26:127.0.0.1 GET /api/updates?_=1408469176316&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:57:26:127.0.0.1 GET /api/updates?_=1408469176317&sid=26347e3b2d34e94cde21d2f3fb34399a
17:26:58:26:127.0.0.1 GET /api/updates?_=1408469176318&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:00:26:127.0.0.1 GET /api/updates?_=1408469176319&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:01:26:127.0.0.1 GET /api/updates?_=1408469176320&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:02:26:127.0.0.1 GET /api/updates?_=1408469176321&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:04:26:127.0.0.1 GET /api/updates?_=1408469176322&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:05:26:127.0.0.1 GET /api/updates?_=1408469176323&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:06:26:127.0.0.1 GET /api/updates?_=1408469176324&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:07:26:127.0.0.1 GET /api/updates?_=1408469176325&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:08:26:127.0.0.1 GET /api/updates?_=1408469176326&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:09:26:127.0.0.1 GET /api/updates?_=1408469176327&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:11:27:127.0.0.1 GET /api/updates?_=1408469176328&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:12:26:127.0.0.1 GET /api/updates?_=1408469176329&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:13:26:127.0.0.1 GET /api/updates?_=1408469176330&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:15:26:127.0.0.1 GET /api/updates?_=1408469176331&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:15:WU02:FS02:Connecting to 171.67.108.201:80
17:27:15:WARNING:WU02:FS02:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
17:27:15:WU02:FS02:Connecting to 171.64.65.160:80
17:27:15:WARNING:WU02:FS02:Failed to get assignment from '171.64.65.160:80': Empty work server assignment
17:27:15:ERROR:WU02:FS02:Exception: Could not get an assignment
17:27:17:26:127.0.0.1 GET /api/updates?_=1408469176332&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:19:26:127.0.0.1 GET /api/updates?_=1408469176333&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:20:26:127.0.0.1 GET /api/updates?_=1408469176334&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:22:27:127.0.0.1 GET /api/updates?_=1408469176335&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:25:27:127.0.0.1 GET /api/updates?_=1408469176336&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:27:26:127.0.0.1 GET /api/updates?_=1408469176337&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:29:WU00:FS00:0xa3:Completed 300000 out of 500000 steps  (60%)
17:27:30:26:127.0.0.1 GET /api/updates?_=1408469176338&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:32:27:127.0.0.1 GET /api/updates?_=1408469176339&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:33:26:127.0.0.1 GET /api/updates?_=1408469176340&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:35:27:127.0.0.1 GET /api/updates?_=1408469176341&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:37:26:127.0.0.1 GET /api/updates?_=1408469176342&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:38:26:127.0.0.1 GET /api/updates?_=1408469176343&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:39:26:127.0.0.1 GET /api/updates?_=1408469176344&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:41:26:127.0.0.1 GET /api/updates?_=1408469176345&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:42:26:127.0.0.1 GET /api/updates?_=1408469176346&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:43:26:127.0.0.1 GET /api/updates?_=1408469176347&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:45:26:127.0.0.1 GET /api/updates?_=1408469176348&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:47:26:127.0.0.1 GET /api/updates?_=1408469176349&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:48:26:127.0.0.1 GET /api/updates?_=1408469176350&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:49:26:127.0.0.1 GET /api/updates?_=1408469176351&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:51:26:127.0.0.1 GET /api/updates?_=1408469176352&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:52:26:127.0.0.1 GET /api/updates?_=1408469176353&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:53:26:127.0.0.1 GET /api/updates?_=1408469176354&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:55:27:127.0.0.1 GET /api/updates?_=1408469176355&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:57:27:127.0.0.1 GET /api/updates?_=1408469176356&sid=26347e3b2d34e94cde21d2f3fb34399a
17:27:59:27:127.0.0.1 GET /api/updates?_=1408469176357&sid=26347e3b2d34e94cde21d2f3fb34399a
17:28:00:27:127.0.0.1 GET /api/updates?_=1408469176358&sid=26347e3b2d34e94cde21d2f3fb34399a
17:28:02:26:127.0.0.1 GET /api/updates?_=1408469176359&sid=26347e3b2d34e94cde21d2f3fb34399a
17:28:03:27:127.0.0.1 GET /api/updates?_=1408469176360&sid=26347e3b2d34e94cde21d2f3fb34399a
17:28:05:26:127.0.0.1 GET /api/updates?_=1408469176361&sid=26347e3b2d34e94cde21d2f3fb34399a
17:28:07:27:127.0.0.1 GET /api/updates?_=1408469176362&sid=26347e3b2d34e94cde21d2f3fb34399a
17:28:09:27:127.0.0.1 GET /api/updates?_=1408469176363&sid=26347e3b2d34e94cde21d2f3fb34399a
17:28:10:27:127.0.0.1 GET /api/updates?_=1408469176364&sid=26347e3b2d34e94cde21d2f3fb34399a
17:28:11:27:127.0.0.1 GET /api/updates?_=1408469176365&sid=26347e3b2d34e94cde21d2f3fb34399a
17:28:13:27:127.0.0.1 GET /api/updates?_=1408469176366&sid=26347e3b2d34e94cde21d2f3fb34399a
17:28:15:27:127.0.0.1 GET /api/updates?_=1408469176367&sid=26347e3b2d34e94cde21d2f3fb34399a
17:28:52:WU02:FS02:Connecting to 171.67.108.201:80
17:28:52:WARNING:WU02:FS02:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
17:28:52:WU02:FS02:Connecting to 171.64.65.160:80
17:28:53:WARNING:WU02:FS02:Failed to get assignment from '171.64.65.160:80': Empty work server assignment
17:28:53:ERROR:WU02:FS02:Exception: Could not get an assignment
17:29:18:WU01:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
17:29:18:WU01:FS01:0x17:Saving result file logfile_01.txt
17:29:18:WU01:FS01:0x17:Saving result file log.txt
17:29:18:WU01:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
17:29:19:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
17:29:19:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:10467 run:0 clone:94 gen:28 core:0x17 unit:0x0000002d538b3db9538bbc4b9626b20b
17:29:19:WU01:FS01:Uploading 2.32KiB to 140.163.4.233
17:29:19:WU01:FS01:Connecting to 140.163.4.233:8080
17:29:19:WU01:FS01:Upload complete
17:29:19:WU01:FS01:Server responded WORK_ACK (400)
17:29:19:WU01:FS01:Cleaning up
17:29:19:WU01:FS01:Connecting to 171.67.108.201:80
17:29:19:WU01:FS01:Assigned to work server 140.163.4.233
17:29:19:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:GK107 [Quadro K600] from 140.163.4.233
17:29:19:WU01:FS01:Connecting to 140.163.4.233:8080
17:29:19:WU01:FS01:Downloading 4.31MiB
17:29:20:WU01:FS01:Download complete
17:29:20:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:10468 run:0 clone:165 gen:21 core:0x17 unit:0x00000028538b3db9538caebc7c2785b2
17:29:20:WU01:FS01:Starting
17:29:20:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/ZZeppo/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe -dir 01 -suffix 01 -version 704 -lifeline 4732 -checkpoint 10 -gpu 1 -gpu-vendor nvidia
17:29:20:WU01:FS01:Started FahCore on PID 3332
17:29:20:WU01:FS01:Core PID:4212
17:29:20:WU01:FS01:FahCore 0x17 started
17:29:21:WU01:FS01:0x17:*********************** Log Started 2014-08-19T17:29:20Z ***********************
17:29:21:WU01:FS01:0x17:Project: 10468 (Run 0, Clone 165, Gen 21)
17:29:21:WU01:FS01:0x17:Unit: 0x00000028538b3db9538caebc7c2785b2
17:29:21:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
17:29:21:WU01:FS01:0x17:Machine: 1
17:29:21:WU01:FS01:0x17:Reading tar file state.xml
17:29:21:WU01:FS01:0x17:Reading tar file system.xml
17:29:22:WU01:FS01:0x17:Reading tar file integrator.xml
17:29:22:WU01:FS01:0x17:Reading tar file core.xml
17:29:22:WU01:FS01:0x17:Digital signatures verified
17:29:22:WU01:FS01:0x17:Folding@home GPU core17
17:29:22:WU01:FS01:0x17:Version 0.0.52
17:31:29:WU02:FS02:Connecting to 171.67.108.201:80
17:31:29:WARNING:WU02:FS02:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
17:31:29:WU02:FS02:Connecting to 171.64.65.160:80
17:31:30:WARNING:WU02:FS02:Failed to get assignment from '171.64.65.160:80': Empty work server assignment
17:31:30:ERROR:WU02:FS02:Exception: Could not get an assignment
17:32:32:WU01:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
17:32:32:WU01:FS01:0x17:Saving result file logfile_01.txt
17:32:32:WU01:FS01:0x17:Saving result file log.txt
17:32:32:WU01:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
17:32:32:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
17:32:32:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:10468 run:0 clone:165 gen:21 core:0x17 unit:0x00000028538b3db9538caebc7c2785b2
17:32:32:WU01:FS01:Uploading 2.33KiB to 140.163.4.233
17:32:32:WU01:FS01:Connecting to 140.163.4.233:8080
17:32:32:WU01:FS01:Upload complete
17:32:32:WU01:FS01:Server responded WORK_ACK (400)
17:32:32:WU01:FS01:Cleaning up
17:32:32:WU03:FS01:Connecting to 171.67.108.201:80
17:32:33:WU03:FS01:Assigned to work server 140.163.4.233
17:32:33:WU03:FS01:Requesting new work unit for slot 01: READY gpu:1:GK107 [Quadro K600] from 140.163.4.233
17:32:33:WU03:FS01:Connecting to 140.163.4.233:8080
17:32:33:WU03:FS01:Downloading 4.91MiB
17:32:34:WU03:FS01:Download complete
17:32:34:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:10469 run:0 clone:485 gen:20 core:0x17 unit:0x00000038538b3db9538f455caa2acf58
17:32:34:WU03:FS01:Starting
17:32:34:WU03:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/ZZeppo/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe -dir 03 -suffix 01 -version 704 -lifeline 4732 -checkpoint 10 -gpu 1 -gpu-vendor nvidia
17:32:34:WU03:FS01:Started FahCore on PID 4928
17:32:34:WU03:FS01:Core PID:3460
17:32:34:WU03:FS01:FahCore 0x17 started
17:32:34:WU03:FS01:0x17:*********************** Log Started 2014-08-19T17:32:34Z ***********************
17:32:34:WU03:FS01:0x17:Project: 10469 (Run 0, Clone 485, Gen 20)
17:32:34:WU03:FS01:0x17:Unit: 0x00000038538b3db9538f455caa2acf58
17:32:34:WU03:FS01:0x17:CPU: 0x00000000000000000000000000000000
17:32:34:WU03:FS01:0x17:Machine: 1
17:32:34:WU03:FS01:0x17:Reading tar file state.xml
17:32:35:WU03:FS01:0x17:Reading tar file system.xml
17:32:36:WU03:FS01:0x17:Reading tar file integrator.xml
17:32:36:WU03:FS01:0x17:Reading tar file core.xml
17:32:36:WU03:FS01:0x17:Digital signatures verified
17:32:36:WU03:FS01:0x17:Folding@home GPU core17
17:32:36:WU03:FS01:0x17:Version 0.0.52
17:35:44:WU02:FS02:Connecting to 171.67.108.201:80
17:35:44:WARNING:WU02:FS02:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
17:35:44:WU02:FS02:Connecting to 171.64.65.160:80
17:35:44:WARNING:WU02:FS02:Failed to get assignment from '171.64.65.160:80': Empty work server assignment
17:35:44:ERROR:WU02:FS02:Exception: Could not get an assignment
17:36:37:WU03:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
17:36:37:WU03:FS01:0x17:Saving result file logfile_01.txt
17:36:37:WU03:FS01:0x17:Saving result file log.txt
17:36:37:WU03:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
17:36:38:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
17:36:38:WU03:FS01:Sending unit results: id:03 state:SEND error:FAULTY project:10469 run:0 clone:485 gen:20 core:0x17 unit:0x00000038538b3db9538f455caa2acf58
17:36:38:WU03:FS01:Uploading 2.33KiB to 140.163.4.233
17:36:38:WU03:FS01:Connecting to 140.163.4.233:8080
17:36:38:WU03:FS01:Upload complete
17:36:38:WU03:FS01:Server responded WORK_ACK (400)
17:36:38:WU03:FS01:Cleaning up
17:36:38:WU01:FS01:Connecting to 171.67.108.201:80
17:36:38:WU01:FS01:Assigned to work server 140.163.4.233
17:36:38:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:GK107 [Quadro K600] from 140.163.4.233
17:36:38:WU01:FS01:Connecting to 140.163.4.233:8080
17:36:38:WU01:FS01:Downloading 4.93MiB
17:36:40:WU01:FS01:Download complete
17:36:40:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:10469 run:0 clone:367 gen:29 core:0x17 unit:0x00000031538b3db9538f421f84164707
17:36:40:WU01:FS01:Starting
17:36:40:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/ZZeppo/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe -dir 01 -suffix 01 -version 704 -lifeline 4732 -checkpoint 10 -gpu 1 -gpu-vendor nvidia
17:36:40:WU01:FS01:Started FahCore on PID 4620
17:36:40:WU01:FS01:Core PID:2100
17:36:40:WU01:FS01:FahCore 0x17 started
17:36:40:WU01:FS01:0x17:*********************** Log Started 2014-08-19T17:36:40Z ***********************
17:36:40:WU01:FS01:0x17:Project: 10469 (Run 0, Clone 367, Gen 29)
17:36:40:WU01:FS01:0x17:Unit: 0x00000031538b3db9538f421f84164707
17:36:40:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
17:36:40:WU01:FS01:0x17:Machine: 1
17:36:40:WU01:FS01:0x17:Reading tar file state.xml
17:36:41:WU01:FS01:0x17:Reading tar file system.xml
17:36:42:WU01:FS01:0x17:Reading tar file integrator.xml
17:36:42:WU01:FS01:0x17:Reading tar file core.xml
17:36:42:WU01:FS01:0x17:Digital signatures verified
17:36:42:WU01:FS01:0x17:Folding@home GPU core17
17:36:42:WU01:FS01:0x17:Version 0.0.52
17:40:39:WU01:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
17:40:39:WU01:FS01:0x17:Saving result file logfile_01.txt
17:40:39:WU01:FS01:0x17:Saving result file log.txt
17:40:39:WU01:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
17:40:40:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
17:40:40:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:10469 run:0 clone:367 gen:29 core:0x17 unit:0x00000031538b3db9538f421f84164707
17:40:40:WU01:FS01:Uploading 2.32KiB to 140.163.4.233
17:40:40:WU01:FS01:Connecting to 140.163.4.233:8080
17:40:40:WU01:FS01:Upload complete
17:40:40:WU01:FS01:Server responded WORK_ACK (400)
17:40:40:WU01:FS01:Cleaning up
17:40:40:WU01:FS01:Connecting to 171.67.108.201:80
17:40:40:WU01:FS01:Assigned to work server 140.163.4.233
17:40:40:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:GK107 [Quadro K600] from 140.163.4.233
17:40:40:WU01:FS01:Connecting to 140.163.4.233:8080
17:40:41:WU01:FS01:Downloading 4.27MiB
17:40:41:WU01:FS01:Download complete
17:40:42:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:10467 run:0 clone:349 gen:31 core:0x17 unit:0x00000037538b3db9538bc26930377c21
17:40:42:WU01:FS01:Starting
17:40:42:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/ZZeppo/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe -dir 01 -suffix 01 -version 704 -lifeline 4732 -checkpoint 10 -gpu 1 -gpu-vendor nvidia
17:40:42:WU01:FS01:Started FahCore on PID 120
17:40:42:WU01:FS01:Core PID:2960
17:40:42:WU01:FS01:FahCore 0x17 started
17:40:42:WU01:FS01:0x17:*********************** Log Started 2014-08-19T17:40:42Z ***********************
17:40:42:WU01:FS01:0x17:Project: 10467 (Run 0, Clone 349, Gen 31)
17:40:42:WU01:FS01:0x17:Unit: 0x00000037538b3db9538bc26930377c21
17:40:42:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
17:40:42:WU01:FS01:0x17:Machine: 1
17:40:42:WU01:FS01:0x17:Reading tar file state.xml
17:40:43:WU01:FS01:0x17:Reading tar file system.xml
17:40:44:WU01:FS01:0x17:Reading tar file integrator.xml
17:40:44:WU01:FS01:0x17:Reading tar file core.xml
17:40:44:WU01:FS01:0x17:Digital signatures verified
17:40:44:WU01:FS01:0x17:Folding@home GPU core17
17:40:44:WU01:FS01:0x17:Version 0.0.52
17:41:07:WU00:FS00:0xa3:Completed 305000 out of 500000 steps  (61%)
17:42:35:WU02:FS02:Connecting to 171.67.108.201:80
17:42:35:WARNING:WU02:FS02:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
17:42:35:WU02:FS02:Connecting to 171.64.65.160:80
17:42:35:WARNING:WU02:FS02:Failed to get assignment from '171.64.65.160:80': Empty work server assignment
17:42:35:ERROR:WU02:FS02:Exception: Could not get an assignment
17:43:38:WU01:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
17:43:38:WU01:FS01:0x17:Saving result file logfile_01.txt
17:43:38:WU01:FS01:0x17:Saving result file log.txt
17:43:38:WU01:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
17:43:38:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
17:43:38:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:10467 run:0 clone:349 gen:31 core:0x17 unit:0x00000037538b3db9538bc26930377c21
17:43:38:WU01:FS01:Uploading 2.32KiB to 140.163.4.233
17:43:38:WU01:FS01:Connecting to 140.163.4.233:8080
17:43:38:WU01:FS01:Upload complete
17:43:38:WU01:FS01:Server responded WORK_ACK (400)
17:43:38:WU01:FS01:Cleaning up
17:43:38:WU01:FS01:Connecting to 171.67.108.201:80
17:43:39:WU01:FS01:Assigned to work server 140.163.4.233
17:43:39:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:GK107 [Quadro K600] from 140.163.4.233
17:43:39:WU01:FS01:Connecting to 140.163.4.233:8080
17:43:39:WU01:FS01:Downloading 4.93MiB
17:43:39:WU01:FS01:Download complete
17:43:39:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:10469 run:0 clone:443 gen:24 core:0x17 unit:0x00000032538b3db9538f4436ff086304
17:43:39:WU01:FS01:Starting
17:43:39:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/ZZeppo/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe -dir 01 -suffix 01 -version 704 -lifeline 4732 -checkpoint 10 -gpu 1 -gpu-vendor nvidia
17:43:39:WU01:FS01:Started FahCore on PID 3588
17:43:40:WU01:FS01:Core PID:1780
17:43:40:WU01:FS01:FahCore 0x17 started
17:43:40:WU01:FS01:0x17:*********************** Log Started 2014-08-19T17:43:40Z ***********************
17:43:40:WU01:FS01:0x17:Project: 10469 (Run 0, Clone 443, Gen 24)
17:43:40:WU01:FS01:0x17:Unit: 0x00000032538b3db9538f4436ff086304
17:43:40:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
17:43:40:WU01:FS01:0x17:Machine: 1
17:43:40:WU01:FS01:0x17:Reading tar file state.xml
17:43:41:WU01:FS01:0x17:Reading tar file system.xml
17:43:42:WU01:FS01:0x17:Reading tar file integrator.xml
17:43:42:WU01:FS01:0x17:Reading tar file core.xml
17:43:42:WU01:FS01:0x17:Digital signatures verified
17:43:42:WU01:FS01:0x17:Folding@home GPU core17
17:43:42:WU01:FS01:0x17:Version 0.0.52
17:47:39:WU01:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
17:47:39:WU01:FS01:0x17:Saving result file logfile_01.txt
17:47:39:WU01:FS01:0x17:Saving result file log.txt
17:47:39:WU01:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
17:47:40:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
17:47:40:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:10469 run:0 clone:443 gen:24 core:0x17 unit:0x00000032538b3db9538f4436ff086304
17:47:40:WU01:FS01:Uploading 2.33KiB to 140.163.4.233
17:47:40:WU01:FS01:Connecting to 140.163.4.233:8080
17:47:40:WU01:FS01:Upload complete
17:47:40:WU01:FS01:Server responded WORK_ACK (400)
17:47:40:WU01:FS01:Cleaning up
17:47:40:WU03:FS01:Connecting to 171.67.108.201:80
17:47:40:WU03:FS01:Assigned to work server 140.163.4.233
17:47:40:WU03:FS01:Requesting new work unit for slot 01: READY gpu:1:GK107 [Quadro K600] from 140.163.4.233
17:47:40:WU03:FS01:Connecting to 140.163.4.233:8080
17:47:40:WU03:FS01:Downloading 4.93MiB
17:47:42:WU03:FS01:Download complete
17:47:42:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:10469 run:0 clone:362 gen:27 core:0x17 unit:0x0000003a538b3db9538f41fb2858db41
17:47:42:WU03:FS01:Starting
17:47:42:WU03:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/ZZeppo/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe -dir 03 -suffix 01 -version 704 -lifeline 4732 -checkpoint 10 -gpu 1 -gpu-vendor nvidia
17:47:42:WU03:FS01:Started FahCore on PID 4000
17:47:42:WU03:FS01:Core PID:4972
17:47:42:WU03:FS01:FahCore 0x17 started
17:47:42:WU03:FS01:0x17:*********************** Log Started 2014-08-19T17:47:42Z ***********************
17:47:42:WU03:FS01:0x17:Project: 10469 (Run 0, Clone 362, Gen 27)
17:47:42:WU03:FS01:0x17:Unit: 0x0000003a538b3db9538f41fb2858db41
17:47:42:WU03:FS01:0x17:CPU: 0x00000000000000000000000000000000
17:47:42:WU03:FS01:0x17:Machine: 1
17:47:42:WU03:FS01:0x17:Reading tar file state.xml
17:47:43:WU03:FS01:0x17:Reading tar file system.xml
17:47:44:WU03:FS01:0x17:Reading tar file integrator.xml
17:47:44:WU03:FS01:0x17:Reading tar file core.xml
17:47:44:WU03:FS01:0x17:Digital signatures verified
17:47:44:WU03:FS01:0x17:Folding@home GPU core17
17:47:44:WU03:FS01:0x17:Version 0.0.52
17:51:40:WU03:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
17:51:40:WU03:FS01:0x17:Saving result file logfile_01.txt
17:51:40:WU03:FS01:0x17:Saving result file log.txt
17:51:40:WU03:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
17:51:40:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
17:51:40:WU03:FS01:Sending unit results: id:03 state:SEND error:FAULTY project:10469 run:0 clone:362 gen:27 core:0x17 unit:0x0000003a538b3db9538f41fb2858db41
17:51:40:WU03:FS01:Uploading 2.33KiB to 140.163.4.233
17:51:40:WU03:FS01:Connecting to 140.163.4.233:8080
17:51:40:WU03:FS01:Upload complete
17:51:40:WU03:FS01:Server responded WORK_ACK (400)
17:51:40:WU03:FS01:Cleaning up
17:51:40:WU01:FS01:Connecting to 171.67.108.201:80
17:51:41:WU01:FS01:Assigned to work server 140.163.4.233
17:51:41:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:GK107 [Quadro K600] from 140.163.4.233
17:51:41:WU01:FS01:Connecting to 140.163.4.233:8080
17:51:41:WU01:FS01:Downloading 4.93MiB
17:51:42:WU01:FS01:Download complete
17:51:42:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:10469 run:0 clone:8 gen:22 core:0x17 unit:0x00000028538b3db95388d23d0b334b80
17:51:42:WU01:FS01:Starting
17:51:42:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/ZZeppo/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe -dir 01 -suffix 01 -version 704 -lifeline 4732 -checkpoint 10 -gpu 1 -gpu-vendor nvidia
17:51:42:WU01:FS01:Started FahCore on PID 4892
17:51:42:WU01:FS01:Core PID:1812
17:51:42:WU01:FS01:FahCore 0x17 started
17:51:42:WU01:FS01:0x17:*********************** Log Started 2014-08-19T17:51:42Z ***********************
17:51:42:WU01:FS01:0x17:Project: 10469 (Run 0, Clone 8, Gen 22)
17:51:42:WU01:FS01:0x17:Unit: 0x00000028538b3db95388d23d0b334b80
17:51:42:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
17:51:42:WU01:FS01:0x17:Machine: 1
17:51:42:WU01:FS01:0x17:Reading tar file state.xml
17:51:43:WU01:FS01:0x17:Reading tar file system.xml
17:51:44:WU01:FS01:0x17:Reading tar file integrator.xml
17:51:44:WU01:FS01:0x17:Reading tar file core.xml
17:51:44:WU01:FS01:0x17:Digital signatures verified
17:51:44:WU01:FS01:0x17:Folding@home GPU core17
17:51:44:WU01:FS01:0x17:Version 0.0.52
17:53:40:WU02:FS02:Connecting to 171.67.108.201:80
17:53:41:WARNING:WU02:FS02:Failed to get assignment from '171.67.108.201:80': Empty work server assignment
17:53:41:WU02:FS02:Connecting to 171.64.65.160:80
17:53:41:WARNING:WU02:FS02:Failed to get assignment from '171.64.65.160:80': Empty work server assignment
17:53:41:ERROR:WU02:FS02:Exception: Could not get an assignment
17:54:36:WU00:FS00:0xa3:Completed 310000 out of 500000 steps  (62%)
17:55:40:WU01:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
17:55:40:WU01:FS01:0x17:Saving result file logfile_01.txt
17:55:40:WU01:FS01:0x17:Saving result file log.txt
17:55:40:WU01:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
17:55:40:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
17:55:40:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:10469 run:0 clone:8 gen:22 core:0x17 unit:0x00000028538b3db95388d23d0b334b80
17:55:40:WU01:FS01:Uploading 2.31KiB to 140.163.4.233
17:55:40:WU01:FS01:Connecting to 140.163.4.233:8080
17:55:41:WU01:FS01:Upload complete
17:55:41:WU01:FS01:Server responded WORK_ACK (400)
17:55:41:WU01:FS01:Cleaning up
17:55:41:WU03:FS01:Connecting to 171.67.108.201:80
17:55:41:WU03:FS01:Assigned to work server 140.163.4.233
17:55:41:WU03:FS01:Requesting new work unit for slot 01: READY gpu:1:GK107 [Quadro K600] from 140.163.4.233
17:55:41:WU03:FS01:Connecting to 140.163.4.233:8080
17:55:41:WU03:FS01:Downloading 4.26MiB
17:55:44:WU03:FS01:Download complete
17:55:44:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:10467 run:0 clone:38 gen:24 core:0x17 unit:0x00000034538b3db95388cf110290a50c
17:55:44:WU03:FS01:Starting
17:55:44:WU03:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/ZZeppo/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe -dir 03 -suffix 01 -version 704 -lifeline 4732 -checkpoint 10 -gpu 1 -gpu-vendor nvidia
17:55:44:WU03:FS01:Started FahCore on PID 2044
17:55:44:WU03:FS01:Core PID:2692
17:55:44:WU03:FS01:FahCore 0x17 started
17:55:44:WU03:FS01:0x17:*********************** Log Started 2014-08-19T17:55:44Z ***********************
17:55:44:WU03:FS01:0x17:Project: 10467 (Run 0, Clone 38, Gen 24)
17:55:44:WU03:FS01:0x17:Unit: 0x00000034538b3db95388cf110290a50c
17:55:44:WU03:FS01:0x17:CPU: 0x00000000000000000000000000000000
17:55:44:WU03:FS01:0x17:Machine: 1
17:55:44:WU03:FS01:0x17:Reading tar file state.xml
17:55:45:WU03:FS01:0x17:Reading tar file system.xml
17:55:46:WU03:FS01:0x17:Reading tar file integrator.xml
17:55:46:WU03:FS01:0x17:Reading tar file core.xml
17:55:46:WU03:FS01:0x17:Digital signatures verified
17:55:46:WU03:FS01:0x17:Folding@home GPU core17
17:55:46:WU03:FS01:0x17:Version 0.0.52
17:58:30:28:127.0.0.1 GET /api/updates?_=1408469176369&sid=26347e3b2d34e94cde21d2f3fb34399a
17:58:31:28:127.0.0.1 GET /api/updates?_=1408469176370&sid=26347e3b2d34e94cde21d2f3fb34399a
17:58:32:29:127.0.0.1 GET /api/updates?_=1408469176371&sid=26347e3b2d34e94cde21d2f3fb34399a
17:58:33:29:127.0.0.1 GET /api/updates?_=1408469176372&sid=26347e3b2d34e94cde21d2f3fb34399a
17:58:34:28:127.0.0.1 GET /api/updates?_=1408469176373&sid=26347e3b2d34e94cde21d2f3fb34399a
17:58:36:29:127.0.0.1 GET /api/updates?_=1408469176374&sid=26347e3b2d34e94cde21d2f3fb34399a
17:58:37:28:127.0.0.1 GET /api/updates?_=1408469176375&sid=26347e3b2d34e94cde21d2f3fb34399a
17:58:39:29:127.0.0.1 GET /api/updates?_=1408469176376&sid=26347e3b2d34e94cde21d2f3fb34399a
17:58:41:WU03:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
17:58:41:WU03:FS01:0x17:Saving result file logfile_01.txt
17:58:41:WU03:FS01:0x17:Saving result file log.txt
17:58:41:WU03:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
17:58:41:28:127.0.0.1 GET /api/updates?_=1408469176377&sid=26347e3b2d34e94cde21d2f3fb34399a
17:58:41:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
17:58:41:WU03:FS01:Sending unit results: id:03 state:SEND error:FAULTY project:10467 run:0 clone:38 gen:24 core:0x17 unit:0x00000034538b3db95388cf110290a50c
17:58:41:WU03:FS01:Uploading 2.33KiB to 140.163.4.233
17:58:41:WU03:FS01:Connecting to 140.163.4.233:8080
17:58:41:WU03:FS01:Upload complete
17:58:41:WU03:FS01:Server responded WORK_ACK (400)
17:58:41:WU03:FS01:Cleaning up
17:58:42:WU01:FS01:Connecting to 171.67.108.201:80
17:58:42:WU01:FS01:Assigned to work server 140.163.4.233
17:58:42:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:GK107 [Quadro K600] from 140.163.4.233
17:58:42:WU01:FS01:Connecting to 140.163.4.233:8080
17:58:42:WU01:FS01:Downloading 4.31MiB
17:58:43:28:127.0.0.1 GET /api/updates?_=1408469176378&sid=26347e3b2d34e94cde21d2f3fb34399a
17:58:43:WU01:FS01:Download complete
17:58:43:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:10468 run:0 clone:326 gen:21 core:0x17 unit:0x00000021538b3db9538cb2c7724fe8f8
17:58:43:WU01:FS01:Starting
17:58:43:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/ZZeppo/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe -dir 01 -suffix 01 -version 704 -lifeline 4732 -checkpoint 10 -gpu 1 -gpu-vendor nvidia
17:58:43:WU01:FS01:Started FahCore on PID 2648
17:58:43:WU01:FS01:Core PID:5032
17:58:43:WU01:FS01:FahCore 0x17 started
17:58:44:WU01:FS01:0x17:*********************** Log Started 2014-08-19T17:58:43Z ***********************
17:58:44:WU01:FS01:0x17:Project: 10468 (Run 0, Clone 326, Gen 21)
17:58:44:WU01:FS01:0x17:Unit: 0x00000021538b3db9538cb2c7724fe8f8
17:58:44:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
17:58:44:WU01:FS01:0x17:Machine: 1
17:58:44:WU01:FS01:0x17:Reading tar file state.xml
17:58:45:WU01:FS01:0x17:Reading tar file system.xml
17:58:45:28:127.0.0.1 GET /api/updates?_=1408469176379&sid=26347e3b2d34e94cde21d2f3fb34399a
17:58:46:WU01:FS01:0x17:Reading tar file integrator.xml
17:58:46:WU01:FS01:0x17:Reading tar file core.xml
17:58:46:WU01:FS01:0x17:Digital signatures verified
17:58:46:WU01:FS01:0x17:Folding@home GPU core17
17:58:46:WU01:FS01:0x17:Version 0.0.52
17:58:46:28:127.0.0.1 GET /api/updates?_=1408469176380&sid=26347e3b2d34e94cde21d2f3fb34399a
17:58:48:29:127.0.0.1 GET /api/updates?_=1408469176381&sid=26347e3b2d34e94cde21d2f3fb34399a
17:58:51:29:127.0.0.1 GET /api/updates?_=1408469176382&sid=26347e3b2d34e94cde21d2f3fb34399a
17:58:53:28:127.0.0.1 GET /api/updates?_=1408469176383&sid=26347e3b2d34e94cde21d2f3fb34399a
17:58:56:28:127.0.0.1 GET /api/updates?_=1408469176384&sid=26347e3b2d34e94cde21d2f3fb34399a
17:58:58:29:127.0.0.1 GET /api/updates?_=1408469176385&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:01:29:127.0.0.1 GET /api/updates?_=1408469176386&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:03:28:127.0.0.1 GET /api/updates?_=1408469176387&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:06:28:127.0.0.1 GET /api/updates?_=1408469176388&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:08:29:127.0.0.1 GET /api/updates?_=1408469176389&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:11:29:127.0.0.1 GET /api/updates?_=1408469176390&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:13:28:127.0.0.1 GET /api/updates?_=1408469176391&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:16:28:127.0.0.1 GET /api/updates?_=1408469176392&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:18:29:127.0.0.1 GET /api/updates?_=1408469176393&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:21:29:127.0.0.1 GET /api/updates?_=1408469176394&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:23:28:127.0.0.1 GET /api/updates?_=1408469176395&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:26:28:127.0.0.1 GET /api/updates?_=1408469176396&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:28:29:127.0.0.1 GET /api/updates?_=1408469176397&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:31:29:127.0.0.1 GET /api/updates?_=1408469176398&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:33:28:127.0.0.1 GET /api/updates?_=1408469176399&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:37:28:127.0.0.1 GET /api/updates?_=1408469176400&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:39:28:127.0.0.1 GET /api/updates?_=1408469176401&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:40:28:127.0.0.1 GET /api/updates?_=1408469176402&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:42:29:127.0.0.1 GET /api/updates?_=1408469176403&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:45:28:127.0.0.1 GET /api/updates?_=1408469176404&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:47:29:127.0.0.1 GET /api/updates?_=1408469176405&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:49:28:127.0.0.1 GET /api/updates?_=1408469176406&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:51:29:127.0.0.1 GET /api/updates?_=1408469176407&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:53:29:127.0.0.1 GET /api/updates?_=1408469176408&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:55:29:127.0.0.1 GET /api/updates?_=1408469176409&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:57:29:127.0.0.1 GET /api/updates?_=1408469176410&sid=26347e3b2d34e94cde21d2f3fb34399a
17:59:59:29:127.0.0.1 GET /api/updates?_=1408469176411&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:00:29:127.0.0.1 GET /api/updates?_=1408469176412&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:01:29:127.0.0.1 GET /api/updates?_=1408469176413&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:03:29:127.0.0.1 GET /api/updates?_=1408469176414&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:05:29:127.0.0.1 GET /api/updates?_=1408469176415&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:06:29:127.0.0.1 GET /api/updates?_=1408469176416&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:07:29:127.0.0.1 GET /api/updates?_=1408469176417&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:09:29:127.0.0.1 GET /api/updates?_=1408469176418&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:11:29:127.0.0.1 GET /api/updates?_=1408469176419&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:13:29:127.0.0.1 GET /api/updates?_=1408469176420&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:14:29:127.0.0.1 GET /api/updates?_=1408469176421&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:15:29:127.0.0.1 GET /api/updates?_=1408469176422&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:16:29:127.0.0.1 GET /api/updates?_=1408469176423&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:17:29:127.0.0.1 GET /api/updates?_=1408469176424&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:19:29:127.0.0.1 GET /api/updates?_=1408469176425&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:20:29:127.0.0.1 GET /api/updates?_=1408469176426&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:21:29:127.0.0.1 GET /api/updates?_=1408469176427&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:23:29:127.0.0.1 GET /api/updates?_=1408469176428&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:24:29:127.0.0.1 GET /api/updates?_=1408469176429&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:25:29:127.0.0.1 GET /api/updates?_=1408469176430&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:27:29:127.0.0.1 GET /api/updates?_=1408469176431&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:29:29:127.0.0.1 GET /api/updates?_=1408469176432&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:30:29:127.0.0.1 GET /api/updates?_=1408469176433&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:31:30:127.0.0.1 GET /api/updates?_=1408469176434&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:33:30:127.0.0.1 GET /api/updates?_=1408469176435&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:35:29:127.0.0.1 GET /api/updates?_=1408469176436&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:36:30:127.0.0.1 GET /api/updates?_=1408469176437&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:38:29:127.0.0.1 GET /api/updates?_=1408469176438&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:39:30:127.0.0.1 GET /api/updates?_=1408469176439&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:41:29:127.0.0.1 GET /api/updates?_=1408469176440&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:44:29:127.0.0.1 GET /api/updates?_=1408469176441&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:46:30:127.0.0.1 GET /api/updates?_=1408469176442&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:47:29:127.0.0.1 GET /api/updates?_=1408469176443&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:49:30:127.0.0.1 GET /api/updates?_=1408469176444&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:52:29:127.0.0.1 GET /api/updates?_=1408469176445&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:54:30:127.0.0.1 GET /api/updates?_=1408469176446&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:57:29:127.0.0.1 GET /api/updates?_=1408469176447&sid=26347e3b2d34e94cde21d2f3fb34399a
18:00:59:30:127.0.0.1 GET /api/updates?_=1408469176448&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:01:29:127.0.0.1 GET /api/updates?_=1408469176449&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:03:29:127.0.0.1 GET /api/updates?_=1408469176450&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:05:29:127.0.0.1 GET /api/updates?_=1408469176451&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:07:29:127.0.0.1 GET /api/updates?_=1408469176452&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:09:29:127.0.0.1 GET /api/updates?_=1408469176453&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:10:29:127.0.0.1 GET /api/updates?_=1408469176454&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:11:29:127.0.0.1 GET /api/updates?_=1408469176455&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:13:29:127.0.0.1 GET /api/updates?_=1408469176456&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:15:29:127.0.0.1 GET /api/updates?_=1408469176457&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:17:29:127.0.0.1 GET /api/updates?_=1408469176458&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:18:29:127.0.0.1 GET /api/updates?_=1408469176459&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:19:29:127.0.0.1 GET /api/updates?_=1408469176460&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:21:30:127.0.0.1 GET /api/updates?_=1408469176461&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:22:30:127.0.0.1 GET /api/updates?_=1408469176462&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:23:30:127.0.0.1 GET /api/updates?_=1408469176463&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:25:30:127.0.0.1 GET /api/updates?_=1408469176464&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:27:30:127.0.0.1 GET /api/updates?_=1408469176465&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:28:30:127.0.0.1 GET /api/updates?_=1408469176466&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:29:30:127.0.0.1 GET /api/updates?_=1408469176467&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:30:29:127.0.0.1 GET /api/updates?_=1408469176468&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:32:30:127.0.0.1 GET /api/updates?_=1408469176469&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:35:29:127.0.0.1 GET /api/updates?_=1408469176470&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:37:30:127.0.0.1 GET /api/updates?_=1408469176471&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:39:29:127.0.0.1 GET /api/updates?_=1408469176472&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:40:29:127.0.0.1 GET /api/updates?_=1408469176473&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:41:29:127.0.0.1 GET /api/updates?_=1408469176474&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:43:29:127.0.0.1 GET /api/updates?_=1408469176475&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:45:29:127.0.0.1 GET /api/updates?_=1408469176476&sid=26347e3b2d34e94cde21d2f3fb34399a
18:01:46:WU01:FS01:0x17:ERROR:exception: Error invoking kernel computeNonbonded: clEnqueueNDRangeKernel (-5)
18:01:46:WU01:FS01:0x17:Saving result file logfile_01.txt
18:01:46:WU01:FS01:0x17:Saving result file log.txt
18:01:46:WU01:FS01:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
18:01:47:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
18:01:47:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:10468 run:0 clone:326 gen:21 core:0x17 unit:0x00000021538b3db9538cb2c7724fe8f8
18:01:47:WU01:FS01:Uploading 2.33KiB to 140.163.4.233
18:01:47:WU01:FS01:Connecting to 140.163.4.233:8080
18:01:47:WU01:FS01:Upload complete
18:01:47:WU01:FS01:Server responded WORK_ACK (400)
18:01:47:WU01:FS01:Cleaning up
Drivers do support OpenCL:

Code: Select all

Release Highlights:
ODE Driver
This is the fourth release for the R331 drivers, the sixth ‘Optimal Drivers for Enterprise’ [ODE]. ODE branches are dedicated to relatively long term stability for ISV certification, OEMs, and Enterprise customers.

Existing Support:

Support for CUDA 5.5
Support for the Open Computing Language (OpenCL) 1.1*
Support for DirectX11

Is there a way to force these GPUs to pull any work? Oddly enough the K600 will randomly get work assignments, but they always fail out... Most recent was project 10467.

Thanks!
Image
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Constant WU Failures on Both GPUs

Post by bruce »

I can think of three possibilities.
1) The combination of your hardware and your driver version isn't capable of folding. If that's true, the Pande Group will figure out that they need to limit the assignments so they won't continue to fail.
2) If that has already happened to some projects, it will be increasingly difficult to get assignments.
3) If your failures are due to driver conflicts or hardware failures (including overclocking/overheating/low voltages,etc.), there may or may not be any good solutions but it will be trial-and-error.
Top
Nirvana
Posts: 21
Joined: Tue Aug 19, 2014 4:20 pm

Re: Constant WU Failures on Both GPUs

Post by Nirvana »

Hmm, I can try removing one of the GPUs later today. I'm not over-clocking anything so that rules out that and voltages. Maybe overheating...but highly unlikely, their temps average in the mid 50ºC range.

Hopefully this is something that can get resolved. I have this machine running almost 24/7 and would love to rack up some more points! :)


Thanks!
Image
Top
P5-133XL
Posts: 2948
Joined: Sun Dec 02, 2007 4:36 am
Hardware configuration: Machine #1:

Intel Q9450; 2x2GB=8GB Ram; Gigabyte GA-X48-DS4 Motherboard; PC Power and Cooling Q750 PS; 2x GTX 460; Windows Server 2008 X64 (SP1).

Machine #2:

Intel Q6600; 2x2GB=4GB Ram; Gigabyte GA-X48-DS4 Motherboard; PC Power and Cooling Q750 PS; 2x GTX 460 video card; Windows 7 X64.

Machine 3:

Dell Dimension 8400, 3.2GHz P4 4x512GB Ram, Video card GTX 460, Windows 7 X32

I am currently folding just on the 5x GTX 460's for aprox. 70K PPD
Location: Salem. OR USA

Re: Constant WU Failures on Both GPUs

Post by P5-133XL »

It would be nice if you reverted the verbosity setting back to the default of 3. It makes it much easier to read the logs.
Image
Top
Joe_H
Site Admin
Posts: 7937
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Constant WU Failures on Both GPUs

Post by Joe_H »

Yes, the release notes for the driver says OpenCL is supported, but it is an optional install in some nVidia installers. Please check to see if it was actually installed. I am not as familiar with the software versions for Quadro support, is that version the most recent one?

One way to test if the newer card can fold using Core_17 at all is to run FAHBench. That includes testing with actual WU calculations.

If power is okay you don't need to remove one of the cards, just disable that folding slot. Then use a utility to check and see which card is actually being used when a Core_17 WU is attempted. It is possible that the folding client has the GPU indices swapped and is associating the two folding slots with the wrong cards. There are directions available here on the forum for manually assigning the GPU index values to get the right card associated with each slot.

If you do change the hardware setup, it is best to uninstall the client including data and then reinstall. If you have an active CPU WU you can set it to Finish and do the reinstall after it is completed.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Top
7im
Posts: 10179
Joined: Thu Nov 29, 2007 4:30 pm
Hardware configuration: Intel i7-4770K @ 4.5 GHz, 16 GB DDR3-2133 Corsair Vengence (black/red), EVGA GTX 760 @ 1200 MHz, on an Asus Maximus VI Hero MB (black/red), in a blacked out Antec P280 Tower, with a Xigmatek Night Hawk (black) HSF, Seasonic 760w Platinum (black case, sleeves, wires), 4 SilenX 120mm Case fans with silicon fan gaskets and silicon mounts (all black), a 512GB Samsung SSD (black), and a 2TB Black Western Digital HD (silver/black).
Location: Arizona
Contact:
Contact 7im

Re: Constant WU Failures on Both GPUs

Post by 7im »

As a test, install the regular Geforce driver, not the Quadra driver. We know the Geforce driver works with folding, and it supports the K600.
How to provide enough information to get helpful support
Tell me and I forget. Teach me and I remember. Involve me and I learn.
Top
Nirvana
Posts: 21
Joined: Tue Aug 19, 2014 4:20 pm

Re: Constant WU Failures on Both GPUs

Post by Nirvana »

P5-133XL wrote:It would be nice if you reverted the verbosity setting back to the default of 3. It makes it much easier to read the logs.
Done
7im wrote:As a test, install the regular Geforce driver, not the Quadra driver. We know the Geforce driver works with folding, and it supports the K600.
I've been installing the "performance driver".
Joe_H wrote:Yes, the release notes for the driver says OpenCL is supported, but it is an optional install in some nVidia installers. Please check to see if it was actually installed. I am not as familiar with the software versions for Quadro support, is that version the most recent one?

One way to test if the newer card can fold using Core_17 at all is to run FAHBench. That includes testing with actual WU calculations.
Seems like the link for FAHBench is dead... I can't find a mirror for it either.

Thanks!
Image
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Constant WU Failures on Both GPUs

Post by bruce »

I'm trying to track down somebody who knows what's up with FAHBench.
Top
7im
Posts: 10179
Joined: Thu Nov 29, 2007 4:30 pm
Hardware configuration: Intel i7-4770K @ 4.5 GHz, 16 GB DDR3-2133 Corsair Vengence (black/red), EVGA GTX 760 @ 1200 MHz, on an Asus Maximus VI Hero MB (black/red), in a blacked out Antec P280 Tower, with a Xigmatek Night Hawk (black) HSF, Seasonic 760w Platinum (black case, sleeves, wires), 4 SilenX 120mm Case fans with silicon fan gaskets and silicon mounts (all black), a 512GB Samsung SSD (black), and a 2TB Black Western Digital HD (silver/black).
Location: Arizona
Contact:
Contact 7im

Re: Constant WU Failures on Both GPUs

Post by 7im »

The source of FAHBench is GitHub now: https://github.com/SimTk/fahbench
How to provide enough information to get helpful support
Tell me and I forget. Teach me and I remember. Involve me and I learn.
Top
Joe_H
Site Admin
Posts: 7937
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Constant WU Failures on Both GPUs

Post by Joe_H »

But as best as I can tell just the source code is there, you have to compile from that after also having a few other software tools present as well.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Top
Nirvana
Posts: 21
Joined: Tue Aug 19, 2014 4:20 pm

Re: Constant WU Failures on Both GPUs

Post by Nirvana »

Thanks for all the replies guys, I really appreciate it.

I did get things working after removing the old FX580 GPU. Looks like that was the problem all along. I'll try to add the 2nd GPU back in later on for a higher PPD, but if it doesn't work who cares.

I guess you can set this to solved or closed if you want.


Thanks again!
Image
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Constant WU Failures on Both GPUs

Post by bruce »

According to our information, the FX580 is supported, though most people download the standard drivers from NVidia. Can you extract the portions of the log that apply to that WU so we can try to find something definitive about what caused the problem.
Top
Nirvana
Posts: 21
Joined: Tue Aug 19, 2014 4:20 pm

Re: Constant WU Failures on Both GPUs

Post by Nirvana »

As its a Quadro card, the only drivers I could find on Nvidia's site were Quadro related. Even GeForce experience wouldn't grab drivers for it. Screenshot: http://gyazo.com/dfb98756926bc3a844c53aff9921736e
bruce wrote:Can you extract the portions of the log that apply to that WU so we can try to find something definitive about what caused the problem.
Sure, let me install the card again. I'll get back later today.
Image
Top
Post Reply

Return to “GPU Projects and FahCores”