GPU WU issues spreading to CPU WUs

[LonePalm]

Roger that. We'll see how it goes.

[Joe_H]

The first login in your post viewtopic.php?f=18&t=25248&view=unread#p252054 shows your client making its first connection to the AS and getting connected to a WS at 100% , just as you set it:

Code: Select all

6:58:46:WU00:FS00:0x17:Completed 2500000 out of 2500000 steps (100%)
06:58:47:WU02:FS00:Connecting to assign-GPU.stanford.edu:80
06:58:47:WU02:FS00:News: Welcome to Folding@Home
06:58:47:WU02:FS00:Assigned to work server 171.64.65.69
06:58:47:WU02:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:Tahiti XT [Radeon HD 7970] from 171.64.65.69
06:58:47:WU02:FS00:Connecting to 171.64.65.69:8080
06:58:51:ERROR:WU02:FS00:Exception: Server did not assign work unit
06:58:51:WU02:FS00:Connecting to assign-GPU.stanford.edu:80
06:58:51:WU02:FS00:News: Welcome to Folding@Home
06:58:51:WU02:FS00:Assigned to work server 171.64.65.69
06:58:51:WU02:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:Tahiti XT [Radeon HD 7970] from 171.64.65.69
06:58:51:WU02:FS00:Connecting to 171.64.65.69:8080
06:58:55:ERROR:WU02:FS00:Exception: Server did not assign work unit
06:59:06:WU00:FS00:0x17:Saving result file logfile_01.txt

For an unknown reason, your system was not assigned a WU for those couple hours. Checking the log for that server from the server status page, it appears to have had work and was assigning to others during that time period. Perhaps the server was out of WU's designated to run on your type of GPU. Possibly as a coincidence to the restart, you were assigned a WU from the same server afterwards:

Code: Select all

09:03:26:WU00:FS00:News: Welcome to Folding@Home
09:03:26:WU00:FS00:Assigned to work server 171.64.65.69
09:03:26:WU00:FS00:Requesting new work unit for slot 00: READY gpu:0:Tahiti XT [Radeon HD 7970] from 171.64.65.69
09:03:26:WU00:FS00:Connecting to 171.64.65.69:8080
09:03:27:WU00:FS00:Downloading 4.18MiB
09:03:29:WU00:FS00:Download complete
09:03:29:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:8900 run:547 clone:3 gen:9 core:0x17 unit:0x00000011028c126651a6a066c884d9ac

You may not have needed to reboot, we can not tell as we do not know why the prior connections did not get an assignment.

I suspect changing the next-unit-percentage setting may not make a difference. However in changing it do not remove it completely and save, change it to 99 and save. Removing the setting completely and saving appears to be one way to trigger the premature download bug mentioned earlier.

[LonePalm]

Joe,
If I don't reboot, I don't get a GPU WU.

[7im]

What is the longest you have waited before rebooting and not gotten a new WU?

[LonePalm]

Longest wait was six to Seven hours.

There was no WU this afternoon after I changed to "next-unit-percentage 99". I'll try "98"

I rebooted and immediately got a WU.

Code: Select all

22:11:34:WU01:FS00:0x17:Completed 2400000 out of 2500000 steps (96%)
22:16:04:WU01:FS00:0x17:Completed 2425000 out of 2500000 steps (97%)
22:17:13:WU00:FS01:0xa4:Completed 2150000 out of 2500000 steps  (86%)
22:20:14:WU01:FS00:0x17:Completed 2450000 out of 2500000 steps (98%)
22:20:15:WU02:FS00:Connecting to assign-GPU.stanford.edu:80
22:20:15:WU02:FS00:News: Welcome to Folding@Home
22:20:15:WU02:FS00:Assigned to work server 171.64.65.69
22:20:15:WU02:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:Tahiti XT [Radeon HD 7970] from 171.64.65.69
22:20:15:WU02:FS00:Connecting to 171.64.65.69:8080
22:20:19:ERROR:WU02:FS00:Exception: Server did not assign work unit
22:20:19:WU02:FS00:Connecting to assign-GPU.stanford.edu:80
22:20:19:WU02:FS00:News: Welcome to Folding@Home
22:20:19:WU02:FS00:Assigned to work server 171.64.65.69
22:20:19:WU02:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:Tahiti XT [Radeon HD 7970] from 171.64.65.69
22:20:19:WU02:FS00:Connecting to 171.64.65.69:8080
22:20:23:ERROR:WU02:FS00:Exception: Server did not assign work unit
22:21:19:WU02:FS00:Connecting to assign-GPU.stanford.edu:80
22:21:19:WU02:FS00:News: Welcome to Folding@Home
22:21:19:WU02:FS00:Assigned to work server 171.64.65.69
22:21:19:WU02:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:Tahiti XT [Radeon HD 7970] from 171.64.65.69
22:21:19:WU02:FS00:Connecting to 171.64.65.69:8080
22:21:23:ERROR:WU02:FS00:Exception: Server did not assign work unit
22:22:56:WU02:FS00:Connecting to assign-GPU.stanford.edu:80
22:22:57:WU02:FS00:News: Welcome to Folding@Home
22:22:57:WU02:FS00:Assigned to work server 171.64.65.69
22:22:57:WU02:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:Tahiti XT [Radeon HD 7970] from 171.64.65.69
22:22:57:WU02:FS00:Connecting to 171.64.65.69:8080
22:23:00:ERROR:WU02:FS00:Exception: Server did not assign work unit
22:24:43:WU01:FS00:0x17:Completed 2475000 out of 2500000 steps (99%)
22:25:34:WU02:FS00:Connecting to assign-GPU.stanford.edu:80
22:25:34:WU02:FS00:News: Welcome to Folding@Home
22:25:34:WU02:FS00:Assigned to work server 171.64.65.69
22:25:34:WU02:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:Tahiti XT [Radeon HD 7970] from 171.64.65.69
22:25:34:WU02:FS00:Connecting to 171.64.65.69:8080
22:25:38:ERROR:WU02:FS00:Exception: Server did not assign work unit
22:28:53:WU01:FS00:0x17:Completed 2500000 out of 2500000 steps (100%)
22:29:12:WU01:FS00:0x17:Saving result file logfile_01.txt
22:29:13:WU01:FS00:0x17:Saving result file checkpointState.xml
22:29:15:WU01:FS00:0x17:Saving result file checkpt.crc
22:29:15:WU01:FS00:0x17:Saving result file log.txt
22:29:15:WU01:FS00:0x17:Saving result file positions.xtc
22:29:18:WU01:FS00:0x17:Folding@home Core Shutdown: FINISHED_UNIT
22:29:18:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
22:29:19:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:8900 run:461 clone:5 gen:2 core:0x17 unit:0x00000006028c126651a68d4a21a4de34
22:29:19:WU01:FS00:Uploading 12.96MiB to 171.64.65.69
22:29:19:WU01:FS00:Connecting to 171.64.65.69:8080
22:29:25:WU01:FS00:Upload 15.43%
22:29:31:WU01:FS00:Upload 26.52%
22:29:37:WU01:FS00:Upload 41.47%
22:29:43:WU01:FS00:Upload 53.52%
22:29:48:WU02:FS00:Connecting to assign-GPU.stanford.edu:80
22:29:49:WU01:FS00:Upload 65.58%
22:29:49:WU02:FS00:News: Welcome to Folding@Home
22:29:49:WU02:FS00:Assigned to work server 171.64.65.69
22:29:49:WU02:FS00:Requesting new work unit for slot 00: READY gpu:0:Tahiti XT [Radeon HD 7970] from 171.64.65.69
22:29:49:WU02:FS00:Connecting to 171.64.65.69:8080
22:29:52:ERROR:WU02:FS00:Exception: Server did not assign work unit
22:29:55:WU01:FS00:Upload 77.15%
22:30:01:WU01:FS00:Upload 88.24%
22:30:15:WU01:FS00:Upload complete
22:30:15:WU01:FS00:Server responded WORK_ACK (400)
22:30:15:WU01:FS00:Final credit estimate, 27933.00 points
22:30:15:WU01:FS00:Cleaning up
22:34:07:WU00:FS01:0xa4:Completed 2175000 out of 2500000 steps  (87%)
22:36:39:WU02:FS00:Connecting to assign-GPU.stanford.edu:80
22:36:40:WU02:FS00:News: Welcome to Folding@Home
22:36:40:WU02:FS00:Assigned to work server 171.64.65.69
22:36:40:WU02:FS00:Requesting new work unit for slot 00: READY gpu:0:Tahiti XT [Radeon HD 7970] from 171.64.65.69
22:36:40:WU02:FS00:Connecting to 171.64.65.69:8080
22:36:43:ERROR:WU02:FS00:Exception: Server did not assign work unit
22:40:03:WARNING:WU00:FS01:FahCore crashed with Windows unhandled exception code 0xUNKNOWN_ENUM, searching for this code online may provide more information
22:40:03:WARNING:WU00:FS01:FahCore returned: UNKNOWN_ENUM (1073807364 = 0x40010004)
22:40:03:WU00:FS01:Starting
22:40:03:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 703 -lifeline 4836 -checkpoint 15 -np 7
22:40:03:WU00:FS01:Started FahCore on PID 9284

Code: Select all

*********************** Log Started 2013-11-17T22:43:21Z ***********************
22:43:21:************************* Folding@home Client *************************
22:43:21:      Website: http://folding.stanford.edu/
22:43:21:    Copyright: (c) 2009-2013 Stanford University
22:43:21:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:43:21:         Args: --open-web-control
22:43:21:       Config: C:/ProgramData/FAHClient/config.xml
22:43:21:******************************** Build ********************************
22:43:21:      Version: 7.3.6
22:43:21:         Date: Feb 18 2013
22:43:21:         Time: 15:25:17
22:43:21:      SVN Rev: 3923
22:43:21:       Branch: fah/trunk/client
22:43:21:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
22:43:21:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
22:43:21:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
22:43:21:     Platform: win32 XP
22:43:21:         Bits: 32
22:43:21:         Mode: Release
22:43:21:******************************* System ********************************
22:43:21:          CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
22:43:21:       CPU ID: GenuineIntel Family 6 Model 58 Stepping 9
22:43:21:         CPUs: 8
22:43:21:       Memory: 7.96GiB
22:43:21:  Free Memory: 6.60GiB
22:43:21:      Threads: WINDOWS_THREADS
22:43:21:  Has Battery: false
22:43:21:   On Battery: false
22:43:21:   UTC offset: -5
22:43:21:          PID: 4984
22:43:21:          CWD: C:/ProgramData/FAHClient
22:43:21:           OS: Windows 7 Professional
22:43:21:      OS Arch: AMD64
22:43:21:         GPUs: 1
22:43:21:        GPU 0: ATI:5 Tahiti XT [Radeon HD 7970]
22:43:21:         CUDA: Not detected
22:43:21:Win32 Service: false
22:43:21:***********************************************************************
22:43:21:<config>
22:43:21:  <!-- Folding Core -->
22:43:21:  <core-priority v='low'/>
22:43:21:
22:43:21:  <!-- Folding Slot Configuration -->
22:43:21:  <power v='full'/>
22:43:21:
22:43:21:  <!-- Network -->
22:43:21:  <proxy v=':8080'/>
22:43:21:
22:43:21:  <!-- User Information -->
22:43:21:  <passkey v='********************************'/>
22:43:21:  <team v='36120'/>
22:43:21:  <user v='LonePalm'/>
22:43:21:
22:43:21:  <!-- Folding Slots -->
22:43:21:  <slot id='0' type='GPU'>
22:43:21:    <next-unit-percentage v='99'/>
22:43:21:  </slot>
22:43:21:  <slot id='1' type='CPU'>
22:43:21:    <cpus v='-1'/>
22:43:21:    <next-unit-percentage v='99'/>
22:43:21:  </slot>
22:43:21:</config>
22:43:21:Trying to access database...
22:43:23:Successfully acquired database lock
22:43:23:Enabled folding slot 00: READY gpu:0:Tahiti XT [Radeon HD 7970]
22:43:23:Enabled folding slot 01: READY cpu:7
22:43:27:WU00:FS01:Starting
22:43:27:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 703 -lifeline 4984 -checkpoint 15 -np 7
22:43:27:WU00:FS01:Started FahCore on PID 5704
22:43:28:WU00:FS01:Core PID:5776
22:43:28:WU00:FS01:FahCore 0xa4 started
22:43:29:WU00:FS01:0xa4:
22:43:29:WU00:FS01:0xa4:*------------------------------*
22:43:29:WU00:FS01:0xa4:Folding@Home Gromacs GB Core
22:43:29:WU00:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
22:43:29:WU00:FS01:0xa4:
22:43:29:WU00:FS01:0xa4:Preparing to commence simulation
22:43:29:WU00:FS01:0xa4:- Ensuring status. Please wait.
22:43:29:WU01:FS00:Connecting to assign-GPU.stanford.edu:80
22:43:30:WU01:FS00:News: Welcome to Folding@Home
22:43:30:WU01:FS00:Assigned to work server 171.64.65.69
22:43:30:WU01:FS00:Requesting new work unit for slot 00: READY gpu:0:Tahiti XT [Radeon HD 7970] from 171.64.65.69
22:43:30:WU01:FS00:Connecting to 171.64.65.69:8080
22:43:31:WU01:FS00:Downloading 4.18MiB
22:43:34:WU01:FS00:Download complete
22:43:34:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:8900 run:620 clone:0 gen:230 core:0x17 unit:0x0000011a028c126651a6b085d91bf1f2
22:43:34:WU01:FS00:Starting
22:43:34:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/ATI/R600/Core_17.fah/FahCore_17.exe -dir 01 -suffix 01 -version 703 -lifeline 4984 -checkpoint 15 -gpu 0 -gpu-vendor ati
22:43:34:WU01:FS00:Started FahCore on PID 6112
22:43:34:WU01:FS00:Core PID:6124
22:43:34:WU01:FS00:FahCore 0x17 started
22:43:36:WU01:FS00:0x17:*********************** Log Started 2013-11-17T22:43:36Z ***********************
22:43:36:WU01:FS00:0x17:Project: 8900 (Run 620, Clone 0, Gen 230)
22:43:36:WU01:FS00:0x17:Unit: 0x0000011a028c126651a6b085d91bf1f2
22:43:36:WU01:FS00:0x17:CPU: 0x00000000000000000000000000000000
22:43:36:WU01:FS00:0x17:Machine: 0
22:43:36:WU01:FS00:0x17:Reading tar file state.xml
22:43:37:WU01:FS00:0x17:Reading tar file system.xml
22:43:37:WU01:FS00:0x17:Reading tar file integrator.xml
22:43:37:WU01:FS00:0x17:Reading tar file core.xml
22:43:37:WU01:FS00:0x17:Digital signatures verified
22:43:38:WU00:FS01:0xa4:- Looking at optimizations...
22:43:38:WU00:FS01:0xa4:- Working with standard loops on this execution.
22:43:38:WU00:FS01:0xa4:- Previous termination of core was improper.
22:43:38:WU00:FS01:0xa4:- Going to use standard loops.
22:43:38:WU00:FS01:0xa4:- Files status OK
22:43:39:WU00:FS01:0xa4:- Expanded 546999 -> 846444 (decompressed 154.7 percent)
22:43:39:WU00:FS01:0xa4:Called DecompressByteArray: compressed_data_size=546999 data_size=846444, decompressed_data_size=846444 diff=0
22:43:39:WU00:FS01:0xa4:- Digital signature verified
22:43:39:WU00:FS01:0xa4:
22:43:39:WU00:FS01:0xa4:Project: 7647 (Run 170, Clone 0, Gen 156)
22:43:39:WU00:FS01:0xa4:
22:43:39:WU00:FS01:0xa4:Entering M.D.
22:43:45:WU00:FS01:0xa4:Using Gromacs checkpoints
22:43:45:WU00:FS01:0xa4:Mapping NT from 7 to 7 
22:43:51:WU00:FS01:0xa4:Resuming from checkpoint
22:43:51:WU00:FS01:0xa4:Verified 00/wudata_01.log
22:43:59:WU00:FS01:0xa4:Verified 00/wudata_01.trr
22:43:59:WU00:FS01:0xa4:Verified 00/wudata_01.xtc
22:43:59:WU00:FS01:0xa4:Verified 00/wudata_01.edr
22:44:00:WU00:FS01:0xa4:Completed 2164760 out of 2500000 steps  (86%)
22:46:49:WU01:FS00:0x17:Completed 0 out of 2500000 steps (0%)
22:51:27:WU01:FS00:0x17:Completed 25000 out of 2500000 steps (1%)
22:52:08:WU00:FS01:0xa4:Completed 2175000 out of 2500000 steps  (87%)
22:55:36:WU01:FS00:0x17:Completed 50000 out of 2500000 steps (2%)

[PantherX]

In Advanced Control (AKA FAHControl), under the Work Panel on the Right side, what is being displayed for the following when the GPU Slot hangs-up:
Waiting On:
Attempts:
Next Attempt:

Have you tried a fresh installation of V7.3.6? If not, I would suggest that you do it but chose "Only Me" during the installation. With a fresh installation, we can eliminate the possibility that your CPUId is causing an issue since you will be assigned a new one for the fresh installation.

[LonePalm]

All right. I'll give that a shot tomorrow.

BTW - I am the only person using this computer.

[bruce]

Have you had any communications failures downloading? There's a known bug in V7.3.6 whereby after a communications error, the client does not recover without stopping and restarting the ciient. See https://fah-web.stanford.edu/projects/F ... ticket/983.

Note: Not being assigned a WU is not a communications error, but unplugging the ethernet cable or resetting the router DURING a download is and it would (semi-)permanently disable further downloading.

[LonePalm]

Bruce,
No communications failures. Since Comcast replaced a section of fiber about 8 months ago we have not had a loss of service despite some very energetic weather in the mean time.

Everything I have stays screwed down. I built my first computer in 1982 and became compulsive about clean connections and things staying connected. Cats and small children will pull on anything.

All,
I did get a GPU WU at the 98% mark about 0600Z today. It failed to get one at 1320Z but did get one after a reboot. This gets stranger and stranger.

I will try a clean install in a few hours.

[LonePalm]

OK,
I did a fresh install of V7.3.6 yesterday afternoon. I installed FAH to start automatically and for me only. The GPU got a WU on its own about 2325Z but failed to get a WU today at about 0645Z. I rebooted the computer at about 1100Z and went back to bed.

Since the reboot of 0645Z I have not been able to see my folding via the Advanced Control. It only shows an orange 'connecting' without actually making a connection. I can see that folding is taking place via Windows Task Manager.

I have tried another fresh install to no avail. Whenever I use the Web Access to view the folding my user information is GONE. I reenter my name, team, and passkey and save.

Should the Advanced Control port be set to 36330 or 8080? I have tried both without success.

I have no idea what could have changed to cause this.

[bruce]

The default port for Advanced Control (aka FAHControl) to talk to FAHClient is 36330. The port used by Web Control (aka WEBControl) to talk to FAHClient is 7396.
For uploading and downloading, the only ports that Stanford servers listen to are 8080 and 80.

Your Firewall software might block any of these connections.

If you change 36330, you must do it two places and must restart the client. Changing the default is almost never useful.

[LonePalm]

I tried turning off my firewall. No dice.

Right now everything is a plain vanilla, clean installation of 7.3.6.

I only get an orange 'connecting' that never connects with anything.

[Joe_H]

Is there a FAHClient process shown as running? Altogether there should be FAHCore_nn, FAHCore_wrapper and FAHClient processes running. Without a FAHClient process there is nothing for FAHControl or the WebControl to connect to.

[LonePalm]

Yes, I can see both the CPU and GPU clients running in Windows task manager.

Before when one or both weren't running it would still connect and show as 'Ready'.

[Joe_H]

Yes, I can see both the CPU and GPU clients running in Windows task manager.

Before when one or both weren't running it would still connect and show as 'Ready'.

Specifically can you see a FAHClient process in Task Manager? There should be just one of that process running. The CPU and GPU cores are separate processes and should have an associated Core wrapper process for each.