Page 1 of 1
7808 (2,180,57) - ETEA 11.30 days!
Posted: Fri Mar 01, 2013 5:01 am
by telecentricity
Hi,
I'm currently running PRCG 7808 (2, 180, 57) on a quad core i7-2600K. I am getting TPF of 4h 08 min and an ETA of 11.30 days with PPD of 141. This is way out of line from my other runs (usually ETA 8-12h, TPF of 3-4min and PPD of 6k to 8k). I've tried pausing and also rebooting my system to no effect. My GPU folding slot is running fine at a reasonable speed. I read about another 7808 that had a bad WU and I wonder if this is a similar problem. I'm attaching my log and configuration (since my reboot). Thanks very much for your help,
Gary
Code: Select all
*********************** Log Started 2013-03-01T04:47:23Z ***********************
04:47:23:************************* Foldinghome Client *************************
04:47:23: Website: http://folding.stanford.edu/
04:47:23: Copyright: (c) 2009-2013 Stanford University
04:47:23: Author: Joseph Coffland <josephcauldrondevelopment.com>
04:47:23: Args:
04:47:23: Config: C:/ProgramData/FAHClient/config.xml
04:47:23:******************************** Build ********************************
04:47:23: Version: 7.3.6
04:47:23: Date: Feb 18 2013
04:47:23: Time: 15:25:17
04:47:23: SVN Rev: 3923
04:47:23: Branch: fah/trunk/client
04:47:23: Compiler: Intel(R) C++ MSVC 1500 mode 1200
04:47:23: Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
04:47:23: /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
04:47:23: Platform: win32 XP
04:47:23: Bits: 32
04:47:23: Mode: Release
04:47:23:******************************* System ********************************
04:47:23: CPU: Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
04:47:23: CPU ID: GenuineIntel Family 6 Model 42 Stepping 7
04:47:23: CPUs: 8
04:47:23: Memory: 31.90GiB
04:47:23: Free Memory: 28.48GiB
04:47:23: Threads: WINDOWS_THREADS
04:47:23: Has Battery: false
04:47:23: On Battery: false
04:47:23: UTC offset: -6
04:47:23: PID: 6592
04:47:23: CWD: C:/ProgramData/FAHClient
04:47:23: OS: Windows 8 Enterprise
04:47:23: OS Arch: AMD64
04:47:23: GPUs: 1
04:47:23: GPU 0: NVIDIA:2 GF110 [GeForce GTX 570 HD]
04:47:23: CUDA: 2.0
04:47:23: CUDA Driver: 5000
04:47:23:Win32 Service: false
04:47:23:***********************************************************************
04:47:23:<config>
04:47:23: <!-- Folding Slot Configuration -->
04:47:23: <power v='full'/>
04:47:23:
04:47:23: <!-- Network -->
04:47:23: <proxy v=':8080'/>
04:47:23:
04:47:23: <!-- User Information -->
04:47:23: <passkey v='********************************'/>
04:47:23: <team v='40051'/>
04:47:23: <user v='telecentricity'/>
04:47:23:
04:47:23: <!-- Folding Slots -->
04:47:23: <slot id='0' type='GPU'/>
04:47:23: <slot id='1' type='CPU'/>
04:47:23:</config>
04:47:23:Trying to access database...
*********************** Log Started 2013-03-01T04:47:23Z ***********************
04:47:23:************************* Foldinghome Client *************************
04:47:23: Website: http://folding.stanford.edu/
04:47:23: Copyright: (c) 2009-2013 Stanford University
04:47:23: Author: Joseph Coffland <josephcauldrondevelopment.com>
04:47:23: Args:
04:47:23: Config: C:/ProgramData/FAHClient/config.xml
04:47:23:******************************** Build ********************************
04:47:23: Version: 7.3.6
04:47:23: Date: Feb 18 2013
04:47:23: Time: 15:25:17
04:47:23: SVN Rev: 3923
04:47:23: Branch: fah/trunk/client
04:47:23: Compiler: Intel(R) C++ MSVC 1500 mode 1200
04:47:23: Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
04:47:23: /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
04:47:23: Platform: win32 XP
04:47:23: Bits: 32
04:47:23: Mode: Release
04:47:23:******************************* System ********************************
04:47:23: CPU: Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
04:47:23: CPU ID: GenuineIntel Family 6 Model 42 Stepping 7
04:47:23: CPUs: 8
04:47:23: Memory: 31.90GiB
04:47:23: Free Memory: 28.48GiB
04:47:23: Threads: WINDOWS_THREADS
04:47:23: Has Battery: false
04:47:23: On Battery: false
04:47:23: UTC offset: -6
04:47:23: PID: 6592
04:47:23: CWD: C:/ProgramData/FAHClient
04:47:23: OS: Windows 8 Enterprise
04:47:23: OS Arch: AMD64
04:47:23: GPUs: 1
04:47:23: GPU 0: NVIDIA:2 GF110 [GeForce GTX 570 HD]
04:47:23: CUDA: 2.0
04:47:23: CUDA Driver: 5000
04:47:23:Win32 Service: false
04:47:23:***********************************************************************
04:47:23:<config>
04:47:23: <!-- Folding Slot Configuration -->
04:47:23: <power v='full'/>
04:47:23:
04:47:23: <!-- Network -->
04:47:23: <proxy v=':8080'/>
04:47:23:
04:47:23: <!-- User Information -->
04:47:23: <passkey v='********************************'/>
04:47:23: <team v='40051'/>
04:47:23: <user v='telecentricity'/>
04:47:23:
04:47:23: <!-- Folding Slots -->
04:47:23: <slot id='0' type='GPU'/>
04:47:23: <slot id='1' type='CPU'/>
04:47:23:</config>
04:47:23:Trying to access database...
04:47:23:Successfully acquired database lock
04:47:23:Enabled folding slot 00: READY gpu:0:GF110 [GeForce GTX 570 HD]
04:47:23:Enabled folding slot 01: READY cpu:8
04:47:23:WU01:FS01:Starting
04:47:23:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 01 -suffix 01 -version 703 -lifeline 6592 -checkpoint 15 -np 8
04:47:23:WU01:FS01:Started FahCore on PID 8188
04:47:23:WU01:FS01:Core PID:5956
04:47:23:WU01:FS01:FahCore 0xa4 started
04:47:23:WU00:FS00:Starting
04:47:23:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 00 -suffix 01 -version 703 -lifeline 6592 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
04:47:23:WU00:FS00:Started FahCore on PID 6572
04:47:23:WU00:FS00:Core PID:6368
04:47:23:WU00:FS00:FahCore 0x15 started
04:47:24:WU01:FS01:0xa4:
04:47:24:WU01:FS01:0xa4:*------------------------------*
04:47:24:WU01:FS01:0xa4:Folding@Home Gromacs GB Core
04:47:24:WU01:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
04:47:24:WU01:FS01:0xa4:
04:47:24:WU01:FS01:0xa4:Preparing to commence simulation
04:47:24:WU01:FS01:0xa4:- Ensuring status. Please wait.
04:47:24:WU00:FS00:0x15:
04:47:24:WU00:FS00:0x15:*------------------------------*
04:47:24:WU00:FS00:0x15:Folding@Home GPU Core
04:47:24:WU00:FS00:0x15:Version 2.25 (Wed May 9 17:03:01 EDT 2012)
04:47:24:WU00:FS00:0x15:Build host AmoebaRemote
04:47:24:WU00:FS00:0x15:Board Type NVIDIA/CUDA
04:47:24:WU00:FS00:0x15:Core 15
04:47:24:WU00:FS00:0x15:
04:47:24:WU00:FS00:0x15:Window's signal control handler registered.
04:47:24:WU00:FS00:0x15:Preparing to commence simulation
04:47:24:WU00:FS00:0x15:- Ensuring status. Please wait.
04:47:33:WU01:FS01:0xa4:- Looking at optimizations...
04:47:33:WU01:FS01:0xa4:- Working with standard loops on this execution.
04:47:33:WU01:FS01:0xa4:- Previous termination of core was improper.
04:47:33:WU01:FS01:0xa4:- Files status OK
04:47:33:WU00:FS00:0x15:- Looking at optimizations...
04:47:33:WU00:FS00:0x15:- Working with standard loops on this execution.
04:47:33:WU00:FS00:0x15:- Previous termination of core was improper.
04:47:33:WU00:FS00:0x15:- Files status OK
04:47:33:WU00:FS00:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
04:47:33:WU00:FS00:0x15:- Expanded 58812 -> 259314 (decompressed 440.9 percent)
04:47:33:WU00:FS00:0x15:Called DecompressByteArray: compressed_data_size=58812 data_size=259314, decompressed_data_size=259314 diff=0
04:47:33:WU00:FS00:0x15:- Digital signature verified
04:47:33:WU00:FS00:0x15:
04:47:33:WU00:FS00:0x15:Project: 8071 (Run 118, Clone 11, Gen 4)
04:47:33:WU00:FS00:0x15:
04:47:33:WU00:FS00:0x15:Entering M.D.
04:47:33:WU01:FS01:0xa4:- Expanded 2054096 -> 5365960 (decompressed 261.2 percent)
04:47:33:WU01:FS01:0xa4:Called DecompressByteArray: compressed_data_size=2054096 data_size=5365960, decompressed_data_size=5365960 diff=0
04:47:33:WU01:FS01:0xa4:- Digital signature verified
04:47:33:WU01:FS01:0xa4:
04:47:33:WU01:FS01:0xa4:Project: 7808 (Run 2, Clone 180, Gen 57)
04:47:33:WU01:FS01:0xa4:
04:47:33:WU01:FS01:0xa4:Entering M.D.
04:47:35:WU00:FS00:0x15:Will resume from checkpoint file 00/wudata_01.ckp
04:47:35:WU00:FS00:0x15:Tpr hash 00/wudata_01.tpr: 4137678891 2023613714 3735999060 4019548950 179042560
04:47:35:WU00:FS00:0x15:GPU device id=0
04:47:35:WU00:FS00:0x15:Working on Glycine aRginine prOline Methionine Alanine Cystine Serine t= 118.00000
04:47:35:WU00:FS00:0x15:Client config unavailable.
04:47:35:WU00:FS00:0x15:Starting GUI Server
04:47:39:WU01:FS01:0xa4:Using Gromacs checkpoints
04:47:39:WU01:FS01:0xa4:Mapping NT from 8 to 8
04:47:40:WU01:FS01:0xa4:Resuming from checkpoint
04:47:40:WU01:FS01:0xa4:Verified 01/wudata_01.log
04:47:40:WU01:FS01:0xa4:Verified 01/wudata_01.trr
04:47:40:WU01:FS01:0xa4:Verified 01/wudata_01.xtc
04:47:40:WU01:FS01:0xa4:Verified 01/wudata_01.edr
04:47:40:WU01:FS01:0xa4:Completed 518840 out of 1500000 steps (34%)
04:47:43:3:127.0.0.1:New Web connection
04:48:37:WARNING:WU00:FS00:Detected clock skew (1 mins 04 secs), adjusting time estimates
04:48:37:WU00:FS00:0x15:Resuming from checkpoint
04:48:37:WU00:FS00:0x15:fcCheckPointResume: retreived and current tpr file hash:
04:48:37:WU00:FS00:0x15: 0 4137678891 4137678891
04:48:37:WU00:FS00:0x15: 1 2023613714 2023613714
04:48:37:WU00:FS00:0x15: 2 3735999060 3735999060
04:48:37:WU00:FS00:0x15: 3 4019548950 4019548950
04:48:37:WU00:FS00:0x15: 4 179042560 179042560
04:48:37:WU00:FS00:0x15:fcCheckPointResume: file hashes same.
04:48:37:WU00:FS00:0x15:fcCheckPointResume: state restored.
04:48:37:WU00:FS00:0x15:fcCheckPointResume: name 00/wudata_01.log Verified 00/wudata_01.log
04:48:37:WU00:FS00:0x15:fcCheckPointResume: name 00/wudata_01.trr Verified 00/wudata_01.trr
04:48:37:WU00:FS00:0x15:fcCheckPointResume: name 00/wudata_01.xtc Verified 00/wudata_01.xtc
04:48:37:WU00:FS00:0x15:fcCheckPointResume: name 00/wudata_01.edr Verified 00/wudata_01.edr
04:48:37:WU00:FS00:0x15:fcCheckPointResume: state restored 2
04:48:37:WU00:FS00:0x15:Resumed from checkpoint
04:48:37:WU00:FS00:0x15:Setting checkpoint frequency: 500000
04:48:37:WU00:FS00:0x15:Completed 12000001 out of 50000000 steps (24%).
04:51:10:WU00:FS00:0x15:Completed 12500000 out of 50000000 steps (25%).
04:53:44:WU00:FS00:0x15:Completed 13000000 out of 50000000 steps (26%).
Re: 7808 (2,180,57) - ETEA 11.30 days!
Posted: Fri Mar 01, 2013 5:30 am
by bruce
The ETA doesn't match what I see in that log. Give it some time to make a reasonable estimation.
I see that you progressed from 25% to 26% in 2m34s. If that's a typical rate of progress, you can complete 100% in 4h16m40s. You've already completed 26% so it should be completed about the time you expect it to be completed. GPU Project 8071 has a preferred deadline of 10 days so 4 hours is really very fast.
Project 7808 has a preferred deadline of 25 days, so 11 days is not unreasonable for a 4c8t CPU.
Re: 7808 (2,180,57) - ETA 11.30 days!
Posted: Fri Mar 01, 2013 6:45 am
by Napoleon
Welcome to the forum, telecentricity.
The 2m34s is for the GPU slot. One thing I've noticed about P807x GPU WUs is that they're quite CPU intensive in real life, no matter what e.g Task Manager might say - seriously unbalancing SMP work. Dropping back to CPU:6 should do the trick for the A4 WU (P7808). CPU:7 may fail for some projects/WUs, yet many people use it nevertheless. I'd recommend CPU:6, but ultimately it's your call.
Re: 7808 (2,180,57) - ETEA 11.30 days!
Posted: Fri Mar 01, 2013 11:37 am
by mmonnin
This is the first time this project has reported 4-5 hour TPFs...
Re: 7808 (2,180,57) - ETEA 11.30 days!
Posted: Fri Mar 01, 2013 12:20 pm
by Qinsp
I've the Status page estimated TPF off by huge amounts.
Is the TPF you are seeing in the log or estimate?
Re: 7808 (2,180,57) - ETEA 11.30 days!
Posted: Fri Mar 01, 2013 12:42 pm
by telecentricity
Thanks for the fast replies. I dropped the CPU from 8 down to 6 per Napoleon. Now I'm getting TPF of 9:22 with ETA of 9h38m. Wow! Not sure why that worked but it did. I think this was real - it had stumbled along at TPF 4h for a couple of days before I rebooted. There is another thread below by ptschnack which reproduces the problem. Thanks for the help,
Gary
Re: 7808 (2,180,57) - ETEA 11.30 days!
Posted: Fri Mar 01, 2013 8:04 pm
by bruce
Note that when you use all of your CPUs (in this case, 8) any interruptions from other applications will cause a major loss of performance since SMP runs at the speed of the slowest thread. By changing the setting to 6, you've left two virtual cores available for other things that might be running on your system. Check performance monitor. You should see the FahCore using a consistent total of 6/8 = 75% of the CPU time. You should also see essentially 0% for other applications. If you're seeing a few percent consistently going to anything else that's running, that's probably the complete explanation of the problem. SMP can never perform well unless all of its threads are balanced (aka "symmetric").
When you restart the client, the previous log is backed up to the subdirectory "logs" in the FAH data directory. Please post a segment section of one of them showing the 4h TPF. (Unfortunately there is no evidence in the log if the previous paragraph applies to your system.) The log posted above shows the project restarting at (34%) [Actually 34.72%] but does not show additional messages from which we can calculate a TPF.
Re: 7808 (2,180,57) - ETEA 11.30 days!
Posted: Sat Mar 02, 2013 2:37 am
by telecentricity
Bruce,
Thanks very much for your explanation and your patience. Below is the first of two logs prior to the one I submitted, starting with the download. I hope they help you analyze the problem.
Kind regards,
Gary
Code: Select all
*********************** Log Started 2013-02-28T00:40:16Z ***********************
00:40:16:************************* Folding@home Client *************************
00:40:16: Website: http://folding.stanford.edu/
00:40:16: Copyright: (c) 2009-2013 Stanford University
00:40:16: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
00:40:16: Args:
00:40:16: Config: C:/ProgramData/FAHClient/config.xml
00:40:16:******************************** Build ********************************
00:40:16: Version: 7.3.6
00:40:16: Date: Feb 18 2013
00:40:16: Time: 15:25:17
00:40:16: SVN Rev: 3923
00:40:16: Branch: fah/trunk/client
00:40:16: Compiler: Intel(R) C++ MSVC 1500 mode 1200
00:40:16: Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
00:40:16: /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
00:40:16: Platform: win32 XP
00:40:16: Bits: 32
00:40:16: Mode: Release
00:40:16:******************************* System ********************************
00:40:16: CPU: Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
00:40:16: CPU ID: GenuineIntel Family 6 Model 42 Stepping 7
00:40:16: CPUs: 8
00:40:16: Memory: 31.90GiB
00:40:16: Free Memory: 28.52GiB
00:40:16: Threads: WINDOWS_THREADS
00:40:16: Has Battery: false
00:40:16: On Battery: false
00:40:16: UTC offset: -6
00:40:16: PID: 8388
00:40:16: CWD: C:/ProgramData/FAHClient
00:40:16: OS: Windows 8 Enterprise
00:40:16: OS Arch: AMD64
00:40:16: GPUs: 1
00:40:16: GPU 0: NVIDIA:2 GF110 [GeForce GTX 570 HD]
00:40:16: CUDA: 2.0
00:40:16: CUDA Driver: 5000
00:40:16:Win32 Service: false
00:40:16:***********************************************************************
00:40:16:<config>
00:40:16: <!-- Folding Slot Configuration -->
00:40:16: <power v='full'/>
00:40:16:
00:40:16: <!-- Network -->
00:40:16: <proxy v=':8080'/>
00:40:16:
00:40:16: <!-- User Information -->
00:40:16: <passkey v='********************************'/>
00:40:16: <team v='40051'/>
00:40:16: <user v='telecentricity'/>
00:40:16:
00:40:16: <!-- Folding Slots -->
00:40:16: <slot id='0' type='GPU'/>
00:40:16: <slot id='1' type='CPU'/>
00:40:16:</config>
00:40:16:Trying to access database...
00:40:16:Successfully acquired database lock
00:40:16:Enabled folding slot 00: READY gpu:0:GF110 [GeForce GTX 570 HD]
00:40:16:Enabled folding slot 01: READY cpu:8
00:40:16:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:7624 run:335 clone:3 gen:40 core:0x15 unit:0x00000030664f2dd14fe61276ae9eebf5
00:40:16:WU00:FS00:Uploading 807.72KiB to 171.64.65.105
00:40:16:WU01:FS01:Starting
00:40:16:WU00:FS00:Connecting to 171.64.65.105:8080
00:40:16:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 01 -suffix 01 -version 703 -lifeline 8388 -checkpoint 15 -np 8
00:40:16:WU01:FS01:Started FahCore on PID 9300
00:40:16:WU01:FS01:Core PID:9312
00:40:16:WU01:FS01:FahCore 0xa4 started
00:40:16:WU02:FS00:Connecting to assign-GPU.stanford.edu:80
00:40:16:WU01:FS01:0xa4:
00:40:16:WU01:FS01:0xa4:*------------------------------*
00:40:16:WU01:FS01:0xa4:Folding@Home Gromacs GB Core
00:40:16:WU01:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
00:40:16:WU01:FS01:0xa4:
00:40:16:WU01:FS01:0xa4:Preparing to commence simulation
00:40:16:WU01:FS01:0xa4:- Ensuring status. Please wait.
00:40:17:WU02:FS00:News: Welcome to Folding@Home
00:40:17:WU02:FS00:Assigned to work server 171.64.65.105
00:40:17:WU02:FS00:Requesting new work unit for slot 00: READY gpu:0:GF110 [GeForce GTX 570 HD] from 171.64.65.105
00:40:17:WU02:FS00:Connecting to 171.64.65.105:8080
00:40:17:WU00:FS00:Upload complete
00:40:17:WU00:FS00:Server responded WORK_ACK (400)
00:40:17:WU00:FS00:Final credit estimate, 14093.00 points
00:40:17:WU00:FS00:Cleaning up
00:40:18:WU02:FS00:Downloading 122.13KiB
00:40:18:WU02:FS00:Download complete
00:40:18:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:7625 run:709 clone:1 gen:9 core:0x15 unit:0x0000000c664f2dd14fe619371e442d39
00:40:18:WU02:FS00:Starting
00:40:18:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 02 -suffix 01 -version 703 -lifeline 8388 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
00:40:18:WU02:FS00:Started FahCore on PID 9464
00:40:18:WU02:FS00:Core PID:9476
00:40:18:WU02:FS00:FahCore 0x15 started
00:40:19:WU02:FS00:0x15:
00:40:19:WU02:FS00:0x15:*------------------------------*
00:40:19:WU02:FS00:0x15:Folding@Home GPU Core
00:40:19:WU02:FS00:0x15:Version 2.25 (Wed May 9 17:03:01 EDT 2012)
00:40:19:WU02:FS00:0x15:Build host AmoebaRemote
00:40:19:WU02:FS00:0x15:Board Type NVIDIA/CUDA
00:40:19:WU02:FS00:0x15:Core 15
00:40:19:WU02:FS00:0x15:
00:40:19:WU02:FS00:0x15:Window's signal control handler registered.
00:40:19:WU02:FS00:0x15:Preparing to commence simulation
00:40:19:WU02:FS00:0x15:- Looking at optimizations...
00:40:19:WU02:FS00:0x15:DeleteFrameFiles: successfully deleted file=02/wudata_01.ckp
00:40:19:WU02:FS00:0x15:- Created dyn
00:40:19:WU02:FS00:0x15:- Files status OK
00:40:19:WU02:FS00:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
00:40:19:WU02:FS00:0x15:- Expanded 124547 -> 502918 (decompressed 403.7 percent)
00:40:19:WU02:FS00:0x15:Called DecompressByteArray: compressed_data_size=124547 data_size=502918, decompressed_data_size=502918 diff=0
00:40:19:WU02:FS00:0x15:- Digital signature verified
00:40:19:WU02:FS00:0x15:
00:40:19:WU02:FS00:0x15:Project: 7625 (Run 709, Clone 1, Gen 9)
00:40:19:WU02:FS00:0x15:
00:40:19:WU02:FS00:0x15:Assembly optimizations on if available.
00:40:19:WU02:FS00:0x15:Entering M.D.
00:40:21:WU02:FS00:0x15:Tpr hash 02/wudata_01.tpr: 3275045972 1625723097 2082118532 2286997234 4269661401
00:40:21:WU02:FS00:0x15:GPU device id=0
00:40:21:WU02:FS00:0x15:Working on Protein
00:40:21:WU02:FS00:0x15:Client config unavailable.
00:40:21:WU02:FS00:0x15:Starting GUI Server
00:40:26:WU01:FS01:0xa4:- Looking at optimizations...
00:40:26:WU01:FS01:0xa4:- Working with standard loops on this execution.
00:40:26:WU01:FS01:0xa4:- Previous termination of core was improper.
00:40:26:WU01:FS01:0xa4:- Files status OK
00:40:26:WU01:FS01:0xa4:- Expanded 2054096 -> 5365960 (decompressed 261.2 percent)
00:40:26:WU01:FS01:0xa4:Called DecompressByteArray: compressed_data_size=2054096 data_size=5365960, decompressed_data_size=5365960 diff=0
00:40:26:WU01:FS01:0xa4:- Digital signature verified
00:40:26:WU01:FS01:0xa4:
00:40:26:WU01:FS01:0xa4:Project: 7808 (Run 2, Clone 180, Gen 57)
00:40:26:WU01:FS01:0xa4:
00:40:26:WU01:FS01:0xa4:Entering M.D.
00:40:26:3:127.0.0.1:New Web connection
00:40:32:WU01:FS01:0xa4:Using Gromacs checkpoints
00:40:32:WU01:FS01:0xa4:Mapping NT from 8 to 8
00:40:32:WU01:FS01:0xa4:Resuming from checkpoint
00:40:32:WU01:FS01:0xa4:Verified 01/wudata_01.log
00:40:32:WU01:FS01:0xa4:Verified 01/wudata_01.trr
00:40:32:WU01:FS01:0xa4:Verified 01/wudata_01.xtc
00:40:32:WU01:FS01:0xa4:Verified 01/wudata_01.edr
00:40:32:WU01:FS01:0xa4:Completed 427320 out of 1500000 steps (28%)
00:41:23:WU02:FS00:0x15:Setting checkpoint frequency: 400000
00:41:23:WU02:FS00:0x15:Completed 3 out of 40000000 steps (0%).
00:48:07:WU02:FS00:0x15:Completed 400000 out of 40000000 steps (1%).
00:54:51:WU02:FS00:0x15:Completed 800000 out of 40000000 steps (2%).
01:01:34:WU02:FS00:0x15:Completed 1200000 out of 40000000 steps (3%).
01:08:17:WU02:FS00:0x15:Completed 1600000 out of 40000000 steps (4%).
01:15:01:WU02:FS00:0x15:Completed 2000000 out of 40000000 steps (5%).
01:21:44:WU02:FS00:0x15:Completed 2400000 out of 40000000 steps (6%).
01:21:44:WARNING:WU01:FS01:Detected clock skew (3 mins 46 secs), adjusting time estimates
01:21:44:WARNING:WU02:FS00:Detected clock skew (3 mins 46 secs), adjusting time estimates
01:22:35:WU01:FS01:0xa4:Completed 435000 out of 1500000 steps (29%)
01:28:28:WU02:FS00:0x15:Completed 2800000 out of 40000000 steps (7%).
01:35:13:WU02:FS00:0x15:Completed 3200000 out of 40000000 steps (8%).
01:41:57:WU02:FS00:0x15:Completed 3600000 out of 40000000 steps (9%).
01:48:40:WU02:FS00:0x15:Completed 4000000 out of 40000000 steps (10%).
01:55:23:WU02:FS00:0x15:Completed 4400000 out of 40000000 steps (11%).
02:02:06:WU02:FS00:0x15:Completed 4800000 out of 40000000 steps (12%).
02:08:52:WU02:FS00:0x15:Completed 5200000 out of 40000000 steps (13%).
02:15:36:WU02:FS00:0x15:Completed 5600000 out of 40000000 steps (14%).
02:22:20:WU02:FS00:0x15:Completed 6000000 out of 40000000 steps (15%).
02:29:04:WU02:FS00:0x15:Completed 6400000 out of 40000000 steps (16%).
02:35:47:WU02:FS00:0x15:Completed 6800000 out of 40000000 steps (17%).
02:42:31:WU02:FS00:0x15:Completed 7200000 out of 40000000 steps (18%).
02:49:14:WU02:FS00:0x15:Completed 7600000 out of 40000000 steps (19%).
02:55:58:WU02:FS00:0x15:Completed 8000000 out of 40000000 steps (20%).
03:02:41:WU02:FS00:0x15:Completed 8400000 out of 40000000 steps (21%).
03:09:25:WU02:FS00:0x15:Completed 8800000 out of 40000000 steps (22%).
03:16:08:WU02:FS00:0x15:Completed 9200000 out of 40000000 steps (23%).
03:22:52:WU02:FS00:0x15:Completed 9600000 out of 40000000 steps (24%).
03:29:35:WU02:FS00:0x15:Completed 10000000 out of 40000000 steps (25%).
03:36:19:WU02:FS00:0x15:Completed 10400000 out of 40000000 steps (26%).
03:43:02:WU02:FS00:0x15:Completed 10800000 out of 40000000 steps (27%).
03:49:46:WU02:FS00:0x15:Completed 11200000 out of 40000000 steps (28%).
03:56:29:WU02:FS00:0x15:Completed 11600000 out of 40000000 steps (29%).
04:03:13:WU02:FS00:0x15:Completed 12000000 out of 40000000 steps (30%).
04:09:56:WU02:FS00:0x15:Completed 12400000 out of 40000000 steps (31%).
04:16:40:WU02:FS00:0x15:Completed 12800000 out of 40000000 steps (32%).
04:23:23:WU02:FS00:0x15:Completed 13200000 out of 40000000 steps (33%).
04:30:07:WU02:FS00:0x15:Completed 13600000 out of 40000000 steps (34%).
04:36:50:WU02:FS00:0x15:Completed 14000000 out of 40000000 steps (35%).
04:43:33:WU02:FS00:0x15:Completed 14400000 out of 40000000 steps (36%).
04:50:17:WU02:FS00:0x15:Completed 14800000 out of 40000000 steps (37%).
04:57:00:WU02:FS00:0x15:Completed 15200000 out of 40000000 steps (38%).
05:03:43:WU02:FS00:0x15:Completed 15600000 out of 40000000 steps (39%).
05:10:27:WU02:FS00:0x15:Completed 16000000 out of 40000000 steps (40%).
05:17:10:WU02:FS00:0x15:Completed 16400000 out of 40000000 steps (41%).
05:23:53:WU02:FS00:0x15:Completed 16800000 out of 40000000 steps (42%).
05:30:36:WU02:FS00:0x15:Completed 17200000 out of 40000000 steps (43%).
05:37:19:WU02:FS00:0x15:Completed 17600000 out of 40000000 steps (44%).
05:44:03:WU02:FS00:0x15:Completed 18000000 out of 40000000 steps (45%).
05:50:46:WU02:FS00:0x15:Completed 18400000 out of 40000000 steps (46%).
05:57:29:WU02:FS00:0x15:Completed 18800000 out of 40000000 steps (47%).
06:04:12:WU02:FS00:0x15:Completed 19200000 out of 40000000 steps (48%).
06:10:56:WU02:FS00:0x15:Completed 19600000 out of 40000000 steps (49%).
06:17:39:WU02:FS00:0x15:Completed 20000000 out of 40000000 steps (50%).
06:24:22:WU02:FS00:0x15:Completed 20400000 out of 40000000 steps (51%).
06:31:05:WU02:FS00:0x15:Completed 20800000 out of 40000000 steps (52%).
06:37:49:WU02:FS00:0x15:Completed 21200000 out of 40000000 steps (53%).
******************************* Date: 2013-02-28 *******************************
06:44:32:WU02:FS00:0x15:Completed 21600000 out of 40000000 steps (54%).
06:51:15:WU02:FS00:0x15:Completed 22000000 out of 40000000 steps (55%).
06:57:58:WU02:FS00:0x15:Completed 22400000 out of 40000000 steps (56%).
07:04:42:WU02:FS00:0x15:Completed 22800000 out of 40000000 steps (57%).
07:11:25:WU02:FS00:0x15:Completed 23200000 out of 40000000 steps (58%).
07:18:08:WU02:FS00:0x15:Completed 23600000 out of 40000000 steps (59%).
07:24:52:WU02:FS00:0x15:Completed 24000000 out of 40000000 steps (60%).
07:31:35:WU02:FS00:0x15:Completed 24400000 out of 40000000 steps (61%).
07:31:49:WU01:FS01:0xa4:Completed 450000 out of 1500000 steps (30%)
07:38:18:WU02:FS00:0x15:Completed 24800000 out of 40000000 steps (62%).
07:45:01:WU02:FS00:0x15:Completed 25200000 out of 40000000 steps (63%).
07:51:45:WU02:FS00:0x15:Completed 25600000 out of 40000000 steps (64%).
07:58:28:WU02:FS00:0x15:Completed 26000000 out of 40000000 steps (65%).
08:05:12:WU02:FS00:0x15:Completed 26400000 out of 40000000 steps (66%).
08:11:55:WU02:FS00:0x15:Completed 26800000 out of 40000000 steps (67%).
08:18:38:WU02:FS00:0x15:Completed 27200000 out of 40000000 steps (68%).
08:25:21:WU02:FS00:0x15:Completed 27600000 out of 40000000 steps (69%).
08:32:05:WU02:FS00:0x15:Completed 28000000 out of 40000000 steps (70%).
08:38:48:WU02:FS00:0x15:Completed 28400000 out of 40000000 steps (71%).
08:45:31:WU02:FS00:0x15:Completed 28800000 out of 40000000 steps (72%).
08:52:15:WU02:FS00:0x15:Completed 29200000 out of 40000000 steps (73%).
08:58:58:WU02:FS00:0x15:Completed 29600000 out of 40000000 steps (74%).
09:05:41:WU02:FS00:0x15:Completed 30000000 out of 40000000 steps (75%).
09:12:24:WU02:FS00:0x15:Completed 30400000 out of 40000000 steps (76%).
09:19:08:WU02:FS00:0x15:Completed 30800000 out of 40000000 steps (77%).
09:26:49:WU02:FS00:0x15:Completed 31200000 out of 40000000 steps (78%).
09:40:34:WU02:FS00:0x15:Completed 31600000 out of 40000000 steps (79%).
09:49:43:WU02:FS00:0x15:Completed 32000000 out of 40000000 steps (80%).
09:56:27:WU02:FS00:0x15:Completed 32400000 out of 40000000 steps (81%).
10:03:13:WU02:FS00:0x15:Completed 32800000 out of 40000000 steps (82%).
10:09:56:WU02:FS00:0x15:Completed 33200000 out of 40000000 steps (83%).
10:13:07:52:127.0.0.1:New Web connection
10:13:11:WARNING:Exception: 49:127.0.0.1: Send error: 10053: An established connection was aborted by the software in your host machine.
10:16:42:WU02:FS00:0x15:Completed 33600000 out of 40000000 steps (84%).
10:18:54:WARNING:Exception: 57:127.0.0.1: Send error: 10053: An established connection was aborted by the software in your host machine.
10:19:15:61:127.0.0.1:New Web connection
10:23:28:WU02:FS00:0x15:Completed 34000000 out of 40000000 steps (85%).
10:30:13:WU02:FS00:0x15:Completed 34400000 out of 40000000 steps (86%).
10:36:58:WU02:FS00:0x15:Completed 34800000 out of 40000000 steps (87%).
10:37:33:WU01:FS01:0xa4:Completed 465000 out of 1500000 steps (31%)
10:43:42:WU02:FS00:0x15:Completed 35200000 out of 40000000 steps (88%).
10:50:27:WU02:FS00:0x15:Completed 35600000 out of 40000000 steps (89%).
10:57:11:WU02:FS00:0x15:Completed 36000000 out of 40000000 steps (90%).
11:03:56:WU02:FS00:0x15:Completed 36400000 out of 40000000 steps (91%).
11:10:39:WU02:FS00:0x15:Completed 36800000 out of 40000000 steps (92%).
11:17:23:WU02:FS00:0x15:Completed 37200000 out of 40000000 steps (93%).
11:24:06:WU02:FS00:0x15:Completed 37600000 out of 40000000 steps (94%).
11:30:50:WU02:FS00:0x15:Completed 38000000 out of 40000000 steps (95%).
11:37:33:WU02:FS00:0x15:Completed 38400000 out of 40000000 steps (96%).
11:44:17:WU02:FS00:0x15:Completed 38800000 out of 40000000 steps (97%).
11:51:00:WU02:FS00:0x15:Completed 39200000 out of 40000000 steps (98%).
11:51:01:WU00:FS00:Connecting to assign-GPU.stanford.edu:80
11:51:01:WU00:FS00:News: Welcome to Folding@Home
11:51:01:WU00:FS00:Assigned to work server 171.64.65.105
11:51:01:WU00:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GF110 [GeForce GTX 570 HD] from 171.64.65.105
11:51:01:WU00:FS00:Connecting to 171.64.65.105:8080
11:51:01:WU00:FS00:Downloading 124.36KiB
11:51:02:WU00:FS00:Download complete
11:51:02:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:7626 run:44 clone:0 gen:97 core:0x15 unit:0x00000080664f2dd14fe619fecfc47e9d
11:57:44:WU02:FS00:0x15:Completed 39600000 out of 40000000 steps (99%).
12:04:27:WU02:FS00:0x15:Completed 40000000 out of 40000000 steps (100%).
12:04:28:WU02:FS00:0x15:Finished fah_main status=0
12:04:28:WU02:FS00:0x15:Successful run
12:04:28:WU02:FS00:0x15:DynamicWrapper: Finished Work Unit: sleep=10000
12:04:38:WU02:FS00:0x15:Reserved 761532 bytes for xtc file; Cosm status=0
12:04:38:WU02:FS00:0x15:Allocated 761532 bytes for xtc file
12:04:38:WU02:FS00:0x15:- Reading up to 761532 from "02/wudata_01.xtc": Read 761532
12:04:38:WU02:FS00:0x15:Read 761532 bytes from xtc file; available packet space=785668932
12:04:38:WU02:FS00:0x15:xtc file hash check passed.
12:04:38:WU02:FS00:0x15:Reserved 47832 47832 785668932 bytes for arc file=<02/wudata_01.trr> Cosm status=0
12:04:38:WU02:FS00:0x15:Allocated 47832 bytes for arc file
12:04:38:WU02:FS00:0x15:- Reading up to 47832 from "02/wudata_01.trr": Read 47832
12:04:38:WU02:FS00:0x15:Read 47832 bytes from arc file; available packet space=785621100
12:04:38:WU02:FS00:0x15:trr file hash check passed.
12:04:38:WU02:FS00:0x15:Allocated 544 bytes for edr file
12:04:38:WU02:FS00:0x15:Read bedfile
12:04:38:WU02:FS00:0x15:edr file hash check passed.
12:04:38:WU02:FS00:0x15:Allocated 36746 bytes for logfile
12:04:38:WU02:FS00:0x15:Read logfile
12:04:38:WU02:FS00:0x15:GuardedRun: success in DynamicWrapper
12:04:38:WU02:FS00:0x15:GuardedRun: done
12:04:38:WU02:FS00:0x15:Run: GuardedRun completed.
12:04:41:WU02:FS00:0x15:+ Opened results file
12:04:41:WU02:FS00:0x15:- Writing 847166 bytes of core data to disk...
12:04:41:WU02:FS00:0x15:Done: 846654 -> 818598 (compressed to 96.6 percent)
12:04:41:WU02:FS00:0x15: ... Done.
12:04:41:WU02:FS00:0x15:DeleteFrameFiles: successfully deleted file=02/wudata_01.ckp
12:04:42:WU02:FS00:0x15:Shutting down core
12:04:42:WU02:FS00:0x15:
12:04:42:WU02:FS00:0x15:Folding@home Core Shutdown: FINISHED_UNIT
12:04:42:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
12:04:42:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:7625 run:709 clone:1 gen:9 core:0x15 unit:0x0000000c664f2dd14fe619371e442d39
12:04:42:WU02:FS00:Uploading 799.91KiB to 171.64.65.105
12:04:42:WU02:FS00:Connecting to 171.64.65.105:8080
12:04:42:WU00:FS00:Starting
12:04:42:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 00 -suffix 01 -version 703 -lifeline 8388 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
12:04:42:WU00:FS00:Started FahCore on PID 10672
12:04:42:WU00:FS00:Core PID:6388
12:04:42:WU00:FS00:FahCore 0x15 started
12:04:43:WU00:FS00:0x15:
12:04:43:WU00:FS00:0x15:*------------------------------*
12:04:43:WU00:FS00:0x15:Folding@Home GPU Core
12:04:43:WU00:FS00:0x15:Version 2.25 (Wed May 9 17:03:01 EDT 2012)
12:04:43:WU00:FS00:0x15:Build host AmoebaRemote
12:04:43:WU00:FS00:0x15:Board Type NVIDIA/CUDA
12:04:43:WU00:FS00:0x15:Core 15
12:04:43:WU00:FS00:0x15:
12:04:43:WU00:FS00:0x15:Window's signal control handler registered.
12:04:43:WU00:FS00:0x15:Preparing to commence simulation
12:04:43:WU00:FS00:0x15:- Looking at optimizations...
12:04:43:WU00:FS00:0x15:DeleteFrameFiles: successfully deleted file=00/wudata_01.ckp
12:04:43:WU00:FS00:0x15:- Created dyn
12:04:43:WU00:FS00:0x15:- Files status OK
12:04:43:WU00:FS00:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
12:04:43:WU00:FS00:0x15:- Expanded 126828 -> 507182 (decompressed 399.8 percent)
12:04:43:WU00:FS00:0x15:Called DecompressByteArray: compressed_data_size=126828 data_size=507182, decompressed_data_size=507182 diff=0
12:04:43:WU00:FS00:0x15:- Digital signature verified
12:04:43:WU00:FS00:0x15:
12:04:43:WU00:FS00:0x15:Project: 7626 (Run 44, Clone 0, Gen 97)
12:04:43:WU00:FS00:0x15:
12:04:43:WU00:FS00:0x15:Assembly optimizations on if available.
12:04:43:WU00:FS00:0x15:Entering M.D.
12:04:44:WU02:FS00:Upload complete
12:04:44:WU02:FS00:Server responded WORK_ACK (400)
12:04:44:WU02:FS00:Final credit estimate, 14093.00 points
12:04:44:WU02:FS00:Cleaning up
12:04:44:WU00:FS00:0x15:Tpr hash 00/wudata_01.tpr: 2293198413 3759896912 710603028 1508577764 398535787
12:04:44:WU00:FS00:0x15:GPU device id=0
12:04:44:WU00:FS00:0x15:Working on Protein
12:04:44:WU00:FS00:0x15:Client config unavailable.
12:04:45:WU00:FS00:0x15:Starting GUI Server
12:05:46:WU00:FS00:0x15:Setting checkpoint frequency: 400000
12:05:46:WU00:FS00:0x15:Completed 3 out of 40000000 steps (0%).
12:12:29:WU00:FS00:0x15:Completed 400000 out of 40000000 steps (1%).
12:19:12:WU00:FS00:0x15:Completed 800000 out of 40000000 steps (2%).
12:25:56:WU00:FS00:0x15:Completed 1200000 out of 40000000 steps (3%).
12:32:40:WU00:FS00:0x15:Completed 1600000 out of 40000000 steps (4%).
12:39:23:WU00:FS00:0x15:Completed 2000000 out of 40000000 steps (5%).
******************************* Date: 2013-02-28 *******************************
12:46:06:WU00:FS00:0x15:Completed 2400000 out of 40000000 steps (6%).
12:52:50:WU00:FS00:0x15:Completed 2800000 out of 40000000 steps (7%).
12:59:34:WU00:FS00:0x15:Completed 3200000 out of 40000000 steps (8%).
13:06:17:WU00:FS00:0x15:Completed 3600000 out of 40000000 steps (9%).
13:13:01:WU00:FS00:0x15:Completed 4000000 out of 40000000 steps (10%).
13:19:44:WU00:FS00:0x15:Completed 4400000 out of 40000000 steps (11%).
13:26:28:WU00:FS00:0x15:Completed 4800000 out of 40000000 steps (12%).
13:33:11:WU00:FS00:0x15:Completed 5200000 out of 40000000 steps (13%).
13:39:55:WU00:FS00:0x15:Completed 5600000 out of 40000000 steps (14%).
13:46:38:WU00:FS00:0x15:Completed 6000000 out of 40000000 steps (15%).
13:53:21:WU00:FS00:0x15:Completed 6400000 out of 40000000 steps (16%).
14:00:05:WU00:FS00:0x15:Completed 6800000 out of 40000000 steps (17%).
14:06:48:WU00:FS00:0x15:Completed 7200000 out of 40000000 steps (18%).
14:13:31:WU00:FS00:0x15:Completed 7600000 out of 40000000 steps (19%).
14:20:15:WU00:FS00:0x15:Completed 8000000 out of 40000000 steps (20%).
14:26:58:WU00:FS00:0x15:Completed 8400000 out of 40000000 steps (21%).
14:33:41:WU00:FS00:0x15:Completed 8800000 out of 40000000 steps (22%).
14:40:24:WU00:FS00:0x15:Completed 9200000 out of 40000000 steps (23%).
14:47:08:WU00:FS00:0x15:Completed 9600000 out of 40000000 steps (24%).
14:53:51:WU00:FS00:0x15:Completed 10000000 out of 40000000 steps (25%).
15:00:34:WU00:FS00:0x15:Completed 10400000 out of 40000000 steps (26%).
15:07:18:WU00:FS00:0x15:Completed 10800000 out of 40000000 steps (27%).
15:14:01:WU00:FS00:0x15:Completed 11200000 out of 40000000 steps (28%).
15:20:44:WU00:FS00:0x15:Completed 11600000 out of 40000000 steps (29%).
15:27:28:WU00:FS00:0x15:Completed 12000000 out of 40000000 steps (30%).
15:34:11:WU00:FS00:0x15:Completed 12400000 out of 40000000 steps (31%).
15:40:54:WU00:FS00:0x15:Completed 12800000 out of 40000000 steps (32%).
15:47:37:WU00:FS00:0x15:Completed 13200000 out of 40000000 steps (33%).
15:54:21:WU00:FS00:0x15:Completed 13600000 out of 40000000 steps (34%).
16:01:04:WU00:FS00:0x15:Completed 14000000 out of 40000000 steps (35%).
16:07:47:WU00:FS00:0x15:Completed 14400000 out of 40000000 steps (36%).
16:14:31:WU00:FS00:0x15:Completed 14800000 out of 40000000 steps (37%).
16:21:14:WU00:FS00:0x15:Completed 15200000 out of 40000000 steps (38%).
16:27:57:WU00:FS00:0x15:Completed 15600000 out of 40000000 steps (39%).
16:34:41:WU00:FS00:0x15:Completed 16000000 out of 40000000 steps (40%).
16:41:24:WU00:FS00:0x15:Completed 16400000 out of 40000000 steps (41%).
16:44:20:WU01:FS01:0xa4:Completed 480000 out of 1500000 steps (32%)
16:48:07:WU00:FS00:0x15:Completed 16800000 out of 40000000 steps (42%).
16:54:51:WU00:FS00:0x15:Completed 17200000 out of 40000000 steps (43%).
17:01:34:WU00:FS00:0x15:Completed 17600000 out of 40000000 steps (44%).
17:08:17:WU00:FS00:0x15:Completed 18000000 out of 40000000 steps (45%).
17:15:01:WU00:FS00:0x15:Completed 18400000 out of 40000000 steps (46%).
17:21:44:WU00:FS00:0x15:Completed 18800000 out of 40000000 steps (47%).
17:28:27:WU00:FS00:0x15:Completed 19200000 out of 40000000 steps (48%).
17:35:10:WU00:FS00:0x15:Completed 19600000 out of 40000000 steps (49%).
17:41:54:WU00:FS00:0x15:Completed 20000000 out of 40000000 steps (50%).
17:48:37:WU00:FS00:0x15:Completed 20400000 out of 40000000 steps (51%).
17:55:20:WU00:FS00:0x15:Completed 20800000 out of 40000000 steps (52%).
18:02:04:WU00:FS00:0x15:Completed 21200000 out of 40000000 steps (53%).
18:08:47:WU00:FS00:0x15:Completed 21600000 out of 40000000 steps (54%).
18:15:30:WU00:FS00:0x15:Completed 22000000 out of 40000000 steps (55%).
18:22:14:WU00:FS00:0x15:Completed 22400000 out of 40000000 steps (56%).
18:28:57:WU00:FS00:0x15:Completed 22800000 out of 40000000 steps (57%).
18:35:40:WU00:FS00:0x15:Completed 23200000 out of 40000000 steps (58%).
18:42:24:WU00:FS00:0x15:Completed 23600000 out of 40000000 steps (59%).
******************************* Date: 2013-02-28 *******************************
18:49:07:WU00:FS00:0x15:Completed 24000000 out of 40000000 steps (60%).
18:55:50:WU00:FS00:0x15:Completed 24400000 out of 40000000 steps (61%).
19:02:33:WU00:FS00:0x15:Completed 24800000 out of 40000000 steps (62%).
19:09:17:WU00:FS00:0x15:Completed 25200000 out of 40000000 steps (63%).
19:16:00:WU00:FS00:0x15:Completed 25600000 out of 40000000 steps (64%).
19:22:44:WU00:FS00:0x15:Completed 26000000 out of 40000000 steps (65%).
19:29:27:WU00:FS00:0x15:Completed 26400000 out of 40000000 steps (66%).
19:36:10:WU00:FS00:0x15:Completed 26800000 out of 40000000 steps (67%).
19:42:53:WU00:FS00:0x15:Completed 27200000 out of 40000000 steps (68%).
19:49:37:WU00:FS00:0x15:Completed 27600000 out of 40000000 steps (69%).
19:56:20:WU00:FS00:0x15:Completed 28000000 out of 40000000 steps (70%).
20:03:04:WU00:FS00:0x15:Completed 28400000 out of 40000000 steps (71%).
20:09:47:WU00:FS00:0x15:Completed 28800000 out of 40000000 steps (72%).
20:16:30:WU00:FS00:0x15:Completed 29200000 out of 40000000 steps (73%).
20:23:13:WU00:FS00:0x15:Completed 29600000 out of 40000000 steps (74%).
20:29:57:WU00:FS00:0x15:Completed 30000000 out of 40000000 steps (75%).
20:36:40:WU00:FS00:0x15:Completed 30400000 out of 40000000 steps (76%).
20:43:23:WU00:FS00:0x15:Completed 30800000 out of 40000000 steps (77%).
20:50:07:WU00:FS00:0x15:Completed 31200000 out of 40000000 steps (78%).
20:52:58:WU01:FS01:0xa4:Completed 495000 out of 1500000 steps (33%)
20:56:51:WU00:FS00:0x15:Completed 31600000 out of 40000000 steps (79%).
21:03:35:WU00:FS00:0x15:Completed 32000000 out of 40000000 steps (80%).
21:10:19:WU00:FS00:0x15:Completed 32400000 out of 40000000 steps (81%).
21:17:03:WU00:FS00:0x15:Completed 32800000 out of 40000000 steps (82%).
21:23:47:WU00:FS00:0x15:Completed 33200000 out of 40000000 steps (83%).
21:30:31:WU00:FS00:0x15:Completed 33600000 out of 40000000 steps (84%).
21:37:14:WU00:FS00:0x15:Completed 34000000 out of 40000000 steps (85%).
21:43:58:WU00:FS00:0x15:Completed 34400000 out of 40000000 steps (86%).
21:50:41:WU00:FS00:0x15:Completed 34800000 out of 40000000 steps (87%).
21:57:34:WU00:FS00:0x15:Completed 35200000 out of 40000000 steps (88%).
22:04:17:WU00:FS00:0x15:Completed 35600000 out of 40000000 steps (89%).
22:11:03:WU00:FS00:0x15:Completed 36000000 out of 40000000 steps (90%).
22:17:46:WU00:FS00:0x15:Completed 36400000 out of 40000000 steps (91%).
22:24:30:WU00:FS00:0x15:Completed 36800000 out of 40000000 steps (92%).
22:31:19:WU00:FS00:0x15:Completed 37200000 out of 40000000 steps (93%).
22:38:06:WU00:FS00:0x15:Completed 37600000 out of 40000000 steps (94%).
22:44:52:WU00:FS00:0x15:Completed 38000000 out of 40000000 steps (95%).
22:51:36:WU00:FS00:0x15:Completed 38400000 out of 40000000 steps (96%).
22:58:29:WU00:FS00:0x15:Completed 38800000 out of 40000000 steps (97%).
23:05:13:WU00:FS00:0x15:Completed 39200000 out of 40000000 steps (98%).
23:05:14:WU02:FS00:Connecting to assign-GPU.stanford.edu:80
23:05:14:WU02:FS00:News: Welcome to Folding@Home
23:05:14:WU02:FS00:Assigned to work server 171.67.108.36
23:05:14:WU02:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GF110 [GeForce GTX 570 HD] from 171.67.108.36
23:05:14:WU02:FS00:Connecting to 171.67.108.36:8080
23:05:15:WU02:FS00:Downloading 57.49KiB
23:05:15:WU02:FS00:Download complete
23:05:15:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:8071 run:241 clone:4 gen:10 core:0x15 unit:0x0000000c6652edb45122e225b8e119e7
23:12:02:WU00:FS00:0x15:Completed 39600000 out of 40000000 steps (99%).
23:18:46:WU00:FS00:0x15:Completed 40000000 out of 40000000 steps (100%).
23:18:47:WU00:FS00:0x15:Finished fah_main status=0
23:18:47:WU00:FS00:0x15:Successful run
23:18:47:WU00:FS00:0x15:DynamicWrapper: Finished Work Unit: sleep=10000
23:18:57:WU00:FS00:0x15:Reserved 767852 bytes for xtc file; Cosm status=0
23:18:57:WU00:FS00:0x15:Allocated 767852 bytes for xtc file
23:18:57:WU00:FS00:0x15:- Reading up to 767852 from "00/wudata_01.xtc": Read 767852
23:18:57:WU00:FS00:0x15:Read 767852 bytes from xtc file; available packet space=785662612
23:18:57:WU00:FS00:0x15:xtc file hash check passed.
23:18:57:WU00:FS00:0x15:Reserved 48264 48264 785662612 bytes for arc file=<00/wudata_01.trr> Cosm status=0
23:18:57:WU00:FS00:0x15:Allocated 48264 bytes for arc file
23:18:57:WU00:FS00:0x15:- Reading up to 48264 from "00/wudata_01.trr": Read 48264
23:18:57:WU00:FS00:0x15:Read 48264 bytes from arc file; available packet space=785614348
23:18:57:WU00:FS00:0x15:trr file hash check passed.
23:18:57:WU00:FS00:0x15:Allocated 544 bytes for edr file
23:18:57:WU00:FS00:0x15:Read bedfile
23:18:57:WU00:FS00:0x15:edr file hash check passed.
23:18:57:WU00:FS00:0x15:Allocated 36748 bytes for logfile
23:18:57:WU00:FS00:0x15:Read logfile
23:18:57:WU00:FS00:0x15:GuardedRun: success in DynamicWrapper
23:18:57:WU00:FS00:0x15:GuardedRun: done
23:18:57:WU00:FS00:0x15:Run: GuardedRun completed.
23:18:59:WU00:FS00:0x15:+ Opened results file
23:18:59:WU00:FS00:0x15:- Writing 853920 bytes of core data to disk...
23:18:59:WU00:FS00:0x15:Done: 853408 -> 825253 (compressed to 96.7 percent)
23:18:59:WU00:FS00:0x15: ... Done.
23:18:59:WU00:FS00:0x15:DeleteFrameFiles: successfully deleted file=00/wudata_01.ckp
23:18:59:WU00:FS00:0x15:Shutting down core
23:18:59:WU00:FS00:0x15:
23:18:59:WU00:FS00:0x15:Folding@home Core Shutdown: FINISHED_UNIT
23:18:59:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
23:18:59:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:7626 run:44 clone:0 gen:97 core:0x15 unit:0x00000080664f2dd14fe619fecfc47e9d
23:18:59:WU00:FS00:Uploading 806.41KiB to 171.64.65.105
23:18:59:WU00:FS00:Connecting to 171.64.65.105:8080
23:19:00:WU02:FS00:Starting
23:19:00:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 02 -suffix 01 -version 703 -lifeline 8388 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
23:19:00:WU02:FS00:Started FahCore on PID 1156
23:19:00:WU02:FS00:Core PID:11132
23:19:00:WU02:FS00:FahCore 0x15 started
23:19:00:WU02:FS00:0x15:
23:19:00:WU02:FS00:0x15:*------------------------------*
23:19:00:WU02:FS00:0x15:Folding@Home GPU Core
23:19:00:WU02:FS00:0x15:Version 2.25 (Wed May 9 17:03:01 EDT 2012)
23:19:00:WU02:FS00:0x15:Build host AmoebaRemote
23:19:00:WU02:FS00:0x15:Board Type NVIDIA/CUDA
23:19:00:WU02:FS00:0x15:Core 15
23:19:00:WU02:FS00:0x15:
23:19:00:WU02:FS00:0x15:Window's signal control handler registered.
23:19:00:WU02:FS00:0x15:Preparing to commence simulation
23:19:00:WU02:FS00:0x15:- Looking at optimizations...
23:19:00:WU02:FS00:0x15:DeleteFrameFiles: successfully deleted file=02/wudata_01.ckp
23:19:00:WU02:FS00:0x15:- Created dyn
23:19:00:WU02:FS00:0x15:- Files status OK
23:19:00:WU02:FS00:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
23:19:00:WU02:FS00:0x15:- Expanded 58354 -> 259314 (decompressed 444.3 percent)
23:19:00:WU02:FS00:0x15:Called DecompressByteArray: compressed_data_size=58354 data_size=259314, decompressed_data_size=259314 diff=0
23:19:00:WU02:FS00:0x15:- Digital signature verified
23:19:00:WU02:FS00:0x15:
23:19:00:WU02:FS00:0x15:Project: 8071 (Run 241, Clone 4, Gen 10)
23:19:00:WU02:FS00:0x15:
23:19:00:WU02:FS00:0x15:Assembly optimizations on if available.
23:19:00:WU02:FS00:0x15:Entering M.D.
23:19:01:WU00:FS00:Upload complete
23:19:01:WU00:FS00:Server responded WORK_ACK (400)
23:19:01:WU00:FS00:Final credit estimate, 14093.00 points
23:19:01:WU00:FS00:Cleaning up
23:19:02:WU02:FS00:0x15:Tpr hash 02/wudata_01.tpr: 2936282967 4001514755 1064928222 926171899 3862711347
23:19:02:WU02:FS00:0x15:GPU device id=0
23:19:02:WU02:FS00:0x15:Working on Glycine aRginine prOline Methionine Alanine Cystine Serine t= 241.00000
23:19:02:WU02:FS00:0x15:Client config unavailable.
23:19:02:WU02:FS00:0x15:Starting GUI Server
23:20:03:WU02:FS00:0x15:Setting checkpoint frequency: 500000
23:20:03:WU02:FS00:0x15:Completed 3 out of 50000000 steps (0%).
23:22:34:WU02:FS00:0x15:Completed 500000 out of 50000000 steps (1%).
23:25:06:WU02:FS00:0x15:Completed 1000000 out of 50000000 steps (2%).
23:27:37:WU02:FS00:0x15:Completed 1500000 out of 50000000 steps (3%).
23:30:09:WU02:FS00:0x15:Completed 2000000 out of 50000000 steps (4%).
23:32:40:WU02:FS00:0x15:Completed 2500000 out of 50000000 steps (5%).
23:35:12:WU02:FS00:0x15:Completed 3000000 out of 50000000 steps (6%).
23:37:44:WU02:FS00:0x15:Completed 3500000 out of 50000000 steps (7%).
23:40:15:WU02:FS00:0x15:Completed 4000000 out of 50000000 steps (8%).
23:42:47:WU02:FS00:0x15:Completed 4500000 out of 50000000 steps (9%).
23:45:19:WU02:FS00:0x15:Completed 5000000 out of 50000000 steps (10%).
23:47:50:WU02:FS00:0x15:Completed 5500000 out of 50000000 steps (11%).
23:50:22:WU02:FS00:0x15:Completed 6000000 out of 50000000 steps (12%).
23:52:54:WU02:FS00:0x15:Completed 6500000 out of 50000000 steps (13%).
23:55:25:WU02:FS00:0x15:Completed 7000000 out of 50000000 steps (14%).
23:58:19:WU02:FS00:0x15:Completed 7500000 out of 50000000 steps (15%).
00:00:51:WU02:FS00:0x15:Completed 8000000 out of 50000000 steps (16%).
00:03:22:WU02:FS00:0x15:Completed 8500000 out of 50000000 steps (17%).
00:05:54:WU02:FS00:0x15:Completed 9000000 out of 50000000 steps (18%).
00:08:25:WU02:FS00:0x15:Completed 9500000 out of 50000000 steps (19%).
00:10:57:WU02:FS00:0x15:Completed 10000000 out of 50000000 steps (20%).
00:13:29:WU02:FS00:0x15:Completed 10500000 out of 50000000 steps (21%).
00:16:00:WU02:FS00:0x15:Completed 11000000 out of 50000000 steps (22%).
00:18:32:WU02:FS00:0x15:Completed 11500000 out of 50000000 steps (23%).
00:21:03:WU02:FS00:0x15:Completed 12000000 out of 50000000 steps (24%).
00:23:35:WU02:FS00:0x15:Completed 12500000 out of 50000000 steps (25%).
00:26:06:WU02:FS00:0x15:Completed 13000000 out of 50000000 steps (26%).
00:28:38:WU02:FS00:0x15:Completed 13500000 out of 50000000 steps (27%).
00:31:10:WU02:FS00:0x15:Completed 14000000 out of 50000000 steps (28%).
00:33:41:WU02:FS00:0x15:Completed 14500000 out of 50000000 steps (29%).
00:36:12:WU02:FS00:0x15:Completed 15000000 out of 50000000 steps (30%).
00:38:44:WU02:FS00:0x15:Completed 15500000 out of 50000000 steps (31%).
00:41:16:WU02:FS00:0x15:Completed 16000000 out of 50000000 steps (32%).
00:43:47:WU02:FS00:0x15:Completed 16500000 out of 50000000 steps (33%).
00:46:19:WU02:FS00:0x15:Completed 17000000 out of 50000000 steps (34%).
00:48:50:WU02:FS00:0x15:Completed 17500000 out of 50000000 steps (35%).
******************************* Date: 2013-03-01 *******************************
00:51:22:WU02:FS00:0x15:Completed 18000000 out of 50000000 steps (36%).
00:53:53:WU02:FS00:0x15:Completed 18500000 out of 50000000 steps (37%).
00:56:25:WU02:FS00:0x15:Completed 19000000 out of 50000000 steps (38%).
00:58:57:WU02:FS00:0x15:Completed 19500000 out of 50000000 steps (39%).
01:01:28:WU02:FS00:0x15:Completed 20000000 out of 50000000 steps (40%).
01:04:00:WU02:FS00:0x15:Completed 20500000 out of 50000000 steps (41%).
01:06:31:WU02:FS00:0x15:Completed 21000000 out of 50000000 steps (42%).
01:09:03:WU02:FS00:0x15:Completed 21500000 out of 50000000 steps (43%).
01:11:34:WU02:FS00:0x15:Completed 22000000 out of 50000000 steps (44%).
01:14:06:WU02:FS00:0x15:Completed 22500000 out of 50000000 steps (45%).
01:16:38:WU02:FS00:0x15:Completed 23000000 out of 50000000 steps (46%).
01:19:09:WU02:FS00:0x15:Completed 23500000 out of 50000000 steps (47%).
01:21:41:WU02:FS00:0x15:Completed 24000000 out of 50000000 steps (48%).
01:24:12:WU02:FS00:0x15:Completed 24500000 out of 50000000 steps (49%).
01:26:44:WU02:FS00:0x15:Completed 25000000 out of 50000000 steps (50%).
01:29:15:WU02:FS00:0x15:Completed 25500000 out of 50000000 steps (51%).
01:31:47:WU02:FS00:0x15:Completed 26000000 out of 50000000 steps (52%).
01:34:19:WU02:FS00:0x15:Completed 26500000 out of 50000000 steps (53%).
01:36:50:WU02:FS00:0x15:Completed 27000000 out of 50000000 steps (54%).
01:39:22:WU02:FS00:0x15:Completed 27500000 out of 50000000 steps (55%).
01:41:53:WU02:FS00:0x15:Completed 28000000 out of 50000000 steps (56%).
01:44:25:WU02:FS00:0x15:Completed 28500000 out of 50000000 steps (57%).
01:46:56:WU02:FS00:0x15:Completed 29000000 out of 50000000 steps (58%).
01:49:28:WU02:FS00:0x15:Completed 29500000 out of 50000000 steps (59%).
01:52:00:WU02:FS00:0x15:Completed 30000000 out of 50000000 steps (60%).
01:54:31:WU02:FS00:0x15:Completed 30500000 out of 50000000 steps (61%).
01:57:03:WU02:FS00:0x15:Completed 31000000 out of 50000000 steps (62%).
01:59:34:WU02:FS00:0x15:Completed 31500000 out of 50000000 steps (63%).
02:02:06:WU02:FS00:0x15:Completed 32000000 out of 50000000 steps (64%).
02:04:37:WU02:FS00:0x15:Completed 32500000 out of 50000000 steps (65%).
02:07:09:WU02:FS00:0x15:Completed 33000000 out of 50000000 steps (66%).
02:09:41:WU02:FS00:0x15:Completed 33500000 out of 50000000 steps (67%).
02:12:12:WU02:FS00:0x15:Completed 34000000 out of 50000000 steps (68%).
02:14:44:WU02:FS00:0x15:Completed 34500000 out of 50000000 steps (69%).
02:17:15:WU02:FS00:0x15:Completed 35000000 out of 50000000 steps (70%).
02:19:47:WU02:FS00:0x15:Completed 35500000 out of 50000000 steps (71%).
02:22:19:WU02:FS00:0x15:Completed 36000000 out of 50000000 steps (72%).
02:24:51:WU02:FS00:0x15:Completed 36500000 out of 50000000 steps (73%).
02:27:22:WU02:FS00:0x15:Completed 37000000 out of 50000000 steps (74%).
02:29:54:WU02:FS00:0x15:Completed 37500000 out of 50000000 steps (75%).
02:32:26:WU02:FS00:0x15:Completed 38000000 out of 50000000 steps (76%).
02:33:06:WU01:FS01:0xa4:Completed 510000 out of 1500000 steps (34%)
02:34:58:WU02:FS00:0x15:Completed 38500000 out of 50000000 steps (77%).
02:37:29:WU02:FS00:0x15:Completed 39000000 out of 50000000 steps (78%).
02:40:01:WU02:FS00:0x15:Completed 39500000 out of 50000000 steps (79%).
02:42:32:WU02:FS00:0x15:Completed 40000000 out of 50000000 steps (80%).
02:45:04:WU02:FS00:0x15:Completed 40500000 out of 50000000 steps (81%).
02:47:36:WU02:FS00:0x15:Completed 41000000 out of 50000000 steps (82%).
02:50:07:WU02:FS00:0x15:Completed 41500000 out of 50000000 steps (83%).
02:52:39:WU02:FS00:0x15:Completed 42000000 out of 50000000 steps (84%).
02:55:10:WU02:FS00:0x15:Completed 42500000 out of 50000000 steps (85%).
02:57:43:WU02:FS00:0x15:Completed 43000000 out of 50000000 steps (86%).
03:00:14:WU02:FS00:0x15:Completed 43500000 out of 50000000 steps (87%).
03:02:47:WU02:FS00:0x15:Completed 44000000 out of 50000000 steps (88%).
03:05:18:WU02:FS00:0x15:Completed 44500000 out of 50000000 steps (89%).
03:07:50:WU02:FS00:0x15:Completed 45000000 out of 50000000 steps (90%).
03:10:21:WU02:FS00:0x15:Completed 45500000 out of 50000000 steps (91%).
03:12:53:WU02:FS00:0x15:Completed 46000000 out of 50000000 steps (92%).
03:15:24:WU02:FS00:0x15:Completed 46500000 out of 50000000 steps (93%).
03:17:56:WU02:FS00:0x15:Completed 47000000 out of 50000000 steps (94%).
03:20:28:WU02:FS00:0x15:Completed 47500000 out of 50000000 steps (95%).
03:22:59:WU02:FS00:0x15:Completed 48000000 out of 50000000 steps (96%).
03:25:31:WU02:FS00:0x15:Completed 48500000 out of 50000000 steps (97%).
03:28:02:WU02:FS00:0x15:Completed 49000000 out of 50000000 steps (98%).
03:28:03:WU00:FS00:Connecting to assign-GPU.stanford.edu:80
03:28:03:WU00:FS00:News: Welcome to Folding@Home
03:28:03:WU00:FS00:Assigned to work server 171.67.108.36
03:28:03:WU00:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GF110 [GeForce GTX 570 HD] from 171.67.108.36
03:28:03:WU00:FS00:Connecting to 171.67.108.36:8080
03:28:04:WU00:FS00:Downloading 57.93KiB
03:28:04:WU00:FS00:Download complete
03:28:04:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:8071 run:118 clone:11 gen:4 core:0x15 unit:0x000000046652edb4512f2d6fa261427a
03:30:36:WU02:FS00:0x15:Completed 49500000 out of 50000000 steps (99%).
03:33:08:WU02:FS00:0x15:Completed 50000000 out of 50000000 steps (100%).
03:33:08:WU02:FS00:0x15:Finished fah_main status=0
03:33:08:WU02:FS00:0x15:Successful run
03:33:08:WU02:FS00:0x15:DynamicWrapper: Finished Work Unit: sleep=10000
03:33:18:WU02:FS00:0x15:Reserved 324800 bytes for xtc file; Cosm status=0
03:33:18:WU02:FS00:0x15:Allocated 324800 bytes for xtc file
03:33:18:WU02:FS00:0x15:- Reading up to 324800 from "02/wudata_01.xtc": Read 324800
03:33:18:WU02:FS00:0x15:Read 324800 bytes from xtc file; available packet space=786105664
03:33:18:WU02:FS00:0x15:xtc file hash check passed.
03:33:18:WU02:FS00:0x15:Reserved 20256 20256 786105664 bytes for arc file=<02/wudata_01.trr> Cosm status=0
03:33:18:WU02:FS00:0x15:Allocated 20256 bytes for arc file
03:33:18:WU02:FS00:0x15:- Reading up to 20256 from "02/wudata_01.trr": Read 20256
03:33:18:WU02:FS00:0x15:Read 20256 bytes from arc file; available packet space=786085408
03:33:18:WU02:FS00:0x15:trr file hash check passed.
03:33:18:WU02:FS00:0x15:Allocated 544 bytes for edr file
03:33:18:WU02:FS00:0x15:Read bedfile
03:33:18:WU02:FS00:0x15:edr file hash check passed.
03:33:18:WU02:FS00:0x15:Allocated 36804 bytes for logfile
03:33:18:WU02:FS00:0x15:Read logfile
03:33:18:WU02:FS00:0x15:GuardedRun: success in DynamicWrapper
03:33:18:WU02:FS00:0x15:GuardedRun: done
03:33:18:WU02:FS00:0x15:Run: GuardedRun completed.
03:33:22:WU02:FS00:0x15:+ Opened results file
03:33:22:WU02:FS00:0x15:- Writing 382916 bytes of core data to disk...
03:33:22:WU02:FS00:0x15:Done: 382404 -> 351452 (compressed to 91.9 percent)
03:33:22:WU02:FS00:0x15: ... Done.
03:33:22:WU02:FS00:0x15:DeleteFrameFiles: successfully deleted file=02/wudata_01.ckp
03:33:22:WU02:FS00:0x15:Shutting down core
03:33:22:WU02:FS00:0x15:
03:33:22:WU02:FS00:0x15:Folding@home Core Shutdown: FINISHED_UNIT
03:33:22:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
03:33:22:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:8071 run:241 clone:4 gen:10 core:0x15 unit:0x0000000c6652edb45122e225b8e119e7
03:33:22:WU02:FS00:Uploading 343.71KiB to 171.67.108.36
03:33:22:WU02:FS00:Connecting to 171.67.108.36:8080
03:33:23:WU00:FS00:Starting
03:33:23:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 00 -suffix 01 -version 703 -lifeline 8388 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
03:33:23:WU00:FS00:Started FahCore on PID 10224
03:33:23:WU00:FS00:Core PID:9208
03:33:23:WU00:FS00:FahCore 0x15 started
03:33:23:WU00:FS00:0x15:
03:33:23:WU00:FS00:0x15:*------------------------------*
03:33:23:WU00:FS00:0x15:Folding@Home GPU Core
03:33:23:WU00:FS00:0x15:Version 2.25 (Wed May 9 17:03:01 EDT 2012)
03:33:23:WU00:FS00:0x15:Build host AmoebaRemote
03:33:23:WU00:FS00:0x15:Board Type NVIDIA/CUDA
03:33:23:WU00:FS00:0x15:Core 15
03:33:23:WU00:FS00:0x15:
03:33:23:WU00:FS00:0x15:Window's signal control handler registered.
03:33:23:WU00:FS00:0x15:Preparing to commence simulation
03:33:23:WU00:FS00:0x15:- Looking at optimizations...
03:33:23:WU00:FS00:0x15:DeleteFrameFiles: successfully deleted file=00/wudata_01.ckp
03:33:23:WU00:FS00:0x15:- Created dyn
03:33:23:WU00:FS00:0x15:- Files status OK
03:33:23:WU00:FS00:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
03:33:23:WU00:FS00:0x15:- Expanded 58812 -> 259314 (decompressed 440.9 percent)
03:33:23:WU00:FS00:0x15:Called DecompressByteArray: compressed_data_size=58812 data_size=259314, decompressed_data_size=259314 diff=0
03:33:23:WU00:FS00:0x15:- Digital signature verified
03:33:23:WU00:FS00:0x15:
03:33:23:WU00:FS00:0x15:Project: 8071 (Run 118, Clone 11, Gen 4)
03:33:23:WU00:FS00:0x15:
03:33:23:WU00:FS00:0x15:Assembly optimizations on if available.
03:33:23:WU00:FS00:0x15:Entering M.D.
03:33:24:WU02:FS00:Upload complete
03:33:24:WU02:FS00:Server responded WORK_ACK (400)
03:33:24:WU02:FS00:Final credit estimate, 3874.00 points
03:33:24:WU02:FS00:Cleaning up
03:33:25:WU00:FS00:0x15:Tpr hash 00/wudata_01.tpr: 4137678891 2023613714 3735999060 4019548950 179042560
03:33:25:WU00:FS00:0x15:GPU device id=0
03:33:25:WU00:FS00:0x15:Working on Glycine aRginine prOline Methionine Alanine Cystine Serine t= 118.00000
03:33:25:WU00:FS00:0x15:Client config unavailable.
03:33:25:WU00:FS00:0x15:Starting GUI Server
03:34:25:WU00:FS00:0x15:Setting checkpoint frequency: 500000
03:34:25:WU00:FS00:0x15:Completed 3 out of 50000000 steps (0%).
03:36:57:WU00:FS00:0x15:Completed 500000 out of 50000000 steps (1%).
03:39:28:WU00:FS00:0x15:Completed 1000000 out of 50000000 steps (2%).
03:42:00:WU00:FS00:0x15:Completed 1500000 out of 50000000 steps (3%).
03:44:31:WU00:FS00:0x15:Completed 2000000 out of 50000000 steps (4%).
03:47:03:WU00:FS00:0x15:Completed 2500000 out of 50000000 steps (5%).
03:49:35:WU00:FS00:0x15:Completed 3000000 out of 50000000 steps (6%).
03:52:06:WU00:FS00:0x15:Completed 3500000 out of 50000000 steps (7%).
03:54:38:WU00:FS00:0x15:Completed 4000000 out of 50000000 steps (8%).
03:57:09:WU00:FS00:0x15:Completed 4500000 out of 50000000 steps (9%).
03:59:41:WU00:FS00:0x15:Completed 5000000 out of 50000000 steps (10%).
04:02:12:WU00:FS00:0x15:Completed 5500000 out of 50000000 steps (11%).
04:04:44:WU00:FS00:0x15:Completed 6000000 out of 50000000 steps (12%).
04:07:16:WU00:FS00:0x15:Completed 6500000 out of 50000000 steps (13%).
04:09:47:WU00:FS00:0x15:Completed 7000000 out of 50000000 steps (14%).
04:12:19:WU00:FS00:0x15:Completed 7500000 out of 50000000 steps (15%).
04:14:50:WU00:FS00:0x15:Completed 8000000 out of 50000000 steps (16%).
04:17:22:WU00:FS00:0x15:Completed 8500000 out of 50000000 steps (17%).
04:19:33:FS00:Shutting core down
04:19:33:FS01:Shutting core down
04:19:40:WU01:FS01:0xa4:Client no longer detected. Shutting down core
04:19:40:WU01:FS01:0xa4:
04:19:40:WU01:FS01:0xa4:Folding@home Core Shutdown: CLIENT_DIED
04:19:40:WU01:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
04:19:40:WU01:FS01:Starting
04:19:40:WARNING:WU01:FS01:Changed SMP threads from 8 to 4 this can cause some work units to fail
04:19:40:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 01 -suffix 01 -version 703 -lifeline 8388 -checkpoint 15 -np 4
04:19:40:WU01:FS01:Started FahCore on PID 7784
04:19:40:WU01:FS01:Core PID:10260
04:19:40:WU01:FS01:FahCore 0xa4 started
04:19:41:WU01:FS01:0xa4:
04:19:41:WU01:FS01:0xa4:*------------------------------*
04:19:41:WU01:FS01:0xa4:Folding@Home Gromacs GB Core
04:19:41:WU01:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
04:19:41:WU01:FS01:0xa4:
04:19:41:WU01:FS01:0xa4:Preparing to commence simulation
04:19:41:WU01:FS01:0xa4:- Looking at optimizations...
04:19:41:WU01:FS01:0xa4:- Files status OK
04:19:41:WU01:FS01:0xa4:- Expanded 2054096 -> 5365960 (decompressed 261.2 percent)
04:19:41:WU01:FS01:0xa4:Called DecompressByteArray: compressed_data_size=2054096 data_size=5365960, decompressed_data_size=5365960 diff=0
04:19:41:WU01:FS01:0xa4:- Digital signature verified
04:19:41:WU01:FS01:0xa4:
04:19:41:WU01:FS01:0xa4:Project: 7808 (Run 2, Clone 180, Gen 57)
04:19:41:WU01:FS01:0xa4:
04:19:41:WU01:FS01:0xa4:Assembly optimizations on if available.
04:19:41:WU01:FS01:0xa4:Entering M.D.
04:19:43:WU00:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
04:19:47:WU01:FS01:0xa4:Using Gromacs checkpoints
04:19:47:WU01:FS01:0xa4:Mapping NT from 4 to 4
04:19:47:WU01:FS01:0xa4:Resuming from checkpoint
04:19:47:WU01:FS01:0xa4:Verified 01/wudata_01.log
04:19:47:WU01:FS01:0xa4:Verified 01/wudata_01.trr
04:19:47:WU01:FS01:0xa4:Verified 01/wudata_01.xtc
04:19:47:WU01:FS01:0xa4:Verified 01/wudata_01.edr
04:19:47:WU01:FS01:0xa4:Completed 514750 out of 1500000 steps (34%)
04:23:41:WARNING:Exception: 138:127.0.0.1: Send error: 10053: An established connection was aborted by the software in your host machine.
04:23:53:FS01:Shutting core down
04:23:53:WU00:FS00:Starting
04:23:53:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 00 -suffix 01 -version 703 -lifeline 8388 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
04:23:53:WU00:FS00:Started FahCore on PID 9384
04:23:53:WU00:FS00:Core PID:2400
04:23:53:WU00:FS00:FahCore 0x15 started
04:23:54:WU00:FS00:0x15:
04:23:54:WU00:FS00:0x15:*------------------------------*
04:23:54:WU00:FS00:0x15:Folding@Home GPU Core
04:23:54:WU00:FS00:0x15:Version 2.25 (Wed May 9 17:03:01 EDT 2012)
04:23:54:WU00:FS00:0x15:Build host AmoebaRemote
04:23:54:WU00:FS00:0x15:Board Type NVIDIA/CUDA
04:23:54:WU00:FS00:0x15:Core 15
04:23:54:WU00:FS00:0x15:
04:23:54:WU00:FS00:0x15:Window's signal control handler registered.
04:23:54:WU00:FS00:0x15:Preparing to commence simulation
04:23:54:WU00:FS00:0x15:- Looking at optimizations...
04:23:54:WU00:FS00:0x15:- Files status OK
04:23:54:WU00:FS00:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
04:23:54:WU00:FS00:0x15:- Expanded 58812 -> 259314 (decompressed 440.9 percent)
04:23:54:WU00:FS00:0x15:Called DecompressByteArray: compressed_data_size=58812 data_size=259314, decompressed_data_size=259314 diff=0
04:23:54:WU00:FS00:0x15:- Digital signature verified
04:23:54:WU00:FS00:0x15:
04:23:54:WU00:FS00:0x15:Project: 8071 (Run 118, Clone 11, Gen 4)
04:23:54:WU00:FS00:0x15:
04:23:54:WU00:FS00:0x15:Assembly optimizations on if available.
04:23:54:WU00:FS00:0x15:Entering M.D.
04:23:55:WU00:FS00:0x15:Will resume from checkpoint file 00/wudata_01.ckp
04:23:55:WU00:FS00:0x15:Tpr hash 00/wudata_01.tpr: 4137678891 2023613714 3735999060 4019548950 179042560
04:23:55:WU00:FS00:0x15:GPU device id=0
04:23:55:WU00:FS00:0x15:Working on Glycine aRginine prOline Methionine Alanine Cystine Serine t= 118.00000
04:23:55:WU00:FS00:0x15:Client config unavailable.
04:23:56:WU00:FS00:0x15:Starting GUI Server
04:24:01:WU01:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
04:24:01:WU01:FS01:Starting
04:24:01:WARNING:WU01:FS01:Changed SMP threads from 4 to 8 this can cause some work units to fail
04:24:01:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 01 -suffix 01 -version 703 -lifeline 8388 -checkpoint 15 -np 8
04:24:01:WU01:FS01:Started FahCore on PID 4452
04:24:01:WU01:FS01:Core PID:4080
04:24:01:WU01:FS01:FahCore 0xa4 started
04:24:02:WU01:FS01:0xa4:
04:24:02:WU01:FS01:0xa4:*------------------------------*
04:24:02:WU01:FS01:0xa4:Folding@Home Gromacs GB Core
04:24:02:WU01:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
04:24:02:WU01:FS01:0xa4:
04:24:02:WU01:FS01:0xa4:Preparing to commence simulation
04:24:02:WU01:FS01:0xa4:- Looking at optimizations...
04:24:02:WU01:FS01:0xa4:- Files status OK
04:24:02:WU01:FS01:0xa4:- Expanded 2054096 -> 5365960 (decompressed 261.2 percent)
04:24:02:WU01:FS01:0xa4:Called DecompressByteArray: compressed_data_size=2054096 data_size=5365960, decompressed_data_size=5365960 diff=0
04:24:02:WU01:FS01:0xa4:- Digital signature verified
04:24:02:WU01:FS01:0xa4:
04:24:02:WU01:FS01:0xa4:Project: 7808 (Run 2, Clone 180, Gen 57)
04:24:02:WU01:FS01:0xa4:
04:24:02:WU01:FS01:0xa4:Assembly optimizations on if available.
04:24:02:WU01:FS01:0xa4:Entering M.D.
04:24:03:141:127.0.0.1:New Web connection
04:24:08:WU01:FS01:0xa4:Using Gromacs checkpoints
04:24:08:WU01:FS01:0xa4:Mapping NT from 8 to 8
04:24:08:WU01:FS01:0xa4:Resuming from checkpoint
04:24:08:WU01:FS01:0xa4:Verified 01/wudata_01.log
04:24:08:WU01:FS01:0xa4:Verified 01/wudata_01.trr
04:24:08:WU01:FS01:0xa4:Verified 01/wudata_01.xtc
04:24:08:WU01:FS01:0xa4:Verified 01/wudata_01.edr
04:24:08:WU01:FS01:0xa4:Completed 514750 out of 1500000 steps (34%)
04:24:57:WARNING:WU00:FS00:Detected clock skew (1 mins 04 secs), adjusting time estimates
04:24:57:WU00:FS00:0x15:Resuming from checkpoint
04:24:57:WU00:FS00:0x15:fcCheckPointResume: retreived and current tpr file hash:
04:24:57:WU00:FS00:0x15: 0 4137678891 4137678891
04:24:57:WU00:FS00:0x15: 1 2023613714 2023613714
04:24:57:WU00:FS00:0x15: 2 3735999060 3735999060
04:24:57:WU00:FS00:0x15: 3 4019548950 4019548950
04:24:57:WU00:FS00:0x15: 4 179042560 179042560
04:24:57:WU00:FS00:0x15:fcCheckPointResume: file hashes same.
04:24:57:WU00:FS00:0x15:fcCheckPointResume: state restored.
04:24:57:WU00:FS00:0x15:fcCheckPointResume: name 00/wudata_01.log Verified 00/wudata_01.log
04:24:57:WU00:FS00:0x15:fcCheckPointResume: name 00/wudata_01.trr Verified 00/wudata_01.trr
04:24:57:WU00:FS00:0x15:fcCheckPointResume: name 00/wudata_01.xtc Verified 00/wudata_01.xtc
04:24:57:WU00:FS00:0x15:fcCheckPointResume: name 00/wudata_01.edr Verified 00/wudata_01.edr
04:24:57:WU00:FS00:0x15:fcCheckPointResume: state restored 2
04:24:57:WU00:FS00:0x15:Resumed from checkpoint
04:24:57:WU00:FS00:0x15:Setting checkpoint frequency: 500000
04:24:57:WU00:FS00:0x15:Completed 8500001 out of 50000000 steps (17%).
04:25:06:FS01:Paused
04:25:06:FS01:Shutting core down
04:25:09:FS01:Unpaused
04:25:12:WU01:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
04:25:12:WU01:FS01:Starting
04:25:12:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 01 -suffix 01 -version 703 -lifeline 8388 -checkpoint 15 -np 8
04:25:12:WU01:FS01:Started FahCore on PID 10176
04:25:12:WU01:FS01:Core PID:7836
04:25:12:WU01:FS01:FahCore 0xa4 started
04:25:12:WU01:FS01:0xa4:
04:25:12:WU01:FS01:0xa4:*------------------------------*
04:25:12:WU01:FS01:0xa4:Folding@Home Gromacs GB Core
04:25:12:WU01:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
04:25:12:WU01:FS01:0xa4:
04:25:12:WU01:FS01:0xa4:Preparing to commence simulation
04:25:12:WU01:FS01:0xa4:- Looking at optimizations...
04:25:12:WU01:FS01:0xa4:- Files status OK
04:25:12:WU01:FS01:0xa4:- Expanded 2054096 -> 5365960 (decompressed 261.2 percent)
04:25:12:WU01:FS01:0xa4:Called DecompressByteArray: compressed_data_size=2054096 data_size=5365960, decompressed_data_size=5365960 diff=0
04:25:12:WU01:FS01:0xa4:- Digital signature verified
04:25:12:WU01:FS01:0xa4:
04:25:12:WU01:FS01:0xa4:Project: 7808 (Run 2, Clone 180, Gen 57)
04:25:12:WU01:FS01:0xa4:
04:25:12:WU01:FS01:0xa4:Assembly optimizations on if available.
04:25:12:WU01:FS01:0xa4:Entering M.D.
04:25:18:WU01:FS01:0xa4:Using Gromacs checkpoints
04:25:18:WU01:FS01:0xa4:Mapping NT from 8 to 8
04:25:19:WU01:FS01:0xa4:Resuming from checkpoint
04:25:19:WU01:FS01:0xa4:Verified 01/wudata_01.log
04:25:19:WU01:FS01:0xa4:Verified 01/wudata_01.trr
04:25:19:WU01:FS01:0xa4:Verified 01/wudata_01.xtc
04:25:19:WU01:FS01:0xa4:Verified 01/wudata_01.edr
04:25:19:WU01:FS01:0xa4:Completed 514750 out of 1500000 steps (34%)
04:27:30:WU00:FS00:0x15:Completed 9000000 out of 50000000 steps (18%).
04:30:03:WU00:FS00:0x15:Completed 9500000 out of 50000000 steps (19%).
04:32:36:WU00:FS00:0x15:Completed 10000000 out of 50000000 steps (20%).
04:35:09:WU00:FS00:0x15:Completed 10500000 out of 50000000 steps (21%).
04:37:42:WU00:FS00:0x15:Completed 11000000 out of 50000000 steps (22%).
04:40:16:WU00:FS00:0x15:Completed 11500000 out of 50000000 steps (23%).
04:42:48:WU00:FS00:0x15:Completed 12000000 out of 50000000 steps (24%).
04:43:58:WARNING:WU01:FS01:FahCore crashed with Windows unhandled exception code 0xUNKNOWN_ENUM, searching for this code online may provide more information
04:43:58:WARNING:WU01:FS01:FahCore returned: UNKNOWN_ENUM (1073807364 = 0x40010004)
04:43:58:WARNING:WU00:FS00:FahCore crashed with Windows unhandled exception code 0xUNKNOWN_ENUM, searching for this code online may provide more information
04:43:58:WARNING:WU00:FS00:FahCore returned: UNKNOWN_ENUM (1073807364 = 0x40010004)
04:43:59:WU01:FS01:Starting
04:43:59:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 01 -suffix 01 -version 703 -lifeline 8388 -checkpoint 15 -np 8
04:43:59:WU01:FS01:Started FahCore on PID 8600
Re: 7808 (2,180,57) - ETEA 11.30 days!
Posted: Sat Mar 02, 2013 8:25 am
by bruce
There is an extremely wide varation in frame time.
{00:40:32:WU01:FS01:0xa4:Completed 427320 out of 1500000 steps (28%)}
> 01:22:35:WU01:FS01:0xa4:Completed 435000 out of 1500000 steps (29%)
> 07:31:49:WU01:FS01:0xa4:Completed 450000 out of 1500000 steps (30%)
> 10:37:33:WU01:FS01:0xa4:Completed 465000 out of 1500000 steps (31%)
> 16:44:20:WU01:FS01:0xa4:Completed 480000 out of 1500000 steps (32%)
> 20:52:58:WU01:FS01:0xa4:Completed 495000 out of 1500000 steps (33%)
> 02:33:06:WU01:FS01:0xa4:Completed 510000 out of 1500000 steps (34%)
{04:19:47:WU01:FS01:0xa4:Completed 514750 out of 1500000 steps (34%)}
{04:25:19:WU01:FS01:0xa4:Completed 514750 out of 1500000 steps (34%)}
Evaluating the duration of each of the full one percent intervals gives:
> 01:22:35
---------------6h9m
> 07:31:49
---------------3h6m
> 10:37:33
---------------6h7m
> 16:44:20
---------------4h9m
> 20:52:58
---------------5h41m
> 02:33:06
There's no possible way to establish an average. There's just too much variation. I strongly suspect that the FAHCore was competing with some other heavy application.
Re: 7808 (2,180,57) - ETEA 11.30 days!
Posted: Sat Mar 02, 2013 6:04 pm
by P5-133XL
When there are wide variations in frame times spread throughout a log, my immediate assumption is that there is an external process stealing CPU time from SMP-folding causing at least one SMP thread to pause for an extended time. The solution is to decrease the number of CPU cores that the SMP slot is using so it won't be paused by that external process.
Even a small amount of CPU being used will cause large delays to SMP folding. The folding cores are highly synchronized and run at a very low priority so that most anything takes priority causing a SMP thread to be suspended. Since all the threads are synchronized, the other folding threads go into a waiting loop for the one thread to become active again. As such even a small amount of non-folding CPU usage can have a very large effect of folding TPF's.
If you decrease the number of folding threads/cores used to allow for non-folding CPU use then the folding thread never gets suspended and TPF's will drastically improve.
Re: 7808 (2,180,57) - ETEA 11.30 days!
Posted: Sat Mar 02, 2013 7:03 pm
by telecentricity
Since I changed from CPU:8 to CPU:6 it has been rock steady. I think the advice of Napoleon, Bruce and others was spot on. Thanks very much for the help.
Kind regards,
Gary
Re: 7808 (2,180,57) - ETEA 11.30 days!
Posted: Sat Mar 02, 2013 8:25 pm
by bruce
bruce wrote:By changing the setting to 6, you've left two virtual cores available for other things that might be running on your system. Check performance monitor. You should see the FahCore using a consistent total of 6/8 = 75% of the CPU time. You should also see essentially 0% for other applications. If you're seeing a few percent consistently going to anything else that's running, that's probably the complete explanation of the problem. SMP can never perform well unless all of its threads are balanced (aka "symmetric").
Assuming FAH is using 75% of your logical cores, how much more that 75% total is being used and by what?