Project 7611 (0, 23, 259) - High TPF

[ChelseaOilman]

This sounds just like the same issue with some of the bigadv WUs. In the case of the bigadv WUs you can look in the log and see the number of steps are doubled and your frame times double. With these WUs it looks like you can only tell you have a problem by noticing your frame times double. Is this just a coincidence or is something going on that creates this problem?

[tjlane]

Hi All,

We are actively looking into the source of this issue - right now, we are not sure. I have alerted the rest of the project managers to check their projects for similar issues, so if there is a deeper problem we should find it.

Thanks,

TJ

[Windego]

Good Morning,

Running a 3770K @ 4.5, W7 Pro x64, noticed this morning this project was downloaded and has an extremely high TFP (33-35mins) with low estimated points credit (3520) and ETA of 2.05 days currently. I've searched and read problems about this project before previously and others having extremely high TPF like mine, so I am wondering if there is indeed a problem again with this WU. I've restarted the system to ensure there are no issues and the core is at 99% CPU usage to no avail in correcting the high TPF. I've read others typically are around 5mins with this project so I am curious as to should I just let it continue or stop it? Log is from after my reboot.

Code: Select all

*********************** Log Started 2012-10-02T14:08:33Z ***********************
14:08:33:WU01:FS00:Starting
14:08:33:WU01:FS00:Running FahCore: F:\FAHClient/FAHCoreWrapper.exe f:/fahdata/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 01 -suffix 01 -version 701 -lifeline 4176 -checkpoint 15 -np 8
14:08:33:WU01:FS00:Started FahCore on PID 4496
14:08:34:WU01:FS00:Core PID:4580
14:08:34:WU01:FS00:FahCore 0xa4 started
14:08:34:WU01:FS00:0xa4:
14:08:34:WU01:FS00:0xa4:*------------------------------*
14:08:34:WU01:FS00:0xa4:Folding@Home Gromacs GB Core
14:08:34:WU01:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
14:08:34:WU01:FS00:0xa4:
14:08:34:WU01:FS00:0xa4:Preparing to commence simulation
14:08:34:WU01:FS00:0xa4:- Looking at optimizations...
14:08:34:WU01:FS00:0xa4:- Files status OK
14:08:34:WU01:FS00:0xa4:- Expanded 29850 -> 644556 (decompressed 2159.3 percent)
14:08:34:WU01:FS00:0xa4:Called DecompressByteArray: compressed_data_size=29850 data_size=644556, decompressed_data_size=644556 diff=0
14:08:34:WU01:FS00:0xa4:- Digital signature verified
14:08:34:WU01:FS00:0xa4:
14:08:34:WU01:FS00:0xa4:Project: 7611 (Run 0, Clone 23, Gen 259)
14:08:34:WU01:FS00:0xa4:
14:08:34:WU01:FS00:0xa4:Assembly optimizations on if available.
14:08:34:WU01:FS00:0xa4:Entering M.D.
14:08:40:WU01:FS00:0xa4:Using Gromacs checkpoints
14:08:40:WU01:FS00:0xa4:Mapping NT from 8 to 8 
14:08:40:WU01:FS00:0xa4:Resuming from checkpoint
14:08:40:WU01:FS00:0xa4:Verified 01/wudata_01.log
14:08:40:WU01:FS00:0xa4:Verified 01/wudata_01.trr
14:08:40:WU01:FS00:0xa4:Verified 01/wudata_01.xtc
14:08:40:WU01:FS00:0xa4:Verified 01/wudata_01.edr
14:08:40:WU01:FS00:0xa4:Completed 171240 out of 2000000 steps  (8%)
*********************** Log Started 2012-10-02T14:08:33Z ***********************
14:08:33:WU01:FS00:Starting
14:08:33:WU01:FS00:Running FahCore: F:\FAHClient/FAHCoreWrapper.exe f:/fahdata/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 01 -suffix 01 -version 701 -lifeline 4176 -checkpoint 15 -np 8
14:08:33:WU01:FS00:Started FahCore on PID 4496
14:08:34:WU01:FS00:Core PID:4580
14:08:34:WU01:FS00:FahCore 0xa4 started
14:08:34:WU01:FS00:0xa4:
14:08:34:WU01:FS00:0xa4:*------------------------------*
14:08:34:WU01:FS00:0xa4:Folding@Home Gromacs GB Core
14:08:34:WU01:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
14:08:34:WU01:FS00:0xa4:
14:08:34:WU01:FS00:0xa4:Preparing to commence simulation
14:08:34:WU01:FS00:0xa4:- Looking at optimizations...
14:08:34:WU01:FS00:0xa4:- Files status OK
14:08:34:WU01:FS00:0xa4:- Expanded 29850 -> 644556 (decompressed 2159.3 percent)
14:08:34:WU01:FS00:0xa4:Called DecompressByteArray: compressed_data_size=29850 data_size=644556, decompressed_data_size=644556 diff=0
14:08:34:WU01:FS00:0xa4:- Digital signature verified
14:08:34:WU01:FS00:0xa4:
14:08:34:WU01:FS00:0xa4:Project: 7611 (Run 0, Clone 23, Gen 259)
14:08:34:WU01:FS00:0xa4:
14:08:34:WU01:FS00:0xa4:Assembly optimizations on if available.
14:08:34:WU01:FS00:0xa4:Entering M.D.
14:08:40:WU01:FS00:0xa4:Using Gromacs checkpoints
14:08:40:WU01:FS00:0xa4:Mapping NT from 8 to 8 
14:08:40:WU01:FS00:0xa4:Resuming from checkpoint
14:08:40:WU01:FS00:0xa4:Verified 01/wudata_01.log
14:08:40:WU01:FS00:0xa4:Verified 01/wudata_01.trr
14:08:40:WU01:FS00:0xa4:Verified 01/wudata_01.xtc
14:08:40:WU01:FS00:0xa4:Verified 01/wudata_01.edr
14:08:40:WU01:FS00:0xa4:Completed 171240 out of 2000000 steps  (8%)
14:23:32:WU01:FS00:0xa4:Completed 180000 out of 2000000 steps  (9%)
*********************** Log Started 2012-10-02T14:08:33Z ***********************
14:08:33:WU01:FS00:Starting
14:08:33:WU01:FS00:Running FahCore: F:\FAHClient/FAHCoreWrapper.exe f:/fahdata/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 01 -suffix 01 -version 701 -lifeline 4176 -checkpoint 15 -np 8
14:08:33:WU01:FS00:Started FahCore on PID 4496
14:08:34:WU01:FS00:Core PID:4580
14:08:34:WU01:FS00:FahCore 0xa4 started
14:08:34:WU01:FS00:0xa4:
14:08:34:WU01:FS00:0xa4:*------------------------------*
14:08:34:WU01:FS00:0xa4:Folding@Home Gromacs GB Core
14:08:34:WU01:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
14:08:34:WU01:FS00:0xa4:
14:08:34:WU01:FS00:0xa4:Preparing to commence simulation
14:08:34:WU01:FS00:0xa4:- Looking at optimizations...
14:08:34:WU01:FS00:0xa4:- Files status OK
14:08:34:WU01:FS00:0xa4:- Expanded 29850 -> 644556 (decompressed 2159.3 percent)
14:08:34:WU01:FS00:0xa4:Called DecompressByteArray: compressed_data_size=29850 data_size=644556, decompressed_data_size=644556 diff=0
14:08:34:WU01:FS00:0xa4:- Digital signature verified
14:08:34:WU01:FS00:0xa4:
14:08:34:WU01:FS00:0xa4:Project: 7611 (Run 0, Clone 23, Gen 259)
14:08:34:WU01:FS00:0xa4:
14:08:34:WU01:FS00:0xa4:Assembly optimizations on if available.
14:08:34:WU01:FS00:0xa4:Entering M.D.
14:08:40:WU01:FS00:0xa4:Using Gromacs checkpoints
14:08:40:WU01:FS00:0xa4:Mapping NT from 8 to 8 
14:08:40:WU01:FS00:0xa4:Resuming from checkpoint
14:08:40:WU01:FS00:0xa4:Verified 01/wudata_01.log
14:08:40:WU01:FS00:0xa4:Verified 01/wudata_01.trr
14:08:40:WU01:FS00:0xa4:Verified 01/wudata_01.xtc
14:08:40:WU01:FS00:0xa4:Verified 01/wudata_01.edr
14:08:40:WU01:FS00:0xa4:Completed 171240 out of 2000000 steps  (8%)
14:23:32:WU01:FS00:0xa4:Completed 180000 out of 2000000 steps  (9%)
14:57:04:WU01:FS00:0xa4:Completed 200000 out of 2000000 steps  (10%)

Mod edit - added Code tags to log

[Joe_H]

What is the normal TPF on your system processing WU's from project 7611? Do double check that there is no other process continuously taking a small percentage of CPU time, that can have a disproportionate effect on the TPF. Could you also post the beginning of the log that shows the System configuration? It does look though that you may have received one of these problem WU's that were identified previously. I have merged your post with the prior thread on this topic.

[Windego]

What is the normal TPF on your system processing WU's from project 7611? Do double check that there is no other process continuously taking a small percentage of CPU time, that can have a disproportionate effect on the TPF. Could you also post the beginning of the log that shows the System configuration? It does look though that you may have received one of these problem WU's that were identified previously. I have merged your post with the prior thread on this topic.

Here's the config. I've turned off the GPU slot and there is no difference in TFP. Next highest process is System Idle.

Code: Select all

*********************** Log Started 2012-10-02T14:08:33Z ***********************
14:08:33:************************* Folding@home Client *************************
14:08:33:      Website: http://folding.stanford.edu/
14:08:33:    Copyright: (c) 2009-2012 Stanford University
14:08:33:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
14:08:33:         Args: --lifeline 3384 --command-port=36330
14:08:33:       Config: f:/fahdata/config.xml
14:08:33:******************************** Build ********************************
14:08:33:      Version: 7.1.52
14:08:33:         Date: Mar 20 2012
14:08:33:         Time: 19:37:42
14:08:33:      SVN Rev: 3515
14:08:33:       Branch: fah/trunk/client
14:08:33:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
14:08:33:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
14:08:33:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT
14:08:33:     Platform: win32 XP
14:08:33:         Bits: 32
14:08:33:         Mode: Release
14:08:33:******************************* System ********************************
14:08:33:          CPU: Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz
14:08:33:       CPU ID: GenuineIntel Family 6 Model 58 Stepping 9
14:08:33:         CPUs: 8
14:08:33:       Memory: 15.95GiB
14:08:33:  Free Memory: 14.26GiB
14:08:33:      Threads: WINDOWS_THREADS
14:08:33:   On Battery: false
14:08:33:   UTC offset: -6
14:08:33:          PID: 4176
14:08:33:          CWD: f:/fahdata
14:08:33:           OS: Windows 7 Professional
14:08:33:      OS Arch: AMD64
14:08:33:         GPUs: 1
14:08:33:        GPU 0: NVIDIA:3 GK107 [GeForce GTX 670]
14:08:33:         CUDA: 3.0
14:08:33:  CUDA Driver: 5000
14:08:33:Win32 Service: false
14:08:33:***********************************************************************
14:08:33:<config>
14:08:33:  <!-- Network -->
14:08:33:  <proxy v=':8080'/>
14:08:33:
14:08:33:  <!-- User Information -->
14:08:33:  <passkey v='********************************'/>
14:08:33:  <team v='33'/>
14:08:33:  <user v='Windego'/>
14:08:33:
14:08:33:  <!-- Folding Slots -->
14:08:33:  <slot id='0' type='SMP'>
14:08:33:    <client-type v='advanced'/>
14:08:33:    <cpus v='-1'/>
14:08:33:    <max-packet-size v='small'/>
14:08:33:  </slot>
14:08:33:  <slot id='1' type='GPU'>
14:08:33:    <client-type v='beta'/>
14:08:33:    <gpu-index v='0'/>
14:08:33:  </slot>
14:08:33:</config>

[patonb]

Got this unit this morning and am running 59min tpf on my i7 at 4.0Ghz for a whooping ppd of 700. Usually its 15000ppd

Code: Select all

*********************** Log Started 2012-10-28T16:22:35Z ***********************
16:22:35:************************* Folding@home Client *************************
16:22:35:      Website: http://folding.stanford.edu/
16:22:35:    Copyright: (c) 2009-2012 Stanford University
16:22:35:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:22:35:         Args: --lifeline 4736 --command-port=36330
16:22:35:       Config: C:/ProgramData/FAHClient/config.xml
16:22:35:******************************** Build ********************************
16:22:35:      Version: 7.2.9
16:22:35:         Date: Oct 3 2012
16:22:35:         Time: 18:05:48
16:22:35:      SVN Rev: 3578
16:22:35:       Branch: fah/trunk/client
16:22:35:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
16:22:35:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
16:22:35:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
16:22:35:     Platform: win32 XP
16:22:35:         Bits: 32
16:22:35:         Mode: Release
16:22:35:******************************* System ********************************
16:22:35:          CPU: Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
16:22:35:       CPU ID: GenuineIntel Family 6 Model 26 Stepping 5
16:22:35:         CPUs: 8
16:22:35:       Memory: 6.00GiB
16:22:35:  Free Memory: 4.30GiB
16:22:35:      Threads: WINDOWS_THREADS
16:22:35:   On Battery: false
16:22:35:   UTC offset: -4
16:22:35:          PID: 4080
16:22:35:          CWD: C:/ProgramData/FAHClient
16:22:35:           OS: Windows (TM) Vista Ultimate
16:22:35:      OS Arch: AMD64
16:22:35:         GPUs: 1
16:22:35:        GPU 0: NVIDIA:2 GF114 [GeForce GTX 560 Ti]
16:22:35:         CUDA: 2.1
16:22:35:  CUDA Driver: 4020
16:22:35:Win32 Service: false
16:22:35:***********************************************************************
16:22:35:<config>
16:22:35:  <!-- Network -->
16:22:35:  <proxy v=':8080'/>
16:22:35:
16:22:35:  <!-- Remote Command Server -->
16:22:35:  <command-allow v='127.0.0.1 192.168.1.1-192.168.1.255'/>
16:22:35:  <command-allow-no-pass v='127.0.0.1 192.168.1.1-192.168.1.255'/>
16:22:35:
16:22:35:  <!-- User Information -->
16:22:35:  <passkey v='********************************'/>
16:22:35:  <team v='12864'/>
16:22:35:  <user v='patonb'/>
16:22:35:
16:22:35:  <!-- Folding Slots -->
16:22:35:  <slot id='1' type='GPU'/>
16:22:35:  <slot id='0' type='SMP'>
16:22:35:    <cpus v='-1'/>
16:22:35:  </slot>
16:22:35:</config>

Code: Select all

11:19:27:WU01:FS00:0xa4:
11:19:27:WU01:FS00:0xa4:Project: 7611 (Run 2, Clone 72, Gen 153)
11:19:27:WU01:FS00:0xa4:
11:19:27:WU01:FS00:0xa4:Assembly optimizations on if available.
11:19:27:WU01:FS00:0xa4:Entering M.D.
11:19:32:WU00:FS00:Upload 21.78%
11:19:32:WU01:FS00:0xa4:Mapping NT from 8 to 8 
11:19:33:WU01:FS00:0xa4:Completed 0 out of 2000000 steps  (0%)
11:19:38:WU00:FS00:Upload 39.93%
11:19:44:WU00:FS00:Upload 61.71%
11:19:50:WU00:FS00:Upload 79.86%
11:19:56:WU00:FS00:Upload 100.00%
11:19:56:WU00:FS00:Upload complete
11:19:56:WU00:FS00:Server responded WORK_ACK (400)
11:19:56:WU00:FS00:Final credit estimate, 1535.00 points
11:19:56:WU00:FS00:Cleaning up
12:17:44:WU01:FS00:0xa4:Completed 20000 out of 2000000 steps  (1%)
13:16:06:WU01:FS00:0xa4:Completed 40000 out of 2000000 steps  (2%)
14:14:36:WU01:FS00:0xa4:Completed 60000 out of 2000000 steps  (3%)
15:00:07:Lost lifeline PID 4736, exiting
15:00:08:FS00:Shutting core down
15:00:08:Server connection id=3 ended
15:00:16:Clean exit
15:00:16:WU01:FS00:0xa4:Client no longer detected. Shutting down core 
15:00:16:WU01:FS00:0xa4:
15:00:16:WU01:FS00:0xa4:Folding@home Core Shutdown: CLIENT_DIED

[Joe_H]

Unless there was something else taking up CPU and causing the WU to run slow, you may have run into a Project 7611 WU with the same problem as previously identified in this thread. I have merged your post with the existing thread on high TPF WU's.

[patonb]

Thanks..

My other 7611 according to hfm take 7 1/2 mins tpf.

I rebooted and its still 58mins+ tpf.

Is this a dump and go, or am I needing to wait it out. though 4 days is extremely long time.

[bruce]

If you're certain it's the WU and not some other task stealing CPU time from the FahCore, it's a Report/Dump/Go. You've completed step 1.

[patonb]

a4 core using 99-100% cpu.

So Ker-plop

[Joe_H]

Someone else has completed the WU and turned it in, so getting rid of it should not effect the science. Normally I would not recommend dumping, but I don't think it will matter this time.

P.S. I will send a message to tjlane to let him know.

[patonb]

As a side note, I'm running right on par again with my new unit picked up.

I too don't dump units just cause of points.

[tjlane]

Hi All,

I am sorry for the re-occurence of this issue. Unfortunately I can't always guarantee this issue won't crop up, but I've done my best to mitigate it. The root cause is a bug in the A4 core, and I've reported it to the correct people. I think that because this issue occurs in >.1% of WUs, it hasn't been a priority for our dev team, which is usually swamped.

This issue will be resolved eventually, but in the mean time please feel free to dump these WUs. Not only are the points bad, but when this issue occurs the returned WU is meaningless. Therefore it's beneficial for the science (and the donor) to dump the WU.

I do apologize and will bug the core dev team again. Please let me know if there are any further questions or concerns.

Thanks,

TJ

[patonb]

Thanks for keeping up on it.

[Jesse_V]

A friend of mine got this on v2.27 of the A4 core. TPF is 2 hours 40 minutes, and the WU -- 7645 (489, 0, 14) -- will complete in 9 days, past the deadline.