Progress decreases and gets stuck at 99.99%

[dszimmerman]

Hi,
I am new to Folding@Home, and I am excited to participate toward this great project.
I have been folding a WU over the past two days, but the progress keeps fluctuating. It will go up to 30% and then suddenly decrease to 9% (without me closing or touching the program). Occasionally, the progress will reach 99.99% and stay there. I have attached the log for reference. If it's of any use, I'm running Ubuntu 64 bit with an Intel Celeron CPU N3050 @ 1.60GHz x 2 processor and an Intel HD Graphics 400. Thank you for your help and for supporting this project!

Code: Select all

*********************** Log Started 2021-02-06T01:44:37Z ***********************
01:44:37:******************************* libFAH ********************************
01:44:37:       Date: Oct 20 2020
01:44:37:       Time: 20:36:39
01:44:37:   Revision: 5ca109d295a6245e2a2f590b3d0085ad5e567aeb
01:44:37:     Branch: master
01:44:37:   Compiler: GNU 8.3.0
01:44:37:    Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
01:44:37:             -fdata-sections -O3 -funroll-loops -fno-pie
01:44:37:   Platform: linux2 5.8.0-1-amd64
01:44:37:       Bits: 64
01:44:37:       Mode: Release
01:44:37:****************************** FAHClient ******************************
01:44:37:    Version: 7.6.21
01:44:37:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
01:44:37:  Copyright: 2020 foldingathome.org
01:44:37:   Homepage: https://foldingathome.org/
01:44:37:       Date: Oct 20 2020
01:44:37:       Time: 20:39:00
01:44:37:   Revision: 6efbf0e138e22d3963e6a291f78dcb9c6422a278
01:44:37:     Branch: master
01:44:37:   Compiler: GNU 8.3.0
01:44:37:    Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
01:44:37:             -fdata-sections -O3 -funroll-loops -fno-pie
01:44:37:   Platform: linux2 5.8.0-1-amd64
01:44:37:       Bits: 64
01:44:37:       Mode: Release
01:44:37:       Args: --child /etc/fahclient/config.xml --run-as fahclient
01:44:37:             --pid-file=/var/run/fahclient.pid --daemon
01:44:37:     Config: /etc/fahclient/config.xml
01:44:37:******************************** CBang ********************************
01:44:37:       Date: Oct 20 2020
01:44:37:       Time: 18:37:59
01:44:37:   Revision: 7e4ce85225d7eaeb775e87c31740181ca603de60
01:44:37:     Branch: master
01:44:37:   Compiler: GNU 8.3.0
01:44:37:    Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
01:44:37:             -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
01:44:37:   Platform: linux2 5.8.0-1-amd64
01:44:37:       Bits: 64
01:44:37:       Mode: Release
01:44:37:******************************* System ********************************
01:44:37:        CPU: Intel(R) Celeron(R) CPU N3050 @ 1.60GHz
01:44:37:     CPU ID: GenuineIntel Family 6 Model 76 Stepping 3
01:44:37:       CPUs: 2
01:44:37:     Memory: 3.76GiB
01:44:37:Free Memory: 3.15GiB
01:44:37:    Threads: POSIX_THREADS
01:44:37: OS Version: 5.4
01:44:37:Has Battery: false
01:44:37: On Battery: false
01:44:37: UTC Offset: -8
01:44:37:        PID: 1294
01:44:37:        CWD: /var/lib/fahclient
01:44:37:         OS: Linux 5.4.0-65-generic x86_64
01:44:37:    OS Arch: AMD64
01:44:37:       GPUs: 0
01:44:37:       CUDA: Not detected: Failed to open dynamic library 'libcuda.so':
01:44:37:             libcuda.so: cannot open shared object file: No such file or
01:44:37:             directory
01:44:37:     OpenCL: Not detected: Failed to open dynamic library 'libOpenCL.so':
01:44:37:             libOpenCL.so: cannot open shared object file: No such file or
01:44:37:             directory
01:44:37:***********************************************************************
01:44:37:<config>
01:44:37:  <!-- Client Control -->
01:44:37:  <fold-anon v='true'/>
01:44:37:
01:44:37:  <!-- Folding Core -->
01:44:37:  <checkpoint v='5'/>
01:44:37:
01:44:37:  <!-- Folding Slot Configuration -->
01:44:37:  <cause v='HIGH_PRIORITY'/>
01:44:37:  <gpu v='false'/>
01:44:37:
01:44:37:  <!-- Network -->
01:44:37:  <proxy v=':8080'/>
01:44:37:
01:44:37:  <!-- Slot Control -->
01:44:37:  <power v='full'/>
01:44:37:
01:44:37:  <!-- User Information -->
01:44:37:  <team v='228216'/>
01:44:37:  <user v='DanDan'/>
01:44:37:
01:44:37:  <!-- Folding Slots -->
01:44:37:  <slot id='0' type='CPU'/>
01:44:37:</config>
01:44:37:Trying to access database...
01:44:37:Successfully acquired database lock
01:44:37:FS00:Initialized folding slot 00: cpu:2
01:44:37:WU00:FS00:Starting
01:44:37:WARNING:WU00:FS00:AS lowered CPUs from 2 to 1
01:44:37:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit-sse2/a7-0.0.19/Core_a7.fah/FahCore_a7 -dir 00 -suffix 01 -version 706 -lifeline 1294 -checkpoint 5 -np 1
01:44:37:WU00:FS00:Started FahCore on PID 1313
01:44:37:WU00:FS00:Core PID:1317
01:44:37:WU00:FS00:FahCore 0xa7 started
01:44:38:WU00:FS00:0xa7:*********************** Log Started 2021-02-06T01:44:37Z ***********************
01:44:38:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
01:44:38:WU00:FS00:0xa7:       Type: 0xa7
01:44:38:WU00:FS00:0xa7:       Core: Gromacs
01:44:38:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 1313 -checkpoint 5 -np 1
01:44:38:WU00:FS00:0xa7:************************************ CBang *************************************
01:44:38:WU00:FS00:0xa7:       Date: Nov 27 2019
01:44:38:WU00:FS00:0xa7:       Time: 11:26:54
01:44:38:WU00:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
01:44:38:WU00:FS00:0xa7:     Branch: master
01:44:38:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
01:44:38:WU00:FS00:0xa7:    Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
01:44:38:WU00:FS00:0xa7:             -fno-pie -fPIC
01:44:38:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
01:44:38:WU00:FS00:0xa7:       Bits: 64
01:44:38:WU00:FS00:0xa7:       Mode: Release
01:44:38:WU00:FS00:0xa7:************************************ System ************************************
01:44:38:WU00:FS00:0xa7:        CPU: Intel(R) Celeron(R) CPU N3050 @ 1.60GHz
01:44:38:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 76 Stepping 3
01:44:38:WU00:FS00:0xa7:       CPUs: 2
01:44:38:WU00:FS00:0xa7:     Memory: 3.76GiB
01:44:38:WU00:FS00:0xa7:Free Memory: 3.15GiB
01:44:38:WU00:FS00:0xa7:    Threads: POSIX_THREADS
01:44:38:WU00:FS00:0xa7: OS Version: 5.4
01:44:38:WU00:FS00:0xa7:Has Battery: false
01:44:38:WU00:FS00:0xa7: On Battery: false
01:44:38:WU00:FS00:0xa7: UTC Offset: -8
01:44:38:WU00:FS00:0xa7:        PID: 1317
01:44:38:WU00:FS00:0xa7:        CWD: /var/lib/fahclient/work
01:44:38:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
01:44:38:WU00:FS00:0xa7:    Version: 0.0.19
01:44:38:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
01:44:38:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
01:44:38:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
01:44:38:WU00:FS00:0xa7:       Date: Nov 26 2019
01:44:38:WU00:FS00:0xa7:       Time: 00:41:43
01:44:38:WU00:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
01:44:38:WU00:FS00:0xa7:     Branch: master
01:44:38:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
01:44:38:WU00:FS00:0xa7:    Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
01:44:38:WU00:FS00:0xa7:             -fno-pie
01:44:38:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
01:44:38:WU00:FS00:0xa7:       Bits: 64
01:44:38:WU00:FS00:0xa7:       Mode: Release
01:44:38:WU00:FS00:0xa7:************************************ Build *************************************
01:44:38:WU00:FS00:0xa7:       SIMD: sse2
01:44:38:WU00:FS00:0xa7:********************************************************************************
01:44:38:WU00:FS00:0xa7:Project: 16927 (Run 5, Clone 940, Gen 93)
01:44:38:WU00:FS00:0xa7:Unit: 0x00000000000000000000000000000000
01:44:38:WU00:FS00:0xa7:Digital signatures verified
01:44:38:WU00:FS00:0xa7:Calling: mdrun -s frame93.tpr -o frame93.trr -cpi state.cpt -cpt 5 -nt 1
01:44:41:WU00:FS00:0xa7:Steps: first=46500000 total=500000
01:44:44:WU00:FS00:0xa7:Completed 84961 out of 500000 steps (16%)
01:45:31:WU00:FS00:0xa7:Completed 85000 out of 500000 steps (17%)
03:19:58:WU00:FS00:0xa7:Completed 90000 out of 500000 steps (18%)
04:14:04:Removing old file 'configs/config-20210205-003544.xml'
04:14:04:Saving configuration to /etc/fahclient/config.xml
04:14:04:<config>
04:14:04:  <!-- Client Control -->
04:14:04:  <fold-anon v='true'/>
04:14:04:
04:14:04:  <!-- Folding Core -->
04:14:04:  <checkpoint v='5'/>
04:14:04:
04:14:04:  <!-- Folding Slot Configuration -->
04:14:04:  <cause v='HIGH_PRIORITY'/>
04:14:04:  <gpu v='false'/>
04:14:04:
04:14:04:  <!-- Network -->
04:14:04:  <proxy v=':8080'/>
04:14:04:
04:14:04:  <!-- Slot Control -->
04:14:04:  <power v='full'/>
04:14:04:
04:14:04:  <!-- User Information -->
04:14:04:  <team v='228216'/>
04:14:04:  <user v='DanDanDan'/>
04:14:04:
04:14:04:  <!-- Folding Slots -->
04:14:04:  <slot id='0' type='CPU'/>
04:14:04:</config>
04:15:05:Removing old file 'configs/config-20210205-004049.xml'
04:15:05:Saving configuration to /etc/fahclient/config.xml
04:15:05:<config>
04:15:05:  <!-- Client Control -->
04:15:05:  <fold-anon v='true'/>
04:15:05:
04:15:05:  <!-- Folding Core -->
04:15:05:  <checkpoint v='5'/>
04:15:05:
04:15:05:  <!-- Folding Slot Configuration -->
04:15:05:  <cause v='HIGH_PRIORITY'/>
04:15:05:  <gpu v='false'/>
04:15:05:
04:15:05:  <!-- Network -->
04:15:05:  <proxy v=':8080'/>
04:15:05:
04:15:05:  <!-- Slot Control -->
04:15:05:  <power v='full'/>
04:15:05:
04:15:05:  <!-- User Information -->
04:15:05:  <team v='228216'/>
04:15:05:  <user v='DanZDanZ'/>
04:15:05:
04:15:05:  <!-- Folding Slots -->
04:15:05:  <slot id='0' type='CPU'/>
04:15:05:</config>
04:33:33:FS00:Paused
04:33:33:FS00:Shutting core down
04:33:33:WU00:FS00:0xa7:Caught signal SIGINT(2) on PID 1317
04:33:33:WU00:FS00:0xa7:Exiting, please wait. . .
04:33:57:WU00:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
04:34:24:Removing old file 'configs/config-20210205-004352.xml'
04:34:24:Saving configuration to /etc/fahclient/config.xml
04:34:24:<config>
04:34:24:  <!-- Client Control -->
04:34:24:  <fold-anon v='true'/>
04:34:24:
04:34:24:  <!-- Folding Core -->
04:34:24:  <checkpoint v='5'/>
04:34:24:
04:34:24:  <!-- Folding Slot Configuration -->
04:34:24:  <cause v='HIGH_PRIORITY'/>
04:34:24:  <gpu v='false'/>
04:34:24:
04:34:24:  <!-- Network -->
04:34:24:  <proxy v=':8080'/>
04:34:24:
04:34:24:  <!-- Slot Control -->
04:34:24:  <power v='full'/>
04:34:24:
04:34:24:  <!-- User Information -->
04:34:24:  <team v='228216'/>
04:34:24:  <user v='DanZDanZ'/>
04:34:24:
04:34:24:  <!-- Folding Slots -->
04:34:24:  <slot id='0' type='CPU'>
04:34:24:    <paused v='true'/>
04:34:24:  </slot>
04:34:24:</config>
04:34:36:FS00:Unpaused
04:34:36:WU00:FS00:Starting
04:34:36:WARNING:WU00:FS00:AS lowered CPUs from 2 to 1
04:34:36:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit-sse2/a7-0.0.19/Core_a7.fah/FahCore_a7 -dir 00 -suffix 01 -version 706 -lifeline 1294 -checkpoint 5 -np 1
04:34:36:WU00:FS00:Started FahCore on PID 6824
04:34:37:WU00:FS00:Core PID:6828
04:34:37:WU00:FS00:FahCore 0xa7 started
04:34:37:WU00:FS00:0xa7:*********************** Log Started 2021-02-06T04:34:37Z ***********************
04:34:37:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
04:34:37:WU00:FS00:0xa7:       Type: 0xa7
04:34:37:WU00:FS00:0xa7:       Core: Gromacs
04:34:37:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 6824 -checkpoint 5 -np 1
04:34:37:WU00:FS00:0xa7:************************************ CBang *************************************
04:34:37:WU00:FS00:0xa7:       Date: Nov 27 2019
04:34:37:WU00:FS00:0xa7:       Time: 11:26:54
04:34:37:WU00:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
04:34:37:WU00:FS00:0xa7:     Branch: master
04:34:37:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
04:34:37:WU00:FS00:0xa7:    Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
04:34:37:WU00:FS00:0xa7:             -fno-pie -fPIC
04:34:37:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
04:34:37:WU00:FS00:0xa7:       Bits: 64
04:34:37:WU00:FS00:0xa7:       Mode: Release
04:34:37:WU00:FS00:0xa7:************************************ System ************************************
04:34:37:WU00:FS00:0xa7:        CPU: Intel(R) Celeron(R) CPU N3050 @ 1.60GHz
04:34:37:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 76 Stepping 3
04:34:37:WU00:FS00:0xa7:       CPUs: 2
04:34:37:WU00:FS00:0xa7:     Memory: 3.76GiB
04:34:37:WU00:FS00:0xa7:Free Memory: 1.06GiB
04:34:37:WU00:FS00:0xa7:    Threads: POSIX_THREADS
04:34:37:WU00:FS00:0xa7: OS Version: 5.4
04:34:37:WU00:FS00:0xa7:Has Battery: false
04:34:37:WU00:FS00:0xa7: On Battery: false
04:34:37:WU00:FS00:0xa7: UTC Offset: -8
04:34:37:WU00:FS00:0xa7:        PID: 6828
04:34:37:WU00:FS00:0xa7:        CWD: /var/lib/fahclient/work
04:34:37:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
04:34:37:WU00:FS00:0xa7:    Version: 0.0.19
04:34:37:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
04:34:37:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
04:34:37:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
04:34:37:WU00:FS00:0xa7:       Date: Nov 26 2019
04:34:37:WU00:FS00:0xa7:       Time: 00:41:43
04:34:37:WU00:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
04:34:37:WU00:FS00:0xa7:     Branch: master
04:34:37:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
04:34:37:WU00:FS00:0xa7:    Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
04:34:37:WU00:FS00:0xa7:             -fno-pie
04:34:37:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
04:34:37:WU00:FS00:0xa7:       Bits: 64
04:34:37:WU00:FS00:0xa7:       Mode: Release
04:34:37:WU00:FS00:0xa7:************************************ Build *************************************
04:34:37:WU00:FS00:0xa7:       SIMD: sse2
04:34:37:WU00:FS00:0xa7:********************************************************************************
04:34:37:WU00:FS00:0xa7:Project: 16927 (Run 5, Clone 940, Gen 93)
04:34:37:WU00:FS00:0xa7:Unit: 0x00000000000000000000000000000000
04:34:37:WU00:FS00:0xa7:Digital signatures verified
04:34:37:WU00:FS00:0xa7:Calling: mdrun -s frame93.tpr -o frame93.trr -cpi state.cpt -cpt 5 -nt 1
04:34:38:WU00:FS00:0xa7:Steps: first=46500000 total=500000
04:34:42:WU00:FS00:0xa7:Completed 93922 out of 500000 steps (18%)
04:35:25:Removing old file 'configs/config-20210205-005059.xml'
04:35:25:Saving configuration to /etc/fahclient/config.xml
04:35:25:<config>
04:35:25:  <!-- Client Control -->
04:35:25:  <fold-anon v='true'/>
04:35:25:
04:35:25:  <!-- Folding Core -->
04:35:25:  <checkpoint v='5'/>
04:35:25:
04:35:25:  <!-- Folding Slot Configuration -->
04:35:25:  <cause v='HIGH_PRIORITY'/>
04:35:25:  <gpu v='false'/>
04:35:25:
04:35:25:  <!-- Network -->
04:35:25:  <proxy v=':8080'/>
04:35:25:
04:35:25:  <!-- Slot Control -->
04:35:25:  <power v='full'/>
04:35:25:
04:35:25:  <!-- User Information -->
04:35:25:  <team v='228216'/>
04:35:25:  <user v='DanZDanZ'/>
04:35:25:
04:35:25:  <!-- Folding Slots -->
04:35:25:  <slot id='0' type='CPU'/>
04:35:25:</config>
04:54:29:WU00:FS00:0xa7:Completed 95000 out of 500000 steps (19%)

[JimboPalmer]

Welcome to Folding@Home.

Sadly the N3050 is not going to have much performance in F@H.
https://en.wikichip.org/wiki/intel/celeron/n3050

It lacks several 'modern' Floating Point Math features. Currently AVX offers twice the speed of SSE2, and AVX2 offers 60% more speed than AVX.
Despite being recent, your CPU lacks AVX and AVX2, so must use the much weaker, older SSE2 code.
04:34:37:WU00:FS00:0xa7: SIMD: sse2
https://en.wikipedia.org/wiki/SSE2
https://en.wikipedia.org/wiki/Advanced_ ... Extensions
While your N3050 is a dual core part, F@H keeps turning it down to a single core

04:34:37:WU00:FS00:0xa7:Calling: mdrun -s frame93.tpr -o frame93.trr -cpi state.cpt -cpt 5 -nt 1
This says that your checkpoint interval is 5% so if your CPU has an error, F@H may drop back by 5% and try again, so you can see 'reverses' in the forward progress.
(it also shows the number of threads as 1)

[bruce]

It's difficult to diagnose your system from what you've posted.

With a processor as old as your Celeron, I'd guess that you've experienced an overheating condition. Dust has problably accumulated on the fins of the heatsink, degrading it's ability to keep a busy CPU from overheating. The thermal paste under the heatsink may also have degraded over the years.

In fact, there once was an error that interrupted processing at about 9% and it's not really progressing beyond that point. The software that reports progress between checkpoints continues reporting progress when, in fact, no progress is being made ... up to 99.99% where it suddenly figures out that it can't go past 100%.

We can't tell if you've gotten a bad WU, or if it's thermally limited. I would dump the WU to rule out the former and/or figure out how to improve your CPU's cooling.

[dszimmerman]

Thanks for the advice and suggestions. I'll try these and see if it helps! Thanks.