16950 slowing down at the end

Moderators: Site Moderators, FAHC Science Team

Post Reply
JimF
Posts: 651
Joined: Thu Jan 21, 2010 2:03 pm

16950 slowing down at the end

Post by JimF »

I don't think that I have ever seen this before. But P16950 on my Ryzen 3950X (Ubuntu 18.04) started at TPF of 3 minutes, and is slowing down at the end to 30 minutes.
I am wondering whether it will finish or not.

Code: Select all

14:59:54:WU00:FS01:Connecting to 129.32.209.203:8080
15:00:12:WU02:FS01:Download 100.00%
15:00:12:WU02:FS01:Download complete
15:00:12:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:16950 run:48 clone:4 gen:12 core:0xa8 unit:0x000000040000000c0000423600000030
15:00:12:WU02:FS01:Starting
15:00:12:WU02:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit-avx2-256/a8-0.0.9/Core_a8.fah/FahCore_a8 -dir 02 -suffix 01 -version 706 -lifeline 1569 -checkpoint 15 -np 31
15:00:12:WU02:FS01:Started FahCore on PID 11871
15:00:12:WU02:FS01:Core PID:11875
15:00:12:WU02:FS01:FahCore 0xa8 started
15:00:12:WU02:FS01:0xa8:*********************** Log Started 2021-01-09T15:00:12Z ***********************
15:00:12:WU02:FS01:0xa8:************************** Gromacs Folding@home Core ***************************
15:00:12:WU02:FS01:0xa8:       Core: Gromacs
15:00:12:WU02:FS01:0xa8:       Type: 0xa8
15:00:12:WU02:FS01:0xa8:    Version: 0.0.9
15:00:12:WU02:FS01:0xa8:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
15:00:12:WU02:FS01:0xa8:  Copyright: 2020 foldingathome.org
15:00:12:WU02:FS01:0xa8:   Homepage: https://foldingathome.org/
15:00:12:WU02:FS01:0xa8:       Date: Oct 28 2020
15:00:12:WU02:FS01:0xa8:       Time: 22:15:07
15:00:12:WU02:FS01:0xa8:   Compiler: GNU 8.3.0
15:00:12:WU02:FS01:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
15:00:12:WU02:FS01:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie
15:00:12:WU02:FS01:0xa8:   Platform: linux2 4.15.0-108-generic
15:00:12:WU02:FS01:0xa8:       Bits: 64
15:00:12:WU02:FS01:0xa8:       Mode: Release
15:00:12:WU02:FS01:0xa8:       SIMD: avx2_256
15:00:12:WU02:FS01:0xa8:     OpenMP: ON
15:00:12:WU02:FS01:0xa8:       CUDA: OFF
15:00:12:WU02:FS01:0xa8:       Args: -dir 02 -suffix 01 -version 706 -lifeline 11871 -checkpoint 15 -np
15:00:12:WU02:FS01:0xa8:             31
15:00:12:WU02:FS01:0xa8:************************************ libFAH ************************************
15:00:12:WU02:FS01:0xa8:       Date: Oct 28 2020
15:00:12:WU02:FS01:0xa8:       Time: 22:12:00
15:00:12:WU02:FS01:0xa8:   Compiler: GNU 8.3.0
15:00:12:WU02:FS01:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
15:00:12:WU02:FS01:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie
15:00:12:WU02:FS01:0xa8:   Platform: linux2 4.15.0-108-generic
15:00:12:WU02:FS01:0xa8:       Bits: 64
15:00:12:WU02:FS01:0xa8:       Mode: Release
15:00:12:WU02:FS01:0xa8:************************************ CBang *************************************
15:00:12:WU02:FS01:0xa8:       Date: Oct 28 2020
15:00:12:WU02:FS01:0xa8:       Time: 22:11:46
15:00:12:WU02:FS01:0xa8:   Compiler: GNU 8.3.0
15:00:12:WU02:FS01:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
15:00:12:WU02:FS01:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
15:00:12:WU02:FS01:0xa8:   Platform: linux2 4.15.0-108-generic
15:00:12:WU02:FS01:0xa8:       Bits: 64
15:00:12:WU02:FS01:0xa8:       Mode: Release
15:00:12:WU02:FS01:0xa8:************************************ System ************************************
15:00:12:WU02:FS01:0xa8:        CPU: AMD Ryzen 9 3950X 16-Core Processor
15:00:12:WU02:FS01:0xa8:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
15:00:12:WU02:FS01:0xa8:       CPUs: 32
15:00:12:WU02:FS01:0xa8:     Memory: 46.99GiB
15:00:12:WU02:FS01:0xa8:Free Memory: 40.08GiB
15:00:12:WU02:FS01:0xa8:    Threads: POSIX_THREADS
15:00:12:WU02:FS01:0xa8: OS Version: 5.4
15:00:12:WU02:FS01:0xa8:Has Battery: false
15:00:12:WU02:FS01:0xa8: On Battery: false
15:00:12:WU02:FS01:0xa8: UTC Offset: -5
15:00:12:WU02:FS01:0xa8:        PID: 11875
15:00:12:WU02:FS01:0xa8:        CWD: /var/lib/fahclient/work
15:00:12:WU02:FS01:0xa8:********************************************************************************
15:00:12:WU02:FS01:0xa8:Project: 16950 (Run 48, Clone 4, Gen 12)
15:00:12:WU02:FS01:0xa8:Unit: 0x00000000000000000000000000000000
15:00:12:WU02:FS01:0xa8:Reading tar file core.xml
15:00:12:WU02:FS01:0xa8:Reading tar file frame12.tpr
15:00:12:WU02:FS01:0xa8:Digital signatures verified
15:00:12:WU02:FS01:0xa8:Calling: mdrun -c frame12.gro -s frame12.tpr -x frame12.xtc -cpt 15 -nt 31 -ntmpi 1
15:00:12:WU02:FS01:0xa8:Steps: first=60000000 total=65000000
15:00:13:WU02:FS01:0xa8:Completed 1 out of 5000000 steps (0%)
15:00:28:WU00:FS01:Upload complete
15:00:28:WU00:FS01:Server responded WORK_ACK (400)
15:00:28:WU00:FS01:Final credit estimate, 6057.00 points
15:00:28:WU00:FS01:Cleaning up
15:03:10:WU02:FS01:0xa8:Completed 50000 out of 5000000 steps (1%)
15:06:06:WU02:FS01:0xa8:Completed 100000 out of 5000000 steps (2%)
15:09:03:WU02:FS01:0xa8:Completed 150000 out of 5000000 steps (3%)
15:11:59:WU02:FS01:0xa8:Completed 200000 out of 5000000 steps (4%)
15:14:54:WU02:FS01:0xa8:Completed 250000 out of 5000000 steps (5%)
15:17:49:WU02:FS01:0xa8:Completed 300000 out of 5000000 steps (6%)
15:20:44:WU02:FS01:0xa8:Completed 350000 out of 5000000 steps (7%)
15:23:40:WU02:FS01:0xa8:Completed 400000 out of 5000000 steps (8%)
15:26:36:WU02:FS01:0xa8:Completed 450000 out of 5000000 steps (9%)
15:29:31:WU02:FS01:0xa8:Completed 500000 out of 5000000 steps (10%)
15:32:26:WU02:FS01:0xa8:Completed 550000 out of 5000000 steps (11%)
15:35:21:WU02:FS01:0xa8:Completed 600000 out of 5000000 steps (12%)
15:38:16:WU02:FS01:0xa8:Completed 650000 out of 5000000 steps (13%)
15:41:13:WU02:FS01:0xa8:Completed 700000 out of 5000000 steps (14%)
15:44:07:WU02:FS01:0xa8:Completed 750000 out of 5000000 steps (15%)
******************************* Date: 2021-01-09 *******************************
15:47:03:WU02:FS01:0xa8:Completed 800000 out of 5000000 steps (16%)
15:49:58:WU02:FS01:0xa8:Completed 850000 out of 5000000 steps (17%)
15:52:54:WU02:FS01:0xa8:Completed 900000 out of 5000000 steps (18%)
15:55:51:WU02:FS01:0xa8:Completed 950000 out of 5000000 steps (19%)
15:58:44:WU02:FS01:0xa8:Completed 1000000 out of 5000000 steps (20%)
16:01:39:WU02:FS01:0xa8:Completed 1050000 out of 5000000 steps (21%)
16:04:35:WU02:FS01:0xa8:Completed 1100000 out of 5000000 steps (22%)
16:07:31:WU02:FS01:0xa8:Completed 1150000 out of 5000000 steps (23%)
16:10:28:WU02:FS01:0xa8:Completed 1200000 out of 5000000 steps (24%)
16:13:26:WU02:FS01:0xa8:Completed 1250000 out of 5000000 steps (25%)
16:16:22:WU02:FS01:0xa8:Completed 1300000 out of 5000000 steps (26%)
16:19:17:WU02:FS01:0xa8:Completed 1350000 out of 5000000 steps (27%)
16:22:12:WU02:FS01:0xa8:Completed 1400000 out of 5000000 steps (28%)
16:25:08:WU02:FS01:0xa8:Completed 1450000 out of 5000000 steps (29%)
16:28:04:WU02:FS01:0xa8:Completed 1500000 out of 5000000 steps (30%)
16:30:59:WU02:FS01:0xa8:Completed 1550000 out of 5000000 steps (31%)
16:33:54:WU02:FS01:0xa8:Completed 1600000 out of 5000000 steps (32%)
16:36:49:WU02:FS01:0xa8:Completed 1650000 out of 5000000 steps (33%)
16:39:45:WU02:FS01:0xa8:Completed 1700000 out of 5000000 steps (34%)
16:42:41:WU02:FS01:0xa8:Completed 1750000 out of 5000000 steps (35%)
16:45:36:WU02:FS01:0xa8:Completed 1800000 out of 5000000 steps (36%)
16:48:33:WU02:FS01:0xa8:Completed 1850000 out of 5000000 steps (37%)
16:51:28:WU02:FS01:0xa8:Completed 1900000 out of 5000000 steps (38%)
16:54:24:WU02:FS01:0xa8:Completed 1950000 out of 5000000 steps (39%)
16:57:21:WU02:FS01:0xa8:Completed 2000000 out of 5000000 steps (40%)
17:00:16:WU02:FS01:0xa8:Completed 2050000 out of 5000000 steps (41%)
17:03:11:WU02:FS01:0xa8:Completed 2100000 out of 5000000 steps (42%)
17:06:07:WU02:FS01:0xa8:Completed 2150000 out of 5000000 steps (43%)
17:09:03:WU02:FS01:0xa8:Completed 2200000 out of 5000000 steps (44%)
17:11:59:WU02:FS01:0xa8:Completed 2250000 out of 5000000 steps (45%)
17:14:56:WU02:FS01:0xa8:Completed 2300000 out of 5000000 steps (46%)
17:17:52:WU02:FS01:0xa8:Completed 2350000 out of 5000000 steps (47%)
17:20:48:WU02:FS01:0xa8:Completed 2400000 out of 5000000 steps (48%)
17:23:43:WU02:FS01:0xa8:Completed 2450000 out of 5000000 steps (49%)
17:26:38:WU02:FS01:0xa8:Completed 2500000 out of 5000000 steps (50%)
17:29:35:WU02:FS01:0xa8:Completed 2550000 out of 5000000 steps (51%)
17:32:31:WU02:FS01:0xa8:Completed 2600000 out of 5000000 steps (52%)
17:35:27:WU02:FS01:0xa8:Completed 2650000 out of 5000000 steps (53%)
17:38:23:WU02:FS01:0xa8:Completed 2700000 out of 5000000 steps (54%)
17:41:18:WU02:FS01:0xa8:Completed 2750000 out of 5000000 steps (55%)
17:44:14:WU02:FS01:0xa8:Completed 2800000 out of 5000000 steps (56%)
17:47:09:WU02:FS01:0xa8:Completed 2850000 out of 5000000 steps (57%)
17:50:04:WU02:FS01:0xa8:Completed 2900000 out of 5000000 steps (58%)
17:52:58:WU02:FS01:0xa8:Completed 2950000 out of 5000000 steps (59%)
17:55:54:WU02:FS01:0xa8:Completed 3000000 out of 5000000 steps (60%)
17:58:50:WU02:FS01:0xa8:Completed 3050000 out of 5000000 steps (61%)
18:01:45:WU02:FS01:0xa8:Completed 3100000 out of 5000000 steps (62%)
18:04:42:WU02:FS01:0xa8:Completed 3150000 out of 5000000 steps (63%)
18:07:36:WU02:FS01:0xa8:Completed 3200000 out of 5000000 steps (64%)
18:10:33:WU02:FS01:0xa8:Completed 3250000 out of 5000000 steps (65%)
18:13:28:WU02:FS01:0xa8:Completed 3300000 out of 5000000 steps (66%)
18:16:24:WU02:FS01:0xa8:Completed 3350000 out of 5000000 steps (67%)
18:19:21:WU02:FS01:0xa8:Completed 3400000 out of 5000000 steps (68%)
18:22:17:WU02:FS01:0xa8:Completed 3450000 out of 5000000 steps (69%)
18:25:11:WU02:FS01:0xa8:Completed 3500000 out of 5000000 steps (70%)
18:28:07:WU02:FS01:0xa8:Completed 3550000 out of 5000000 steps (71%)
18:31:02:WU02:FS01:0xa8:Completed 3600000 out of 5000000 steps (72%)
18:33:57:WU02:FS01:0xa8:Completed 3650000 out of 5000000 steps (73%)
18:36:53:WU02:FS01:0xa8:Completed 3700000 out of 5000000 steps (74%)
18:39:48:WU02:FS01:0xa8:Completed 3750000 out of 5000000 steps (75%)
18:42:43:WU02:FS01:0xa8:Completed 3800000 out of 5000000 steps (76%)
18:45:38:WU02:FS01:0xa8:Completed 3850000 out of 5000000 steps (77%)
18:48:30:WU02:FS01:0xa8:Completed 3900000 out of 5000000 steps (78%)
18:51:26:WU02:FS01:0xa8:Completed 3950000 out of 5000000 steps (79%)
18:54:21:WU02:FS01:0xa8:Completed 4000000 out of 5000000 steps (80%)
18:57:16:WU02:FS01:0xa8:Completed 4050000 out of 5000000 steps (81%)
19:00:13:WU02:FS01:0xa8:Completed 4100000 out of 5000000 steps (82%)
19:03:09:WU02:FS01:0xa8:Completed 4150000 out of 5000000 steps (83%)
19:06:06:WU02:FS01:0xa8:Completed 4200000 out of 5000000 steps (84%)
19:09:02:WU02:FS01:0xa8:Completed 4250000 out of 5000000 steps (85%)
19:11:57:WU02:FS01:0xa8:Completed 4300000 out of 5000000 steps (86%)
19:14:53:WU02:FS01:0xa8:Completed 4350000 out of 5000000 steps (87%)
19:17:49:WU02:FS01:0xa8:Completed 4400000 out of 5000000 steps (88%)
19:20:45:WU02:FS01:0xa8:Completed 4450000 out of 5000000 steps (89%)
19:23:42:WU02:FS01:0xa8:Completed 4500000 out of 5000000 steps (90%)
19:26:37:WU02:FS01:0xa8:Completed 4550000 out of 5000000 steps (91%)
19:29:34:WU02:FS01:0xa8:Completed 4600000 out of 5000000 steps (92%)
19:32:30:WU02:FS01:0xa8:Completed 4650000 out of 5000000 steps (93%)
19:35:26:WU02:FS01:0xa8:Completed 4700000 out of 5000000 steps (94%)
19:38:23:WU02:FS01:0xa8:Completed 4750000 out of 5000000 steps (95%)
19:40:41:FS01:Finishing
19:41:19:WU02:FS01:0xa8:Completed 4800000 out of 5000000 steps (96%)
20:09:23:WU02:FS01:0xa8:Completed 4850000 out of 5000000 steps (97%)
20:39:35:WU02:FS01:0xa8:Completed 4900000 out of 5000000 steps (98%)
Top
JimF
Posts: 651
Joined: Thu Jan 21, 2010 2:03 pm

Re: 16950 slowing down at the end

Post by JimF »

OK, it finally finished up and uploaded apparently OK. It was just a bit unusual.

Code: Select all

20:48:45:FS01:Finishing
21:11:32:WU02:FS01:0xa8:Completed 4950000 out of 5000000 steps (99%)
******************************* Date: 2021-01-09 *******************************
21:49:56:WU02:FS01:0xa8:Completed 5000000 out of 5000000 steps (100%)
21:49:56:WU02:FS01:0xa8:Saving result file ../logfile_01.txt
21:49:56:WU02:FS01:0xa8:Saving result file dhdl.xvg
21:49:56:WU02:FS01:0xa8:Saving result file frame12.gro
21:49:56:WU02:FS01:0xa8:Saving result file frame12.xtc
21:49:56:WU02:FS01:0xa8:Saving result file md.log
21:49:56:WU02:FS01:0xa8:Saving result file science.log
21:49:56:WU02:FS01:0xa8:Saving result file state.cpt
21:49:56:WU02:FS01:0xa8:Folding@home Core Shutdown: FINISHED_UNIT
21:49:56:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
21:49:56:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:16950 run:48 clone:4 gen:12 core:0xa8 unit:0x000000040000000c0000423600000030
21:49:56:WU02:FS01:Uploading 21.69MiB to 129.32.209.203
21:49:56:WU02:FS01:Connecting to 129.32.209.203:8080
21:50:02:WU02:FS01:Upload 54.74%
21:50:43:WU02:FS01:Upload complete
21:50:43:WU02:FS01:Server responded WORK_ACK (400)
21:50:43:WU02:FS01:Final credit estimate, 66427.00 points
21:50:43:WU02:FS01:Cleaning up
Top
psaam0001
Posts: 378
Joined: Mon May 18, 2020 2:02 am
Location: Ruckersville, Virginia, USA

Re: 16950 slowing down at the end

Post by psaam0001 »

I've seen some CPU WU's on each of my systems appear to start slow, but then they appear to reach a decent completion speed after the 2% mark is reached.

But I'm not worried. I'll just keep calm and fold on. After all, COVID's demise is on my schedule--until further notice.

Paul
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: 16950 slowing down at the end

Post by bruce »

FAHCore_a8 is relatively new to me, but with some older cores and with older projects, this was explained by noting that the shape of the protein is changing. The motions of each atom is influenced mostly by the nearby atoms. If the atoms are sparse, the calculations are faster than if the atoms are closely packed because more calculations must be performed during each time-step.

In this particular case, that may or may not be the reason.

It does make it difficult to assign an appropriate number of points to all WUs in the project.
Top
JimF
Posts: 651
Joined: Thu Jan 21, 2010 2:03 pm

Re: 16950 slowing down at the end

Post by JimF »

bruce wrote:It does make it difficult to assign an appropriate number of points to all WUs in the project.
True, that was another anomaly, but not one that I am worried about. I get concerned about "stuck" work units, which are prevalent on some projects.
I have not seen them here though. If it is just a result of the workload, I can live with that.
Top
Ultrafire3
Posts: 2
Joined: Sat Oct 31, 2020 7:12 pm

Re: 16950 slowing down at the end

Post by Ultrafire3 »

I've had this issue with GPU WUs if I suddenly do something GPU intensive and then stop. Even after I free up the GPU, the TPF stays tripled in length.
Top
PantherX
Site Moderator
Posts: 6986
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:
Contact PantherX

Re: 16950 slowing down at the end

Post by PantherX »

Welcome to the F@H Forum Ultrafire3,

Please note that the triple TPF will gradually come back to normal once at least 4% of the WU has been completed without any interruptions. Generally, the TFP being displayed uses the last 3% to calculate it thus, the "lag" in updating it.
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Top
Post Reply

Return to “Issues with a specific WU”