WUs that slow way down at the end

Moderators: Site Moderators, FAHC Science Team

Post Reply
bikeaddict
Posts: 210
Joined: Sun May 03, 2020 1:20 am

WUs that slow way down at the end

Post by bikeaddict »

Occasionally I see WUs that start with PPD estimate around 400-500K and then suddenly drop to 20-30K for the last few percent and it takes many hours to finish.

Project: 12422 (Run 66, Clone 0, Gen 23) is an example that I have currently running.

Here are the time intervals by percent:

Code: Select all

18:03:01:WU00:FS01:0xa8:Steps: first=115000000 total=120000000
18:03:05:WU00:FS01:0xa8:Completed 1 out of 5000000 steps (0%)
18:13:24:WU00:FS01:0xa8:Completed 50000 out of 5000000 steps (1%)
18:23:44:WU00:FS01:0xa8:Completed 100000 out of 5000000 steps (2%)
18:34:07:WU00:FS01:0xa8:Completed 150000 out of 5000000 steps (3%)
18:44:27:WU00:FS01:0xa8:Completed 200000 out of 5000000 steps (4%)
18:54:48:WU00:FS01:0xa8:Completed 250000 out of 5000000 steps (5%)
19:05:10:WU00:FS01:0xa8:Completed 300000 out of 5000000 steps (6%)
19:15:31:WU00:FS01:0xa8:Completed 350000 out of 5000000 steps (7%)
19:25:51:WU00:FS01:0xa8:Completed 400000 out of 5000000 steps (8%)
19:36:12:WU00:FS01:0xa8:Completed 450000 out of 5000000 steps (9%)
19:46:33:WU00:FS01:0xa8:Completed 500000 out of 5000000 steps (10%)
19:56:54:WU00:FS01:0xa8:Completed 550000 out of 5000000 steps (11%)
20:07:15:WU00:FS01:0xa8:Completed 600000 out of 5000000 steps (12%)
20:17:36:WU00:FS01:0xa8:Completed 650000 out of 5000000 steps (13%)
20:27:57:WU00:FS01:0xa8:Completed 700000 out of 5000000 steps (14%)
20:38:18:WU00:FS01:0xa8:Completed 750000 out of 5000000 steps (15%)
20:48:40:WU00:FS01:0xa8:Completed 800000 out of 5000000 steps (16%)
20:59:00:WU00:FS01:0xa8:Completed 850000 out of 5000000 steps (17%)
21:09:15:WU00:FS01:0xa8:Completed 900000 out of 5000000 steps (18%)
21:19:32:WU00:FS01:0xa8:Completed 950000 out of 5000000 steps (19%)
21:29:51:WU00:FS01:0xa8:Completed 1000000 out of 5000000 steps (20%)
21:40:10:WU00:FS01:0xa8:Completed 1050000 out of 5000000 steps (21%)
21:50:33:WU00:FS01:0xa8:Completed 1100000 out of 5000000 steps (22%)
22:00:54:WU00:FS01:0xa8:Completed 1150000 out of 5000000 steps (23%)
22:11:15:WU00:FS01:0xa8:Completed 1200000 out of 5000000 steps (24%)
22:21:36:WU00:FS01:0xa8:Completed 1250000 out of 5000000 steps (25%)
22:31:57:WU00:FS01:0xa8:Completed 1300000 out of 5000000 steps (26%)
22:42:19:WU00:FS01:0xa8:Completed 1350000 out of 5000000 steps (27%)
22:52:34:WU00:FS01:0xa8:Completed 1400000 out of 5000000 steps (28%)
23:02:47:WU00:FS01:0xa8:Completed 1450000 out of 5000000 steps (29%)
23:13:00:WU00:FS01:0xa8:Completed 1500000 out of 5000000 steps (30%)
23:23:16:WU00:FS01:0xa8:Completed 1550000 out of 5000000 steps (31%)
23:33:34:WU00:FS01:0xa8:Completed 1600000 out of 5000000 steps (32%)
23:43:51:WU00:FS01:0xa8:Completed 1650000 out of 5000000 steps (33%)
23:54:12:WU00:FS01:0xa8:Completed 1700000 out of 5000000 steps (34%)
00:04:34:WU00:FS01:0xa8:Completed 1750000 out of 5000000 steps (35%)
00:14:55:WU00:FS01:0xa8:Completed 1800000 out of 5000000 steps (36%)
00:25:17:WU00:FS01:0xa8:Completed 1850000 out of 5000000 steps (37%)
00:35:38:WU00:FS01:0xa8:Completed 1900000 out of 5000000 steps (38%)
00:46:02:WU00:FS01:0xa8:Completed 1950000 out of 5000000 steps (39%)
00:56:29:WU00:FS01:0xa8:Completed 2000000 out of 5000000 steps (40%)
01:06:55:WU00:FS01:0xa8:Completed 2050000 out of 5000000 steps (41%)
01:17:20:WU00:FS01:0xa8:Completed 2100000 out of 5000000 steps (42%)
01:27:45:WU00:FS01:0xa8:Completed 2150000 out of 5000000 steps (43%)
01:38:03:WU00:FS01:0xa8:Completed 2200000 out of 5000000 steps (44%)
01:48:22:WU00:FS01:0xa8:Completed 2250000 out of 5000000 steps (45%)
01:58:43:WU00:FS01:0xa8:Completed 2300000 out of 5000000 steps (46%)
02:09:03:WU00:FS01:0xa8:Completed 2350000 out of 5000000 steps (47%)
02:19:24:WU00:FS01:0xa8:Completed 2400000 out of 5000000 steps (48%)
02:29:45:WU00:FS01:0xa8:Completed 2450000 out of 5000000 steps (49%)
02:40:07:WU00:FS01:0xa8:Completed 2500000 out of 5000000 steps (50%)
02:50:28:WU00:FS01:0xa8:Completed 2550000 out of 5000000 steps (51%)
03:00:49:WU00:FS01:0xa8:Completed 2600000 out of 5000000 steps (52%)
03:11:10:WU00:FS01:0xa8:Completed 2650000 out of 5000000 steps (53%)
03:21:32:WU00:FS01:0xa8:Completed 2700000 out of 5000000 steps (54%)
03:31:52:WU00:FS01:0xa8:Completed 2750000 out of 5000000 steps (55%)
03:42:13:WU00:FS01:0xa8:Completed 2800000 out of 5000000 steps (56%)
03:52:35:WU00:FS01:0xa8:Completed 2850000 out of 5000000 steps (57%)
04:02:56:WU00:FS01:0xa8:Completed 2900000 out of 5000000 steps (58%)
04:13:29:WU00:FS01:0xa8:Completed 2950000 out of 5000000 steps (59%)
04:23:50:WU00:FS01:0xa8:Completed 3000000 out of 5000000 steps (60%)
04:34:11:WU00:FS01:0xa8:Completed 3050000 out of 5000000 steps (61%)
04:44:31:WU00:FS01:0xa8:Completed 3100000 out of 5000000 steps (62%)
04:54:52:WU00:FS01:0xa8:Completed 3150000 out of 5000000 steps (63%)
05:05:14:WU00:FS01:0xa8:Completed 3200000 out of 5000000 steps (64%)
05:15:35:WU00:FS01:0xa8:Completed 3250000 out of 5000000 steps (65%)
05:25:56:WU00:FS01:0xa8:Completed 3300000 out of 5000000 steps (66%)
05:36:18:WU00:FS01:0xa8:Completed 3350000 out of 5000000 steps (67%)
05:46:37:WU00:FS01:0xa8:Completed 3400000 out of 5000000 steps (68%)
05:56:56:WU00:FS01:0xa8:Completed 3450000 out of 5000000 steps (69%)
06:07:17:WU00:FS01:0xa8:Completed 3500000 out of 5000000 steps (70%)
06:17:38:WU00:FS01:0xa8:Completed 3550000 out of 5000000 steps (71%)
06:28:00:WU00:FS01:0xa8:Completed 3600000 out of 5000000 steps (72%)
06:38:21:WU00:FS01:0xa8:Completed 3650000 out of 5000000 steps (73%)
06:48:43:WU00:FS01:0xa8:Completed 3700000 out of 5000000 steps (74%)
06:59:04:WU00:FS01:0xa8:Completed 3750000 out of 5000000 steps (75%)
07:09:25:WU00:FS01:0xa8:Completed 3800000 out of 5000000 steps (76%)
07:19:46:WU00:FS01:0xa8:Completed 3850000 out of 5000000 steps (77%)
07:30:08:WU00:FS01:0xa8:Completed 3900000 out of 5000000 steps (78%)
07:40:29:WU00:FS01:0xa8:Completed 3950000 out of 5000000 steps (79%)
07:50:50:WU00:FS01:0xa8:Completed 4000000 out of 5000000 steps (80%)
08:01:11:WU00:FS01:0xa8:Completed 4050000 out of 5000000 steps (81%)
08:11:34:WU00:FS01:0xa8:Completed 4100000 out of 5000000 steps (82%)
08:21:55:WU00:FS01:0xa8:Completed 4150000 out of 5000000 steps (83%)
08:32:16:WU00:FS01:0xa8:Completed 4200000 out of 5000000 steps (84%)
08:42:37:WU00:FS01:0xa8:Completed 4250000 out of 5000000 steps (85%)
08:52:58:WU00:FS01:0xa8:Completed 4300000 out of 5000000 steps (86%)
09:03:19:WU00:FS01:0xa8:Completed 4350000 out of 5000000 steps (87%)
09:13:35:WU00:FS01:0xa8:Completed 4400000 out of 5000000 steps (88%)
09:23:54:WU00:FS01:0xa8:Completed 4450000 out of 5000000 steps (89%)
09:34:15:WU00:FS01:0xa8:Completed 4500000 out of 5000000 steps (90%)
09:44:36:WU00:FS01:0xa8:Completed 4550000 out of 5000000 steps (91%)
09:55:00:WU00:FS01:0xa8:Completed 4600000 out of 5000000 steps (92%)
11:59:54:WU00:FS01:0xa8:Completed 4650000 out of 5000000 steps (93%)
14:04:52:WU00:FS01:0xa8:Completed 4700000 out of 5000000 steps (94%)
16:09:46:WU00:FS01:0xa8:Completed 4750000 out of 5000000 steps (95%)
About 10 minutes per percent then jumps to a couple hours after 92% and shows completion estimates of 8+ hours. This only happens on my old 14-core or 16-core Xeons but not on a modern 16-core Ryzen.

Just wanted to check if this is intended behavior or if there is a problem with these WUs.
Joe_H
Site Admin
Posts: 7922
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: WUs that slow way down at the end

Post by Joe_H »

I have seen slowdowns in the beginning, middle and end of WUs. Barring a slowdown from the CPU being throttled for thermal or other reasons, usually this indicates a shift in conformation of the protein being modeled that takes longer to calculate interactions.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Post Reply