I have been seeing a deterioration of performance with download and upload of SMP projects over time. I test my speed at over 50Mbit/s, and I don't see this as much with GPU projects. See example below, where each packet is about 4.26% of the upload:
I looked into this and was wondering if you're still seeing this issue? The server status page still says there is a high net load, but things appear to be working now on my side. I tried to alleviate the pressure on my work server and that seems to be working, though the server status page has not yet adjusted to this.
The server status page was last updated at 14:00. (It's now 14:45 so something is wrong, which I'll report.)
The updated information captured at 14:20 (before you fixed it) showed it was still bad. I can't see any updates captured since then, such as at 14:40.
It seems to be updated to 15:00:00 now, so at least it is updating in some capacity even if it's not as frequently as it should.
I guess I'm just having trouble seeing the connection between the high net load (which it seems to be doing, but it might just be slowing things down because there are too many jobs out right now) and the dumping jobs, which I'm not seeing on my side today. If that's still happening though, please let me know and I'll see if I can figure out the continued issue.
Is the system clock on that (virtual) machine correct? I noticed on Server Stat that vspg14b, vspg14c, vspg14d, and vspg14e all show worsening Diff Time values from -460 on Dec 22 to currently -535. If that is seconds, then the clock could be off by nearly 9 minutes.
It is doing much better, but last summer I remember uploads would usually take less than a few seconds for A4 projects. It is my impression that the download and uploads have been getting progressively worse since sometime in October. This is my last A4 upload:
20:45:46:WU01:FS00:0xa4:Completed 250000 out of 250000 steps (100%)
20:45:47:WU00:FS00:Connecting to 171.67.108.45:8080
20:45:47:WU01:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
20:45:47:WU00:FS00:Assigned to work server 171.64.65.124
20:45:47:WU00:FS00:Requesting new work unit for slot 00: RUNNING cpu:10 from 171.64.65.124
20:45:47:WU00:FS00:Connecting to 171.64.65.124:8080
20:45:49:ERROR:WU00:FS00:Exception: Server did not assign work unit
20:45:49:WU00:FS00:Connecting to 171.67.108.45:8080
20:45:50:WU00:FS00:Assigned to work server 171.64.65.124
20:45:50:WU00:FS00:Requesting new work unit for slot 00: RUNNING cpu:10 from 171.64.65.124
20:45:50:WU00:FS00:Connecting to 171.64.65.124:8080
20:45:56:WU00:FS00:Downloading 806.96KiB
20:45:57:WU01:FS00:0xa4:
20:45:57:WU01:FS00:0xa4:Finished Work Unit:
20:45:57:WU01:FS00:0xa4:- Reading up to 811536 from "01/wudata_01.trr": Read 811536
20:45:57:WU01:FS00:0xa4:trr file hash check passed.
20:45:57:WU01:FS00:0xa4:- Reading up to 745956 from "01/wudata_01.xtc": Read 745956
20:45:57:WU01:FS00:0xa4:xtc file hash check passed.
20:45:57:WU01:FS00:0xa4:edr file hash check passed.
20:45:57:WU01:FS00:0xa4:logfile size: 22828
20:45:57:WU01:FS00:0xa4:Leaving Run
20:46:00:WU01:FS00:0xa4:- Writing 1582808 bytes of core data to disk...
20:46:00:WU01:FS00:0xa4:Done: 1582296 -> 1537441 (compressed to 97.1 percent)
20:46:00:WU01:FS00:0xa4: ... Done.
20:46:00:WU01:FS00:0xa4:- Shutting down core
20:46:00:WU01:FS00:0xa4:
20:46:00:WU01:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
20:46:01:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
20:46:01:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:9022 run:348 clone:11 gen:27 core:0xa4 unit:0x00000020ab40417c55e8a25a2845533d
20:46:01:WU01:FS00:Uploading 1.47MiB to 171.64.65.124
20:46:01:WU01:FS00:Connecting to 171.64.65.124:8080
20:46:02:WU00:FS00:Download 7.93%
20:46:12:WU00:FS00:Download 23.79%
20:46:25:WU00:FS00:Download 31.72%
20:46:30:WU01:FS00:Upload 12.78%
20:46:42:WU00:FS00:Download 39.66%
20:47:10:WU00:FS00:Download 47.59%
20:47:19:WU00:FS00:Download 63.45%
20:47:27:WU00:FS00:Download 71.38%
20:47:27:WU01:FS00:Upload 17.04%
20:47:33:WU00:FS00:Download 79.31%
20:47:44:WU00:FS00:Download 95.17%
20:47:46:WU00:FS00:Download complete
20:47:46:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9028 run:527 clone:10 gen:10 core:0xa4 unit:0x0000000bab40417c55f393a96d7ba9c6
20:47:46:WU00:FS00:Starting
20:47:46:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win2/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/beta/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 704 -lifeline 1172 -checkpoint 15 -np 10
20:47:46:WU00:FS00:Started FahCore on PID 1716
20:47:46:WU00:FS00:Core PID:3892
20:47:46:WU00:FS00:FahCore 0xa4 started
20:47:47:WU00:FS00:0xa4:
20:47:47:WU00:FS00:0xa4:*------------------------------*
20:47:47:WU00:FS00:0xa4:Folding@Home Gromacs GB Core
20:47:47:WU00:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
20:47:47:WU00:FS00:0xa4:
20:47:47:WU00:FS00:0xa4:Preparing to commence simulation
20:47:47:WU00:FS00:0xa4:- Looking at optimizations...
20:47:47:WU00:FS00:0xa4:- Created dyn
20:47:47:WU00:FS00:0xa4:- Files status OK
20:47:47:WU00:FS00:0xa4:- Expanded 825811 -> 1402860 (decompressed 169.8 percent)
20:47:47:WU00:FS00:0xa4:Called DecompressByteArray: compressed_data_size=825811 data_size=1402860, decompressed_data_size=1402860 diff=0
20:47:47:WU00:FS00:0xa4:- Digital signature verified
20:47:47:WU00:FS00:0xa4:
20:47:47:WU00:FS00:0xa4:Project: 9028 (Run 527, Clone 10, Gen 10)
20:47:47:WU00:FS00:0xa4:
20:47:47:WU00:FS00:0xa4:Assembly optimizations on if available.
20:47:47:WU00:FS00:0xa4:Entering M.D.
20:47:52:WU01:FS00:Upload 21.31%
20:47:53:WU00:FS00:0xa4:Mapping NT from 10 to 10
20:47:53:WU00:FS00:0xa4:Completed 0 out of 250000 steps (0%)
20:48:12:WU01:FS00:Upload 25.57%
20:48:47:WU00:FS00:0xa4:Completed 2500 out of 250000 steps (1%)
20:49:19:WU01:FS00:Upload 29.83%
20:49:38:WU01:FS00:Upload 34.09%
20:49:42:WU00:FS00:0xa4:Completed 5000 out of 250000 steps (2%)
20:50:23:WU01:FS00:Upload 38.35%
20:50:36:WU00:FS00:0xa4:Completed 7500 out of 250000 steps (3%)
20:50:54:WU01:FS00:Upload 42.61%
20:51:31:WU00:FS00:0xa4:Completed 10000 out of 250000 steps (4%)
20:52:25:WU00:FS00:0xa4:Completed 12500 out of 250000 steps (5%)
20:53:20:WU00:FS00:0xa4:Completed 15000 out of 250000 steps (6%)
20:54:06:WU01:FS00:Upload 46.87%
20:54:14:WU00:FS00:0xa4:Completed 17500 out of 250000 steps (7%)
20:54:49:WU01:FS00:Upload 51.13%
20:55:05:WU01:FS00:Upload 55.40%
20:55:09:WU00:FS00:0xa4:Completed 20000 out of 250000 steps (8%)
20:55:46:WU01:FS00:Upload 59.66%
20:55:58:WU01:FS00:Upload 63.92%
20:56:03:WU00:FS00:0xa4:Completed 22500 out of 250000 steps (9%)
20:56:36:WU01:FS00:Upload 68.18%
20:56:58:WU00:FS00:0xa4:Completed 25000 out of 250000 steps (10%)
20:57:08:WU01:FS00:Upload 72.44%
20:57:46:WU01:FS00:Upload 80.96%
20:57:52:WU00:FS00:0xa4:Completed 27500 out of 250000 steps (11%)
20:58:04:WU01:FS00:Upload 85.22%
20:58:18:WU01:FS00:Upload 89.49%
20:58:47:WU00:FS00:0xa4:Completed 30000 out of 250000 steps (12%)
20:59:14:WU01:FS00:Upload 93.75%
20:59:33:WU01:FS00:Upload 100.00%
20:59:41:WU00:FS00:0xa4:Completed 32500 out of 250000 steps (13%)
21:00:36:WU00:FS00:0xa4:Completed 35000 out of 250000 steps (14%)
21:00:56:WU01:FS00:Upload complete
21:00:56:WU01:FS00:Server responded WORK_ACK (400)
21:00:56:WU01:FS00:Final credit estimate, 1892.00 points
21:00:56:WU01:FS00:Cleaning up
21:01:31:WU00:FS00:0xa4:Completed 37500 out of 250000 steps (15%)
21:02:26:WU00:FS00:0xa4:Completed 40000 out of 250000 steps (16%)
21:03:20:WU00:FS00:0xa4:Completed 42500 out of 250000 steps (17%)
21:04:15:WU00:FS00:0xa4:Completed 45000 out of 250000 steps (18%)
21:05:09:WU00:FS00:0xa4:Completed 47500 out of 250000 steps (19%)
21:06:05:WU00:FS00:0xa4:Completed 50000 out of 250000 steps (20%)
21:06:59:WU00:FS00:0xa4:Completed 52500 out of 250000 steps (21%)
21:07:53:WU00:FS00:0xa4:Completed 55000 out of 250000 steps (22%)
21:08:48:WU00:FS00:0xa4:Completed 57500 out of 250000 steps (23%)
21:09:42:WU00:FS00:0xa4:Completed 60000 out of 250000 steps (24%)
21:10:36:WU00:FS00:0xa4:Completed 62500 out of 250000 steps (25%)
21:11:30:WU00:FS00:0xa4:Completed 65000 out of 250000 steps (26%)
21:12:25:WU00:FS00:0xa4:Completed 67500 out of 250000 steps (27%)
21:13:19:WU00:FS00:0xa4:Completed 70000 out of 250000 steps (28%)
21:14:14:WU00:FS00:0xa4:Completed 72500 out of 250000 steps (29%)
21:15:08:WU00:FS00:0xa4:Completed 75000 out of 250000 steps (30%)
21:16:02:WU00:FS00:0xa4:Completed 77500 out of 250000 steps (31%)
21:16:56:WU00:FS00:0xa4:Completed 80000 out of 250000 steps (32%)
21:17:51:WU00:FS00:0xa4:Completed 82500 out of 250000 steps (33%)
21:18:46:WU00:FS00:0xa4:Completed 85000 out of 250000 steps (34%)
21:19:40:WU00:FS00:0xa4:Completed 87500 out of 250000 steps (35%)
21:20:35:WU00:FS00:0xa4:Completed 90000 out of 250000 steps (36%)
21:21:29:WU00:FS00:0xa4:Completed 92500 out of 250000 steps (37%)
21:22:23:WU00:FS00:0xa4:Completed 95000 out of 250000 steps (38%)
21:23:17:WU00:FS00:0xa4:Completed 97500 out of 250000 steps (39%)
21:24:12:WU00:FS00:0xa4:Completed 100000 out of 250000 steps (40%)
21:25:06:WU00:FS00:0xa4:Completed 102500 out of 250000 steps (41%)
21:26:01:WU00:FS00:0xa4:Completed 105000 out of 250000 steps (42%)
21:26:55:WU00:FS00:0xa4:Completed 107500 out of 250000 steps (43%)
21:27:49:WU00:FS00:0xa4:Completed 110000 out of 250000 steps (44%)
21:28:43:WU00:FS00:0xa4:Completed 112500 out of 250000 steps (45%)
21:29:38:WU00:FS00:0xa4:Completed 115000 out of 250000 steps (46%)
21:30:32:WU00:FS00:0xa4:Completed 117500 out of 250000 steps (47%)
21:31:26:WU00:FS00:0xa4:Completed 120000 out of 250000 steps (48%)
21:32:20:WU00:FS00:0xa4:Completed 122500 out of 250000 steps (49%)
21:33:14:WU00:FS00:0xa4:Completed 125000 out of 250000 steps (50%)
21:34:08:WU00:FS00:0xa4:Completed 127500 out of 250000 steps (51%)
21:35:02:WU00:FS00:0xa4:Completed 130000 out of 250000 steps (52%)
21:35:56:WU00:FS00:0xa4:Completed 132500 out of 250000 steps (53%)
21:36:50:WU00:FS00:0xa4:Completed 135000 out of 250000 steps (54%)
21:37:44:WU00:FS00:0xa4:Completed 137500 out of 250000 steps (55%)
21:38:38:WU00:FS00:0xa4:Completed 140000 out of 250000 steps (56%)
21:39:32:WU00:FS00:0xa4:Completed 142500 out of 250000 steps (57%)
21:40:26:WU00:FS00:0xa4:Completed 145000 out of 250000 steps (58%)
21:41:21:WU00:FS00:0xa4:Completed 147500 out of 250000 steps (59%)
21:42:15:WU00:FS00:0xa4:Completed 150000 out of 250000 steps (60%)
21:43:09:WU00:FS00:0xa4:Completed 152500 out of 250000 steps (61%)
21:44:03:WU00:FS00:0xa4:Completed 155000 out of 250000 steps (62%)
21:44:57:WU00:FS00:0xa4:Completed 157500 out of 250000 steps (63%)
21:45:52:WU00:FS00:0xa4:Completed 160000 out of 250000 steps (64%)
21:46:46:WU00:FS00:0xa4:Completed 162500 out of 250000 steps (65%)
21:47:40:WU00:FS00:0xa4:Completed 165000 out of 250000 steps (66%)
21:48:35:WU00:FS00:0xa4:Completed 167500 out of 250000 steps (67%)
21:49:29:WU00:FS00:0xa4:Completed 170000 out of 250000 steps (68%)
21:50:24:WU00:FS00:0xa4:Completed 172500 out of 250000 steps (69%)
21:51:18:WU00:FS00:0xa4:Completed 175000 out of 250000 steps (70%)
21:52:13:WU00:FS00:0xa4:Completed 177500 out of 250000 steps (71%)
21:53:08:WU00:FS00:0xa4:Completed 180000 out of 250000 steps (72%)
21:54:02:WU00:FS00:0xa4:Completed 182500 out of 250000 steps (73%)
21:54:57:WU00:FS00:0xa4:Completed 185000 out of 250000 steps (74%)
21:55:52:WU00:FS00:0xa4:Completed 187500 out of 250000 steps (75%)
21:56:46:WU00:FS00:0xa4:Completed 190000 out of 250000 steps (76%)
21:57:40:WU00:FS00:0xa4:Completed 192500 out of 250000 steps (77%)
21:58:34:WU00:FS00:0xa4:Completed 195000 out of 250000 steps (78%)
21:59:29:WU00:FS00:0xa4:Completed 197500 out of 250000 steps (79%)
22:00:23:WU00:FS00:0xa4:Completed 200000 out of 250000 steps (80%)
22:01:18:WU00:FS00:0xa4:Completed 202500 out of 250000 steps (81%)
22:02:12:WU00:FS00:0xa4:Completed 205000 out of 250000 steps (82%)
22:03:07:WU00:FS00:0xa4:Completed 207500 out of 250000 steps (83%)
22:04:01:WU00:FS00:0xa4:Completed 210000 out of 250000 steps (84%)
22:04:56:WU00:FS00:0xa4:Completed 212500 out of 250000 steps (85%)
22:05:50:WU00:FS00:0xa4:Completed 215000 out of 250000 steps (86%)
22:06:44:WU00:FS00:0xa4:Completed 217500 out of 250000 steps (87%)
22:07:39:WU00:FS00:0xa4:Completed 220000 out of 250000 steps (88%)
22:08:34:WU00:FS00:0xa4:Completed 222500 out of 250000 steps (89%)
22:09:29:WU00:FS00:0xa4:Completed 225000 out of 250000 steps (90%)
22:10:24:WU00:FS00:0xa4:Completed 227500 out of 250000 steps (91%)
22:11:18:WU00:FS00:0xa4:Completed 230000 out of 250000 steps (92%)
22:12:13:WU00:FS00:0xa4:Completed 232500 out of 250000 steps (93%)
22:13:07:WU00:FS00:0xa4:Completed 235000 out of 250000 steps (94%)
22:14:02:WU00:FS00:0xa4:Completed 237500 out of 250000 steps (95%)
22:14:56:WU00:FS00:0xa4:Completed 240000 out of 250000 steps (96%)
22:15:50:WU00:FS00:0xa4:Completed 242500 out of 250000 steps (97%)
22:16:45:WU00:FS00:0xa4:Completed 245000 out of 250000 steps (98%)
22:17:39:WU00:FS00:0xa4:Completed 247500 out of 250000 steps (99%)
22:18:34:WU00:FS00:0xa4:Completed 250000 out of 250000 steps (100%)
22:18:34:WU00:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
22:18:34:WU02:FS00:Connecting to 171.67.108.45:8080
22:18:34:WU02:FS00:Assigned to work server 171.64.65.124
22:18:34:WU02:FS00:Requesting new work unit for slot 00: RUNNING cpu:10 from 171.64.65.124
22:18:34:WU02:FS00:Connecting to 171.64.65.124:8080
22:18:44:WU00:FS00:0xa4:
22:18:44:WU00:FS00:0xa4:Finished Work Unit:
22:18:44:WU00:FS00:0xa4:- Reading up to 811536 from "00/wudata_01.trr": Read 811536
22:18:44:WU00:FS00:0xa4:trr file hash check passed.
22:18:44:WU00:FS00:0xa4:- Reading up to 746216 from "00/wudata_01.xtc": Read 746216
22:18:44:WU00:FS00:0xa4:xtc file hash check passed.
22:18:44:WU00:FS00:0xa4:edr file hash check passed.
22:18:44:WU00:FS00:0xa4:logfile size: 22831
22:18:44:WU00:FS00:0xa4:Leaving Run
22:18:48:WU00:FS00:0xa4:- Writing 1583071 bytes of core data to disk...
22:18:48:WU00:FS00:0xa4:Done: 1582559 -> 1538111 (compressed to 97.1 percent)
22:18:48:WU00:FS00:0xa4: ... Done.
22:18:48:WU00:FS00:0xa4:- Shutting down core
22:18:48:WU00:FS00:0xa4:
22:18:48:WU00:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
22:18:49:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
22:18:49:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:9028 run:527 clone:10 gen:10 core:0xa4 unit:0x0000000bab40417c55f393a96d7ba9c6
22:18:49:WU00:FS00:Uploading 1.47MiB to 171.64.65.124
22:18:49:WU00:FS00:Connecting to 171.64.65.124:8080
22:19:10:WU02:FS00:Downloading 806.70KiB
22:19:21:WU02:FS00:Download 7.93%
22:19:35:WU02:FS00:Download 23.80%
22:19:40:WU00:FS00:Upload 12.78%
22:19:49:WU02:FS00:Download 39.67%
22:19:57:WU02:FS00:Download 47.60%
22:20:05:WU00:FS00:Upload 17.04%
22:20:09:WU02:FS00:Download 55.53%
22:20:26:WU02:FS00:Download 63.47%
22:20:33:WU02:FS00:Download 71.40%
22:20:42:WU02:FS00:Download 79.34%
22:20:53:WU02:FS00:Download 95.20%
22:20:56:WU00:FS00:Upload 21.30%
22:21:02:WU02:FS00:Download 100.00%
22:21:02:WU02:FS00:Download complete
22:21:02:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:9026 run:789 clone:12 gen:7 core:0xa4 unit:0x00000009ab40417c55f39c36e0aa4646
22:21:02:WU02:FS00:Starting
22:21:02:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win2/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/beta/Core_a4.fah/FahCore_a4.exe -dir 02 -suffix 01 -version 704 -lifeline 1172 -checkpoint 15 -np 10
22:21:02:WU02:FS00:Started FahCore on PID 5028
22:21:02:WU02:FS00:Core PID:2168
22:21:02:WU02:FS00:FahCore 0xa4 started
22:21:03:WU02:FS00:0xa4:
22:21:03:WU02:FS00:0xa4:*------------------------------*
22:21:03:WU02:FS00:0xa4:Folding@Home Gromacs GB Core
22:21:03:WU02:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
22:21:03:WU02:FS00:0xa4:
22:21:03:WU02:FS00:0xa4:Preparing to commence simulation
22:21:03:WU02:FS00:0xa4:- Looking at optimizations...
22:21:03:WU02:FS00:0xa4:- Created dyn
22:21:03:WU02:FS00:0xa4:- Files status OK
22:21:03:WU02:FS00:0xa4:- Expanded 825553 -> 1403132 (decompressed 169.9 percent)
22:21:03:WU02:FS00:0xa4:Called DecompressByteArray: compressed_data_size=825553 data_size=1403132, decompressed_data_size=1403132 diff=0
22:21:03:WU02:FS00:0xa4:- Digital signature verified
22:21:03:WU02:FS00:0xa4:
22:21:03:WU02:FS00:0xa4:Project: 9026 (Run 789, Clone 12, Gen 7)
22:21:03:WU02:FS00:0xa4:
22:21:03:WU02:FS00:0xa4:Assembly optimizations on if available.
22:21:03:WU02:FS00:0xa4:Entering M.D.
22:21:09:WU02:FS00:0xa4:Mapping NT from 10 to 10
22:21:09:WU02:FS00:0xa4:Completed 0 out of 250000 steps (0%)
22:21:18:WU00:FS00:Upload 25.56%
22:21:48:WU00:FS00:Upload 29.82%
22:22:04:WU02:FS00:0xa4:Completed 2500 out of 250000 steps (1%)
22:22:47:WU00:FS00:Upload 34.08%
22:22:59:WU02:FS00:0xa4:Completed 5000 out of 250000 steps (2%)
22:23:05:WU00:FS00:Upload 38.33%
22:23:35:WU00:FS00:Upload 42.59%
22:23:54:WU02:FS00:0xa4:Completed 7500 out of 250000 steps (3%)
22:24:08:WU00:FS00:Upload 46.85%
22:24:35:WU00:FS00:Upload 51.11%
22:24:48:WU02:FS00:0xa4:Completed 10000 out of 250000 steps (4%)
22:25:09:WU00:FS00:Upload 55.37%
22:25:43:WU02:FS00:0xa4:Completed 12500 out of 250000 steps (5%)
22:25:44:WU00:FS00:Upload 59.63%
22:26:24:WU00:FS00:Upload 63.89%
22:26:38:WU02:FS00:0xa4:Completed 15000 out of 250000 steps (6%)
22:27:04:WU00:FS00:Upload 68.15%
22:27:33:WU02:FS00:0xa4:Completed 17500 out of 250000 steps (7%)
22:27:47:WU00:FS00:Upload 72.41%
22:28:14:WU00:FS00:Upload 76.67%
22:28:28:WU02:FS00:0xa4:Completed 20000 out of 250000 steps (8%)
22:28:47:WU00:FS00:Upload 85.19%
22:29:18:WU00:FS00:Upload 89.45%
22:29:23:WU02:FS00:0xa4:Completed 22500 out of 250000 steps (9%)
22:29:39:WU00:FS00:Upload 93.71%
22:30:08:WU00:FS00:Upload 97.97%
22:30:17:WU02:FS00:0xa4:Completed 25000 out of 250000 steps (10%)
22:30:19:WU00:FS00:Upload 100.00%
22:31:12:WU02:FS00:0xa4:Completed 27500 out of 250000 steps (11%)
22:32:07:WU02:FS00:0xa4:Completed 30000 out of 250000 steps (12%)
22:32:17:WU00:FS00:Upload complete
22:32:17:WU00:FS00:Server responded WORK_QUIT (404)
22:32:17:WARNING:WU00:FS00:Server did not like results, dumping
22:32:17:WU00:FS00:Cleaning up
22:33:02:WU02:FS00:0xa4:Completed 32500 out of 250000 steps (13%)
I am still seeing very slow uploads and downloads to this WS more than half the time. Just before posting, one of my systems just finished getting a new assignment and uploading a completed WU to 171.64.65.124. The download took about 4 minutes, normal is less than 30 seconds. The upload was 10 minutes, and the acknowledgement for the returned WU took nearly 5 minutes to show up in the client log. Normal time to upload and get an ACK is a couple minutes.
iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
16:25:12:WU01:FS00:0xa4:Completed 250000 out of 250000 steps (100%)
16:25:12:WU01:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
16:25:13:WU02:FS00:Connecting to 171.67.108.45:8080
16:25:13:WU02:FS00:Assigned to work server 171.64.65.124
16:25:13:WU02:FS00:Requesting new work unit for slot 00: RUNNING cpu:24 from 171.64.65.124
16:25:13:WU02:FS00:Connecting to 171.64.65.124:8080
16:25:22:WU01:FS00:0xa4:
16:25:22:WU01:FS00:0xa4:Finished Work Unit:
16:25:22:WU01:FS00:0xa4:- Reading up to 811608 from "01/wudata_01.trr": Read 811608
16:25:22:WU01:FS00:0xa4:trr file hash check passed.
16:25:22:WU01:FS00:0xa4:- Reading up to 746708 from "01/wudata_01.xtc": Read 746708
16:25:22:WU01:FS00:0xa4:xtc file hash check passed.
16:25:22:WU01:FS00:0xa4:edr file hash check passed.
16:25:22:WU01:FS00:0xa4:logfile size: 23087
16:25:22:WU01:FS00:0xa4:Leaving Run
16:25:26:WU01:FS00:0xa4:- Writing 1583891 bytes of core data to disk...
16:25:27:WU01:FS00:0xa4:Done: 1583379 -> 1538509 (compressed to 97.1 percent)
16:25:27:WU01:FS00:0xa4: ... Done.
16:25:27:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
16:25:27:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:9025 run:271 clone:13 gen:2 core:0xa4 unit:0x00000002ab40417c55f3a3e97f8c3f67
16:25:27:WU01:FS00:Uploading 1.47MiB to 171.64.65.124
16:25:27:WU01:FS00:Connecting to 171.64.65.124:8080
16:26:04:WU02:FS00:Downloading 807.02KiB
16:26:15:WU02:FS00:Download 7.93%
16:26:26:WU02:FS00:Download 15.86%
16:26:41:WU02:FS00:Download 23.79%
16:27:02:WU01:FS00:Upload 12.77%
16:27:06:WU02:FS00:Download 31.72%
16:27:19:WU02:FS00:Download 39.65%
16:27:26:WU02:FS00:Download 47.58%
16:27:32:WU02:FS00:Download 55.51%
16:27:37:WU01:FS00:Upload 17.03%
16:27:43:WU02:FS00:Download 63.44%
16:27:50:WU02:FS00:Download 71.37%
16:27:57:WU02:FS00:Download 79.30%
16:28:12:WU02:FS00:Download 95.16%
16:28:16:WU01:FS00:Upload 21.29%
16:28:24:WU02:FS00:Download 100.00%
16:28:24:WU02:FS00:Download complete
16:28:24:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:9026 run:917 clone:9 gen:1 core:0xa4 unit:0x00000003ab40417c55f39b9470d3caee
16:28:24:WU02:FS00:Starting
16:28:24:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/beta/Core_a4.fah/FahCore_a4.exe -dir 02 -suffix 01 -version 704 -lifeline 5260 -checkpoint 15 -np 24
16:28:24:WU02:FS00:Started FahCore on PID 708
16:28:24:WU02:FS00:Core PID:7048
16:28:24:WU02:FS00:FahCore 0xa4 started
16:28:25:WU02:FS00:0xa4:
16:28:25:WU02:FS00:0xa4:*------------------------------*
16:28:25:WU02:FS00:0xa4:Folding@Home Gromacs GB Core
16:28:25:WU02:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
16:28:25:WU02:FS00:0xa4:
16:28:25:WU02:FS00:0xa4:Preparing to commence simulation
16:28:25:WU02:FS00:0xa4:- Looking at optimizations...
16:28:25:WU02:FS00:0xa4:- Created dyn
16:28:25:WU02:FS00:0xa4:- Files status OK
16:28:25:WU02:FS00:0xa4:- Expanded 825881 -> 1403132 (decompressed 169.8 percent)
16:28:25:WU02:FS00:0xa4:Called DecompressByteArray: compressed_data_size=825881 data_size=1403132, decompressed_data_size=1403132 diff=0
16:28:25:WU02:FS00:0xa4:- Digital signature verified
16:28:25:WU02:FS00:0xa4:
16:28:25:WU02:FS00:0xa4:Project: 9026 (Run 917, Clone 9, Gen 1)
16:28:25:WU02:FS00:0xa4:
16:28:25:WU02:FS00:0xa4:Assembly optimizations on if available.
16:28:25:WU02:FS00:0xa4:Entering M.D.
16:28:31:WU02:FS00:0xa4:Mapping NT from 24 to 24
16:28:31:WU02:FS00:0xa4:Completed 0 out of 250000 steps (0%)
16:29:05:WU02:FS00:0xa4:Completed 2500 out of 250000 steps (1%)
16:29:12:WU01:FS00:Upload 25.55%
16:29:29:WU01:FS00:Upload 29.81%
16:29:39:WU02:FS00:0xa4:Completed 5000 out of 250000 steps (2%)
16:29:46:WU01:FS00:Upload 34.07%
16:30:13:WU02:FS00:0xa4:Completed 7500 out of 250000 steps (3%)
16:30:29:WU01:FS00:Upload 38.32%
16:30:48:WU02:FS00:0xa4:Completed 10000 out of 250000 steps (4%)
16:31:22:WU02:FS00:0xa4:Completed 12500 out of 250000 steps (5%)
16:31:37:WU01:FS00:Upload 42.58%
16:31:48:WU01:FS00:Upload 46.84%
16:31:56:WU02:FS00:0xa4:Completed 15000 out of 250000 steps (6%)
16:32:20:WU01:FS00:Upload 51.10%
16:32:30:WU02:FS00:0xa4:Completed 17500 out of 250000 steps (7%)
16:33:00:WU01:FS00:Upload 55.36%
16:33:05:WU02:FS00:0xa4:Completed 20000 out of 250000 steps (8%)
16:33:18:WU01:FS00:Upload 59.62%
16:33:39:WU02:FS00:0xa4:Completed 22500 out of 250000 steps (9%)
16:33:45:WU01:FS00:Upload 63.87%
16:34:13:WU02:FS00:0xa4:Completed 25000 out of 250000 steps (10%)
16:34:18:WU01:FS00:Upload 68.13%
16:34:47:WU02:FS00:0xa4:Completed 27500 out of 250000 steps (11%)
16:35:21:WU02:FS00:0xa4:Completed 30000 out of 250000 steps (12%)
16:35:38:WU01:FS00:Upload 72.39%
16:35:54:WU01:FS00:Upload 76.65%
16:35:56:WU02:FS00:0xa4:Completed 32500 out of 250000 steps (13%)
16:36:26:WU01:FS00:Upload 80.91%
16:36:30:WU02:FS00:0xa4:Completed 35000 out of 250000 steps (14%)
16:37:04:WU02:FS00:0xa4:Completed 37500 out of 250000 steps (15%)
16:37:31:WU01:FS00:Upload 85.17%
16:37:38:WU02:FS00:0xa4:Completed 40000 out of 250000 steps (16%)
16:37:48:WU01:FS00:Upload 89.42%
16:38:12:WU02:FS00:0xa4:Completed 42500 out of 250000 steps (17%)
16:38:17:WU01:FS00:Upload 93.68%
16:38:37:WU01:FS00:Upload 97.94%
16:38:46:WU02:FS00:0xa4:Completed 45000 out of 250000 steps (18%)
16:39:15:WU01:FS00:Upload 100.00%
16:39:21:WU02:FS00:0xa4:Completed 47500 out of 250000 steps (19%)
16:39:55:WU02:FS00:0xa4:Completed 50000 out of 250000 steps (20%)
16:40:29:WU02:FS00:0xa4:Completed 52500 out of 250000 steps (21%)
16:41:04:WU02:FS00:0xa4:Completed 55000 out of 250000 steps (22%)
16:41:24:WU01:FS00:Upload complete
16:41:24:WU01:FS00:Server responded WORK_ACK (400)
16:41:24:WU01:FS00:Final credit estimate, 2258.00 points
16:41:24:WU01:FS00:Cleaning up
16:41:38:WU02:FS00:0xa4:Completed 57500 out of 250000 steps (23%)
16:42:12:WU02:FS00:0xa4:Completed 60000 out of 250000 steps (24%)
**********Truncated**********
17:25:37:WU02:FS00:0xa4:Completed 250000 out of 250000 steps (100%)
17:25:37:WU01:FS00:Connecting to 171.67.108.45:8080
17:25:37:WU01:FS00:Assigned to work server 171.64.65.124
17:25:37:WU02:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
17:25:37:WU01:FS00:Requesting new work unit for slot 00: RUNNING cpu:24 from 171.64.65.124
17:25:37:WU01:FS00:Connecting to 171.64.65.124:8080
17:25:47:WU02:FS00:0xa4:
17:25:47:WU02:FS00:0xa4:Finished Work Unit:
17:25:47:WU02:FS00:0xa4:- Reading up to 811560 from "02/wudata_01.trr": Read 811560
17:25:47:WU02:FS00:0xa4:trr file hash check passed.
17:25:47:WU02:FS00:0xa4:- Reading up to 746072 from "02/wudata_01.xtc": Read 746072
17:25:47:WU02:FS00:0xa4:xtc file hash check passed.
17:25:47:WU02:FS00:0xa4:edr file hash check passed.
17:25:47:WU02:FS00:0xa4:logfile size: 23085
17:25:47:WU02:FS00:0xa4:Leaving Run
17:25:50:WU02:FS00:0xa4:- Writing 1583205 bytes of core data to disk...
17:25:50:WU02:FS00:0xa4:Done: 1582693 -> 1538138 (compressed to 97.1 percent)
17:25:50:WU02:FS00:0xa4: ... Done.
17:25:50:WU02:FS00:0xa4:- Shutting down core
17:25:50:WU02:FS00:0xa4:
17:25:50:WU02:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
17:25:51:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
17:25:51:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:9026 run:917 clone:9 gen:1 core:0xa4 unit:0x00000003ab40417c55f39b9470d3caee
17:25:51:WU02:FS00:Uploading 1.47MiB to 171.64.65.124
17:25:51:WU02:FS00:Connecting to 171.64.65.124:8080
17:25:58:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
17:25:58:WU01:FS00:Connecting to 171.64.65.124:80
17:26:10:WU02:FS00:Upload 8.52%
17:26:10:WARNING:WU02:FS00:Exception: Failed to send results to work server: Transfer failed
17:26:10:WU02:FS00:Trying to send results to collection server
17:26:10:WU02:FS00:Uploading 1.47MiB to 171.67.108.46
17:26:10:WU02:FS00:Connecting to 171.67.108.46:8080
17:26:11:WU02:FS00:Upload complete
17:26:11:WU02:FS00:Server responded WORK_ACK (400)
17:26:11:WU02:FS00:Final credit estimate, 2742.00 points
17:26:11:WU02:FS00:Cleaning up
17:27:04:ERROR:WU01:FS00:Exception: Server did not assign work unit
17:27:04:WU01:FS00:Connecting to 171.67.108.45:8080
17:27:04:WU01:FS00:Assigned to work server 171.64.65.124
17:27:04:WU01:FS00:Requesting new work unit for slot 00: READY cpu:24 from 171.64.65.124
17:27:04:WU01:FS00:Connecting to 171.64.65.124:8080
17:27:25:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
17:27:25:WU01:FS00:Connecting to 171.64.65.124:80
17:28:26:WU01:FS00:Downloading 644.27KiB
17:28:33:WU01:FS00:Download 9.93%
17:28:46:WU01:FS00:Download 29.80%
17:29:00:WU01:FS00:Download 49.67%
17:29:06:WU01:FS00:Download 59.60%
17:29:21:WU01:FS00:Download 79.47%
17:29:30:WU01:FS00:Download 89.40%
17:29:42:WU01:FS00:Download 99.34%
17:29:42:WU01:FS00:Download complete
17:29:42:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9025 run:619 clone:9 gen:0 core:0xa4 unit:0x00000002ab40417c55f3a1fad058dffc
17:29:42:WU01:FS00:Starting
17:29:42:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/beta/Core_a4.fah/FahCore_a4.exe -dir 01 -suffix 01 -version 704 -lifeline 5260 -checkpoint 15 -np 24
17:29:42:WU01:FS00:Started FahCore on PID 4512
17:29:42:WU01:FS00:Core PID:7684
17:29:42:WU01:FS00:FahCore 0xa4 started
17:29:42:WU01:FS00:0xa4:
17:29:42:WU01:FS00:0xa4:*------------------------------*
17:29:42:WU01:FS00:0xa4:Folding@Home Gromacs GB Core
17:29:42:WU01:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
17:29:42:WU01:FS00:0xa4:
17:29:42:WU01:FS00:0xa4:Preparing to commence simulation
17:29:42:WU01:FS00:0xa4:- Looking at optimizations...
17:29:42:WU01:FS00:0xa4:- Created dyn
17:29:42:WU01:FS00:0xa4:- Files status OK
17:29:42:WU01:FS00:0xa4:- Expanded 659225 -> 1402156 (decompressed 212.6 percent)
17:29:42:WU01:FS00:0xa4:Called DecompressByteArray: compressed_data_size=659225 data_size=1402156, decompressed_data_size=1402156 diff=0
17:29:42:WU01:FS00:0xa4:- Digital signature verified
17:29:42:WU01:FS00:0xa4:
17:29:42:WU01:FS00:0xa4:Project: 9025 (Run 619, Clone 9, Gen 0)
17:29:42:WU01:FS00:0xa4:
17:29:42:WU01:FS00:0xa4:Assembly optimizations on if available.
17:29:42:WU01:FS00:0xa4:Entering M.D.
17:29:48:WU01:FS00:0xa4:Mapping NT from 24 to 24
17:29:48:WU01:FS00:0xa4:Completed 0 out of 250000 steps (0%)
17:30:22:WU01:FS00:0xa4:Completed 2500 out of 250000 steps (1%)
17:30:56:WU01:FS00:0xa4:Completed 5000 out of 250000 steps (2%)
*********Truncated************
18:26:49:WU01:FS00:0xa4:Completed 247500 out of 250000 steps (99%)
18:27:24:WU01:FS00:0xa4:Completed 250000 out of 250000 steps (100%)
18:27:25:WU01:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
18:27:25:WU04:FS00:Connecting to 171.67.108.45:8080
18:27:25:WU04:FS00:Assigned to work server 171.64.65.124
18:27:25:WU04:FS00:Requesting new work unit for slot 00: RUNNING cpu:24 from 171.64.65.124
18:27:25:WU04:FS00:Connecting to 171.64.65.124:8080
18:27:35:WU01:FS00:0xa4:
18:27:35:WU01:FS00:0xa4:Finished Work Unit:
18:27:35:WU01:FS00:0xa4:- Reading up to 811608 from "01/wudata_01.trr": Read 811608
18:27:35:WU01:FS00:0xa4:trr file hash check passed.
18:27:35:WU01:FS00:0xa4:- Reading up to 746088 from "01/wudata_01.xtc": Read 746088
18:27:35:WU01:FS00:0xa4:xtc file hash check passed.
18:27:35:WU01:FS00:0xa4:edr file hash check passed.
18:27:35:WU01:FS00:0xa4:logfile size: 23255
18:27:35:WU01:FS00:0xa4:Leaving Run
18:27:37:WU01:FS00:0xa4:- Writing 1583439 bytes of core data to disk...
18:27:37:WU01:FS00:0xa4:Done: 1582927 -> 1538133 (compressed to 97.1 percent)
18:27:37:WU01:FS00:0xa4: ... Done.
18:27:38:WU01:FS00:0xa4:- Shutting down core
18:27:38:WU01:FS00:0xa4:
18:27:38:WU01:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
18:27:38:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
18:27:38:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:9025 run:619 clone:9 gen:0 core:0xa4 unit:0x00000002ab40417c55f3a1fad058dffc
18:27:38:WU01:FS00:Uploading 1.47MiB to 171.64.65.124
18:27:38:WU01:FS00:Connecting to 171.64.65.124:8080
18:28:30:WU04:FS00:Downloading 807.01KiB
18:28:41:WU04:FS00:Download 7.93%
18:28:53:WU04:FS00:Download 15.86%
18:29:00:WU01:FS00:Upload 12.78%
18:29:05:WU04:FS00:Download 23.79%
18:29:16:WU04:FS00:Download 31.72%
18:29:26:WU04:FS00:Download 39.65%
18:29:28:WU01:FS00:Upload 17.04%
18:29:34:WU04:FS00:Download 47.58%
18:29:41:WU04:FS00:Download 55.51%
18:29:50:WU04:FS00:Download 63.44%
18:29:57:WU04:FS00:Download 71.37%
18:30:06:WU01:FS00:Upload 21.30%
18:30:06:WU04:FS00:Download 79.30%
18:30:19:WU04:FS00:Download 87.24%
18:30:31:WU04:FS00:Download 95.17%
18:30:35:WU04:FS00:Download complete
18:30:35:WU04:FS00:Received Unit: id:04 state:DOWNLOAD error:NO_ERROR project:9026 run:444 clone:9 gen:7 core:0xa4 unit:0x0000000aab40417c55f39dfc6abc8344
18:30:35:WU04:FS00:Starting
18:30:35:WU04:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/beta/Core_a4.fah/FahCore_a4.exe -dir 04 -suffix 01 -version 704 -lifeline 5260 -checkpoint 15 -np 24
18:30:35:WU04:FS00:Started FahCore on PID 6984
18:30:35:WU04:FS00:Core PID:11004
18:30:35:WU04:FS00:FahCore 0xa4 started
18:30:35:WU01:FS00:Upload 25.56%
18:30:36:WU04:FS00:0xa4:
18:30:36:WU04:FS00:0xa4:*------------------------------*
18:30:36:WU04:FS00:0xa4:Folding@Home Gromacs GB Core
18:30:36:WU04:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
18:30:36:WU04:FS00:0xa4:
18:30:36:WU04:FS00:0xa4:Preparing to commence simulation
18:30:36:WU04:FS00:0xa4:- Looking at optimizations...
18:30:36:WU04:FS00:0xa4:- Created dyn
18:30:36:WU04:FS00:0xa4:- Files status OK
18:30:36:WU04:FS00:0xa4:- Expanded 825870 -> 1403132 (decompressed 169.8 percent)
18:30:36:WU04:FS00:0xa4:Called DecompressByteArray: compressed_data_size=825870 data_size=1403132, decompressed_data_size=1403132 diff=0
18:30:36:WU04:FS00:0xa4:- Digital signature verified
18:30:36:WU04:FS00:0xa4:
18:30:36:WU04:FS00:0xa4:Project: 9026 (Run 444, Clone 9, Gen 7)
18:30:36:WU04:FS00:0xa4:
18:30:36:WU04:FS00:0xa4:Assembly optimizations on if available.
18:30:36:WU04:FS00:0xa4:Entering M.D.
18:30:42:WU04:FS00:0xa4:Mapping NT from 24 to 24
18:30:42:WU04:FS00:0xa4:Completed 0 out of 250000 steps (0%)
18:31:01:WU01:FS00:Upload 29.82%
18:31:16:WU04:FS00:0xa4:Completed 2500 out of 250000 steps (1%)
18:31:40:WU01:FS00:Upload 34.07%
18:31:51:WU04:FS00:0xa4:Completed 5000 out of 250000 steps (2%)
18:32:25:WU04:FS00:0xa4:Completed 7500 out of 250000 steps (3%)
18:32:25:WU01:FS00:Upload 38.33%
18:32:59:WU04:FS00:0xa4:Completed 10000 out of 250000 steps (4%)
18:33:15:WU01:FS00:Upload 42.59%
18:33:34:WU04:FS00:0xa4:Completed 12500 out of 250000 steps (5%)
18:33:50:WU01:FS00:Upload 46.85%
18:34:08:WU04:FS00:0xa4:Completed 15000 out of 250000 steps (6%)
18:34:31:WU01:FS00:Upload 51.11%
18:34:42:WU04:FS00:0xa4:Completed 17500 out of 250000 steps (7%)
18:35:03:WU01:FS00:Upload 55.37%
18:35:17:WU04:FS00:0xa4:Completed 20000 out of 250000 steps (8%)
18:35:43:WU01:FS00:Upload 59.63%
18:35:51:WU04:FS00:0xa4:Completed 22500 out of 250000 steps (9%)
18:36:01:WU01:FS00:Upload 63.89%
18:36:26:WU04:FS00:0xa4:Completed 25000 out of 250000 steps (10%)
18:36:39:WU01:FS00:Upload 68.15%
18:36:39:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
18:36:39:WU01:FS00:Trying to send results to collection server
18:36:39:WU01:FS00:Uploading 1.47MiB to 171.67.108.46
18:36:39:WU01:FS00:Connecting to 171.67.108.46:8080
18:37:00:WU04:FS00:0xa4:Completed 27500 out of 250000 steps (11%)
18:37:00:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
18:37:00:WU01:FS00:Connecting to 171.67.108.46:80
18:37:21:ERROR:WU01:FS00:Exception: Failed to connect to 171.67.108.46:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
18:37:22:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:9025 run:619 clone:9 gen:0 core:0xa4 unit:0x00000002ab40417c55f3a1fad058dffc
18:37:22:WU01:FS00:Uploading 1.47MiB to 171.64.65.124
18:37:22:WU01:FS00:Connecting to 171.64.65.124:8080
18:37:35:WU04:FS00:0xa4:Completed 30000 out of 250000 steps (12%)
18:37:43:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
18:37:43:WU01:FS00:Connecting to 171.64.65.124:80
18:38:04:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to connect to 171.64.65.124:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
18:38:04:WU01:FS00:Trying to send results to collection server
18:38:04:WU01:FS00:Uploading 1.47MiB to 171.67.108.46
18:38:04:WU01:FS00:Connecting to 171.67.108.46:8080
18:38:09:WU04:FS00:0xa4:Completed 32500 out of 250000 steps (13%)
18:38:25:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
18:38:25:WU01:FS00:Connecting to 171.67.108.46:80
18:38:34:ERROR:WU01:FS00:Exception: Failed to connect to 171.67.108.46:80: No connection could be made because the target machine actively refused it.
18:38:34:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:9025 run:619 clone:9 gen:0 core:0xa4 unit:0x00000002ab40417c55f3a1fad058dffc
18:38:34:WU01:FS00:Uploading 1.47MiB to 171.64.65.124
18:38:34:WU01:FS00:Connecting to 171.64.65.124:8080
18:38:43:WU04:FS00:0xa4:Completed 35000 out of 250000 steps (14%)
18:39:17:WU04:FS00:0xa4:Completed 37500 out of 250000 steps (15%)
18:39:46:WU01:FS00:Upload 12.78%
18:39:52:WU04:FS00:0xa4:Completed 40000 out of 250000 steps (16%)
18:40:09:WU01:FS00:Upload 17.04%
18:40:26:WU04:FS00:0xa4:Completed 42500 out of 250000 steps (17%)
18:40:37:WU01:FS00:Upload 21.30%
18:41:00:WU04:FS00:0xa4:Completed 45000 out of 250000 steps (18%)
18:41:35:WU04:FS00:0xa4:Completed 47500 out of 250000 steps (19%)
18:41:36:WU01:FS00:Upload 25.56%
18:42:03:WU01:FS00:Upload 29.82%
18:42:09:WU04:FS00:0xa4:Completed 50000 out of 250000 steps (20%)
18:42:44:WU04:FS00:0xa4:Completed 52500 out of 250000 steps (21%)
18:42:49:WU01:FS00:Upload 34.07%
18:43:18:WU04:FS00:0xa4:Completed 55000 out of 250000 steps (22%)
18:43:28:WU01:FS00:Upload 38.33%
18:43:52:WU04:FS00:0xa4:Completed 57500 out of 250000 steps (23%)
18:43:58:WU01:FS00:Upload 42.59%
18:44:26:WU04:FS00:0xa4:Completed 60000 out of 250000 steps (24%)
18:44:55:WU01:FS00:Upload 46.85%
18:45:01:WU04:FS00:0xa4:Completed 62500 out of 250000 steps (25%)
18:45:35:WU04:FS00:0xa4:Completed 65000 out of 250000 steps (26%)
18:45:42:WU01:FS00:Upload 51.11%
18:46:08:WU01:FS00:Upload 55.37%
18:46:09:WU04:FS00:0xa4:Completed 67500 out of 250000 steps (27%)
18:46:44:WU04:FS00:0xa4:Completed 70000 out of 250000 steps (28%)
18:47:16:WU01:FS00:Upload 59.63%
18:47:18:WU04:FS00:0xa4:Completed 72500 out of 250000 steps (29%)
18:47:39:WU01:FS00:Upload 63.89%
18:47:53:WU04:FS00:0xa4:Completed 75000 out of 250000 steps (30%)
18:47:57:WU01:FS00:Upload 68.15%
18:48:12:WU01:FS00:Upload 72.41%
18:48:27:WU04:FS00:0xa4:Completed 77500 out of 250000 steps (31%)
18:49:01:WU04:FS00:0xa4:Completed 80000 out of 250000 steps (32%)
18:49:03:WU01:FS00:Upload 76.67%
18:49:36:WU04:FS00:0xa4:Completed 82500 out of 250000 steps (33%)
18:50:10:WU04:FS00:0xa4:Completed 85000 out of 250000 steps (34%)
18:50:18:WU01:FS00:Upload 80.93%
18:50:45:WU04:FS00:0xa4:Completed 87500 out of 250000 steps (35%)
18:50:49:WU01:FS00:Upload 85.19%
18:51:10:WU01:FS00:Upload 89.45%
18:51:19:WU04:FS00:0xa4:Completed 90000 out of 250000 steps (36%)
18:51:53:WU04:FS00:0xa4:Completed 92500 out of 250000 steps (37%)
18:52:14:WU01:FS00:Upload 93.71%
18:52:27:WU04:FS00:0xa4:Completed 95000 out of 250000 steps (38%)
18:52:31:WU01:FS00:Upload 97.96%
18:53:02:WU04:FS00:0xa4:Completed 97500 out of 250000 steps (39%)
18:53:36:WU04:FS00:0xa4:Completed 100000 out of 250000 steps (40%)
18:54:10:WU04:FS00:0xa4:Completed 102500 out of 250000 steps (41%)
18:54:45:WU04:FS00:0xa4:Completed 105000 out of 250000 steps (42%)
18:54:54:WU01:FS00:Upload complete
18:54:54:WU01:FS00:Server responded WORK_QUIT (404)
18:54:54:WARNING:WU01:FS00:Server did not like results, dumping
18:54:54:WU01:FS00:Cleaning up
18:55:19:WU04:FS00:0xa4:Completed 107500 out of 250000 steps (43%)
***********Truncated*************
19:27:54:WU04:FS00:0xa4:Completed 250000 out of 250000 steps (100%)
19:27:55:WU04:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
19:27:55:WU00:FS00:Connecting to 171.67.108.45:8080
19:27:55:WU00:FS00:Assigned to work server 171.64.65.124
19:27:56:WU00:FS00:Requesting new work unit for slot 00: RUNNING cpu:24 from 171.64.65.124
19:27:56:WU00:FS00:Connecting to 171.64.65.124:8080
19:28:05:WU04:FS00:0xa4:
19:28:05:WU04:FS00:0xa4:Finished Work Unit:
19:28:05:WU04:FS00:0xa4:- Reading up to 811560 from "04/wudata_01.trr": Read 811560
19:28:05:WU04:FS00:0xa4:trr file hash check passed.
19:28:05:WU04:FS00:0xa4:- Reading up to 746004 from "04/wudata_01.xtc": Read 746004
19:28:05:WU04:FS00:0xa4:xtc file hash check passed.
19:28:05:WU04:FS00:0xa4:edr file hash check passed.
19:28:05:WU04:FS00:0xa4:logfile size: 23094
19:28:05:WU04:FS00:0xa4:Leaving Run
19:28:06:WU04:FS00:0xa4:- Writing 1583146 bytes of core data to disk...
19:28:06:WU04:FS00:0xa4:Done: 1582634 -> 1537861 (compressed to 97.1 percent)
19:28:06:WU04:FS00:0xa4: ... Done.
19:28:06:WU04:FS00:0xa4:- Shutting down core
19:28:06:WU04:FS00:0xa4:
19:28:06:WU04:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
19:28:07:WU04:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
19:28:07:WU04:FS00:Sending unit results: id:04 state:SEND error:NO_ERROR project:9026 run:444 clone:9 gen:7 core:0xa4 unit:0x0000000aab40417c55f39dfc6abc8344
19:28:07:WU04:FS00:Uploading 1.47MiB to 171.64.65.124
19:28:07:WU04:FS00:Connecting to 171.64.65.124:8080
19:28:16:WU04:FS00:Upload 4.26%
19:29:02:WU00:FS00:Downloading 806.13KiB
19:29:12:WU00:FS00:Download 7.94%
19:29:26:WU00:FS00:Download 15.88%
19:29:39:WU00:FS00:Download 23.82%
19:29:45:WU00:FS00:Download 31.76%
19:29:52:WU00:FS00:Download 39.70%
19:29:58:WU04:FS00:Upload 12.78%
19:29:59:WU00:FS00:Download 47.64%
19:30:12:WU00:FS00:Download 63.51%
19:30:22:WU00:FS00:Download 71.45%
19:30:23:WU04:FS00:Upload 17.04%
19:30:28:WU00:FS00:Download 79.39%
19:30:35:WU00:FS00:Download 87.33%
19:30:44:WU00:FS00:Download 95.27%
19:30:44:WU04:FS00:Upload 21.30%
19:30:44:WU00:FS00:Download complete
19:30:44:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9029 run:147 clone:11 gen:39 core:0xa4 unit:0x0000002cab40417c55f3909ddff62523
19:30:44:WU00:FS00:Starting
19:30:44:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win1/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/beta/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 704 -lifeline 5260 -checkpoint 15 -np 24
19:30:45:WU00:FS00:Started FahCore on PID 11116
19:30:45:WU00:FS00:Core PID:4368
19:30:45:WU00:FS00:FahCore 0xa4 started
19:30:45:WU00:FS00:0xa4:
19:30:45:WU00:FS00:0xa4:*------------------------------*
19:30:45:WU00:FS00:0xa4:Folding@Home Gromacs GB Core
19:30:45:WU00:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
19:30:45:WU00:FS00:0xa4:
19:30:45:WU00:FS00:0xa4:Preparing to commence simulation
19:30:45:WU00:FS00:0xa4:- Looking at optimizations...
19:30:45:WU00:FS00:0xa4:- Created dyn
19:30:45:WU00:FS00:0xa4:- Files status OK
19:30:45:WU00:FS00:0xa4:- Expanded 824960 -> 1398040 (decompressed 169.4 percent)
19:30:45:WU00:FS00:0xa4:Called DecompressByteArray: compressed_data_size=824960 data_size=1398040, decompressed_data_size=1398040 diff=0
19:30:45:WU00:FS00:0xa4:- Digital signature verified
19:30:45:WU00:FS00:0xa4:
19:30:45:WU00:FS00:0xa4:Project: 9029 (Run 147, Clone 11, Gen 39)
19:30:45:WU00:FS00:0xa4:
19:30:45:WU00:FS00:0xa4:Assembly optimizations on if available.
19:30:45:WU00:FS00:0xa4:Entering M.D.
19:30:51:WU00:FS00:0xa4:Mapping NT from 24 to 24
19:30:51:WU00:FS00:0xa4:Completed 0 out of 250000 steps (0%)
19:31:01:WU04:FS00:Upload 25.56%
19:31:25:WU00:FS00:0xa4:Completed 2500 out of 250000 steps (1%)
19:31:59:WU00:FS00:0xa4:Completed 5000 out of 250000 steps (2%)
19:32:09:WU04:FS00:Upload 29.82%
19:32:34:WU00:FS00:0xa4:Completed 7500 out of 250000 steps (3%)
19:32:54:WU04:FS00:Upload 34.08%
19:33:08:WU00:FS00:0xa4:Completed 10000 out of 250000 steps (4%)
19:33:41:WU00:FS00:0xa4:Completed 12500 out of 250000 steps (5%)
19:33:55:WU04:FS00:Upload 38.34%
19:34:08:WU04:FS00:Upload 42.60%
19:34:15:WU00:FS00:0xa4:Completed 15000 out of 250000 steps (6%)
19:34:49:WU00:FS00:0xa4:Completed 17500 out of 250000 steps (7%)
19:34:51:WU04:FS00:Upload 46.86%
19:35:04:WU04:FS00:Upload 51.12%
19:35:23:WU00:FS00:0xa4:Completed 20000 out of 250000 steps (8%)
19:35:57:WU00:FS00:0xa4:Completed 22500 out of 250000 steps (9%)
19:36:27:WU04:FS00:Upload 55.38%
19:36:31:WU00:FS00:0xa4:Completed 25000 out of 250000 steps (10%)
19:36:33:WU04:FS00:Upload 59.64%
19:36:55:WU04:FS00:Upload 63.90%
19:37:05:WU00:FS00:0xa4:Completed 27500 out of 250000 steps (11%)
19:37:39:WU00:FS00:0xa4:Completed 30000 out of 250000 steps (12%)
19:37:41:WU04:FS00:Upload 68.16%
19:38:00:WU04:FS00:Upload 72.42%
19:38:13:WU00:FS00:0xa4:Completed 32500 out of 250000 steps (13%)
19:38:47:WU00:FS00:0xa4:Completed 35000 out of 250000 steps (14%)
19:39:00:WU04:FS00:Upload 76.68%
19:39:21:WU00:FS00:0xa4:Completed 37500 out of 250000 steps (15%)
19:39:24:WU04:FS00:Upload 80.94%
19:39:55:WU00:FS00:0xa4:Completed 40000 out of 250000 steps (16%)
19:39:55:WU04:FS00:Upload 85.20%
19:40:10:WU04:FS00:Upload 89.46%
19:40:28:WU00:FS00:0xa4:Completed 42500 out of 250000 steps (17%)
19:40:37:WU04:FS00:Upload 93.72%
19:41:02:WU00:FS00:0xa4:Completed 45000 out of 250000 steps (18%)
19:41:15:WU04:FS00:Upload 97.98%
19:41:29:WU04:FS00:Upload 100.00%
19:41:36:WU00:FS00:0xa4:Completed 47500 out of 250000 steps (19%)
19:42:10:WU00:FS00:0xa4:Completed 50000 out of 250000 steps (20%)
19:42:44:WU00:FS00:0xa4:Completed 52500 out of 250000 steps (21%)
19:42:51:WU04:FS00:Upload complete
19:42:51:WU04:FS00:Server responded WORK_ACK (400)
19:42:51:WU04:FS00:Final credit estimate, 2275.00 points
19:42:51:WU04:FS00:Cleaning up
19:43:19:WU00:FS00:0xa4:Completed 55000 out of 250000 steps (22%)
Another example of slow transfer ... the long delay for the last 2% or so and the 20 minutes total to upload suggests to me that the server has some serious connection problems.
Then the unit gets dumped. Could it be that there are error in the transfer rather than the processing that are responsible for the unit getting dumped?
So it appears that there was a hidden variable in my WS that was limiting how many jobs could be open and uploaded at once. That was where this problem was coming from, as well as why the net load was so high. I believe we've fixed this successfully, so there shouldn't be more problems. Thanks for your heads up and patience!