129.74.85.15-16 keeps failing upload at 96-99%

[warp-9.9]

This started sometime overnight, and for the past 4 & 1/2 hours the upload has been failing (2 times shown below, 11 total, truncated due to post size restrictions). It bounces back and forth between 129.74.85.15 and 129.74.85.16. I've included a hand scrubbed log file for this work unit and folding slot only, from a resume at about 85% through completion to most recent failure. Then I exited the client. I started the client again and the same thing keeps happening.

One odd thing. The log file indicates that this is project 8004, but the user interface (Status -> Work Unit) indicates project 7035.

• Any way to determine what is wrong and salvage the result of my processing time?
• Is it data corruption somehow?
• How do I verify data correctness or corruption?
• How long should I let it try until I delete it?
• Any ideas how to determine what went wrong?

Code: Select all

14:57:12:WU02:FS01:Starting
14:57:12:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 02 -suffix 01 -version 701 -lifeline 3396 -checkpoint 3 -np 8
14:57:12:WU02:FS01:Started FahCore on PID 3676
14:57:12:WU02:FS01:Core PID:3024
14:57:12:WU02:FS01:FahCore 0xa4 started
14:57:13:WU02:FS01:0xa4:
14:57:13:WU02:FS01:0xa4:*------------------------------*
14:57:13:WU02:FS01:0xa4:Folding@Home Gromacs GB Core
14:57:13:WU02:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
14:57:13:WU02:FS01:0xa4:
14:57:13:WU02:FS01:0xa4:Preparing to commence simulation
14:57:13:WU02:FS01:0xa4:- Ensuring status. Please wait.
14:57:22:WU02:FS01:0xa4:- Looking at optimizations...
14:57:22:WU02:FS01:0xa4:- Working with standard loops on this execution.
14:57:22:WU02:FS01:0xa4:- Previous termination of core was improper.
14:57:22:WU02:FS01:0xa4:- Going to use standard loops.
14:57:22:WU02:FS01:0xa4:- Files status OK
14:57:22:WU02:FS01:0xa4:- Expanded 543494 -> 1303872 (decompressed 239.9 percent)
14:57:22:WU02:FS01:0xa4:Called DecompressByteArray: compressed_data_size=543494 data_size=1303872, decompressed_data_size=1303872 diff=0
14:57:22:WU02:FS01:0xa4:- Digital signature verified
14:57:22:WU02:FS01:0xa4:
14:57:22:WU02:FS01:0xa4:Project: 8004 (Run 173, Clone 1, Gen 279)
14:57:22:WU02:FS01:0xa4:
14:57:22:WU02:FS01:0xa4:Entering M.D.
14:57:28:WU02:FS01:0xa4:Using Gromacs checkpoints
14:57:28:WU02:FS01:0xa4:Mapping NT from 8 to 8 
14:57:28:WU02:FS01:0xa4:Resuming from checkpoint
14:57:28:WU02:FS01:0xa4:Verified 02/wudata_01.log
14:57:28:WU02:FS01:0xa4:Verified 02/wudata_01.trr
14:57:28:WU02:FS01:0xa4:Verified 02/wudata_01.xtc
14:57:28:WU02:FS01:0xa4:Verified 02/wudata_01.edr
14:57:28:WU02:FS01:0xa4:Completed 210140 out of 250000 steps  (84%)
14:58:31:WU02:FS01:0xa4:Completed 212500 out of 250000 steps  (85%)
14:59:57:WU02:FS01:0xa4:Completed 215000 out of 250000 steps  (86%)
15:01:00:WU02:FS01:0xa4:Completed 217500 out of 250000 steps  (87%)
15:01:57:WU02:FS01:0xa4:Completed 220000 out of 250000 steps  (88%)
15:02:55:WU02:FS01:0xa4:Completed 222500 out of 250000 steps  (89%)
15:04:11:WU02:FS01:0xa4:Completed 225000 out of 250000 steps  (90%)
15:05:10:WU02:FS01:0xa4:Completed 227500 out of 250000 steps  (91%)
15:06:21:WU02:FS01:0xa4:Completed 230000 out of 250000 steps  (92%)
15:07:32:WU02:FS01:0xa4:Completed 232500 out of 250000 steps  (93%)
15:08:25:WU02:FS01:0xa4:Completed 235000 out of 250000 steps  (94%)
15:09:22:WU02:FS01:0xa4:Completed 237500 out of 250000 steps  (95%)
15:10:16:WU02:FS01:0xa4:Completed 240000 out of 250000 steps  (96%)
15:11:09:WU02:FS01:0xa4:Completed 242500 out of 250000 steps  (97%)
15:12:02:WU02:FS01:0xa4:Completed 245000 out of 250000 steps  (98%)
15:12:55:WU02:FS01:0xa4:Completed 247500 out of 250000 steps  (99%)
04:28:19:WU02:FS01:0xa4:Completed 25000000 out of 25000000 steps  (100%)
04:28:19:WU02:FS01:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
04:28:29:WU02:FS01:0xa4:
04:28:29:WU02:FS01:0xa4:Finished Work Unit:
04:28:29:WU02:FS01:0xa4:- Reading up to 17021328 from "02/wudata_01.trr": Read 17021328
04:28:29:WU02:FS01:0xa4:trr file hash check passed.
04:28:29:WU02:FS01:0xa4:- Reading up to 1681992 from "02/wudata_01.xtc": Read 1681992
04:28:29:WU02:FS01:0xa4:xtc file hash check passed.
04:28:29:WU02:FS01:0xa4:edr file hash check passed.
04:28:29:WU02:FS01:0xa4:logfile size: 142237
04:28:29:WU02:FS01:0xa4:Leaving Run
04:28:32:WU02:FS01:0xa4:- Writing 18927369 bytes of core data to disk...
04:28:37:WU02:FS01:0xa4:Done: 18926857 -> 17506462 (compressed to 92.4 percent)
04:28:37:WU02:FS01:0xa4:  ... Done.
04:28:45:WU02:FS01:0xa4:- Shutting down core
04:28:45:WU02:FS01:0xa4:
04:28:45:WU02:FS01:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
04:28:45:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
04:28:45:WU02:FS01:Sending unit results: id:02 state:SEND error:OK project:7035 run:0 clone:209 gen:1 core:0xa4 unit:0x000000010001329c50294f5a14313472
04:28:45:WU02:FS01:Uploading 16.70MiB to 129.74.85.15
04:28:45:WU02:FS01:Connecting to 129.74.85.15:8080
04:28:51:WU02:FS01:Upload 0.75%
04:28:59:WU02:FS01:Upload 1.50%
04:29:06:WU02:FS01:Upload 2.25%
04:29:13:WU02:FS01:Upload 2.99%
04:29:19:WU02:FS01:Upload 3.74%
04:29:26:WU02:FS01:Upload 4.49%
04:29:33:WU02:FS01:Upload 5.24%
04:29:40:WU02:FS01:Upload 5.99%
04:29:46:WU02:FS01:Upload 6.74%
04:29:53:WU02:FS01:Upload 7.49%
04:30:00:WU02:FS01:Upload 8.24%
04:30:07:WU02:FS01:Upload 8.98%
04:30:13:WU02:FS01:Upload 9.73%
04:30:20:WU02:FS01:Upload 10.48%
04:30:27:WU02:FS01:Upload 11.23%
04:30:34:WU02:FS01:Upload 11.98%
04:30:40:WU02:FS01:Upload 12.73%
04:30:47:WU02:FS01:Upload 13.48%
04:30:54:WU02:FS01:Upload 14.23%
04:31:01:WU02:FS01:Upload 14.97%
04:31:08:WU02:FS01:Upload 15.72%
04:31:14:WU02:FS01:Upload 16.47%
04:31:21:WU02:FS01:Upload 17.22%
04:31:28:WU02:FS01:Upload 17.97%
04:31:35:WU02:FS01:Upload 18.72%
04:31:41:WU02:FS01:Upload 19.47%
04:31:48:WU02:FS01:Upload 20.21%
04:31:55:WU02:FS01:Upload 20.96%
04:32:02:WU02:FS01:Upload 21.71%
04:32:08:WU02:FS01:Upload 22.46%
04:32:15:WU02:FS01:Upload 23.21%
04:32:22:WU02:FS01:Upload 23.96%
04:32:29:WU02:FS01:Upload 24.71%
04:32:35:WU02:FS01:Upload 25.46%
04:32:42:WU02:FS01:Upload 26.20%
04:32:49:WU02:FS01:Upload 26.95%
04:32:56:WU02:FS01:Upload 27.70%
04:33:02:WU02:FS01:Upload 28.45%
04:33:09:WU02:FS01:Upload 29.20%
04:33:16:WU02:FS01:Upload 29.95%
04:33:23:WU02:FS01:Upload 30.70%
04:33:29:WU02:FS01:Upload 31.44%
04:33:36:WU02:FS01:Upload 32.19%
04:33:43:WU02:FS01:Upload 32.94%
04:33:50:WU02:FS01:Upload 33.69%
04:33:56:WU02:FS01:Upload 34.44%
04:34:03:WU02:FS01:Upload 35.19%
04:34:10:WU02:FS01:Upload 35.94%
04:34:17:WU02:FS01:Upload 36.69%
04:34:23:WU02:FS01:Upload 37.43%
04:34:30:WU02:FS01:Upload 38.18%
04:34:37:WU02:FS01:Upload 38.93%
04:34:44:WU02:FS01:Upload 39.68%
04:34:50:WU02:FS01:Upload 40.43%
04:34:57:WU02:FS01:Upload 41.18%
04:35:04:WU02:FS01:Upload 41.93%
04:35:11:WU02:FS01:Upload 42.68%
04:35:17:WU02:FS01:Upload 43.42%
04:35:24:WU02:FS01:Upload 44.17%
04:35:31:WU02:FS01:Upload 44.92%
04:35:38:WU02:FS01:Upload 45.67%
04:35:45:WU02:FS01:Upload 46.42%
04:35:51:WU02:FS01:Upload 47.17%
04:35:58:WU02:FS01:Upload 47.92%
04:36:05:WU02:FS01:Upload 48.66%
04:36:12:WU02:FS01:Upload 49.41%
04:36:18:WU02:FS01:Upload 50.16%
04:36:25:WU02:FS01:Upload 50.91%
04:36:32:WU02:FS01:Upload 51.66%
04:36:42:WU02:FS01:Upload 52.41%
04:36:49:WU02:FS01:Upload 53.16%
04:36:56:WU02:FS01:Upload 53.91%
04:37:02:WU02:FS01:Upload 54.65%
04:37:09:WU02:FS01:Upload 55.40%
04:37:16:WU02:FS01:Upload 56.15%
04:37:23:WU02:FS01:Upload 56.90%
04:37:29:WU02:FS01:Upload 57.65%
04:37:36:WU02:FS01:Upload 58.40%
04:37:43:WU02:FS01:Upload 59.15%
04:37:50:WU02:FS01:Upload 59.89%
04:37:56:WU02:FS01:Upload 60.64%
04:38:03:WU02:FS01:Upload 61.39%
04:38:10:WU02:FS01:Upload 62.14%
04:38:17:WU02:FS01:Upload 62.89%
04:38:23:WU02:FS01:Upload 63.64%
04:38:30:WU02:FS01:Upload 64.39%
04:38:37:WU02:FS01:Upload 65.14%
04:38:44:WU02:FS01:Upload 65.88%
04:38:51:WU02:FS01:Upload 66.63%
04:38:57:WU02:FS01:Upload 67.38%
04:39:04:WU02:FS01:Upload 68.13%
04:39:11:WU02:FS01:Upload 68.88%
04:39:18:WU02:FS01:Upload 69.63%
04:39:24:WU02:FS01:Upload 70.38%
04:39:31:WU02:FS01:Upload 71.13%
04:39:38:WU02:FS01:Upload 71.87%
04:39:45:WU02:FS01:Upload 72.62%
04:39:51:WU02:FS01:Upload 73.37%
04:39:58:WU02:FS01:Upload 74.12%
04:40:05:WU02:FS01:Upload 74.87%
04:40:12:WU02:FS01:Upload 75.62%
04:40:18:WU02:FS01:Upload 76.37%
04:40:25:WU02:FS01:Upload 77.11%
04:40:32:WU02:FS01:Upload 77.86%
04:40:39:WU02:FS01:Upload 78.61%
04:40:46:WU02:FS01:Upload 79.36%
04:40:52:WU02:FS01:Upload 80.11%
04:40:59:WU02:FS01:Upload 80.86%
04:41:06:WU02:FS01:Upload 81.61%
04:41:13:WU02:FS01:Upload 82.36%
04:41:20:WU02:FS01:Upload 83.10%
04:41:26:WU02:FS01:Upload 83.85%
04:41:33:WU02:FS01:Upload 84.60%
04:41:40:WU02:FS01:Upload 85.35%
04:41:47:WU02:FS01:Upload 86.10%
04:41:53:WU02:FS01:Upload 86.85%
04:42:00:WU02:FS01:Upload 87.60%
04:42:07:WU02:FS01:Upload 88.34%
04:42:14:WU02:FS01:Upload 89.09%
04:42:21:WU02:FS01:Upload 89.84%
04:42:27:WU02:FS01:Upload 90.59%
04:42:34:WU02:FS01:Upload 91.34%
04:42:41:WU02:FS01:Upload 92.09%
04:42:48:WU02:FS01:Upload 92.84%
04:42:54:WU02:FS01:Upload 93.59%
04:43:01:WU02:FS01:Upload 94.33%
04:43:08:WU02:FS01:Upload 95.08%
04:43:14:WU02:FS01:Upload 95.83%
04:43:21:WU02:FS01:Upload 96.58%
04:43:28:WU02:FS01:Upload 97.33%
04:43:35:WU02:FS01:Upload 98.08%
04:43:42:WU02:FS01:Upload 98.83%
04:43:44:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
04:43:44:WU02:FS01:Trying to send results to collection server
04:43:44:WU02:FS01:Uploading 16.70MiB to 129.74.85.16
04:43:44:WU02:FS01:Connecting to 129.74.85.16:8080
04:43:50:WU02:FS01:Upload 0.75%
04:43:57:WU02:FS01:Upload 1.50%
04:44:04:WU02:FS01:Upload 2.25%
04:44:10:WU02:FS01:Upload 2.99%
04:44:17:WU02:FS01:Upload 3.74%
04:44:24:WU02:FS01:Upload 4.49%
04:44:30:WU02:FS01:Upload 4.87%
04:44:38:WU02:FS01:Upload 5.24%
04:44:45:WU02:FS01:Upload 5.62%
04:44:51:WU02:FS01:Upload 5.99%
04:44:58:WU02:FS01:Upload 6.36%
04:45:05:WU02:FS01:Upload 6.74%
04:45:12:WU02:FS01:Upload 7.49%
04:45:21:WU02:FS01:Upload 8.24%
04:45:28:WU02:FS01:Upload 8.98%
04:45:35:WU02:FS01:Upload 9.73%
04:45:41:WU02:FS01:Upload 10.48%
04:45:48:WU02:FS01:Upload 11.23%
04:45:55:WU02:FS01:Upload 11.98%
04:46:02:WU02:FS01:Upload 12.73%
04:46:08:WU02:FS01:Upload 13.48%
04:46:15:WU02:FS01:Upload 14.23%
04:46:22:WU02:FS01:Upload 14.97%
04:46:29:WU02:FS01:Upload 15.72%
04:46:35:WU02:FS01:Upload 16.47%
04:46:42:WU02:FS01:Upload 17.22%
04:46:49:WU02:FS01:Upload 17.97%
04:46:56:WU02:FS01:Upload 18.72%
04:47:02:WU02:FS01:Upload 19.47%
04:47:09:WU02:FS01:Upload 20.21%
04:47:16:WU02:FS01:Upload 20.96%
04:47:23:WU02:FS01:Upload 21.71%
04:47:29:WU02:FS01:Upload 22.46%
04:47:36:WU02:FS01:Upload 23.21%
04:47:43:WU02:FS01:Upload 23.96%
04:47:50:WU02:FS01:Upload 24.71%
04:47:56:WU02:FS01:Upload 25.46%
04:48:03:WU02:FS01:Upload 26.20%
04:48:10:WU02:FS01:Upload 26.95%
04:48:17:WU02:FS01:Upload 27.70%
04:48:23:WU02:FS01:Upload 28.45%
04:48:30:WU02:FS01:Upload 29.20%
04:48:37:WU02:FS01:Upload 29.95%
04:48:44:WU02:FS01:Upload 30.70%
04:48:50:WU02:FS01:Upload 31.44%
04:48:57:WU02:FS01:Upload 32.19%
04:49:04:WU02:FS01:Upload 32.94%
04:49:11:WU02:FS01:Upload 33.69%
04:49:17:WU02:FS01:Upload 34.44%
04:49:24:WU02:FS01:Upload 35.19%
04:49:31:WU02:FS01:Upload 35.94%
04:49:37:WU02:FS01:Upload 36.69%
04:49:44:WU02:FS01:Upload 37.43%
04:49:51:WU02:FS01:Upload 38.18%
04:49:58:WU02:FS01:Upload 38.93%
04:50:05:WU02:FS01:Upload 39.68%
04:50:11:WU02:FS01:Upload 40.43%
04:50:18:WU02:FS01:Upload 41.18%
04:50:25:WU02:FS01:Upload 41.93%
04:50:32:WU02:FS01:Upload 42.68%
04:50:38:WU02:FS01:Upload 43.42%
04:50:45:WU02:FS01:Upload 44.17%
04:50:52:WU02:FS01:Upload 44.92%
04:50:59:WU02:FS01:Upload 45.67%
04:51:05:WU02:FS01:Upload 46.42%
04:51:12:WU02:FS01:Upload 47.17%
04:51:19:WU02:FS01:Upload 47.92%
04:51:26:WU02:FS01:Upload 48.66%
04:51:32:WU02:FS01:Upload 49.41%
04:51:39:WU02:FS01:Upload 50.16%
04:51:46:WU02:FS01:Upload 50.91%
04:51:53:WU02:FS01:Upload 51.66%
04:51:59:WU02:FS01:Upload 52.41%
04:52:06:WU02:FS01:Upload 53.16%
04:52:13:WU02:FS01:Upload 53.91%
04:52:20:WU02:FS01:Upload 54.65%
04:52:26:WU02:FS01:Upload 55.40%
04:52:33:WU02:FS01:Upload 56.15%
04:52:40:WU02:FS01:Upload 56.90%
04:52:47:WU02:FS01:Upload 57.65%
04:52:53:WU02:FS01:Upload 58.40%
04:53:00:WU02:FS01:Upload 59.15%
04:53:07:WU02:FS01:Upload 59.89%
04:53:14:WU02:FS01:Upload 60.64%
04:53:20:WU02:FS01:Upload 61.39%
04:53:27:WU02:FS01:Upload 62.14%
04:53:34:WU02:FS01:Upload 62.89%
04:53:41:WU02:FS01:Upload 63.64%
04:53:48:WU02:FS01:Upload 64.39%
04:53:54:WU02:FS01:Upload 65.14%
04:54:01:WU02:FS01:Upload 65.88%
04:54:08:WU02:FS01:Upload 66.63%
04:54:15:WU02:FS01:Upload 67.38%
04:54:21:WU02:FS01:Upload 68.13%
04:54:28:WU02:FS01:Upload 68.88%
04:54:35:WU02:FS01:Upload 69.63%
04:54:41:WU02:FS01:Upload 70.38%
04:54:48:WU02:FS01:Upload 71.13%
04:54:55:WU02:FS01:Upload 71.87%
04:55:02:WU02:FS01:Upload 72.62%
04:55:09:WU02:FS01:Upload 73.37%
04:55:15:WU02:FS01:Upload 74.12%
04:55:22:WU02:FS01:Upload 74.87%
04:55:29:WU02:FS01:Upload 75.62%
04:55:36:WU02:FS01:Upload 76.37%
04:55:43:WU02:FS01:Upload 77.11%
04:55:49:WU02:FS01:Upload 77.86%
04:55:56:WU02:FS01:Upload 78.61%
04:56:03:WU02:FS01:Upload 79.36%
04:56:10:WU02:FS01:Upload 80.11%
04:56:16:WU02:FS01:Upload 80.86%
04:56:23:WU02:FS01:Upload 81.61%
04:56:30:WU02:FS01:Upload 82.36%
04:56:37:WU02:FS01:Upload 83.10%
04:56:43:WU02:FS01:Upload 83.85%
04:56:50:WU02:FS01:Upload 84.60%
04:56:57:WU02:FS01:Upload 85.35%
04:57:04:WU02:FS01:Upload 86.10%
04:57:11:WU02:FS01:Upload 86.85%
04:57:17:WU02:FS01:Upload 87.60%
04:57:24:WU02:FS01:Upload 88.34%
04:57:31:WU02:FS01:Upload 89.09%
04:57:37:WU02:FS01:Upload 89.84%
04:57:44:WU02:FS01:Upload 90.59%
04:57:51:WU02:FS01:Upload 91.34%
04:57:58:WU02:FS01:Upload 92.09%
04:58:04:WU02:FS01:Upload 92.84%
04:58:11:WU02:FS01:Upload 93.59%
04:58:18:WU02:FS01:Upload 94.33%
04:58:25:WU02:FS01:Upload 95.08%
04:58:32:WU02:FS01:Upload 95.83%
04:58:39:WU02:FS01:Upload 96.58%
04:58:43:ERROR:WU02:FS01:Exception: Transfer failed
04:58:43:WU02:FS01:Sending unit results: id:02 state:SEND error:OK project:7035 run:0 clone:209 gen:1 core:0xa4 unit:0x000000010001329c50294f5a14313472
04:58:43:WU02:FS01:Uploading 16.70MiB to 129.74.85.15
04:58:43:WU02:FS01:Connecting to 129.74.85.15:8080
04:58:49:WU02:FS01:Upload 0.75%

[bruce]

One odd thing. The log file indicates that this is project 8004, but the user interface (Status -> Work Unit) indicates project 7035.

The project being processed is not the same as the project being uploaded.

[*]Any way to determine what is wrong and salvage the result of my processing time?[/*]
See below.

[*]Is it data corruption somehow?[/*]
Probably not. See below.

[*]How do I verify data correctness or corruption?[/*]
I don't know of any method available to us.

[*]How long should I let it try until I delete it?[/*]
See below.

[*]Any ideas how to determine what went wrong?[/*]
I'll ask. See below.

Code: Select all

04:28:45:WU02:FS01:Sending unit results: id:02 state:SEND error:OK project:7035 run:0 clone:209 gen:1 core:0xa4 unit:0x000000010001329c50294f5a14313472
04:28:45:WU02:FS01:Uploading 16.70MiB to 129.74.85.15
04:28:45:WU02:FS01:Connecting to 129.74.85.15:8080
. . . ,.
04:43:42:WU02:FS01:Upload 98.83%
04:43:44:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed

The fact that exactly 15 minutes elapsed between the start of the upload and the termination of the upload probably means that the server is set to abort any connection that takes more than 15 minutes. (There is some number associated with every server.) If my guess is correct, you either need for your internet connection to be very slightly faster, for the result to be slightly smaller, or we need to get somebody to increase the 15 minute limit. You can't really do anything about any of those factors so I'll ask the servers' owner about #3.

[abdulwahidc]

Hi

I've changed the timeout on our server to 30 minutes.

Please let me know if there are any other issues.

[kiore]

Hi

I've changed the timeout on our server to 30 minutes.

Please let me know if there are any other issues.

Those of us with poor connectivity thank you .