Code: Select all
06:16:22:WU01:FS00:Assigned to work server 128.143.231.202
06:16:22:WU01:FS00:Requesting new work unit for slot 00: RUNNING cpu:36 from 128.143.231.202
06:16:22:WU01:FS00:Connecting to 128.143.231.202:8080
06:16:23:WU01:FS00:Downloading 512B
06:16:23:WU01:FS00:Download complete
06:16:23:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:6096 run:9 clone:85 gen:119 core:0xa3 unit:0x000000cf0a3b1e594f1afb7efee33d1e
06:16:32:WU00:FS00:0xa3:
06:16:32:WU00:FS00:0xa3:Finished Work Unit:
06:16:32:WU00:FS00:0xa3:- Reading up to 12102120 from "00/wudata_01.trr": Read 12102120
06:16:32:WU00:FS00:0xa3:trr file hash check passed.
06:16:32:WU00:FS00:0xa3:edr file hash check passed.
06:16:32:WU00:FS00:0xa3:logfile size: 54541
06:16:32:WU00:FS00:0xa3:Leaving Run
06:16:32:WU00:FS00:0xa3:- Writing 12190337 bytes of core data to disk...
06:16:34:WU00:FS00:0xa3:Done: 12189825 -> 11292087 (compressed to 92.6 percent)
06:16:34:WU00:FS00:0xa3: ... Done.
06:18:12:WU00:FS00:0xa3:- Shutting down core
06:18:12:WU00:FS00:0xa3:
06:18:12:WU00:FS00:0xa3:Folding@home Core Shutdown: FINISHED_UNIT
06:18:24:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
06:18:24:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:6096 run:2 clone:12 gen:152 core:0xa3 unit:0x000000e90a3b1e594f1af862b00337bd
06:18:24:WU00:FS00:Uploading 10.77MiB to 128.143.231.202
06:18:24:WU00:FS00:Connecting to 128.143.231.202:8080
06:18:24:WU01:FS00:Starting
06:18:24:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 01 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
06:18:24:WU01:FS00:Started FahCore on PID 27794
06:18:24:WU01:FS00:Core PID:27798
06:18:24:WU01:FS00:FahCore 0xa3 started
[93m06:18:25:WARNING:WU01:FS00:FahCore returned: FILE_IO_ERROR (117 = 0x75)[0m
[93m06:18:25:WARNING:WU01:FS00:Fatal error, dumping[0m
06:18:25:WU01:FS00:Sending unit results: id:01 state:SEND error:DUMPED project:6096 run:9 clone:85 gen:119 core:0xa3 unit:0x000000cf0a3b1e594f1afb7efee33d1e
06:18:25:WU01:FS00:Connecting to 128.143.231.202:8080
06:18:26:WU02:FS00:Connecting to 171.67.108.200:8080
06:18:26:WU01:FS00:Server responded WORK_ACK (400)
06:18:26:WU01:FS00:Cleaning up
06:18:28:WU02:FS00:Assigned to work server 128.143.231.202
06:18:28:WU02:FS00:Requesting new work unit for slot 00: READY cpu:36 from 128.143.231.202
06:18:28:WU02:FS00:Connecting to 128.143.231.202:8080
06:18:29:WU02:FS00:Downloading 512B
06:18:29:WU02:FS00:Download complete
06:18:29:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:6096 run:6 clone:37 gen:105 core:0xa3 unit:0x000000da0a3b1e594f1afa1a4b265d81
06:18:29:WU02:FS00:Starting
06:18:29:WU02:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 02 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
06:18:29:WU02:FS00:Started FahCore on PID 27799
06:18:29:WU02:FS00:Core PID:27803
06:18:29:WU02:FS00:FahCore 0xa3 started
[93m06:18:30:WARNING:WU02:FS00:FahCore returned: FILE_IO_ERROR (117 = 0x75)[0m
[93m06:18:30:WARNING:WU02:FS00:Fatal error, dumping[0m
06:18:30:WU02:FS00:Sending unit results: id:02 state:SEND error:DUMPED project:6096 run:6 clone:37 gen:105 core:0xa3 unit:0x000000da0a3b1e594f1afa1a4b265d81
06:18:30:WU02:FS00:Connecting to 128.143.231.202:8080
06:18:30:WU00:FS00:Upload 15.67%
06:18:31:WU01:FS00:Connecting to 171.67.108.200:8080
06:18:32:WU02:FS00:Server responded WORK_ACK (400)
06:18:32:WU02:FS00:Cleaning up
06:18:34:WU01:FS00:Assigned to work server 128.143.231.202
06:18:34:WU01:FS00:Requesting new work unit for slot 00: READY cpu:36 from 128.143.231.202
06:18:34:WU01:FS00:Connecting to 128.143.231.202:8080
06:18:36:WU00:FS00:Upload 31.34%
06:18:37:WU01:FS00:Downloading 512B
06:18:37:WU01:FS00:Download complete
06:18:37:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:6096 run:9 clone:62 gen:126 core:0xa3 unit:0x000000d00a3b1e594f1afb68d01a81f7
06:18:37:WU01:FS00:Starting
06:18:37:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 01 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
06:18:37:WU01:FS00:Started FahCore on PID 27804
06:18:37:WU01:FS00:Core PID:27808
06:18:37:WU01:FS00:FahCore 0xa3 started
[93m06:18:37:WARNING:WU01:FS00:FahCore returned: FILE_IO_ERROR (117 = 0x75)[0m
[93m06:18:37:WARNING:WU01:FS00:Fatal error, dumping[0m
06:18:37:WU01:FS00:Sending unit results: id:01 state:SEND error:DUMPED project:6096 run:9 clone:62 gen:126 core:0xa3 unit:0x000000d00a3b1e594f1afb68d01a81f7
06:18:37:WU01:FS00:Connecting to 128.143.231.202:8080
06:18:40:WU02:FS00:Connecting to 171.67.108.200:8080
06:18:41:WU01:FS00:Server responded WORK_ACK (400)
06:18:41:WU01:FS00:Cleaning up
06:18:42:WU00:FS00:Upload 48.17%
06:18:43:WU02:FS00:Assigned to work server 128.143.231.202
06:18:43:WU02:FS00:Requesting new work unit for slot 00: READY cpu:36 from 128.143.231.202
06:18:43:WU02:FS00:Connecting to 128.143.231.202:8080
06:18:45:WU02:FS00:Downloading 512B
06:18:45:WU02:FS00:Download complete
06:18:45:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:6096 run:5 clone:44 gen:102 core:0xa3 unit:0x000000df0a3b1e594f1af9b8ca236eb5
06:18:45:WU02:FS00:Starting
06:18:45:WU02:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 02 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
06:18:45:WU02:FS00:Started FahCore on PID 27809
06:18:45:WU02:FS00:Core PID:27813
06:18:45:WU02:FS00:FahCore 0xa3 started
[93m06:18:46:WARNING:WU02:FS00:FahCore returned: FILE_IO_ERROR (117 = 0x75)[0m
[93m06:18:46:WARNING:WU02:FS00:Fatal error, dumping[0m
06:18:46:WU02:FS00:Sending unit results: id:02 state:SEND error:DUMPED project:6096 run:5 clone:44 gen:102 core:0xa3 unit:0x000000df0a3b1e594f1af9b8ca236eb5
06:18:46:WU02:FS00:Connecting to 128.143.231.202:8080
06:18:48:WU00:FS00:Upload 61.52%
06:18:48:WU01:FS00:Connecting to 171.67.108.200:8080
06:18:49:WU02:FS00:Server responded WORK_ACK (400)
06:18:49:WU02:FS00:Cleaning up
06:18:52:WU01:FS00:Assigned to work server 128.143.231.202
06:18:52:WU01:FS00:Requesting new work unit for slot 00: READY cpu:36 from 128.143.231.202
06:18:52:WU01:FS00:Connecting to 128.143.231.202:8080
06:18:54:WU00:FS00:Upload 76.61%
06:18:55:WU01:FS00:Downloading 3.64MiB
06:19:00:WU01:FS00:Download complete
06:19:00:WU00:FS00:Upload 92.27%
06:19:00:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:6096 run:2 clone:19 gen:101 core:0xa3 unit:0x000000c10a3b1e594f1af86be3722fc2
06:19:00:WU01:FS00:Starting
06:19:00:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 01 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
06:19:00:WU01:FS00:Started FahCore on PID 27814
06:19:00:WU01:FS00:Core PID:27818
06:19:00:WU01:FS00:FahCore 0xa3 started
06:19:01:WU01:FS00:0xa3:
06:19:01:WU01:FS00:0xa3:*------------------------------*
06:19:01:WU01:FS00:0xa3:Folding@Home Gromacs SMP Core
06:19:01:WU01:FS00:0xa3:Version 2.27 (Dec. 15, 2010)
06:19:01:WU01:FS00:0xa3:
06:19:01:WU01:FS00:0xa3:Preparing to commence simulation
06:19:01:WU01:FS00:0xa3:- Looking at optimizations...
06:19:01:WU01:FS00:0xa3:- Created dyn
06:19:01:WU01:FS00:0xa3:- Files status OK
06:19:01:WU01:FS00:0xa3:- Expanded 3812275 -> 4169088 (decompressed 109.3 percent)
06:19:01:WU01:FS00:0xa3:Called DecompressByteArray: compressed_data_size=3812275 data_size=4169088, decompressed_data_size=4169088 diff=0
06:19:01:WU01:FS00:0xa3:- Digital signature verified
06:19:01:WU01:FS00:0xa3:
06:19:01:WU01:FS00:0xa3:Project: 6096 (Run 2, Clone 19, Gen 101)
06:19:01:WU01:FS00:0xa3:
06:19:01:WU01:FS00:0xa3:Assembly optimizations on if available.
06:19:01:WU01:FS00:0xa3:Entering M.D.
06:19:06:WU00:FS00:Upload complete
06:19:06:WU00:FS00:Server responded WORK_ACK (400)
06:19:06:WU00:FS00:Final credit estimate, 24001.00 points
06:19:06:WU00:FS00:Cleaning up
06:19:07:WU01:FS00:0xa3:Mapping NT from 36 to 32
06:19:07:WU01:FS00:0xa3:Completed 0 out of 500000 steps (0%)
06:21:31:WU01:FS00:0xa3:Completed 5000 out of 500000 steps (1%)
06:23:54:WU01:FS00:0xa3:Completed 10000 out of 500000 steps (2%)
06:26:17:WU01:FS00:0xa3:Completed 15000 out of 500000 steps (3%)
06:28:40:WU01:FS00:0xa3:Completed 20000 out of 500000 steps (4%)
06:31:03:WU01:FS00:0xa3:Completed 25000 out of 500000 steps (5%)
06:33:25:WU01:FS00:0xa3:Completed 30000 out of 500000 steps (6%)
06:35:48:WU01:FS00:0xa3:Completed 35000 out of 500000 steps (7%)
======= processing continues and I complete
======= this and a couple more Project 6096s
======= and it happens again
14:15:44:WU01:FS00:Assigned to work server 128.143.231.202
14:15:44:WU01:FS00:Requesting new work unit for slot 00: RUNNING cpu:36 from 128.143.231.202
14:15:44:WU01:FS00:Connecting to 128.143.231.202:8080
14:15:44:WU01:FS00:Downloading 512B
14:15:44:WU01:FS00:Download complete
14:15:45:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:6096 run:2 clone:1 gen:199 core:0xa3 unit:0x000001200a3b1e594f1af855f152d9c8
14:15:49:WU00:FS00:0xa3:
14:15:49:WU00:FS00:0xa3:Finished Work Unit:
14:15:49:WU00:FS00:0xa3:- Reading up to 12102120 from "00/wudata_01.trr": Read 12102120
14:15:49:WU00:FS00:0xa3:trr file hash check passed.
14:15:49:WU00:FS00:0xa3:edr file hash check passed.
14:15:49:WU00:FS00:0xa3:logfile size: 54540
14:15:49:WU00:FS00:0xa3:Leaving Run
14:15:49:WU00:FS00:0xa3:- Writing 12190336 bytes of core data to disk...
14:15:51:WU00:FS00:0xa3:Done: 12189824 -> 11293543 (compressed to 92.6 percent)
14:15:51:WU00:FS00:0xa3: ... Done.
14:17:28:WU00:FS00:0xa3:- Shutting down core
14:17:28:WU00:FS00:0xa3:
14:17:28:WU00:FS00:0xa3:Folding@home Core Shutdown: FINISHED_UNIT
14:17:40:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
14:17:40:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:6096 run:3 clone:51 gen:112 core:0xa3 unit:0x000000cf0a3b1e594f1af8f50581727e
14:17:40:WU00:FS00:Uploading 10.77MiB to 128.143.231.202
14:17:40:WU00:FS00:Connecting to 128.143.231.202:8080
14:17:40:WU01:FS00:Starting
14:17:40:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 01 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
14:17:40:WU01:FS00:Started FahCore on PID 29795
14:17:40:WU01:FS00:Core PID:29799
14:17:40:WU01:FS00:FahCore 0xa3 started
[93m14:17:41:WARNING:WU01:FS00:FahCore returned: FILE_IO_ERROR (117 = 0x75)[0m
[93m14:17:41:WARNING:WU01:FS00:Fatal error, dumping[0m
14:17:41:WU01:FS00:Sending unit results: id:01 state:SEND error:DUMPED project:6096 run:2 clone:1 gen:199 core:0xa3 unit:0x000001200a3b1e594f1af855f152d9c8
14:17:41:WU01:FS00:Connecting to 128.143.231.202:8080
14:17:42:WU02:FS00:Connecting to 171.67.108.200:8080
14:17:43:WU01:FS00:Server responded WORK_ACK (400)
14:17:43:WU01:FS00:Cleaning up
14:17:44:WU02:FS00:Assigned to work server 128.143.231.202
14:17:44:WU02:FS00:Requesting new work unit for slot 00: READY cpu:36 from 128.143.231.202
14:17:44:WU02:FS00:Connecting to 128.143.231.202:8080
14:17:45:WU02:FS00:Downloading 512B
14:17:45:WU02:FS00:Download complete
14:17:45:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:6096 run:8 clone:21 gen:100 core:0xa3 unit:0x000000c40a3b1e594f1afad62969b48e
14:17:45:WU02:FS00:Starting
14:17:45:WU02:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 02 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
14:17:45:WU02:FS00:Started FahCore on PID 29800
14:17:45:WU02:FS00:Core PID:29804
14:17:45:WU02:FS00:FahCore 0xa3 started
[93m14:17:45:WARNING:WU02:FS00:FahCore returned: FILE_IO_ERROR (117 = 0x75)[0m
[93m14:17:45:WARNING:WU02:FS00:Fatal error, dumping[0m
14:17:45:WU02:FS00:Sending unit results: id:02 state:SEND error:DUMPED project:6096 run:8 clone:21 gen:100 core:0xa3 unit:0x000000c40a3b1e594f1afad62969b48e
14:17:45:WU02:FS00:Connecting to 128.143.231.202:8080
14:17:46:WU00:FS00:Upload 11.61%
14:17:46:WU01:FS00:Connecting to 171.67.108.200:8080
14:17:47:WU02:FS00:Server responded WORK_ACK (400)
14:17:47:WU02:FS00:Cleaning up
14:17:48:WU01:FS00:Assigned to work server 128.143.231.202
14:17:48:WU01:FS00:Requesting new work unit for slot 00: READY cpu:36 from 128.143.231.202
14:17:48:WU01:FS00:Connecting to 128.143.231.202:8080
14:17:50:WU01:FS00:Downloading 512B
14:17:50:WU01:FS00:Download complete
14:17:50:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:6096 run:6 clone:94 gen:100 core:0xa3 unit:0x000000da0a3b1e594f1afa5395b3d7bb
14:17:50:WU01:FS00:Starting
14:17:50:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 01 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
14:17:50:WU01:FS00:Started FahCore on PID 29805
14:17:50:WU01:FS00:Core PID:29809
14:17:50:WU01:FS00:FahCore 0xa3 started
[93m14:17:50:WARNING:WU01:FS00:FahCore returned: FILE_IO_ERROR (117 = 0x75)[0m
[93m14:17:50:WARNING:WU01:FS00:Fatal error, dumping[0m
14:17:50:WU01:FS00:Sending unit results: id:01 state:SEND error:DUMPED project:6096 run:6 clone:94 gen:100 core:0xa3 unit:0x000000da0a3b1e594f1afa5395b3d7bb
14:17:50:WU01:FS00:Connecting to 128.143.231.202:8080
14:17:52:WU00:FS00:Upload 29.59%
14:17:53:WU02:FS00:Connecting to 171.67.108.200:8080
14:17:54:WU01:FS00:Server responded WORK_ACK (400)
14:17:54:WU01:FS00:Cleaning up
14:17:58:WU02:FS00:Assigned to work server 128.143.231.202
14:17:58:WU02:FS00:Requesting new work unit for slot 00: READY cpu:36 from 128.143.231.202
14:17:58:WU02:FS00:Connecting to 128.143.231.202:8080
14:17:59:WU00:FS00:Upload 43.52%
14:18:00:WU02:FS00:Downloading 512B
14:18:00:WU02:FS00:Download complete
14:18:00:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:6096 run:0 clone:33 gen:101 core:0xa3 unit:0x000000c50a3b1e594f1af7ab3a939476
14:18:01:WU02:FS00:Starting
14:18:01:WU02:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 02 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
14:18:01:WU02:FS00:Started FahCore on PID 29810
14:18:01:WU02:FS00:Core PID:29814
14:18:01:WU02:FS00:FahCore 0xa3 started
[93m14:18:01:WARNING:WU02:FS00:FahCore returned: FILE_IO_ERROR (117 = 0x75)[0m
[93m14:18:01:WARNING:WU02:FS00:Fatal error, dumping[0m
14:18:01:WU02:FS00:Sending unit results: id:02 state:SEND error:DUMPED project:6096 run:0 clone:33 gen:101 core:0xa3 unit:0x000000c50a3b1e594f1af7ab3a939476
14:18:01:WU02:FS00:Connecting to 128.143.231.202:8080
14:18:03:WU01:FS00:Connecting to 171.67.108.200:8080
14:18:04:WU02:FS00:Server responded WORK_ACK (400)
14:18:04:WU02:FS00:Cleaning up
14:18:05:WU00:FS00:Upload 60.93%
14:18:06:WU01:FS00:Assigned to work server 128.143.231.202
14:18:06:WU01:FS00:Requesting new work unit for slot 00: READY cpu:36 from 128.143.231.202
14:18:06:WU01:FS00:Connecting to 128.143.231.202:8080
14:18:11:WU00:FS00:Upload 76.02%
14:18:13:WU01:FS00:Downloading 3.64MiB
14:18:17:WU00:FS00:Upload 89.94%
14:18:17:WU01:FS00:Download complete
14:18:17:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:6096 run:2 clone:37 gen:98 core:0xa3 unit:0x000000c30a3b1e594f1af88007871bf5
14:18:17:WU01:FS00:Starting
14:18:17:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 01 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
14:18:17:WU01:FS00:Started FahCore on PID 29815
14:18:17:WU01:FS00:Core PID:29819
14:18:17:WU01:FS00:FahCore 0xa3 started
14:18:18:WU01:FS00:0xa3:
14:18:18:WU01:FS00:0xa3:*------------------------------*
14:18:18:WU01:FS00:0xa3:Folding@Home Gromacs SMP Core
14:18:18:WU01:FS00:0xa3:Version 2.27 (Dec. 15, 2010)
14:18:18:WU01:FS00:0xa3:
14:18:18:WU01:FS00:0xa3:Preparing to commence simulation
14:18:18:WU01:FS00:0xa3:- Looking at optimizations...
14:18:18:WU01:FS00:0xa3:- Created dyn
14:18:18:WU01:FS00:0xa3:- Files status OK
14:18:18:WU01:FS00:0xa3:- Expanded 3811644 -> 4169088 (decompressed 109.3 percent)
14:18:18:WU01:FS00:0xa3:Called DecompressByteArray: compressed_data_size=3811644 data_size=4169088, decompressed_data_size=4169088 diff=0
14:18:18:WU01:FS00:0xa3:- Digital signature verified
14:18:18:WU01:FS00:0xa3:
14:18:18:WU01:FS00:0xa3:Project: 6096 (Run 2, Clone 37, Gen 98)
14:18:18:WU01:FS00:0xa3:
14:18:18:WU01:FS00:0xa3:Assembly optimizations on if available.
14:18:18:WU01:FS00:0xa3:Entering M.D.
14:18:24:WU00:FS00:Upload complete
14:18:24:WU00:FS00:Server responded WORK_ACK (400)
14:18:24:WU00:FS00:Final credit estimate, 24082.00 points
14:18:24:WU00:FS00:Cleaning up
14:18:24:WU01:FS00:0xa3:Mapping NT from 36 to 32
14:18:24:WU01:FS00:0xa3:Completed 0 out of 500000 steps (0%)
14:20:48:WU01:FS00:0xa3:Completed 5000 out of 500000 steps (1%)
14:23:11:WU01:FS00:0xa3:Completed 10000 out of 500000 steps (2%)
14:25:35:WU01:FS00:0xa3:Completed 15000 out of 500000 steps (3%)
14:27:58:WU01:FS00:0xa3:Completed 20000 out of 500000 steps (4%)
======= processing continues and I complete
======= this and several more Project 6095s and 6096s
======= and it happens again
02:34:45:WU01:FS00:0xa3:Completed 500000 out of 500000 steps (100%)
02:34:45:WU00:FS00:Connecting to 171.67.108.200:8080
02:34:45:WU00:FS00:Assigned to work server 128.143.231.202
02:34:45:WU00:FS00:Requesting new work unit for slot 00: RUNNING cpu:36 from 128.143.231.202
02:34:45:WU00:FS00:Connecting to 128.143.231.202:8080
02:34:46:WU00:FS00:Downloading 512B
02:34:46:WU00:FS00:Download complete
02:34:46:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:6096 run:1 clone:52 gen:114 core:0xa3 unit:0x000000bd0a3b1e594f1af827ce75a21b
02:34:46:WU01:FS00:0xa3:DynamicWrapper: Finished Work Unit: sleep=10000
02:34:56:WU01:FS00:0xa3:
02:34:56:WU01:FS00:0xa3:Finished Work Unit:
02:34:56:WU01:FS00:0xa3:- Reading up to 12092904 from "01/wudata_01.trr": Read 12092904
02:34:56:WU01:FS00:0xa3:trr file hash check passed.
02:34:56:WU01:FS00:0xa3:edr file hash check passed.
02:34:56:WU01:FS00:0xa3:logfile size: 54604
02:34:56:WU01:FS00:0xa3:Leaving Run
02:34:59:WU01:FS00:0xa3:- Writing 12181184 bytes of core data to disk...
02:35:01:WU01:FS00:0xa3:Done: 12180672 -> 11292742 (compressed to 92.7 percent)
02:35:01:WU01:FS00:0xa3: ... Done.
02:36:38:WU01:FS00:0xa3:- Shutting down core
02:36:38:WU01:FS00:0xa3:
02:36:38:WU01:FS00:0xa3:Folding@home Core Shutdown: FINISHED_UNIT
02:36:50:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
02:36:50:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:6095 run:29 clone:24 gen:103 core:0xa3 unit:0x000000850a3b1e594f25cedd2acb0474
02:36:50:WU01:FS00:Uploading 10.77MiB to 128.143.231.202
02:36:50:WU00:FS00:Starting
02:36:50:WU01:FS00:Connecting to 128.143.231.202:8080
02:36:50:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 00 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
02:36:50:WU00:FS00:Started FahCore on PID 32464
02:36:50:WU00:FS00:Core PID:32468
02:36:50:WU00:FS00:FahCore 0xa3 started
[93m02:36:51:WARNING:WU00:FS00:FahCore returned: FILE_IO_ERROR (117 = 0x75)[0m
[93m02:36:51:WARNING:WU00:FS00:Fatal error, dumping[0m
02:36:51:WU00:FS00:Sending unit results: id:00 state:SEND error:DUMPED project:6096 run:1 clone:52 gen:114 core:0xa3 unit:0x000000bd0a3b1e594f1af827ce75a21b
02:36:51:WU00:FS00:Connecting to 128.143.231.202:8080
02:36:52:WU02:FS00:Connecting to 171.67.108.200:8080
02:36:52:WU00:FS00:Server responded WORK_ACK (400)
02:36:52:WU00:FS00:Cleaning up
02:36:53:WU02:FS00:Assigned to work server 128.143.231.202
02:36:53:WU02:FS00:Requesting new work unit for slot 00: READY cpu:36 from 128.143.231.202
02:36:53:WU02:FS00:Connecting to 128.143.231.202:8080
02:36:55:WU02:FS00:Downloading 512B
02:36:55:WU02:FS00:Download complete
02:36:55:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:6096 run:3 clone:51 gen:113 core:0xa3 unit:0x000000d70a3b1e594f1af8f50581727e
02:36:55:WU02:FS00:Starting
02:36:55:WU02:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 02 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
02:36:55:WU02:FS00:Started FahCore on PID 32469
02:36:55:WU02:FS00:Core PID:32473
02:36:55:WU02:FS00:FahCore 0xa3 started
[93m02:36:55:WARNING:WU02:FS00:FahCore returned: FILE_IO_ERROR (117 = 0x75)[0m
[93m02:36:55:WARNING:WU02:FS00:Fatal error, dumping[0m
02:36:55:WU02:FS00:Sending unit results: id:02 state:SEND error:DUMPED project:6096 run:3 clone:51 gen:113 core:0xa3 unit:0x000000d70a3b1e594f1af8f50581727e
02:36:55:WU02:FS00:Connecting to 128.143.231.202:8080
02:36:56:WU01:FS00:Upload 14.51%
02:36:56:WU00:FS00:Connecting to 171.67.108.200:8080
02:36:57:WU02:FS00:Server responded WORK_ACK (400)
02:36:57:WU02:FS00:Cleaning up
02:36:58:WU00:FS00:Assigned to work server 128.143.231.202
02:36:58:WU00:FS00:Requesting new work unit for slot 00: READY cpu:36 from 128.143.231.202
02:36:58:WU00:FS00:Connecting to 128.143.231.202:8080
02:37:01:WU00:FS00:Downloading 512B
02:37:01:WU00:FS00:Download complete
02:37:01:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:6095 run:11 clone:96 gen:56 core:0xa3 unit:0x0000004b0a3b1e594f25c63774afb259
02:37:01:WU00:FS00:Starting
02:37:01:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 00 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
02:37:01:WU00:FS00:Started FahCore on PID 32474
02:37:01:WU00:FS00:Core PID:32478
02:37:01:WU00:FS00:FahCore 0xa3 started
[93m02:37:02:WARNING:WU00:FS00:FahCore returned: FILE_IO_ERROR (117 = 0x75)[0m
[93m02:37:02:WARNING:WU00:FS00:Fatal error, dumping[0m
02:37:02:WU00:FS00:Sending unit results: id:00 state:SEND error:DUMPED project:6095 run:11 clone:96 gen:56 core:0xa3 unit:0x0000004b0a3b1e594f25c63774afb259
02:37:02:WU00:FS00:Connecting to 128.143.231.202:8080
02:37:02:WU01:FS00:Upload 31.92%
02:37:04:WU02:FS00:Connecting to 171.67.108.200:8080
02:37:05:WU00:FS00:Server responded WORK_ACK (400)
02:37:05:WU00:FS00:Cleaning up
02:37:07:WU02:FS00:Assigned to work server 128.143.231.202
02:37:07:WU02:FS00:Requesting new work unit for slot 00: READY cpu:36 from 128.143.231.202
02:37:07:WU02:FS00:Connecting to 128.143.231.202:8080
02:37:08:WU01:FS00:Upload 44.68%
02:37:10:WU02:FS00:Downloading 512B
02:37:10:WU02:FS00:Download complete
02:37:10:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:6095 run:24 clone:13 gen:55 core:0xa3 unit:0x000000520a3b1e594f25cc48203a3af9
02:37:10:WU02:FS00:Starting
02:37:10:WU02:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 02 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
02:37:10:WU02:FS00:Started FahCore on PID 32479
02:37:10:WU02:FS00:Core PID:32483
02:37:10:WU02:FS00:FahCore 0xa3 started
[93m02:37:10:WARNING:WU02:FS00:FahCore returned: FILE_IO_ERROR (117 = 0x75)[0m
[93m02:37:10:WARNING:WU02:FS00:Fatal error, dumping[0m
02:37:10:WU02:FS00:Sending unit results: id:02 state:SEND error:DUMPED project:6095 run:24 clone:13 gen:55 core:0xa3 unit:0x000000520a3b1e594f25cc48203a3af9
02:37:10:WU02:FS00:Connecting to 128.143.231.202:8080
02:37:12:WU00:FS00:Connecting to 171.67.108.200:8080
02:37:13:WU02:FS00:Server responded WORK_ACK (400)
02:37:13:WU02:FS00:Cleaning up
02:37:14:WU01:FS00:Upload 61.51%
02:37:14:WU00:FS00:Assigned to work server 128.143.231.202
02:37:14:WU00:FS00:Requesting new work unit for slot 00: READY cpu:36 from 128.143.231.202
02:37:14:WU00:FS00:Connecting to 128.143.231.202:8080
02:37:17:WU00:FS00:Downloading 512B
02:37:17:WU00:FS00:Download complete
02:37:17:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:6096 run:5 clone:81 gen:112 core:0xa3 unit:0x000000c60a3b1e594f1af9e04bbf032a
02:37:17:WU00:FS00:Starting
02:37:17:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 00 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
02:37:17:WU00:FS00:Started FahCore on PID 32484
02:37:17:WU00:FS00:Core PID:32488
02:37:17:WU00:FS00:FahCore 0xa3 started
[93m02:37:17:WARNING:WU00:FS00:FahCore returned: FILE_IO_ERROR (117 = 0x75)[0m
[93m02:37:17:WARNING:WU00:FS00:Fatal error, dumping[0m
02:37:17:WU00:FS00:Sending unit results: id:00 state:SEND error:DUMPED project:6096 run:5 clone:81 gen:112 core:0xa3 unit:0x000000c60a3b1e594f1af9e04bbf032a
02:37:17:WU00:FS00:Connecting to 128.143.231.202:8080
02:37:19:WU02:FS00:Connecting to 171.67.108.200:8080
02:37:20:WU01:FS00:Upload 76.02%
02:37:20:WU00:FS00:Server responded WORK_ACK (400)
02:37:20:WU00:FS00:Cleaning up
02:37:22:WU02:FS00:Assigned to work server 128.143.231.202
02:37:22:WU02:FS00:Requesting new work unit for slot 00: READY cpu:36 from 128.143.231.202
02:37:22:WU02:FS00:Connecting to 128.143.231.202:8080
02:37:25:WU02:FS00:Downloading 512B
02:37:25:WU02:FS00:Download complete
02:37:25:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:6095 run:5 clone:17 gen:55 core:0xa3 unit:0x000000560a3b1e594f1af588020ba946
02:37:25:WU02:FS00:Starting
02:37:25:WU02:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 02 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
02:37:25:WU02:FS00:Started FahCore on PID 32489
02:37:25:WU02:FS00:Core PID:32493
02:37:25:WU02:FS00:FahCore 0xa3 started
[93m02:37:26:WARNING:WU02:FS00:FahCore returned: FILE_IO_ERROR (117 = 0x75)[0m
[93m02:37:26:WARNING:WU02:FS00:Fatal error, dumping[0m
02:37:26:WU02:FS00:Sending unit results: id:02 state:SEND error:DUMPED project:6095 run:5 clone:17 gen:55 core:0xa3 unit:0x000000560a3b1e594f1af588020ba946
02:37:26:WU02:FS00:Connecting to 128.143.231.202:8080
02:37:27:WU01:FS00:Upload 91.11%
02:37:28:WU00:FS00:Connecting to 171.67.108.200:8080
02:37:29:WU02:FS00:Server responded WORK_ACK (400)
02:37:29:WU02:FS00:Cleaning up
02:37:31:WU00:FS00:Assigned to work server 128.143.231.202
02:37:31:WU00:FS00:Requesting new work unit for slot 00: READY cpu:36 from 128.143.231.202
02:37:31:WU00:FS00:Connecting to 128.143.231.202:8080
02:37:33:WU01:FS00:Upload complete
02:37:33:WU01:FS00:Server responded WORK_ACK (400)
02:37:33:WU01:FS00:Final credit estimate, 25827.00 points
02:37:33:WU01:FS00:Cleaning up
02:37:34:WU00:FS00:Downloading 3.64MiB
02:37:38:WU00:FS00:Download complete
02:37:39:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:6096 run:2 clone:45 gen:116 core:0xa3 unit:0x000000d00a3b1e594f1af888d8e06db2
02:37:39:WU00:FS00:Starting
02:37:39:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a3.fah/FahCore_a3 -dir 00 -suffix 01 -version 704 -lifeline 1780 -checkpoint 15 -np 36
02:37:39:WU00:FS00:Started FahCore on PID 32494
02:37:39:WU00:FS00:Core PID:32498
02:37:39:WU00:FS00:FahCore 0xa3 started
02:37:39:WU00:FS00:0xa3:
02:37:39:WU00:FS00:0xa3:*------------------------------*
02:37:39:WU00:FS00:0xa3:Folding@Home Gromacs SMP Core
02:37:39:WU00:FS00:0xa3:Version 2.27 (Dec. 15, 2010)
02:37:39:WU00:FS00:0xa3:
02:37:39:WU00:FS00:0xa3:Preparing to commence simulation
02:37:39:WU00:FS00:0xa3:- Looking at optimizations...
02:37:39:WU00:FS00:0xa3:- Created dyn
02:37:39:WU00:FS00:0xa3:- Files status OK
02:37:39:WU00:FS00:0xa3:- Expanded 3812224 -> 4169088 (decompressed 109.3 percent)
02:37:39:WU00:FS00:0xa3:Called DecompressByteArray: compressed_data_size=3812224 data_size=4169088, decompressed_data_size=4169088 diff=0
02:37:39:WU00:FS00:0xa3:- Digital signature verified
02:37:39:WU00:FS00:0xa3:
02:37:39:WU00:FS00:0xa3:Project: 6096 (Run 2, Clone 45, Gen 116)
02:37:39:WU00:FS00:0xa3:
02:37:39:WU00:FS00:0xa3:Assembly optimizations on if available.
02:37:39:WU00:FS00:0xa3:Entering M.D.
02:37:45:WU00:FS00:0xa3:Mapping NT from 36 to 32
02:37:46:WU00:FS00:0xa3:Completed 0 out of 500000 steps (0%)
02:40:09:WU00:FS00:0xa3:Completed 5000 out of 500000 steps (1%)
02:42:33:WU00:FS00:0xa3:Completed 10000 out of 500000 steps (2%)
02:44:56:WU00:FS00:0xa3:Completed 15000 out of 500000 steps (3%)
02:47:19:WU00:FS00:0xa3:Completed 20000 out of 500000 steps (4%)
02:49:43:WU00:FS00:0xa3:Completed 25000 out of 500000 steps (5%)