Project: 2665 (Run 3, Clone 611, Gen 51)

[ChrisDTC]

I stopped it because I have a pretty good idea what is going to happen again

Code: Select all

[07:42:09] + Processing work unit
[07:42:09] Work type a1 not eligible for variable processors
[07:42:09] Core required: FahCore_a1.exe
[07:42:09] Core found.
[07:42:09] Using generic mpiexec calls
[07:42:09] Working on queue slot 00 [October 5 07:42:09 UTC]
[07:42:09] + Working ...
[07:42:09] - Calling 'mpiexec -np 4 -channel auto -host 127.0.0.1 FahCore_a1.exe -dir work/ -suffix 00 -checkpoint 3 -service -verbose -lifeline 4396 -version 622'

[07:42:10] 
[07:42:10] *------------------------------*
[07:42:10] Folding@Home Gromacs SMP Core
[07:42:10] Version 1.74 (March 10, 2007)
[07:42:10] 
[07:42:10] Preparing to commence simulation
[07:42:10] - Looking at optimizations...
[07:42:10] Examination of work files indicates 8 consecutive improper terminations of core.
[07:42:16] - Expanded 4721790 -> 24426905 (decompressed 517.3 percent)
[07:42:16] - Data doesn't match checksum.
[07:42:16] - Starting from initial work packet
[07:42:16] 
[07:42:16] Project: 2665 (Run 3, Clone 611, Gen 51)
[07:42:16] 
[07:42:19] Assembly optimizations on if available.
[07:42:19] Entering M.D.
[07:42:41] ring M.D.
[07:42:48] Rejecting checkpoint
[07:42:50] Protein: HGG in water
[07:42:50] Writing local files
[07:42:59] Extra SSE boost OK.
[07:43:00] Writing local files
[07:43:00] Completed 0 out of 250000 steps  (0 percent)
[07:46:01] Timered checkpoint triggered.
[07:49:00] Timered checkpoint triggered.
[07:52:02] Timered checkpoint triggered.
[07:55:02] Timered checkpoint triggered.
[07:58:02] Timered checkpoint triggered.
[08:01:02] Timered checkpoint triggered.
[08:04:02] Timered checkpoint triggered.
[08:06:46] Writing local files
[08:06:46] Completed 2500 out of 250000 steps  (1 percent)
[08:09:46] Timered checkpoint triggered.
[08:12:46] Timered checkpoint triggered.
[08:15:47] Timered checkpoint triggered.
[08:18:46] Timered checkpoint triggered.
[08:21:47] Timered checkpoint triggered.
[08:24:48] Timered checkpoint triggered.
[08:27:50] Timered checkpoint triggered.
[08:30:29] Writing local files
[08:30:29] Completed 5000 out of 250000 steps  (2 percent)
[08:33:30] Timered checkpoint triggered.
[08:36:29] Timered checkpoint triggered.
[08:39:30] Timered checkpoint triggered.
[08:42:30] Timered checkpoint triggered.
[08:45:30] Timered checkpoint triggered.
[08:48:31] Timered checkpoint triggered.
[08:51:31] Timered checkpoint triggered.
[08:54:16] Writing local files
[08:54:17] Completed 7500 out of 250000 steps  (3 percent)
[08:57:17] Timered checkpoint triggered.
[09:00:18] Timered checkpoint triggered.
[09:03:18] Timered checkpoint triggered.
[09:06:20] Timered checkpoint triggered.
[09:09:21] Timered checkpoint triggered.
[09:12:22] Timered checkpoint triggered.
[09:15:23] Timered checkpoint triggered.
[09:18:00] Writing local files
[09:18:01] Completed 10000 out of 250000 steps  (4 percent)
[09:21:02] Timered checkpoint triggered.
[09:24:03] Timered checkpoint triggered.
[09:27:04] Timered checkpoint triggered.
[09:30:06] Timered checkpoint triggered.
[09:33:06] Timered checkpoint triggered.
[09:36:07] Timered checkpoint triggered.
[09:39:08] Timered checkpoint triggered.
[09:41:46] Writing local files
[09:41:46] Completed 12500 out of 250000 steps  (5 percent)
[09:44:47] Timered checkpoint triggered.
[09:47:48] Timered checkpoint triggered.
[09:50:49] Timered checkpoint triggered.
[09:53:49] Timered checkpoint triggered.
[09:56:49] Timered checkpoint triggered.
[09:59:49] Timered checkpoint triggered.
[10:02:49] Timered checkpoint triggered.
[10:05:32] Writing local files
[10:05:32] Completed 15000 out of 250000 steps  (6 percent)
[10:08:33] Timered checkpoint triggered.
[10:11:32] Timered checkpoint triggered.
[10:14:32] Timered checkpoint triggered.
[10:17:32] Timered checkpoint triggered.
[10:20:32] Timered checkpoint triggered.
[10:23:33] Timered checkpoint triggered.
[10:26:32] Timered checkpoint triggered.
[10:29:17] Writing local files
[10:29:18] Completed 17500 out of 250000 steps  (7 percent)
[10:32:19] Timered checkpoint triggered.
[10:35:18] Timered checkpoint triggered.
[10:38:18] Timered checkpoint triggered.
[10:41:19] Timered checkpoint triggered.
[10:44:18] Timered checkpoint triggered.
[10:45:57] - Autosending finished units... [October 5 10:45:57 UTC]
[10:45:57] Trying to send all finished work units
[10:45:57] + No unsent completed units remaining.
[10:45:57] - Autosend completed
[10:47:19] Timered checkpoint triggered.
[10:50:19] Timered checkpoint triggered.
[10:53:06] Writing local files
[10:53:07] Completed 20000 out of 250000 steps  (8 percent)
[10:56:07] Timered checkpoint triggered.
[10:59:07] Timered checkpoint triggered.
[11:02:07] Timered checkpoint triggered.
[11:05:07] Timered checkpoint triggered.
[11:08:07] Timered checkpoint triggered.
[11:11:08] Timered checkpoint triggered.
[11:14:07] Timered checkpoint triggered.
[11:16:51] Writing local files
[11:16:51] Completed 22500 out of 250000 steps  (9 percent)
[11:19:52] Timered checkpoint triggered.
[11:22:51] Timered checkpoint triggered.
[11:25:52] Timered checkpoint triggered.
[11:28:51] Timered checkpoint triggered.
[11:31:51] Timered checkpoint triggered.
[11:34:52] Timered checkpoint triggered.
[11:37:52] Timered checkpoint triggered.
[11:40:39] Writing local files
[11:40:39] Completed 25000 out of 250000 steps  (10 percent)
[11:43:40] Timered checkpoint triggered.
[11:46:41] Timered checkpoint triggered.
[11:49:42] Timered checkpoint triggered.
[11:52:43] Timered checkpoint triggered.
[11:55:44] Timered checkpoint triggered.
[11:58:45] Timered checkpoint triggered.
[12:01:45] Timered checkpoint triggered.
[12:04:27] Writing local files
[12:04:27] Completed 27500 out of 250000 steps  (11 percent)
[12:07:28] Timered checkpoint triggered.
[12:10:29] Timered checkpoint triggered.
[12:13:30] Timered checkpoint triggered.
[12:16:31] Timered checkpoint triggered.
[12:19:31] Timered checkpoint triggered.
[12:22:32] Timered checkpoint triggered.
[12:25:33] Timered checkpoint triggered.
[12:28:14] Writing local files
[12:28:15] Completed 30000 out of 250000 steps  (12 percent)
[12:31:15] Timered checkpoint triggered.
[12:34:16] Timered checkpoint triggered.
[12:37:17] Timered checkpoint triggered.
[12:40:18] Timered checkpoint triggered.
[12:43:19] Timered checkpoint triggered.
[12:46:21] Timered checkpoint triggered.
[12:49:22] Timered checkpoint triggered.
[12:52:05] Writing local files
[12:52:05] Completed 32500 out of 250000 steps  (13 percent)
[12:55:06] Timered checkpoint triggered.
[12:58:07] Timered checkpoint triggered.
[13:01:08] Timered checkpoint triggered.
[13:04:09] Timered checkpoint triggered.
[13:07:11] Timered checkpoint triggered.
[13:10:12] Timered checkpoint triggered.
[13:13:13] Timered checkpoint triggered.
[13:15:50] Writing local files
[13:15:50] Completed 35000 out of 250000 steps  (14 percent)
[13:18:51] Timered checkpoint triggered.
[13:21:51] Timered checkpoint triggered.
[13:24:52] Timered checkpoint triggered.
[13:27:52] Timered checkpoint triggered.
[13:30:52] Timered checkpoint triggered.
[13:33:52] Timered checkpoint triggered.
[13:36:53] Timered checkpoint triggered.
[13:39:36] Writing local files
[13:39:36] Completed 37500 out of 250000 steps  (15 percent)
[13:42:36] Timered checkpoint triggered.
[13:45:37] Timered checkpoint triggered.
[13:48:36] Timered checkpoint triggered.
[13:51:37] Timered checkpoint triggered.
[13:54:36] Timered checkpoint triggered.
[13:57:36] Timered checkpoint triggered.
[14:00:37] Timered checkpoint triggered.
[14:03:23] Writing local files
[14:03:24] Completed 40000 out of 250000 steps  (16 percent)
[14:06:24] Timered checkpoint triggered.
[14:09:23] Timered checkpoint triggered.
[14:12:23] Timered checkpoint triggered.
[14:15:24] Timered checkpoint triggered.
[14:18:24] Timered checkpoint triggered.
[14:21:23] Timered checkpoint triggered.
[14:24:23] Timered checkpoint triggered.
[14:27:11] Writing local files
[14:27:11] Completed 42500 out of 250000 steps  (17 percent)
[14:30:12] Timered checkpoint triggered.
[14:33:12] Timered checkpoint triggered.
[14:36:13] Timered checkpoint triggered.
[14:39:13] Timered checkpoint triggered.
[14:42:13] Timered checkpoint triggered.
[14:45:13] Timered checkpoint triggered.
[14:48:14] Timered checkpoint triggered.
[14:51:02] Writing local files
[14:51:02] Completed 45000 out of 250000 steps  (18 percent)
[14:54:02] Timered checkpoint triggered.
[14:57:02] Timered checkpoint triggered.
[15:00:03] Timered checkpoint triggered.
[15:03:02] Timered checkpoint triggered.
[15:06:02] Timered checkpoint triggered.
[15:09:03] Timered checkpoint triggered.
[15:12:03] Timered checkpoint triggered.
[15:14:50] Writing local files
[15:14:50] Completed 47500 out of 250000 steps  (19 percent)
[15:17:51] Timered checkpoint triggered.
[15:20:50] Timered checkpoint triggered.
[15:23:51] Timered checkpoint triggered.
[15:26:52] Timered checkpoint triggered.
[15:29:53] Timered checkpoint triggered.
[15:32:54] Timered checkpoint triggered.
[15:35:53] Timered checkpoint triggered.
[15:38:38] Writing local files
[15:38:38] Completed 50000 out of 250000 steps  (20 percent)
[15:41:39] Timered checkpoint triggered.
[15:44:40] Timered checkpoint triggered.
[15:47:41] Timered checkpoint triggered.
[15:50:42] Timered checkpoint triggered.
[15:53:42] Timered checkpoint triggered.
[15:56:43] Timered checkpoint triggered.
[15:59:44] Timered checkpoint triggered.
[16:02:26] Writing local files
[16:02:26] Completed 52500 out of 250000 steps  (21 percent)
[16:05:26] Timered checkpoint triggered.
[16:08:27] Timered checkpoint triggered.
[16:11:28] Timered checkpoint triggered.
[16:14:29] Timered checkpoint triggered.
[16:17:30] Timered checkpoint triggered.
[16:20:31] Timered checkpoint triggered.
[16:23:33] Timered checkpoint triggered.
[16:26:13] Writing local files
[16:26:13] Completed 55000 out of 250000 steps  (22 percent)
[16:29:14] Timered checkpoint triggered.
[16:32:15] Timered checkpoint triggered.
[16:35:16] Timered checkpoint triggered.
[16:38:17] Timered checkpoint triggered.
[16:41:18] Timered checkpoint triggered.
[16:44:18] Timered checkpoint triggered.
[16:45:56] - Autosending finished units... [October 5 16:45:56 UTC]
[16:45:56] Trying to send all finished work units
[16:45:56] + No unsent completed units remaining.
[16:45:56] - Autosend completed
[16:47:19] Timered checkpoint triggered.
[16:50:01] Writing local files
[16:50:02] Completed 57500 out of 250000 steps  (23 percent)
[16:53:01] Timered checkpoint triggered.
[16:56:02] Timered checkpoint triggered.
[16:59:01] Timered checkpoint triggered.
[17:02:01] Timered checkpoint triggered.
[17:05:01] Timered checkpoint triggered.
[17:08:01] Timered checkpoint triggered.
[17:11:02] Timered checkpoint triggered.
[17:13:49] Writing local files
[17:13:50] Completed 60000 out of 250000 steps  (24 percent)
[17:16:50] Timered checkpoint triggered.
[17:19:50] Timered checkpoint triggered.
[17:22:49] Timered checkpoint triggered.
[17:25:50] Timered checkpoint triggered.
[17:28:50] Timered checkpoint triggered.
[17:31:49] Timered checkpoint triggered.
[17:34:50] Timered checkpoint triggered.
[17:37:37] Writing local files
[17:37:38] Completed 62500 out of 250000 steps  (25 percent)
[17:40:38] Timered checkpoint triggered.
[17:43:37] Timered checkpoint triggered.
[17:46:37] Timered checkpoint triggered.
[17:49:38] Timered checkpoint triggered.
[17:52:38] Timered checkpoint triggered.
[17:55:38] Timered checkpoint triggered.
[17:58:39] Timered checkpoint triggered.
[18:01:26] Writing local files
[18:01:26] Completed 65000 out of 250000 steps  (26 percent)
[18:04:27] Timered checkpoint triggered.
[18:07:27] Timered checkpoint triggered.
[18:10:26] Timered checkpoint triggered.
[18:13:27] Timered checkpoint triggered.
[18:16:28] Timered checkpoint triggered.
[18:19:29] Timered checkpoint triggered.
[18:22:29] Timered checkpoint triggered.
[18:25:14] Writing local files
[18:25:14] Completed 67500 out of 250000 steps  (27 percent)
[18:28:15] Timered checkpoint triggered.
[18:31:15] Timered checkpoint triggered.
[18:34:16] Timered checkpoint triggered.
[18:37:16] Timered checkpoint triggered.
[18:40:18] Timered checkpoint triggered.
[18:43:18] Timered checkpoint triggered.
[18:46:18] Timered checkpoint triggered.
[18:49:05] Writing local files
[18:49:05] Completed 70000 out of 250000 steps  (28 percent)
[18:52:05] Timered checkpoint triggered.
[18:55:06] Timered checkpoint triggered.
[18:58:08] Timered checkpoint triggered.
[19:01:08] Timered checkpoint triggered.
[19:04:09] Timered checkpoint triggered.
[19:07:09] Timered checkpoint triggered.
[19:10:10] Timered checkpoint triggered.
[19:12:52] Writing local files
[19:12:52] Completed 72500 out of 250000 steps  (29 percent)
[19:15:53] Timered checkpoint triggered.
[19:18:54] Timered checkpoint triggered.
[19:21:55] Timered checkpoint triggered.
[19:24:56] Timered checkpoint triggered.
[19:27:57] Timered checkpoint triggered.
[19:30:59] Timered checkpoint triggered.
[19:34:00] Timered checkpoint triggered.
[19:36:41] Writing local files
[19:36:41] Completed 75000 out of 250000 steps  (30 percent)
[19:39:42] Timered checkpoint triggered.
[19:42:43] Timered checkpoint triggered.
[19:45:45] Timered checkpoint triggered.
[19:48:45] Timered checkpoint triggered.
[19:51:46] Timered checkpoint triggered.
[19:54:47] Timered checkpoint triggered.
[19:57:48] Timered checkpoint triggered.
[20:00:40] Writing local files
[20:00:40] Completed 77500 out of 250000 steps  (31 percent)
[20:03:41] Timered checkpoint triggered.
[20:06:42] Timered checkpoint triggered.
[20:09:41] Timered checkpoint triggered.
[20:12:41] Timered checkpoint triggered.
[20:15:41] Timered checkpoint triggered.
[20:18:41] Timered checkpoint triggered.
[20:21:42] Timered checkpoint triggered.
[20:24:33] Writing local files
[20:24:33] Completed 80000 out of 250000 steps  (32 percent)
[20:27:33] Timered checkpoint triggered.
[20:30:33] Timered checkpoint triggered.
[20:33:33] Timered checkpoint triggered.
[20:36:34] Timered checkpoint triggered.
[20:39:34] Timered checkpoint triggered.
[20:42:34] Timered checkpoint triggered.
[20:45:34] Timered checkpoint triggered.
[20:48:22] Writing local files
[20:48:23] Completed 82500 out of 250000 steps  (33 percent)
[20:51:23] Timered checkpoint triggered.
[20:54:23] Timered checkpoint triggered.
[20:57:23] Timered checkpoint triggered.
[21:00:22] Timered checkpoint triggered.
[21:03:24] Timered checkpoint triggered.
[21:06:24] Timered checkpoint triggered.
[21:09:23] Timered checkpoint triggered.
[21:12:08] Writing local files
[21:12:08] Completed 85000 out of 250000 steps  (34 percent)
[21:15:09] Timered checkpoint triggered.
[21:18:08] Timered checkpoint triggered.
[21:21:09] Timered checkpoint triggered.
[21:24:10] Timered checkpoint triggered.
[21:27:11] Timered checkpoint triggered.
[21:30:11] Timered checkpoint triggered.
[21:33:10] Timered checkpoint triggered.
[21:35:56] Writing local files
[21:35:56] Completed 87500 out of 250000 steps  (35 percent)
[21:38:57] Timered checkpoint triggered.
[21:41:57] Timered checkpoint triggered.
[21:44:58] Timered checkpoint triggered.
[21:47:59] Timered checkpoint triggered.
[21:51:00] Timered checkpoint triggered.
[21:54:00] Timered checkpoint triggered.
[21:57:01] Timered checkpoint triggered.
[21:59:44] Writing local files
[21:59:44] Completed 90000 out of 250000 steps  (36 percent)
[22:02:45] Timered checkpoint triggered.
[22:05:46] Timered checkpoint triggered.
[22:08:47] Timered checkpoint triggered.
[22:11:49] Timered checkpoint triggered.
[22:14:49] Timered checkpoint triggered.
[22:17:49] Timered checkpoint triggered.
[22:20:51] Timered checkpoint triggered.
[22:23:30] Writing local files
[22:23:30] Completed 92500 out of 250000 steps  (37 percent)
[22:26:31] Timered checkpoint triggered.
[22:29:32] Timered checkpoint triggered.
[22:32:33] Timered checkpoint triggered.
[22:35:34] Timered checkpoint triggered.
[22:38:35] Timered checkpoint triggered.
[22:41:36] Timered checkpoint triggered.
[22:44:37] Timered checkpoint triggered.
[22:45:56] - Autosending finished units... [October 5 22:45:56 UTC]
[22:45:56] Trying to send all finished work units
[22:45:56] + No unsent completed units remaining.
[22:45:56] - Autosend completed
[22:47:20] Writing local files
[22:47:20] Completed 95000 out of 250000 steps  (38 percent)
[22:50:21] Timered checkpoint triggered.
[22:53:22] Timered checkpoint triggered.
[22:56:23] Timered checkpoint triggered.
[22:59:24] Timered checkpoint triggered.
[23:02:25] Timered checkpoint triggered.
[23:05:27] Timered checkpoint triggered.
[23:08:28] Timered checkpoint triggered.
[23:11:10] Writing local files
[23:11:10] Completed 97500 out of 250000 steps  (39 percent)
[23:14:10] Timered checkpoint triggered.
[23:17:10] Timered checkpoint triggered.
[23:19:19] Warning:  long 1-4 interactions
[23:19:21] Gromacs cannot continue further.
[23:19:21] Going to send back what have done.
[23:19:21] logfile size: 75706
[23:19:21] - Writing 76242 bytes of core data to disk...
[23:19:21]   ... Done.
[23:21:21] 
[23:21:21] Folding@home Core Shutdown: EARLY_UNIT_END
[23:21:21] 
[23:21:21] Folding@home Core Shutdown: EARLY_UNIT_END
[23:21:25] CoreStatus = 7B (123)
[23:21:25] Client-core communications error: ERROR 0x7b
[23:21:25] Deleting current work unit & continuing...
[23:21:25] Using generic mpiexec calls
[23:23:28] - Warning: Could not delete all work unit files (0): Core returned invalid code
[23:23:28] Trying to send all finished work units
[23:23:28] + No unsent completed units remaining.
[23:23:28] - Preparing to get new work unit...
[23:23:28] + Attempting to get work packet
[23:23:28] - Will indicate memory of 3068 MB
[23:23:28] - Connecting to assignment server
[23:23:28] Connecting to http://assign.stanford.edu:8080/
[23:23:28] Posted data.
[23:23:28] Initial: 40AB; - Successful: assigned to (171.64.65.64).
[23:23:28] + News From Folding@Home: Welcome to Folding@Home
[23:23:29] Loaded queue successfully.
[23:23:29] Connecting to http://171.64.65.64:8080/
[23:23:34] Posted data.
[23:23:34] Initial: 0000; - Receiving payload (expected size: 4722302)
[23:23:41] - Downloaded at ~658 kB/s
[23:23:41] - Averaged speed for that direction ~693 kB/s
[23:23:41] + Received work.
[23:23:41] + Closed connections
[23:23:46] 
[23:23:46] + Processing work unit
[23:23:46] Work type a1 not eligible for variable processors
[23:23:46] Core required: FahCore_a1.exe
[23:23:46] Core found.
[23:23:46] Using generic mpiexec calls
[23:23:46] Working on queue slot 01 [October 5 23:23:46 UTC]
[23:23:46] + Working ...
[23:23:46] - Calling 'mpiexec -np 4 -channel auto -host 127.0.0.1 FahCore_a1.exe -dir work/ -suffix 01 -checkpoint 3 -service -verbose -lifeline 4396 -version 622'

[23:23:46] 
[23:23:46] *------------------------------*
[23:23:46] Folding@Home Gromacs SMP Core
[23:23:46] Version 1.74 (March 10, 2007)
[23:23:46] 
[23:23:46] Preparing to commence simulation
[23:23:46] - Ensuring status. Please wait.
[23:23:52] - Data doesn't match checksum.
[23:23:52] - Starting from initial work packet
[23:23:52] 
[23:23:52] Project: 2665 (Run 3, Clone 611, Gen 51)
[23:23:52] 
[23:23:53] Entering M.D.
[23:24:13] t: 2665 (Run 3, Clone 611, Gen 51)
[23:24:13] 
[23:24:17] Entering M.D.
[23:24:25] Rejecting checkpoint
[23:24:27] Protein: HGG in water
[23:24:27] Writing local files
[23:24:36] Extra SSE boost OK.
[23:24:36] Writing local files
[23:24:36] Completed 0 out of 250000 steps  (0 percent)
[23:27:37] Timered checkpoint triggered.
[23:30:37] Timered checkpoint triggered.
[23:33:38] Timered checkpoint triggered.
[23:36:39] Timered checkpoint triggered.
[23:39:39] Timered checkpoint triggered.
[23:42:40] Timered checkpoint triggered.
[23:45:41] Timered checkpoint triggered.
[23:48:24] Writing local files
[23:48:24] Completed 2500 out of 250000 steps  (1 percent)
[23:51:25] Timered checkpoint triggered.
[23:54:26] Timered checkpoint triggered.
[23:57:27] Timered checkpoint triggered.
[00:00:28] Timered checkpoint triggered.
[00:03:29] Timered checkpoint triggered.
[00:06:29] Timered checkpoint triggered.
[00:09:30] Timered checkpoint triggered.
[00:12:09] Writing local files
[00:12:09] Completed 5000 out of 250000 steps  (2 percent)
[00:15:10] Timered checkpoint triggered.
[00:18:11] Timered checkpoint triggered.
[00:21:12] Timered checkpoint triggered.
[00:24:13] Timered checkpoint triggered.
[00:27:14] Timered checkpoint triggered.
[00:30:15] Timered checkpoint triggered.
[00:33:16] Timered checkpoint triggered.
[00:35:55] Writing local files
[00:35:55] Completed 7500 out of 250000 steps  (3 percent)
[00:38:56] Timered checkpoint triggered.
[00:41:57] Timered checkpoint triggered.
[00:44:58] Timered checkpoint triggered.
[00:48:00] Timered checkpoint triggered.
[00:51:02] Timered checkpoint triggered.
[00:54:02] Timered checkpoint triggered.
[00:57:03] Timered checkpoint triggered.
[00:59:43] Writing local files
[00:59:43] Completed 10000 out of 250000 steps  (4 percent)
[01:02:43] Timered checkpoint triggered.
[01:05:43] Timered checkpoint triggered.
[01:08:44] Timered checkpoint triggered.
[01:11:44] Timered checkpoint triggered.
[01:14:44] Timered checkpoint triggered.
[01:17:45] Timered checkpoint triggered.
[01:20:44] Timered checkpoint triggered.
[01:23:27] Writing local files
[01:23:28] Completed 12500 out of 250000 steps  (5 percent)
[01:26:28] Timered checkpoint triggered.
[01:29:28] Timered checkpoint triggered.
[01:32:30] Timered checkpoint triggered.
[01:35:30] Timered checkpoint triggered.
[01:38:29] Timered checkpoint triggered.
[01:41:30] Timered checkpoint triggered.
[01:44:29] Timered checkpoint triggered.
[01:47:14] Writing local files
[01:47:14] Completed 15000 out of 250000 steps  (6 percent)
[01:50:15] Timered checkpoint triggered.
[01:53:14] Timered checkpoint triggered.
[01:56:14] Timered checkpoint triggered.
[01:59:14] Timered checkpoint triggered.
[02:02:14] Timered checkpoint triggered.
[02:05:15] Timered checkpoint triggered.
[02:08:15] Timered checkpoint triggered.
[02:11:00] Writing local files
[02:11:00] Completed 17500 out of 250000 steps  (7 percent)
[02:14:01] Timered checkpoint triggered.
[02:17:00] Timered checkpoint triggered.
[02:20:01] Timered checkpoint triggered.
[02:23:02] Timered checkpoint triggered.
[02:26:03] Timered checkpoint triggered.
[02:29:03] Timered checkpoint triggered.
[02:32:04] Timered checkpoint triggered.
[02:34:50] Writing local files
[02:34:50] Completed 20000 out of 250000 steps  (8 percent)
[02:37:50] Timered checkpoint triggered.
[02:40:50] Timered checkpoint triggered.
[02:43:51] Timered checkpoint triggered.
[02:46:51] Timered checkpoint triggered.
[02:49:52] Timered checkpoint triggered.
[02:52:53] Timered checkpoint triggered.
[02:55:53] Timered checkpoint triggered.
[02:58:36] Writing local files
[02:58:36] Completed 22500 out of 250000 steps  (9 percent)
[03:01:37] Timered checkpoint triggered.
[03:04:37] Timered checkpoint triggered.
[03:07:38] Timered checkpoint triggered.
[03:10:39] Timered checkpoint triggered.
[03:13:40] Timered checkpoint triggered.
[03:16:40] Timered checkpoint triggered.
[03:19:41] Timered checkpoint triggered.
[03:22:24] Writing local files
[03:22:24] Completed 25000 out of 250000 steps  (10 percent)
[03:25:25] Timered checkpoint triggered.
[03:28:26] Timered checkpoint triggered.
[03:31:27] Timered checkpoint triggered.
[03:34:28] Timered checkpoint triggered.
[03:37:29] Timered checkpoint triggered.
[03:40:30] Timered checkpoint triggered.
[03:43:31] Timered checkpoint triggered.
[03:46:12] Writing local files
[03:46:12] Completed 27500 out of 250000 steps  (11 percent)
[03:49:13] Timered checkpoint triggered.
[03:52:14] Timered checkpoint triggered.
[03:55:15] Timered checkpoint triggered.
[03:58:16] Timered checkpoint triggered.
[04:01:17] Timered checkpoint triggered.
[04:04:18] Timered checkpoint triggered.
[04:07:20] Timered checkpoint triggered.
[04:10:00] Writing local files
[04:10:00] Completed 30000 out of 250000 steps  (12 percent)
[04:13:01] Timered checkpoint triggered.
[04:16:02] Timered checkpoint triggered.
[04:19:01] Timered checkpoint triggered.
[04:22:01] Timered checkpoint triggered.
[04:25:02] Timered checkpoint triggered.
[04:28:01] Timered checkpoint triggered.
[04:31:01] Timered checkpoint triggered.
[04:33:50] Writing local files
[04:33:51] Completed 32500 out of 250000 steps  (13 percent)
[04:36:51] Timered checkpoint triggered.
[04:39:51] Timered checkpoint triggered.
[04:42:51] Timered checkpoint triggered.
[04:45:52] Timered checkpoint triggered.
[04:45:56] - Autosending finished units... [October 6 04:45:56 UTC]
[04:45:56] Trying to send all finished work units
[04:45:56] + No unsent completed units remaining.
[04:45:56] - Autosend completed
[04:48:52] Timered checkpoint triggered.
[04:51:53] Timered checkpoint triggered.
[04:54:53] Timered checkpoint triggered.
[04:57:36] Writing local files
[04:57:37] Completed 35000 out of 250000 steps  (14 percent)
[05:00:37] Timered checkpoint triggered.
[05:03:37] Timered checkpoint triggered.
[05:06:37] Timered checkpoint triggered.
[05:09:38] Timered checkpoint triggered.
[05:12:37] Timered checkpoint triggered.
[05:15:37] Timered checkpoint triggered.
[05:18:37] Timered checkpoint triggered.
[05:21:25] Writing local files
[05:21:25] Completed 37500 out of 250000 steps  (15 percent)
[05:24:25] Timered checkpoint triggered.
[05:27:25] Timered checkpoint triggered.
[05:30:25] Timered checkpoint triggered.
[05:33:25] Timered checkpoint triggered.
[05:36:26] Timered checkpoint triggered.
[05:39:25] Timered checkpoint triggered.
[05:42:26] Timered checkpoint triggered.
[05:45:12] Writing local files
[05:45:12] Completed 40000 out of 250000 steps  (16 percent)
[05:48:12] Timered checkpoint triggered.
[05:51:13] Timered checkpoint triggered.
[05:54:13] Timered checkpoint triggered.
[05:57:12] Timered checkpoint triggered.
[06:00:13] Timered checkpoint triggered.
[06:03:13] Timered checkpoint triggered.
[06:06:13] Timered checkpoint triggered.
[06:09:01] Writing local files
[06:09:01] Completed 42500 out of 250000 steps  (17 percent)
[06:12:02] Timered checkpoint triggered.
[06:15:02] Timered checkpoint triggered.
[06:18:02] Timered checkpoint triggered.
[06:21:02] Timered checkpoint triggered.
[06:24:04] Timered checkpoint triggered.
[06:27:03] Timered checkpoint triggered.
[06:30:04] Timered checkpoint triggered.
[06:33:06] Timered checkpoint triggered.
[06:36:02] Writing local files
[06:36:02] Completed 45000 out of 250000 steps  (18 percent)
[06:39:03] Timered checkpoint triggered.
[06:42:04] Timered checkpoint triggered.
[06:45:05] Timered checkpoint triggered.
[06:48:06] Timered checkpoint triggered.
[06:51:06] Timered checkpoint triggered.
[06:54:07] Timered checkpoint triggered.
[06:57:08] Timered checkpoint triggered.
[06:59:53] Writing local files
[06:59:53] Completed 47500 out of 250000 steps  (19 percent)
[07:02:54] Timered checkpoint triggered.
[07:05:55] Timered checkpoint triggered.
[07:08:56] Timered checkpoint triggered.
[07:11:57] Timered checkpoint triggered.
[07:14:58] Timered checkpoint triggered.
[07:17:59] Timered checkpoint triggered.
[07:21:00] Timered checkpoint triggered.
[07:23:41] Writing local files
[07:23:41] Completed 50000 out of 250000 steps  (20 percent)
[07:26:42] Timered checkpoint triggered.
[07:29:43] Timered checkpoint triggered.
[07:32:44] Timered checkpoint triggered.
[07:35:46] Timered checkpoint triggered.
[07:38:46] Timered checkpoint triggered.
[07:41:46] Timered checkpoint triggered.
[07:44:46] Timered checkpoint triggered.
[07:47:42] Writing local files
[07:47:43] Completed 52500 out of 250000 steps  (21 percent)
[07:50:42] Timered checkpoint triggered.
[07:53:43] Timered checkpoint triggered.
[07:56:42] Timered checkpoint triggered.
[07:59:42] Timered checkpoint triggered.
[08:02:43] Timered checkpoint triggered.
[08:05:42] Timered checkpoint triggered.
[08:08:42] Timered checkpoint triggered.
[08:11:29] Writing local files
[08:11:30] Completed 55000 out of 250000 steps  (22 percent)
[08:14:29] Timered checkpoint triggered.
[08:17:29] Timered checkpoint triggered.
[08:20:29] Timered checkpoint triggered.
[08:23:29] Timered checkpoint triggered.
[08:26:30] Timered checkpoint triggered.
[08:29:29] Timered checkpoint triggered.
[08:32:29] Timered checkpoint triggered.
[08:35:17] Writing local files
[08:35:18] Completed 57500 out of 250000 steps  (23 percent)
[08:38:17] Timered checkpoint triggered.
[08:41:18] Timered checkpoint triggered.
[08:44:18] Timered checkpoint triggered.
[08:47:17] Timered checkpoint triggered.
[08:50:18] Timered checkpoint triggered.
[08:53:17] Timered checkpoint triggered.
[08:56:18] Timered checkpoint triggered.
[08:59:04] Writing local files
[08:59:04] Completed 60000 out of 250000 steps  (24 percent)
[09:02:04] Timered checkpoint triggered.
[09:05:05] Timered checkpoint triggered.
[09:08:05] Timered checkpoint triggered.
[09:11:05] Timered checkpoint triggered.
[09:14:05] Timered checkpoint triggered.
[09:17:05] Timered checkpoint triggered.
[09:20:06] Timered checkpoint triggered.
[09:22:51] Writing local files
[09:22:51] Completed 62500 out of 250000 steps  (25 percent)
[09:25:52] Timered checkpoint triggered.
[09:28:52] Timered checkpoint triggered.
[09:31:52] Timered checkpoint triggered.
[09:34:52] Timered checkpoint triggered.
[09:37:52] Timered checkpoint triggered.
[09:40:51] Timered checkpoint triggered.
[09:43:52] Timered checkpoint triggered.
[09:46:39] Writing local files
[09:46:39] Completed 65000 out of 250000 steps  (26 percent)
[09:49:40] Timered checkpoint triggered.
[09:52:41] Timered checkpoint triggered.
[09:55:42] Timered checkpoint triggered.
[09:58:43] Timered checkpoint triggered.
[10:01:43] Timered checkpoint triggered.
[10:04:43] Timered checkpoint triggered.
[10:07:43] Timered checkpoint triggered.
[10:10:28] Writing local files
[10:10:28] Completed 67500 out of 250000 steps  (27 percent)
[10:13:29] Timered checkpoint triggered.
[10:16:29] Timered checkpoint triggered.
[10:19:31] Timered checkpoint triggered.
[10:22:31] Timered checkpoint triggered.
[10:25:32] Timered checkpoint triggered.
[10:28:32] Timered checkpoint triggered.
[10:31:33] Timered checkpoint triggered.
[10:34:17] Writing local files
[10:34:17] Completed 70000 out of 250000 steps  (28 percent)
[10:37:18] Timered checkpoint triggered.
[10:40:19] Timered checkpoint triggered.
[10:43:20] Timered checkpoint triggered.
[10:45:56] - Autosending finished units... [October 6 10:45:56 UTC]
[10:45:56] Trying to send all finished work units
[10:45:56] + No unsent completed units remaining.
[10:45:56] - Autosend completed
[10:46:21] Timered checkpoint triggered.
[10:49:22] Timered checkpoint triggered.
[10:52:23] Timered checkpoint triggered.
[10:55:24] Timered checkpoint triggered.
[10:58:05] Writing local files
[10:58:05] Completed 72500 out of 250000 steps  (29 percent)
[11:01:06] Timered checkpoint triggered.
[11:04:07] Timered checkpoint triggered.
[11:07:08] Timered checkpoint triggered.
[11:10:10] Timered checkpoint triggered.
[11:13:10] Timered checkpoint triggered.
[11:16:11] Timered checkpoint triggered.
[11:19:13] Timered checkpoint triggered.
[11:21:58] Writing local files
[11:21:58] Completed 75000 out of 250000 steps  (30 percent)
[11:24:58] Timered checkpoint triggered.
[11:27:59] Timered checkpoint triggered.
[11:30:58] Timered checkpoint triggered.
[11:33:59] Timered checkpoint triggered.
[11:36:58] Timered checkpoint triggered.
[11:39:58] Timered checkpoint triggered.
[11:42:59] Timered checkpoint triggered.
[11:45:46] Writing local files
[11:45:46] Completed 77500 out of 250000 steps  (31 percent)
[11:48:46] Timered checkpoint triggered.
[11:51:47] Timered checkpoint triggered.
[11:54:47] Timered checkpoint triggered.
[11:57:47] Timered checkpoint triggered.
[12:00:47] Timered checkpoint triggered.
[12:03:47] Timered checkpoint triggered.
[12:06:48] Timered checkpoint triggered.
[12:09:34] Writing local files
[12:09:34] Completed 80000 out of 250000 steps  (32 percent)
[12:12:35] Timered checkpoint triggered.
[12:15:34] Timered checkpoint triggered.
[12:18:34] Timered checkpoint triggered.
[12:21:35] Timered checkpoint triggered.
[12:24:34] Timered checkpoint triggered.
[12:27:35] Timered checkpoint triggered.
[12:30:36] Timered checkpoint triggered.
[12:33:21] Writing local files
[12:33:21] Completed 82500 out of 250000 steps  (33 percent)
[12:36:22] Timered checkpoint triggered.
[12:39:22] Timered checkpoint triggered.
[12:42:22] Timered checkpoint triggered.
[12:45:22] Timered checkpoint triggered.
[12:48:22] Timered checkpoint triggered.
[12:51:22] Timered checkpoint triggered.
[12:54:22] Timered checkpoint triggered.
[12:57:09] Writing local files
[12:57:09] Completed 85000 out of 250000 steps  (34 percent)
[13:00:09] Timered checkpoint triggered.
[13:03:10] Timered checkpoint triggered.
[13:06:10] Timered checkpoint triggered.
[13:09:12] Timered checkpoint triggered.
[13:12:12] Timered checkpoint triggered.
[13:15:13] Timered checkpoint triggered.
[13:18:13] Timered checkpoint triggered.
[13:21:16] Timered checkpoint triggered.
[13:23:18] Writing local files
[13:23:18] Completed 87500 out of 250000 steps  (35 percent)
[13:26:19] Timered checkpoint triggered.
[13:29:18] Timered checkpoint triggered.
[13:32:19] Timered checkpoint triggered.
[13:35:19] Timered checkpoint triggered.
[13:38:20] Timered checkpoint triggered.
[13:41:21] Timered checkpoint triggered.
[13:44:22] Timered checkpoint triggered.
[13:47:22] Timered checkpoint triggered.
[13:47:40] Writing local files
[13:47:41] Completed 90000 out of 250000 steps  (36 percent)
[13:50:41] Timered checkpoint triggered.
[13:53:42] Timered checkpoint triggered.
[13:56:43] Timered checkpoint triggered.
[13:59:45] Timered checkpoint triggered.
[14:02:45] Timered checkpoint triggered.
[14:05:46] Timered checkpoint triggered.
[14:08:47] Timered checkpoint triggered.
[14:11:48] Timered checkpoint triggered.
[14:11:51] Writing local files
[14:11:51] Completed 92500 out of 250000 steps  (37 percent)
[14:14:52] Timered checkpoint triggered.
[14:17:53] Timered checkpoint triggered.
[14:20:55] Timered checkpoint triggered.
[14:23:55] Timered checkpoint triggered.
[14:26:56] Timered checkpoint triggered.
[14:29:57] Timered checkpoint triggered.
[14:32:58] Timered checkpoint triggered.
[14:35:59] Timered checkpoint triggered.
[14:36:01] Writing local files
[14:36:01] Completed 95000 out of 250000 steps  (38 percent)
[14:39:02] Timered checkpoint triggered.
[14:42:03] Timered checkpoint triggered.
[14:45:04] Timered checkpoint triggered.
[14:48:05] Timered checkpoint triggered.
[14:51:06] Timered checkpoint triggered.
[14:54:06] Timered checkpoint triggered.
[14:57:07] Timered checkpoint triggered.
[15:00:05] Writing local files
[15:00:05] Completed 97500 out of 250000 steps  (39 percent)
[15:03:06] Timered checkpoint triggered.
[15:06:06] Timered checkpoint triggered.
[15:08:27] Warning:  long 1-4 interactions
[15:08:30] Gromacs cannot continue further.
[15:08:30] Going to send back what have done.
[15:08:30] logfile size: 99397
[15:08:30] - Writing 99933 bytes of core data to disk...
[15:08:30]   ... Done.
[15:08:30] - Failed to delete work/wudata_01.arc
[15:08:30] No C.P. to delete.
[15:08:30] Warning:  check for stray files
[15:08:30] 
[15:08:30] Folding@home Core Shutdown: EARLY_UNIT_END
[15:08:30] 
[15:08:30] Folding@home Core Shutdown: EARLY_UNIT_END
[15:08:33] CoreStatus = 7B (123)
[15:08:33] Client-core communications error: ERROR 0x7b
[15:08:33] Deleting current work unit & continuing...
[15:08:33] Using generic mpiexec calls
[15:10:36] - Warning: Could not delete all work unit files (1): Core returned invalid code
[15:10:36] Trying to send all finished work units
[15:10:36] + No unsent completed units remaining.
[15:10:36] - Preparing to get new work unit...
[15:10:36] + Attempting to get work packet
[15:10:36] - Will indicate memory of 3068 MB
[15:10:36] - Connecting to assignment server
[15:10:36] Connecting to http://assign.stanford.edu:8080/
[15:10:36] Posted data.
[15:10:36] Initial: 40AB; - Successful: assigned to (171.64.65.64).
[15:10:36] + News From Folding@Home: Welcome to Folding@Home
[15:10:36] Loaded queue successfully.
[15:10:36] Connecting to http://171.64.65.64:8080/
[15:10:41] Posted data.
[15:10:41] Initial: 0000; - Receiving payload (expected size: 4722302)
[15:10:48] - Downloaded at ~658 kB/s
[15:10:48] - Averaged speed for that direction ~686 kB/s
[15:10:48] + Received work.
[15:10:48] + Closed connections
[15:10:53] 
[15:10:53] + Processing work unit
[15:10:53] Work type a1 not eligible for variable processors
[15:10:53] Core required: FahCore_a1.exe
[15:10:53] Core found.
[15:10:53] Using generic mpiexec calls
[15:10:53] Working on queue slot 02 [October 6 15:10:53 UTC]
[15:10:53] + Working ...
[15:10:53] - Calling 'mpiexec -np 4 -channel auto -host 127.0.0.1 FahCore_a1.exe -dir work/ -suffix 02 -checkpoint 3 -service -verbose -lifeline 4396 -version 622'

[15:10:53] 
[15:10:53] *------------------------------*
[15:10:53] Folding@Home Gromacs SMP Core
[15:10:53] Version 1.74 (March 10, 2007)
[15:10:53] 
[15:10:53] Preparing to commence simulation
[15:10:53] - Ensuring status. Please wait.
[15:11:00] - Data doesn't match checksum.
[15:11:00] - Starting from initial work packet
[15:11:00] 
[15:11:00] Project: 2665 (Run 3, Clone 611, Gen 51)
[15:11:00] 
[15:11:00] Entering M.D.
[15:11:21] rom initial work packet
[15:11:21] 
[15:11:21] Project: 2665 (Run 3, Clone 611, Gen 51)
[15:11:21] 
[15:11:26] Entering M.D.
[15:11:32] Rejecting checkpoint
[15:11:34] Protein: HGG in water
[15:11:34] Writing local files
[15:11:43] Extra SSE boost OK.
[15:11:44] Writing local files
[15:11:44] Completed 0 out of 250000 steps  (0 percent)
[15:14:45] Timered checkpoint triggered.
[15:17:46] Timered checkpoint triggered.
[15:20:46] Timered checkpoint triggered.
[15:23:46] Timered checkpoint triggered.
[15:26:46] Timered checkpoint triggered.
[15:29:47] Timered checkpoint triggered.
[15:32:48] Timered checkpoint triggered.
[15:35:48] Timered checkpoint triggered.
[15:36:27] Writing local files
[15:36:28] Completed 2500 out of 250000 steps  (1 percent)
[15:39:30] Timered checkpoint triggered.
[15:42:30] Timered checkpoint triggered.
[15:45:30] Timered checkpoint triggered.
[15:48:29] Timered checkpoint triggered.
[15:51:30] Timered checkpoint triggered.
[15:54:31] Timered checkpoint triggered.
[15:57:32] Timered checkpoint triggered.
[16:00:33] Timered checkpoint triggered.
[16:01:51] Writing local files
[16:01:52] Completed 5000 out of 250000 steps  (2 percent)
[16:04:54] Timered checkpoint triggered.
[16:07:55] Timered checkpoint triggered.
[16:10:57] Timered checkpoint triggered.
[16:13:57] Timered checkpoint triggered.
[16:16:57] Timered checkpoint triggered.
[16:19:57] Timered checkpoint triggered.
[16:22:57] Timered checkpoint triggered.
[16:25:58] Timered checkpoint triggered.
[16:26:36] Writing local files
[16:26:36] Completed 7500 out of 250000 steps  (3 percent)
[16:29:37] Timered checkpoint triggered.
[16:32:38] Timered checkpoint triggered.
[16:35:39] Timered checkpoint triggered.
[16:38:40] Timered checkpoint triggered.
[16:41:41] Timered checkpoint triggered.
[16:44:42] Timered checkpoint triggered.
[16:45:56] - Autosending finished units... [October 6 16:45:56 UTC]
[16:45:56] Trying to send all finished work units
[16:45:56] + No unsent completed units remaining.
[16:45:56] - Autosend completed
[16:47:43] Timered checkpoint triggered.
[16:50:44] Timered checkpoint triggered.
[16:51:00] Writing local files
[16:51:00] Completed 10000 out of 250000 steps  (4 percent)
[16:54:01] Timered checkpoint triggered.
[16:57:02] Timered checkpoint triggered.
[17:00:03] Timered checkpoint triggered.
[17:03:04] Timered checkpoint triggered.
[17:06:05] Timered checkpoint triggered.
[17:09:06] Timered checkpoint triggered.
[17:12:07] Timered checkpoint triggered.
[17:15:08] Timered checkpoint triggered.
[17:16:45] Writing local files
[17:16:45] Completed 12500 out of 250000 steps  (5 percent)
[17:19:46] Timered checkpoint triggered.
[17:22:47] Timered checkpoint triggered.
[17:25:48] Timered checkpoint triggered.
[17:28:49] Timered checkpoint triggered.
[17:31:49] Timered checkpoint triggered.
[17:34:50] Timered checkpoint triggered.
[17:37:51] Timered checkpoint triggered.
[17:40:52] Timered checkpoint triggered.
[17:42:08] Writing local files
[17:42:08] Completed 15000 out of 250000 steps  (6 percent)
[17:45:09] Timered checkpoint triggered.
[17:48:10] Timered checkpoint triggered.
[17:51:11] Timered checkpoint triggered.
[17:54:12] Timered checkpoint triggered.
[17:57:13] Timered checkpoint triggered.
[18:00:13] Timered checkpoint triggered.
[18:03:14] Timered checkpoint triggered.
[18:06:15] Timered checkpoint triggered.
[18:07:47] Writing local files
[18:07:47] Completed 17500 out of 250000 steps  (7 percent)
[18:10:48] Timered checkpoint triggered.
[18:13:49] Timered checkpoint triggered.
[18:16:49] Timered checkpoint triggered.
[18:19:49] Timered checkpoint triggered.
[18:22:51] Timered checkpoint triggered.
[18:25:51] Timered checkpoint triggered.
[18:28:52] Timered checkpoint triggered.
[18:31:52] Timered checkpoint triggered.
[18:34:31] Writing local files
[18:34:31] Completed 20000 out of 250000 steps  (8 percent)
[18:37:32] Timered checkpoint triggered.
[18:40:33] Timered checkpoint triggered.
[18:43:34] Timered checkpoint triggered.
[18:46:34] Timered checkpoint triggered.
[18:49:35] Timered checkpoint triggered.
[18:52:36] Timered checkpoint triggered.
[18:55:37] Timered checkpoint triggered.
[18:58:38] Timered checkpoint triggered.
[19:00:11] Writing local files
[19:00:11] Completed 22500 out of 250000 steps  (9 percent)
[19:03:12] Timered checkpoint triggered.
[19:06:12] Timered checkpoint triggered.
[19:09:13] Timered checkpoint triggered.
[19:12:14] Timered checkpoint triggered.
[19:15:16] Timered checkpoint triggered.
[19:18:17] Timered checkpoint triggered.
[19:21:18] Timered checkpoint triggered.
[19:24:18] Timered checkpoint triggered.
[19:25:50] Writing local files
[19:25:51] Completed 25000 out of 250000 steps  (10 percent)
[19:28:52] Timered checkpoint triggered.
[19:28:57] Service stop request received.
[19:28:57] ***** Got a SIGTERM signal (2)
[19:28:57] Killing all core threads
[19:28:57] Killing 3 cores
[19:28:57] Killing core 0
[19:28:57] Killing core 1
[19:28:57] Killing core 2

Folding@Home Client Shutdown.

[toTOW]

Someone returned it for partial credit.

[ChrisDTC]

Someone returned it for partial credit.

Where do you see this stuff?

[toTOW]

This is a moderator tool to help Stanford deal with issues with WUs.

[ChrisDTC]

Code: Select all

[19:35:47] *------------------------------*
[19:35:47] Folding@Home Gromacs SMP Core
[19:35:47] Version 1.74 (March 10, 2007)
[19:35:47] 
[19:35:47] Preparing to commence simulation
[19:35:47] - Ensuring status. Please wait.
[19:36:04] - Looking at optimizations...
[19:36:04] - Working with standard loops on this execution.
[19:36:04] - Previous termination of core was improper.
[19:36:04] - Going to use standard loops.
[19:36:04] - Files status OK
[19:36:22] (decompressed 517.3 percent)
[19:36:23] - Starting from initial work packet
[19:36:23] 
[19:36:23] Project: 2665 (Run 3, Clone 611, Gen 51)
[19:36:23] 
[19:36:24] 65 (Run 3, Clone 611, Gen 51)
[19:36:24] 
[19:36:26] Entering M.D.
[19:36:32] Rejecting checkpoint
[19:36:34] Protein: HGG in water
[19:36:35] Writing local files
[19:36:45] Extra SSE boost OK.
[19:36:45] Writing local files
[19:36:46] Completed 0 out of 250000 steps  (0 percent)
[19:39:45] Timered checkpoint triggered.
[19:42:47] Timered checkpoint triggered.
[19:45:46] Timered checkpoint triggered.
[19:48:47] Timered checkpoint triggered.
[19:51:48] Timered checkpoint triggered.
[19:54:47] Timered checkpoint triggered.
[19:57:48] Timered checkpoint triggered.
[20:00:49] Timered checkpoint triggered.
[20:00:55] Writing local files
[20:00:55] Completed 2500 out of 250000 steps  (1 percent)
[20:03:56] Timered checkpoint triggered.
[20:06:57] Timered checkpoint triggered.
[20:09:58] Timered checkpoint triggered.
[20:12:59] Timered checkpoint triggered.
[20:16:01] Timered checkpoint triggered.
[20:19:02] Timered checkpoint triggered.
[20:22:03] Timered checkpoint triggered.
[20:24:40] Writing local files
[20:24:40] Completed 5000 out of 250000 steps  (2 percent)
[20:27:42] Timered checkpoint triggered.
[20:30:43] Timered checkpoint triggered.
[20:33:44] Timered checkpoint triggered.
[20:36:46] Timered checkpoint triggered.
[20:39:48] Timered checkpoint triggered.
[20:42:49] Timered checkpoint triggered.
[20:45:50] Timered checkpoint triggered.
[20:48:27] Writing local files
[20:48:27] Completed 7500 out of 250000 steps  (3 percent)
[20:51:28] Timered checkpoint triggered.
[20:54:28] Timered checkpoint triggered.
[20:57:29] Timered checkpoint triggered.
[21:00:29] Timered checkpoint triggered.
[21:03:29] Timered checkpoint triggered.
[21:06:29] Timered checkpoint triggered.
[21:09:30] Timered checkpoint triggered.
[21:12:12] Writing local files
[21:12:12] Completed 10000 out of 250000 steps  (4 percent)
[21:15:12] Timered checkpoint triggered.
[21:18:13] Timered checkpoint triggered.
[21:21:12] Timered checkpoint triggered.
[21:24:12] Timered checkpoint triggered.
[21:27:13] Timered checkpoint triggered.
[21:30:12] Timered checkpoint triggered.
[21:33:13] Timered checkpoint triggered.
[21:35:58] Writing local files
[21:35:59] Completed 12500 out of 250000 steps  (5 percent)
[21:38:58] Timered checkpoint triggered.
[21:41:59] Timered checkpoint triggered.
[21:44:58] Timered checkpoint triggered.
[21:47:59] Timered checkpoint triggered.
[21:51:00] Timered checkpoint triggered.
[21:53:59] Timered checkpoint triggered.
[21:57:00] Timered checkpoint triggered.
[21:59:51] Writing local files
[21:59:52] Completed 15000 out of 250000 steps  (6 percent)
[22:02:52] Timered checkpoint triggered.
[22:05:51] Timered checkpoint triggered.
[22:08:52] Timered checkpoint triggered.
[22:11:51] Timered checkpoint triggered.
[22:14:52] Timered checkpoint triggered.
[22:17:53] Timered checkpoint triggered.
[22:20:52] Timered checkpoint triggered.
[22:23:37] Writing local files
[22:23:37] Completed 17500 out of 250000 steps  (7 percent)
[22:26:38] Timered checkpoint triggered.
[22:29:38] Timered checkpoint triggered.
[22:32:38] Timered checkpoint triggered.
[22:35:38] Timered checkpoint triggered.
[22:38:37] Timered checkpoint triggered.
[22:41:38] Timered checkpoint triggered.
[22:44:38] Timered checkpoint triggered.
[22:47:25] Writing local files
[22:47:26] Completed 20000 out of 250000 steps  (8 percent)
[22:50:26] Timered checkpoint triggered.
[22:53:25] Timered checkpoint triggered.
[22:56:26] Timered checkpoint triggered.
[22:59:26] Timered checkpoint triggered.
[23:02:27] Timered checkpoint triggered.
[23:05:28] Timered checkpoint triggered.
[23:08:29] Timered checkpoint triggered.
[23:11:13] Writing local files
[23:11:13] Completed 22500 out of 250000 steps  (9 percent)
[23:14:14] Timered checkpoint triggered.
[23:17:15] Timered checkpoint triggered.
[23:20:16] Timered checkpoint triggered.
[23:23:16] Timered checkpoint triggered.
[23:26:16] Timered checkpoint triggered.
[23:29:17] Timered checkpoint triggered.
[23:32:17] Timered checkpoint triggered.
[23:35:11] Writing local files
[23:35:11] Completed 25000 out of 250000 steps  (10 percent)
[23:38:12] Timered checkpoint triggered.
[23:41:13] Timered checkpoint triggered.
[23:44:14] Timered checkpoint triggered.
[23:47:15] Timered checkpoint triggered.
[23:50:16] Timered checkpoint triggered.
[23:53:17] Timered checkpoint triggered.
[23:56:18] Timered checkpoint triggered.
[23:59:07] Writing local files
[23:59:08] Completed 27500 out of 250000 steps  (11 percent)
[00:02:08] Timered checkpoint triggered.
[00:05:10] Timered checkpoint triggered.
[00:08:11] Timered checkpoint triggered.
[00:11:13] Timered checkpoint triggered.
[00:14:13] Timered checkpoint triggered.
[00:17:14] Timered checkpoint triggered.
[00:20:16] Timered checkpoint triggered.
[00:23:01] Writing local files
[00:23:02] Completed 30000 out of 250000 steps  (12 percent)
[00:26:01] Timered checkpoint triggered.
[00:29:02] Timered checkpoint triggered.
[00:32:02] Timered checkpoint triggered.
[00:35:03] Timered checkpoint triggered.
[00:38:04] Timered checkpoint triggered.
[00:41:04] Timered checkpoint triggered.
[00:44:03] Timered checkpoint triggered.
[00:46:52] Writing local files
[00:46:52] Completed 32500 out of 250000 steps  (13 percent)
[00:49:51] Timered checkpoint triggered.
[00:52:52] Timered checkpoint triggered.
[00:55:52] Timered checkpoint triggered.
[00:58:52] Timered checkpoint triggered.
[01:01:53] Timered checkpoint triggered.
[01:04:52] Timered checkpoint triggered.
[01:07:53] Timered checkpoint triggered.
[01:10:39] Writing local files
[01:10:39] Completed 35000 out of 250000 steps  (14 percent)
[01:13:39] Timered checkpoint triggered.
[01:16:40] Timered checkpoint triggered.
[01:19:39] Timered checkpoint triggered.
[01:22:39] Timered checkpoint triggered.
[01:25:40] Timered checkpoint triggered.
[01:28:39] Timered checkpoint triggered.
[01:31:06] - Autosending finished units... [October 7 01:31:06 UTC]
[01:31:06] Trying to send all finished work units
[01:31:06] + No unsent completed units remaining.
[01:31:06] - Autosend completed
[01:31:39] Timered checkpoint triggered.
[01:34:26] Writing local files
[01:34:26] Completed 37500 out of 250000 steps  (15 percent)
[01:37:26] Timered checkpoint triggered.
[01:40:27] Timered checkpoint triggered.
[01:43:27] Timered checkpoint triggered.
[01:46:26] Timered checkpoint triggered.
[01:49:27] Timered checkpoint triggered.
[01:52:26] Timered checkpoint triggered.
[01:55:27] Timered checkpoint triggered.
[01:58:14] Writing local files
[01:58:14] Completed 40000 out of 250000 steps  (16 percent)
[02:01:15] Timered checkpoint triggered.
[02:04:15] Timered checkpoint triggered.
[02:07:15] Timered checkpoint triggered.
[02:10:16] Timered checkpoint triggered.
[02:13:16] Timered checkpoint triggered.
[02:16:17] Timered checkpoint triggered.
[02:19:18] Timered checkpoint triggered.
[02:22:03] Writing local files
[02:22:03] Completed 42500 out of 250000 steps  (17 percent)
[02:25:04] Timered checkpoint triggered.
[02:28:04] Timered checkpoint triggered.
[02:31:05] Timered checkpoint triggered.
[02:34:06] Timered checkpoint triggered.
[02:37:07] Timered checkpoint triggered.
[02:40:08] Timered checkpoint triggered.
[02:43:09] Timered checkpoint triggered.
[02:45:54] Writing local files
[02:45:54] Completed 45000 out of 250000 steps  (18 percent)
[02:48:55] Timered checkpoint triggered.
[02:51:55] Timered checkpoint triggered.
[02:54:56] Timered checkpoint triggered.
[02:57:57] Timered checkpoint triggered.
[03:00:58] Timered checkpoint triggered.
[03:04:00] Timered checkpoint triggered.
[03:07:01] Timered checkpoint triggered.
[03:09:43] Writing local files
[03:09:43] Completed 47500 out of 250000 steps  (19 percent)
[03:12:44] Timered checkpoint triggered.
[03:15:45] Timered checkpoint triggered.
[03:18:46] Timered checkpoint triggered.
[03:21:47] Timered checkpoint triggered.
[03:24:48] Timered checkpoint triggered.
[03:27:49] Timered checkpoint triggered.
[03:30:51] Timered checkpoint triggered.
[03:33:32] Writing local files
[03:33:32] Completed 50000 out of 250000 steps  (20 percent)
[03:36:33] Timered checkpoint triggered.
[03:39:34] Timered checkpoint triggered.
[03:42:35] Timered checkpoint triggered.
[03:45:36] Timered checkpoint triggered.
[03:48:36] Timered checkpoint triggered.
[03:51:36] Timered checkpoint triggered.
[03:54:36] Timered checkpoint triggered.
[03:57:20] Writing local files
[03:57:20] Completed 52500 out of 250000 steps  (21 percent)
[04:00:20] Timered checkpoint triggered.
[04:03:20] Timered checkpoint triggered.
[04:06:20] Timered checkpoint triggered.
[04:09:20] Timered checkpoint triggered.
[04:12:21] Timered checkpoint triggered.
[04:15:20] Timered checkpoint triggered.
[04:18:21] Timered checkpoint triggered.
[04:21:07] Writing local files
[04:21:07] Completed 55000 out of 250000 steps  (22 percent)
[04:24:07] Timered checkpoint triggered.
[04:27:08] Timered checkpoint triggered.
[04:30:07] Timered checkpoint triggered.
[04:33:07] Timered checkpoint triggered.
[04:36:08] Timered checkpoint triggered.
[04:39:07] Timered checkpoint triggered.
[04:42:08] Timered checkpoint triggered.
[04:44:54] Writing local files
[04:44:55] Completed 57500 out of 250000 steps  (23 percent)
[04:47:54] Timered checkpoint triggered.
[04:50:55] Timered checkpoint triggered.
[04:53:54] Timered checkpoint triggered.
[04:56:55] Timered checkpoint triggered.
[04:59:55] Timered checkpoint triggered.
[05:02:54] Timered checkpoint triggered.
[05:05:55] Timered checkpoint triggered.
[05:08:41] Writing local files
[05:08:41] Completed 60000 out of 250000 steps  (24 percent)
[05:11:42] Timered checkpoint triggered.
[05:14:42] Timered checkpoint triggered.
[05:17:41] Timered checkpoint triggered.
[05:20:42] Timered checkpoint triggered.
[05:23:42] Timered checkpoint triggered.
[05:26:42] Timered checkpoint triggered.
[05:29:43] Timered checkpoint triggered.
[05:32:31] Writing local files
[05:32:31] Completed 62500 out of 250000 steps  (25 percent)
[05:35:31] Timered checkpoint triggered.
[05:38:31] Timered checkpoint triggered.
[05:41:32] Timered checkpoint triggered.
[05:44:31] Timered checkpoint triggered.
[05:47:32] Timered checkpoint triggered.
[05:50:33] Timered checkpoint triggered.
[05:53:33] Timered checkpoint triggered.
[05:56:20] Writing local files
[05:56:20] Completed 65000 out of 250000 steps  (26 percent)
[05:59:20] Timered checkpoint triggered.
[06:02:21] Timered checkpoint triggered.
[06:05:23] Timered checkpoint triggered.
[06:08:24] Timered checkpoint triggered.
[06:11:25] Timered checkpoint triggered.
[06:14:26] Timered checkpoint triggered.
[06:17:26] Timered checkpoint triggered.
[06:20:09] Writing local files
[06:20:09] Completed 67500 out of 250000 steps  (27 percent)
[06:23:09] Timered checkpoint triggered.
[06:26:10] Timered checkpoint triggered.
[06:29:11] Timered checkpoint triggered.
[06:32:23] Timered checkpoint triggered.
[06:35:24] Timered checkpoint triggered.
[06:38:25] Timered checkpoint triggered.
[06:41:27] Timered checkpoint triggered.
[06:44:27] Timered checkpoint triggered.
[06:47:03] Writing local files
[06:47:03] Completed 70000 out of 250000 steps  (28 percent)
[06:50:05] Timered checkpoint triggered.
[06:53:05] Timered checkpoint triggered.
[06:56:06] Timered checkpoint triggered.
[06:59:07] Timered checkpoint triggered.
[07:02:08] Timered checkpoint triggered.
[07:05:09] Timered checkpoint triggered.
[07:08:11] Timered checkpoint triggered.
[07:10:51] Writing local files
[07:10:51] Completed 72500 out of 250000 steps  (29 percent)
[07:13:51] Timered checkpoint triggered.
[07:16:51] Timered checkpoint triggered.
[07:19:51] Timered checkpoint triggered.
[07:22:52] Timered checkpoint triggered.
[07:25:51] Timered checkpoint triggered.
[07:28:51] Timered checkpoint triggered.
[07:31:06] - Autosending finished units... [October 7 07:31:06 UTC]
[07:31:06] Trying to send all finished work units
[07:31:06] + No unsent completed units remaining.
[07:31:06] - Autosend completed
[07:31:52] Timered checkpoint triggered.
[07:34:39] Writing local files
[07:34:40] Completed 75000 out of 250000 steps  (30 percent)
[07:37:39] Timered checkpoint triggered.
[07:40:40] Timered checkpoint triggered.
[07:43:39] Timered checkpoint triggered.
[07:46:40] Timered checkpoint triggered.
[07:49:40] Timered checkpoint triggered.
[07:52:40] Timered checkpoint triggered.
[07:55:40] Timered checkpoint triggered.
[07:58:28] Writing local files
[07:58:28] Completed 77500 out of 250000 steps  (31 percent)
[08:01:28] Timered checkpoint triggered.
[08:04:28] Timered checkpoint triggered.
[08:07:29] Timered checkpoint triggered.
[08:10:29] Timered checkpoint triggered.
[08:13:28] Timered checkpoint triggered.
[08:16:29] Timered checkpoint triggered.
[08:19:29] Timered checkpoint triggered.
[08:22:20] Writing local files
[08:22:21] Completed 80000 out of 250000 steps  (32 percent)
[08:25:20] Timered checkpoint triggered.
[08:28:21] Timered checkpoint triggered.
[08:31:22] Timered checkpoint triggered.
[08:34:22] Timered checkpoint triggered.
[08:37:22] Timered checkpoint triggered.
[08:40:22] Timered checkpoint triggered.
[08:43:23] Timered checkpoint triggered.
[08:46:09] Writing local files
[08:46:09] Completed 82500 out of 250000 steps  (33 percent)
[08:49:09] Timered checkpoint triggered.
[08:52:10] Timered checkpoint triggered.
[08:55:10] Timered checkpoint triggered.
[08:58:09] Timered checkpoint triggered.
[09:01:11] Timered checkpoint triggered.
[09:04:11] Timered checkpoint triggered.
[09:07:11] Timered checkpoint triggered.
[09:09:56] Writing local files
[09:09:56] Completed 85000 out of 250000 steps  (34 percent)
[09:12:56] Timered checkpoint triggered.
[09:15:57] Timered checkpoint triggered.
[09:18:59] Timered checkpoint triggered.
[09:21:59] Timered checkpoint triggered.
[09:25:00] Timered checkpoint triggered.
[09:28:01] Timered checkpoint triggered.
[09:31:02] Timered checkpoint triggered.
[09:33:43] Writing local files
[09:33:43] Completed 87500 out of 250000 steps  (35 percent)
[09:36:44] Timered checkpoint triggered.
[09:39:45] Timered checkpoint triggered.
[09:42:46] Timered checkpoint triggered.
[09:45:47] Timered checkpoint triggered.
[09:48:48] Timered checkpoint triggered.
[09:51:49] Timered checkpoint triggered.
[09:54:50] Timered checkpoint triggered.
[09:57:30] Writing local files
[09:57:30] Completed 90000 out of 250000 steps  (36 percent)
[10:00:31] Timered checkpoint triggered.
[10:03:32] Timered checkpoint triggered.
[10:06:33] Timered checkpoint triggered.
[10:09:34] Timered checkpoint triggered.
[10:12:35] Timered checkpoint triggered.
[10:15:37] Timered checkpoint triggered.
[10:18:37] Timered checkpoint triggered.
[10:21:16] Writing local files
[10:21:17] Completed 92500 out of 250000 steps  (37 percent)
[10:24:18] Timered checkpoint triggered.
[10:27:18] Timered checkpoint triggered.
[10:30:19] Timered checkpoint triggered.
[10:33:19] Timered checkpoint triggered.
[10:36:19] Timered checkpoint triggered.
[10:39:20] Timered checkpoint triggered.
[10:42:19] Timered checkpoint triggered.
[10:45:20] Writing local files
[10:45:20] Completed 95000 out of 250000 steps  (38 percent)
[10:48:21] Timered checkpoint triggered.
[10:51:20] Timered checkpoint triggered.
[10:54:21] Timered checkpoint triggered.
[10:57:21] Timered checkpoint triggered.
[11:00:20] Timered checkpoint triggered.
[11:03:21] Timered checkpoint triggered.
[11:06:21] Timered checkpoint triggered.
[11:09:10] Writing local files
[11:09:10] Completed 97500 out of 250000 steps  (39 percent)
[11:12:10] Timered checkpoint triggered.
[11:15:10] Timered checkpoint triggered.
[11:17:18] Warning:  long 1-4 interactions
[11:17:21] Gromacs cannot continue further.
[11:17:21] Going to send back what have done.
[11:17:21] logfile size: 71097
[11:17:21] - Writing 71633 bytes of core data to disk...
[11:17:21]   ... Done.
[11:17:21] - Failed to delete work/wudata_03.arc
[11:17:21] No C.P. to delete.
[11:17:21] Warning:  check for stray files
[11:17:21] 
[11:17:21] Folding@home Core Shutdown: EARLY_UNIT_END
[11:17:21] 
[11:17:21] Folding@home Core Shutdown: EARLY_UNIT_END
[11:17:24] CoreStatus = 7B (123)
[11:17:24] Client-core communications error: ERROR 0x7b
[11:17:24] Deleting current work unit & continuing...
[11:17:24] Using generic mpiexec calls
[11:19:26] - Warning: Could not delete all work unit files (3): Core returned invalid code
[11:19:26] Trying to send all finished work units
[11:19:26] + No unsent completed units remaining.
[11:19:26] - Preparing to get new work unit...
[11:19:26] + Attempting to get work packet
[11:19:26] - Will indicate memory of 3068 MB
[11:19:26] - Connecting to assignment server
[11:19:26] Connecting to http://assign.stanford.edu:8080/
[11:19:27] Posted data.
[11:19:27] Initial: 40AB; - Successful: assigned to (171.64.65.64).
[11:19:27] + News From Folding@Home: Welcome to Folding@Home
[11:19:27] Loaded queue successfully.
[11:19:27] Connecting to http://171.64.65.64:8080/
[11:19:32] Posted data.
[11:19:32] Initial: 0000; - Receiving payload (expected size: 4722302)
[11:19:38] - Downloaded at ~768 kB/s
[11:19:38] - Averaged speed for that direction ~716 kB/s
[11:19:38] + Received work.
[11:19:38] + Closed connections
[11:19:43] 
[11:19:43] + Processing work unit
[11:19:43] Work type a1 not eligible for variable processors
[11:19:43] Core required: FahCore_a1.exe
[11:19:43] Core found.
[11:19:43] Using generic mpiexec calls
[11:19:43] Working on queue slot 04 [October 7 11:19:43 UTC]
[11:19:43] + Working ...
[11:19:43] - Calling 'mpiexec -np 4 -channel auto -host 127.0.0.1 FahCore_a1.exe -dir work/ -suffix 04 -checkpoint 3 -service -verbose -lifeline 4600 -version 622'

[11:19:43] 

It says it sent in what it finished, but it downloaded the same WU again, and it will most certainly fail again

[toTOW]

Use qfix : viewtopic.php?f=19&t=6042