171.67.108.45 and 171.64.65.35
Posted: Wed Sep 13, 2017 8:44 pm
Hey,
Just started having problems uploading work and receiving jobs.
Last job and following errors:
Got 4 CPU servers and 1 server with 2 GPUs experiencing this.
Maybe related to the fixing procedures for the stats servers, but I thought I'd mention it.
Just started having problems uploading work and receiving jobs.
Last job and following errors:
Code: Select all
19:02:01:WU02:FS01:Starting
19:02:01:WU02:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 02 -suffix 01 -version 704 -lifeline 1527 -checkpoint 15 -gpu 1 -gpu-vendor nvidia
19:02:01:WU02:FS01:Started FahCore on PID 8552
19:02:01:WU02:FS01:Core PID:8556
19:02:01:WU02:FS01:FahCore 0x21 started
19:02:01:WU02:FS01:0x21:*********************** Log Started 2017-09-13T19:02:01Z ***********************
19:02:01:WU02:FS01:0x21:Project: 9415 (Run 1680, Clone 0, Gen 344)
19:02:01:WU02:FS01:0x21:Unit: 0x00000187ab436c9d585e06d8baf4a0ca
19:02:01:WU02:FS01:0x21:CPU: 0x00000000000000000000000000000000
19:02:01:WU02:FS01:0x21:Machine: 1
19:02:01:WU02:FS01:0x21:Reading tar file core.xml
19:02:01:WU02:FS01:0x21:Reading tar file integrator.xml
19:02:01:WU02:FS01:0x21:Reading tar file state.xml
19:02:01:WU02:FS01:0x21:Reading tar file system.xml
19:02:01:WU02:FS01:0x21:Digital signatures verified
19:02:01:WU02:FS01:0x21:Folding@home GPU Core21 Folding@home Core
19:02:01:WU02:FS01:0x21:Version 0.0.18
19:02:02:WU02:FS01:0x21:Completed 0 out of 6250000 steps (0%)
19:02:02:WU02:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
19:02:03:WU03:FS01:Upload complete
19:02:03:WU03:FS01:Server responded WORK_ACK (400)
19:02:03:WU03:FS01:Final credit estimate, 53093.00 points
19:02:03:WU03:FS01:Cleaning up
19:02:42:WU02:FS01:0x21:Completed 62500 out of 6250000 steps (1%)
19:03:21:WU02:FS01:0x21:Completed 125000 out of 6250000 steps (2%)
19:04:01:WU02:FS01:0x21:Completed 187500 out of 6250000 steps (3%)
19:04:41:WU02:FS01:0x21:Completed 250000 out of 6250000 steps (4%)
19:05:21:WU02:FS01:0x21:Completed 312500 out of 6250000 steps (5%)
19:06:00:WU02:FS01:0x21:Completed 375000 out of 6250000 steps (6%)
19:06:40:WU02:FS01:0x21:Completed 437500 out of 6250000 steps (7%)
19:07:20:WU02:FS01:0x21:Completed 500000 out of 6250000 steps (8%)
19:07:59:WU02:FS01:0x21:Completed 562500 out of 6250000 steps (9%)
19:08:39:WU02:FS01:0x21:Completed 625000 out of 6250000 steps (10%)
19:09:19:WU02:FS01:0x21:Completed 687500 out of 6250000 steps (11%)
19:09:59:WU02:FS01:0x21:Completed 750000 out of 6250000 steps (12%)
19:10:39:WU02:FS01:0x21:Completed 812500 out of 6250000 steps (13%)
19:11:19:WU02:FS01:0x21:Completed 875000 out of 6250000 steps (14%)
19:11:58:WU02:FS01:0x21:Completed 937500 out of 6250000 steps (15%)
19:12:38:WU02:FS01:0x21:Completed 1000000 out of 6250000 steps (16%)
19:13:18:WU02:FS01:0x21:Completed 1062500 out of 6250000 steps (17%)
19:13:58:WU02:FS01:0x21:Completed 1125000 out of 6250000 steps (18%)
19:14:37:WU02:FS01:0x21:Completed 1187500 out of 6250000 steps (19%)
19:15:17:WU02:FS01:0x21:Completed 1250000 out of 6250000 steps (20%)
19:15:57:WU02:FS01:0x21:Completed 1312500 out of 6250000 steps (21%)
19:16:36:WU02:FS01:0x21:Completed 1375000 out of 6250000 steps (22%)
19:17:16:WU02:FS01:0x21:Completed 1437500 out of 6250000 steps (23%)
19:17:56:WU02:FS01:0x21:Completed 1500000 out of 6250000 steps (24%)
19:18:36:WU02:FS01:0x21:Completed 1562500 out of 6250000 steps (25%)
19:19:15:WU02:FS01:0x21:Completed 1625000 out of 6250000 steps (26%)
19:19:55:WU02:FS01:0x21:Completed 1687500 out of 6250000 steps (27%)
19:20:35:WU02:FS01:0x21:Completed 1750000 out of 6250000 steps (28%)
19:21:15:WU02:FS01:0x21:Completed 1812500 out of 6250000 steps (29%)
19:21:55:WU02:FS01:0x21:Completed 1875000 out of 6250000 steps (30%)
19:22:34:WU02:FS01:0x21:Completed 1937500 out of 6250000 steps (31%)
19:23:14:WU02:FS01:0x21:Completed 2000000 out of 6250000 steps (32%)
19:23:54:WU02:FS01:0x21:Completed 2062500 out of 6250000 steps (33%)
19:24:34:WU02:FS01:0x21:Completed 2125000 out of 6250000 steps (34%)
19:25:13:WU02:FS01:0x21:Completed 2187500 out of 6250000 steps (35%)
19:25:53:WU02:FS01:0x21:Completed 2250000 out of 6250000 steps (36%)
19:26:33:WU02:FS01:0x21:Completed 2312500 out of 6250000 steps (37%)
19:27:12:WU02:FS01:0x21:Completed 2375000 out of 6250000 steps (38%)
19:27:52:WU02:FS01:0x21:Completed 2437500 out of 6250000 steps (39%)
19:28:32:WU02:FS01:0x21:Completed 2500000 out of 6250000 steps (40%)
19:29:12:WU02:FS01:0x21:Completed 2562500 out of 6250000 steps (41%)
19:29:51:WU02:FS01:0x21:Completed 2625000 out of 6250000 steps (42%)
19:30:31:WU02:FS01:0x21:Completed 2687500 out of 6250000 steps (43%)
19:31:11:WU02:FS01:0x21:Completed 2750000 out of 6250000 steps (44%)
19:31:50:WU02:FS01:0x21:Completed 2812500 out of 6250000 steps (45%)
19:32:30:WU02:FS01:0x21:Completed 2875000 out of 6250000 steps (46%)
19:33:10:WU02:FS01:0x21:Completed 2937500 out of 6250000 steps (47%)
19:33:49:WU02:FS01:0x21:Completed 3000000 out of 6250000 steps (48%)
19:34:29:WU02:FS01:0x21:Completed 3062500 out of 6250000 steps (49%)
19:35:09:WU02:FS01:0x21:Completed 3125000 out of 6250000 steps (50%)
19:35:48:WU02:FS01:0x21:Completed 3187500 out of 6250000 steps (51%)
19:36:28:WU02:FS01:0x21:Completed 3250000 out of 6250000 steps (52%)
19:37:08:WU02:FS01:0x21:Completed 3312500 out of 6250000 steps (53%)
19:37:48:WU02:FS01:0x21:Completed 3375000 out of 6250000 steps (54%)
19:38:28:WU02:FS01:0x21:Completed 3437500 out of 6250000 steps (55%)
19:39:07:WU02:FS01:0x21:Completed 3500000 out of 6250000 steps (56%)
19:39:47:WU02:FS01:0x21:Completed 3562500 out of 6250000 steps (57%)
19:40:27:WU02:FS01:0x21:Completed 3625000 out of 6250000 steps (58%)
19:41:06:WU02:FS01:0x21:Completed 3687500 out of 6250000 steps (59%)
19:41:46:WU02:FS01:0x21:Completed 3750000 out of 6250000 steps (60%)
19:42:26:WU02:FS01:0x21:Completed 3812500 out of 6250000 steps (61%)
19:43:06:WU02:FS01:0x21:Completed 3875000 out of 6250000 steps (62%)
19:43:45:WU02:FS01:0x21:Completed 3937500 out of 6250000 steps (63%)
19:44:25:WU02:FS01:0x21:Completed 4000000 out of 6250000 steps (64%)
19:45:05:WU02:FS01:0x21:Completed 4062500 out of 6250000 steps (65%)
19:45:44:WU02:FS01:0x21:Completed 4125000 out of 6250000 steps (66%)
19:46:24:WU02:FS01:0x21:Completed 4187500 out of 6250000 steps (67%)
19:47:04:WU02:FS01:0x21:Completed 4250000 out of 6250000 steps (68%)
19:47:44:WU02:FS01:0x21:Completed 4312500 out of 6250000 steps (69%)
19:48:23:WU02:FS01:0x21:Completed 4375000 out of 6250000 steps (70%)
19:49:03:WU02:FS01:0x21:Completed 4437500 out of 6250000 steps (71%)
19:49:43:WU02:FS01:0x21:Completed 4500000 out of 6250000 steps (72%)
19:50:23:WU02:FS01:0x21:Completed 4562500 out of 6250000 steps (73%)
19:51:02:WU02:FS01:0x21:Completed 4625000 out of 6250000 steps (74%)
19:51:42:WU02:FS01:0x21:Completed 4687500 out of 6250000 steps (75%)
19:52:22:WU02:FS01:0x21:Completed 4750000 out of 6250000 steps (76%)
19:53:02:WU02:FS01:0x21:Completed 4812500 out of 6250000 steps (77%)
19:53:41:WU02:FS01:0x21:Completed 4875000 out of 6250000 steps (78%)
19:54:21:WU02:FS01:0x21:Completed 4937500 out of 6250000 steps (79%)
19:55:00:WU02:FS01:0x21:Completed 5000000 out of 6250000 steps (80%)
19:55:40:WU02:FS01:0x21:Completed 5062500 out of 6250000 steps (81%)
19:56:20:WU02:FS01:0x21:Completed 5125000 out of 6250000 steps (82%)
19:57:00:WU02:FS01:0x21:Completed 5187500 out of 6250000 steps (83%)
19:57:40:WU02:FS01:0x21:Completed 5250000 out of 6250000 steps (84%)
19:58:19:WU02:FS01:0x21:Completed 5312500 out of 6250000 steps (85%)
19:58:59:WU02:FS01:0x21:Completed 5375000 out of 6250000 steps (86%)
19:59:39:WU02:FS01:0x21:Completed 5437500 out of 6250000 steps (87%)
20:00:18:WU02:FS01:0x21:Completed 5500000 out of 6250000 steps (88%)
20:00:58:WU02:FS01:0x21:Completed 5562500 out of 6250000 steps (89%)
20:01:38:WU02:FS01:0x21:Completed 5625000 out of 6250000 steps (90%)
20:02:18:WU02:FS01:0x21:Completed 5687500 out of 6250000 steps (91%)
20:02:57:WU02:FS01:0x21:Completed 5750000 out of 6250000 steps (92%)
20:03:38:WU02:FS01:0x21:Completed 5812500 out of 6250000 steps (93%)
20:04:18:WU02:FS01:0x21:Completed 5875000 out of 6250000 steps (94%)
20:04:57:WU02:FS01:0x21:Completed 5937500 out of 6250000 steps (95%)
20:05:37:WU02:FS01:0x21:Completed 6000000 out of 6250000 steps (96%)
20:06:17:WU02:FS01:0x21:Completed 6062500 out of 6250000 steps (97%)
20:06:57:WU02:FS01:0x21:Completed 6125000 out of 6250000 steps (98%)
20:07:36:WU02:FS01:0x21:Completed 6187500 out of 6250000 steps (99%)
20:07:39:WU01:FS01:Connecting to 171.67.108.45:80
20:08:16:WU02:FS01:0x21:Completed 6250000 out of 6250000 steps (100%)
20:08:16:WU02:FS01:0x21:Saving result file logfile_01.txt
20:08:16:WU02:FS01:0x21:Saving result file checkpointState.xml
20:08:16:WU02:FS01:0x21:Saving result file checkpt.crc
20:08:16:WU02:FS01:0x21:Saving result file log.txt
20:08:16:WU02:FS01:0x21:Saving result file positions.xtc
20:08:16:WU02:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
20:08:17:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
20:08:17:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:9415 run:1680 clone:0 gen:344 core:0x21 unit:0x00000187ab436c9d585e06d8baf4a0ca
20:08:17:WU02:FS01:Uploading 7.80MiB to 171.67.108.157
20:08:17:WU02:FS01:Connecting to 171.67.108.157:8080
20:08:20:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.45:80': 10002: Received short response, expected 272 bytes, got 0
20:08:20:WU01:FS01:Connecting to 171.64.65.35:80
20:09:06:WARNING:WU01:FS01:Failed to get assignment from '171.64.65.35:80': 10002: Received short response, expected 272 bytes, got 0
20:09:06:ERROR:WU01:FS01:Exception: Could not get an assignment
20:09:06:WU01:FS01:Connecting to 171.67.108.45:80
20:09:50:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.45:80': 10002: Received short response, expected 272 bytes, got 0
20:09:50:WU01:FS01:Connecting to 171.64.65.35:80
20:10:24:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
20:10:24:WU02:FS01:Connecting to 171.67.108.157:80
20:10:24:WU02:FS01:Upload 0.80%
20:10:34:WARNING:WU01:FS01:Failed to get assignment from '171.64.65.35:80': 10002: Received short response, expected 272 bytes, got 0
20:10:34:ERROR:WU01:FS01:Exception: Could not get an assignment
20:10:34:WU01:FS01:Connecting to 171.67.108.45:80
20:11:10:WU02:FS01:Upload 3.20%
20:11:10:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
20:11:10:WU02:FS01:Trying to send results to collection server
20:11:10:WU02:FS01:Uploading 7.80MiB to 171.67.108.46
20:11:10:WU02:FS01:Connecting to 171.67.108.46:8080
20:11:22:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.45:80': 10002: Received short response, expected 272 bytes, got 0
20:11:22:WU01:FS01:Connecting to 171.64.65.35:80
20:12:07:WARNING:WU01:FS01:Failed to get assignment from '171.64.65.35:80': 10002: Received short response, expected 272 bytes, got 0
20:12:07:ERROR:WU01:FS01:Exception: Could not get an assignment
20:12:11:WU01:FS01:Connecting to 171.67.108.45:80
20:12:52:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.45:80': 10002: Received short response, expected 272 bytes, got 0
20:12:52:WU01:FS01:Connecting to 171.64.65.35:80
20:13:17:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
20:13:17:WU02:FS01:Connecting to 171.67.108.46:80
20:13:17:WU02:FS01:Upload 0.80%
20:13:37:WARNING:WU01:FS01:Failed to get assignment from '171.64.65.35:80': 10002: Received short response, expected 272 bytes, got 0
20:13:37:ERROR:WU01:FS01:Exception: Could not get an assignment
20:13:58:WU02:FS01:Upload 3.20%
20:13:58:ERROR:WU02:FS01:Exception: Transfer failed
20:13:58:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:9415 run:1680 clone:0 gen:344 core:0x21 unit:0x00000187ab436c9d585e06d8baf4a0ca
20:13:58:WU02:FS01:Uploading 7.80MiB to 171.67.108.157
20:13:58:WU02:FS01:Connecting to 171.67.108.157:8080
20:14:49:WU01:FS01:Connecting to 171.67.108.45:80
20:15:34:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.45:80': 10002: Received short response, expected 272 bytes, got 0
20:15:34:WU01:FS01:Connecting to 171.64.65.35:80
20:16:06:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
20:16:06:WU02:FS01:Connecting to 171.67.108.157:80
20:16:06:WU02:FS01:Upload 0.80%
20:16:22:WARNING:WU01:FS01:Failed to get assignment from '171.64.65.35:80': 10002: Received short response, expected 272 bytes, got 0
20:16:22:ERROR:WU01:FS01:Exception: Could not get an assignment
20:16:55:WU02:FS01:Upload 3.20%
20:16:55:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
20:16:55:WU02:FS01:Trying to send results to collection server
20:16:55:WU02:FS01:Uploading 7.80MiB to 171.67.108.46
20:16:55:WU02:FS01:Connecting to 171.67.108.46:8080
20:19:02:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
20:19:02:WU02:FS01:Connecting to 171.67.108.46:80
20:19:02:WU02:FS01:Upload 0.80%
20:19:03:WU01:FS01:Connecting to 171.67.108.45:80
20:19:44:WU02:FS01:Upload 3.20%
20:19:44:ERROR:WU02:FS01:Exception: Transfer failed
20:19:44:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:9415 run:1680 clone:0 gen:344 core:0x21 unit:0x00000187ab436c9d585e06d8baf4a0ca
20:19:44:WU02:FS01:Uploading 7.80MiB to 171.67.108.157
20:19:44:WU02:FS01:Connecting to 171.67.108.157:8080
20:19:44:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.45:80': 10002: Received short response, expected 272 bytes, got 0
20:19:44:WU01:FS01:Connecting to 171.64.65.35:80
20:20:28:WARNING:WU01:FS01:Failed to get assignment from '171.64.65.35:80': 10002: Received short response, expected 272 bytes, got 0
20:20:28:ERROR:WU01:FS01:Exception: Could not get an assignment
20:21:51:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
20:21:51:WU02:FS01:Connecting to 171.67.108.157:80
20:21:51:WU02:FS01:Upload 0.80%
20:22:40:WU02:FS01:Upload 3.20%
20:22:40:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
20:22:40:WU02:FS01:Trying to send results to collection server
20:22:40:WU02:FS01:Uploading 7.80MiB to 171.67.108.46
20:22:40:WU02:FS01:Connecting to 171.67.108.46:8080
20:24:47:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
20:24:47:WU02:FS01:Connecting to 171.67.108.46:80
20:24:47:WU02:FS01:Upload 0.80%
20:25:30:WU02:FS01:Upload 3.20%
20:25:30:ERROR:WU02:FS01:Exception: Transfer failed
20:25:30:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:9415 run:1680 clone:0 gen:344 core:0x21 unit:0x00000187ab436c9d585e06d8baf4a0ca
20:25:30:WU02:FS01:Uploading 7.80MiB to 171.67.108.157
20:25:30:WU02:FS01:Connecting to 171.67.108.157:8080
20:25:55:WU01:FS01:Connecting to 171.67.108.45:80
20:26:42:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.45:80': 10002: Received short response, expected 272 bytes, got 0
20:26:42:WU01:FS01:Connecting to 171.64.65.35:80
20:27:31:WARNING:WU01:FS01:Failed to get assignment from '171.64.65.35:80': 10002: Received short response, expected 272 bytes, got 0
20:27:31:ERROR:WU01:FS01:Exception: Could not get an assignment
20:27:37:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
20:27:37:WU02:FS01:Connecting to 171.67.108.157:80
20:27:37:WU02:FS01:Upload 0.80%
20:28:22:WU02:FS01:Upload 3.20%
20:28:22:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
20:28:22:WU02:FS01:Trying to send results to collection server
20:28:22:WU02:FS01:Uploading 7.80MiB to 171.67.108.46
20:28:22:WU02:FS01:Connecting to 171.67.108.46:8080
20:30:29:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
20:30:29:WU02:FS01:Connecting to 171.67.108.46:80
20:30:29:WU02:FS01:Upload 0.80%
20:31:16:WU02:FS01:Upload 3.20%
20:31:16:ERROR:WU02:FS01:Exception: Transfer failed
20:31:16:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:9415 run:1680 clone:0 gen:344 core:0x21 unit:0x00000187ab436c9d585e06d8baf4a0ca
20:31:16:WU02:FS01:Uploading 7.80MiB to 171.67.108.157
20:31:16:WU02:FS01:Connecting to 171.67.108.157:8080
20:33:24:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
20:33:24:WU02:FS01:Connecting to 171.67.108.157:80
20:33:24:WU02:FS01:Upload 0.80%
20:34:10:WU02:FS01:Upload 3.20%
20:34:10:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
20:34:10:WU02:FS01:Trying to send results to collection server
20:34:10:WU02:FS01:Uploading 7.80MiB to 171.67.108.46
20:34:10:WU02:FS01:Connecting to 171.67.108.46:8080
20:36:17:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
20:36:17:WU02:FS01:Connecting to 171.67.108.46:80
20:36:17:WU02:FS01:Upload 0.80%
20:36:58:WU02:FS01:Upload 3.20%
20:36:58:ERROR:WU02:FS01:Exception: Transfer failed
20:36:58:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:9415 run:1680 clone:0 gen:344 core:0x21 unit:0x00000187ab436c9d585e06d8baf4a0ca
20:36:58:WU02:FS01:Uploading 7.80MiB to 171.67.108.157
20:36:58:WU02:FS01:Connecting to 171.67.108.157:8080
20:37:00:WU01:FS01:Connecting to 171.67.108.45:80
20:37:46:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.45:80': 10002: Received short response, expected 272 bytes, got 0
20:37:46:WU01:FS01:Connecting to 171.64.65.35:80
20:38:28:WARNING:WU01:FS01:Failed to get assignment from '171.64.65.35:80': 10002: Received short response, expected 272 bytes, got 0
20:38:28:ERROR:WU01:FS01:Exception: Could not get an assignment
20:39:06:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
20:39:06:WU02:FS01:Connecting to 171.67.108.157:80
20:39:06:WU02:FS01:Upload 0.80%
20:39:46:WU02:FS01:Upload 3.20%
20:39:46:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
20:39:46:WU02:FS01:Trying to send results to collection server
20:39:46:WU02:FS01:Uploading 7.80MiB to 171.67.108.46
20:39:46:WU02:FS01:Connecting to 171.67.108.46:8080
Maybe related to the fixing procedures for the stats servers, but I thought I'd mention it.