40.114.52.201 & 128.252.203.10 - Stuck, Not Refreshing

[ToeBlister]

Windows 10. Running GPU only WUs.

I have been having the issue where the FAHClient got stuck in a infinite state where the next server poll doesn't get triggered.
It happened in either of this scenario:

Scenario 1:
FAHClient connects to either of those WS.
Attempts to pull a WU. Fails because bandwidth issues (No WUs for this config)
FAHClient supposes to poll again in some set interval. But the timer does show. The Next Attempt is Unknown.
Only way to exit this state is to kill the FAHClient and restart the app.

Scenario 2:
FAHClient connects to either of those WS.
None of the the WS gave a response.
The Next Attempt is Unknown.
Only way to exit this state is to kill the FAHClient and restart the app.

Pausing and unpausing does not work.

[toTOW]

This is probably the timeout bug but a log would be more explicit ...

Restart the client (might require to kill it with task manager) or reboot the system.

[noprob]

rebooted as suggested with server 40.114.52.201 just hung and status of server is good / ok

what to do?

*EDIT*

This is for my Xeon w5580 quad core CPU WU as stated in the FAHcontrol

also doing a GTX 660 FTW 2GB gpu on the same workstation with no issues same Work server.

https://drive.google.com/file/d/18f2Dat ... sp=sharing

*2nd EDIT*

After sometime and another reboot I now have a WU for my CPU.

https://drive.google.com/open?id=1k7ipY ... ztk0crFhoJ

[ToeBlister]

It happened again and this time I captured the log.

Code: Select all

16:07:11:WU02:FS00:FahCore 0x22 started
16:07:12:WU02:FS00:0x22:*********************** Log Started 2020-04-04T16:07:12Z ***********************
16:07:12:WU02:FS00:0x22:*************************** Core22 Folding@home Core ***************************
16:07:12:WU02:FS00:0x22:       Type: 0x22
16:07:12:WU02:FS00:0x22:       Core: Core22
16:07:12:WU02:FS00:0x22:    Website: https://foldingathome.org/
16:07:12:WU02:FS00:0x22:  Copyright: (c) 2009-2018 foldingathome.org
16:07:12:WU02:FS00:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
16:07:12:WU02:FS00:0x22:             <rafal.wiewiora@choderalab.org>
16:07:12:WU02:FS00:0x22:       Args: -dir 02 -suffix 01 -version 705 -lifeline 15320 -checkpoint 15
16:07:12:WU02:FS00:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
16:07:12:WU02:FS00:0x22:             0 -gpu 0
16:07:12:WU02:FS00:0x22:     Config: <none>
16:07:12:WU02:FS00:0x22:************************************ Build *************************************
16:07:12:WU02:FS00:0x22:    Version: 0.0.2
16:07:12:WU02:FS00:0x22:       Date: Dec 6 2019
16:07:12:WU02:FS00:0x22:       Time: 21:30:31
16:07:12:WU02:FS00:0x22: Repository: Git
16:07:12:WU02:FS00:0x22:   Revision: abeb39247cc72df5af0f63723edafadb23d5dfbe
16:07:12:WU02:FS00:0x22:     Branch: HEAD
16:07:12:WU02:FS00:0x22:   Compiler: Visual C++ 2008
16:07:12:WU02:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
16:07:12:WU02:FS00:0x22:   Platform: win32 10
16:07:12:WU02:FS00:0x22:       Bits: 64
16:07:12:WU02:FS00:0x22:       Mode: Release
16:07:12:WU02:FS00:0x22:************************************ System ************************************
16:07:12:WU02:FS00:0x22:        CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
16:07:12:WU02:FS00:0x22:     CPU ID: GenuineIntel Family 6 Model 158 Stepping 10
16:07:12:WU02:FS00:0x22:       CPUs: 12
16:07:12:WU02:FS00:0x22:     Memory: 31.86GiB
16:07:12:WU02:FS00:0x22:Free Memory: 25.71GiB
16:07:12:WU02:FS00:0x22:    Threads: WINDOWS_THREADS
16:07:12:WU02:FS00:0x22: OS Version: 6.2
16:07:12:WU02:FS00:0x22:Has Battery: true
16:07:12:WU02:FS00:0x22: On Battery: false
16:07:12:WU02:FS00:0x22: UTC Offset: 8
16:07:12:WU02:FS00:0x22:        PID: 5440
16:07:12:WU02:FS00:0x22:        CWD: C:\Users\X\AppData\Roaming\FAHClient\work
16:07:12:WU02:FS00:0x22:         OS: Windows 10 Home
16:07:12:WU02:FS00:0x22:    OS Arch: AMD64
16:07:12:WU02:FS00:0x22:********************************************************************************
16:07:12:WU02:FS00:0x22:Project: 14541 (Run 0, Clone 402, Gen 42)
16:07:12:WU02:FS00:0x22:Unit: 0x0000002e0d5262775e7d2523aa9e1515
16:07:12:WU02:FS00:0x22:Reading tar file core.xml
16:07:12:WU02:FS00:0x22:Reading tar file integrator.xml
16:07:12:WU02:FS00:0x22:Reading tar file state.xml
16:07:12:WU02:FS00:0x22:Reading tar file system.xml
16:07:12:WU02:FS00:0x22:Digital signatures verified
16:07:12:WU02:FS00:0x22:Folding@home GPU Core22 Folding@home Core
16:07:12:WU02:FS00:0x22:Version 0.0.2
16:07:17:WU01:FS00:Upload 16.86%
16:07:18:WU02:FS00:0x22:Completed 0 out of 8000000 steps (0%)
16:07:18:WU02:FS00:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
16:07:23:WU01:FS00:Upload 19.89%
16:07:29:WU01:FS00:Upload 26.30%
16:07:35:WU01:FS00:Upload 35.06%
16:07:41:WU01:FS00:Upload 43.15%
16:07:47:WU01:FS00:Upload 51.92%
16:07:53:WU01:FS00:Upload 60.68%
16:07:59:WU01:FS00:Upload 69.45%
16:08:05:WU01:FS00:Upload 77.20%
16:08:11:WU01:FS00:Upload 85.96%
16:08:17:WU01:FS00:Upload 94.73%
16:08:25:WU01:FS00:Upload complete
16:08:25:WU01:FS00:Server responded WORK_ACK (400)
16:08:25:WU01:FS00:Final credit estimate, 89795.00 points
16:08:25:WU01:FS00:Cleaning up
16:08:56:WU02:FS00:0x22:Completed 80000 out of 8000000 steps (1%)
16:10:35:WU02:FS00:0x22:Completed 160000 out of 8000000 steps (2%)
16:12:14:WU02:FS00:0x22:Completed 240000 out of 8000000 steps (3%)
16:13:53:WU02:FS00:0x22:Completed 320000 out of 8000000 steps (4%)
16:15:31:WU02:FS00:0x22:Completed 400000 out of 8000000 steps (5%)
16:17:11:WU02:FS00:0x22:Completed 480000 out of 8000000 steps (6%)
16:18:50:WU02:FS00:0x22:Completed 560000 out of 8000000 steps (7%)
16:20:29:WU02:FS00:0x22:Completed 640000 out of 8000000 steps (8%)
16:22:08:WU02:FS00:0x22:Completed 720000 out of 8000000 steps (9%)
16:23:46:WU02:FS00:0x22:Completed 800000 out of 8000000 steps (10%)
16:25:26:WU02:FS00:0x22:Completed 880000 out of 8000000 steps (11%)
16:27:05:WU02:FS00:0x22:Completed 960000 out of 8000000 steps (12%)
16:28:43:WU02:FS00:0x22:Completed 1040000 out of 8000000 steps (13%)
16:30:23:WU02:FS00:0x22:Completed 1120000 out of 8000000 steps (14%)
16:32:01:WU02:FS00:0x22:Completed 1200000 out of 8000000 steps (15%)
16:33:40:WU02:FS00:0x22:Completed 1280000 out of 8000000 steps (16%)
16:35:20:WU02:FS00:0x22:Completed 1360000 out of 8000000 steps (17%)
16:36:58:WU02:FS00:0x22:Completed 1440000 out of 8000000 steps (18%)
16:38:38:WU02:FS00:0x22:Completed 1520000 out of 8000000 steps (19%)
16:40:16:WU02:FS00:0x22:Completed 1600000 out of 8000000 steps (20%)
16:41:56:WU02:FS00:0x22:Completed 1680000 out of 8000000 steps (21%)
16:43:35:WU02:FS00:0x22:Completed 1760000 out of 8000000 steps (22%)
16:45:14:WU02:FS00:0x22:Completed 1840000 out of 8000000 steps (23%)
16:46:53:WU02:FS00:0x22:Completed 1920000 out of 8000000 steps (24%)
16:48:30:WU02:FS00:0x22:Completed 2000000 out of 8000000 steps (25%)
16:50:07:WU02:FS00:0x22:Completed 2080000 out of 8000000 steps (26%)
16:51:44:WU02:FS00:0x22:Completed 2160000 out of 8000000 steps (27%)
16:53:21:WU02:FS00:0x22:Completed 2240000 out of 8000000 steps (28%)
16:54:57:WU02:FS00:0x22:Completed 2320000 out of 8000000 steps (29%)
16:56:35:WU02:FS00:0x22:Completed 2400000 out of 8000000 steps (30%)
16:58:14:WU02:FS00:0x22:Completed 2480000 out of 8000000 steps (31%)
16:59:53:WU02:FS00:0x22:Completed 2560000 out of 8000000 steps (32%)
17:01:31:WU02:FS00:0x22:Completed 2640000 out of 8000000 steps (33%)
17:03:11:WU02:FS00:0x22:Completed 2720000 out of 8000000 steps (34%)
17:04:49:WU02:FS00:0x22:Completed 2800000 out of 8000000 steps (35%)
17:06:28:WU02:FS00:0x22:Completed 2880000 out of 8000000 steps (36%)
17:08:07:WU02:FS00:0x22:Completed 2960000 out of 8000000 steps (37%)
17:09:46:WU02:FS00:0x22:Completed 3040000 out of 8000000 steps (38%)
17:11:25:WU02:FS00:0x22:Completed 3120000 out of 8000000 steps (39%)
17:13:04:WU02:FS00:0x22:Completed 3200000 out of 8000000 steps (40%)
17:14:43:WU02:FS00:0x22:Completed 3280000 out of 8000000 steps (41%)
17:16:23:WU02:FS00:0x22:Completed 3360000 out of 8000000 steps (42%)
17:18:01:WU02:FS00:0x22:Completed 3440000 out of 8000000 steps (43%)
17:19:41:WU02:FS00:0x22:Completed 3520000 out of 8000000 steps (44%)
17:21:19:WU02:FS00:0x22:Completed 3600000 out of 8000000 steps (45%)
17:22:59:WU02:FS00:0x22:Completed 3680000 out of 8000000 steps (46%)
17:24:38:WU02:FS00:0x22:Completed 3760000 out of 8000000 steps (47%)
17:26:17:WU02:FS00:0x22:Completed 3840000 out of 8000000 steps (48%)
17:27:56:WU02:FS00:0x22:Completed 3920000 out of 8000000 steps (49%)
17:29:34:WU02:FS00:0x22:Completed 4000000 out of 8000000 steps (50%)
17:31:14:WU02:FS00:0x22:Completed 4080000 out of 8000000 steps (51%)
17:32:53:WU02:FS00:0x22:Completed 4160000 out of 8000000 steps (52%)
17:34:31:WU02:FS00:0x22:Completed 4240000 out of 8000000 steps (53%)
17:36:11:WU02:FS00:0x22:Completed 4320000 out of 8000000 steps (54%)
17:37:49:WU02:FS00:0x22:Completed 4400000 out of 8000000 steps (55%)
17:39:28:WU02:FS00:0x22:Completed 4480000 out of 8000000 steps (56%)
17:41:08:WU02:FS00:0x22:Completed 4560000 out of 8000000 steps (57%)
17:42:46:WU02:FS00:0x22:Completed 4640000 out of 8000000 steps (58%)
17:44:26:WU02:FS00:0x22:Completed 4720000 out of 8000000 steps (59%)
17:46:04:WU02:FS00:0x22:Completed 4800000 out of 8000000 steps (60%)
17:47:44:WU02:FS00:0x22:Completed 4880000 out of 8000000 steps (61%)
17:49:23:WU02:FS00:0x22:Completed 4960000 out of 8000000 steps (62%)
17:51:01:WU02:FS00:0x22:Completed 5040000 out of 8000000 steps (63%)
17:52:41:WU02:FS00:0x22:Completed 5120000 out of 8000000 steps (64%)
17:54:19:WU02:FS00:0x22:Completed 5200000 out of 8000000 steps (65%)
17:55:59:WU02:FS00:0x22:Completed 5280000 out of 8000000 steps (66%)
17:57:38:WU02:FS00:0x22:Completed 5360000 out of 8000000 steps (67%)
17:59:17:WU02:FS00:0x22:Completed 5440000 out of 8000000 steps (68%)
18:00:56:WU02:FS00:0x22:Completed 5520000 out of 8000000 steps (69%)
18:02:35:WU02:FS00:0x22:Completed 5600000 out of 8000000 steps (70%)
18:04:14:WU02:FS00:0x22:Completed 5680000 out of 8000000 steps (71%)
18:05:53:WU02:FS00:0x22:Completed 5760000 out of 8000000 steps (72%)
18:07:32:WU02:FS00:0x22:Completed 5840000 out of 8000000 steps (73%)
18:09:11:WU02:FS00:0x22:Completed 5920000 out of 8000000 steps (74%)
18:10:49:WU02:FS00:0x22:Completed 6000000 out of 8000000 steps (75%)
18:12:29:WU02:FS00:0x22:Completed 6080000 out of 8000000 steps (76%)
18:14:08:WU02:FS00:0x22:Completed 6160000 out of 8000000 steps (77%)
18:15:46:WU02:FS00:0x22:Completed 6240000 out of 8000000 steps (78%)
18:17:26:WU02:FS00:0x22:Completed 6320000 out of 8000000 steps (79%)
18:19:04:WU02:FS00:0x22:Completed 6400000 out of 8000000 steps (80%)
18:20:43:WU02:FS00:0x22:Completed 6480000 out of 8000000 steps (81%)
18:22:23:WU02:FS00:0x22:Completed 6560000 out of 8000000 steps (82%)
18:24:01:WU02:FS00:0x22:Completed 6640000 out of 8000000 steps (83%)
18:25:40:WU02:FS00:0x22:Completed 6720000 out of 8000000 steps (84%)
18:27:19:WU02:FS00:0x22:Completed 6800000 out of 8000000 steps (85%)
18:28:58:WU02:FS00:0x22:Completed 6880000 out of 8000000 steps (86%)
18:30:37:WU02:FS00:0x22:Completed 6960000 out of 8000000 steps (87%)
18:32:15:WU02:FS00:0x22:Completed 7040000 out of 8000000 steps (88%)
18:33:55:WU02:FS00:0x22:Completed 7120000 out of 8000000 steps (89%)
18:35:33:WU02:FS00:0x22:Completed 7200000 out of 8000000 steps (90%)
18:37:13:WU02:FS00:0x22:Completed 7280000 out of 8000000 steps (91%)
18:38:52:WU02:FS00:0x22:Completed 7360000 out of 8000000 steps (92%)
18:40:31:WU02:FS00:0x22:Completed 7440000 out of 8000000 steps (93%)
******************************* Date: 2020-04-04 *******************************
18:42:11:WU02:FS00:0x22:Completed 7520000 out of 8000000 steps (94%)
18:43:49:WU02:FS00:0x22:Completed 7600000 out of 8000000 steps (95%)
18:45:29:WU02:FS00:0x22:Completed 7680000 out of 8000000 steps (96%)
18:47:09:WU02:FS00:0x22:Completed 7760000 out of 8000000 steps (97%)
18:48:47:WU02:FS00:0x22:Completed 7840000 out of 8000000 steps (98%)
18:50:27:WU02:FS00:0x22:Completed 7920000 out of 8000000 steps (99%)
18:50:27:WU01:FS00:Connecting to 65.254.110.245:8080
18:50:27:WU01:FS00:Assigned to work server 128.252.203.4
18:50:27:WU01:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:TU106M [GeForce RTX 2060 Mobile] from 128.252.203.4
18:50:27:WU01:FS00:Connecting to 128.252.203.4:8080
18:50:28:WU01:FS00:Downloading 38.41MiB
18:50:34:WU01:FS00:Download 15.13%
18:50:40:WU01:FS00:Download 37.58%
18:50:46:WU01:FS00:Download 59.22%
18:50:52:WU01:FS00:Download 80.05%
18:50:57:WU01:FS00:Download complete
18:50:57:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14543 run:0 clone:50 gen:20 core:0x22 unit:0x0000001980fccb045e7fc1c1d51ebc5f
18:52:05:WU02:FS00:0x22:Completed 8000000 out of 8000000 steps (100%)
18:52:07:WU02:FS00:0x22:Saving result file ..\logfile_01.txt
18:52:07:WU02:FS00:0x22:Saving result file checkpointState.xml
18:52:07:WU02:FS00:0x22:Saving result file checkpt.crc
18:52:07:WU02:FS00:0x22:Saving result file positions.xtc
18:52:07:WU02:FS00:0x22:Saving result file science.log
18:52:07:WU02:FS00:0x22:Folding@home Core Shutdown: FINISHED_UNIT
18:52:07:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
18:52:07:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14541 run:0 clone:402 gen:42 core:0x22 unit:0x0000002e0d5262775e7d2523aa9e1515
18:52:07:WU02:FS00:Uploading 18.54MiB to 13.82.98.119
18:52:07:WU02:FS00:Connecting to 13.82.98.119:8080
18:52:07:WU01:FS00:Starting
18:52:07:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" "C:\Users\X\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe" -dir 01 -suffix 01 -version 705 -lifeline 18744 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
18:52:07:WU01:FS00:Started FahCore on PID 12992
18:52:07:WU01:FS00:Core PID:21556
18:52:07:WU01:FS00:FahCore 0x22 started
18:52:08:WU01:FS00:0x22:*********************** Log Started 2020-04-04T18:52:08Z ***********************
18:52:08:WU01:FS00:0x22:*************************** Core22 Folding@home Core ***************************
18:52:08:WU01:FS00:0x22:       Type: 0x22
18:52:08:WU01:FS00:0x22:       Core: Core22
18:52:08:WU01:FS00:0x22:    Website: https://foldingathome.org/
18:52:08:WU01:FS00:0x22:  Copyright: (c) 2009-2018 foldingathome.org
18:52:08:WU01:FS00:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
18:52:08:WU01:FS00:0x22:             <rafal.wiewiora@choderalab.org>
18:52:08:WU01:FS00:0x22:       Args: -dir 01 -suffix 01 -version 705 -lifeline 12992 -checkpoint 15
18:52:08:WU01:FS00:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
18:52:08:WU01:FS00:0x22:             0 -gpu 0
18:52:08:WU01:FS00:0x22:     Config: <none>
18:52:08:WU01:FS00:0x22:************************************ Build *************************************
18:52:08:WU01:FS00:0x22:    Version: 0.0.2
18:52:08:WU01:FS00:0x22:       Date: Dec 6 2019
18:52:08:WU01:FS00:0x22:       Time: 21:30:31
18:52:08:WU01:FS00:0x22: Repository: Git
18:52:08:WU01:FS00:0x22:   Revision: abeb39247cc72df5af0f63723edafadb23d5dfbe
18:52:08:WU01:FS00:0x22:     Branch: HEAD
18:52:08:WU01:FS00:0x22:   Compiler: Visual C++ 2008
18:52:08:WU01:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
18:52:08:WU01:FS00:0x22:   Platform: win32 10
18:52:08:WU01:FS00:0x22:       Bits: 64
18:52:08:WU01:FS00:0x22:       Mode: Release
18:52:08:WU01:FS00:0x22:************************************ System ************************************
18:52:08:WU01:FS00:0x22:        CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
18:52:08:WU01:FS00:0x22:     CPU ID: GenuineIntel Family 6 Model 158 Stepping 10
18:52:08:WU01:FS00:0x22:       CPUs: 12
18:52:08:WU01:FS00:0x22:     Memory: 31.86GiB
18:52:08:WU01:FS00:0x22:Free Memory: 25.72GiB
18:52:08:WU01:FS00:0x22:    Threads: WINDOWS_THREADS
18:52:08:WU01:FS00:0x22: OS Version: 6.2
18:52:08:WU01:FS00:0x22:Has Battery: true
18:52:08:WU01:FS00:0x22: On Battery: false
18:52:08:WU01:FS00:0x22: UTC Offset: 8
18:52:08:WU01:FS00:0x22:        PID: 21556
18:52:08:WU01:FS00:0x22:        CWD: C:\Users\X\AppData\Roaming\FAHClient\work
18:52:08:WU01:FS00:0x22:         OS: Windows 10 Home
18:52:08:WU01:FS00:0x22:    OS Arch: AMD64
18:52:08:WU01:FS00:0x22:********************************************************************************
18:52:08:WU01:FS00:0x22:Project: 14543 (Run 0, Clone 50, Gen 20)
18:52:08:WU01:FS00:0x22:Unit: 0x0000001980fccb045e7fc1c1d51ebc5f
18:52:08:WU01:FS00:0x22:Reading tar file core.xml
18:52:08:WU01:FS00:0x22:Reading tar file integrator.xml
18:52:08:WU01:FS00:0x22:Reading tar file state.xml
18:52:08:WU01:FS00:0x22:Reading tar file system.xml
18:52:09:WU01:FS00:0x22:Digital signatures verified
18:52:09:WU01:FS00:0x22:Folding@home GPU Core22 Folding@home Core
18:52:09:WU01:FS00:0x22:Version 0.0.2
18:52:13:WU02:FS00:Upload 13.82%
18:52:19:WU02:FS00:Upload 21.24%
18:52:22:WU01:FS00:0x22:Completed 0 out of 4000000 steps (0%)
18:52:23:WU01:FS00:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
18:52:25:WU02:FS00:Upload 29.33%
18:52:31:WU02:FS00:Upload 37.76%
18:52:37:WU02:FS00:Upload 46.53%
18:52:43:WU02:FS00:Upload 55.63%
18:52:49:WU02:FS00:Upload 64.06%
18:52:55:WU02:FS00:Upload 72.15%
18:53:01:WU02:FS00:Upload 80.92%
18:53:07:WU02:FS00:Upload 89.35%
18:53:13:WU02:FS00:Upload 98.45%
18:53:19:WU02:FS00:Upload complete
18:53:19:WU02:FS00:Server responded WORK_ACK (400)
18:53:19:WU02:FS00:Final credit estimate, 89290.00 points
18:53:19:WU02:FS00:Cleaning up
18:54:55:WU01:FS00:0x22:Completed 40000 out of 4000000 steps (1%)
18:57:32:WU01:FS00:0x22:Completed 80000 out of 4000000 steps (2%)
19:00:08:WU01:FS00:0x22:Completed 120000 out of 4000000 steps (3%)
19:02:44:WU01:FS00:0x22:Completed 160000 out of 4000000 steps (4%)
19:05:17:WU01:FS00:0x22:Completed 200000 out of 4000000 steps (5%)
19:07:53:WU01:FS00:0x22:Completed 240000 out of 4000000 steps (6%)
19:10:29:WU01:FS00:0x22:Completed 280000 out of 4000000 steps (7%)
19:13:06:WU01:FS00:0x22:Completed 320000 out of 4000000 steps (8%)
19:15:42:WU01:FS00:0x22:Completed 360000 out of 4000000 steps (9%)
19:18:15:WU01:FS00:0x22:Completed 400000 out of 4000000 steps (10%)
19:20:51:WU01:FS00:0x22:Completed 440000 out of 4000000 steps (11%)
19:23:27:WU01:FS00:0x22:Completed 480000 out of 4000000 steps (12%)
19:26:04:WU01:FS00:0x22:Completed 520000 out of 4000000 steps (13%)
19:28:40:WU01:FS00:0x22:Completed 560000 out of 4000000 steps (14%)
19:31:13:WU01:FS00:0x22:Completed 600000 out of 4000000 steps (15%)
19:33:49:WU01:FS00:0x22:Completed 640000 out of 4000000 steps (16%)
19:36:26:WU01:FS00:0x22:Completed 680000 out of 4000000 steps (17%)
19:39:02:WU01:FS00:0x22:Completed 720000 out of 4000000 steps (18%)
19:41:38:WU01:FS00:0x22:Completed 760000 out of 4000000 steps (19%)
19:44:11:WU01:FS00:0x22:Completed 800000 out of 4000000 steps (20%)
19:46:47:WU01:FS00:0x22:Completed 840000 out of 4000000 steps (21%)
19:49:24:WU01:FS00:0x22:Completed 880000 out of 4000000 steps (22%)
19:52:00:WU01:FS00:0x22:Completed 920000 out of 4000000 steps (23%)
19:54:37:WU01:FS00:0x22:Completed 960000 out of 4000000 steps (24%)
19:57:09:WU01:FS00:0x22:Completed 1000000 out of 4000000 steps (25%)
19:59:46:WU01:FS00:0x22:Completed 1040000 out of 4000000 steps (26%)
20:02:22:WU01:FS00:0x22:Completed 1080000 out of 4000000 steps (27%)
20:04:58:WU01:FS00:0x22:Completed 1120000 out of 4000000 steps (28%)
20:07:35:WU01:FS00:0x22:Completed 1160000 out of 4000000 steps (29%)
20:10:08:WU01:FS00:0x22:Completed 1200000 out of 4000000 steps (30%)
20:12:44:WU01:FS00:0x22:Completed 1240000 out of 4000000 steps (31%)
20:15:20:WU01:FS00:0x22:Completed 1280000 out of 4000000 steps (32%)
20:17:57:WU01:FS00:0x22:Completed 1320000 out of 4000000 steps (33%)
20:20:33:WU01:FS00:0x22:Completed 1360000 out of 4000000 steps (34%)
20:23:06:WU01:FS00:0x22:Completed 1400000 out of 4000000 steps (35%)
20:25:43:WU01:FS00:0x22:Completed 1440000 out of 4000000 steps (36%)
20:28:20:WU01:FS00:0x22:Completed 1480000 out of 4000000 steps (37%)
20:30:56:WU01:FS00:0x22:Completed 1520000 out of 4000000 steps (38%)
20:33:32:WU01:FS00:0x22:Completed 1560000 out of 4000000 steps (39%)
20:36:05:WU01:FS00:0x22:Completed 1600000 out of 4000000 steps (40%)
20:38:42:WU01:FS00:0x22:Completed 1640000 out of 4000000 steps (41%)
20:41:18:WU01:FS00:0x22:Completed 1680000 out of 4000000 steps (42%)
20:43:55:WU01:FS00:0x22:Completed 1720000 out of 4000000 steps (43%)
20:46:31:WU01:FS00:0x22:Completed 1760000 out of 4000000 steps (44%)
20:49:03:WU01:FS00:0x22:Completed 1800000 out of 4000000 steps (45%)
20:51:39:WU01:FS00:0x22:Completed 1840000 out of 4000000 steps (46%)
20:54:16:WU01:FS00:0x22:Completed 1880000 out of 4000000 steps (47%)
20:56:52:WU01:FS00:0x22:Completed 1920000 out of 4000000 steps (48%)
20:59:28:WU01:FS00:0x22:Completed 1960000 out of 4000000 steps (49%)
21:02:00:WU01:FS00:0x22:Completed 2000000 out of 4000000 steps (50%)
21:04:35:WU01:FS00:0x22:Completed 2040000 out of 4000000 steps (51%)
21:07:11:WU01:FS00:0x22:Completed 2080000 out of 4000000 steps (52%)
21:09:47:WU01:FS00:0x22:Completed 2120000 out of 4000000 steps (53%)
21:12:23:WU01:FS00:0x22:Completed 2160000 out of 4000000 steps (54%)
21:14:55:WU01:FS00:0x22:Completed 2200000 out of 4000000 steps (55%)
21:17:31:WU01:FS00:0x22:Completed 2240000 out of 4000000 steps (56%)
21:20:07:WU01:FS00:0x22:Completed 2280000 out of 4000000 steps (57%)
21:22:43:WU01:FS00:0x22:Completed 2320000 out of 4000000 steps (58%)
21:25:18:WU01:FS00:0x22:Completed 2360000 out of 4000000 steps (59%)
21:27:51:WU01:FS00:0x22:Completed 2400000 out of 4000000 steps (60%)
21:30:26:WU01:FS00:0x22:Completed 2440000 out of 4000000 steps (61%)
21:33:02:WU01:FS00:0x22:Completed 2480000 out of 4000000 steps (62%)
21:35:38:WU01:FS00:0x22:Completed 2520000 out of 4000000 steps (63%)
21:38:14:WU01:FS00:0x22:Completed 2560000 out of 4000000 steps (64%)
21:40:47:WU01:FS00:0x22:Completed 2600000 out of 4000000 steps (65%)
21:43:23:WU01:FS00:0x22:Completed 2640000 out of 4000000 steps (66%)
21:45:59:WU01:FS00:0x22:Completed 2680000 out of 4000000 steps (67%)
21:48:35:WU01:FS00:0x22:Completed 2720000 out of 4000000 steps (68%)
21:51:11:WU01:FS00:0x22:Completed 2760000 out of 4000000 steps (69%)
21:53:43:WU01:FS00:0x22:Completed 2800000 out of 4000000 steps (70%)
21:56:19:WU01:FS00:0x22:Completed 2840000 out of 4000000 steps (71%)
21:58:55:WU01:FS00:0x22:Completed 2880000 out of 4000000 steps (72%)
22:01:31:WU01:FS00:0x22:Completed 2920000 out of 4000000 steps (73%)
22:04:07:WU01:FS00:0x22:Completed 2960000 out of 4000000 steps (74%)
22:06:40:WU01:FS00:0x22:Completed 3000000 out of 4000000 steps (75%)
22:09:16:WU01:FS00:0x22:Completed 3040000 out of 4000000 steps (76%)
22:11:52:WU01:FS00:0x22:Completed 3080000 out of 4000000 steps (77%)
22:14:28:WU01:FS00:0x22:Completed 3120000 out of 4000000 steps (78%)
22:17:04:WU01:FS00:0x22:Completed 3160000 out of 4000000 steps (79%)
22:19:36:WU01:FS00:0x22:Completed 3200000 out of 4000000 steps (80%)
22:22:12:WU01:FS00:0x22:Completed 3240000 out of 4000000 steps (81%)
22:24:49:WU01:FS00:0x22:Completed 3280000 out of 4000000 steps (82%)
22:27:25:WU01:FS00:0x22:Completed 3320000 out of 4000000 steps (83%)
22:30:00:WU01:FS00:0x22:Completed 3360000 out of 4000000 steps (84%)
22:32:33:WU01:FS00:0x22:Completed 3400000 out of 4000000 steps (85%)
22:35:08:WU01:FS00:0x22:Completed 3440000 out of 4000000 steps (86%)
22:37:44:WU01:FS00:0x22:Completed 3480000 out of 4000000 steps (87%)
22:40:20:WU01:FS00:0x22:Completed 3520000 out of 4000000 steps (88%)
22:42:56:WU01:FS00:0x22:Completed 3560000 out of 4000000 steps (89%)
22:45:28:WU01:FS00:0x22:Completed 3600000 out of 4000000 steps (90%)
22:48:04:WU01:FS00:0x22:Completed 3640000 out of 4000000 steps (91%)
22:50:41:WU01:FS00:0x22:Completed 3680000 out of 4000000 steps (92%)
22:53:17:WU01:FS00:0x22:Completed 3720000 out of 4000000 steps (93%)
22:55:53:WU01:FS00:0x22:Completed 3760000 out of 4000000 steps (94%)
22:58:25:WU01:FS00:0x22:Completed 3800000 out of 4000000 steps (95%)
23:01:01:WU01:FS00:0x22:Completed 3840000 out of 4000000 steps (96%)
23:03:38:WU01:FS00:0x22:Completed 3880000 out of 4000000 steps (97%)
23:06:14:WU01:FS00:0x22:Completed 3920000 out of 4000000 steps (98%)
23:08:50:WU01:FS00:0x22:Completed 3960000 out of 4000000 steps (99%)
23:08:51:WU00:FS00:Connecting to 65.254.110.245:8080
23:08:51:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
23:08:51:WU00:FS00:Connecting to 18.218.241.186:80
23:08:52:WU00:FS00:Assigned to work server 40.114.52.201
23:08:52:WU00:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:TU106M [GeForce RTX 2060 Mobile] from 40.114.52.201
23:08:52:WU00:FS00:Connecting to 40.114.52.201:8080
23:11:23:WU01:FS00:0x22:Completed 4000000 out of 4000000 steps (100%)
23:11:26:WU01:FS00:0x22:Saving result file ..\logfile_01.txt
23:11:26:WU01:FS00:0x22:Saving result file checkpointState.xml
23:11:26:WU01:FS00:0x22:Saving result file checkpt.crc
23:11:26:WU01:FS00:0x22:Saving result file positions.xtc
23:11:27:WU01:FS00:0x22:Saving result file science.log
23:11:27:WU01:FS00:0x22:Folding@home Core Shutdown: FINISHED_UNIT
23:11:27:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
23:11:27:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14543 run:0 clone:50 gen:20 core:0x22 unit:0x0000001980fccb045e7fc1c1d51ebc5f
23:11:27:WU01:FS00:Uploading 39.21MiB to 128.252.203.4
23:11:27:WU01:FS00:Connecting to 128.252.203.4:8080
23:11:33:WU01:FS00:Upload 12.27%
23:11:39:WU01:FS00:Upload 24.23%
23:11:45:WU01:FS00:Upload 36.34%
23:11:51:WU01:FS00:Upload 47.97%
23:11:57:WU01:FS00:Upload 59.77%
23:12:03:WU01:FS00:Upload 71.72%
23:12:09:WU01:FS00:Upload 83.36%
23:12:15:WU01:FS00:Upload 95.15%
23:12:19:WU01:FS00:Upload complete
23:12:19:WU01:FS00:Server responded WORK_ACK (400)
23:12:19:WU01:FS00:Final credit estimate, 146329.00 points
23:12:19:WU01:FS00:Cleaning up
00:17:37:FS00:Paused
00:17:43:FS00:Unpaused

[Rainmaker]

I also have had a completed WU since yesterday, still unable to upload to 40.114.52.201. I've rebooted several times. Log excerpt:

Code: Select all

11:13:33:WU02:FS01:Starting
11:13:33:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\rainm\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 705 -lifeline 11872 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
11:13:33:WU02:FS01:Started FahCore on PID 9276
11:13:33:WU02:FS01:Core PID:3080
11:13:33:WU02:FS01:FahCore 0x22 started
11:13:33:WU02:FS01:0x22:*********************** Log Started 2020-04-09T11:13:33Z ***********************
11:13:33:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
11:13:33:WU02:FS01:0x22:       Type: 0x22
11:13:33:WU02:FS01:0x22:       Core: Core22
11:13:33:WU02:FS01:0x22:    Website: https://foldingathome.org/
11:13:33:WU02:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
11:13:33:WU02:FS01:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
11:13:33:WU02:FS01:0x22:             <rafal.wiewiora@choderalab.org>
11:13:33:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 705 -lifeline 9276 -checkpoint 15
11:13:33:WU02:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
11:13:33:WU02:FS01:0x22:     Config: <none>
11:13:33:WU02:FS01:0x22:************************************ Build *************************************
11:13:33:WU02:FS01:0x22:    Version: 0.0.2
11:13:33:WU02:FS01:0x22:       Date: Dec 6 2019
11:13:33:WU02:FS01:0x22:       Time: 21:30:31
11:13:33:WU02:FS01:0x22: Repository: Git
11:13:33:WU02:FS01:0x22:   Revision: abeb39247cc72df5af0f63723edafadb23d5dfbe
11:13:33:WU02:FS01:0x22:     Branch: HEAD
11:13:33:WU02:FS01:0x22:   Compiler: Visual C++ 2008
11:13:33:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
11:13:33:WU02:FS01:0x22:   Platform: win32 10
11:13:33:WU02:FS01:0x22:       Bits: 64
11:13:33:WU02:FS01:0x22:       Mode: Release
11:13:33:WU02:FS01:0x22:************************************ System ************************************
11:13:33:WU02:FS01:0x22:        CPU: AMD Ryzen Threadripper 3960X 24-Core Processor
11:13:33:WU02:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 49 Stepping 0
11:13:33:WU02:FS01:0x22:       CPUs: 48
11:13:33:WU02:FS01:0x22:     Memory: 31.88GiB
11:13:33:WU02:FS01:0x22:Free Memory: 28.64GiB
11:13:33:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
11:13:33:WU02:FS01:0x22: OS Version: 6.2
11:13:33:WU02:FS01:0x22:Has Battery: false
11:13:33:WU02:FS01:0x22: On Battery: false
11:13:33:WU02:FS01:0x22: UTC Offset: 1
11:13:33:WU02:FS01:0x22:        PID: 3080
11:13:33:WU02:FS01:0x22:        CWD: C:\Users\rainm\AppData\Roaming\FAHClient\work
11:13:33:WU02:FS01:0x22:         OS: Windows 10 Pro for Workstations
11:13:33:WU02:FS01:0x22:    OS Arch: AMD64
11:13:33:WU02:FS01:0x22:********************************************************************************
11:13:33:WU02:FS01:0x22:Project: 11777 (Run 0, Clone 30605, Gen 15)
11:13:33:WU02:FS01:0x22:Unit: 0x0000001f287234c95e7930b514528715
11:13:33:WU02:FS01:0x22:Reading tar file core.xml
11:13:33:WU02:FS01:0x22:Reading tar file integrator.xml
11:13:33:WU02:FS01:0x22:Reading tar file state.xml
11:13:33:WU02:FS01:0x22:Reading tar file system.xml
11:13:33:WU02:FS01:0x22:Digital signatures verified
11:13:33:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
11:13:33:WU02:FS01:0x22:Version 0.0.2
11:13:41:WU02:FS01:0x22:Completed 0 out of 2000000 steps (0%)
11:13:41:WU02:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
11:13:54:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
11:13:54:WU00:FS01:Connecting to 40.114.52.201:80
11:14:15:WARNING:WU00:FS01:Exception: Failed to send results to work server: Failed to connect to 40.114.52.201:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

[rwh202]

Not seen too many posts on 40.114.52.201 but it's being particularly troublesome for me with multiple failed uploads.

On some machines it will build up a backlog of 6 failed to send (hung at 99 odd percent) such that other client connections will no longer be possible.

A restart will often see some manage to upload - especially if they try and upload before the network is initialised, so fall back to the collection server which works more consistently.

Here's the log showing a full recovery after a restart, although the client is still erroneously trying to get an assignment - it hasn't recognised that the previous one was successful and both slots are currently in work:

Code: Select all

*********************** Log Started 2020-04-09T18:01:44Z ***********************
18:01:44:************************* Folding@home Client *************************
18:01:44:    Website: http://folding.stanford.edu/
18:01:44:  Copyright: (c) 2009-2014 Stanford University
18:01:44:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:01:44:       Args: --child --lifeline 1177 /etc/fahclient/config.xml --run-as
18:01:44:             fahclient --pid-file=/var/run/fahclient.pid --daemon
18:01:44:     Config: /etc/fahclient/config.xml
18:01:44:******************************** Build ********************************
18:01:44:    Version: 7.4.4
18:01:44:       Date: Mar 4 2014
18:01:44:       Time: 12:02:38
18:01:44:    SVN Rev: 4130
18:01:44:     Branch: fah/trunk/client
18:01:44:   Compiler: GNU 4.4.7
18:01:44:    Options: -std=gnu++98 -O3 -funroll-loops -mfpmath=sse -ffast-math
18:01:44:             -fno-unsafe-math-optimizations -msse2
18:01:44:   Platform: linux2 3.2.0-1-amd64
18:01:44:       Bits: 64
18:01:44:       Mode: Release
18:01:44:******************************* System ********************************
18:01:44:        CPU: Intel(R) Pentium(R) CPU G2020 @ 2.90GHz
18:01:44:     CPU ID: GenuineIntel Family 6 Model 58 Stepping 9
18:01:44:       CPUs: 2
18:01:44:     Memory: 7.75GiB
18:01:44:Free Memory: 7.28GiB
18:01:44:    Threads: POSIX_THREADS
18:01:44: OS Version: 3.19
18:01:44:Has Battery: false
18:01:44: On Battery: false
18:01:44: UTC Offset: 1
18:01:44:        PID: 1179
18:01:44:        CWD: /var/lib/fahclient
18:01:44:         OS: Linux 3.19.0-32-generic x86_64
18:01:44:    OS Arch: AMD64
18:01:44:       GPUs: 2
18:01:44:      GPU 0: NVIDIA:8 GP102 [TITAN Xp] 12150
18:01:44:      GPU 1: NVIDIA:8 GP102 [TITAN Xp] 12150
18:01:44:       CUDA: 6.1
18:01:44:CUDA Driver: 9010
18:01:44:***********************************************************************
18:01:44:<config>
18:01:44:  <!-- Client Control -->
18:01:44:  <fold-anon v='true'/>
18:01:44:
18:01:44:  <!-- Folding Slot Configuration -->
18:01:44:  <client-type v='beta'/>
18:01:44:
18:01:44:  <!-- HTTP Server -->
18:01:44:  <allow v='127.0.0.1 192.168.1.0/24'/>
18:01:44:
18:01:44:  <!-- Network -->
18:01:44:  <proxy v=':8080'/>
18:01:44:
18:01:44:  <!-- Remote Command Server -->
18:01:44:  <password v='******'/>
18:01:44:
18:01:44:  <!-- Slot Control -->
18:01:44:  <power v='full'/>
18:01:44:
18:01:44:  <!-- User Information -->
18:01:44:  <passkey v='********************************'/>
18:01:44:  <team v='224497'/>
18:01:44:  <user v='TML_ALL_1Fb7XpTZN9gefApL87L51CY34aT7DZKCUu'/>
18:01:44:
18:01:44:  <!-- Work Unit Control -->
18:01:44:  <next-unit-percentage v='100'/>
18:01:44:
18:01:44:  <!-- Folding Slots -->
18:01:44:  <slot id='1' type='GPU'/>
18:01:44:  <slot id='2' type='GPU'/>
18:01:44:</config>
18:01:44:Switching to user fahclient
18:01:44:Trying to access database...
18:01:44:Successfully acquired database lock
18:01:44:Enabled folding slot 01: READY gpu:0:GP102 [TITAN Xp] 12150
18:01:44:Enabled folding slot 02: READY gpu:1:GP102 [TITAN Xp] 12150
18:01:44:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11778 run:0 clone:6876 gen:19 core:0x22 unit:0x00000026287234c95e743262966f272e
18:01:44:WU00:FS01:Uploading 23.06MiB to 40.114.52.201
18:01:44:WU00:FS01:Connecting to 40.114.52.201:8080
\x1b[93m18:01:44:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU00:FS01:Connecting to 40.114.52.201:80
\x1b[93m18:01:44:WARNING:WU00:FS01:Exception: Failed to send results to work server: Failed to connect to 40.114.52.201:80: Network is unreachable\x1b[0m
18:01:44:WU00:FS01:Trying to send results to collection server
18:01:44:WU00:FS01:Uploading 23.06MiB to 52.224.109.74
18:01:44:WU00:FS01:Connecting to 52.224.109.74:8080
\x1b[93m18:01:44:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU00:FS01:Connecting to 52.224.109.74:80
\x1b[91m18:01:44:ERROR:WU00:FS01:Exception: Failed to connect to 52.224.109.74:80: Network is unreachable\x1b[0m
18:01:44:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:11777 run:0 clone:33961 gen:0 core:0x22 unit:0x00000001287234c95e7930550917b66c
18:01:44:WU03:FS02:Uploading 23.06MiB to 40.114.52.201
18:01:44:WU03:FS02:Connecting to 40.114.52.201:8080
\x1b[93m18:01:44:WARNING:WU03:FS02:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU03:FS02:Connecting to 40.114.52.201:80
\x1b[93m18:01:44:WARNING:WU03:FS02:Exception: Failed to send results to work server: Failed to connect to 40.114.52.201:80: Network is unreachable\x1b[0m
18:01:44:WU03:FS02:Trying to send results to collection server
18:01:44:WU03:FS02:Uploading 23.06MiB to 52.224.109.74
18:01:44:WU03:FS02:Connecting to 52.224.109.74:8080
\x1b[93m18:01:44:WARNING:WU03:FS02:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU03:FS02:Connecting to 52.224.109.74:80
\x1b[91m18:01:44:ERROR:WU03:FS02:Exception: Failed to connect to 52.224.109.74:80: Network is unreachable\x1b[0m
18:01:44:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11759 run:0 clone:1428 gen:31 core:0x22 unit:0x0000004080fccb0a5e6d7c8eb8da243b
18:01:44:WU01:FS01:Uploading 49.93MiB to 128.252.203.10
18:01:44:WU01:FS01:Connecting to 128.252.203.10:8080
\x1b[93m18:01:44:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU01:FS01:Connecting to 128.252.203.10:80
\x1b[93m18:01:44:WARNING:WU01:FS01:Exception: Failed to send results to work server: Failed to connect to 128.252.203.10:80: Network is unreachable\x1b[0m
18:01:44:WU01:FS01:Trying to send results to collection server
18:01:44:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11778 run:0 clone:6876 gen:19 core:0x22 unit:0x00000026287234c95e743262966f272e
18:01:44:WU01:FS01:Uploading 49.93MiB to 52.224.109.74
18:01:44:WU01:FS01:Connecting to 52.224.109.74:8080
\x1b[93m18:01:44:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU01:FS01:Connecting to 52.224.109.74:80
18:01:44:WU00:FS01:Uploading 23.06MiB to 40.114.52.201
18:01:44:WU00:FS01:Connecting to 40.114.52.201:8080
\x1b[93m18:01:44:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU00:FS01:Connecting to 40.114.52.201:80
\x1b[93m18:01:44:WARNING:WU00:FS01:Exception: Failed to send results to work server: Failed to connect to 40.114.52.201:80: Network is unreachable\x1b[0m
18:01:44:WU00:FS01:Trying to send results to collection server
18:01:44:WU00:FS01:Uploading 23.06MiB to 52.224.109.74
18:01:44:WU00:FS01:Connecting to 52.224.109.74:8080
\x1b[93m18:01:44:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU00:FS01:Connecting to 52.224.109.74:80
\x1b[91m18:01:44:ERROR:WU00:FS01:Exception: Failed to connect to 52.224.109.74:80: Network is unreachable\x1b[0m
18:01:44:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:11777 run:0 clone:33961 gen:0 core:0x22 unit:0x00000001287234c95e7930550917b66c
18:01:44:WU03:FS02:Uploading 23.06MiB to 40.114.52.201
18:01:44:WU03:FS02:Connecting to 40.114.52.201:8080
\x1b[93m18:01:44:WARNING:WU03:FS02:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU03:FS02:Connecting to 40.114.52.201:80
\x1b[93m18:01:44:WARNING:WU03:FS02:Exception: Failed to send results to work server: Failed to connect to 40.114.52.201:80: Network is unreachable\x1b[0m
18:01:44:WU03:FS02:Trying to send results to collection server
18:01:44:WU03:FS02:Uploading 23.06MiB to 52.224.109.74
18:01:44:WU03:FS02:Connecting to 52.224.109.74:8080
\x1b[93m18:01:44:WARNING:WU03:FS02:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU03:FS02:Connecting to 52.224.109.74:80
\x1b[91m18:01:44:ERROR:WU03:FS02:Exception: Failed to connect to 52.224.109.74:80: Network is unreachable\x1b[0m
18:01:44:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:11781 run:0 clone:4356 gen:28 core:0x22 unit:0x0000002d0d5a98395e73c4fe33e70b0a
18:01:44:WU04:FS01:Uploading 55.24MiB to 13.90.152.57
18:01:44:WU04:FS01:Connecting to 13.90.152.57:8080
\x1b[93m18:01:44:WARNING:WU04:FS01:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU04:FS01:Connecting to 13.90.152.57:80
\x1b[93m18:01:44:WARNING:WU04:FS01:Exception: Failed to send results to work server: Failed to connect to 13.90.152.57:80: Network is unreachable\x1b[0m
18:01:44:WU04:FS01:Trying to send results to collection server
18:01:44:WU02:FS02:Sending unit results: id:02 state:SEND error:NO_ERROR project:13877 run:0 clone:690 gen:14 core:0x22 unit:0x0000001234e06d4a5e80cfed3832a723
18:01:44:WU04:FS01:Uploading 55.24MiB to 52.224.109.74
18:01:44:WU04:FS01:Connecting to 52.224.109.74:8080
\x1b[93m18:01:44:WARNING:WU04:FS01:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU04:FS01:Connecting to 52.224.109.74:80
18:01:44:WU02:FS02:Uploading 48.11MiB to 52.224.109.74
18:01:44:WU02:FS02:Connecting to 52.224.109.74:8080
\x1b[93m18:01:44:WARNING:WU02:FS02:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU02:FS02:Connecting to 52.224.109.74:80
\x1b[93m18:01:44:WARNING:WU02:FS02:Exception: Failed to send results to work server: Failed to connect to 52.224.109.74:80: Network is unreachable\x1b[0m
18:01:44:WU07:FS01:Starting
18:01:44:WU07:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/beta/Core_22.fah/FahCore_22 -dir 07 -suffix 01 -version 704 -lifeline 1179 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
18:01:44:WU07:FS01:Started FahCore on PID 1189
18:01:44:WU07:FS01:Core PID:1193
18:01:44:WU07:FS01:FahCore 0x22 started
18:01:44:WU06:FS02:Starting
18:01:44:WU06:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/beta/Core_22.fah/FahCore_22 -dir 06 -suffix 01 -version 704 -lifeline 1179 -checkpoint 15 -gpu 1 -gpu-vendor nvidia
18:01:44:WU06:FS02:Started FahCore on PID 1355
18:01:44:WU06:FS02:Core PID:1359
18:01:44:WU06:FS02:FahCore 0x22 started
18:01:44:WU05:FS02:Sending unit results: id:05 state:SEND error:NO_ERROR project:11777 run:0 clone:19211 gen:16 core:0x22 unit:0x0000001c287234c95e774a5b994238ac
18:01:44:WU05:FS02:Uploading 23.06MiB to 40.114.52.201
18:01:44:WU05:FS02:Connecting to 40.114.52.201:8080
\x1b[93m18:01:44:WARNING:WU05:FS02:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU05:FS02:Connecting to 40.114.52.201:80
\x1b[93m18:01:44:WARNING:WU05:FS02:Exception: Failed to send results to work server: Failed to connect to 40.114.52.201:80: Network is unreachable\x1b[0m
18:01:44:WU05:FS02:Trying to send results to collection server
18:01:44:WU05:FS02:Uploading 23.06MiB to 52.224.109.74
18:01:44:WU05:FS02:Connecting to 52.224.109.74:8080
\x1b[93m18:01:44:WARNING:WU05:FS02:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU05:FS02:Connecting to 52.224.109.74:80
\x1b[91m18:01:44:ERROR:WU05:FS02:Exception: Failed to connect to 52.224.109.74:80: Network is unreachable\x1b[0m
18:01:44:WU07:FS01:0x22:*********************** Log Started 2020-04-09T18:01:44Z ***********************
18:01:44:WU07:FS01:0x22:*************************** Core22 Folding@home Core ***************************
18:01:44:WU07:FS01:0x22:       Type: 0x22
18:01:44:WU07:FS01:0x22:       Core: Core22
18:01:44:WU07:FS01:0x22:    Website: https://foldingathome.org/
18:01:44:WU07:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
18:01:44:WU07:FS01:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
18:01:44:WU07:FS01:0x22:             <rafal.wiewiora@choderalab.org>
18:01:44:WU07:FS01:0x22:       Args: -dir 07 -suffix 01 -version 704 -lifeline 1189 -checkpoint 15 -gpu
18:01:44:WU07:FS01:0x22:             0 -gpu-vendor nvidia
18:01:44:WU07:FS01:0x22:     Config: <none>
18:01:44:WU07:FS01:0x22:************************************ Build *************************************
18:01:44:WU07:FS01:0x22:    Version: 0.0.2
18:01:44:WU07:FS01:0x22:       Date: Dec 6 2019
18:01:44:WU07:FS01:0x22:       Time: 21:20:17
18:01:44:WU07:FS01:0x22: Repository: Git
18:01:44:WU07:FS01:0x22:   Revision: f87d92b58abdf7e6bf2e173cfbc4dc3e837c7042
18:01:44:WU07:FS01:0x22:     Branch: core22
18:01:44:WU07:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
18:01:44:WU07:FS01:0x22:    Options: -std=gnu++98 -O3 -funroll-loops
18:01:44:WU07:FS01:0x22:   Platform: linux2 4.9.87-linuxkit-aufs
18:01:44:WU07:FS01:0x22:       Bits: 64
18:01:44:WU07:FS01:0x22:       Mode: Release
18:01:44:WU07:FS01:0x22:************************************ System ************************************
18:01:44:WU07:FS01:0x22:        CPU: Intel(R) Pentium(R) CPU G2020 @ 2.90GHz
18:01:44:WU07:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 58 Stepping 9
18:01:44:WU07:FS01:0x22:       CPUs: 2
18:01:44:WU07:FS01:0x22:     Memory: 7.75GiB
18:01:44:WU07:FS01:0x22:Free Memory: 7.24GiB
18:01:44:WU07:FS01:0x22:    Threads: POSIX_THREADS
18:01:44:WU07:FS01:0x22: OS Version: 3.19
18:01:44:WU07:FS01:0x22:Has Battery: false
18:01:44:WU07:FS01:0x22: On Battery: false
18:01:44:WU07:FS01:0x22: UTC Offset: 1
18:01:44:WU07:FS01:0x22:        PID: 1193
18:01:44:WU07:FS01:0x22:        CWD: /var/lib/fahclient/work
18:01:44:WU07:FS01:0x22:         OS: Linux 3.19.0-32-generic x86_64
18:01:44:WU07:FS01:0x22:    OS Arch: AMD64
18:01:44:WU07:FS01:0x22:********************************************************************************
18:01:44:WU07:FS01:0x22:Project: 11777 (Run 0, Clone 33782, Gen 1)
18:01:44:WU07:FS01:0x22:Unit: 0x00000004287234c95e79305af262ad4c
18:01:44:WU07:FS01:0x22:Reading tar file core.xml
18:01:44:WU07:FS01:0x22:Reading tar file integrator.xml
18:01:44:WU07:FS01:0x22:Reading tar file state.xml
18:01:44:WU07:FS01:0x22:Reading tar file system.xml
18:01:44:WU07:FS01:0x22:Digital signatures verified
18:01:44:WU07:FS01:0x22:Folding@home GPU Core22 Folding@home Core
18:01:44:WU07:FS01:0x22:Version 0.0.2
18:01:44:WU02:FS02:Sending unit results: id:02 state:SEND error:NO_ERROR project:13877 run:0 clone:690 gen:14 core:0x22 unit:0x0000001234e06d4a5e80cfed3832a723
18:01:44:WU02:FS02:Uploading 48.11MiB to 52.224.109.74
\x1b[91m18:01:44:ERROR:WU01:FS01:Exception: Failed to connect to 52.224.109.74:80: Network is unreachable\x1b[0m
18:01:44:WU02:FS02:Connecting to 52.224.109.74:8080
18:01:44:WU05:FS02:Sending unit results: id:05 state:SEND error:NO_ERROR project:11777 run:0 clone:19211 gen:16 core:0x22 unit:0x0000001c287234c95e774a5b994238ac
\x1b[93m18:01:44:WARNING:WU02:FS02:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU02:FS02:Connecting to 52.224.109.74:80
18:01:44:WU05:FS02:Uploading 23.06MiB to 40.114.52.201
18:01:44:WU05:FS02:Connecting to 40.114.52.201:8080
\x1b[93m18:01:44:WARNING:WU05:FS02:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU05:FS02:Connecting to 40.114.52.201:80
\x1b[93m18:01:44:WARNING:WU05:FS02:Exception: Failed to send results to work server: Failed to connect to 40.114.52.201:80: Network is unreachable\x1b[0m
18:01:44:WU05:FS02:Trying to send results to collection server
18:01:44:WU05:FS02:Uploading 23.06MiB to 52.224.109.74
18:01:44:WU05:FS02:Connecting to 52.224.109.74:8080
\x1b[93m18:01:44:WARNING:WU05:FS02:WorkServer connection failed on port 8080 trying 80\x1b[0m
18:01:44:WU05:FS02:Connecting to 52.224.109.74:80
\x1b[91m18:01:44:ERROR:WU05:FS02:Exception: Failed to connect to 52.224.109.74:80: Network is unreachable\x1b[0m
\x1b[93m18:01:44:WARNING:WU08:FS01:Exception: Could not get IP address for assign-GPU.stanford.edu: Temporary failure in name resolution\x1b[0m
\x1b[91m18:01:44:ERROR:WU08:FS01:Exception: Could not get an assignment\x1b[0m
18:01:45:WU06:FS02:0x22:*********************** Log Started 2020-04-09T18:01:44Z ***********************
18:01:45:WU06:FS02:0x22:*************************** Core22 Folding@home Core ***************************
18:01:45:WU06:FS02:0x22:       Type: 0x22
18:01:45:WU06:FS02:0x22:       Core: Core22
18:01:45:WU06:FS02:0x22:    Website: https://foldingathome.org/
18:01:45:WU06:FS02:0x22:  Copyright: (c) 2009-2018 foldingathome.org
18:01:45:WU06:FS02:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
18:01:45:WU06:FS02:0x22:             <rafal.wiewiora@choderalab.org>
18:01:45:WU06:FS02:0x22:       Args: -dir 06 -suffix 01 -version 704 -lifeline 1355 -checkpoint 15 -gpu
18:01:45:WU06:FS02:0x22:             1 -gpu-vendor nvidia
18:01:45:WU06:FS02:0x22:     Config: <none>
18:01:45:WU06:FS02:0x22:************************************ Build *************************************
18:01:45:WU06:FS02:0x22:    Version: 0.0.2
18:01:45:WU06:FS02:0x22:       Date: Dec 6 2019
18:01:45:WU06:FS02:0x22:       Time: 21:20:17
18:01:45:WU06:FS02:0x22: Repository: Git
18:01:45:WU06:FS02:0x22:   Revision: f87d92b58abdf7e6bf2e173cfbc4dc3e837c7042
18:01:45:WU06:FS02:0x22:     Branch: core22
18:01:45:WU06:FS02:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
18:01:45:WU06:FS02:0x22:    Options: -std=gnu++98 -O3 -funroll-loops
18:01:45:WU06:FS02:0x22:   Platform: linux2 4.9.87-linuxkit-aufs
18:01:45:WU06:FS02:0x22:       Bits: 64
18:01:45:WU06:FS02:0x22:       Mode: Release
18:01:45:WU06:FS02:0x22:************************************ System ************************************
18:01:45:WU06:FS02:0x22:        CPU: Intel(R) Pentium(R) CPU G2020 @ 2.90GHz
18:01:45:WU06:FS02:0x22:     CPU ID: GenuineIntel Family 6 Model 58 Stepping 9
18:01:45:WU06:FS02:0x22:       CPUs: 2
18:01:45:WU06:FS02:0x22:     Memory: 7.75GiB
18:01:45:WU06:FS02:0x22:Free Memory: 7.13GiB
18:01:45:WU06:FS02:0x22:    Threads: POSIX_THREADS
18:01:45:WU06:FS02:0x22: OS Version: 3.19
18:01:45:WU06:FS02:0x22:Has Battery: false
18:01:45:WU06:FS02:0x22: On Battery: false
18:01:45:WU06:FS02:0x22: UTC Offset: 1
18:01:45:WU06:FS02:0x22:        PID: 1359
18:01:45:WU06:FS02:0x22:        CWD: /var/lib/fahclient/work
18:01:45:WU06:FS02:0x22:         OS: Linux 3.19.0-32-generic x86_64
18:01:45:WU06:FS02:0x22:    OS Arch: AMD64
18:01:45:WU06:FS02:0x22:********************************************************************************
18:01:45:WU06:FS02:0x22:Project: 14431 (Run 0, Clone 82, Gen 3)
18:01:45:WU06:FS02:0x22:Unit: 0x000000030d5262775e8b4d597f4d7f65
18:01:45:WU06:FS02:0x22:Reading tar file core.xml
18:01:45:WU06:FS02:0x22:Reading tar file integrator.xml
18:01:45:WU06:FS02:0x22:Reading tar file state.xml
18:01:45:WU06:FS02:0x22:Reading tar file system.xml
18:01:46:WU06:FS02:0x22:Digital signatures verified
18:01:46:WU06:FS02:0x22:Folding@home GPU Core22 Folding@home Core
18:01:46:WU06:FS02:0x22:Version 0.0.2
18:01:54:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11759 run:0 clone:1428 gen:31 core:0x22 unit:0x0000004080fccb0a5e6d7c8eb8da243b
18:01:54:WU01:FS01:Uploading 49.93MiB to 128.252.203.10
18:01:54:WU01:FS01:Connecting to 128.252.203.10:8080
18:01:54:WU08:FS01:Connecting to 65.254.110.245:80
\x1b[93m18:01:54:WARNING:WU02:FS02:Exception: Failed to send results to work server: Failed to connect to 52.224.109.74:80: Network is unreachable\x1b[0m
\x1b[91m18:01:55:ERROR:WU04:FS01:Exception: Failed to connect to 52.224.109.74:80: Network is unreachable\x1b[0m
\x1b[93m18:01:55:WARNING:WU08:FS01:Failed to get assignment from '65.254.110.245:80': Empty work server assignment\x1b[0m
18:01:55:WU08:FS01:Connecting to 18.218.241.186:80
18:01:55:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:11781 run:0 clone:4356 gen:28 core:0x22 unit:0x0000002d0d5a98395e73c4fe33e70b0a
18:01:55:WU04:FS01:Uploading 55.24MiB to 13.90.152.57
18:01:55:WU04:FS01:Connecting to 13.90.152.57:8080
\x1b[93m18:01:56:WARNING:WU08:FS01:Failed to get assignment from '18.218.241.186:80': Empty work server assignment\x1b[0m
\x1b[91m18:01:56:ERROR:WU08:FS01:Exception: Could not get an assignment\x1b[0m
18:02:03:WU07:FS01:0x22:Completed 0 out of 2000000 steps (0%)
18:02:03:WU07:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
18:02:09:WU01:FS01:Upload 0.13%
18:02:09:WU06:FS02:0x22:Completed 0 out of 2000000 steps (0%)
18:02:09:WU06:FS02:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
18:02:44:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11778 run:0 clone:6876 gen:19 core:0x22 unit:0x00000026287234c95e743262966f272e
18:02:44:WU00:FS01:Uploading 23.06MiB to 40.114.52.201
18:02:44:WU00:FS01:Connecting to 40.114.52.201:8080
18:02:44:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:11777 run:0 clone:33961 gen:0 core:0x22 unit:0x00000001287234c95e7930550917b66c
18:02:44:WU03:FS02:Uploading 23.06MiB to 40.114.52.201
18:02:44:WU03:FS02:Connecting to 40.114.52.201:8080
18:02:45:WU02:FS02:Sending unit results: id:02 state:SEND error:NO_ERROR project:13877 run:0 clone:690 gen:14 core:0x22 unit:0x0000001234e06d4a5e80cfed3832a723
18:02:45:WU02:FS02:Uploading 48.11MiB to 52.224.109.74
18:02:45:WU02:FS02:Connecting to 52.224.109.74:8080
18:02:45:WU05:FS02:Sending unit results: id:05 state:SEND error:NO_ERROR project:11777 run:0 clone:19211 gen:16 core:0x22 unit:0x0000001c287234c95e774a5b994238ac
18:02:45:WU05:FS02:Uploading 23.06MiB to 40.114.52.201
18:02:45:WU05:FS02:Connecting to 40.114.52.201:8080
18:02:50:WU00:FS01:Upload 5.69%
18:02:50:WU03:FS02:Upload 12.47%
18:02:51:WU05:FS02:Upload 13.01%
18:02:51:WU02:FS02:Upload 6.63%
18:02:54:WU04:FS01:Upload 0.11%
18:02:56:WU00:FS01:Upload 17.35%
18:02:56:WU03:FS02:Upload 24.93%
18:02:57:WU02:FS02:Upload 11.82%
18:02:57:WU05:FS02:Upload 25.48%
18:02:57:WU07:FS01:0x22:Completed 20000 out of 2000000 steps (1%)
18:03:01:WU01:FS01:Upload 0.63%
18:03:02:WU03:FS02:Upload 37.13%
18:03:02:WU00:FS01:Upload 29.55%
18:03:03:WU02:FS02:Upload 16.89%
18:03:03:WU05:FS02:Upload 36.86%
18:03:07:WU01:FS01:Upload 3.25%
18:03:08:WU00:FS01:Upload 40.12%
18:03:08:WU03:FS02:Upload 48.78%
18:03:09:WU05:FS02:Upload 44.72%
18:03:09:WU02:FS02:Upload 21.44%
18:03:10:WU04:FS01:Upload 0.23%
18:03:13:WU01:FS01:Upload 7.26%
18:03:14:WU00:FS01:Upload 50.69%
18:03:14:WU03:FS02:Upload 58.81%
18:03:15:WU05:FS02:Upload 52.31%
18:03:15:WU02:FS02:Upload 25.07%
18:03:16:WU04:FS01:Upload 1.58%
18:03:19:WU01:FS01:Upload 12.02%
18:03:20:WU00:FS01:Upload 60.18%
18:03:20:WU03:FS02:Upload 68.83%
18:03:21:WU05:FS02:Upload 61.79%
18:03:21:WU02:FS02:Upload 27.54%
18:03:22:WU04:FS01:Upload 2.83%
18:03:25:WU01:FS01:Upload 16.15%
18:03:26:WU03:FS02:Upload 80.21%
18:03:26:WU00:FS01:Upload 68.31%
18:03:27:WU05:FS02:Upload 73.72%
18:03:27:WU02:FS02:Upload 30.01%
18:03:28:WU04:FS01:Upload 4.07%
18:03:31:WU01:FS01:Upload 19.90%
18:03:32:WU00:FS01:Upload 76.71%
18:03:32:WU03:FS02:Upload 88.34%
18:03:33:WU05:FS02:Upload 84.83%
18:03:33:WU02:FS02:Upload 34.04%
18:03:34:WU04:FS01:Upload 5.54%
18:03:37:WU01:FS01:Upload 23.16%
18:03:38:WU00:FS01:Upload 87.55%
18:03:38:WU03:FS02:Upload 98.37%
18:03:39:WU02:FS02:Upload 37.55%
18:03:39:WU05:FS02:Upload 93.77%
18:03:39:WU03:FS02:Upload complete
18:03:39:WU03:FS02:Server responded WORK_ACK (400)
18:03:39:WU03:FS02:Final credit estimate, 17429.00 points
18:03:39:WU03:FS02:Cleaning up
18:03:39:WU08:FS01:Connecting to 65.254.110.245:80
\x1b[93m18:03:40:WARNING:WU08:FS01:Failed to get assignment from '65.254.110.245:80': Empty work server assignment\x1b[0m
18:03:40:WU08:FS01:Connecting to 18.218.241.186:80
\x1b[93m18:03:40:WARNING:WU08:FS01:Failed to get assignment from '18.218.241.186:80': Empty work server assignment\x1b[0m
\x1b[91m18:03:40:ERROR:WU08:FS01:Exception: Could not get an assignment\x1b[0m
18:03:40:WU04:FS01:Upload 6.90%
18:03:42:WU05:FS02:Upload complete
18:03:42:WU05:FS02:Server responded WORK_ACK (400)
18:03:42:WU05:FS02:Final credit estimate, 29136.00 points
18:03:42:WU05:FS02:Cleaning up
18:03:43:WU01:FS01:Upload 26.41%
18:03:43:WU00:FS01:Upload complete
18:03:43:WU00:FS01:Server responded WORK_ACK (400)
18:03:43:WU00:FS01:Final credit estimate, 16931.00 points
18:03:43:WU00:FS01:Cleaning up
18:03:45:WU02:FS02:Upload 39.62%
18:03:46:WU04:FS01:Upload 8.26%
18:03:49:WU01:FS01:Upload 35.55%
18:03:50:WU07:FS01:0x22:Completed 40000 out of 2000000 steps (2%)
18:03:51:WU02:FS02:Upload 43.52%
18:03:52:WU04:FS01:Upload 9.39%
18:03:55:WU01:FS01:Upload 47.06%
18:03:57:WU02:FS02:Upload 47.94%
18:03:58:WU04:FS01:Upload 10.41%
18:04:00:WU06:FS02:0x22:Completed 20000 out of 2000000 steps (1%)
18:04:01:WU01:FS01:Upload 56.08%
18:04:03:WU02:FS02:Upload 53.01%
18:04:04:WU04:FS01:Upload 11.65%
18:04:07:WU01:FS01:Upload 66.84%
18:04:09:WU02:FS02:Upload 56.51%
18:04:10:WU04:FS01:Upload 12.79%
18:04:13:WU01:FS01:Upload 76.10%
18:04:15:WU02:FS02:Upload 61.06%
18:04:16:WU04:FS01:Upload 14.14%
18:04:19:WU01:FS01:Upload 89.37%
18:04:21:WU02:FS02:Upload 65.48%
18:04:22:WU04:FS01:Upload 15.50%
18:04:25:WU01:FS01:Upload 96.13%
18:04:27:WU02:FS02:Upload 71.19%
18:04:28:WU04:FS01:Upload 16.75%
18:04:28:WU01:FS01:Upload complete
18:04:28:WU01:FS01:Server responded WORK_ACK (400)
18:04:28:WU01:FS01:Final credit estimate, 30914.00 points
18:04:28:WU01:FS01:Cleaning up
18:04:33:WU02:FS02:Upload 76.52%
18:04:34:WU04:FS01:Upload 17.99%
18:04:39:WU02:FS02:Upload 81.33%
18:04:40:WU04:FS01:Upload 19.46%
18:04:45:WU07:FS01:0x22:Completed 60000 out of 2000000 steps (3%)
18:04:45:WU02:FS02:Upload 85.35%
18:04:46:WU04:FS01:Upload 21.16%
18:04:52:WU04:FS01:Upload 22.40%
18:04:52:WU02:FS02:Upload 88.47%
18:04:58:WU04:FS01:Upload 23.76%
18:04:58:WU02:FS02:Upload 93.93%
18:05:04:WU02:FS02:Upload 99.90%
18:05:04:WU04:FS01:Upload 25.35%
18:05:05:WU02:FS02:Upload complete
18:05:05:WU02:FS02:Server responded WORK_ACK (400)
18:05:05:WU02:FS02:Final credit estimate, 79886.00 points
18:05:05:WU02:FS02:Cleaning up
18:05:10:WU04:FS01:Upload 26.48%
18:05:16:WU04:FS01:Upload 27.72%
18:05:16:WU08:FS01:Connecting to 65.254.110.245:80
\x1b[93m18:05:17:WARNING:WU08:FS01:Failed to get assignment from '65.254.110.245:80': Empty work server assignment\x1b[0m
18:05:17:WU08:FS01:Connecting to 18.218.241.186:80
\x1b[93m18:05:17:WARNING:WU08:FS01:Failed to get assignment from '18.218.241.186:80': Empty work server assignment\x1b[0m
\x1b[91m18:05:17:ERROR:WU08:FS01:Exception: Could not get an assignment\x1b[0m
18:05:22:WU04:FS01:Upload 29.19%
18:05:30:WU04:FS01:Upload 30.10%
18:05:36:WU04:FS01:Upload 31.12%
18:05:38:WU07:FS01:0x22:Completed 80000 out of 2000000 steps (4%)
18:05:42:WU04:FS01:Upload 32.59%
18:05:48:WU04:FS01:Upload 33.94%
18:05:50:WU06:FS02:0x22:Completed 40000 out of 2000000 steps (2%)
18:05:54:WU04:FS01:Upload 35.19%
18:06:00:WU04:FS01:Upload 36.77%
18:06:06:WU04:FS01:Upload 38.24%
18:06:12:WU04:FS01:Upload 40.28%
18:06:18:WU04:FS01:Upload 41.98%
18:06:24:WU04:FS01:Upload 43.45%
18:06:30:WU04:FS01:Upload 45.26%
18:06:32:WU07:FS01:0x22:Completed 100000 out of 2000000 steps (5%)
18:06:36:WU04:FS01:Upload 46.84%
18:06:42:WU04:FS01:Upload 48.31%
18:06:48:WU04:FS01:Upload 49.67%
18:06:54:WU04:FS01:Upload 51.03%
18:07:00:WU04:FS01:Upload 52.61%
18:07:06:WU04:FS01:Upload 54.09%
18:07:12:WU04:FS01:Upload 55.44%
18:07:18:WU04:FS01:Upload 56.91%
18:07:24:WU04:FS01:Upload 58.72%
18:07:27:WU07:FS01:0x22:Completed 120000 out of 2000000 steps (6%)
18:07:30:WU04:FS01:Upload 60.20%
18:07:36:WU04:FS01:Upload 61.44%
18:07:40:WU06:FS02:0x22:Completed 60000 out of 2000000 steps (3%)
18:07:42:WU04:FS01:Upload 62.80%
18:07:48:WU04:FS01:Upload 64.16%
18:07:54:WU08:FS01:Connecting to 65.254.110.245:80
18:07:54:WU04:FS01:Upload 65.74%
\x1b[93m18:07:54:WARNING:WU08:FS01:Failed to get assignment from '65.254.110.245:80': Empty work server assignment\x1b[0m
18:07:54:WU08:FS01:Connecting to 18.218.241.186:80
\x1b[93m18:07:55:WARNING:WU08:FS01:Failed to get assignment from '18.218.241.186:80': Empty work server assignment\x1b[0m
\x1b[91m18:07:55:ERROR:WU08:FS01:Exception: Could not get an assignment\x1b[0m
18:08:00:WU04:FS01:Upload 67.10%
18:08:06:WU04:FS01:Upload 68.57%
18:08:12:WU04:FS01:Upload 69.93%
18:08:18:WU04:FS01:Upload 71.40%
18:08:21:WU07:FS01:0x22:Completed 140000 out of 2000000 steps (7%)
18:08:24:WU04:FS01:Upload 72.75%
18:08:30:WU04:FS01:Upload 74.23%
18:08:36:WU04:FS01:Upload 75.58%
18:08:42:WU04:FS01:Upload 77.17%
18:08:48:WU04:FS01:Upload 78.53%
18:08:54:WU04:FS01:Upload 80.11%
18:09:00:WU04:FS01:Upload 81.47%
18:09:06:WU04:FS01:Upload 83.16%
18:09:12:WU04:FS01:Upload 85.09%
18:09:18:WU04:FS01:Upload 86.33%
18:09:18:WU07:FS01:0x22:Completed 160000 out of 2000000 steps (8%)
18:09:24:WU04:FS01:Upload 87.80%
18:09:29:WU06:FS02:0x22:Completed 80000 out of 2000000 steps (4%)
18:09:30:WU04:FS01:Upload 89.16%
18:09:36:WU04:FS01:Upload 90.29%
18:09:42:WU04:FS01:Upload 91.65%
18:09:48:WU04:FS01:Upload 92.90%
18:09:54:WU04:FS01:Upload 94.37%
18:10:00:WU04:FS01:Upload 95.84%
18:10:06:WU04:FS01:Upload 97.31%
18:10:11:WU07:FS01:0x22:Completed 180000 out of 2000000 steps (9%)
18:10:12:WU04:FS01:Upload 98.55%
18:10:21:WU04:FS01:Upload complete
18:10:21:WU04:FS01:Server responded WORK_ACK (400)
18:10:21:WU04:FS01:Final credit estimate, 34316.00 points
18:10:21:WU04:FS01:Cleaning up
18:11:05:WU07:FS01:0x22:Completed 200000 out of 2000000 steps (10%)
18:11:20:WU06:FS02:0x22:Completed 100000 out of 2000000 steps (5%)
18:12:01:WU07:FS01:0x22:Completed 220000 out of 2000000 steps (11%)
18:12:08:WU08:FS01:Connecting to 65.254.110.245:80
\x1b[93m18:12:08:WARNING:WU08:FS01:Failed to get assignment from '65.254.110.245:80': Empty work server assignment\x1b[0m
18:12:08:WU08:FS01:Connecting to 18.218.241.186:80
18:12:09:WU08:FS01:Assigned to work server 128.252.203.10
18:12:09:WU08:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GP102 [TITAN Xp] 12150 from 128.252.203.10
18:12:09:WU08:FS01:Connecting to 128.252.203.10:8080
\x1b[91m18:12:22:ERROR:WU08:FS01:Exception: 10002: Received short response, expected 512 bytes, got 0\x1b[0m
18:12:57:WU07:FS01:0x22:Completed 240000 out of 2000000 steps (12%)
18:13:11:WU06:FS02:0x22:Completed 120000 out of 2000000 steps (6%)
18:13:53:WU07:FS01:0x22:Completed 260000 out of 2000000 steps (13%)
18:14:47:WU07:FS01:0x22:Completed 280000 out of 2000000 steps (14%)
18:15:01:WU06:FS02:0x22:Completed 140000 out of 2000000 steps (7%)
18:15:41:WU07:FS01:0x22:Completed 300000 out of 2000000 steps (15%)
18:16:38:WU07:FS01:0x22:Completed 320000 out of 2000000 steps (16%)
18:16:51:WU06:FS02:0x22:Completed 160000 out of 2000000 steps (8%)
18:17:32:WU07:FS01:0x22:Completed 340000 out of 2000000 steps (17%)
18:18:28:WU07:FS01:0x22:Completed 360000 out of 2000000 steps (18%)
18:18:41:WU06:FS02:0x22:Completed 180000 out of 2000000 steps (9%)
18:18:59:WU08:FS01:Connecting to 65.254.110.245:80
\x1b[93m18:19:00:WARNING:WU08:FS01:Failed to get assignment from '65.254.110.245:80': Empty work server assignment\x1b[0m
18:19:00:WU08:FS01:Connecting to 18.218.241.186:80
\x1b[93m18:19:00:WARNING:WU08:FS01:Failed to get assignment from '18.218.241.186:80': Empty work server assignment\x1b[0m
\x1b[91m18:19:00:ERROR:WU08:FS01:Exception: Could not get an assignment\x1b[0m
18:19:23:WU07:FS01:0x22:Completed 380000 out of 2000000 steps (19%)
18:20:17:WU07:FS01:0x22:Completed 400000 out of 2000000 steps (20%)
18:20:32:WU06:FS02:0x22:Completed 200000 out of 2000000 steps (10%)