99.99% and stuck

qe4 · Post by **qe4** » Mon Apr 27, 2020 12:55 pm

Hi. I only run GPU WU's from my Ubuntu server with 4 AMD RX 480. Everything was working fine yesterday, but today the last 3 WU's have had the same issue: They get stuck at 99.99% and don't upload.

I am pretty new to this so there are a few things I haven't fingured out yet. Like:
- How do I interact with the client (other than the web-interface which is working fine but only has limited options)?
- How do I cancel/remove a WU that is stuck?

ajm · Post by **ajm** » Mon Apr 27, 2020 2:49 pm

Up to Ubuntu 18, you can install Python2 and then FAHControl, which gives you that interaction.

Code: Select all

wget https://download.foldingathome.org/releases/public/release/fahcontrol/debian-stable-64bit/v7.5/fahcontrol_7.5.1-1_all.deb
sudo dpkg -i fahcontrol_7.5.1-1_all.deb

Otherwise all you really can do that I'm aware of is edit /etc/fahclient/config.xml
I'm not sure, but I think that stopping the client, removing the gpu slot in config.xml and restarting the client should dump the WU. Alternatively, you can set <gpu value="false"/>, which should eliminate the GPU slots.

qe4 · Post by **qe4** » Mon Apr 27, 2020 4:50 pm

But doesn't FAHControl require GUI (I am on Ubuntu server)?

ajm · Post by **ajm** » Mon Apr 27, 2020 5:04 pm

Yes, it does, sorry.

Maybe you can edit /etc/fahclient/config.xml, adding

Code: Select all

<!-- Remote Command Server -->
<command-allow-no-pass v="127.0.0.1 192.168.2.59"/>

where 192.168.2.59 would be replaced by the IP of another machine of the network with a GUI (Linux, Win, or Mac), where you would run FAHControl (with no local slot allocated). From there, you could add a client with the IP of your server and thus control FAH on the server. For it to work, FAHClient would need to be restarted on the server.

HaloJones · Post by **HaloJones** » Mon Apr 27, 2020 5:10 pm

there are commands that can be issued via telnet although I'm struggling to find details of how to use it

ajm · Post by **ajm** » Mon Apr 27, 2020 5:13 pm

Yes, I haven't found them either...
And sorry again, the code must be more complete than what I give above:

Code: Select all

<!-- HTTP Server -->
<allow v="127.0.0.1 192.168.2.59"/>
<!-- Network -->
<proxy v=":8080"/>
<!-- Remote Command Server -->
<command-allow-no-pass v="127.0.0.1 192.168.2.59"/>

In the <config> section.

ajm · Post by **ajm** » Mon Apr 27, 2020 5:51 pm

There's something here: https://github.com/FoldingAtHome/fah-co ... Client-API

Post by **Joe_H** » Mon Apr 27, 2020 9:11 pm

Using telnet into the client is mostly documented by the help command at the prompt, telnet 127.0.0.1 36330 enters you into a command entry field.

Post by **Joe_H** » Mon Apr 27, 2020 9:13 pm

Getting back to the original post, the WU going to 99.99% and then sitting there during processing is usually an indication the video driver crashed. That could be from a bug, or more commonly from overheating or an excessive overclock.

qe4 · Post by **qe4** » Mon Apr 27, 2020 10:07 pm

I managed to delete the stuck WU's by deleting the work folder and running

Code: Select all

$ FAHclient --dump ALL

I got fresh WU for my GPU's but got the same problem. 2 of the WU's have completed but they are not uploaded to the collection server. Here is a log file from one of them:

Code: Select all

*********************** Log Started 2020-04-27T18:02:19Z ***********************
*************************** Core22 Folding@home Core ***************************
       Type: 0x22
       Core: Core22
    Website: https://foldingathome.org/
  Copyright: (c) 2009-2018 foldingathome.org
     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
             <rafal.wiewiora@choderalab.org>
       Args: -dir 03 -suffix 01 -version 706 -lifeline 3112 -checkpoint 15
             -gpu-vendor amd -opencl-platform 0 -opencl-device 3 -gpu 3
     Config: <none>
************************************ Build *************************************
    Version: 0.0.5
       Date: Apr 22 2020
       Time: 03:57:11
 Repository: Git
   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
     Branch: HEAD
   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
    Options: -std=c++11 -O3 -funroll-loops
   Platform: linux2 4.19.76-linuxkit
       Bits: 64
       Mode: Release
************************************ System ************************************
        CPU: Intel(R) Core(TM) i5-4430 CPU @ 3.00GHz
     CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
       CPUs: 4
     Memory: 7.48GiB
Free Memory: 3.06GiB
    Threads: POSIX_THREADS
 OS Version: 4.4
Has Battery: false
 On Battery: false
 UTC Offset: 0
        PID: 3116
        CWD: /home/abcabc/abc/foldingathome/work
         OS: Linux 4.4.0-130-generic x86_64
    OS Arch: AMD64
********************************************************************************
Project: 11743 (Run 0, Clone 2700, Gen 61)
Unit: 0x000000608ca304f15e69d11615081141
Reading tar file core.xml
Reading tar file integrator.xml
Reading tar file state.xml
Reading tar file system.xml
Digital signatures verified
Folding@home GPU Core22 Folding@home Core
Version 0.0.5
Completed 0 out of 2000000 steps (0%)
Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
Completed 20000 out of 2000000 steps (1%)
Completed 40000 out of 2000000 steps (2%)
Completed 60000 out of 2000000 steps (3%)
Completed 80000 out of 2000000 steps (4%)
Completed 100000 out of 2000000 steps (5%)
Completed 120000 out of 2000000 steps (6%)
Completed 140000 out of 2000000 steps (7%)
Completed 160000 out of 2000000 steps (8%)
Completed 180000 out of 2000000 steps (9%)
Completed 200000 out of 2000000 steps (10%)
Completed 220000 out of 2000000 steps (11%)
Completed 240000 out of 2000000 steps (12%)
Completed 260000 out of 2000000 steps (13%)
Completed 280000 out of 2000000 steps (14%)
Completed 300000 out of 2000000 steps (15%)
Completed 320000 out of 2000000 steps (16%)
Completed 340000 out of 2000000 steps (17%)
Completed 360000 out of 2000000 steps (18%)
Completed 380000 out of 2000000 steps (19%)
Completed 400000 out of 2000000 steps (20%)
Completed 420000 out of 2000000 steps (21%)
Completed 440000 out of 2000000 steps (22%)
Completed 460000 out of 2000000 steps (23%)
Completed 480000 out of 2000000 steps (24%)
Completed 500000 out of 2000000 steps (25%)
Completed 520000 out of 2000000 steps (26%)
Completed 540000 out of 2000000 steps (27%)
Completed 560000 out of 2000000 steps (28%)
Completed 580000 out of 2000000 steps (29%)
Completed 600000 out of 2000000 steps (30%)
Completed 620000 out of 2000000 steps (31%)
Completed 640000 out of 2000000 steps (32%)
Completed 660000 out of 2000000 steps (33%)
Completed 680000 out of 2000000 steps (34%)
Completed 700000 out of 2000000 steps (35%)
Completed 720000 out of 2000000 steps (36%)
Completed 740000 out of 2000000 steps (37%)
Completed 760000 out of 2000000 steps (38%)
Completed 780000 out of 2000000 steps (39%)
Completed 800000 out of 2000000 steps (40%)
Completed 820000 out of 2000000 steps (41%)
Completed 840000 out of 2000000 steps (42%)
Completed 860000 out of 2000000 steps (43%)
Completed 880000 out of 2000000 steps (44%)
Completed 900000 out of 2000000 steps (45%)
Completed 920000 out of 2000000 steps (46%)
Completed 940000 out of 2000000 steps (47%)
Completed 960000 out of 2000000 steps (48%)
Completed 980000 out of 2000000 steps (49%)
Completed 1000000 out of 2000000 steps (50%)
Completed 1020000 out of 2000000 steps (51%)
Completed 1040000 out of 2000000 steps (52%)
Completed 1060000 out of 2000000 steps (53%)
Completed 1080000 out of 2000000 steps (54%)
Completed 1100000 out of 2000000 steps (55%)
Completed 1120000 out of 2000000 steps (56%)
Completed 1140000 out of 2000000 steps (57%)
Completed 1160000 out of 2000000 steps (58%)
Completed 1180000 out of 2000000 steps (59%)
Completed 1200000 out of 2000000 steps (60%)
Completed 1220000 out of 2000000 steps (61%)
Completed 1240000 out of 2000000 steps (62%)
Completed 1260000 out of 2000000 steps (63%)
Completed 1280000 out of 2000000 steps (64%)
Completed 1300000 out of 2000000 steps (65%)
Completed 1320000 out of 2000000 steps (66%)
Completed 1340000 out of 2000000 steps (67%)
Completed 1360000 out of 2000000 steps (68%)
Completed 1380000 out of 2000000 steps (69%)
Completed 1400000 out of 2000000 steps (70%)
Completed 1420000 out of 2000000 steps (71%)
Completed 1440000 out of 2000000 steps (72%)
Completed 1460000 out of 2000000 steps (73%)
Completed 1480000 out of 2000000 steps (74%)
Completed 1500000 out of 2000000 steps (75%)
Completed 1520000 out of 2000000 steps (76%)
Completed 1540000 out of 2000000 steps (77%)
Completed 1560000 out of 2000000 steps (78%)
Completed 1580000 out of 2000000 steps (79%)
Completed 1600000 out of 2000000 steps (80%)
Completed 1620000 out of 2000000 steps (81%)
Completed 1640000 out of 2000000 steps (82%)
Completed 1660000 out of 2000000 steps (83%)
Completed 1680000 out of 2000000 steps (84%)
Completed 1700000 out of 2000000 steps (85%)
Completed 1720000 out of 2000000 steps (86%)
Completed 1740000 out of 2000000 steps (87%)
Completed 1760000 out of 2000000 steps (88%)
Completed 1780000 out of 2000000 steps (89%)
Completed 1800000 out of 2000000 steps (90%)
Completed 1820000 out of 2000000 steps (91%)
Completed 1840000 out of 2000000 steps (92%)
Completed 1860000 out of 2000000 steps (93%)
Completed 1880000 out of 2000000 steps (94%)
Completed 1900000 out of 2000000 steps (95%)
Completed 1920000 out of 2000000 steps (96%)
Completed 1940000 out of 2000000 steps (97%)
Completed 1960000 out of 2000000 steps (98%)
Completed 1980000 out of 2000000 steps (99%)
Completed 2000000 out of 2000000 steps (100%)
Saving result file ../logfile_01.txt
Saving result file checkpointState.xml
Saving result file checkpt.crc
Saving result file positions.xtc
Saving result file science.log
Folding@home Core Shutdown: FINISHED_UNIT

When I pause the folding I get a whole bunch of these lines in the main log:

Code: Select all

21:54:21:WARNING:FS00:Killing WU00
21:54:21:WARNING:FS01:Killing WU01
21:54:21:WARNING:FS02:Killing WU02
21:54:21:WARNING:FS03:Killing WU03
21:54:21:WARNING:FS00:Killing WU00
21:54:21:WARNING:FS01:Killing WU01
21:54:21:WARNING:FS02:Killing WU02
21:54:21:WARNING:FS03:Killing WU03
21:54:21:WARNING:FS00:Killing WU00
21:54:21:WARNING:FS01:Killing WU01
...

Post by **Joe_H** » Mon Apr 27, 2020 10:15 pm

That is the log file from the work directory of the WU, what does the log.txt file have for the client as a whole?

qe4 · Post by **qe4** » Mon Apr 27, 2020 10:35 pm

Here is the log file. I have restarted the client a couple of times. I am running it from command line, so not as a service.

Code: Select all

*********************** Log Started 2020-04-27T17:59:21Z ***********************
17:59:21:****************************** FAHClient ******************************
17:59:21:        Version: 7.6.9
17:59:21:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
17:59:21:      Copyright: 2020 foldingathome.org
17:59:21:       Homepage: https://foldingathome.org/
17:59:21:           Date: Apr 17 2020
17:59:21:           Time: 18:11:26
17:59:21:       Revision: 398c2b17fa535e0cc6c9d10856b2154c32771646
17:59:21:         Branch: master
17:59:21:       Compiler: GNU 8.3.0
17:59:21:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
17:59:21:                 -funroll-loops -fno-pie
17:59:21:       Platform: linux2 4.19.0-5-amd64
17:59:21:           Bits: 64
17:59:21:           Mode: Release
17:59:21:           Args: --config /etc/fahclient/config.xml
17:59:21:         Config: /etc/fahclient/config.xml
17:59:21:******************************** CBang ********************************
17:59:21:           Date: Apr 17 2020
17:59:21:           Time: 18:10:13
17:59:21:       Revision: 2fb0be7809c5e45287a122ca5fbc15b5ae859a3b
17:59:21:         Branch: master
17:59:21:       Compiler: GNU 8.3.0
17:59:21:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
17:59:21:                 -funroll-loops -fno-pie -fPIC
17:59:21:       Platform: linux2 4.19.0-5-amd64
17:59:21:           Bits: 64
17:59:21:           Mode: Release
17:59:21:******************************* System ********************************
17:59:21:            CPU: Intel(R) Core(TM) i5-4430 CPU @ 3.00GHz
17:59:21:         CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
17:59:21:           CPUs: 4
17:59:21:         Memory: 7.48GiB
17:59:21:    Free Memory: 3.51GiB
17:59:21:        Threads: POSIX_THREADS
17:59:21:     OS Version: 4.4
17:59:21:    Has Battery: false
17:59:21:     On Battery: false
17:59:21:     UTC Offset: 0
17:59:21:            PID: 2849
17:59:21:            CWD: /home/aaabbbb/cccddd/foldingathome
17:59:21:             OS: Linux 4.4.0-130-generic x86_64
17:59:21:        OS Arch: AMD64
17:59:21:           GPUs: 4
17:59:21:          GPU 0: Bus:1 Slot:0 Func:0 AMD:5 Hawaii [Radeon R9 200/300X Series]
17:59:21:          GPU 1: Bus:2 Slot:0 Func:0 AMD:5 Ellesmere XT [Radeon RX
17:59:21:                 470/480/570/580/590]
17:59:21:          GPU 2: Bus:3 Slot:0 Func:0 AMD:5 Ellesmere XT [Radeon RX
17:59:21:                 470/480/570/580/590]
17:59:21:          GPU 3: Bus:4 Slot:0 Func:0 AMD:5 Ellesmere XT [Radeon RX
17:59:21:                 470/480/570/580/590]
17:59:21:           CUDA: Not detected: Failed to open dynamic library 'libcuda.so':
17:59:21:                 libcuda.so: cannot open shared object file: No such file or
17:59:21:                 directory
17:59:21:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:2580.4
17:59:21:OpenCL Device 1: Platform:0 Device:1 Bus:2 Slot:0 Compute:1.2 Driver:2580.4
17:59:21:OpenCL Device 2: Platform:0 Device:2 Bus:3 Slot:0 Compute:1.2 Driver:2580.4
17:59:21:OpenCL Device 3: Platform:0 Device:3 Bus:4 Slot:0 Compute:1.2 Driver:2580.4
17:59:21:******************************* libFAH ********************************
17:59:21:           Date: Apr 15 2020
17:59:21:           Time: 21:43:24
17:59:21:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
17:59:21:         Branch: master
17:59:21:       Compiler: GNU 8.3.0
17:59:21:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
17:59:21:                 -funroll-loops -fno-pie
17:59:21:       Platform: linux2 4.19.0-5-amd64
17:59:21:           Bits: 64
17:59:21:           Mode: Release
17:59:21:***********************************************************************
17:59:21:<config>
17:59:21:  <!-- Folding Core -->
17:59:21:  <cpu-usage v='20'/>
17:59:21:
17:59:21:  <!-- HTTP Server -->
17:59:21:  <allow v='127.0.0.1 xxx.yyy.zzz.aaa aaa.bbb.ccc.ddd'/>
17:59:21:
17:59:21:  <!-- Slot Control -->
17:59:21:  <power v='FULL'/>
17:59:21:
17:59:21:  <!-- User Information -->
17:59:21:  <passkey v='*****'/>
17:59:21:  <team v='244483'/>
17:59:21:  <user v='MyUsername'/>
17:59:21:
17:59:21:  <!-- Web Server -->
17:59:21:  <web-allow v='127.0.0.1 xxx.yyy.zzz.aaa aaa.bbb.ccc.ddd'/>
17:59:21:
17:59:21:  <!-- Folding Slots -->
17:59:21:  <slot id='0' type='GPU'>
17:59:21:    <paused v='true'/>
17:59:21:  </slot>
17:59:21:  <slot id='1' type='GPU'>
17:59:21:    <paused v='true'/>
17:59:21:  </slot>
17:59:21:  <slot id='2' type='GPU'>
17:59:21:    <paused v='true'/>
17:59:21:  </slot>
17:59:21:  <slot id='3' type='GPU'>
17:59:21:    <paused v='true'/>
17:59:21:  </slot>
17:59:21:</config>
17:59:21:Trying to access database...
17:59:21:Successfully acquired database lock
17:59:21:Enabled folding slot 00: PAUSED gpu:0:Hawaii [Radeon R9 200/300X Series] (by user)
17:59:21:Enabled folding slot 01: PAUSED gpu:1:Ellesmere XT [Radeon RX 470/480/570/580/590] (by user)
17:59:21:Enabled folding slot 02: PAUSED gpu:2:Ellesmere XT [Radeon RX 470/480/570/580/590] (by user)
17:59:21:Enabled folding slot 03: PAUSED gpu:3:Ellesmere XT [Radeon RX 470/480/570/580/590] (by user)
17:59:33:2:62.198.23.116:New Web session
17:59:49:FS00:Unpaused
17:59:49:FS01:Unpaused
17:59:49:FS02:Unpaused
17:59:49:FS03:Unpaused
17:59:49:WU00:FS00:Connecting to 65.254.110.245:80
17:59:49:WU01:FS01:Connecting to 65.254.110.245:80
17:59:50:WU00:FS00:Connecting to 65.254.110.245:80
17:59:50:WU02:FS02:Connecting to 65.254.110.245:80
17:59:50:WU01:FS01:Connecting to 65.254.110.245:80
17:59:50:WU03:FS03:Connecting to 65.254.110.245:80
[93m17:59:50:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
17:59:50:WU00:FS00:Connecting to 18.218.241.186:80
[93m17:59:50:WARNING:WU02:FS02:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
17:59:50:WU02:FS02:Connecting to 18.218.241.186:80
[93m17:59:50:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
17:59:50:WU01:FS01:Connecting to 18.218.241.186:80
[93m17:59:51:WARNING:WU03:FS03:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
17:59:51:WU03:FS03:Connecting to 18.218.241.186:80
[93m17:59:51:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
17:59:51:WU00:FS00:Connecting to 65.254.110.245:80
[93m17:59:51:WARNING:WU02:FS02:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
17:59:51:WU02:FS02:Connecting to 65.254.110.245:80
[93m17:59:51:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
17:59:51:WU01:FS01:Connecting to 65.254.110.245:80
[93m17:59:51:WARNING:WU03:FS03:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
17:59:51:WU03:FS03:Connecting to 65.254.110.245:80
[93m17:59:51:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
17:59:51:WU00:FS00:Connecting to 18.218.241.186:80
[93m17:59:51:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
17:59:51:WU01:FS01:Connecting to 18.218.241.186:80
[93m17:59:51:WARNING:WU02:FS02:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
17:59:51:WU02:FS02:Connecting to 18.218.241.186:80
[93m17:59:52:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
[91m17:59:52:ERROR:WU00:FS00:Exception: Could not get an assignment[0m
[93m17:59:52:WARNING:WU02:FS02:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
[91m17:59:52:ERROR:WU02:FS02:Exception: Could not get an assignment[0m
[93m17:59:52:WARNING:WU03:FS03:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
17:59:52:WU03:FS03:Connecting to 18.218.241.186:80
[93m17:59:52:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
[91m17:59:52:ERROR:WU01:FS01:Exception: Could not get an assignment[0m
17:59:52:WU02:FS02:Connecting to 65.254.110.245:80
17:59:52:WU00:FS00:Connecting to 65.254.110.245:80
17:59:52:WU01:FS01:Connecting to 65.254.110.245:80
[93m17:59:52:WARNING:WU03:FS03:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
[91m17:59:52:ERROR:WU03:FS03:Exception: Could not get an assignment[0m
17:59:53:WU03:FS03:Connecting to 65.254.110.245:80
[93m17:59:53:WARNING:WU02:FS02:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
17:59:53:WU02:FS02:Connecting to 18.218.241.186:80
[93m17:59:53:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
17:59:53:WU00:FS00:Connecting to 18.218.241.186:80
[93m17:59:53:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
17:59:53:WU01:FS01:Connecting to 18.218.241.186:80
[93m17:59:53:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
17:59:53:WU00:FS00:Connecting to 65.254.110.245:80
[93m17:59:53:WARNING:WU02:FS02:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
17:59:53:WU02:FS02:Connecting to 65.254.110.245:80
[93m17:59:53:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
17:59:53:WU01:FS01:Connecting to 65.254.110.245:80
[93m17:59:53:WARNING:WU03:FS03:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
17:59:53:WU03:FS03:Connecting to 18.218.241.186:80
[93m17:59:54:WARNING:WU03:FS03:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
17:59:54:WU03:FS03:Connecting to 65.254.110.245:80
[93m17:59:54:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
17:59:54:WU01:FS01:Connecting to 18.218.241.186:80
[93m17:59:54:WARNING:WU02:FS02:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
17:59:54:WU02:FS02:Connecting to 18.218.241.186:80
[93m17:59:54:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
17:59:54:WU00:FS00:Connecting to 18.218.241.186:80
[93m17:59:54:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
[91m17:59:54:ERROR:WU00:FS00:Exception: Could not get an assignment[0m
[93m17:59:54:WARNING:WU02:FS02:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
[91m17:59:54:ERROR:WU02:FS02:Exception: Could not get an assignment[0m
[93m17:59:54:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
[91m17:59:54:ERROR:WU01:FS01:Exception: Could not get an assignment[0m
[93m17:59:54:WARNING:WU03:FS03:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
17:59:54:WU03:FS03:Connecting to 18.218.241.186:80
[93m17:59:55:WARNING:WU03:FS03:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
[91m17:59:55:ERROR:WU03:FS03:Exception: Could not get an assignment[0m
18:00:22:Saving configuration to /etc/fahclient/config.xml
18:00:22:<config>
18:00:22:  <!-- Folding Core -->
18:00:22:  <cpu-usage v='20'/>
18:00:22:
18:00:22:  <!-- HTTP Server -->
18:00:22:  <allow v='127.0.0.1 xxx.yyy.zzz.aaa aaa.bbb.ccc.ddd'/>
18:00:22:
18:00:22:  <!-- Slot Control -->
18:00:22:  <power v='FULL'/>
18:00:22:
18:00:22:  <!-- User Information -->
18:00:22:  <passkey v='*****'/>
18:00:22:  <team v='244483'/>
18:00:22:  <user v='MyUsername'/>
18:00:22:
18:00:22:  <!-- Web Server -->
18:00:22:  <web-allow v='127.0.0.1 xxx.yyy.zzz.aaa aaa.bbb.ccc.ddd'/>
18:00:22:
18:00:22:  <!-- Folding Slots -->
18:00:22:  <slot id='0' type='GPU'/>
18:00:22:  <slot id='1' type='GPU'/>
18:00:22:  <slot id='2' type='GPU'/>
18:00:22:  <slot id='3' type='GPU'/>
18:00:22:</config>
18:00:52:WU00:FS00:Connecting to 65.254.110.245:80
18:00:52:WU01:FS01:Connecting to 65.254.110.245:80
[93m18:00:53:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:00:53:WU00:FS00:Connecting to 18.218.241.186:80
18:00:53:WU02:FS02:Connecting to 65.254.110.245:80
[93m18:00:53:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:00:53:WU01:FS01:Connecting to 18.218.241.186:80
18:00:53:WU03:FS03:Connecting to 65.254.110.245:80
[93m18:00:53:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
18:00:53:WU00:FS00:Connecting to 65.254.110.245:80
[93m18:00:53:WARNING:WU02:FS02:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:00:53:WU02:FS02:Connecting to 18.218.241.186:80
[93m18:00:53:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
18:00:53:WU01:FS01:Connecting to 65.254.110.245:80
[93m18:00:53:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:00:53:WU00:FS00:Connecting to 18.218.241.186:80
[93m18:00:53:WARNING:WU03:FS03:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:00:53:WU03:FS03:Connecting to 18.218.241.186:80
[93m18:00:53:WARNING:WU02:FS02:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
18:00:53:WU02:FS02:Connecting to 65.254.110.245:80
18:00:54:WU01:FS01:Assigned to work server 128.252.203.10
18:00:54:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:Ellesmere XT [Radeon RX 470/480/570/580/590] from 128.252.203.10
18:00:54:WU01:FS01:Connecting to 128.252.203.10:8080
18:00:54:WU00:FS00:Assigned to work server 128.252.203.10
18:00:54:WU00:FS00:Requesting new work unit for slot 00: READY gpu:0:Hawaii [Radeon R9 200/300X Series] from 128.252.203.10
18:00:54:WU00:FS00:Connecting to 128.252.203.10:8080
18:00:54:WU03:FS03:Assigned to work server 140.163.4.241
18:00:54:WU03:FS03:Requesting new work unit for slot 03: READY gpu:3:Ellesmere XT [Radeon RX 470/480/570/580/590] from 140.163.4.241
18:00:54:WU03:FS03:Connecting to 140.163.4.241:8080
[93m18:00:54:WARNING:WU02:FS02:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:00:54:WU02:FS02:Connecting to 18.218.241.186:80
[93m18:00:54:WARNING:WU02:FS02:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
[91m18:00:54:ERROR:WU02:FS02:Exception: Could not get an assignment[0m
18:01:40:WU01:FS01:Downloading 51.19MiB
18:01:46:WU01:FS01:Download 17.09%
18:01:52:WU01:FS01:Download 35.16%
18:01:58:WU01:FS01:Download 54.33%
18:02:04:WU01:FS01:Download 72.27%
18:02:10:WU01:FS01:Download 89.85%
18:02:14:WU03:FS03:Downloading 4.52MiB
18:02:14:WU01:FS01:Download complete
18:02:14:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:11763 run:0 clone:10408 gen:35 core:0x22 unit:0x0000004480fccb0a5e711344dcc320b6
18:02:14:WU01:FS01:Starting
18:02:14:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /home/aaabbbb/cccddd/foldingathome/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 01 -suffix 01 -version 706 -lifeline 2849 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 1 -gpu 1
18:02:14:WU01:FS01:Started FahCore on PID 3097
18:02:14:WU01:FS01:Core PID:3101
18:02:14:WU01:FS01:FahCore 0x22 started
18:02:16:WU01:FS01:0x22:*********************** Log Started 2020-04-27T18:02:15Z ***********************
18:02:16:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
18:02:16:WU01:FS01:0x22:       Type: 0x22
18:02:16:WU01:FS01:0x22:       Core: Core22
18:02:16:WU01:FS01:0x22:    Website: https://foldingathome.org/
18:02:16:WU01:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
18:02:16:WU01:FS01:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
18:02:16:WU01:FS01:0x22:             <rafal.wiewiora@choderalab.org>
18:02:16:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 3097 -checkpoint 15
18:02:16:WU01:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 1 -gpu 1
18:02:16:WU01:FS01:0x22:     Config: <none>
18:02:16:WU01:FS01:0x22:************************************ Build *************************************
18:02:16:WU01:FS01:0x22:    Version: 0.0.5
18:02:16:WU01:FS01:0x22:       Date: Apr 22 2020
18:02:16:WU01:FS01:0x22:       Time: 03:57:11
18:02:16:WU01:FS01:0x22: Repository: Git
18:02:16:WU01:FS01:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
18:02:16:WU01:FS01:0x22:     Branch: HEAD
18:02:16:WU01:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
18:02:16:WU01:FS01:0x22:    Options: -std=c++11 -O3 -funroll-loops
18:02:16:WU01:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
18:02:16:WU01:FS01:0x22:       Bits: 64
18:02:16:WU01:FS01:0x22:       Mode: Release
18:02:16:WU01:FS01:0x22:************************************ System ************************************
18:02:16:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i5-4430 CPU @ 3.00GHz
18:02:16:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
18:02:16:WU01:FS01:0x22:       CPUs: 4
18:02:16:WU01:FS01:0x22:     Memory: 7.48GiB
18:02:16:WU01:FS01:0x22:Free Memory: 3.42GiB
18:02:16:WU01:FS01:0x22:    Threads: POSIX_THREADS
18:02:16:WU01:FS01:0x22: OS Version: 4.4
18:02:16:WU01:FS01:0x22:Has Battery: false
18:02:16:WU01:FS01:0x22: On Battery: false
18:02:16:WU01:FS01:0x22: UTC Offset: 0
18:02:16:WU01:FS01:0x22:        PID: 3101
18:02:16:WU01:FS01:0x22:        CWD: /home/aaabbbb/cccddd/foldingathome/work
18:02:16:WU01:FS01:0x22:         OS: Linux 4.4.0-130-generic x86_64
18:02:16:WU01:FS01:0x22:    OS Arch: AMD64
18:02:16:WU01:FS01:0x22:********************************************************************************
18:02:16:WU01:FS01:0x22:Project: 11763 (Run 0, Clone 10408, Gen 35)
18:02:16:WU01:FS01:0x22:Unit: 0x0000004480fccb0a5e711344dcc320b6
18:02:16:WU01:FS01:0x22:Reading tar file core.xml
18:02:16:WU01:FS01:0x22:Reading tar file integrator.xml
18:02:16:WU01:FS01:0x22:Reading tar file state.xml
18:02:16:WU01:FS01:0x22:Reading tar file system.xml
18:02:16:WU01:FS01:0x22:Digital signatures verified
18:02:16:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
18:02:16:WU01:FS01:0x22:Version 0.0.5
18:02:19:WU03:FS03:Download complete
18:02:19:WU03:FS03:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:11743 run:0 clone:2700 gen:61 core:0x22 unit:0x000000608ca304f15e69d11615081141
18:02:19:WU03:FS03:Starting
18:02:19:WU03:FS03:Running FahCore: /usr/bin/FAHCoreWrapper /home/aaabbbb/cccddd/foldingathome/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 03 -suffix 01 -version 706 -lifeline 2849 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 3 -gpu 3
18:02:19:WU03:FS03:Started FahCore on PID 3112
18:02:19:WU03:FS03:Core PID:3116
18:02:19:WU03:FS03:FahCore 0x22 started
18:02:20:WU03:FS03:0x22:*********************** Log Started 2020-04-27T18:02:19Z ***********************
18:02:20:WU03:FS03:0x22:*************************** Core22 Folding@home Core ***************************
18:02:20:WU03:FS03:0x22:       Type: 0x22
18:02:20:WU03:FS03:0x22:       Core: Core22
18:02:20:WU03:FS03:0x22:    Website: https://foldingathome.org/
18:02:20:WU03:FS03:0x22:  Copyright: (c) 2009-2018 foldingathome.org
18:02:20:WU03:FS03:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
18:02:20:WU03:FS03:0x22:             <rafal.wiewiora@choderalab.org>
18:02:20:WU03:FS03:0x22:       Args: -dir 03 -suffix 01 -version 706 -lifeline 3112 -checkpoint 15
18:02:20:WU03:FS03:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 3 -gpu 3
18:02:20:WU03:FS03:0x22:     Config: <none>
18:02:20:WU03:FS03:0x22:************************************ Build *************************************
18:02:20:WU03:FS03:0x22:    Version: 0.0.5
18:02:20:WU03:FS03:0x22:       Date: Apr 22 2020
18:02:20:WU03:FS03:0x22:       Time: 03:57:11
18:02:20:WU03:FS03:0x22: Repository: Git
18:02:20:WU03:FS03:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
18:02:20:WU03:FS03:0x22:     Branch: HEAD
18:02:20:WU03:FS03:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
18:02:20:WU03:FS03:0x22:    Options: -std=c++11 -O3 -funroll-loops
18:02:20:WU03:FS03:0x22:   Platform: linux2 4.19.76-linuxkit
18:02:20:WU03:FS03:0x22:       Bits: 64
18:02:20:WU03:FS03:0x22:       Mode: Release
18:02:20:WU03:FS03:0x22:************************************ System ************************************
18:02:20:WU03:FS03:0x22:        CPU: Intel(R) Core(TM) i5-4430 CPU @ 3.00GHz
18:02:20:WU03:FS03:0x22:     CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
18:02:20:WU03:FS03:0x22:       CPUs: 4
18:02:20:WU03:FS03:0x22:     Memory: 7.48GiB
18:02:20:WU03:FS03:0x22:Free Memory: 3.06GiB
18:02:20:WU03:FS03:0x22:    Threads: POSIX_THREADS
18:02:20:WU03:FS03:0x22: OS Version: 4.4
18:02:20:WU03:FS03:0x22:Has Battery: false
18:02:20:WU03:FS03:0x22: On Battery: false
18:02:20:WU03:FS03:0x22: UTC Offset: 0
18:02:20:WU03:FS03:0x22:        PID: 3116
18:02:20:WU03:FS03:0x22:        CWD: /home/aaabbbb/cccddd/foldingathome/work
18:02:20:WU03:FS03:0x22:         OS: Linux 4.4.0-130-generic x86_64
18:02:20:WU03:FS03:0x22:    OS Arch: AMD64
18:02:20:WU03:FS03:0x22:********************************************************************************
18:02:20:WU03:FS03:0x22:Project: 11743 (Run 0, Clone 2700, Gen 61)
18:02:20:WU03:FS03:0x22:Unit: 0x000000608ca304f15e69d11615081141
18:02:20:WU03:FS03:0x22:Reading tar file core.xml
18:02:20:WU03:FS03:0x22:Reading tar file integrator.xml
18:02:20:WU03:FS03:0x22:Reading tar file state.xml
18:02:20:WU03:FS03:0x22:Reading tar file system.xml
18:02:20:WU03:FS03:0x22:Digital signatures verified
18:02:20:WU03:FS03:0x22:Folding@home GPU Core22 Folding@home Core
18:02:20:WU03:FS03:0x22:Version 0.0.5
[91m18:02:22:ERROR:WU00:FS00:Exception: 10002: Received short response, expected 512 bytes, got 0[0m
18:02:27:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
18:02:27:WU01:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
18:02:28:WU03:FS03:0x22:Completed 0 out of 2000000 steps (0%)
18:02:28:WU03:FS03:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
18:02:29:WU00:FS00:Connecting to 65.254.110.245:80
[93m18:02:30:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:02:30:WU00:FS00:Connecting to 18.218.241.186:80
18:02:30:WU02:FS02:Connecting to 65.254.110.245:80
[93m18:02:30:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
18:02:30:WU00:FS00:Connecting to 65.254.110.245:80
[93m18:02:30:WARNING:WU02:FS02:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:02:30:WU02:FS02:Connecting to 18.218.241.186:80
[93m18:02:31:WARNING:WU02:FS02:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
18:02:31:WU02:FS02:Connecting to 65.254.110.245:80
[93m18:02:31:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:02:31:WU00:FS00:Connecting to 18.218.241.186:80
[93m18:02:31:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
[91m18:02:31:ERROR:WU00:FS00:Exception: Could not get an assignment[0m
[93m18:02:31:WARNING:WU02:FS02:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:02:31:WU02:FS02:Connecting to 18.218.241.186:80
18:02:32:WU02:FS02:Assigned to work server 13.82.98.119
18:02:32:WU02:FS02:Requesting new work unit for slot 02: READY gpu:2:Ellesmere XT [Radeon RX 470/480/570/580/590] from 13.82.98.119
18:02:32:WU02:FS02:Connecting to 13.82.98.119:8080
18:02:32:WU02:FS02:Downloading 13.26MiB
18:02:38:WU02:FS02:Download 72.11%
18:02:40:WU02:FS02:Download complete
18:02:40:WU02:FS02:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:14538 run:0 clone:1664 gen:82 core:0x22 unit:0x0000006c0d5262775e7b9cdbfcff3d82
18:02:40:WU02:FS02:Starting
18:02:40:WU02:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /home/aaabbbb/cccddd/foldingathome/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 02 -suffix 01 -version 706 -lifeline 2849 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 2 -gpu 2
18:02:40:WU02:FS02:Started FahCore on PID 3177
18:02:40:WU02:FS02:Core PID:3181
18:02:40:WU02:FS02:FahCore 0x22 started
18:02:40:WU02:FS02:0x22:*********************** Log Started 2020-04-27T18:02:40Z ***********************
18:02:40:WU02:FS02:0x22:*************************** Core22 Folding@home Core ***************************
18:02:40:WU02:FS02:0x22:       Type: 0x22
18:02:40:WU02:FS02:0x22:       Core: Core22
18:02:40:WU02:FS02:0x22:    Website: https://foldingathome.org/
18:02:40:WU02:FS02:0x22:  Copyright: (c) 2009-2018 foldingathome.org
18:02:40:WU02:FS02:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
18:02:40:WU02:FS02:0x22:             <rafal.wiewiora@choderalab.org>
18:02:40:WU02:FS02:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 3177 -checkpoint 15
18:02:40:WU02:FS02:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 2 -gpu 2
18:02:40:WU02:FS02:0x22:     Config: <none>
18:02:40:WU02:FS02:0x22:************************************ Build *************************************
18:02:40:WU02:FS02:0x22:    Version: 0.0.5
18:02:40:WU02:FS02:0x22:       Date: Apr 22 2020
18:02:40:WU02:FS02:0x22:       Time: 03:57:11
18:02:40:WU02:FS02:0x22: Repository: Git
18:02:40:WU02:FS02:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
18:02:40:WU02:FS02:0x22:     Branch: HEAD
18:02:40:WU02:FS02:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
18:02:40:WU02:FS02:0x22:    Options: -std=c++11 -O3 -funroll-loops
18:02:40:WU02:FS02:0x22:   Platform: linux2 4.19.76-linuxkit
18:02:40:WU02:FS02:0x22:       Bits: 64
18:02:40:WU02:FS02:0x22:       Mode: Release
18:02:40:WU02:FS02:0x22:************************************ System ************************************
18:02:40:WU02:FS02:0x22:        CPU: Intel(R) Core(TM) i5-4430 CPU @ 3.00GHz
18:02:40:WU02:FS02:0x22:     CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
18:02:40:WU02:FS02:0x22:       CPUs: 4
18:02:40:WU02:FS02:0x22:     Memory: 7.48GiB
18:02:40:WU02:FS02:0x22:Free Memory: 2.60GiB
18:02:40:WU02:FS02:0x22:    Threads: POSIX_THREADS
18:02:40:WU02:FS02:0x22: OS Version: 4.4
18:02:40:WU02:FS02:0x22:Has Battery: false
18:02:40:WU02:FS02:0x22: On Battery: false
18:02:40:WU02:FS02:0x22: UTC Offset: 0
18:02:40:WU02:FS02:0x22:        PID: 3181
18:02:40:WU02:FS02:0x22:        CWD: /home/aaabbbb/cccddd/foldingathome/work
18:02:40:WU02:FS02:0x22:         OS: Linux 4.4.0-130-generic x86_64
18:02:40:WU02:FS02:0x22:    OS Arch: AMD64
18:02:40:WU02:FS02:0x22:********************************************************************************
18:02:40:WU02:FS02:0x22:Project: 14538 (Run 0, Clone 1664, Gen 82)
18:02:40:WU02:FS02:0x22:Unit: 0x0000006c0d5262775e7b9cdbfcff3d82
18:02:40:WU02:FS02:0x22:Reading tar file core.xml
18:02:40:WU02:FS02:0x22:Reading tar file integrator.xml
18:02:40:WU02:FS02:0x22:Reading tar file state.xml
18:02:40:WU02:FS02:0x22:Reading tar file system.xml
18:02:40:WU02:FS02:0x22:Digital signatures verified
18:02:40:WU02:FS02:0x22:Folding@home GPU Core22 Folding@home Core
18:02:40:WU02:FS02:0x22:Version 0.0.5
18:02:45:WU02:FS02:0x22:Completed 0 out of 4000000 steps (0%)
18:02:45:WU02:FS02:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
18:04:15:WU01:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
18:04:45:WU03:FS03:0x22:Completed 20000 out of 2000000 steps (1%)
18:05:07:WU00:FS00:Connecting to 65.254.110.245:80
[93m18:05:07:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:05:07:WU00:FS00:Connecting to 18.218.241.186:80
18:06:02:WU01:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
18:07:00:WU03:FS03:0x22:Completed 40000 out of 2000000 steps (2%)
[93m18:07:14:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': Failed to connect to 18.218.241.186:80: Connection timed out[0m
18:07:14:WU00:FS00:Connecting to 65.254.110.245:80
[93m18:07:15:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:07:15:WU00:FS00:Connecting to 18.218.241.186:80
[93m18:07:15:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
[91m18:07:15:ERROR:WU00:FS00:Exception: Could not get an assignment[0m
18:07:47:WU01:FS01:0x22:Completed 30000 out of 1000000 steps (3%)
18:09:17:WU03:FS03:0x22:Completed 60000 out of 2000000 steps (3%)
18:09:21:WU00:FS00:Connecting to 65.254.110.245:80
[93m18:09:21:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:09:21:WU00:FS00:Connecting to 18.218.241.186:80
[93m18:09:22:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
18:09:22:WU00:FS00:Connecting to 65.254.110.245:80
[93m18:09:22:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:09:22:WU00:FS00:Connecting to 18.218.241.186:80
[93m18:09:23:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
[91m18:09:23:ERROR:WU00:FS00:Exception: Could not get an assignment[0m
18:09:33:WU01:FS01:0x22:Completed 40000 out of 1000000 steps (4%)
18:11:19:WU01:FS01:0x22:Completed 50000 out of 1000000 steps (5%)
18:11:34:WU03:FS03:0x22:Completed 80000 out of 2000000 steps (4%)
18:13:08:WU01:FS01:0x22:Completed 60000 out of 1000000 steps (6%)
18:13:49:WU03:FS03:0x22:Completed 100000 out of 2000000 steps (5%)
18:14:54:WU01:FS01:0x22:Completed 70000 out of 1000000 steps (7%)
18:16:06:WU03:FS03:0x22:Completed 120000 out of 2000000 steps (6%)
18:16:12:WU00:FS00:Connecting to 65.254.110.245:80
[93m18:16:13:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:16:13:WU00:FS00:Connecting to 18.218.241.186:80
[93m18:16:13:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
18:16:13:WU00:FS00:Connecting to 65.254.110.245:80
[93m18:16:14:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:16:14:WU00:FS00:Connecting to 18.218.241.186:80
[93m18:16:14:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
[91m18:16:14:ERROR:WU00:FS00:Exception: Could not get an assignment[0m
18:16:40:WU01:FS01:0x22:Completed 80000 out of 1000000 steps (8%)
18:18:23:WU03:FS03:0x22:Completed 140000 out of 2000000 steps (7%)
18:18:26:WU01:FS01:0x22:Completed 90000 out of 1000000 steps (9%)
18:20:12:WU01:FS01:0x22:Completed 100000 out of 1000000 steps (10%)
18:20:40:WU03:FS03:0x22:Completed 160000 out of 2000000 steps (8%)
18:22:01:WU01:FS01:0x22:Completed 110000 out of 1000000 steps (11%)
18:22:54:WU03:FS03:0x22:Completed 180000 out of 2000000 steps (9%)
18:23:47:WU01:FS01:0x22:Completed 120000 out of 1000000 steps (12%)
18:25:08:WU03:FS03:0x22:Completed 200000 out of 2000000 steps (10%)
18:25:34:WU01:FS01:0x22:Completed 130000 out of 1000000 steps (13%)
18:27:18:WU00:FS00:Connecting to 65.254.110.245:80
[93m18:27:18:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:27:18:WU00:FS00:Connecting to 18.218.241.186:80
[93m18:27:19:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
18:27:19:WU00:FS00:Connecting to 65.254.110.245:80
[93m18:27:19:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:27:19:WU00:FS00:Connecting to 18.218.241.186:80
[93m18:27:20:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
[91m18:27:20:ERROR:WU00:FS00:Exception: Could not get an assignment[0m
18:27:20:WU01:FS01:0x22:Completed 140000 out of 1000000 steps (14%)
18:27:25:WU03:FS03:0x22:Completed 220000 out of 2000000 steps (11%)
18:29:06:WU01:FS01:0x22:Completed 150000 out of 1000000 steps (15%)
18:29:42:WU03:FS03:0x22:Completed 240000 out of 2000000 steps (12%)
18:30:55:WU01:FS01:0x22:Completed 160000 out of 1000000 steps (16%)
18:32:00:WU03:FS03:0x22:Completed 260000 out of 2000000 steps (13%)
18:32:42:WU01:FS01:0x22:Completed 170000 out of 1000000 steps (17%)
18:34:16:WU03:FS03:0x22:Completed 280000 out of 2000000 steps (14%)
18:34:27:WU01:FS01:0x22:Completed 180000 out of 1000000 steps (18%)
18:36:13:WU01:FS01:0x22:Completed 190000 out of 1000000 steps (19%)
18:36:33:WU03:FS03:0x22:Completed 300000 out of 2000000 steps (15%)
18:37:59:WU01:FS01:0x22:Completed 200000 out of 1000000 steps (20%)
18:38:50:WU03:FS03:0x22:Completed 320000 out of 2000000 steps (16%)
18:39:47:WU01:FS01:0x22:Completed 210000 out of 1000000 steps (21%)
18:41:03:WU03:FS03:0x22:Completed 340000 out of 2000000 steps (17%)
18:41:33:WU01:FS01:0x22:Completed 220000 out of 1000000 steps (22%)
18:43:18:WU03:FS03:0x22:Completed 360000 out of 2000000 steps (18%)
18:43:19:WU01:FS01:0x22:Completed 230000 out of 1000000 steps (23%)
18:45:04:WU01:FS01:0x22:Completed 240000 out of 1000000 steps (24%)
18:45:15:WU00:FS00:Connecting to 65.254.110.245:80
[93m18:45:15:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:45:15:WU00:FS00:Connecting to 18.218.241.186:80
[93m18:45:16:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
18:45:16:WU00:FS00:Connecting to 65.254.110.245:80
[93m18:45:16:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
18:45:16:WU00:FS00:Connecting to 18.218.241.186:80
[93m18:45:17:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
[91m18:45:17:ERROR:WU00:FS00:Exception: Could not get an assignment[0m
18:45:33:WU03:FS03:0x22:Completed 380000 out of 2000000 steps (19%)
18:46:50:WU01:FS01:0x22:Completed 250000 out of 1000000 steps (25%)
18:47:50:WU03:FS03:0x22:Completed 400000 out of 2000000 steps (20%)
18:48:39:WU01:FS01:0x22:Completed 260000 out of 1000000 steps (26%)
18:50:08:WU03:FS03:0x22:Completed 420000 out of 2000000 steps (21%)
18:50:25:WU01:FS01:0x22:Completed 270000 out of 1000000 steps (27%)
18:52:12:WU01:FS01:0x22:Completed 280000 out of 1000000 steps (28%)
18:52:25:WU03:FS03:0x22:Completed 440000 out of 2000000 steps (22%)
18:53:58:WU01:FS01:0x22:Completed 290000 out of 1000000 steps (29%)
18:54:40:WU03:FS03:0x22:Completed 460000 out of 2000000 steps (23%)
18:55:44:WU01:FS01:0x22:Completed 300000 out of 1000000 steps (30%)
18:56:54:WU03:FS03:0x22:Completed 480000 out of 2000000 steps (24%)
18:57:32:WU01:FS01:0x22:Completed 310000 out of 1000000 steps (31%)
18:59:09:WU03:FS03:0x22:Completed 500000 out of 2000000 steps (25%)
18:59:19:WU01:FS01:0x22:Completed 320000 out of 1000000 steps (32%)
19:01:05:WU01:FS01:0x22:Completed 330000 out of 1000000 steps (33%)
19:01:24:WU03:FS03:0x22:Completed 520000 out of 2000000 steps (26%)
19:02:51:WU01:FS01:0x22:Completed 340000 out of 1000000 steps (34%)
19:03:38:WU03:FS03:0x22:Completed 540000 out of 2000000 steps (27%)
19:04:36:WU01:FS01:0x22:Completed 350000 out of 1000000 steps (35%)
19:05:53:WU03:FS03:0x22:Completed 560000 out of 2000000 steps (28%)
19:06:25:WU01:FS01:0x22:Completed 360000 out of 1000000 steps (36%)
19:08:06:WU03:FS03:0x22:Completed 580000 out of 2000000 steps (29%)
19:08:10:WU01:FS01:0x22:Completed 370000 out of 1000000 steps (37%)
19:09:56:WU01:FS01:0x22:Completed 380000 out of 1000000 steps (38%)
19:10:20:WU03:FS03:0x22:Completed 600000 out of 2000000 steps (30%)
19:11:42:WU01:FS01:0x22:Completed 390000 out of 1000000 steps (39%)
19:12:36:WU03:FS03:0x22:Completed 620000 out of 2000000 steps (31%)
19:13:28:WU01:FS01:0x22:Completed 400000 out of 1000000 steps (40%)
19:14:17:WU00:FS00:Connecting to 65.254.110.245:80
[93m19:14:17:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
19:14:17:WU00:FS00:Connecting to 18.218.241.186:80
19:14:18:WU00:FS00:Assigned to work server 128.252.203.10
19:14:18:WU00:FS00:Requesting new work unit for slot 00: READY gpu:0:Hawaii [Radeon R9 200/300X Series] from 128.252.203.10
19:14:18:WU00:FS00:Connecting to 128.252.203.10:8080
19:14:52:WU03:FS03:0x22:Completed 640000 out of 2000000 steps (32%)
19:15:14:WU00:FS00:Downloading 50.73MiB
19:15:16:WU01:FS01:0x22:Completed 410000 out of 1000000 steps (41%)
19:15:20:WU00:FS00:Download 16.01%
19:15:26:WU00:FS00:Download 36.34%
19:15:32:WU00:FS00:Download 52.97%
19:15:38:WU00:FS00:Download 68.74%
19:15:44:WU00:FS00:Download 87.71%
19:15:48:WU00:FS00:Download complete
19:15:48:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:11762 run:0 clone:6945 gen:50 core:0x22 unit:0x0000005f80fccb0a5e7113db6c6a06e3
19:15:48:WU00:FS00:Starting
19:15:48:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /home/aaabbbb/cccddd/foldingathome/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 00 -suffix 01 -version 706 -lifeline 2849 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
19:15:48:WU00:FS00:Started FahCore on PID 9526
19:15:48:WU00:FS00:Core PID:9530
19:15:48:WU00:FS00:FahCore 0x22 started
19:15:48:WU00:FS00:0x22:*********************** Log Started 2020-04-27T19:15:48Z ***********************
19:15:48:WU00:FS00:0x22:*************************** Core22 Folding@home Core ***************************
19:15:48:WU00:FS00:0x22:       Type: 0x22
19:15:48:WU00:FS00:0x22:       Core: Core22
19:15:48:WU00:FS00:0x22:    Website: https://foldingathome.org/
19:15:48:WU00:FS00:0x22:  Copyright: (c) 2009-2018 foldingathome.org
19:15:48:WU00:FS00:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
19:15:48:WU00:FS00:0x22:             <rafal.wiewiora@choderalab.org>
19:15:48:WU00:FS00:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 9526 -checkpoint 15
19:15:48:WU00:FS00:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
19:15:48:WU00:FS00:0x22:     Config: <none>
19:15:48:WU00:FS00:0x22:************************************ Build *************************************
19:15:48:WU00:FS00:0x22:    Version: 0.0.5
19:15:48:WU00:FS00:0x22:       Date: Apr 22 2020
19:15:48:WU00:FS00:0x22:       Time: 03:57:11
19:15:48:WU00:FS00:0x22: Repository: Git
19:15:48:WU00:FS00:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
19:15:48:WU00:FS00:0x22:     Branch: HEAD
19:15:48:WU00:FS00:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
19:15:48:WU00:FS00:0x22:    Options: -std=c++11 -O3 -funroll-loops
19:15:48:WU00:FS00:0x22:   Platform: linux2 4.19.76-linuxkit
19:15:48:WU00:FS00:0x22:       Bits: 64
19:15:48:WU00:FS00:0x22:       Mode: Release
19:15:48:WU00:FS00:0x22:************************************ System ************************************
19:15:48:WU00:FS00:0x22:        CPU: Intel(R) Core(TM) i5-4430 CPU @ 3.00GHz
19:15:48:WU00:FS00:0x22:     CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
19:15:48:WU00:FS00:0x22:       CPUs: 4
19:15:48:WU00:FS00:0x22:     Memory: 7.48GiB
19:15:48:WU00:FS00:0x22:Free Memory: 2.10GiB
19:15:48:WU00:FS00:0x22:    Threads: POSIX_THREADS
19:15:48:WU00:FS00:0x22: OS Version: 4.4
19:15:48:WU00:FS00:0x22:Has Battery: false
19:15:48:WU00:FS00:0x22: On Battery: false
19:15:48:WU00:FS00:0x22: UTC Offset: 0
19:15:48:WU00:FS00:0x22:        PID: 9530
19:15:48:WU00:FS00:0x22:        CWD: /home/aaabbbb/cccddd/foldingathome/work
19:15:48:WU00:FS00:0x22:         OS: Linux 4.4.0-130-generic x86_64
19:15:48:WU00:FS00:0x22:    OS Arch: AMD64
19:15:48:WU00:FS00:0x22:********************************************************************************
19:15:48:WU00:FS00:0x22:Project: 11762 (Run 0, Clone 6945, Gen 50)
19:15:48:WU00:FS00:0x22:Unit: 0x0000005f80fccb0a5e7113db6c6a06e3
19:15:48:WU00:FS00:0x22:Reading tar file core.xml
19:15:48:WU00:FS00:0x22:Reading tar file integrator.xml
19:15:48:WU00:FS00:0x22:Reading tar file state.xml
19:15:48:WU00:FS00:0x22:Reading tar file system.xml
19:15:48:WU00:FS00:0x22:Digital signatures verified
19:15:48:WU00:FS00:0x22:Folding@home GPU Core22 Folding@home Core
19:15:48:WU00:FS00:0x22:Version 0.0.5
19:17:03:WU01:FS01:0x22:Completed 420000 out of 1000000 steps (42%)
19:17:08:WU03:FS03:0x22:Completed 660000 out of 2000000 steps (33%)
19:18:50:WU01:FS01:0x22:Completed 430000 out of 1000000 steps (43%)
19:19:23:WU03:FS03:0x22:Completed 680000 out of 2000000 steps (34%)
19:20:37:WU01:FS01:0x22:Completed 440000 out of 1000000 steps (44%)
19:21:38:WU03:FS03:0x22:Completed 700000 out of 2000000 steps (35%)
19:22:24:WU01:FS01:0x22:Completed 450000 out of 1000000 steps (45%)
19:23:53:WU03:FS03:0x22:Completed 720000 out of 2000000 steps (36%)
19:24:12:WU01:FS01:0x22:Completed 460000 out of 1000000 steps (46%)
19:25:59:WU01:FS01:0x22:Completed 470000 out of 1000000 steps (47%)
19:26:07:WU03:FS03:0x22:Completed 740000 out of 2000000 steps (37%)
19:27:45:WU01:FS01:0x22:Completed 480000 out of 1000000 steps (48%)
19:28:25:WU03:FS03:0x22:Completed 760000 out of 2000000 steps (38%)
19:29:31:WU01:FS01:0x22:Completed 490000 out of 1000000 steps (49%)
19:30:42:WU03:FS03:0x22:Completed 780000 out of 2000000 steps (39%)
19:31:17:WU01:FS01:0x22:Completed 500000 out of 1000000 steps (50%)
19:32:58:WU03:FS03:0x22:Completed 800000 out of 2000000 steps (40%)
19:33:05:WU01:FS01:0x22:Completed 510000 out of 1000000 steps (51%)
19:34:52:WU01:FS01:0x22:Completed 520000 out of 1000000 steps (52%)
19:35:16:WU03:FS03:0x22:Completed 820000 out of 2000000 steps (41%)
19:36:38:WU01:FS01:0x22:Completed 530000 out of 1000000 steps (53%)
19:37:31:WU03:FS03:0x22:Completed 840000 out of 2000000 steps (42%)
19:38:25:WU01:FS01:0x22:Completed 540000 out of 1000000 steps (54%)
19:39:47:WU03:FS03:0x22:Completed 860000 out of 2000000 steps (43%)
19:40:11:WU01:FS01:0x22:Completed 550000 out of 1000000 steps (55%)
19:42:00:WU01:FS01:0x22:Completed 560000 out of 1000000 steps (56%)
19:42:03:WU03:FS03:0x22:Completed 880000 out of 2000000 steps (44%)
19:43:46:WU01:FS01:0x22:Completed 570000 out of 1000000 steps (57%)
19:44:19:WU03:FS03:0x22:Completed 900000 out of 2000000 steps (45%)
19:45:32:WU01:FS01:0x22:Completed 580000 out of 1000000 steps (58%)
19:46:37:WU03:FS03:0x22:Completed 920000 out of 2000000 steps (46%)
19:47:18:WU01:FS01:0x22:Completed 590000 out of 1000000 steps (59%)
19:48:51:WU03:FS03:0x22:Completed 940000 out of 2000000 steps (47%)
19:49:04:WU01:FS01:0x22:Completed 600000 out of 1000000 steps (60%)
19:50:53:WU01:FS01:0x22:Completed 610000 out of 1000000 steps (61%)
19:51:08:WU03:FS03:0x22:Completed 960000 out of 2000000 steps (48%)
19:52:38:WU01:FS01:0x22:Completed 620000 out of 1000000 steps (62%)
19:53:22:WU03:FS03:0x22:Completed 980000 out of 2000000 steps (49%)
19:54:26:WU01:FS01:0x22:Completed 630000 out of 1000000 steps (63%)
19:55:37:WU03:FS03:0x22:Completed 1000000 out of 2000000 steps (50%)
19:56:13:WU01:FS01:0x22:Completed 640000 out of 1000000 steps (64%)
19:57:52:WU03:FS03:0x22:Completed 1020000 out of 2000000 steps (51%)
19:57:58:WU01:FS01:0x22:Completed 650000 out of 1000000 steps (65%)
19:59:47:WU01:FS01:0x22:Completed 660000 out of 1000000 steps (66%)
20:00:06:WU03:FS03:0x22:Completed 1040000 out of 2000000 steps (52%)
20:01:33:WU01:FS01:0x22:Completed 670000 out of 1000000 steps (67%)
20:02:23:WU03:FS03:0x22:Completed 1060000 out of 2000000 steps (53%)
20:03:19:WU01:FS01:0x22:Completed 680000 out of 1000000 steps (68%)
20:04:39:WU03:FS03:0x22:Completed 1080000 out of 2000000 steps (54%)
20:05:05:WU01:FS01:0x22:Completed 690000 out of 1000000 steps (69%)
20:06:50:WU01:FS01:0x22:Completed 700000 out of 1000000 steps (70%)
20:06:54:WU03:FS03:0x22:Completed 1100000 out of 2000000 steps (55%)
20:08:39:WU01:FS01:0x22:Completed 710000 out of 1000000 steps (71%)
20:09:11:WU03:FS03:0x22:Completed 1120000 out of 2000000 steps (56%)
20:10:25:WU01:FS01:0x22:Completed 720000 out of 1000000 steps (72%)
20:11:27:WU03:FS03:0x22:Completed 1140000 out of 2000000 steps (57%)
20:12:11:WU01:FS01:0x22:Completed 730000 out of 1000000 steps (73%)
20:13:45:WU03:FS03:0x22:Completed 1160000 out of 2000000 steps (58%)
20:13:57:WU01:FS01:0x22:Completed 740000 out of 1000000 steps (74%)
20:15:42:WU01:FS01:0x22:Completed 750000 out of 1000000 steps (75%)
20:16:01:WU03:FS03:0x22:Completed 1180000 out of 2000000 steps (59%)
20:17:31:WU01:FS01:0x22:Completed 760000 out of 1000000 steps (76%)
20:18:17:WU03:FS03:0x22:Completed 1200000 out of 2000000 steps (60%)
20:19:16:WU01:FS01:0x22:Completed 770000 out of 1000000 steps (77%)
20:20:35:WU03:FS03:0x22:Completed 1220000 out of 2000000 steps (61%)
20:21:02:WU01:FS01:0x22:Completed 780000 out of 1000000 steps (78%)
20:22:48:WU01:FS01:0x22:Completed 790000 out of 1000000 steps (79%)
20:22:51:WU03:FS03:0x22:Completed 1240000 out of 2000000 steps (62%)
20:24:35:WU01:FS01:0x22:Completed 800000 out of 1000000 steps (80%)
20:25:09:WU03:FS03:0x22:Completed 1260000 out of 2000000 steps (63%)
20:26:23:WU01:FS01:0x22:Completed 810000 out of 1000000 steps (81%)
20:27:25:WU03:FS03:0x22:Completed 1280000 out of 2000000 steps (64%)
20:28:09:WU01:FS01:0x22:Completed 820000 out of 1000000 steps (82%)
20:29:42:WU03:FS03:0x22:Completed 1300000 out of 2000000 steps (65%)
20:29:55:WU01:FS01:0x22:Completed 830000 out of 1000000 steps (83%)
20:31:41:WU01:FS01:0x22:Completed 840000 out of 1000000 steps (84%)
20:31:58:WU03:FS03:0x22:Completed 1320000 out of 2000000 steps (66%)
20:33:27:WU01:FS01:0x22:Completed 850000 out of 1000000 steps (85%)
20:34:13:WU03:FS03:0x22:Completed 1340000 out of 2000000 steps (67%)
20:35:15:WU01:FS01:0x22:Completed 860000 out of 1000000 steps (86%)
20:36:28:WU03:FS03:0x22:Completed 1360000 out of 2000000 steps (68%)
20:37:01:WU01:FS01:0x22:Completed 870000 out of 1000000 steps (87%)
20:38:43:WU03:FS03:0x22:Completed 1380000 out of 2000000 steps (69%)
20:38:47:WU01:FS01:0x22:Completed 880000 out of 1000000 steps (88%)
20:40:33:WU01:FS01:0x22:Completed 890000 out of 1000000 steps (89%)
20:41:00:WU03:FS03:0x22:Completed 1400000 out of 2000000 steps (70%)
20:42:19:WU01:FS01:0x22:Completed 900000 out of 1000000 steps (90%)
20:43:18:WU03:FS03:0x22:Completed 1420000 out of 2000000 steps (71%)
20:44:08:WU01:FS01:0x22:Completed 910000 out of 1000000 steps (91%)
20:45:35:WU03:FS03:0x22:Completed 1440000 out of 2000000 steps (72%)
20:45:55:WU01:FS01:0x22:Completed 920000 out of 1000000 steps (92%)
20:47:41:WU01:FS01:0x22:Completed 930000 out of 1000000 steps (93%)
20:47:53:WU03:FS03:0x22:Completed 1460000 out of 2000000 steps (73%)
20:49:27:WU01:FS01:0x22:Completed 940000 out of 1000000 steps (94%)
20:50:06:WU03:FS03:0x22:Completed 1480000 out of 2000000 steps (74%)
20:51:13:WU01:FS01:0x22:Completed 950000 out of 1000000 steps (95%)
20:52:20:WU03:FS03:0x22:Completed 1500000 out of 2000000 steps (75%)
20:53:03:WU01:FS01:0x22:Completed 960000 out of 1000000 steps (96%)
20:54:37:WU03:FS03:0x22:Completed 1520000 out of 2000000 steps (76%)
20:54:50:WU01:FS01:0x22:Completed 970000 out of 1000000 steps (97%)
20:56:36:WU01:FS01:0x22:Completed 980000 out of 1000000 steps (98%)
20:56:52:WU03:FS03:0x22:Completed 1540000 out of 2000000 steps (77%)
20:58:21:WU01:FS01:0x22:Completed 990000 out of 1000000 steps (99%)
20:58:22:WU04:FS01:Connecting to 65.254.110.245:80
[93m20:58:22:WARNING:WU04:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
20:58:22:WU04:FS01:Connecting to 18.218.241.186:80
20:58:23:WU04:FS01:Assigned to work server 140.163.4.231
20:58:23:WU04:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:Ellesmere XT [Radeon RX 470/480/570/580/590] from 140.163.4.231
20:58:23:WU04:FS01:Connecting to 140.163.4.231:8080
20:59:08:WU03:FS03:0x22:Completed 1560000 out of 2000000 steps (78%)
21:00:07:WU01:FS01:0x22:Completed 1000000 out of 1000000 steps (100%)
21:00:10:WU01:FS01:0x22:Saving result file ../logfile_01.txt
21:00:10:WU01:FS01:0x22:Saving result file checkpointState.xml
21:00:10:WU01:FS01:0x22:Saving result file checkpt.crc
21:00:10:WU01:FS01:0x22:Saving result file positions.xtc
21:00:10:WU01:FS01:0x22:Saving result file science.log
21:00:10:WU01:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
[91m21:00:28:ERROR:WU04:FS01:Exception: 10002: Received short response, expected 512 bytes, got 0[0m
21:00:29:WU04:FS01:Connecting to 65.254.110.245:80
[93m21:00:29:WARNING:WU04:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
21:00:29:WU04:FS01:Connecting to 18.218.241.186:80
[93m21:00:30:WARNING:WU04:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
21:00:30:WU04:FS01:Connecting to 65.254.110.245:80
[93m21:00:30:WARNING:WU04:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
21:00:30:WU04:FS01:Connecting to 18.218.241.186:80
21:01:22:WU03:FS03:0x22:Completed 1580000 out of 2000000 steps (79%)
[93m21:02:37:WARNING:WU04:FS01:Failed to get assignment from '18.218.241.186:80': Failed to connect to 18.218.241.186:80: Connection timed out[0m
[91m21:02:37:ERROR:WU04:FS01:Exception: Could not get an assignment[0m
21:02:37:WU04:FS01:Connecting to 65.254.110.245:80
21:02:38:WU04:FS01:Assigned to work server 140.163.4.231
21:02:38:WU04:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:Ellesmere XT [Radeon RX 470/480/570/580/590] from 140.163.4.231
21:02:38:WU04:FS01:Connecting to 140.163.4.231:8080
21:03:35:WU03:FS03:0x22:Completed 1600000 out of 2000000 steps (80%)
[93m21:04:45:WARNING:WU04:FS01:WorkServer connection failed on port 8080 trying 80[0m
21:04:45:WU04:FS01:Connecting to 140.163.4.231:80
21:05:50:WU03:FS03:0x22:Completed 1620000 out of 2000000 steps (81%)
[91m21:06:20:ERROR:WU04:FS01:Exception: 10002: Received short response, expected 512 bytes, got 0[0m
21:06:20:WU04:FS01:Connecting to 65.254.110.245:80
[93m21:06:21:WARNING:WU04:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
21:06:21:WU04:FS01:Connecting to 18.218.241.186:80
[93m21:06:21:WARNING:WU04:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
21:06:21:WU04:FS01:Connecting to 65.254.110.245:80
21:06:22:WU04:FS01:Assigned to work server 128.252.203.10
21:06:22:WU04:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:Ellesmere XT [Radeon RX 470/480/570/580/590] from 128.252.203.10
21:06:22:WU04:FS01:Connecting to 128.252.203.10:8080
21:07:21:WU04:FS01:Downloading 51.19MiB
21:07:27:WU04:FS01:Download 12.21%
21:07:33:WU04:FS01:Download 28.45%
21:07:39:WU04:FS01:Download 43.95%
21:07:45:WU04:FS01:Download 58.48%
21:07:51:WU04:FS01:Download 65.07%
21:07:57:WU04:FS01:Download 78.99%
21:08:03:WU04:FS01:Download 93.64%
21:08:03:WU03:FS03:0x22:Completed 1640000 out of 2000000 steps (82%)
21:08:05:WU04:FS01:Download complete
21:08:05:WU04:FS01:Received Unit: id:04 state:DOWNLOAD error:NO_ERROR project:11763 run:0 clone:14400 gen:9 core:0x22 unit:0x0000001980fccb0a5e7113189522230e
21:10:18:WU03:FS03:0x22:Completed 1660000 out of 2000000 steps (83%)
21:12:31:WU03:FS03:0x22:Completed 1680000 out of 2000000 steps (84%)
21:14:44:WU03:FS03:0x22:Completed 1700000 out of 2000000 steps (85%)
21:16:59:WU03:FS03:0x22:Completed 1720000 out of 2000000 steps (86%)
21:19:12:WU03:FS03:0x22:Completed 1740000 out of 2000000 steps (87%)
21:21:26:WU03:FS03:0x22:Completed 1760000 out of 2000000 steps (88%)
21:23:40:WU03:FS03:0x22:Completed 1780000 out of 2000000 steps (89%)
21:25:54:WU03:FS03:0x22:Completed 1800000 out of 2000000 steps (90%)
21:28:12:WU03:FS03:0x22:Completed 1820000 out of 2000000 steps (91%)
21:30:29:WU03:FS03:0x22:Completed 1840000 out of 2000000 steps (92%)
21:32:46:WU03:FS03:0x22:Completed 1860000 out of 2000000 steps (93%)
21:35:02:WU03:FS03:0x22:Completed 1880000 out of 2000000 steps (94%)
21:37:16:WU03:FS03:0x22:Completed 1900000 out of 2000000 steps (95%)
21:39:31:WU03:FS03:0x22:Completed 1920000 out of 2000000 steps (96%)
21:41:45:WU03:FS03:0x22:Completed 1940000 out of 2000000 steps (97%)
21:43:59:WU03:FS03:0x22:Completed 1960000 out of 2000000 steps (98%)
21:46:12:WU03:FS03:0x22:Completed 1980000 out of 2000000 steps (99%)
21:46:13:WU05:FS03:Connecting to 65.254.110.245:80
[93m21:46:14:WARNING:WU05:FS03:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
21:46:14:WU05:FS03:Connecting to 18.218.241.186:80
[93m21:48:21:WARNING:WU05:FS03:Failed to get assignment from '18.218.241.186:80': Failed to connect to 18.218.241.186:80: Connection timed out[0m
21:48:21:WU05:FS03:Connecting to 65.254.110.245:80
[93m21:48:21:WARNING:WU05:FS03:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
21:48:21:WU05:FS03:Connecting to 18.218.241.186:80
21:48:26:WU03:FS03:0x22:Completed 2000000 out of 2000000 steps (100%)
21:48:27:WU03:FS03:0x22:Saving result file ../logfile_01.txt
21:48:27:WU03:FS03:0x22:Saving result file checkpointState.xml
21:48:29:WU03:FS03:0x22:Saving result file checkpt.crc
21:48:29:WU03:FS03:0x22:Saving result file positions.xtc
21:48:30:WU03:FS03:0x22:Saving result file science.log
21:48:30:WU03:FS03:0x22:Folding@home Core Shutdown: FINISHED_UNIT
[93m21:50:29:WARNING:WU05:FS03:Failed to get assignment from '18.218.241.186:80': Failed to connect to 18.218.241.186:80: Connection timed out[0m
[91m21:50:29:ERROR:WU05:FS03:Exception: Could not get an assignment[0m
21:50:29:WU05:FS03:Connecting to 65.254.110.245:80
[93m21:50:29:WARNING:WU05:FS03:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
21:50:29:WU05:FS03:Connecting to 18.218.241.186:80
[93m21:50:30:WARNING:WU05:FS03:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
21:50:30:WU05:FS03:Connecting to 65.254.110.245:80
[93m21:50:30:WARNING:WU05:FS03:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
21:50:30:WU05:FS03:Connecting to 18.218.241.186:80
[93m21:50:31:WARNING:WU05:FS03:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
[91m21:50:31:ERROR:WU05:FS03:Exception: Could not get an assignment[0m
21:51:29:WU05:FS03:Connecting to 65.254.110.245:80
[93m21:51:29:WARNING:WU05:FS03:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
21:51:29:WU05:FS03:Connecting to 18.218.241.186:80
[93m21:51:30:WARNING:WU05:FS03:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
21:51:30:WU05:FS03:Connecting to 65.254.110.245:80
[93m21:51:30:WARNING:WU05:FS03:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration[0m
21:51:30:WU05:FS03:Connecting to 18.218.241.186:80
21:53:20:FS00:Paused
21:53:20:FS01:Paused
21:53:20:FS02:Paused
21:53:20:FS03:Paused
21:53:20:FS00:Shutting core down
21:53:20:FS01:Shutting core down
21:53:20:FS02:Shutting core down
21:53:20:FS03:Shutting core down
21:53:20:WU00:FS00:0x22:Caught signal SIGINT(2) on PID 9530
21:53:20:WU00:FS00:0x22:Exiting, please wait. . .
[93m21:53:38:WARNING:WU05:FS03:Failed to get assignment from '18.218.241.186:80': Failed to connect to 18.218.241.186:80: Connection timed out[0m
[91m21:53:38:ERROR:WU05:FS03:Exception: Could not get an assignment[0m
21:54:12:Saving configuration to /etc/fahclient/config.xml
21:54:12:<config>
21:54:12:  <!-- Folding Core -->
21:54:12:  <cpu-usage v='20'/>
21:54:12:
21:54:12:  <!-- HTTP Server -->
21:54:12:  <allow v='127.0.0.1 xxx.yyy.zzz.aaa aaa.bbb.ccc.ddd'/>
21:54:12:
21:54:12:  <!-- Slot Control -->
21:54:12:  <power v='FULL'/>
21:54:12:
21:54:12:  <!-- User Information -->
21:54:12:  <passkey v='*****'/>
21:54:12:  <team v='244483'/>
21:54:12:  <user v='MyUsername'/>
21:54:12:
21:54:12:  <!-- Web Server -->
21:54:12:  <web-allow v='127.0.0.1 xxx.yyy.zzz.aaa aaa.bbb.ccc.ddd'/>
21:54:12:
21:54:12:  <!-- Folding Slots -->
21:54:12:  <slot id='0' type='GPU'>
21:54:12:    <paused v='true'/>
21:54:12:  </slot>
21:54:12:  <slot id='1' type='GPU'>
21:54:12:    <paused v='true'/>
21:54:12:  </slot>
21:54:12:  <slot id='2' type='GPU'>
21:54:12:    <paused v='true'/>
21:54:12:  </slot>
21:54:12:  <slot id='3' type='GPU'>
21:54:12:    <paused v='true'/>
21:54:12:  </slot>
21:54:12:</config>
[93m21:54:21:WARNING:FS00:Killing WU00[0m
[93m21:54:21:WARNING:FS01:Killing WU01[0m
[93m21:54:21:WARNING:FS02:Killing WU02[0m
[93m21:54:21:WARNING:FS03:Killing WU03[0m
...
...
[93m21:55:24:WARNING:FS01:Killing WU01[0m
[93m21:55:24:WARNING:FS02:Killing WU02[0m
[93m21:55:24:WARNING:FS03:Killing WU03[0m
21:55:24:FS00:Unpaused
21:55:24:FS01:Unpaused
21:55:24:FS02:Unpaused
21:55:24:FS03:Unpaused
21:55:24:WU05:FS03:Connecting to 65.254.110.245:80
21:55:25:WU05:FS03:Assigned to work server 140.163.4.241
21:55:25:WU05:FS03:Requesting new work unit for slot 03: RUNNING gpu:3:Ellesmere XT [Radeon RX 470/480/570/580/590] from 140.163.4.241
21:55:25:WU05:FS03:Connecting to 140.163.4.241:8080
21:55:54:WU05:FS03:Downloading 7.93MiB
21:56:00:WU05:FS03:Download 76.48%
21:56:01:WU05:FS03:Download complete
21:56:01:WU05:FS03:Received Unit: id:05 state:DOWNLOAD error:NO_ERROR project:11745 run:0 clone:2269 gen:86 core:0x22 unit:0x000000818ca304f15e67f1241bc196ba
21:56:14:Saving configuration to /etc/fahclient/config.xml
...
...
[93m22:09:49:WARNING:FS02:Killing WU02[0m
[93m22:09:49:WARNING:FS03:Killing WU03[0m
[93m22:09:49:WARNING:FS00:Killing WU00[0m
...
...

Nuitari · Post by **Nuitari** » Tue Apr 28, 2020 4:37 am

What does the last few lines of dmesg say when it gets stuck ? Any error in particular about a stuck process?
Does free show any ram available or excessive swap usage?
Which filesystem are you running ? Is it sync mounted, or does it have barriers active?

Post by **Joe_H** » Tue Apr 28, 2020 5:04 am

Going through the log I see two WUs that have completed to 100%, been packed up in the post completion phase, and then no sign of any attempt to send them. They are WU01 and WU03. That I do not understand, and have not idea what is going on there.

Slot 0 for the R9 200/300 series does eventually download a WU - WU00, but I see no sign that it ever progressed beyond 0%. The same happens with WU02 on FS02, it starts and never progresses.

Does the server have sufficient swap space to work with? I see 8 GB configured, each GPU slot can easily use 2 GB of memory at times during processing depending on the size of the WU in terms of number of atoms in the simulation.

Could you post the command that you are using?

Finally, for monitoring and controlling the folding being done on the server, FAHControl can be set up on another Windows, Linux or OS X system with a display to do this. The other system does not need to be set up for folding.

qe4 · Post by **qe4** » Tue Apr 28, 2020 3:09 pm

@Joe_H: Thank you for digging through the logs. I think your description of the problem is spot on.

Both you and Nuitari mentions that it could be swap or RAM. The server has 8GB RAM and I have now increased the swap to 12GB in total.

I have rebooted the server. Then I ran:

Code: Select all

FAHClient --dump ALL

Straight after. I seems that this command does some housekeeping and to my surprise it also uploaded af bunch of WU's. Here is the output from the log:

Code: Select all

$ sudo FAHClient --dump ALL
12:32:37:INFO(1):Read GPUs.txt
12:32:37:****************************** FAHClient ******************************
12:32:37:        Version: 7.6.9
12:32:37:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
12:32:37:      Copyright: 2020 foldingathome.org
12:32:37:       Homepage: https://foldingathome.org/
12:32:37:           Date: Apr 17 2020
12:32:37:           Time: 18:11:26
12:32:37:       Revision: 398c2b17fa535e0cc6c9d10856b2154c32771646
12:32:37:         Branch: master
12:32:37:       Compiler: GNU 8.3.0
12:32:37:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
12:32:37:                 -funroll-loops -fno-pie
12:32:37:       Platform: linux2 4.19.0-5-amd64
12:32:37:           Bits: 64
12:32:37:           Mode: Release
12:32:37:           Args: --dump ALL
12:32:37:         Config: /home/xxx/yyy/foldingathome/config.xml
12:32:37:******************************** CBang ********************************
12:32:37:           Date: Apr 17 2020
12:32:37:           Time: 18:10:13
12:32:37:       Revision: 2fb0be7809c5e45287a122ca5fbc15b5ae859a3b
12:32:37:         Branch: master
12:32:37:       Compiler: GNU 8.3.0
12:32:37:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
12:32:37:                 -funroll-loops -fno-pie -fPIC
12:32:37:       Platform: linux2 4.19.0-5-amd64
12:32:37:           Bits: 64
12:32:37:           Mode: Release
12:32:37:******************************* System ********************************
12:32:37:            CPU: Intel(R) Core(TM) i5-4430 CPU @ 3.00GHz
12:32:37:         CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
12:32:37:           CPUs: 4
12:32:37:         Memory: 7.48GiB
12:32:37:    Free Memory: 3.92GiB
12:32:37:        Threads: POSIX_THREADS
12:32:37:     OS Version: 4.4
12:32:37:    Has Battery: false
12:32:37:     On Battery: false
12:32:37:     UTC Offset: 0
12:32:37:            PID: 3514
12:32:37:            CWD: /home/xxx/yyy/foldingathome
12:32:37:             OS: Linux 4.4.0-130-generic x86_64
12:32:37:        OS Arch: AMD64
12:32:37:           GPUs: 4
12:32:37:          GPU 0: Bus:1 Slot:0 Func:0 AMD:5 Hawaii [Radeon R9 200/300X Series]
12:32:37:          GPU 1: Bus:2 Slot:0 Func:0 AMD:5 Ellesmere XT [Radeon RX
12:32:37:                 470/480/570/580/590]
12:32:37:          GPU 2: Bus:3 Slot:0 Func:0 AMD:5 Ellesmere XT [Radeon RX
12:32:37:                 470/480/570/580/590]
12:32:37:          GPU 3: Bus:4 Slot:0 Func:0 AMD:5 Ellesmere XT [Radeon RX
12:32:37:                 470/480/570/580/590]
12:32:37:           CUDA: Not detected: Failed to open dynamic library 'libcuda.so':
12:32:37:                 libcuda.so: cannot open shared object file: No such file or
12:32:37:                 directory
12:32:37:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:2580.4
12:32:37:OpenCL Device 1: Platform:0 Device:1 Bus:2 Slot:0 Compute:1.2 Driver:2580.4
12:32:37:OpenCL Device 2: Platform:0 Device:2 Bus:3 Slot:0 Compute:1.2 Driver:2580.4
12:32:37:OpenCL Device 3: Platform:0 Device:3 Bus:4 Slot:0 Compute:1.2 Driver:2580.4
12:32:37:******************************* libFAH ********************************
12:32:37:           Date: Apr 15 2020
12:32:37:           Time: 21:43:24
12:32:37:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
12:32:37:         Branch: master
12:32:37:       Compiler: GNU 8.3.0
12:32:37:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
12:32:37:                 -funroll-loops -fno-pie
12:32:37:       Platform: linux2 4.19.0-5-amd64
12:32:37:           Bits: 64
12:32:37:           Mode: Release
12:32:37:***********************************************************************
12:32:37:<config>
12:32:37:  <!-- Folding Slots -->
12:32:37:  <slot id='0' type='CPU'/>
12:32:37:</config>
12:32:37:Trying to access database...
12:32:37:Successfully acquired database lock
12:32:40:Enabled folding slot 00: READY cpu:3
12:32:40:WARNING:WU02:Slot ID 2 no longer exists and there are no other matching slots, dumping
12:32:40:WARNING:WU00:No longer matches Slot 0's configuration and there are no other matching slots, dumping
12:32:40:WARNING:WU01:Slot ID 1 no longer exists and there are no other matching slots, dumping
12:32:40:WARNING:WU04:Slot ID 1 no longer exists and there are no other matching slots, dumping
12:32:40:WARNING:WU03:Slot ID 3 no longer exists and there are no other matching slots, dumping
12:32:40:WARNING:WU05:Slot ID 3 no longer exists and there are no other matching slots, dumping
12:32:40:WARNING:Dumping WU00 per user request
12:32:40:WU02:Sending unit results: id:02 state:SEND error:DUMPED project:14538 run:0 clone:1664 gen:82 core:0x22 unit:0x0000006c0d5262775e7b9cdbfcff3d82
12:32:40:WU02:Connecting to 13.82.98.119:8080
12:32:40:WU00:FS00:Cleaning up
12:32:40:WU01:Sending unit results: id:01 state:SEND error:DUMPED project:11763 run:0 clone:10408 gen:35 core:0x22 unit:0x0000004480fccb0a5e711344dcc320b6
12:32:40:WU01:Uploading 33.37MiB to 128.252.203.10
12:32:40:WU01:Connecting to 128.252.203.10:8080
12:32:40:WU04:Sending unit results: id:04 state:SEND error:DUMPED project:11763 run:0 clone:14400 gen:9 core:0x22 unit:0x0000001980fccb0a5e7113189522230e
12:32:41:WU04:Connecting to 128.252.203.10:8080
12:32:41:WU03:Sending unit results: id:03 state:SEND error:DUMPED project:11743 run:0 clone:2700 gen:61 core:0x22 unit:0x000000608ca304f15e69d11615081141
12:32:41:WU03:Uploading 12.57MiB to 140.163.4.241
12:32:41:WU03:Connecting to 140.163.4.241:8080
12:32:41:WU05:Sending unit results: id:05 state:SEND error:DUMPED project:11745 run:0 clone:2269 gen:86 core:0x22 unit:0x000000818ca304f15e67f1241bc196ba
12:32:41:WU05:Connecting to 140.163.4.241:8080
12:32:54:WU01:Upload 0.19%
12:32:54:WARNING:WU01:Exception: Failed to send results to work server: Transfer failed
12:32:54:WU01:Trying to send results to collection server
12:32:54:WU01:Uploading 33.37MiB to 52.224.109.74
12:32:54:WU01:Connecting to 52.224.109.74:8080
12:33:00:WU01:Upload 8.80%
12:33:06:WU01:Upload 16.30%
12:33:11:WU02:Server responded WORK_ACK (400)
12:33:11:WU02:Cleaning up
12:33:12:WU01:Upload 23.04%
12:33:18:WU01:Upload 32.22%
12:33:24:WU01:Upload 38.21%
12:33:30:WU01:Upload 47.20%
12:33:36:WU01:Upload 56.57%
12:33:40:WU04:Server responded WORK_ACK (400)
12:33:40:WU04:Cleaning up
12:33:42:WU01:Upload 59.76%
12:33:48:WU01:Upload 67.62%
12:33:53:WU05:Server responded WORK_ACK (400)
12:33:53:WU05:Cleaning up
12:33:53:WU03:Upload 0.99%
12:33:54:WU01:Upload 71.56%
12:33:59:WU03:Upload 20.39%
12:34:00:WU01:Upload 80.55%
12:34:05:WU03:Upload 45.25%
12:34:06:WU01:Upload 87.10%
12:34:11:WU03:Upload 71.60%
12:34:12:WU01:Upload 93.85%
12:34:17:WU03:Upload 86.02%
12:34:18:WU01:Upload complete
12:34:18:WU01:Server responded WORK_ACK (400)
12:34:18:WU01:Cleaning up
12:34:21:WU03:Upload complete
12:34:21:WU03:Server responded WORK_ACK (400)
12:34:21:WU03:Cleaning up
12:34:22:Unit processing completed
12:34:23:Clean exit

I like that "Clean exit" at the end

I doesn't appear that my user has been credited with any points even though the uploads seem to have worked as intended (maybe I need to give it more time).

I have now started the client with my usual command:

Code: Select all

sudo FAHClient --config /etc/fahclient/config.xml

The client got WU's for all 4 GPU's but only one is working. Free mem/swap when running the client with one active GPU:

Code: Select all

              total        used        free      shared  buff/cache   available
Mem:           7663        2969         144          13        4549        4347
Swap:         12287           0       12287

So currently 3 WU's are stuck and 1 is working. I see this in the log a few times:

Code: Select all

Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
Following exception occured: Particle coordinate is nan

...they don't seem to be able to recover when this happens. Maybe things will work better if I lower clock speed... Just strange that it was running so well over the weekend.

Folding Forum

99.99% and stuck

99.99% and stuck

Re: 99.99% and stuck

Re: 99.99% and stuck

Re: 99.99% and stuck

Re: 99.99% and stuck

Re: 99.99% and stuck

Re: 99.99% and stuck

Re: 99.99% and stuck

Re: 99.99% and stuck

Re: 99.99% and stuck

Re: 99.99% and stuck

Re: 99.99% and stuck

Re: 99.99% and stuck

Re: 99.99% and stuck

Re: 99.99% and stuck