155.247.166.220 downloads stalled

[info2x]

Still happening as of this morning.

[HaloJones]

Failing as of 1730 UTC but I got redirected elsewhere pretty quickly

Code: Select all

16:28:54:WU01:FS00:Assigned to work server 155.247.166.220
16:28:54:WU01:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GP104 [GeForce GTX 1070] 6463 from 155.247.166.220
16:28:54:WU01:FS00:Connecting to 155.247.166.220:8080
16:29:15:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
16:29:15:WU01:FS00:Connecting to 155.247.166.220:80
16:29:37:ERROR:WU01:FS00:Exception: Failed to connect to 155.247.166.220:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
16:29:37:WU01:FS00:Connecting to 65.254.110.245:80

[bruce]

When I open http://vav4.ocis.temple.edu I do see the server's landing page but it refreshes very slowly. I'm guessing that the campus networking problem mentioned above is still un-solved.

[gordonbb]

Yup, for the last 3 days pretty much every stuck slot has been a result of a download stalling from this server. It's nasty in that the download starts and progresses a few % then stalls which then seems to block the algorithm used to detect stalls. I was on the new client (7.6.9 & .10) and rolled my systems back to 7.5 with no improvement.

Just checked all 15 of my GPUs and not a single one is running a WU from this server.

I observed that if the download is progressing slowly and the current WU completes and starts uploading that seems to cause a stall. So I backed the next-unit percentage off to 95% but that didn't seem to improve things.

What I've been using in Ubuntu to un-stick the slots is:

1. Pause all the slots on the system
2. execute:

Code: Select all

sudo service FAHClient stop (but this does not cleanly stop the client ... so ...)
ps -ef | grep fah (and note the Process ID (PID) of the fahclient)
sudo kill -KILL <PID>
sudo service  FAHClient start

[info2x]

Latest...

Code: Select all

01:29:16:WU00:FS01:0x21:Completed 9900000 out of 10000000 steps (99%)
01:29:17:WU01:FS01:Connecting to 65.254.110.245:80
01:29:17:WU01:FS01:Assigned to work server 155.247.166.220
01:29:17:WU01:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GP104 [GeForce GTX 1070 Ti] 8186 from 155.247.166.220
01:29:17:WU01:FS01:Connecting to 155.247.166.220:8080
01:29:17:WU01:FS01:Downloading 11.60MiB
01:31:39:WU00:FS01:0x21:Completed 10000000 out of 10000000 steps (100%)
01:31:40:WU00:FS01:0x21:Saving result file logfile_01.txt
01:31:40:WU00:FS01:0x21:Saving result file checkpointState.xml
01:31:40:WU00:FS01:0x21:Saving result file checkpt.crc
01:31:40:WU00:FS01:0x21:Saving result file log.txt
01:31:40:WU00:FS01:0x21:Saving result file positions.xtc
01:31:40:WU00:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
01:31:41:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
01:31:41:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:16904 run:11 clone:1 gen:5 core:0x21 unit:0x000000060002894c5ea3647e590f0dba
01:31:41:WU00:FS01:Uploading 12.48MiB to 155.247.166.220
01:31:41:WU00:FS01:Connecting to 155.247.166.220:8080
01:31:48:WU00:FS01:Upload 3.00%
01:31:55:WU00:FS01:Upload 4.51%
01:32:01:WU00:FS01:Upload 8.01%
01:32:07:WU00:FS01:Upload 10.51%
01:32:13:WU00:FS01:Upload 12.02%
01:32:22:WU00:FS01:Upload 14.02%

both the upload and download are frozen. Have no problem pulling up the server page.

[rickoic]

I've been fighting this same problem off and on for at least 2 weeks. Get the Connecting to 155.247.166.220:8080 line and then it will either just sit there forever (or until I reboot) and wait for information to be passed, or it will send me a pittance and then sit there forever (or until I reboot).

Could a NO activity timer be installed some where that would cause it to abort and try again?

Apparently after the server stops sending data it aborts at that end, but doesn't transmit the abortion to the over end.

[Kjetil]

I have no problems 155.247.166.220. but i am on beta flag.

Code: Select all

13:46:15:WU00:FS01:0x21:Version 0.0.20
13:46:17:WU01:FS01:Upload 2.56%
13:46:19:WU00:FS01:0x21:Completed 0 out of 10000000 steps (0%)
13:46:19:WU00:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
13:46:23:WU01:FS01:Upload 5.41%
13:46:29:WU01:FS01:Upload 8.11%
13:46:35:WU01:FS01:Upload 10.96%
13:46:41:WU01:FS01:Upload 13.81%
13:46:47:WU01:FS01:Upload 16.58%
13:46:53:WU01:FS01:Upload 19.43%
13:46:59:WU01:FS01:Upload 22.28%
13:47:05:WU01:FS01:Upload 25.20%
13:47:11:WU01:FS01:Upload 28.12%
13:47:17:WU01:FS01:Upload 30.96%
13:47:23:WU01:FS01:Upload 33.88%
13:47:29:WU01:FS01:Upload 36.59%
13:47:35:WU01:FS01:Upload 39.43%
13:47:35:WU00:FS01:0x21:Completed 100000 out of 10000000 steps (1%)
13:47:41:WU01:FS01:Upload 42.28%
13:47:47:WU01:FS01:Upload 45.20%
13:47:53:WU01:FS01:Upload 47.90%
13:47:59:WU01:FS01:Upload 50.75%
13:48:05:WU01:FS01:Upload 53.60%
13:48:11:WU01:FS01:Upload 56.52%
13:48:17:WU01:FS01:Upload 59.36%
13:48:23:WU01:FS01:Upload 62.28%
13:48:29:WU01:FS01:Upload 65.13%
13:48:35:WU01:FS01:Upload 68.05%
13:48:41:WU01:FS01:Upload 70.89%
13:48:47:WU01:FS01:Upload 73.81%
13:48:51:WU00:FS01:0x21:Completed 200000 out of 10000000 steps (2%)
13:48:53:WU01:FS01:Upload 76.73%
13:48:59:WU01:FS01:Upload 79.36%
13:49:05:WU01:FS01:Upload 82.21%
13:49:11:WU01:FS01:Upload 85.20%
13:49:17:WU01:FS01:Upload 88.05%
13:49:23:WU01:FS01:Upload 90.89%
13:49:29:WU01:FS01:Upload 93.81%
13:49:35:WU01:FS01:Upload 96.73%
13:49:41:WU01:FS01:Upload 99.65%
13:49:42:WU01:FS01:Upload complete
13:49:42:WU01:FS01:Server responded WORK_ACK (400)
13:49:42:WU01:FS01:Final credit estimate, 154565.00 points
13:49:42:WU01:FS01:Cleaning up
13:50:08:WU00:FS01:0x21:Completed 300000 out of 10000000 steps (3%)
13:51:24:WU00:FS01:0x21:Completed 400000 out of 10000000 steps (4%)
13:52:40:WU00:FS01:0x21:Completed 500000 out of 10000000 steps (5%)
13:53:58:WU00:FS01:0x21:Completed 600000 out of 10000000 steps (6%)
13:55:14:WU00:FS01:0x21:Completed 700000 out of 10000000 steps (7%)
13:56:32:WU00:FS01:0x21:Completed 800000 out of 10000000 steps (8%)
13:57:48:WU00:FS01:0x21:Completed 900000 out of 10000000 steps (9%)
13:59:04:WU00:FS01:0x21:Completed 1000000 out of 10000000 steps (10%)
14:00:22:WU00:FS01:0x21:Completed 1100000 out of 10000000 steps (11%)
14:01:38:WU00:FS01:0x21:Completed 1200000 out of 10000000 steps (12%)
14:02:55:WU00:FS01:0x21:Completed 1300000 out of 10000000 steps (13%)
14:04:11:WU00:FS01:0x21:Completed 1400000 out of 10000000 steps (14%)
14:05:27:WU00:FS01:0x21:Completed 1500000 out of 10000000 steps (15%)
14:06:45:WU00:FS01:0x21:Completed 1600000 out of 10000000 steps (16%)
14:08:01:WU00:FS01:0x21:Completed 1700000 out of 10000000 steps (17%)
14:09:19:WU00:FS01:0x21:Completed 1800000 out of 10000000 steps (18%)
14:10:35:WU00:FS01:0x21:Completed 1900000 out of 10000000 steps (19%)

[info2x]

Code: Select all

*********************** Log Started 2020-05-05T14:05:18Z ***********************
14:05:19:WU00:FS01:Connecting to 65.254.110.245:80
14:05:22:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
14:05:22:WU00:FS01:Connecting to 18.218.241.186:80
14:05:23:WU00:FS01:Assigned to work server 155.247.166.220
14:05:23:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070 Ti] 8186 from 155.247.166.220
14:05:23:WU00:FS01:Connecting to 155.247.166.220:8080
14:05:24:WU00:FS01:Downloading 5.13MiB

Just noticed that I seem to be connecting to 18.218.241.186:80 for an assignment. I don't see that on the server list. When I visit the page it looks like a normal assignment server.

[Neil-B]

Just noticed that I seem to be connecting to 18.218.241.186:80 for an assignment. I don't see that on the server list. When I visit the page it looks like a normal assignment server.

It is an Assignment Server see … viewtopic.php?f=18&t=34034&p=323083&hil ... ip#p323085

[info2x]

Just noticed that I seem to be connecting to 18.218.241.186:80 for an assignment. I don't see that on the server list. When I visit the page it looks like a normal assignment server.

It is an Assignment Server see … viewtopic.php?f=18&t=34034&p=323083&hil ... ip#p323085

Ahhh ok. Thanks

[CKWarner]

We're working on the problem. We've seen similar problems before -- they might arise from how the server code deals with stale connections, compounded with network issues on campus. We have restarted the server code; let us know if the problem persists

I'm still getting stalled downloads, and not just from that one server any more.

[PantherX]

...I'm still getting stalled downloads, and not just from that one server any more.

Welcome to the F@H Forum CKWarner,

Please start a new topic and post your log file so we can see what the issue is. If you require guidance, please see this topic: viewtopic.php?f=24&t=26036

[CKWarner]

Welcome to the F@H Forum CKWarner,

Please start a new topic and post your log file so we can see what the issue is.

This is my thread already.

It seems to be only GPU WUs that stall. The log isn't super helpful at the time that it happens, given that it's just showing the percentage and then nothing, although the timestamps could conceivably be useful I suppose. Here's a stall from yesterday, coincidentally from the server in question:

Code: Select all

[93m19:47:17:WARNING:WU03:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration[0m
19:47:17:WU03:FS01:Connecting to 65.254.110.245:80
19:47:17:WU03:FS01:Assigned to work server 155.247.166.220
19:47:17:WU03:FS01:Requesting new work unit for slot 01: READY gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 from 155.247.166.220
19:47:17:WU03:FS01:Connecting to 155.247.166.220:8080
19:47:18:WU03:FS01:Downloading 7.40MiB
19:47:24:WU03:FS01:Download 0.84%
19:47:35:WU03:FS01:Download 2.53%
19:47:42:WU03:FS01:Download 4.22%
19:47:56:WU01:FS00:0xa7:Completed 10000 out of 500000 steps (2%)
19:47:56:WU03:FS01:Download 6.75%
19:50:32:WU01:FS00:0xa7:Completed 15000 out of 500000 steps (3%)
19:53:08:WU01:FS00:0xa7:Completed 20000 out of 500000 steps (4%)
19:55:44:WU01:FS00:0xa7:Completed 25000 out of 500000 steps (5%)
19:58:21:WU01:FS00:0xa7:Completed 30000 out of 500000 steps (6%)
20:00:57:WU01:FS00:0xa7:Completed 35000 out of 500000 steps (7%)

About five minutes later I did the delete the GPU slot procedure I listed earlier in the thread to get the client folding again.

Here's the configuration gubbins from the start of the current log:

Code: Select all

*********************** Log Started 2020-05-06T00:01:21Z ***********************
00:01:21:****************************** FAHClient ******************************
00:01:21:        Version: 7.6.9
00:01:21:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
00:01:21:      Copyright: 2020 foldingathome.org
00:01:21:       Homepage: https://foldingathome.org/
00:01:21:           Date: Apr 17 2020
00:01:21:           Time: 18:11:26
00:01:21:       Revision: 398c2b17fa535e0cc6c9d10856b2154c32771646
00:01:21:         Branch: master
00:01:21:       Compiler: GNU 8.3.0
00:01:21:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
00:01:21:                 -funroll-loops -fno-pie
00:01:21:       Platform: linux2 4.19.0-5-amd64
00:01:21:           Bits: 64
00:01:21:           Mode: Release
00:01:21:           Args: --child /etc/fahclient/config.xml --run-as fahclient
00:01:21:                 --pid-file=/var/run/fahclient.pid --daemon
00:01:21:         Config: /etc/fahclient/config.xml
00:01:21:******************************** CBang ********************************
00:01:21:           Date: Apr 17 2020
00:01:21:           Time: 18:10:13
00:01:21:       Revision: 2fb0be7809c5e45287a122ca5fbc15b5ae859a3b
00:01:21:         Branch: master
00:01:21:       Compiler: GNU 8.3.0
00:01:21:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
00:01:21:                 -funroll-loops -fno-pie -fPIC
00:01:21:       Platform: linux2 4.19.0-5-amd64
00:01:21:           Bits: 64
00:01:21:           Mode: Release
00:01:21:******************************* System ********************************
00:01:21:            CPU: AMD Ryzen 7 2700X Eight-Core Processor
00:01:21:         CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
00:01:21:           CPUs: 16
00:01:21:         Memory: 15.65GiB
00:01:21:    Free Memory: 14.52GiB
00:01:21:        Threads: POSIX_THREADS
00:01:21:     OS Version: 5.3
00:01:21:    Has Battery: false
00:01:21:     On Battery: false
00:01:21:     UTC Offset: 1
00:01:21:            PID: 1170
00:01:21:            CWD: /var/lib/fahclient
00:01:21:             OS: Linux 5.3.0-51-lowlatency x86_64
00:01:21:        OS Arch: AMD64
00:01:21:           GPUs: 1
00:01:21:          GPU 0: Bus:11 Slot:0 Func:0 NVIDIA:8 TU102 [GeForce RTX 2080 Ti Rev.
00:01:21:                 A] M 13448
00:01:21:  CUDA Device 0: Platform:0 Device:0 Bus:11 Slot:0 Compute:7.5 Driver:10.2
00:01:21:OpenCL Device 0: Platform:0 Device:0 Bus:11 Slot:0 Compute:1.2 Driver:440.82
00:01:21:******************************* libFAH ********************************
00:01:21:           Date: Apr 15 2020
00:01:21:           Time: 21:43:24
00:01:21:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
00:01:21:         Branch: master
00:01:21:       Compiler: GNU 8.3.0
00:01:21:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
00:01:21:                 -funroll-loops -fno-pie
00:01:21:       Platform: linux2 4.19.0-5-amd64
00:01:21:           Bits: 64
00:01:21:           Mode: Release
00:01:21:***********************************************************************
00:01:21:<config>
00:01:21:  <!-- Client Control -->
00:01:21:  <fold-anon v='true'/>
00:01:21:
00:01:21:  <!-- HTTP Server -->
00:01:21:  <allow v='127.0.0.1 192.168.1.0/24'/>
00:01:21:
00:01:21:  <!-- Network -->
00:01:21:  <proxy v=':8080'/>
00:01:21:
00:01:21:  <!-- Remote Command Server -->
00:01:21:  <command-allow-no-pass v='127.0.0.1 192.168.1.0/24'/>
00:01:21:
00:01:21:  <!-- Slot Control -->
00:01:21:  <pause-on-start v='true'/>
00:01:21:  <power v='full'/>
00:01:21:
00:01:21:  <!-- User Information -->
00:01:21:  <passkey v='*****'/>
00:01:21:  <team v='14'/>
00:01:21:  <user v='CatKiller'/>
00:01:21:
00:01:21:  <!-- Work Unit Control -->
00:01:21:  <next-unit-percentage v='97'/>
00:01:21:
00:01:21:  <!-- Folding Slots -->
00:01:21:  <slot id='0' type='CPU'>
00:01:21:    <paused v='true'/>
00:01:21:  </slot>
00:01:21:  <slot id='1' type='GPU'>
00:01:21:    <paused v='true'/>
00:01:21:  </slot>
00:01:21:</config>
00:01:21:Trying to access database...
00:01:21:Successfully acquired database lock
00:01:21:Enabled folding slot 00: PAUSED cpu:15 (by user)
00:01:21:Enabled folding slot 01: PAUSED gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 (by user)
01:20:07:FS00:Unpaused
01:20:07:FS01:Unpaused
01:20:07:WU00:FS01:Starting
01:20:07:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 00 -suffix 01 -version 706 -lifeline 1170 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
01:20:07:WU00:FS01:Started FahCore on PID 22171
01:20:07:WU00:FS01:Core PID:22175
01:20:07:WU00:FS01:FahCore 0x22 started
01:20:07:WU01:FS00:Starting
01:20:07:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 01 -suffix 01 -version 706 -lifeline 1170 -checkpoint 15 -np 15
01:20:07:WU01:FS00:Started FahCore on PID 22178
01:20:07:WU01:FS00:Core PID:22182
01:20:07:WU01:FS00:FahCore 0xa7 started
01:20:08:WU00:FS01:0x22:*********************** Log Started 2020-05-06T01:20:07Z ***********************
01:20:08:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
01:20:08:WU00:FS01:0x22:       Type: 0x22
01:20:08:WU00:FS01:0x22:       Core: Core22
01:20:08:WU00:FS01:0x22:    Website: https://foldingathome.org/
01:20:08:WU00:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
01:20:08:WU00:FS01:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
01:20:08:WU00:FS01:0x22:             <rafal.wiewiora@choderalab.org>
01:20:08:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 22171 -checkpoint 15
01:20:08:WU00:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
01:20:08:WU00:FS01:0x22:             0 -gpu 0
01:20:08:WU00:FS01:0x22:     Config: <none>
01:20:08:WU00:FS01:0x22:************************************ Build *************************************
01:20:08:WU00:FS01:0x22:    Version: 0.0.5
01:20:08:WU00:FS01:0x22:       Date: Apr 22 2020
01:20:08:WU00:FS01:0x22:       Time: 03:57:11
01:20:08:WU00:FS01:0x22: Repository: Git
01:20:08:WU00:FS01:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
01:20:08:WU00:FS01:0x22:     Branch: HEAD
01:20:08:WU00:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
01:20:08:WU00:FS01:0x22:    Options: -std=c++11 -O3 -funroll-loops
01:20:08:WU00:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
01:20:08:WU00:FS01:0x22:       Bits: 64
01:20:08:WU00:FS01:0x22:       Mode: Release
01:20:08:WU00:FS01:0x22:************************************ System ************************************
01:20:08:WU00:FS01:0x22:        CPU: AMD Ryzen 7 2700X Eight-Core Processor
01:20:08:WU00:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
01:20:08:WU00:FS01:0x22:       CPUs: 16
01:20:08:WU00:FS01:0x22:     Memory: 15.65GiB
01:20:08:WU00:FS01:0x22:Free Memory: 4.64GiB
01:20:08:WU00:FS01:0x22:    Threads: POSIX_THREADS
01:20:08:WU00:FS01:0x22: OS Version: 5.3
01:20:08:WU00:FS01:0x22:Has Battery: false
01:20:08:WU00:FS01:0x22: On Battery: false
01:20:08:WU00:FS01:0x22: UTC Offset: 1
01:20:08:WU00:FS01:0x22:        PID: 22175
01:20:08:WU00:FS01:0x22:        CWD: /var/lib/fahclient/work
01:20:08:WU00:FS01:0x22:         OS: Linux 5.3.0-51-lowlatency x86_64
01:20:08:WU00:FS01:0x22:    OS Arch: AMD64
01:20:08:WU00:FS01:0x22:********************************************************************************
01:20:08:WU00:FS01:0x22:Project: 16435 (Run 2615, Clone 0, Gen 3)
01:20:08:WU00:FS01:0x22:Unit: 0x0000000903854c135e9a4ef7e6c469f7
01:20:08:WU00:FS01:0x22:Digital signatures verified
01:20:08:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
01:20:08:WU00:FS01:0x22:Version 0.0.5
01:20:08:WU00:FS01:0x22:  Found a checkpoint file
01:20:08:WU01:FS00:0xa7:*********************** Log Started 2020-05-06T01:20:07Z ***********************
01:20:08:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
01:20:08:WU01:FS00:0xa7:       Type: 0xa7
01:20:08:WU01:FS00:0xa7:       Core: Gromacs
01:20:08:WU01:FS00:0xa7:       Args: -dir 01 -suffix 01 -version 706 -lifeline 22178 -checkpoint 15 -np
01:20:08:WU01:FS00:0xa7:             15
01:20:08:WU01:FS00:0xa7:************************************ CBang *************************************
01:20:08:WU01:FS00:0xa7:       Date: Nov 5 2019
01:20:08:WU01:FS00:0xa7:       Time: 06:06:57
01:20:08:WU01:FS00:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
01:20:08:WU01:FS00:0xa7:     Branch: master
01:20:08:WU01:FS00:0xa7:   Compiler: GNU 8.3.0
01:20:08:WU01:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
01:20:08:WU01:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
01:20:08:WU01:FS00:0xa7:       Bits: 64
01:20:08:WU01:FS00:0xa7:       Mode: Release
01:20:08:WU01:FS00:0xa7:************************************ System ************************************
01:20:08:WU01:FS00:0xa7:        CPU: AMD Ryzen 7 2700X Eight-Core Processor
01:20:08:WU01:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
01:20:08:WU01:FS00:0xa7:       CPUs: 16
01:20:08:WU01:FS00:0xa7:     Memory: 15.65GiB
01:20:08:WU01:FS00:0xa7:Free Memory: 4.63GiB
01:20:08:WU01:FS00:0xa7:    Threads: POSIX_THREADS
01:20:08:WU01:FS00:0xa7: OS Version: 5.3
01:20:08:WU01:FS00:0xa7:Has Battery: false
01:20:08:WU01:FS00:0xa7: On Battery: false
01:20:08:WU01:FS00:0xa7: UTC Offset: 1
01:20:08:WU01:FS00:0xa7:        PID: 22182
01:20:08:WU01:FS00:0xa7:        CWD: /var/lib/fahclient/work
01:20:08:WU01:FS00:0xa7:******************************** Build - libFAH ********************************
01:20:08:WU01:FS00:0xa7:    Version: 0.0.18
01:20:08:WU01:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
01:20:08:WU01:FS00:0xa7:  Copyright: 2019 foldingathome.org
01:20:08:WU01:FS00:0xa7:   Homepage: https://foldingathome.org/
01:20:08:WU01:FS00:0xa7:       Date: Nov 5 2019
01:20:08:WU01:FS00:0xa7:       Time: 06:13:26
01:20:08:WU01:FS00:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
01:20:08:WU01:FS00:0xa7:     Branch: master
01:20:08:WU01:FS00:0xa7:   Compiler: GNU 8.3.0
01:20:08:WU01:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
01:20:08:WU01:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
01:20:08:WU01:FS00:0xa7:       Bits: 64
01:20:08:WU01:FS00:0xa7:       Mode: Release
01:20:08:WU01:FS00:0xa7:************************************ Build *************************************
01:20:08:WU01:FS00:0xa7:       SIMD: avx_256
01:20:08:WU01:FS00:0xa7:********************************************************************************
01:20:08:WU01:FS00:0xa7:Project: 16803 (Run 2, Clone 526, Gen 7)
01:20:08:WU01:FS00:0xa7:Unit: 0x0000000a82ed0b915e99f9e1b6d16842
01:20:08:WU01:FS00:0xa7:Digital signatures verified
01:20:08:WU01:FS00:0xa7:Calling: mdrun -s frame7.tpr -o frame7.trr -cpi state.cpt -cpt 15 -nt 15
01:20:08:WU01:FS00:0xa7:Steps: first=3500000 total=500000
01:20:10:WU01:FS00:0xa7:Completed 218852 out of 500000 steps (43%)
01:20:16:WU00:FS01:0x22:Completed 2400000 out of 5000000 steps (48%)
01:20:16:WU00:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
01:20:39:Removing old file 'configs/config-20200504-110640.xml'
01:20:39:Saving configuration to /etc/fahclient/config.xml
01:20:39:<config>
01:20:39:  <!-- Client Control -->
01:20:39:  <fold-anon v='true'/>
01:20:39:
01:20:39:  <!-- HTTP Server -->
01:20:39:  <allow v='127.0.0.1 192.168.1.0/24'/>
01:20:39:
01:20:39:  <!-- Network -->
01:20:39:  <proxy v=':8080'/>
01:20:39:
01:20:39:  <!-- Remote Command Server -->
01:20:39:  <command-allow-no-pass v='127.0.0.1 192.168.1.0/24'/>
01:20:39:
01:20:39:  <!-- Slot Control -->
01:20:39:  <pause-on-start v='true'/>
01:20:39:  <power v='full'/>
01:20:39:
01:20:39:  <!-- User Information -->
01:20:39:  <passkey v='*****'/>
01:20:39:  <team v='14'/>
01:20:39:  <user v='CatKiller'/>
01:20:39:
01:20:39:  <!-- Work Unit Control -->
01:20:39:  <next-unit-percentage v='97'/>
01:20:39:
01:20:39:  <!-- Folding Slots -->
01:20:39:  <slot id='0' type='CPU'/>
01:20:39:  <slot id='1' type='GPU'/>
01:20:39:</config>

[PantherX]

...This is my thread already...

Apologies... I scrolled a few posts up and saw various other logs hence the suggestion.

...It seems to be only GPU WUs that stall. The log isn't super helpful at the time that it happens, given that it's just showing the percentage and then nothing, although the timestamps could conceivably be useful I suppose...

In this case, it was useful as it showed that you encountered a known issue where failure in network connection will cause the client to hang-up (https://github.com/FoldingAtHome/fah-issues/issues/983). Rather than removing the GPU Slot and adding it back in, it would much quicker to restart the client or your system depending on what is easier.

[rickoic]

Still having the stalled downloads from this server. but it does seem to be improving.
Mon 8 of 9 stalled
Tue 7 of 9 stalled
Wed 5 of 9 stalled
Thu 3 of 9 stalled

This when I checked them right after getting up after a nights folding.