Slow WU downloads & failed transfer [140.163.4.231]

[OntosChalmer]

My last WU from mskcc downloaded in 1 hour (for a 16.30 MiB WU, a better time than normal), and had no issues with transfer failure. I'm shutting the FAHC off for now, but I'll see if there are further issues when I fold tomorrow.

EDIT: Can confirm that speed improvement seems somewhat consistent. Better than last time, but still has 1+ hour of downtime between each WU. Better than the longer downtimes I had earlier, though.

[rafwiewiora]

Hi folks - we've spoken to our networking people, no problems they can see - no errors or drops, plenty of bandwidth left (we're only doing 50mbps on a 10gbps port now). Pings and traceroutes are blocked from outside.

Are you still experiencing problems? I'd like to get a sense of how many people are still having trouble, if this is something on our side and if we should keep investigating.

Thanks for your patience!

[OntosChalmer]

10 hours worth of folding download failures. I've had to set up my system to run BOINC on downtime periods due to the simple lack of GPU WUs.

Many messages cut for brevity.

Code: Select all

14:16:24:WU01:FS01:Connecting to 65.254.110.245:8080
14:16:25:WU01:FS01:Assigned to work server 140.163.4.231
14:16:25:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070] 6463 from 140.163.4.231
14:16:25:WU01:FS01:Connecting to 140.163.4.231:8080
14:30:01:WU01:FS01:Downloading 5.42MiB
14:30:49:WU01:FS01:Download 38.08%
15:00:19:WU01:FS01:Download 93.46%
15:01:05:WU01:FS01:Download 94.62%
15:01:29:WU01:FS01:Download 94.62%
15:01:29:ERROR:WU01:FS01:Exception: Transfer failed
15:01:29:WU01:FS01:Connecting to 65.254.110.245:8080
15:01:30:WU01:FS01:Assigned to work server 140.163.4.231
15:01:30:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070] 6463 from 140.163.4.231
15:01:30:WU01:FS01:Connecting to 140.163.4.231:8080
15:22:35:WU01:FS01:Downloading 5.42MiB
15:23:18:WU01:FS01:Download 38.07%
15:45:48:WU01:FS01:Download 77.30%
15:46:27:WU01:FS01:Download 78.46%
15:46:33:WU01:FS01:Download 78.46%
15:46:34:ERROR:WU01:FS01:Exception: Transfer failed
15:46:34:WU01:FS01:Connecting to 65.254.110.245:8080
15:46:34:WU01:FS01:Assigned to work server 140.163.4.231
15:46:34:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070] 6463 from 140.163.4.231
15:46:34:WU01:FS01:Connecting to 140.163.4.231:8080
16:06:46:WU01:FS01:Downloading 5.42MiB
16:07:27:WU01:FS01:Download 38.07%
16:30:49:WU01:FS01:Download 80.76%
16:31:25:WU01:FS01:Download 81.91%
16:31:38:WU01:FS01:Download 81.91%
16:31:38:ERROR:WU01:FS01:Exception: Transfer failed
16:31:38:WU01:FS01:Connecting to 65.254.110.245:8080
16:31:39:WU01:FS01:Assigned to work server 140.163.4.231
16:31:39:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070] 6463 from 140.163.4.231
16:31:39:WU01:FS01:Connecting to 140.163.4.231:8080
16:52:36:WU01:FS01:Downloading 5.42MiB
16:53:17:WU01:FS01:Download 38.08%
17:16:00:WU01:FS01:Download 78.46%
17:16:39:WU01:FS01:Download 79.62%
17:16:43:ERROR:WU01:FS01:Exception: Transfer failed
17:16:43:WU01:FS01:Connecting to 65.254.110.245:8080
17:16:44:WU01:FS01:Assigned to work server 128.252.203.4
17:16:44:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070] 6463 from 128.252.203.4
17:16:44:WU01:FS01:Connecting to 128.252.203.4:8080
17:34:32:WU01:FS01:Downloading 11.65MiB
17:35:05:WU01:FS01:Download 17.70%
18:01:10:WU01:FS01:Download 42.91%
18:01:41:WU01:FS01:Download 43.44%
18:01:50:WU01:FS01:Download 43.44%
18:01:50:ERROR:WU01:FS01:Exception: Transfer failed
18:01:50:WU01:FS01:Connecting to 65.254.110.245:8080
18:01:51:WU01:FS01:Assigned to work server 140.163.4.231
18:01:51:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070] 6463 from 140.163.4.231
18:01:51:WU01:FS01:Connecting to 140.163.4.231:8080
18:22:27:WU01:FS01:Downloading 5.42MiB
18:23:07:WU01:FS01:Download 38.07%
18:32:09:WU01:FS01:Download 54.23%
******************************* Date: 2018-07-04 *******************************
18:32:48:WU01:FS01:Download 55.38%
18:45:43:WU01:FS01:Download 78.46%
18:46:22:WU01:FS01:Download 79.61%
18:46:54:WU01:FS01:Download 79.61%
18:46:54:ERROR:WU01:FS01:Exception: Transfer failed
18:46:55:WU01:FS01:Connecting to 65.254.110.245:8080
18:46:55:WU01:FS01:Assigned to work server 140.163.4.231
18:46:55:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070] 6463 from 140.163.4.231
18:46:55:WU01:FS01:Connecting to 140.163.4.231:8080
19:04:35:WU01:FS01:Downloading 5.42MiB
19:05:10:WU01:FS01:Download 38.07%
19:30:51:WU01:FS01:Download 89.99%
19:31:26:WU01:FS01:Download 91.15%
19:31:59:WU01:FS01:Download 91.15%
19:31:59:ERROR:WU01:FS01:Exception: Transfer failed
19:31:59:WU01:FS01:Connecting to 65.254.110.245:8080
19:32:00:WU01:FS01:Assigned to work server 140.163.4.231
19:32:00:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070] 6463 from 140.163.4.231
19:32:00:WU01:FS01:Connecting to 140.163.4.231:8080
19:52:14:WU01:FS01:Downloading 5.42MiB
19:52:52:WU01:FS01:Download 38.08%
20:15:51:WU01:FS01:Download 80.77%
20:16:29:WU01:FS01:Download 81.93%
20:17:04:WU01:FS01:Download 81.93%
20:17:04:ERROR:WU01:FS01:Exception: Transfer failed
20:17:04:WU01:FS01:Connecting to 65.254.110.245:8080
20:17:05:WU01:FS01:Assigned to work server 140.163.4.231
20:17:05:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070] 6463 from 140.163.4.231
20:17:05:WU01:FS01:Connecting to 140.163.4.231:8080
20:37:25:WU01:FS01:Downloading 5.42MiB
20:38:04:WU01:FS01:Download 38.08%
21:00:57:WU01:FS01:Download 79.62%
21:01:35:WU01:FS01:Download 80.77%
21:02:09:WU01:FS01:Download 80.77%
21:02:09:ERROR:WU01:FS01:Exception: Transfer failed
21:02:09:WU01:FS01:Connecting to 65.254.110.245:8080
21:02:09:WU01:FS01:Assigned to work server 140.163.4.231
21:02:09:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070] 6463 from 140.163.4.231
21:02:09:WU01:FS01:Connecting to 140.163.4.231:8080
21:23:23:WU01:FS01:Downloading 5.42MiB
21:24:05:WU01:FS01:Download 38.08%
21:46:25:WU01:FS01:Download 76.16%
21:47:06:WU01:FS01:Download 77.31%
21:47:13:WU01:FS01:Download 77.31%
21:47:13:ERROR:WU01:FS01:Exception: Transfer failed
21:49:08:WU01:FS01:Connecting to 65.254.110.245:8080
21:49:08:WU01:FS01:Assigned to work server 128.252.203.4
21:49:08:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070] 6463 from 128.252.203.4
21:49:08:WU01:FS01:Connecting to 128.252.203.4:8080
22:10:51:WU01:FS01:Downloading 11.66MiB
22:11:33:WU01:FS01:Download 17.69%
22:33:10:WU01:FS01:Download 34.85%
22:33:51:WU01:FS01:Download 35.38%
22:34:11:WU01:FS01:Download 35.38%
22:34:12:ERROR:WU01:FS01:Exception: Transfer failed
23:05:09:WU01:FS01:Connecting to 65.254.110.245:8080
23:05:09:WU01:FS01:Assigned to work server 140.163.4.231
23:05:09:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070] 6463 from 140.163.4.231
23:05:09:WU01:FS01:Connecting to 140.163.4.231:8080
23:22:49:WU01:FS01:Downloading 5.42MiB
23:23:23:WU01:FS01:Download 38.08%
23:49:19:WU01:FS01:Download 92.31%
23:49:52:WU01:FS01:Download 93.46%
23:50:13:WU01:FS01:Download 93.46%
23:50:13:ERROR:WU01:FS01:Exception: Transfer failed

Tracert details to 140.163.4.231

Code: Select all

Tracing route to plfah1-1.mskcc.org [140.163.4.231]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.1.254
  2     2 ms     2 ms     2 ms  bb119-74-45-254.singnet.com.sg [119.74.45.254]
  3     4 ms     2 ms     2 ms  202.166.123.134
  4    16 ms     2 ms     4 ms  202.166.123.133
  5     2 ms     2 ms     2 ms  ae8-0.tp-cr03.singnet.com.sg [202.166.122.50]
  6     2 ms     2 ms     2 ms  ae4-0.tp-er03.singnet.com.sg [202.166.123.70]
  7     2 ms     2 ms     2 ms  203.208.191.197
  8   187 ms   186 ms   187 ms  203.208.183.46
  9   199 ms   199 ms   199 ms  palo-b1-link.telia.net [80.239.134.85]
 10   246 ms   245 ms   252 ms  nyk-bb3-link.telia.net [62.115.114.4]
 11   259 ms   259 ms   259 ms  nyk-b3-link.telia.net [62.115.140.223]
 12   246 ms   246 ms   246 ms  windstream-ic-310403-nyk-b3.c.telia.net [213.248.95.22]
 13   247 ms   246 ms   254 ms  be1.agr03.nwrk01-nj.us.windstream.net [40.128.248.11]
 14   256 ms   255 ms   255 ms  xe0-0-0-0.pe07.nwrk01-nj.us.windstream.net [40.128.249.132]
 15   246 ms   247 ms   247 ms  74.8.57.6
 16     *        *        *     Request timed out.
 17     *        *        *     Request timed out.
 18     *        *        *     Request timed out.
 19     *        *        *     Request timed out.
 20     *        *        *     Request timed out.
 21     *        *        *     Request timed out.
 22     *        *        *     Request timed out.
 23     *        *        *     Request timed out.
 24     *        *        *     Request timed out.
 25     *        *        *     Request timed out.
 26     *        *        *     Request timed out.
 27     *        *        *     Request timed out.
 28     *        *        *     Request timed out.
 29     *        *        *     Request timed out.
 30     *        *        *     Request timed out.

Trace complete.

[SteveWillis]

I wonder if you could force a different route by using a VPN and could that help?

[rafwiewiora]

@ OntosChalmer thanks a lot for the new report! I'm passing onto the networking folks to have them investigate further...

For now I've restarted the WS - which has been known to help temporarily.

[rafwiewiora]

@ OntosChalmer - I restarted the WS an hour ago, and now we've rebooted the machines as well - would you mind seeing if the problems still persist?

[bruce]

I wonder if you could force a different route by using a VPN and could that help?

It's worth a try, but I'd expect that you'd still get to nwrk01-nj.us.windstream.net and it would still be the same after that ... where we have no useful information.

I'd really like to see what changes in that traceroute when things are working right. I understand that campus servers can be configured to drop pings, so the timeout messages are not necessarily indicative of the problem.

I wonder if an outgoing traceroute from the server 140.163.4.231 itself starts with the same two hops.

Code: Select all

74.8.57.6
xe0-0-0-0.pe07.nwrk01-nj.us.windstream.net [40.128.249.132]

and would it look any different from 140.163.4.232?

SOMETHING is throttling the downloads after a period of (fairly) good operation. Maybe there's a Virtual Machine routing issue we're not seeing from here.

[rafwiewiora]

I understand that campus servers can be configured to drop pings, so the timeout messages are not necessarily indicative of the problem.

They're dropping pings and traceroutes - you won't get anything from those.

I wonder if an outgoing traceroute from the server 140.163.4.231 itself starts with the same two hops.

Appears the outbound are also blocked.

[OntosChalmer]

Code: Select all

03:29:30:WU01:FS01:Connecting to 65.254.110.245:8080
03:29:32:WU01:FS01:Assigned to work server 140.163.4.231
03:29:32:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070] 6463 from 140.163.4.231
03:29:32:WU01:FS01:Connecting to 140.163.4.231:8080
03:51:37:WU01:FS01:Downloading 5.42MiB
03:52:21:WU01:FS01:Download 38.08%
03:53:03:WU01:FS01:Download 39.23%
03:53:46:WU01:FS01:Download 40.39%
03:54:27:WU01:FS01:Download 41.54%
03:55:09:WU01:FS01:Download 42.69%
03:55:51:WU01:FS01:Download 43.85%
03:56:32:WU01:FS01:Download 45.00%
03:57:14:WU01:FS01:Download 46.16%
03:57:57:WU01:FS01:Download 47.31%
03:58:40:WU01:FS01:Download 48.46%
03:59:23:WU01:FS01:Download 49.62%
04:00:05:WU01:FS01:Download 50.77%
04:00:47:WU01:FS01:Download 51.92%
04:01:29:WU01:FS01:Download 53.08%
04:02:12:WU01:FS01:Download 54.23%
04:02:53:WU01:FS01:Download 55.39%
04:03:36:WU01:FS01:Download 56.54%
04:04:18:WU01:FS01:Download 57.69%
04:05:00:WU01:FS01:Download 58.85%
04:05:44:WU01:FS01:Download 60.00%
04:06:27:WU01:FS01:Download 61.16%
04:07:10:WU01:FS01:Download 62.31%
04:07:54:WU01:FS01:Download 63.46%
04:08:36:WU01:FS01:Download 64.62%
04:09:20:WU01:FS01:Download 65.77%
04:10:02:WU01:FS01:Download 66.93%
04:10:44:WU01:FS01:Download 68.08%
04:11:26:WU01:FS01:Download 69.23%
04:12:09:WU01:FS01:Download 70.39%
04:12:52:WU01:FS01:Download 71.54%
04:13:35:WU01:FS01:Download 72.69%
04:14:17:WU01:FS01:Download 73.85%
04:14:36:WU01:FS01:Download 73.85%
04:14:36:ERROR:WU01:FS01:Exception: Transfer failed
04:14:36:WU01:FS01:Connecting to 65.254.110.245:8080
04:14:37:WU01:FS01:Assigned to work server 140.163.4.231
04:14:37:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070] 6463 from 140.163.4.231
04:14:37:WU01:FS01:Connecting to 140.163.4.231:8080

Can confirm that there are no improvements.

Cannot currently afford a reliable VPN, so cannot test with one.

[SteveWillis]

Just fyi mine is 5 euro a month for up to 5 devices at the same time and I'm very happy with it mullvad.net I have it on my phone too, which is a good idea if you use unsecured wifi anywhere.

[lknl]

it's still very difficult to get new GPU WU for me, especially for 16MB WU.
FAH Log: https://pastebin.com/raw/HyahHspW
in normal cases, what is the download time for 16MB GPU WU? if it's expected to be slow then i can only blame my router/ISP for dropping my connection.

[bruce]

i can only blame my router/ISP for dropping my connection.

No, you can't blame your router/ISP. The server code is designed to drop connections that have been connected too long. When SOMETHING makes the download take "too long" that's what is expected to happen. The bottom line is that somebody needs to figure out why this server is performing so poorly.

I presume you have no trouble opening the banner pages on http://140.163.4.231/ or http://140.163.4.231:8080/

In the log you posted I see no entries for max_packet_size, so you're apparently using the default setting.

Just for kicks, what happens if you add a max_packet_size setting of 9 for the GPU slot and you wait for the next assignment to be downloaded?

[bruce]

Please explain why you had this error, This indicates that SOMETHING is happening on your ISP connection.

Code: Select all

07:10:02:WU01:FS01:Connecting to 65.254.110.245:80
07:10:04:WU01:FS01:Assigned to work server 140.163.4.231
07:10:04:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 140.163.4.231
07:10:04:WU01:FS01:Connecting to 140.163.4.231:8080
07:10:04:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
07:10:04:WU01:FS01:Connecting to 140.163.4.231:80
07:10:04:ERROR:WU01:FS01:Exception: Failed to connect to 140.163.4.231:80: A socket operation was attempted to an unreachable network.
07:10:05:ERROR:WU01:FS01:Exception: Could not get IP address for assign-GPU.stanford.edu: No such host is known. 
07:10:05:ERROR:WU01:FS01:Exception: Could not get IP address for assign-GPU2.stanford.edu: No such host is known. 
07:10:05:WARNING:WU01:FS01:Exception: Failed to find any IP addresses for assignment servers
07:10:05:ERROR:WU01:FS01:Exception: Could not get an assignment

The fact that it apparently recovers doesn't change the fact that it actually happened.

[lknl]

I presume you have no trouble opening the banner pages on http://140.163.4.231/ or http://140.163.4.231:8080/

Both are accessible, showing Folding@home, Folding@home Logo, Work Server, Version 9.2.5

In the log you posted I see no entries for max_packet_size, so you're apparently using the default setting.
Just for kicks, what happens if you add a max_packet_size setting of 9 for the GPU slot and you wait for the next assignment to be downloaded?

I don't remember seeing that max_packet_size in FAH guide before, can you point me to the details? Edit: Sorry, I saw the link in your signature viewtopic.php?p=261088&f=24#p261088
For now, this this the setting:

Code: Select all

16:50:08:Saving configuration to config.xml
16:50:08:<config>
16:50:08:  <!-- Folding Core -->
16:50:08:  <checkpoint v='30'/>
16:50:08:
16:50:08:  <!-- Network -->
16:50:08:  <proxy v=':8080'/>
16:50:08:
16:50:08:  <!-- Slot Control -->
16:50:08:  <power v='FULL'/>
16:50:08:
16:50:08:  <!-- User Information -->
16:50:08:  <passkey v='********************************'/>
16:50:08:  <team v='edited'/>
16:50:08:  <user v='edited'/>
16:50:08:
16:50:08:  <!-- Folding Slots -->
16:50:08:  <slot id='0' type='CPU'>
16:50:08:    <next-unit-percentage v='98'/>
16:50:08:  </slot>
16:50:08:  <slot id='1' type='GPU'>
16:50:08:    <max-packet-size v='9'/>
16:50:08:    <next-unit-percentage v='90'/>
16:50:08:  </slot>
16:50:08:</config>

Do I need to restart FAHClient? Edit: uninstalled old ver & installed 7.5.1

Please explain why you had this error, This indicates that SOMETHING is happening on your ISP connection.

Code: Select all

07:10:02:WU01:FS01:Connecting to 65.254.110.245:80
07:10:04:WU01:FS01:Assigned to work server 140.163.4.231
07:10:04:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 140.163.4.231
07:10:04:WU01:FS01:Connecting to 140.163.4.231:8080
07:10:04:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
07:10:04:WU01:FS01:Connecting to 140.163.4.231:80
07:10:04:ERROR:WU01:FS01:Exception: Failed to connect to 140.163.4.231:80: A socket operation was attempted to an unreachable network.
07:10:05:ERROR:WU01:FS01:Exception: Could not get IP address for assign-GPU.stanford.edu: No such host is known. 
07:10:05:ERROR:WU01:FS01:Exception: Could not get IP address for assign-GPU2.stanford.edu: No such host is known. 
07:10:05:WARNING:WU01:FS01:Exception: Failed to find any IP addresses for assignment servers
07:10:05:ERROR:WU01:FS01:Exception: Could not get an assignment

The fact that it apparently recovers doesn't change the fact that it actually happened.

I set FAHClient to start automatically when Windows starts (using Task Scheduler, set to run whether user is logged on or not), I have no proof but I believe this error happens because Windows hasn't connected to network yet when FAHClient starts (after logon, I also get the Windows error message saying that my local network drive is offline, but in fact I can still access it normally). Sometimes I also get that error when there are pending Windows Updates.

I just see new FAH version 7.5.1, let me update first.

Edit: just uninstalled old ver and installed 7.5.1, let me monitor it and get back to you later.

Code: Select all

*********************** Log Started 2018-07-24T17:07:56Z ***********************
17:07:56:************************* Folding@home Client *************************
17:07:56:        Website: https://foldingathome.org/
17:07:56:      Copyright: (c) 2009-2018 foldingathome.org
17:07:56:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
17:07:56:           Args: 
17:07:56:         Config: E:\FAH\config.xml
17:07:56:******************************** Build ********************************
17:07:56:        Version: 7.5.1
17:07:56:           Date: May 11 2018
17:07:56:           Time: 13:06:32
17:07:56:     Repository: Git
17:07:56:       Revision: 4705bf53c635f88b8fe85af7675557e15d491ff0
17:07:56:         Branch: master
17:07:56:       Compiler: Visual C++ 2008
17:07:56:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
17:07:56:       Platform: win32 10
17:07:56:           Bits: 32
17:07:56:           Mode: Release
17:07:56:******************************* System ********************************
17:07:56:            CPU: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz
17:07:56:         CPU ID: GenuineIntel Family 6 Model 158 Stepping 10
17:07:56:           CPUs: 12
17:07:56:         Memory: 31.94GiB
17:07:56:    Free Memory: 20.04GiB
17:07:56:        Threads: WINDOWS_THREADS
17:07:56:     OS Version: 6.2
17:07:56:    Has Battery: false
17:07:56:     On Battery: false
17:07:56:     UTC Offset: 8
17:07:56:            PID: 1164
17:07:56:            CWD: E:\FAH
17:07:56:             OS: Windows 10 Enterprise
17:07:56:        OS Arch: AMD64
17:07:56:           GPUs: 1
17:07:56:          GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:7 GP102 [GeForce GTX 1080 Ti] 11380
17:07:56:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:6.1 Driver:9.2
17:07:56:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:398.36
17:07:56:  Win32 Service: false
17:07:56:***********************************************************************
17:07:56:<config>
17:07:56:  <!-- Folding Core -->
17:07:56:  <checkpoint v='30'/>
17:07:56:
17:07:56:  <!-- Network -->
17:07:56:  <proxy v=':8080'/>
17:07:56:
17:07:56:  <!-- Slot Control -->
17:07:56:  <power v='FULL'/>
17:07:56:
17:07:56:  <!-- User Information -->
17:07:56:  <passkey v='********************************'/>
17:07:56:  <team v='38156'/>
17:07:56:  <user v='VNK'/>
17:07:56:
17:07:56:  <!-- Folding Slots -->
17:07:56:  <slot id='0' type='CPU'>
17:07:56:    <next-unit-percentage v='98'/>
17:07:56:    <paused v='true'/>
17:07:56:  </slot>
17:07:56:  <slot id='1' type='GPU'>
17:07:56:    <max-packet-size v='9'/>
17:07:56:    <next-unit-percentage v='90'/>
17:07:56:  </slot>
17:07:56:</config>
17:07:56:Trying to access database...
17:07:56:Upgrading database schema from version 14 to 16
17:07:56:Successfully acquired database lock
17:07:56:Enabled folding slot 00: PAUSED cpu:11 (by user)
17:07:56:Enabled folding slot 01: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380
17:07:56:ERROR:Exception: Failed to register systray icon: Unspecified error
17:07:56:WU01:FS01:Connecting to 65.254.110.245:8080
17:07:57:WU01:FS01:Assigned to work server 140.163.4.231
17:07:57:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 140.163.4.231
17:07:57:WU01:FS01:Connecting to 140.163.4.231:8080
17:07:59:WU01:FS01:Downloading 5.42MiB
17:08:11:WU01:FS01:Download 1.15%
17:08:24:WU01:FS01:Download 2.31%
17:08:37:WU01:FS01:Download 3.46%
17:08:50:WU01:FS01:Download 4.61%
17:09:02:WU01:FS01:Download 5.77%
17:09:14:WU01:FS01:Download 6.92%
17:09:26:WU01:FS01:Download 8.08%
17:09:38:WU01:FS01:Download 9.23%
17:09:51:WU01:FS01:Download 10.38%
17:10:04:WU01:FS01:Download 11.54%
17:10:17:WU01:FS01:Download 12.69%
17:10:29:WU01:FS01:Download 13.84%
17:10:41:WU01:FS01:Download 15.00%
17:10:48:FS00:Unpaused
17:10:48:WU00:FS00:Starting
17:10:48:WARNING:WU00:FS00:AS lowered CPUs from 11 to 10
17:10:48:WU00:FS00:Running FahCore: E:\FAHClient/FAHCoreWrapper.exe E:\FAH\cores/cores.foldingathome.org/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 705 -lifeline 1164 -checkpoint 30 -np 10
17:10:48:WU00:FS00:Started FahCore on PID 8232
17:10:48:WU00:FS00:Core PID:6556
17:10:48:WU00:FS00:FahCore 0xa4 started
17:10:49:WU00:FS00:0xa4:
17:10:49:WU00:FS00:0xa4:*------------------------------*
17:10:49:WU00:FS00:0xa4:Folding@Home Gromacs GB Core
17:10:49:WU00:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
17:10:49:WU00:FS00:0xa4:
17:10:49:WU00:FS00:0xa4:Preparing to commence simulation
17:10:49:WU00:FS00:0xa4:- Looking at optimizations...
17:10:49:WU00:FS00:0xa4:- Files status OK
17:10:49:WU00:FS00:0xa4:- Expanded 378827 -> 597116 (decompressed 157.6 percent)
17:10:49:WU00:FS00:0xa4:Called DecompressByteArray: compressed_data_size=378827 data_size=597116, decompressed_data_size=597116 diff=0
17:10:49:WU00:FS00:0xa4:- Digital signature verified
17:10:49:WU00:FS00:0xa4:
17:10:49:WU00:FS00:0xa4:Project: 8659 (Run 515, Clone 0, Gen 75)
17:10:49:WU00:FS00:0xa4:
17:10:49:WU00:FS00:0xa4:Assembly optimizations on if available.
17:10:49:WU00:FS00:0xa4:Entering M.D.
17:10:54:WU01:FS01:Download 16.15%
17:10:55:WU00:FS00:0xa4:Using Gromacs checkpoints
17:10:55:WU00:FS00:0xa4:Mapping NT from 10 to 10 
17:10:55:WU00:FS00:0xa4:Resuming from checkpoint
17:10:55:WU00:FS00:0xa4:Verified 00/wudata_01.log
17:10:55:WU00:FS00:0xa4:Verified 00/wudata_01.trr
17:10:55:WU00:FS00:0xa4:Verified 00/wudata_01.xtc
17:10:55:WU00:FS00:0xa4:Verified 00/wudata_01.edr
17:10:55:WU00:FS00:0xa4:Completed 749130 out of 2500000 steps  (29%)
17:10:57:WU00:FS00:0xa4:Completed 750000 out of 2500000 steps  (30%)
17:10:59:Removing old file 'configs/config-20180704-152303.xml'
17:10:59:Saving configuration to config.xml
17:10:59:<config>
17:10:59:  <!-- Folding Core -->
17:10:59:  <checkpoint v='30'/>
17:10:59:
17:10:59:  <!-- Network -->
17:10:59:  <proxy v=':8080'/>
17:10:59:
17:10:59:  <!-- Slot Control -->
17:10:59:  <power v='FULL'/>
17:10:59:
17:10:59:  <!-- User Information -->
17:10:59:  <passkey v='********************************'/>
17:10:59:  <team v='edited'/>
17:10:59:  <user v='edited'/>
17:10:59:
17:10:59:  <!-- Folding Slots -->
17:10:59:  <slot id='0' type='CPU'>
17:10:59:    <next-unit-percentage v='98'/>
17:10:59:  </slot>
17:10:59:  <slot id='1' type='GPU'>
17:10:59:    <max-packet-size v='9'/>
17:10:59:    <next-unit-percentage v='90'/>
17:10:59:  </slot>
17:10:59:</config>
17:11:29:WU01:FS01:Download 17.31%
17:11:58:WU00:FS00:0xa4:Completed 775000 out of 2500000 steps  (31%)
17:12:09:WU01:FS01:Download 18.46%
17:12:51:WU01:FS01:Download 19.61%
17:12:59:WU00:FS00:0xa4:Completed 800000 out of 2500000 steps  (32%)
17:13:33:WU01:FS01:Download 20.77%
17:13:59:WU00:FS00:0xa4:Completed 825000 out of 2500000 steps  (33%)
17:14:14:WU01:FS01:Download 21.92%
17:14:52:WU01:FS01:Download 23.07%
17:14:59:WU00:FS00:0xa4:Completed 850000 out of 2500000 steps  (34%)
17:15:33:WU01:FS01:Download 24.23%
17:15:59:WU00:FS00:0xa4:Completed 875000 out of 2500000 steps  (35%)
17:16:16:WU01:FS01:Download 25.38%
17:16:58:WU01:FS01:Download 26.53%
17:16:59:WU00:FS00:0xa4:Completed 900000 out of 2500000 steps  (36%)
17:17:42:WU01:FS01:Download 27.69%
17:17:59:WU00:FS00:0xa4:Completed 925000 out of 2500000 steps  (37%)
17:18:24:WU01:FS01:Download 28.84%
17:18:59:WU00:FS00:0xa4:Completed 950000 out of 2500000 steps  (38%)
17:19:06:WU01:FS01:Download 30.00%

[bruce]

It sounds like you're trying to somehow overcome the Windows restriction that GPUs can only be used by FAH when the FAHCore is running in the memory-space of the logged on user by (re-)starting FAHClient when user A logs off an user B logs on. I don't know if anybody has done that successfully. I'm sure that Microsoft had a reason to build that restriction into Windows.

Have you though about Linux -- which is designed with genuine multi-user support built in.