A WU just finished on one of my GPU instances, but seems it can't connect to the WorkServer, keeps failing on port 8080 and port 80. GPU is running idle at the moment. Some issues in the background @ FAH?
04:17:22:WU00:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
04:17:22:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
04:17:22:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11751 run:0 clone:1046 gen:0 core:0x22 unit:0x000000038ca304e75e6a803eb583808f
04:17:22:WU00:FS01:Uploading 33.37MiB to 140.163.4.231
04:17:22:WU00:FS01:Connecting to 140.163.4.231:8080
04:18:19:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
04:18:19:WU01:FS01:Connecting to 140.163.4.231:80
04:19:32:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
04:19:32:WU00:FS01:Connecting to 140.163.4.231:80
04:20:11:ERROR:WU01:FS01:Exception: 10002: Received short response, expected 512 bytes, got 0
04:20:12:WU01:FS01:Connecting to 65.254.110.245:80
04:20:12:WU01:FS01:Assigned to work server 140.163.4.231
04:20:12:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:TU104GL [Tesla T4] from 140.163.4.231
04:20:12:WU01:FS01:Connecting to 140.163.4.231:8080
04:21:43:WARNING:WU00:FS01:Exception: Failed to send results to work server: Failed to connect to 140.163.4.231:80: Connection timed out
04:21:43:WU00:FS01:Trying to send results to collection server
04:21:43:WU00:FS01:Uploading 33.37MiB to 128.252.203.4
04:21:43:WU00:FS01:Connecting to 128.252.203.4:8080
04:21:45:WU00:FS01:Upload complete
04:21:45:WU00:FS01:Server responded WORK_ACK (400)
04:21:45:WU00:FS01:Final credit estimate, 64685.00 points
04:21:45:WU00:FS01:Cleaning up
04:22:22:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
04:22:22:WU01:FS01:Connecting to 140.163.4.231:80
04:24:33:ERROR:WU01:FS01:Exception: Failed to connect to 140.163.4.231:80: Connection timed out
04:24:33:WU01:FS01:Connecting to 65.254.110.245:80
04:24:34:WU01:FS01:Assigned to work server 140.163.4.231
04:24:34:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:TU104GL [Tesla T4] from 140.163.4.231
04:24:34:WU01:FS01:Connecting to 140.163.4.231:8080
I had this too, it stopped by itself (I think?) after I closed and started the client. Either the restart helped, or it finally found a WU.
I *think* the problem is simply that right now MANY more are folding, so the servers can't keep up with dishing out WUs. (See also the thread in announcements - they had to add in another server just to keep up with demand of giving out WUs.)
It seems I can't write a signature that both conveys my feelings and doesn't look like a miserable trolling attempt...
I would have thought that the assignment servers would be able to detect when a work server is down and cease pointing clients to it until the ping is re-established. Apparently that is not the case. Or if it is, then for some reason 65.254.110.245 thinks that 140.163.4.231 is still alive - when it clearly isn't.
Hi folks! My GPU today stopped receiving assignments -- no more WUs available! Here's the link to my thread on this forum with some logging information: viewtopic.php?f=18&t=32330
[root@ip-172-31-2-53 ~]# grep "WorkServer connection failed" /var/lib/fahclient/log.txt
04:44:54:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
08:19:09:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:23:15:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:27:37:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:31:59:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:36:21:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:40:44:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
10:58:05:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
[root@ip-172-31-3-58 ~]# grep "WorkServer connection failed" /var/lib/fahclient/log.txt
04:46:02:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
04:46:45:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
04:51:07:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
04:55:29:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
04:59:52:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
05:04:14:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
08:21:31:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:23:30:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
08:25:53:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:30:15:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:37:15:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
Issue appears to have returned throughout the day. At present one of my GPU instances is sitting idle :/. Is there too many GPU's and not enough work? Or is the bottleneck the work server not keeping up?
[root@ip-172-31-2-53 ~]# grep "WorkServer connection failed" /var/lib/fahclient/log.txt
04:44:54:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
08:19:09:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:23:15:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:27:37:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:31:59:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:36:21:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:40:44:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
10:58:05:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
20:52:04:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
22:54:01:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
02:27:52:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
04:24:08:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
[ec2-user@ip-172-31-2-53 ~]$ grep "WorkServer connection failed" /var/lib/fahclient/log.txt
04:44:54:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
08:19:09:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:23:15:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:27:37:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:31:59:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:36:21:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
08:40:44:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
10:58:05:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
20:52:04:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
22:54:01:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
02:27:52:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
04:24:08:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
Looks like we've given the team more resources than they can dish work out to Both my Tesla T4's are running idle, maybe I should build a trigger to shut down these instances until work is available.
[root@ip-172-31-2-53 ~]# grep "Empty work server assignment" /var/lib/fahclient/ log.txt
00:15:22:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
00:15:22:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
00:15:23:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
00:15:23:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
00:16:23:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
00:16:23:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
00:18:00:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
00:18:00:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
00:24:54:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
00:24:54:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
00:37:38:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
00:37:38:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
00:38:38:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
00:38:38:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
00:42:51:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
01:00:48:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
01:00:48:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
01:29:50:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
01:29:50:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
02:16:49:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
02:16:49:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
03:32:49:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
03:32:49:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
04:18:38:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
04:18:38:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
04:18:38:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
04:18:38:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
04:19:39:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
04:19:39:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
05:35:49:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
05:35:49:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
06:22:22:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
06:22:22:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
06:22:22:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
06:22:22:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
06:23:23:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
09:30:08:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
09:30:08:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
09:30:08:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
09:30:09:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
09:31:09:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
09:31:09:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
09:32:46:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
09:35:23:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
09:35:23:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
09:39:37:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
09:39:37:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
09:46:29:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
09:46:29:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
09:57:34:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
09:57:34:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
10:15:31:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
12:22:09:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
12:22:09:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
12:22:09:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
12:22:09:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
12:23:09:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
12:23:09:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
12:24:47:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
12:24:47:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
12:27:24:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
12:27:24:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
12:31:38:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
12:31:38:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
14:16:49:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
14:16:49:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
16:13:17:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
16:14:56:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
16:14:56:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
16:15:56:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
16:15:56:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
16:17:33:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
16:17:33:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
16:20:10:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
16:20:10:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
16:24:25:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
16:24:25:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
16:31:16:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
16:42:22:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
16:42:22:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
17:29:21:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
17:29:21:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
18:16:19:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
18:16:19:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
19:32:20:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
19:32:20:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
20:16:50:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
20:16:50:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
21:35:20:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
21:35:20:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
00:54:20:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
00:54:20:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
02:16:50:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
02:16:50:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
06:16:20:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
06:16:20:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
12:16:20:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
12:16:20:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
[root@ip-172-31-3-58 ~]# grep "Empty work server assignment" /var/lib/fahclient/ log.txt
16:35:46:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
16:40:00:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
16:57:57:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
16:57:57:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
22:10:10:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
22:10:10:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
22:10:11:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
22:10:11:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
02:11:56:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
02:11:56:WARNING:WU02:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
02:11:57:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
02:11:57:WARNING:WU02:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
02:12:57:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
02:12:57:WARNING:WU02:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
02:14:34:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
02:14:34:WARNING:WU02:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
11:17:23:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
11:17:23:WARNING:WU02:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
11:17:23:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
11:17:23:WARNING:WU02:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
11:18:24:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
11:18:24:WARNING:WU02:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
14:14:33:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
14:14:33:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
14:14:33:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
14:14:33:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
16:53:52:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
16:54:02:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
01:04:17:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
01:07:07:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
01:07:07:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
01:08:07:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
01:08:07:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
01:09:44:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
01:09:44:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
01:16:36:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
01:16:36:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
01:23:27:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
01:23:27:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
01:34:32:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
01:34:32:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
01:52:29:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
01:52:29:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
07:46:27:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
07:46:27:WARNING:WU02:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
07:46:27:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
07:46:27:WARNING:WU02:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
07:47:27:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
07:47:27:WARNING:WU02:FS00:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
07:49:05:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:8080': Empty work server assignment
10:56:05:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
10:56:05:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
10:56:05:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
10:56:05:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
10:57:06:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
10:57:06:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
10:58:43:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
10:58:43:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
11:01:20:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
11:01:20:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
11:05:34:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
11:05:34:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
11:12:26:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
11:12:26:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
11:23:31:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
13:31:20:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
13:31:20:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
13:31:21:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
13:31:21:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
13:32:21:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
13:32:21:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
13:33:58:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
13:33:58:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
13:36:35:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
13:36:35:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
13:40:49:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
13:40:49:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
13:47:41:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
13:47:41:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
13:58:46:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
13:58:46:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
14:16:43:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
14:16:43:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
16:51:03:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
16:51:03:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
16:51:03:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
16:51:03:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
16:52:04:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
16:52:04:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
16:53:41:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
16:53:41:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
16:56:18:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
17:00:41:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
17:00:41:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
17:07:32:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
23:03:27:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
23:03:27:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
23:03:27:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
23:03:27:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
23:04:28:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
23:04:28:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
23:06:05:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
23:06:05:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
23:08:42:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
23:08:42:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
23:12:56:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
23:12:56:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
23:19:48:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
03:02:03:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
03:02:03:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
03:02:04:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
03:02:04:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
03:03:04:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
03:03:04:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
03:04:41:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
03:04:41:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
03:07:18:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
03:07:18:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
03:11:32:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
03:11:32:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
03:18:24:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
03:18:24:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
03:29:29:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
03:29:29:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
03:47:26:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
03:47:26:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
04:16:28:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
04:16:28:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
05:03:27:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
05:03:27:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
06:19:28:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
06:19:28:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
08:22:28:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
08:22:28:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
11:41:28:WARNING:WU00:FS01:Failed to get assignment from '65.254.110.245:80': Em pty work server assignment
11:41:28:WARNING:WU00:FS01:Failed to get assignment from '18.218.241.186:80': Em pty work server assignment
aaronbrighton wrote:Issue appears to have returned throughout the day. At present one of my GPU instances is sitting idle :/. Is there too many GPU's and not enough work? Or is the bottleneck the work server not keeping up?
IMHO the work server is the bottleneck, as has been mentioned by several forum admins.
Both my Tesla T4's are running idle, maybe I should build a trigger to shut down these instances until work is available.