Short responses from 18.218.241.186 & 65.254.110.245
Posted: Tue May 05, 2020 6:14 am
I've got 7 active clients (3 CPU, 2x Nvidia 1030, 1xNvidia 1650, 1x Nvidia 970), most of the time they're purring along nicely, but recently I've noticed issues with short responses from a couple of servers (18.xxx and 65.xxx). There appear to be reports of other folk seeing the same, which makes me think it's not an issue on my side.
I'll put the logs at the bottom of the post, but in the way of a theory it looks, to me, like a situation I've seen before with a load balancer or firewall terminating the connection before the data has been served to the client due to being overloaded. This can happen if the firewall/load balancer is overloaded with connections and starts dropping some to service faster or more recent connections (depends on the configuration of the piece of equipment). This isn't a 100% reproducing situation, which makes me think the network kit may be spiking above it's peak capacity occasionally rather than being constantly at it, so to investigate someone would need to dig into p95 load over short periods of time (seconds at most) rather than looking at an average over several seconds or minutes.
Anyway, hope this is useful, and now for the logs;
Machine 1;
05:45:44:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
05:45:44:WU01:FS00:Connecting to 18.218.241.186:80
05:45:45:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
05:45:45:WU01:FS00:Connecting to 65.254.110.245:80
...
05:45:46:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
05:45:46:ERROR:WU01:FS00:Exception: Could not get an assignment
05:47:21:WU01:FS00:Connecting to 65.254.110.245:80
05:47:22:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
05:47:22:WU01:FS00:Connecting to 18.218.241.186:80
05:47:22:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
05:47:22:WU01:FS00:Connecting to 65.254.110.245:80
05:47:23:WU01:FS00:Assigned to work server 128.252.203.10
05:47:23:WU01:FS00:Requesting new work unit for slot 00: READY gpu:0:GM204 [GeForce GTX 970] 3494 from 128.252.203.10
05:47:23:WU01:FS00:Connecting to 128.252.203.10:8080
05:47:54:ERROR:WU01:FS00:Exception: 10002: Received short response, expected 512 bytes, got 0
05:49:58:WU01:FS00:Connecting to 65.254.110.245:80
05:49:59:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
05:49:59:WU01:FS00:Connecting to 18.218.241.186:80
05:49:59:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
05:49:59:WU01:FS00:Connecting to 65.254.110.245:80
05:50:30:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:80': 10002: Received short response, expected 272 bytes, got 0
Machine #2;
17:33:04:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
17:33:04:WU02:FS00:Connecting to 18.218.241.186:80
17:33:05:WARNING:WU02:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
17:33:05:WU02:FS00:Connecting to 65.254.110.245:80
17:33:05:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
17:33:05:WU02:FS00:Connecting to 18.218.241.186:80
17:33:06:WU02:FS00:Assigned to work server 13.82.98.119
17:33:06:WU02:FS00:Requesting new work unit for slot 00: READY gpu:0:TU117 [GeForce GTX 1650] from 13.82.98.119
17:33:06:WU02:FS00:Connecting to 13.82.98.119:8080
17:33:10:ERROR:WU02:FS00:Exception: Server did not assign work unit
23:33:03:WU02:FS00:Connecting to 65.254.110.245:80
23:33:04:WU02:FS00:Assigned to work server 140.163.4.231
23:33:04:WU02:FS00:Requesting new work unit for slot 00: READY gpu:0:TU117 [GeForce GTX 1650] from 140.163.4.231
23:33:04:WU02:FS00:Connecting to 140.163.4.231:8080
23:33:42:ERROR:WU02:FS00:Exception: 10002: Received short response, expected 512 bytes, got 0
05:33:04:WU02:FS00:Connecting to 65.254.110.245:80
05:33:04:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
05:33:04:WU02:FS00:Connecting to 18.218.241.186:80
05:33:05:WARNING:WU02:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
05:33:05:WU02:FS00:Connecting to 65.254.110.245:80
05:33:05:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
05:33:05:WU02:FS00:Connecting to 18.218.241.186:80
05:33:06:WARNING:WU02:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
05:33:06:ERROR:WU02:FS00:Exception: Could not get an assignment
I'll put the logs at the bottom of the post, but in the way of a theory it looks, to me, like a situation I've seen before with a load balancer or firewall terminating the connection before the data has been served to the client due to being overloaded. This can happen if the firewall/load balancer is overloaded with connections and starts dropping some to service faster or more recent connections (depends on the configuration of the piece of equipment). This isn't a 100% reproducing situation, which makes me think the network kit may be spiking above it's peak capacity occasionally rather than being constantly at it, so to investigate someone would need to dig into p95 load over short periods of time (seconds at most) rather than looking at an average over several seconds or minutes.
Anyway, hope this is useful, and now for the logs;
Machine 1;
05:45:44:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
05:45:44:WU01:FS00:Connecting to 18.218.241.186:80
05:45:45:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
05:45:45:WU01:FS00:Connecting to 65.254.110.245:80
...
05:45:46:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
05:45:46:ERROR:WU01:FS00:Exception: Could not get an assignment
05:47:21:WU01:FS00:Connecting to 65.254.110.245:80
05:47:22:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
05:47:22:WU01:FS00:Connecting to 18.218.241.186:80
05:47:22:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
05:47:22:WU01:FS00:Connecting to 65.254.110.245:80
05:47:23:WU01:FS00:Assigned to work server 128.252.203.10
05:47:23:WU01:FS00:Requesting new work unit for slot 00: READY gpu:0:GM204 [GeForce GTX 970] 3494 from 128.252.203.10
05:47:23:WU01:FS00:Connecting to 128.252.203.10:8080
05:47:54:ERROR:WU01:FS00:Exception: 10002: Received short response, expected 512 bytes, got 0
05:49:58:WU01:FS00:Connecting to 65.254.110.245:80
05:49:59:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
05:49:59:WU01:FS00:Connecting to 18.218.241.186:80
05:49:59:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
05:49:59:WU01:FS00:Connecting to 65.254.110.245:80
05:50:30:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:80': 10002: Received short response, expected 272 bytes, got 0
Machine #2;
17:33:04:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
17:33:04:WU02:FS00:Connecting to 18.218.241.186:80
17:33:05:WARNING:WU02:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
17:33:05:WU02:FS00:Connecting to 65.254.110.245:80
17:33:05:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
17:33:05:WU02:FS00:Connecting to 18.218.241.186:80
17:33:06:WU02:FS00:Assigned to work server 13.82.98.119
17:33:06:WU02:FS00:Requesting new work unit for slot 00: READY gpu:0:TU117 [GeForce GTX 1650] from 13.82.98.119
17:33:06:WU02:FS00:Connecting to 13.82.98.119:8080
17:33:10:ERROR:WU02:FS00:Exception: Server did not assign work unit
23:33:03:WU02:FS00:Connecting to 65.254.110.245:80
23:33:04:WU02:FS00:Assigned to work server 140.163.4.231
23:33:04:WU02:FS00:Requesting new work unit for slot 00: READY gpu:0:TU117 [GeForce GTX 1650] from 140.163.4.231
23:33:04:WU02:FS00:Connecting to 140.163.4.231:8080
23:33:42:ERROR:WU02:FS00:Exception: 10002: Received short response, expected 512 bytes, got 0
05:33:04:WU02:FS00:Connecting to 65.254.110.245:80
05:33:04:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
05:33:04:WU02:FS00:Connecting to 18.218.241.186:80
05:33:05:WARNING:WU02:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
05:33:05:WU02:FS00:Connecting to 65.254.110.245:80
05:33:05:WARNING:WU02:FS00:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
05:33:05:WU02:FS00:Connecting to 18.218.241.186:80
05:33:06:WARNING:WU02:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
05:33:06:ERROR:WU02:FS00:Exception: Could not get an assignment