Page 1 of 1

GPU WU Uploads Backing Up P14220 ~ P14250

Posted: Fri Sep 27, 2019 2:52 am
by dfgirl12
I'm starting to experience a different, but related problem of WU taking longer to upload (original posts from: viewtopic.php?f=18&t=31876&p=308997#p308997) than it takes a 2080Ti / 1080Ti to complete them.
It could just be my cable internet connection, but currently having 37+ concurrent 50MB-100MB uploads is choking it to death (fast or slow servers: once the connection is clogged, then none of them get uploaded for hours or days).
My uploads are 4+ WU deep on some machines, and getting worse today.

Example:
<<Image: Had 5 WUs waiting to send for a single folding slot>>

Are there any plans for the larger WU sizes and not enough return data channel capacity to handle it (like longer WU run times)?
Any suggestions, other than turning off half (~20) of the GPUs, or getting a fiber connection (not really available for me yet)?
Or, like a setting to limit only uploads to 1 upload per FAH Client?

Without running a speed test, I know I can typically upload 100-300 KB/s with my connection. But, all it takes is a few slow FAH uploads and it starts backing up when the WUs take ~30 minutes to fold.

Thanks

Re: GPU WU Uploads Backing Up P14220 ~ P14250

Posted: Fri Sep 27, 2019 11:28 am
by dfgirl12
I 'finished' all the current WUs so they were idle, and disconnected half of the PCs from the network. It took 8-12 hours for half of the PCs to clear out the completed WUs. I'm adding in the others, one every 1-2 hours.

This problem has only happened with these larger WUs and short folding times.

I'm going to try FAH setting of: max-queue=2 (which didn't help, or limit the upload queue). Hopefully, something like that will keep the upload queue from spiraling out of control. Any other ideas?

Re: GPU WU Uploads Backing Up P14220 ~ P14250

Posted: Fri Sep 27, 2019 6:02 pm
by bruce
there are several topics discussing the same problem. There have been network problems at temple.edu. The network admins are aware of the problem. See the post here.

The two Work Servers vav3.ocis.temple.edu and vav4.ocis.temple.edu apparently have been throttled (perhaps by a shared campus router) and they require a certain minimum bandwidth to support FAH's communications needs..

Re: GPU WU Uploads Backing Up P14220 ~ P14250

Posted: Sat Sep 28, 2019 2:17 am
by dfgirl12
It looks like with those servers back online again, the congestion is being alleviated with more WU variety. Thanks! :)