core22 0.0.10 released to full FAH!

[JohnChodera]

We're investigating to see why certain RUNs in that project are slow. Will let you know what we find out.

Thanks again!

~ John Chodera // MSKCC

[ajm]

No thank YOU for what you are doing with FAH! It's really great!

But, let me complain a bit more about those other 13415 WUs, because it's really taking fearful proportions. Last night, my 5700XT has done almost nothing else. Presently (13415 (909, 0, 0)), she is producing a PPD of 82k (after over 20% of the WU), that is, about 16-17 times less than what she can generate. This while my two 1050ti, on another kit, obviously much better suited for these 13415 WUs, didn't get any. This is not about me - I still can get a quite comfortable PPD on the whole. But looking at the big picture, it is a terrible waste of resources.

So this morning I tend to think that this GitHub topic https://github.com/FoldingAtHome/fah-issues/issues/1527 should get some priority. And I added a comment there:

How about adding to the client a small benchmark function, that would run automatically each time a slot is created or modified? A few seconds would probably be enough for deciding on solid ground which kind of WUs should best be assigned to the slot, or not.

EDIT: List of 13415 WUs on the 5700XT from this morning on:
13415 (909, 0, 0) -> PPD 82k (~6% capacity)
13415 (231, 26, 0) -> PPD 350k (~25%)
13415 (439, 26, 1) -> PPD 330k (~25%)
13415 (507, 28, 0) -> PPD 340k (~25%)
13415 (998, 24, 1) -> PPD 340k (~25%)
13415 (195, 28, 1) -> PPD 340k (~25%)
13413 (771, 2, 1) -> PPD 320k (~23%)
13415 (974, 31, 1) -> PPD 330k (~25%)
13415 (969, 32, 1) -> PPD 330k (~25%)
13415 (106, 35, 0) -> PPD 330k (~25%)
13415 (459, 36, 0) -> PPD 330k (~25%)

[Nuitari]

Some more datapoints, on RX570. Multiple units completed between 4 to 5h which seems to make sense.
Others have been going on for much much longer, with one of them possibly expiring out. 13415 (911,3,1)
Other slow ones:
13415 (866,9,1)
13415 (830,19,0)
13415 (815,20,0)



Good speed one
13415 (916,20, 0)
project:13415 run:90 clone:8 gen:1
project:13415 run:569 clone:1 gen:0
project:13415 run:344 clone:0 gen:1
project:13415 run:687 clone:3 gen:0
project:13415 run:678 clone:3 gen:0
project:13415 run:393 clone:4 gen:1
project:13415 run:282 clone:6 gen:0
project:13415 run:535 clone:4 gen:1
project:13415 run:425 clone:7 gen:0
project:13415 run:326 clone:8 gen:1
project:13415 run:114 clone:9 gen:1
project:13415 run:104 clone:11 gen:0
project:13415 run:532 clone:9 gen:1
project:13415 run:234 clone:13 gen:0
project:13415 run:149 clone:13 gen:0
project:13415 run:612 clone:13 gen:1
project:13415 run:311 clone:8 gen:1
project:13415 run:893 clone:16 gen:0
project:13415 run:562 clone:17 gen:1

[Nuitari]

Overnight the situation changed to this

RX570: 13415 (911,3,1) 73% TPF of 18m 55s / 8h30m to go
RX570: 13415 (866,9,1) 89% TPF of 11m 15s / 2h to go
RX560: 13415 (830, 19, 0) 65% TPF of 11m / 6h15m to go
RX570: 13415 (815,20,0) 51% TPF of 11m / 9h15m to go
RX570: 13415 (148,30,0) 38% TPF of 2m 46s / 3h to go
RX570: 13415 (730,26,1) 80% TPF of 2m 49s / 1h to go
Carrizo: 13415 (79,32,0) 13% TPF of 3m 24s / 5h to go

One thing that is clearly problematic is that 13415 (79,32,0) is assigned to the Carrizo slot, which is a very very small GPU within the CPU. A8-9600 RADEON R7.

[HugoNotte]

We're investigating to see why certain RUNs in that project are slow. Will let you know what we find out.

Thanks again!

~ John Chodera // MSKCC

I had one of those 13141x WUs a few days ago. It processed quite fast but had loads of PPD / TPF fluctuations throughout, not just in the beginning. My normal PPD on the GTX750TI is just under 100k, I think this WU overall PPD was in the end around 110k, but in between the PPD did go up well above 200k, several times. I often see some kind of variation in the first 5% of a WU, but I haven't picked up fluctuations like that (100%) before.

[ajm]

13414 is normal, actually, as far as i can tell. I have had quite a few of them today, on several cards, and they perform within the excepted PPD range.
There are problems with 13413 and 13415, but I get them only on my AMD card - no occurrence of them on Nvidia cards.

[bruce]

How about adding to the client a small benchmark function, that would run automatically each time a slot is created or modified? A few seconds would probably be enough for deciding on solid ground which kind of WUs should best be assigned to the slot, or not.

I'm not sure that would provide the necessary information. The server already knows what your hardware is, but it doesn't know if you run 24x7 or just overnight so the benchmark probably wouldn't show the server what it needs to know. Moreover, a local benchmark is subject to manipulation by the cherrypickers.

[ajm]

There are always loopholes of course, but it would at least prevent the servers from sending me, for example, whole series of WUs, back to back, that utilize my card to 25% or less.

[Neil-B]

I am sure the servers won't be picking on you - those you will be competing against will most likely be seeing the same ... but at the moment what FAH needs is to get the benchmarking done and sort out the science - if that means using any/all kit below its optimum to get this done then that is what needs to happen !!

From posts in other threads people are working to try and get this work to the right resource, but part of that is knowing which kit is impacted most under certain conditions hence some of the oddities that are being seen ... and it is stuff the FAH team need to be done to sort out how science is generated in the future - they are learning as fast as they can.

Sometimes it isn't efficiency or min/maxing that is important ... at least not in the short term - give the FAH team some time and I am sure all those for whom the points are more obvious than the science will be happy again ... like the bottlenecking in March/April I am sure that the leaness of June will soon be a thing of the past - but for now there will continue to be unhappy posts in theses forums

[ajm]

Well put, Neil, thank you! I'll stop monitoring for the night.

[Curt3g]

Another data point on processing a Project 13414 WU (ongoing). With the RTX 2060, WU's have been taking roughly 2-4 hours (1.0-1.1M PPD). This one looks like it will take somewhat longer. There's plenty of RAM available and I only have 8 of the 16 CPU cores working on Rosetta WU's, with the remaining cores available. And no heat issues, everything seems to be running well.

System Info and Configuration:

Code: Select all

*********************** Log Started 2020-06-17T18:25:03Z ***********************
18:25:03:Trying to access database...
18:25:03:Successfully acquired database lock
18:25:03:Read GPUs.txt
18:25:03:Enabled folding slot 01: PAUSED gpu:0:TU106 [Geforce RTX 2060] (by user)
18:25:04:****************************** FAHClient ******************************
18:25:04:        Version: 7.6.13
18:25:04:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:25:04:      Copyright: 2020 foldingathome.org
18:25:04:       Homepage: https://foldingathome.org/
18:25:04:           Date: Apr 27 2020
18:25:04:           Time: 21:21:01
18:25:04:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
18:25:04:         Branch: master
18:25:04:       Compiler: Visual C++ 2008
18:25:04:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
18:25:04:       Platform: win32 10
18:25:04:           Bits: 32
18:25:04:           Mode: Release
18:25:04:         Config: C:\Users\Home Theater\AppData\Roaming\FAHClient\config.xml
18:25:04:******************************** CBang ********************************
18:25:04:           Date: Apr 24 2020
18:25:04:           Time: 17:07:55
18:25:04:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
18:25:04:         Branch: master
18:25:04:       Compiler: Visual C++ 2008
18:25:04:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
18:25:04:       Platform: win32 10
18:25:04:           Bits: 32
18:25:04:           Mode: Release
18:25:04:******************************* System ********************************
18:25:04:            CPU: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
18:25:04:         CPU ID: GenuineIntel Family 6 Model 158 Stepping 13
18:25:04:           CPUs: 16
18:25:04:         Memory: 63.91GiB
18:25:04:    Free Memory: 60.99GiB
18:25:04:        Threads: WINDOWS_THREADS
18:25:04:     OS Version: 6.2
18:25:04:    Has Battery: false
18:25:04:     On Battery: false
18:25:04:     UTC Offset: -7
18:25:04:            PID: 5676
18:25:04:            CWD: C:\Users\Home Theater\AppData\Roaming\FAHClient
18:25:04:  Win32 Service: false
18:25:04:             OS: Windows 10 Enterprise
18:25:04:        OS Arch: AMD64
18:25:04:           GPUs: 1
18:25:04:          GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:7 TU106 [Geforce RTX 2060]
18:25:04:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:7.5 Driver:11.0
18:25:04:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:445.87
18:25:04:******************************* libFAH ********************************
18:25:04:           Date: Apr 15 2020
18:25:04:           Time: 14:53:14
18:25:04:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
18:25:04:         Branch: master
18:25:04:       Compiler: Visual C++ 2008
18:25:04:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
18:25:04:       Platform: win32 10
18:25:04:           Bits: 32
18:25:04:           Mode: Release
18:25:04:***********************************************************************
18:25:05:<config>
18:25:05:  <!-- Folding Slot Configuration -->
18:25:05:  <cause v='COVID_19'/>
18:25:05:
18:25:05:  <!-- Network -->
18:25:05:  <proxy v=':8080'/>
18:25:05:
18:25:05:  <!-- Slot Control -->
18:25:05:  <power v='full'/>
18:25:05:
18:25:05:  <!-- User Information -->
18:25:05:  <passkey v='*****'/>
18:25:05:  <team v='254342'/>
18:25:05:  <user v='Curt3g'/>
18:25:05:
18:25:05:  <!-- Folding Slots -->
18:25:05:  <slot id='1' type='GPU'>
18:25:05:    <paused v='true'/>
18:25:05:  </slot>
18:25:05:</config>
18:25:48:FS01:Unpaused

And today's get and start up of Project: 13414 (Run 824, Clone 39, Gen 1). Other than snipping the upload steps, I've left in the interleaved completion of the previous WU (sheer laziness to extract ). And have left in the completion steps for the currently running 13414 WU to illustrate the variability in time per frame (correct terminology?).

Code: Select all

12:30:00:WU01:FS01:Connecting to assign1.foldingathome.org:80
12:30:00:WU01:FS01:Assigned to work server 140.163.4.231
12:30:00:WU01:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:TU106 [Geforce RTX 2060] from 140.163.4.231
12:30:00:WU01:FS01:Connecting to 140.163.4.231:8080
12:30:20:ERROR:WU01:FS01:Exception: 10002: Received short response, expected 512 bytes, got 0
12:30:20:WU01:FS01:Connecting to assign1.foldingathome.org:80
12:30:21:WU01:FS01:Assigned to work server 18.188.125.154
12:30:21:WU01:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:TU106 [Geforce RTX 2060] from 18.188.125.154
12:30:21:WU01:FS01:Connecting to 18.188.125.154:8080
12:30:23:WU01:FS01:Downloading 9.42MiB
12:30:29:WU01:FS01:Download 77.64%
12:30:30:WU01:FS01:Download complete
12:30:30:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13414 run:824 clone:39 gen:1 core:0x22 unit:0x0000000112bc7d9a5eed8c58bbe879cb
12:32:56:WU00:FS01:0x22:Completed 2000000 out of 2000000 steps (100%)
12:32:56:WU00:FS01:0x22:Average performance: 3.93174 ns/day
12:33:00:WU00:FS01:0x22:Saving result file ..\logfile_01.txt
12:33:00:WU00:FS01:0x22:Saving result file checkpointState.xml
12:33:02:WU00:FS01:0x22:Saving result file positions.xtc
12:33:04:WU00:FS01:0x22:Saving result file science.log
12:33:04:WU00:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
12:33:05:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
12:33:05:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14466 run:0 clone:742 gen:37 core:0x22 unit:0x0000003e03854c135eb9853136ea37a2
12:33:05:WU00:FS01:Uploading 83.62MiB to 3.133.76.19
12:33:05:WU01:FS01:Starting
12:33:05:WU00:FS01:Connecting to 3.133.76.19:8080
12:33:05:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" "C:\Users\Home Theater\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe" -dir 01 -suffix 01 -version 706 -lifeline 5676 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
12:33:05:WU01:FS01:Started FahCore on PID 22204
12:33:05:WU01:FS01:Core PID:23004
12:33:05:WU01:FS01:FahCore 0x22 started
12:33:06:WU01:FS01:0x22:*********************** Log Started 2020-06-21T12:33:05Z ***********************
12:33:06:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
12:33:06:WU01:FS01:0x22:       Core: Core22
12:33:06:WU01:FS01:0x22:       Type: 0x22
12:33:06:WU01:FS01:0x22:    Version: 0.0.10
12:33:06:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
12:33:06:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
12:33:06:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
12:33:06:WU01:FS01:0x22:       Date: Jun 16 2020
12:33:06:WU01:FS01:0x22:       Time: 14:33:22
12:33:06:WU01:FS01:0x22:   Revision: 147051aad40bcbec7d4b25105bbedfab425f1dc2
12:33:06:WU01:FS01:0x22:     Branch: core22-0.0.10
12:33:06:WU01:FS01:0x22:   Compiler: Visual C++ 2015
12:33:06:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
12:33:06:WU01:FS01:0x22:   Platform: win32 10
12:33:06:WU01:FS01:0x22:       Bits: 64
12:33:06:WU01:FS01:0x22:       Mode: Release
12:33:06:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
12:33:06:WU01:FS01:0x22:             <peastman@stanford.edu>
12:33:06:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 22204 -checkpoint 15
12:33:06:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
12:33:06:WU01:FS01:0x22:             0 -gpu 0
12:33:06:WU01:FS01:0x22:************************************ libFAH ************************************
12:33:06:WU01:FS01:0x22:       Date: Jun 15 2020
12:33:06:WU01:FS01:0x22:       Time: 18:05:04
12:33:06:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
12:33:06:WU01:FS01:0x22:     Branch: HEAD
12:33:06:WU01:FS01:0x22:   Compiler: Visual C++ 2015
12:33:06:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
12:33:06:WU01:FS01:0x22:   Platform: win32 10
12:33:06:WU01:FS01:0x22:       Bits: 64
12:33:06:WU01:FS01:0x22:       Mode: Release
12:33:06:WU01:FS01:0x22:************************************ CBang *************************************
12:33:06:WU01:FS01:0x22:       Date: Jun 16 2020
12:33:06:WU01:FS01:0x22:       Time: 14:31:33
12:33:06:WU01:FS01:0x22:   Revision: 75fcee0b8e713cb47f5191a3689d5f4f07244c7f
12:33:06:WU01:FS01:0x22:     Branch: HEAD
12:33:06:WU01:FS01:0x22:   Compiler: Visual C++ 2015
12:33:06:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
12:33:06:WU01:FS01:0x22:   Platform: win32 10
12:33:06:WU01:FS01:0x22:       Bits: 64
12:33:06:WU01:FS01:0x22:       Mode: Release
12:33:06:WU01:FS01:0x22:************************************ System ************************************
12:33:06:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
12:33:06:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 158 Stepping 13
12:33:06:WU01:FS01:0x22:       CPUs: 16
12:33:06:WU01:FS01:0x22:     Memory: 63.91GiB
12:33:06:WU01:FS01:0x22:Free Memory: 56.94GiB
12:33:06:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
12:33:06:WU01:FS01:0x22: OS Version: 6.2
12:33:06:WU01:FS01:0x22:Has Battery: false
12:33:06:WU01:FS01:0x22: On Battery: false
12:33:06:WU01:FS01:0x22: UTC Offset: -7
12:33:06:WU01:FS01:0x22:        PID: 23004
12:33:06:WU01:FS01:0x22:        CWD: C:\Users\Home Theater\AppData\Roaming\FAHClient\work
12:33:06:WU01:FS01:0x22:********************************************************************************
12:33:06:WU01:FS01:0x22:Project: 13414 (Run 824, Clone 39, Gen 1)
12:33:06:WU01:FS01:0x22:Unit: 0x0000000112bc7d9a5eed8c58bbe879cb
12:33:06:WU01:FS01:0x22:Reading tar file core.xml
12:33:06:WU01:FS01:0x22:Reading tar file integrator.xml
12:33:06:WU01:FS01:0x22:Reading tar file state.xml
12:33:06:WU01:FS01:0x22:Reading tar file system.xml
12:33:06:WU01:FS01:0x22:Digital signatures verified
12:33:06:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
12:33:06:WU01:FS01:0x22:Version 0.0.10
12:33:06:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
12:33:06:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
12:33:06:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
12:33:06:WU01:FS01:0x22:  Global context and integrator variables write interval: 250 steps (0.025%) [4000 total]
12:33:11:WU00:FS01:Upload 2.24%
12:33:17:WU00:FS01:Upload 6.43%
12:33:19:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
12:33:23:WU00:FS01:Upload 10.61%
12:33:29:WU00:FS01:Upload 14.87%
< snipped >
12:35:29:WU00:FS01:Upload 99.41%
12:35:30:WU00:FS01:Upload complete
12:35:30:WU00:FS01:Server responded WORK_ACK (400)
12:35:30:WU00:FS01:Final credit estimate, 227920.00 points
12:35:31:WU00:FS01:Cleaning up
******************************* Date: 2020-06-21 *******************************
12:40:33:WU01:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
12:47:39:WU01:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
12:54:48:WU01:FS01:0x22:Completed 30000 out of 1000000 steps (3%)
13:01:52:WU01:FS01:0x22:Completed 40000 out of 1000000 steps (4%)
13:08:59:WU01:FS01:0x22:Completed 50000 out of 1000000 steps (5%)
13:16:09:WU01:FS01:0x22:Completed 60000 out of 1000000 steps (6%)
13:23:17:WU01:FS01:0x22:Completed 70000 out of 1000000 steps (7%)
13:30:24:WU01:FS01:0x22:Completed 80000 out of 1000000 steps (8%)
13:37:31:WU01:FS01:0x22:Completed 90000 out of 1000000 steps (9%)
13:44:39:WU01:FS01:0x22:Completed 100000 out of 1000000 steps (10%)
13:51:47:WU01:FS01:0x22:Completed 110000 out of 1000000 steps (11%)
13:58:56:WU01:FS01:0x22:Completed 120000 out of 1000000 steps (12%)
14:06:05:WU01:FS01:0x22:Completed 130000 out of 1000000 steps (13%)
14:13:15:WU01:FS01:0x22:Completed 140000 out of 1000000 steps (14%)
14:20:25:WU01:FS01:0x22:Completed 150000 out of 1000000 steps (15%)
14:28:02:WU01:FS01:0x22:Completed 160000 out of 1000000 steps (16%)
14:38:40:WU01:FS01:0x22:Completed 170000 out of 1000000 steps (17%)
14:49:00:WU01:FS01:0x22:Completed 180000 out of 1000000 steps (18%)
14:57:38:WU01:FS01:0x22:Completed 190000 out of 1000000 steps (19%)
15:06:54:WU01:FS01:0x22:Completed 200000 out of 1000000 steps (20%)
15:16:25:WU01:FS01:0x22:Completed 210000 out of 1000000 steps (21%)
15:25:48:WU01:FS01:0x22:Completed 220000 out of 1000000 steps (22%)
15:35:29:WU01:FS01:0x22:Completed 230000 out of 1000000 steps (23%)
15:44:18:WU01:FS01:0x22:Completed 240000 out of 1000000 steps (24%)
15:52:29:WU01:FS01:0x22:Completed 250000 out of 1000000 steps (25%)
16:01:06:WU01:FS01:0x22:Completed 260000 out of 1000000 steps (26%)
16:09:37:WU01:FS01:0x22:Completed 270000 out of 1000000 steps (27%)
16:18:06:WU01:FS01:0x22:Completed 280000 out of 1000000 steps (28%)
16:26:35:WU01:FS01:0x22:Completed 290000 out of 1000000 steps (29%)
16:35:06:WU01:FS01:0x22:Completed 300000 out of 1000000 steps (30%)
16:43:37:WU01:FS01:0x22:Completed 310000 out of 1000000 steps (31%)
16:53:06:WU01:FS01:0x22:Completed 320000 out of 1000000 steps (32%)

Cheers,
Curt

[HaloJones]

I am sure the servers won't be picking on you

They don't "pick on you" but I do find cards get "stuck" onto particular servers so get into a loop of the same units over and over again.

[Neil-B]

So to make it easier to see the above log shows the following TPF values for each frame:

Code: Select all

Frame	TPF
1	00:07:06
2	00:07:09
3	00:07:04
4	00:07:07
5	00:07:10
6	00:07:08
7	00:07:07
8	00:07:07
9	00:07:08
10	00:07:08
11	00:07:09
12	00:07:09
13	00:07:10
14	00:07:10
15	00:07:37
16	00:10:38
17	00:10:20
18	00:08:38
19	00:09:16
20	00:09:31
21	00:09:23
22	00:09:41
23	00:08:49
24	00:08:11
25	00:08:37
26	00:08:31
27	00:08:29
28	00:08:29
29	00:08:31
30	00:08:31
31	00:09:29

[Curt3g]

Good idea. Thanks!

[ajm]

Just a rant after a discussion on https://github.com/FoldingAtHome/fah-issues/issues/1527

There should be a major decision by FAH to orient their efforts. There is a crossroad here: either push towards standard users or towards computer savvy people/IT enterprises.
In order to gain et retain the former, FAH needs to develop such a matching system (between the servers/WUs and the donors' hardware) that would be transparent, efficient and would function largely unattended.
In order to attract the latter, FH would need a large-scale deployment solution that would streamline the installation, fine tuning, monitoring and controlling of hundreds of systems at once. It would be a really big project, or two, rather, and FAH, notoriously, doesn't have the staff.

Maybe talk the IT pros who joined FAH recently into helping programming such nice systems, for standard users AND IT pros? If there is a strong commitment on the side of FAH going forwards, it might take off.