Suggestion for F@H clients

[shankus]

Perhaps there is a good reason this isn't already in place, but let me have a go at this.

I just returned a WU that was well over 30 MB and took over 7 minutes to upload. Only after the upload was complete did the server send me another. Now, even if the WU I returned is junk for some reason, what is the sense in determining that before I get my next?

Now my main machine will do 1% of many WUs in ~3 minutes. If a WU is completed, and the server sends another, THEN uploads the result of the previous, I could have completed 2% in the interim (on one of several per day). While at first glance this may seem trivial, multiply it by the number of contributors and it becomes significant.

Now, this is about like making a Drake equation, but at the time of this post, there are about 350,000 "Active CPUs". Let's divide that by two and guess about 175,000 donors. If we divide 175,000 by 100 (%), we get 1,750 (%). This is the potential loss of percentage of WUs just due to upload BEFORE download. (assuming 1 WU/day/donor)

I don't know if it's reasonable to extrapolate that to 17.5 WUs, but, let's "Drake" on.

If there are 175,000 donors that can only complete one WU/day, that equates to 6,387.5 WUs per year lost. This equals several extra "ghost" donors.

Is my math and reasoning valid? Did I miss anything?

[GreyWhiskers]

My suggestion would be to run the V7 Beta client. One of the big improvements in v7 over the previous Uniprocessor, SMP and GPU clients, besides being able to set up and run any of the client types on your machine from a single GUI, or even run remote clients from the same single GUI, is the fact that it overlaps the download of the new work unit with completion of the previous, and performs upload of the just-completed WU while the new one is starting up.

The default is for the v7 client to start downloading the next WU when the previous reaches 99%. As soon as the previous WU is finished and its data is packaged up for upload, the new work unit starts, in parallel with uploading the old results.

I have set my "next-unit-percentage" parameter in v7 to 100%, because download of the new work unit with my internet connection is always so fast. It rarely takes more than a few seconds to download a new work unit, which can be done between reaching 100% and the wrapup and packaging up the data to complete the processing.

My currently running SMP work unit has a TPF of 7.5 minutes, so I don't need to download the next work unit THAT early.

I've got two examples below from my laptop running one SMP slot under v7.

The first shows completion of a very large work unit.
- It reached 100% at 21:19:40, when it downloaded the next WU, and started wrapping the results of the old one.
- at 21:20:12, the wrapup was completed "FINISHED_UNIT ", and it immediately started processing the new WU
- the same second, it started uploading the finished data.
- It finished the long upload (28.83 MiB) at 21:26:50, while the new WU was finished its 7th frame and well into the 8th.

I think that this scenario was exactly what you were commenting on.

Code: Select all

21:12:00:Unit 01:Completed 2475000 out of 2500000 steps  (99%)
21:19:40:Unit 01:Completed 2500000 out of 2500000 steps  (100%)
21:19:40:Unit 01:DynamicWrapper: Finished Work Unit: sleep=10000
21:19:40:Connecting to assign3.stanford.edu:8080
21:19:40:News: Welcome to Folding@Home
21:19:40:Assigned to work server 171.67.108.58
21:19:40:Requesting new work unit for slot 00: RUNNING smp:8 from 171.67.108.58
21:19:40:Connecting to 171.67.108.58:8080
21:19:40:Slot 00: Downloading 532.30KiB
21:19:41:Slot 00: Download complete
21:19:41:Received Unit: id:00 state:DOWNLOAD error:OK project:8001 run:69 clone:45 gen:3 core:0xa4 unit:0x000000046652edca4ededd7643f089dc
21:19:50:Unit 01:
21:19:50:Unit 01:Finished Work Unit:
21:19:50:Unit 01:- Reading up to 35910936 from "01/wudata_01.trr": Read 35910936
21:19:50:Unit 01:trr file hash check passed.
21:19:50:Unit 01:edr file hash check passed.
21:19:50:Unit 01:logfile size: 59661
21:19:50:Unit 01:Leaving Run
21:19:54:Unit 01:- Writing 36000513 bytes of core data to disk...
21:20:01:Unit 01:Done: 36000001 -> 30226485 (compressed to 83.9 percent)
21:20:02:Unit 01:  ... Done.
21:20:11:Unit 01:- Shutting down core
21:20:11:Unit 01:
21:20:11:Unit 01:Folding@home Core Shutdown: FINISHED_UNIT
21:20:12:FahCore, running Unit 01, returned: FINISHED_UNIT (100 = 0x64)
21:20:12:Sending unit results: id:01 state:SEND error:OK project:7903 run:17 clone:4 gen:12 core:0xa4 unit:0x0000000d00ac9c214ebaaa7087f971cd
21:20:12:Unit 01: Uploading 28.83MiB to 128.113.12.161
21:20:12:Connecting to 128.113.12.161:8080
21:20:12:Starting Unit 00
21:20:12:Running core: "C:/Users/USER/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe" -dir 00 -suffix 01 -lifeline 5868 -version 701 -checkpoint 15 -np 8
21:20:12:Started core on PID 8048
21:20:12:FahCore 0xa4 started
21:20:12:Unit 00:
21:20:12:Unit 00:*------------------------------*
21:20:12:Unit 00:Folding@Home Gromacs GB Core
21:20:12:Unit 00:Version 2.27 (Dec. 15, 2010)
21:20:12:Unit 00:
21:20:12:Unit 00:Preparing to commence simulation
21:20:12:Unit 00:- Looking at optimizations...
21:20:12:Unit 00:- Created dyn
21:20:12:Unit 00:- Files status OK
21:20:12:Unit 00:- Expanded 544567 -> 1305312 (decompressed 239.6 percent)
21:20:12:Unit 00:Called DecompressByteArray: compressed_data_size=544567 data_size=1305312, decompressed_data_size=1305312 diff=0
21:20:12:Unit 00:- Digital signature verified
21:20:12:Unit 00:
21:20:12:Unit 00:Project: 8001 (Run 69, Clone 45, Gen 3)
21:20:12:Unit 00:
21:20:12:Unit 00:Assembly optimizations on if available.
21:20:12:Unit 00:Entering M.D.
21:20:18:Unit 01: 1.46%
21:20:18:Unit 00:Mapping NT from 8 to 8 
21:20:18:Unit 00:Completed 0 out of 250000 steps  (0%)
21:20:24:Unit 01: 2.99%
21:20:30:Unit 01: 4.55%
21:20:36:Unit 01: 6.06%
21:20:42:Unit 01: 7.62%
21:20:48:Unit 01: 9.17%
21:20:54:Unit 01: 10.72%
21:21:00:Unit 01: 12.24%
21:21:06:Unit 01: 13.75%
21:21:09:Unit 00:Completed 2500 out of 250000 steps  (1%)
21:21:12:Unit 01: 15.31%
21:21:18:Unit 01: 16.86%
21:21:24:Unit 01: 18.43%
21:21:30:Unit 01: 19.99%
21:21:36:Unit 01: 21.56%
21:21:42:Unit 01: 23.10%
21:21:48:Unit 01: 24.66%
21:21:54:Unit 01: 26.21%
21:22:00:Unit 01: 27.74%
21:22:01:Unit 00:Completed 5000 out of 250000 steps  (2%)
21:22:06:Unit 01: 29.28%
21:22:12:Unit 01: 30.84%
21:22:18:Unit 01: 32.39%
21:22:24:Unit 01: 33.96%
21:22:30:Unit 01: 35.53%
21:22:36:Unit 01: 37.06%
21:22:42:Unit 01: 38.58%
21:22:48:Unit 01: 40.10%
21:22:54:Unit 01: 41.61%
21:22:55:Unit 00:Completed 7500 out of 250000 steps  (3%)
21:23:00:Unit 01: 43.15%
21:23:06:Unit 01: 44.68%
21:23:12:Unit 01: 46.21%
21:23:18:Unit 01: 47.73%
21:23:24:Unit 01: 49.23%
21:23:30:Unit 01: 50.75%
21:23:36:Unit 01: 52.28%
21:23:42:Unit 01: 53.77%
21:23:48:Unit 01: 55.30%
21:23:48:Unit 00:Completed 10000 out of 250000 steps  (4%)
21:23:54:Unit 01: 56.79%
21:24:00:Unit 01: 58.30%
21:24:06:Unit 01: 59.81%
21:24:12:Unit 01: 61.32%
21:24:18:Unit 01: 62.73%
21:24:24:Unit 01: 64.23%
21:24:30:Unit 01: 65.73%
21:24:36:Unit 01: 67.25%
21:24:41:Unit 00:Completed 12500 out of 250000 steps  (5%)
21:24:42:Unit 01: 68.77%
21:24:48:Unit 01: 70.29%
21:24:54:Unit 01: 71.79%
21:25:00:Unit 01: 73.28%
21:25:06:Unit 01: 74.80%
21:25:12:Unit 01: 76.32%
21:25:18:Unit 01: 77.84%
21:25:24:Unit 01: 79.35%
21:25:30:Unit 01: 80.88%
21:25:35:Unit 00:Completed 15000 out of 250000 steps  (6%)
21:25:36:Unit 01: 82.39%
21:25:42:Unit 01: 83.88%
21:25:48:Unit 01: 85.41%
21:25:54:Unit 01: 86.93%
21:26:00:Unit 01: 88.41%
21:26:06:Unit 01: 89.94%
21:26:12:Unit 01: 91.47%
21:26:18:Unit 01: 92.97%
21:26:24:Unit 01: 94.49%
21:26:27:Unit 00:Completed 17500 out of 250000 steps  (7%)
21:26:30:Unit 01: 95.98%
21:26:36:Unit 01: 97.50%
21:26:42:Unit 01: 99.00%
21:26:50:Unit 01: Upload complete
21:26:50:Server responded WORK_ACK (400)
21:26:50:Final credit estimate, 4262.00 points
21:26:50:Cleaning up Unit 01
21:27:20:Unit 00:Completed 20000 out of 250000 steps  (8%)
21:28:13:Unit 00:Completed 22500 out of 250000 steps  (9%

This example is a more typical one - where the upload is very short.

Same process - at 100%, connection to the Stanford server to get the next WU, while wrapping up the old.
- At 00:27:56:the old unit was FINISHED_UNIT, and the new WU was started at the same time upload of the old was started.
- The upload was complete at 00:28:01, well before the new WU reached 0%. It only had to upload 1.14MiB, so that was done very, very fast.

With v7, in either case, the next Work Unit starts immediately upon the completion of the previous, with both the upload and download processes parallel with the actual work unit processing.

Code: Select all

00:26:50:Unit 01:Completed 247500 out of 250000 steps  (99%)
00:27:41:Unit 01:Completed 250000 out of 250000 steps  (100%)
00:27:42:Unit 01:DynamicWrapper: Finished Work Unit: sleep=10000
00:27:42:Connecting to assign3.stanford.edu:8080
00:27:42:News: Welcome to Folding@Home
00:27:42:Assigned to work server 171.64.65.101
00:27:42:Requesting new work unit for slot 00: RUNNING smp:8 from 171.64.65.101
00:27:42:Connecting to 171.64.65.101:8080
00:27:42:Slot 00: Downloading 296.08KiB
00:27:43:Slot 00: Download complete
00:27:43:Received Unit: id:00 state:DOWNLOAD error:OK project:7600 run:0 clone:21 gen:24 core:0xa4 unit:0x00000056664f2dcd4dee895fea0e6de6
00:27:52:Unit 01:
00:27:52:Unit 01:Finished Work Unit:
00:27:52:Unit 01:- Reading up to 770532 from "01/wudata_01.trr": Read 770532
00:27:52:Unit 01:trr file hash check passed.
00:27:52:Unit 01:- Reading up to 456592 from "01/wudata_01.xtc": Read 456592
00:27:52:Unit 01:xtc file hash check passed.
00:27:52:Unit 01:edr file hash check passed.
00:27:52:Unit 01:logfile size: 22832
00:27:52:Unit 01:Leaving Run
00:27:54:Unit 01:- Writing 1255360 bytes of core data to disk...
00:27:55:Unit 01:Done: 1254848 -> 1194941 (compressed to 95.2 percent)
00:27:55:Unit 01:  ... Done.
00:27:56:Unit 01:- Shutting down core
00:27:56:Unit 01:
00:27:56:Unit 01:Folding@home Core Shutdown: FINISHED_UNIT
00:27:56:FahCore, running Unit 01, returned: FINISHED_UNIT (100 = 0x64)
00:27:56:Sending unit results: id:01 state:SEND error:OK project:8001 run:49 clone:63 gen:3 core:0xa4 unit:0x000000046652edca4ededba8055142e7
00:27:56:Unit 01: Uploading 1.14MiB to 171.67.108.58
00:27:56:Connecting to 171.67.108.58:8080
00:27:56:Starting Unit 00
00:27:56:Running core: "C:/Users/USER/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe" -dir 00 -suffix 01 -lifeline 5868 -version 701 -checkpoint 15 -np 8
00:27:56:Started core on PID 8352
00:27:56:FahCore 0xa4 started
00:27:57:Unit 00:
00:27:57:Unit 00:*------------------------------*
00:27:57:Unit 00:Folding@Home Gromacs GB Core
00:27:57:Unit 00:Version 2.27 (Dec. 15, 2010)
00:27:57:Unit 00:
00:27:57:Unit 00:Preparing to commence simulation
00:27:57:Unit 00:- Looking at optimizations...
00:27:57:Unit 00:- Created dyn
00:27:57:Unit 00:- Files status OK
00:27:57:Unit 00:- Expanded 302673 -> 506024 (decompressed 167.1 percent)
00:27:57:Unit 00:Called DecompressByteArray: compressed_data_size=302673 data_size=506024, decompressed_data_size=506024 diff=0
00:27:57:Unit 00:- Digital signature verified
00:27:57:Unit 00:
00:27:57:Unit 00:Project: 7600 (Run 0, Clone 21, Gen 24)
00:27:57:Unit 00:
00:27:57:Unit 00:Assembly optimizations on if available.
00:27:57:Unit 00:Entering M.D.
00:28:01:Unit 01: Upload complete
00:28:01:Server responded WORK_ACK (400)
00:28:01:Final credit estimate, 1102.00 points
00:28:01:Cleaning up Unit 01
00:28:03:Unit 00:Mapping NT from 8 to 8 
00:28:03:Unit 00:Completed 0 out of 2000000 steps  (0%)
00:35:27:Unit 00:Completed 20000 out of 2000000 steps  (1%)
00:43:21:Unit 00:Completed 40000 out of 2000000 steps  (2%)

[rwh202]

I'm sure someone more experienced will jump in if I'm wrong, but as far as I know, this functionality is included in the v7 client.

By default, a new unit is downloaded at 99% completion (this seems wasteful of points though), but the value can be changed with the 'next-unit-percentage' flag to 100.

Also, 'langouste' is available for older clients (viewtopic.php?f=14&t=11615) that achieves the same thing. This was especially useful for bigadv units (100MB) that took 30 mins to upload.

(Edit: - beaten to it by GreyWhiskers)

[shankus]

*EDIT*
I completely missed what you said about the v7 client, which I've never heard of. Forgive me, I have trouble reading and focusing due to depression. I will look into this client and see how to use it. It looks like a front-end for a multi-core client, I think.
*END EDIT*

I'm using the SMP client, v 6.34. Here is what I show, minus all the extraneous info in the middle:

[05:05:34] Completed 250000 out of 250000 steps (100%)
[05:05:59] + Attempting to send results [December 18 05:05:59
[05:13:40] + Results successfully sent
[05:13:46] - Preparing to get new work unit...
[05:13:46] + Attempting to get work packet
[05:14:00] Completed 0 out of 500000 steps (0%)

That's about 8.5 minutes. This unit is taking me less that 3 minutes/%.

[08:45:23] Completed 330000 out of 500000 steps (66%)
[08:48:17] Completed 335000 out of 500000 steps (67%)

[GreyWhiskers]

@ shankus. Right, that's exactly what all the Stanford Folding at home clients prior to v7 did - serialized the" processing - upload - download - processing" steps.

V7, as rwh202 and I point out, performs the upload and download functions in parallel with the processing, allowing the processing to proceed nearly as efficiently as it can. There might still be some room for improvement, e.g., unwrapping the new WU as soon as it's received, so it can be started as soon as the old WU has stopped using the CPU for the folding, but the parallelism in v7 is great. Since most of our internet connections have a slower upload than download, moving the upload totally out of the serial timeline really helps for larger work units, as your example shows.

So, I'd suggest you give v7 a try and see if it doesn't devote more of your computer's wall clock time to folding.

GW.

[shankus]

I will. Thanks GreyWhiskers. I thought when I wrote the post that some well smarter than myself had brought up the problem already.

I need to get this running on both of my machines, but Alas, my PS3 stands alone. I wrote to Sony begging to be able to use more than one PS3 or upcoming PS4 under the same username, but no reply as of yet.

Of course, I could run more than one PS3 with extra results contributing to the project, but credit going to "Anonymous", but my hubris prevents it.

[PantherX]

...I wrote to Sony begging to be able to use more than one PS3 or upcoming PS4 under the same username, but no reply as of yet.

Of course, I could run more than one PS3 with extra results contributing to the project, but credit going to "Anonymous", but my hubris prevents it.

AFAIK, you can fold with multiple PS3 and they can contribute under the same unsername/team if they are configured identically.

[shankus]

*EDIT*
I did see a Playstation forum here. I bet those folks have all the answers.
*END EDIT*

Sony tech support told me I have to be signed in to PSN, and there can only be one username per PSN account. I assumed they meant I couldn't be signed in on two consoles simultaneously, as using more than one console was the topic of our conversation.

I don't know anyone else with one, and I'm not buying another to find out. I figured I'd build a "Folding Only" machine.

[shankus]

Um, the v7 client is not available in 64 bit for Win OS. Are you guys running this 32 bit software on 64 bit systems? If so, is it slower than just using the -smp client?

[k1wi]

The client is 32bit because there is no need for it to be 64bit (i.e. no need for it to ever use more than 4GB of memory). The reason for this is that all the actual processing is offloaded to the FAH cores.

[PantherX]

The client is 32bit because there is no need for it to be 64bit (i.e. no need for it to ever use more than 4GB of memory). The reason for this is that all the actual processing is offloaded to the FAH cores.

... and FAHCores can be made available in 32 or 64 bit versions and the V7 will download the appropriate version when required.

[codysluder]

The client is 32bit because there is no need for it to be 64bit (i.e. no need for it to ever use more than 4GB of memory). The reason for this is that all the actual processing is offloaded to the FAH cores.

Right. Almost everything that the client software does is download and upload WUs and start the appropriate FahCore. The speed of uploads and downloads is limited by your internet connection, not by anything the client software might do. Using a 64-bit FahCore is a good thing, though.

[Zagen30]

You don't have to be signed into the PSN to fold. I know there were times where I was randomly signed out of the PSN but was still able to send and receive WUs.

[codysluder]

I don't think the PS3 client uses your PSN name. It has been a while since I ran the PS3, but I know team number was set separately and I think FAH Name was on the same screen.

[Zagen30]

codysluder's correct- your PSN username is completely separate from your folding username.