Anatomy of a series of GPU Work Units from the trenches

[GreyWhiskers]

I had been wondering about how my contributions are actually supporting the science at FAH when I wrote the post below last week. I'm a retired "quant" who likes to understand the numbers and relationships of the things I work with.

Re: publication of new paper on FAH (bigadv) results

... Are big molecules broken into many smaller segments that are issued as WUs? What are the "runs" "clones" "generations" exactly? How does one set of runs set up a subsequent set?...

This is an explanation of PRCGs that is easy to understand (http://fahwiki.net/index.php/Runs%2C_Clones_and_Gens).

I went live around 1 April 2011 with my Digital Storm Sandy Bridge system (see sig) including the GTX560Ti GPU card. I have processed 370+ Project 6801 Core 15 GPU WUs, at a rock steady 1:21 TPF, interspersed with four 1xxxx WUs that must have been part of the advmethods projects.

I hadn't really looked at the series of WUs very closely until this morning. I pulled out of the HFM Work History viewer .db3 database with Datadmin 3.4.8 personal and Excel exported as a .prn text file most of my work history since 1 April. Parenthetically, I haven't put this Sandy Bridge rig into FAH v7 because I don't see any way to get the equivalent level of history and understanding yet.

Reading Ensign's terrific paper on "Runs, Clones and Generations" in the FAH Wiki (see PantherX's link in the quote above), I see that the Gens are timesteps along the trajectory of particular clones for a particular run.

The interesting thing about this sequence is that I've consistently since 1 April been getting successive Gens - starting with 0 progressing to 12 in the last day. This shows that for the runs they are working on, all the WUs start at the beginning timestep - Gen 0.

EDIT: There are a couple of what may be "catch up" out of sequence WUs on May 5, 6, but the rest of the series seems consistent.

For Gen0, 1-4 April, I got a set of WUs for Clone 0, 1, 2, 3, 4 successively for various Runs.
Gen 1, 3-6 April
Gen 2, 7-9 April,
up to today
Gen 12, 11-13 May (and that's what I'm folding at the moment). It's on Clone 3, Edit: next two WUs after table below were Clone 4s.

I can just visualize in my mind's eye the trajectory tree growing time step by time step, gen by gen.

This makes clearer this quote from Ensign's paper

Okay, here it is: The CLONE numbers are labels for each trajectory that we run. Each GENeration is another chunk of time along that trajectory. So, say that I benchmark CLONE0, GEN0 (the first 4 ns). That WU is then done, and the FAH software builds a new WU with starting coordinates (and velocities and stuff) where mine left off. Then the new WU -- GEN1 of CLONE0 -- gets sent to you, and you simulate the next 4 ns. And so on. So CLONE is a label for an individual trajectory, and GENerations are time steps along that trajectory.

Code: Select all

Project    Run    Clone     Gen  Download Date
    6801    6848       1      0 4/1/2011 14:23
    6801    7683       1      0 4/1/2011 16:46
    6801    5112       2      0 4/2/2011 0:03
    6801    5911       2      0 4/2/2011 2:27
    6801    6709       2      0 4/2/2011 4:50
    6801    7476       2      0 4/2/2011 7:13
    6801    8278       2      0 4/2/2011 9:36
    6801    9082       2      0 4/2/2011 11:59
    6801    9853       2      0 4/2/2011 14:23
    6801    5657       3      0 4/2/2011 16:46
    6801    6464       3      0 4/2/2011 19:09
    6801    7254       3      0 4/2/2011 21:32
    6801    8012       3      0 4/2/2011 23:56
    6801    8776       3      0 4/3/2011 2:19
    6801    9521       3      0 4/3/2011 4:43
    6801    5234       4      0 4/3/2011 7:06
    6801    6026       4      0 4/3/2011 9:29
    6801    6821       4      0 4/3/2011 11:52
    6801    7601       4      0 4/3/2011 14:16
    6801    8426       4      0 4/3/2011 16:39
    6801    9214       4      0 4/3/2011 19:02
    6801    5000       0      1 4/3/2011 21:25
    6801    5786       0      1 4/3/2011 23:48
    6801    6589       0      1 4/4/2011 2:12
    6801    7365       0      1 4/4/2011 4:35
    6801    8155       0      1 4/4/2011 6:58
    6801    8944       0      1 4/4/2011 9:21
    6801    9734       0      1 4/4/2011 11:44
    6801    5534       1      1 4/4/2011 14:08
    6801    6340       1      1 4/4/2011 16:31
    6801    7135       1      1 4/4/2011 18:54
    6801    7920       1      1 4/4/2011 21:17
    6801    9465       1      1 4/5/2011 2:04
    6801    5245       2      1 4/5/2011 4:27
    6801    6037       2      1 4/5/2011 6:50
    6801    6829       2      1 4/5/2011 9:14
    6801    7646       2      1 4/5/2011 11:37
    6801    8441       2      1 4/5/2011 14:00
    6801    9232       2      1 4/5/2011 16:23
    6801    5042       3      1 4/5/2011 18:46
    6801    9865       2      1 4/5/2011 21:09
    6801    6584       3      1 4/5/2011 23:33
    6801    7356       3      1 4/6/2011 1:56
    6801    8104       3      1 4/6/2011 4:19
    6801    8892       3      1 4/6/2011 6:42
    6801    9707       3      1 4/6/2011 9:06
    6801    5368       4      1 4/6/2011 11:29
    6801    6175       4      1 4/6/2011 13:52
    6801    6960       4      1 4/6/2011 16:15
    6801    7776       4      1 4/6/2011 18:38
    6801    8576       4      1 4/6/2011 21:02
    6801    9345       4      1 4/6/2011 23:25
    6801    5100       0      2 4/7/2011 1:48
    6801    5864       0      2 4/7/2011 4:11
    6801    6601       0      2 4/7/2011 6:34
    6801    7393       0      2 4/7/2011 8:58
    6801    8182       0      2 4/7/2011 11:21
    6801    8979       0      2 4/7/2011 13:44
    6801    9774       0      2 4/7/2011 16:08
    6801    5560       1      2 4/7/2011 18:31
    6801    6341       1      2 4/7/2011 20:54
    6801    7082       1      2 4/7/2011 23:17
    6801    7860       1      2 4/8/2011 1:40
    6801    8565       1      2 4/8/2011 3:59
    6801    9291       1      2 4/8/2011 6:16
    6801    5043       2      2 4/8/2011 8:33
    6801    5767       2      2 4/8/2011 10:49
    6801    6501       2      2 4/8/2011 13:06
    6801    7254       2      2 4/8/2011 15:22
    6801    7996       2      2 4/8/2011 17:39
    6801    8767       2      2 4/8/2011 19:56
    6801    9521       2      2 4/8/2011 22:13
    6801    5211       3      2 4/9/2011 0:29
    6801    5942       3      2 4/9/2011 2:46
    6801    6637       3      2 4/9/2011 5:02
    6801    7350       3      2 4/9/2011 7:19
    6801    8067       3      2 4/9/2011 9:36
    6801    8816       3      2 4/9/2011 11:56
    6801    9565       3      2 4/9/2011 14:13
    6801    5146       4      2 4/9/2011 16:30
    6801    5866       4      2 4/9/2011 18:46
    6801    6601       4      2 4/9/2011 21:03
    6801    7345       4      2 4/9/2011 23:20
    6801    5003       0      3 4/10/2011 7:49
    6801    5777       0      3 4/10/2011 10:06
    6801    6531       0      3 4/10/2011 12:23
    6801    7267       0      3 4/10/2011 14:40
    6801    8006       0      3 4/10/2011 16:56
    6801    8745       0      3 4/10/2011 19:13
    6801    9456       0      3 4/10/2011 21:29
    6801    5136       1      3 4/10/2011 23:46
    6801    5808       1      3 4/11/2011 2:03
    6801    6486       1      3 4/11/2011 4:20
    6801    7138       1      3 4/11/2011 6:36
    6801    7839       1      3 4/11/2011 8:53
    6801    8533       1      3 4/11/2011 11:10
    6801    9258       1      3 4/11/2011 13:26
    6801    9985       1      3 4/11/2011 15:43
    6801    5724       2      3 4/11/2011 18:00
    6801    6452       2      3 4/11/2011 20:17
    6801    7141       2      3 4/11/2011 22:34
    6801    7854       2      3 4/12/2011 0:50
    6801    8537       2      3 4/12/2011 3:07
    6801    9208       2      3 4/12/2011 5:23
    6801    9877       2      3 4/12/2011 7:40
    6801    5579       3      3 4/12/2011 9:56
    6801    6323       3      3 4/12/2011 12:13
    6801    7045       3      3 4/12/2011 14:29
    6801    7772       3      3 4/12/2011 16:46
    6801    8481       3      3 4/12/2011 19:02
    6801    9201       3      3 4/12/2011 21:19
    6801    9931       3      3 4/12/2011 23:36
    6801    5364       4      3 4/13/2011 1:52
    6801    6044       4      3 4/13/2011 4:08
    6801    7094       4      3 4/13/2011 7:35
    6801    7838       4      3 4/13/2011 9:51
    6801    8580       4      3 4/13/2011 12:08
    6801    9331       4      3 4/13/2011 14:24
    6801    5059       0      4 4/13/2011 16:41
    6801    5774       0      4 4/13/2011 18:57
    6801    6507       0      4 4/13/2011 21:14
    6801    7218       0      4 4/13/2011 23:31
    6801    7218       0      4 4/14/2011 0:47
    6801    8306       0      4 4/14/2011 3:04
    6801    9016       0      4 4/14/2011 5:21
    6801    9811       0      4 4/14/2011 7:55
    6801    5496       1      4 4/14/2011 10:12
    6801    6211       1      4 4/14/2011 12:28
    6801    6924       1      4 4/14/2011 14:45
    6801    7645       1      4 4/14/2011 17:01
    6801    9926       1      4 4/15/2011 0:15
    6801    5616       2      4 4/15/2011 2:33
    6801    6322       2      4 4/15/2011 4:51
    6801    7022       2      4 4/15/2011 7:09
    6801    7726       2      4 4/15/2011 9:26
    6801    7412       2      5 4/15/2011 11:44
    6801    9127       2      4 4/15/2011 14:02
    6801    9731       2      4 4/15/2011 16:20
    6801    5383       3      4 4/15/2011 18:37
    6801    6081       3      4 4/15/2011 20:57
    6801    6757       3      4 4/15/2011 23:13
    6801    7424       3      4 4/16/2011 1:29
    6801    8073       3      4 4/16/2011 3:44
    6801    8719       3      4 4/16/2011 6:00
    6801    9389       3      4 4/16/2011 8:16
    6801    2262       4      4 4/16/2011 10:32
    6801    5411       4      4 4/16/2011 12:48
    6801    6117       4      4 4/16/2011 15:04
    6801    6842       4      4 4/16/2011 17:20
    6801    7534       4      4 4/16/2011 19:36
    6801    8221       4      4 4/16/2011 21:52
    6801    8881       4      4 4/17/2011 0:08
    6801    9527       4      4 4/17/2011 2:24
    6801    5182       0      5 4/17/2011 4:40
    6801    5837       0      5 4/17/2011 6:55
    6801    6520       0      5 4/17/2011 9:11
    6801    7226       0      5 4/17/2011 11:28
    6801    7950       0      5 4/17/2011 13:43
    6801    8652       0      5 4/17/2011 16:00
    6801    9356       0      5 4/17/2011 18:15
    6801    5041       1      5 4/17/2011 20:31
    6801    5706       1      5 4/17/2011 22:47
    6801    6372       1      5 4/18/2011 1:03
    6801    7041       1      5 4/18/2011 3:19
    6801    7685       1      5 4/18/2011 5:35
    6801    8328       1      5 4/18/2011 7:51
    6801    8996       1      5 4/18/2011 10:08
    6801    9711       1      5 4/18/2011 12:26
    6801    5416       2      5 4/18/2011 14:44
    6801    6118       2      5 4/18/2011 17:02
    6801    6825       2      5 4/18/2011 19:20
    6801    7539       2      5 4/18/2011 21:38
    6801    6646       3      5 4/19/2011 11:20
    6801    7365       3      5 4/19/2011 13:44
    6801    8075       3      5 4/19/2011 16:01
    6801    8766       3      5 4/19/2011 18:18
    6801    9458       3      5 4/19/2011 20:34
    6801    2855       4      5 4/19/2011 22:51
    6801    5392       4      5 4/20/2011 1:07
    6801    6048       4      5 4/20/2011 3:24
    6801    6707       4      5 4/20/2011 5:41
    6801    7381       4      5 4/20/2011 7:58
    6801    9596       4      5 4/20/2011 15:22
    6801    5245       0      6 4/20/2011 17:42
    6801    5933       0      6 4/20/2011 19:59
    6801    6625       0      6 4/20/2011 22:16
    6801    6395       0      7 4/21/2011 0:32
    6801    7917       0      6 4/21/2011 2:49
    6801    8550       0      6 4/21/2011 5:05
    6801    9202       0      6 4/21/2011 7:22
    6801    9853       0      6 4/21/2011 9:38
    6801    5529       1      6 4/21/2011 11:55
    6801    6221       1      6 4/21/2011 14:11
    6801    6922       1      6 4/21/2011 16:28
    6801    7585       1      6 4/21/2011 18:44
    6801    8248       1      6 4/21/2011 21:01
    6801    8946       1      6 4/21/2011 23:17
    6801    9600       1      6 4/22/2011 1:34
    6801    5237       2      6 4/22/2011 3:50
    6801    5911       2      6 4/22/2011 6:07
    6801    6558       2      6 4/22/2011 8:23
    6801    7250       2      6 4/22/2011 10:40
    6801    7932       2      6 4/22/2011 12:57
    6801    8614       2      6 4/22/2011 15:14
    6801    9320       2      6 4/22/2011 17:30
    6801    9982       2      6 4/22/2011 19:47
    6801    5600       3      6 4/22/2011 22:04
    6801    6238       3      6 4/23/2011 0:20
    6801    6898       3      6 4/23/2011 2:37
    6801    7535       3      6 4/23/2011 4:53
    6801    8164       3      6 4/23/2011 7:10
    6801    8838       3      6 4/23/2011 9:28
    6801    9491       3      6 4/23/2011 11:45
    6801    2699       4      6 4/23/2011 14:01
    6801    5345       4      6 4/23/2011 16:18
    6801    5992       4      6 4/23/2011 18:34
    6801    6622       4      6 4/23/2011 20:51
    6801    7264       4      6 4/23/2011 23:08
    6801    7867       4      6 4/24/2011 1:24
    6801    8502       4      6 4/24/2011 3:41
    6801    9145       4      6 4/24/2011 5:58
    6801    9829       4      6 4/24/2011 8:14
    6801    5502       0      7 4/24/2011 10:31
    6801    6206       0      7 4/24/2011 12:48
    6801    6876       0      7 4/24/2011 15:05
    6801    7553       0      7 4/24/2011 17:21
    6801    8221       0      7 4/24/2011 19:38
    6801    8882       0      7 4/24/2011 21:56
    6801    9536       0      7 4/25/2011 0:13
    6801    5142       1      7 4/25/2011 2:29
    6801    5793       1      7 4/25/2011 4:46
    6801    6438       1      7 4/25/2011 7:03
    6801    7086       1      7 4/25/2011 9:19
    6801    7747       1      7 4/25/2011 11:36
    6801    8357       1      7 4/25/2011 13:53
    6801    9008       1      7 4/25/2011 16:09
    6801    9638       1      7 4/25/2011 18:26
    6801    5297       2      7 4/25/2011 20:43
    6801    5909       2      7 4/25/2011 23:00
    6801    6530       2      7 4/26/2011 1:16
    6801    7173       2      7 4/26/2011 3:33
    6801    7845       2      7 4/26/2011 6:00
    6801    8508       2      7 4/26/2011 8:17
    6801    9155       2      7 4/26/2011 10:33
    6801    7281       2      7 4/26/2011 12:50
    6801    5427       3      7 4/26/2011 15:07
    6801    5032       4      7 4/27/2011 9:19
    6801    5032       4      8 5/1/2011 7:23
    6801    6995       4      8 5/1/2011 9:40
    6801    7784       4      8 5/1/2011 11:56
    6801    8403       4      8 5/1/2011 14:13
    6801    9204       4      8 5/1/2011 16:30
    6801    5030       0      9 5/1/2011 18:46
    6801    5779       0      9 5/1/2011 21:03
    6801    6510       0      9 5/1/2011 23:19
    6801    7089       0      9 5/2/2011 1:36
    6801    7845       0      9 5/2/2011 3:53
    6801    8531       0      9 5/2/2011 6:09
    6801    9021       0      9 5/2/2011 8:30
    6801    9804       0      9 5/2/2011 10:46
    6801    5602       1      9 5/2/2011 13:03
    6801    6410       1      9 5/2/2011 15:19
    6801    7185       1      9 5/2/2011 17:36
    6801    7925       1      9 5/2/2011 19:53
    6801    8680       1      9 5/2/2011 22:09
    6801    9456       1      9 5/3/2011 0:26
    6801    5079       2      9 5/3/2011 2:42
    6801    5734       2      9 5/3/2011 4:59
    6801    6560       2      9 5/3/2011 7:15
    6801    7382       2      9 5/3/2011 9:32
    6801    8463       2      9 5/3/2011 12:59
    6801    9245       2      9 5/3/2011 15:15
    6801    5001       3      9 5/3/2011 17:32
    6801    5791       3      9 5/3/2011 19:48
    6801    6535       3      9 5/3/2011 22:05
    6801    7281       3      9 5/4/2011 0:21
    6801    7886       3      9 5/4/2011 2:38
    6801    8651       3      9 5/4/2011 4:55
    6801    9416       3      9 5/4/2011 7:11
    6801    1741       4      9 5/4/2011 9:28
    6801    5043       4      9 5/4/2011 11:44
    6801    5832       4      9 5/4/2011 14:01
    6801    6621       4      9 5/4/2011 16:18
    6801    7403       4      9 5/4/2011 18:34
    6801    8217       4      9 5/4/2011 20:51
    6801    8933       4      9 5/4/2011 23:08
    6801    9648       4      9 5/5/2011 1:25
    6801    5419       0     10 5/5/2011 3:41
    6801    6166       0     10 5/5/2011 5:58
    6801    6969       0     10 5/5/2011 8:14
    6801    7800       0     10 5/5/2011 10:31
    6801    8632       0     10 5/5/2011 12:47
    6801    9309       0     10 5/5/2011 15:04
    6801    5136       1     10 5/5/2011 17:21
    6801    8387       3      8 5/5/2011 19:37
    6801    8387       3      9 5/5/2011 21:54
    6801    7618       1     10 5/6/2011 0:10
    6801    8444       1     10 5/6/2011 2:27
    6801    9306       1     10 5/6/2011 4:43
    6801    5160       2     10 5/6/2011 7:00
    6801    6396       2     10 5/6/2011 10:39
    6801    7168       2     10 5/6/2011 12:55
    6801    8106       2     10 5/6/2011 15:12
    6801    8898       2     10 5/6/2011 17:29
    6801    9696       2     10 5/6/2011 19:45
    6801    6023       0      9 5/6/2011 22:02
    6801    6343       3     10 5/7/2011 0:18
    6801    7036       3     10 5/7/2011 2:35
    6801    7903       3     10 5/7/2011 4:51
    6801    8664       3     10 5/7/2011 7:08
    6801    9576       3     10 5/7/2011 9:25
    6801    3669       4     10 5/7/2011 11:41
    6801    5382       4     10 5/7/2011 13:58
    6801    6240       4     10 5/7/2011 16:15
    6801    7093       4     10 5/7/2011 18:31
    6801    7961       4     10 5/7/2011 20:48
    6801    8783       4     10 5/7/2011 23:04
    6801    9642       4     10 5/8/2011 1:21
    6801    5445       0     11 5/8/2011 3:38
    6801    6228       0     11 5/8/2011 5:54
    6801    7038       0     11 5/8/2011 8:11
    6801    7654       2      9 5/8/2011 10:27
    6801    7654       2     10 5/8/2011 12:44
    6801    9534       0     11 5/8/2011 15:01
    6801    5375       1     11 5/8/2011 17:17
    6801    6201       1     11 5/8/2011 19:34
    6801    7011       1     11 5/8/2011 21:51
    6801    7847       1     11 5/9/2011 0:07
    6801    8924       1     11 5/9/2011 3:35
    6801    9742       1     11 5/9/2011 5:52
    6801    5606       2     11 5/9/2011 8:09
    6801    6421       2     11 5/9/2011 10:26
    6801    8306       2     11 5/9/2011 16:17
    6801    9108       2     11 5/9/2011 18:34
    6801    9964       2     11 5/9/2011 20:51
    6801    5693       3     11 5/9/2011 23:07
    6801    6487       3     11 5/10/2011 1:24
    6801    7263       3     11 5/10/2011 3:40
    6801    8030       3     11 5/10/2011 5:57
    6801    8793       3     11 5/10/2011 8:14
    6801    5121       4     11 5/10/2011 15:03
    6801    5953       4     11 5/10/2011 17:20
    6801    6755       4     11 5/10/2011 19:37
    6801    7522       4     11 5/10/2011 21:53
    6801    8259       4     11 5/11/2011 0:10
    6801    8987       4     11 5/11/2011 2:27
    6801    9743       4     11 5/11/2011 4:43
    6801    5534       0     12 5/11/2011 7:00
    6801    6283       0     12 5/11/2011 9:16
    6801    7049       0     12 5/11/2011 11:33
    6801    7798       0     12 5/11/2011 13:50
    6801    8578       0     12 5/11/2011 16:06
    6801    9360       0     12 5/11/2011 18:23
    6801    5144       1     12 5/11/2011 20:40
    6801    5905       1     12 5/11/2011 22:56
    6801    6647       1     12 5/12/2011 1:13
    6801    7412       1     12 5/12/2011 3:29
    6801    8149       1     12 5/12/2011 5:46
    6801    8904       1     12 5/12/2011 8:03
    6801    9723       1     12 5/12/2011 10:28
    6801    5482       2     12 5/12/2011 12:45
    6801    6246       2     12 5/12/2011 15:01
    6801    7010       2     12 5/12/2011 17:18
    6801    7773       2     12 5/12/2011 19:35
    6801    8541       2     12 5/12/2011 21:51
    6801    9291       2     12 5/13/2011 0:08
    6801     333       3     12 5/13/2011 2:25
    6801    5655       3     12 5/13/2011 4:41
    6801    6410       3     12 5/13/2011 6:58
    6801    7176       3     12 5/13/2011 9:15
    6801    7958       3     12 5/13/2011 11:31
    6801    8746       3     12 5/13/2011 13:48
    6801    9491       3     12 5/13/2011 16:04

[7im]

This is one of the best depictions of the serial and time sensitive nature of the project, and why the quick return of work units increases the value of the work units. If one of those work units had been held up, as was likely around May 5-6th, it's easy to see the affect it has on the project.

It also shows how meeting short deadlines is important, and why caching of work units for offline processing is not very helpful.

[codysluder]

If this pattern is followed until the project is finished, I think we can conclude that there will be some number of Run,Clone trajectories which reach some Gen number that is defined scientifically as "enough." Any trajectory that is delayed will not be "enough" so there may be a lot of wasted work on those trajectories.

Or, perhaps the project will be tapered off, suspending some trajectories and continuing the slower ones. At that point, the conclusions will have to wait until the slowest of trajectories reach "enough." In other words, if you slow down one of the slower trajectories, you could be slowing down the entire process, no matter how many people complete their work on time.

[GreyWhiskers]

As I've been thinking of this throughout the day, I was wondering if it is asking too much of the FAH project team for each lead to describe in the Stanford Project description page just what the WU strategy is for that project.

Those who define the projects obviously know what their strategy is when they lay it out. I think documenting the strategy makes the science more transparent to the donors, maybe helping the large cadre of donors to realize there is more about FAH than PPD and bonuses.

I would think there are probably several strategy templates that the project leads choose among - and that a lot of boilerplate could be created to make the Project Description page more relevant. There are many of the project pages that represent a huge number of the WUs that I receive that are essentially "content free" not telling me any more than the points and deadlines and core type.