13404 and 13405 low PPD on Nvidia 1080ti on Windows

[clapanse]

My GPU has grabbed a couple of these recently (3x 13405 completed successfully, 1x 13404 complete, 1x 13404 in progress), and I'm seeing PPD values all the way down around 950k for 13405 and 920k for 13404. For reference, on other Core 22 projects, I've seen PPDs more in the range of 1.2-1.4M (recent examples: 11745 @ 1.29M ppd, 14251 @ 1.37M PPD, or 3x16441 and 3x16442 @ 1.35M PPD), and 900k-1M is more in the range for what I tend to get on core 21 (I recently got a 16900 @ 920k ppd and a 16906@ 1085kppd, for example).

Has anyone else noticed this? It also seems to be drawing an abnormally small amount of power - it's only using 67% TDP right now, while I usually see 85-115% TDP usage when folding.

[clapanse]

Also, if it matters at all, here's my current log for 13405 (I just got another one after the 13404 finished):

Code: Select all

18:22:30:WU01:FS01:Started FahCore on PID 9440
18:22:30:WU01:FS01:Core PID:19368
18:22:30:WU01:FS01:FahCore 0x22 started
18:22:31:WU01:FS01:0x22:*********************** Log Started 2020-05-12T18:22:30Z ***********************
18:22:31:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
18:22:31:WU01:FS01:0x22:       Type: 0x22
18:22:31:WU01:FS01:0x22:       Core: Core22
18:22:31:WU01:FS01:0x22:    Website: https://foldingathome.org/
18:22:31:WU01:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
18:22:31:WU01:FS01:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
18:22:31:WU01:FS01:0x22:             <rafal.wiewiora@choderalab.org>
18:22:31:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 9440 -checkpoint 30
18:22:31:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
18:22:31:WU01:FS01:0x22:             0 -gpu 0
18:22:31:WU01:FS01:0x22:     Config: <none>
18:22:31:WU01:FS01:0x22:************************************ Build *************************************
18:22:31:WU01:FS01:0x22:    Version: 0.0.5
18:22:31:WU01:FS01:0x22:       Date: Apr 22 2020
18:22:31:WU01:FS01:0x22:       Time: 04:42:59
18:22:31:WU01:FS01:0x22: Repository: Git
18:22:31:WU01:FS01:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
18:22:31:WU01:FS01:0x22:     Branch: HEAD
18:22:31:WU01:FS01:0x22:   Compiler: Visual C++ 2008
18:22:31:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
18:22:31:WU01:FS01:0x22:   Platform: win32 10
18:22:31:WU01:FS01:0x22:       Bits: 64
18:22:31:WU01:FS01:0x22:       Mode: Release
18:22:31:WU01:FS01:0x22:************************************ System ************************************
18:22:31:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i7-3960X CPU @ 3.30GHz
18:22:31:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 45 Stepping 6
18:22:31:WU01:FS01:0x22:       CPUs: 12
18:22:31:WU01:FS01:0x22:     Memory: 31.92GiB
18:22:31:WU01:FS01:0x22:Free Memory: 16.27GiB
18:22:31:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
18:22:31:WU01:FS01:0x22: OS Version: 6.2
18:22:31:WU01:FS01:0x22:Has Battery: false
18:22:31:WU01:FS01:0x22: On Battery: false
18:22:31:WU01:FS01:0x22: UTC Offset: -6
18:22:31:WU01:FS01:0x22:        PID: 19368
18:22:31:WU01:FS01:0x22:        CWD: C:\Users\Chris\AppData\Roaming\FAHClient\work
18:22:31:WU01:FS01:0x22:         OS: Windows 10 Pro
18:22:31:WU01:FS01:0x22:    OS Arch: AMD64
18:22:31:WU01:FS01:0x22:********************************************************************************
18:22:31:WU01:FS01:0x22:Project: 13405 (Run 517, Clone 88, Gen 0)
18:22:31:WU01:FS01:0x22:Unit: 0x0000000112bc7d9a5eb5846b7218b752
18:22:31:WU01:FS01:0x22:Reading tar file core.xml
18:22:31:WU01:FS01:0x22:Reading tar file integrator.xml
18:22:31:WU01:FS01:0x22:Reading tar file state.xml
18:22:31:WU01:FS01:0x22:Reading tar file system.xml
18:22:32:WU01:FS01:0x22:Digital signatures verified
18:22:32:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
18:22:32:WU01:FS01:0x22:Version 0.0.5
18:22:36:WU00:FS01:Upload 34.02%
18:22:42:WU00:FS01:Upload 82.69%
18:22:49:WU00:FS01:Upload complete
18:22:49:WU00:FS01:Server responded WORK_ACK (400)
18:22:49:WU00:FS01:Final credit estimate, 73812.00 points
18:22:49:WU00:FS01:Cleaning up
18:22:57:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
18:22:57:WU01:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
18:25:31:WU01:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
18:28:00:WU01:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
18:30:28:WU01:FS01:0x22:Completed 30000 out of 1000000 steps (3%)
18:32:55:WU01:FS01:0x22:Completed 40000 out of 1000000 steps (4%)
18:35:19:WU01:FS01:0x22:Completed 50000 out of 1000000 steps (5%)
18:37:45:WU01:FS01:0x22:Completed 60000 out of 1000000 steps (6%)
18:40:16:WU01:FS01:0x22:Completed 70000 out of 1000000 steps (7%)
18:42:50:WU01:FS01:0x22:Completed 80000 out of 1000000 steps (8%)
18:45:18:WU01:FS01:0x22:Completed 90000 out of 1000000 steps (9%)
18:47:45:WU01:FS01:0x22:Completed 100000 out of 1000000 steps (10%)
18:50:14:WU01:FS01:0x22:Completed 110000 out of 1000000 steps (11%)
18:52:39:WU01:FS01:0x22:Completed 120000 out of 1000000 steps (12%)
18:55:03:WU01:FS01:0x22:Completed 130000 out of 1000000 steps (13%)
18:57:29:WU01:FS01:0x22:Completed 140000 out of 1000000 steps (14%)
18:59:53:WU01:FS01:0x22:Completed 150000 out of 1000000 steps (15%)
19:02:16:WU01:FS01:0x22:Completed 160000 out of 1000000 steps (16%)
19:04:40:WU01:FS01:0x22:Completed 170000 out of 1000000 steps (17%)
19:06:56:WU01:FS01:0x22:Completed 180000 out of 1000000 steps (18%)
19:09:14:WU01:FS01:0x22:Completed 190000 out of 1000000 steps (19%)
19:11:30:WU01:FS01:0x22:Completed 200000 out of 1000000 steps (20%)
19:13:45:WU01:FS01:0x22:Completed 210000 out of 1000000 steps (21%)
19:16:03:WU01:FS01:0x22:Completed 220000 out of 1000000 steps (22%)
19:18:18:WU01:FS01:0x22:Completed 230000 out of 1000000 steps (23%)
19:20:34:WU01:FS01:0x22:Completed 240000 out of 1000000 steps (24%)
19:22:49:WU01:FS01:0x22:Completed 250000 out of 1000000 steps (25%)
19:25:21:WU01:FS01:0x22:Completed 260000 out of 1000000 steps (26%)
19:27:47:WU01:FS01:0x22:Completed 270000 out of 1000000 steps (27%)
19:30:13:WU01:FS01:0x22:Completed 280000 out of 1000000 steps (28%)
19:32:39:WU01:FS01:0x22:Completed 290000 out of 1000000 steps (29%)
19:35:05:WU01:FS01:0x22:Completed 300000 out of 1000000 steps (30%)
19:37:31:WU01:FS01:0x22:Completed 310000 out of 1000000 steps (31%)
19:39:57:WU01:FS01:0x22:Completed 320000 out of 1000000 steps (32%)
19:42:23:WU01:FS01:0x22:Completed 330000 out of 1000000 steps (33%)
19:44:49:WU01:FS01:0x22:Completed 340000 out of 1000000 steps (34%)
19:47:15:WU01:FS01:0x22:Completed 350000 out of 1000000 steps (35%)
19:49:41:WU01:FS01:0x22:Completed 360000 out of 1000000 steps (36%)
19:52:07:WU01:FS01:0x22:Completed 370000 out of 1000000 steps (37%)
19:54:34:WU01:FS01:0x22:Completed 380000 out of 1000000 steps (38%)
19:57:00:WU01:FS01:0x22:Completed 390000 out of 1000000 steps (39%)
19:59:26:WU01:FS01:0x22:Completed 400000 out of 1000000 steps (40%)
20:01:52:WU01:FS01:0x22:Completed 410000 out of 1000000 steps (41%)
20:04:18:WU01:FS01:0x22:Completed 420000 out of 1000000 steps (42%)
20:06:44:WU01:FS01:0x22:Completed 430000 out of 1000000 steps (43%)
20:09:10:WU01:FS01:0x22:Completed 440000 out of 1000000 steps (44%)
20:11:36:WU01:FS01:0x22:Completed 450000 out of 1000000 steps (45%)
20:14:02:WU01:FS01:0x22:Completed 460000 out of 1000000 steps (46%)
20:16:28:WU01:FS01:0x22:Completed 470000 out of 1000000 steps (47%)
20:18:54:WU01:FS01:0x22:Completed 480000 out of 1000000 steps (48%)
20:21:20:WU01:FS01:0x22:Completed 490000 out of 1000000 steps (49%)
20:23:46:WU01:FS01:0x22:Completed 500000 out of 1000000 steps (50%)
20:26:02:WU01:FS01:0x22:Completed 510000 out of 1000000 steps (51%)
20:28:16:WU01:FS01:0x22:Completed 520000 out of 1000000 steps (52%)
20:30:28:WU01:FS01:0x22:Completed 530000 out of 1000000 steps (53%)
20:32:41:WU01:FS01:0x22:Completed 540000 out of 1000000 steps (54%)
20:34:54:WU01:FS01:0x22:Completed 550000 out of 1000000 steps (55%)
20:37:08:WU01:FS01:0x22:Completed 560000 out of 1000000 steps (56%)
20:39:20:WU01:FS01:0x22:Completed 570000 out of 1000000 steps (57%)
20:41:33:WU01:FS01:0x22:Completed 580000 out of 1000000 steps (58%)
20:43:46:WU01:FS01:0x22:Completed 590000 out of 1000000 steps (59%)
20:45:59:WU01:FS01:0x22:Completed 600000 out of 1000000 steps (60%)
20:48:13:WU01:FS01:0x22:Completed 610000 out of 1000000 steps (61%)
20:50:26:WU01:FS01:0x22:Completed 620000 out of 1000000 steps (62%)
20:52:39:WU01:FS01:0x22:Completed 630000 out of 1000000 steps (63%)
20:54:52:WU01:FS01:0x22:Completed 640000 out of 1000000 steps (64%)
20:57:05:WU01:FS01:0x22:Completed 650000 out of 1000000 steps (65%)
20:59:18:WU01:FS01:0x22:Completed 660000 out of 1000000 steps (66%)
21:01:32:WU01:FS01:0x22:Completed 670000 out of 1000000 steps (67%)
21:03:45:WU01:FS01:0x22:Completed 680000 out of 1000000 steps (68%)
21:05:58:WU01:FS01:0x22:Completed 690000 out of 1000000 steps (69%)
21:08:11:WU01:FS01:0x22:Completed 700000 out of 1000000 steps (70%)
21:10:24:WU01:FS01:0x22:Completed 710000 out of 1000000 steps (71%)
21:12:37:WU01:FS01:0x22:Completed 720000 out of 1000000 steps (72%)
21:14:50:WU01:FS01:0x22:Completed 730000 out of 1000000 steps (73%)
21:17:03:WU01:FS01:0x22:Completed 740000 out of 1000000 steps (74%)
21:19:17:WU01:FS01:0x22:Completed 750000 out of 1000000 steps (75%)
21:21:47:WU01:FS01:0x22:Completed 760000 out of 1000000 steps (76%)
21:24:15:WU01:FS01:0x22:Completed 770000 out of 1000000 steps (77%)
21:26:43:WU01:FS01:0x22:Completed 780000 out of 1000000 steps (78%)
21:29:11:WU01:FS01:0x22:Completed 790000 out of 1000000 steps (79%)
21:31:38:WU01:FS01:0x22:Completed 800000 out of 1000000 steps (80%)
21:34:06:WU01:FS01:0x22:Completed 810000 out of 1000000 steps (81%)
21:36:33:WU01:FS01:0x22:Completed 820000 out of 1000000 steps (82%)
21:39:01:WU01:FS01:0x22:Completed 830000 out of 1000000 steps (83%)
21:41:27:WU01:FS01:0x22:Completed 840000 out of 1000000 steps (84%)
21:43:53:WU01:FS01:0x22:Completed 850000 out of 1000000 steps (85%)
21:46:19:WU01:FS01:0x22:Completed 860000 out of 1000000 steps (86%)
21:48:46:WU01:FS01:0x22:Completed 870000 out of 1000000 steps (87%)
21:51:12:WU01:FS01:0x22:Completed 880000 out of 1000000 steps (88%)
21:53:38:WU01:FS01:0x22:Completed 890000 out of 1000000 steps (89%)
21:56:05:WU01:FS01:0x22:Completed 900000 out of 1000000 steps (90%)
21:58:30:WU01:FS01:0x22:Completed 910000 out of 1000000 steps (91%)
22:01:08:WU01:FS01:0x22:Completed 920000 out of 1000000 steps (92%)
22:03:44:WU01:FS01:0x22:Completed 930000 out of 1000000 steps (93%)
22:06:23:WU01:FS01:0x22:Completed 940000 out of 1000000 steps (94%)

[Neil-B]

The are a number of different reports about these projects - from odd processing profiles and unusual checkpoints through to Bad WU errors … The projects are something a bit different relating to some "high speed" turnarounds for COVID-Moonshot related research iirc - the researcher has responded to a number of the threads … Your report will assist the researcher understand a bit more about how the WUs for this project are varying.

[clapanse]

Interesting. I just grabbed yet another 13404, so I'm trying pausing my CPU folding right now just to experiment and see if maybe I wasn't leaving enough free CPU power to fully feed the GPU, but I'm glad to hear I'm not the only one with odd experiences here.

[Neil-B]

If you use the search for 13404 and/or 13405 and you will be able to piece together a fairly good view of what the project differences are.

actually just spotted this … viewtopic.php?f=19&t=35209#p333833

[clapanse]

After playing around with my clocks and settings, it does seem to be pretty memory bandwidth bound. Pushing memory clocks up from 5000 to 5600 remedied most of the PPD difference, though I haven't monitored other WUs to see if I get a similar gain there. Definitely glad to see the researcher's responses though.

[JohnChodera]

> After playing around with my clocks and settings, it does seem to be pretty memory bandwidth bound. Pushing memory clocks up from 5000 to 5600 remedied most of the PPD difference, though I haven't monitored other WUs to see if I get a similar gain there.

Interesting! Thanks for this---that may help us tune things on our end too.

~ John Chodera // MSKCC

[bruce]

I, too, am experiencing RAM "thrashing" (aka memory bandwidth saturation) but it's closely related to several other very large GPU projects. (>400K atoms). I've asked for clarification from the owner of those other projects but have not yet received an answer yet. I'm just trying to do too much with a system that used to be adequate.

[clapanse]

> After playing around with my clocks and settings, it does seem to be pretty memory bandwidth bound. Pushing memory clocks up from 5000 to 5600 remedied most of the PPD difference, though I haven't monitored other WUs to see if I get a similar gain there.

Interesting! Thanks for this---that may help us tune things on our end too.

~ John Chodera // MSKCC

Also, just in case that wasn't clear, that's the GPU memory clocks, not system memory. I've usually been running my GPU in P2 power state when folding to save a bit of power (basically full speed/power on the core, but ~10% underclocked memory), but this WU really seems to prefer P0, which is what pushed the clocks up from 5000 to 5600 (or I guess from 10Gbps to 11.2Gbps using Nvidia's method of measuring the clock speed).

[clapanse]

Also, looking through my logs, I have had a few of these fail as bad work units. I'm pretty sure this isn't my system, since no other WUs have failed in similar ways, and my understanding from the other threads are that this is kind of expected with this project. Here's a log of all the 13404s and 13405s that I've run though, just for some additional data:

https://i.imgur.com/XVZbPYF.png

I changed the power state from P2 to P0 at around 8PM UTC on 5/13, so you can see all the WUs after that got a significant boost in PPD (the highlighted WU is the first that was run with the higher clocks), even though GPU-Z only shows ~30-40% memory controller load with the WU running. Other GPU WUs I've received since the change seem to have not seen any performance difference, so this does seem unique to these projects.

[NRT_AntiKytherA]

I changed the power state from P2 to P0 at around 8PM UTC on 5/13

I just checked mine it's already on P0 according to CPU-Z. GPU-Z reports similar memory controller loading to you and TDP is around 75% when folding. GPU clock is 1923MHz, memory 1251MHz. Temp at 75C and no bad work units to date with this project's units, I actually like this loading level for the card as it's not pushing it too hard.

[clapanse]

1251? That sounds a lot like P2 - even on stock clocks, a 1080ti should be sitting at 1375 or so. Might be worth double checking with Nvidia Inspector? I know on default drivers, it forces P2 on compute workloads.

[NRT_AntiKytherA]

It's the default for the Gaming Mode profile set by MSI's Gaming App. I know it's slightly lower than the reference but maybe they did that to achieve stability with the higher GPU clock. I'll keep it as is anyway since it works well and I don't want to upset that and yes inspector says P2 not P0 like CPU-Z.

[clapanse]

Makes sense. In games though, you're running faster - note that the current mem clock is 5006, but the GPU clock is 5505, so in P0 (which is the performance state in games), you'll be running at 5505 instead (which is a normal clock for a factory 1080ti). It does look like there's a bit of a factory overclock on the core though - mine has a default core clock of 1480, vs yours at 1544 (though with how high it boosts, both of those clocks are a little bit misleading - as you can see, boost actually has you way up at 1898).

[clapanse]

So I've gotten several more of these now, and I'm pretty convinced that the memory fixed it for most. My computer did get a 13405(545, 58, 3) just now though, and it's running at well over 3 min/frame for only 513kppd. Not sure if this is useful feedback, but for some reason this particular run seems way slower than other variants of 13405.

Here's my log so far:

Code: Select all

13:34:17:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 18.188.125.154
13:34:17:WU01:FS01:Connecting to 18.188.125.154:8080
13:34:19:WU01:FS01:Downloading 5.98MiB
13:34:20:WU01:FS01:Download complete
13:34:20:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13405 run:545 clone:58 gen:3 core:0x22 unit:0x0000000a12bc7d9a5eb97d3eaa4f0fd2
13:34:20:WU01:FS01:Starting
13:34:20:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Chris\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 18208 -checkpoint 30 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
13:34:22:WU01:FS01:Started FahCore on PID 21880
13:34:22:WU01:FS01:Core PID:18064
13:34:22:WU01:FS01:FahCore 0x22 started
13:34:23:WU01:FS01:0x22:*********************** Log Started 2020-05-15T13:34:22Z ***********************
13:34:23:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
13:34:23:WU01:FS01:0x22:       Type: 0x22
13:34:23:WU01:FS01:0x22:       Core: Core22
13:34:23:WU01:FS01:0x22:    Website: https://foldingathome.org/
13:34:23:WU01:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
13:34:23:WU01:FS01:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
13:34:23:WU01:FS01:0x22:             <rafal.wiewiora@choderalab.org>
13:34:23:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 21880 -checkpoint 30
13:34:23:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
13:34:23:WU01:FS01:0x22:             0 -gpu 0
13:34:23:WU01:FS01:0x22:     Config: <none>
13:34:23:WU01:FS01:0x22:************************************ Build *************************************
13:34:23:WU01:FS01:0x22:    Version: 0.0.5
13:34:23:WU01:FS01:0x22:       Date: Apr 22 2020
13:34:23:WU01:FS01:0x22:       Time: 04:42:59
13:34:23:WU01:FS01:0x22: Repository: Git
13:34:23:WU01:FS01:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
13:34:23:WU01:FS01:0x22:     Branch: HEAD
13:34:23:WU01:FS01:0x22:   Compiler: Visual C++ 2008
13:34:23:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
13:34:23:WU01:FS01:0x22:   Platform: win32 10
13:34:23:WU01:FS01:0x22:       Bits: 64
13:34:23:WU01:FS01:0x22:       Mode: Release
13:34:23:WU01:FS01:0x22:************************************ System ************************************
13:34:23:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i7-3960X CPU @ 3.30GHz
13:34:23:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 45 Stepping 6
13:34:23:WU01:FS01:0x22:       CPUs: 12
13:34:23:WU01:FS01:0x22:     Memory: 31.92GiB
13:34:23:WU01:FS01:0x22:Free Memory: 15.84GiB
13:34:23:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
13:34:23:WU01:FS01:0x22: OS Version: 6.2
13:34:23:WU01:FS01:0x22:Has Battery: false
13:34:23:WU01:FS01:0x22: On Battery: false
13:34:23:WU01:FS01:0x22: UTC Offset: -6
13:34:23:WU01:FS01:0x22:        PID: 18064
13:34:23:WU01:FS01:0x22:        CWD: C:\Users\Chris\AppData\Roaming\FAHClient\work
13:34:23:WU01:FS01:0x22:         OS: Windows 10 Pro
13:34:23:WU01:FS01:0x22:    OS Arch: AMD64
13:34:23:WU01:FS01:0x22:********************************************************************************
13:34:23:WU01:FS01:0x22:Project: 13405 (Run 545, Clone 58, Gen 3)
13:34:23:WU01:FS01:0x22:Unit: 0x0000000a12bc7d9a5eb97d3eaa4f0fd2
13:34:23:WU01:FS01:0x22:Reading tar file core.xml
13:34:23:WU01:FS01:0x22:Reading tar file integrator.xml
13:34:23:WU01:FS01:0x22:Reading tar file state.xml
13:34:23:WU01:FS01:0x22:Reading tar file system.xml
13:34:23:WU01:FS01:0x22:Digital signatures verified
13:34:23:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
13:34:23:WU01:FS01:0x22:Version 0.0.5
13:34:40:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
13:34:40:WU01:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
13:38:02:WU01:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
13:41:22:WU01:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
13:44:43:WU01:FS01:0x22:Completed 30000 out of 1000000 steps (3%)
13:48:05:WU01:FS01:0x22:Completed 40000 out of 1000000 steps (4%)
13:51:25:WU01:FS01:0x22:Completed 50000 out of 1000000 steps (5%)
13:54:44:WU01:FS01:0x22:Completed 60000 out of 1000000 steps (6%)
13:58:05:WU01:FS01:0x22:Completed 70000 out of 1000000 steps (7%)
14:01:26:WU01:FS01:0x22:Completed 80000 out of 1000000 steps (8%)
14:04:47:WU01:FS01:0x22:Completed 90000 out of 1000000 steps (9%)
14:08:07:WU01:FS01:0x22:Completed 100000 out of 1000000 steps (10%)
14:11:27:WU01:FS01:0x22:Completed 110000 out of 1000000 steps (11%)
14:14:47:WU01:FS01:0x22:Completed 120000 out of 1000000 steps (12%)
14:18:07:WU01:FS01:0x22:Completed 130000 out of 1000000 steps (13%)
14:21:27:WU01:FS01:0x22:Completed 140000 out of 1000000 steps (14%)
14:24:44:WU01:FS01:0x22:Completed 150000 out of 1000000 steps (15%)
14:27:58:WU01:FS01:0x22:Completed 160000 out of 1000000 steps (16%)
14:31:11:WU01:FS01:0x22:Completed 170000 out of 1000000 steps (17%)
14:34:25:WU01:FS01:0x22:Completed 180000 out of 1000000 steps (18%)
14:37:39:WU01:FS01:0x22:Completed 190000 out of 1000000 steps (19%)
14:40:53:WU01:FS01:0x22:Completed 200000 out of 1000000 steps (20%)
14:44:06:WU01:FS01:0x22:Completed 210000 out of 1000000 steps (21%)
14:47:20:WU01:FS01:0x22:Completed 220000 out of 1000000 steps (22%)
14:50:33:WU01:FS01:0x22:Completed 230000 out of 1000000 steps (23%)
14:53:45:WU01:FS01:0x22:Completed 240000 out of 1000000 steps (24%)
14:56:59:WU01:FS01:0x22:Completed 250000 out of 1000000 steps (25%)
15:00:29:WU01:FS01:0x22:Completed 260000 out of 1000000 steps (26%)
15:03:54:WU01:FS01:0x22:Completed 270000 out of 1000000 steps (27%)
15:07:20:WU01:FS01:0x22:Completed 280000 out of 1000000 steps (28%)
15:10:46:WU01:FS01:0x22:Completed 290000 out of 1000000 steps (29%)
15:14:13:WU01:FS01:0x22:Completed 300000 out of 1000000 steps (30%)
15:17:38:WU01:FS01:0x22:Completed 310000 out of 1000000 steps (31%)
15:21:03:WU01:FS01:0x22:Completed 320000 out of 1000000 steps (32%)
15:24:29:WU01:FS01:0x22:Completed 330000 out of 1000000 steps (33%)
15:27:54:WU01:FS01:0x22:Completed 340000 out of 1000000 steps (34%)
15:31:20:WU01:FS01:0x22:Completed 350000 out of 1000000 steps (35%)
15:34:45:WU01:FS01:0x22:Completed 360000 out of 1000000 steps (36%)
15:38:10:WU01:FS01:0x22:Completed 370000 out of 1000000 steps (37%)
15:41:35:WU01:FS01:0x22:Completed 380000 out of 1000000 steps (38%)
15:45:00:WU01:FS01:0x22:Completed 390000 out of 1000000 steps (39%)
15:48:24:WU01:FS01:0x22:Completed 400000 out of 1000000 steps (40%)
15:51:49:WU01:FS01:0x22:Completed 410000 out of 1000000 steps (41%)
15:55:12:WU01:FS01:0x22:Completed 420000 out of 1000000 steps (42%)
15:58:39:WU01:FS01:0x22:Completed 430000 out of 1000000 steps (43%)
16:02:08:WU01:FS01:0x22:Completed 440000 out of 1000000 steps (44%)
16:05:38:WU01:FS01:0x22:Completed 450000 out of 1000000 steps (45%)
16:09:10:WU01:FS01:0x22:Completed 460000 out of 1000000 steps (46%)
16:12:36:WU01:FS01:0x22:Completed 470000 out of 1000000 steps (47%)
16:16:03:WU01:FS01:0x22:Completed 480000 out of 1000000 steps (48%)
16:19:31:WU01:FS01:0x22:Completed 490000 out of 1000000 steps (49%)
******************************* Date: 2020-05-15 *******************************
16:22:53:WU01:FS01:0x22:Completed 500000 out of 1000000 steps (50%)
16:26:12:WU01:FS01:0x22:Completed 510000 out of 1000000 steps (51%)
16:29:26:WU01:FS01:0x22:Completed 520000 out of 1000000 steps (52%)
16:32:39:WU01:FS01:0x22:Completed 530000 out of 1000000 steps (53%)
16:35:53:WU01:FS01:0x22:Completed 540000 out of 1000000 steps (54%)
16:39:04:WU01:FS01:0x22:Completed 550000 out of 1000000 steps (55%)
16:42:18:WU01:FS01:0x22:Completed 560000 out of 1000000 steps (56%)
16:45:28:WU01:FS01:0x22:Completed 570000 out of 1000000 steps (57%)
16:48:34:WU01:FS01:0x22:Completed 580000 out of 1000000 steps (58%)
16:51:36:WU01:FS01:0x22:Completed 590000 out of 1000000 steps (59%)
16:54:48:WU01:FS01:0x22:Completed 600000 out of 1000000 steps (60%)
16:57:57:WU01:FS01:0x22:Completed 610000 out of 1000000 steps (61%)
17:01:00:WU01:FS01:0x22:Completed 620000 out of 1000000 steps (62%)
17:04:04:WU01:FS01:0x22:Completed 630000 out of 1000000 steps (63%)
17:07:12:WU01:FS01:0x22:Completed 640000 out of 1000000 steps (64%)
17:10:19:WU01:FS01:0x22:Completed 650000 out of 1000000 steps (65%)
17:13:32:WU01:FS01:0x22:Completed 660000 out of 1000000 steps (66%)
17:16:54:WU01:FS01:0x22:Completed 670000 out of 1000000 steps (67%)
17:20:17:WU01:FS01:0x22:Completed 680000 out of 1000000 steps (68%)
17:23:33:WU01:FS01:0x22:Completed 690000 out of 1000000 steps (69%)
17:26:52:WU01:FS01:0x22:Completed 700000 out of 1000000 steps (70%)
17:30:07:WU01:FS01:0x22:Completed 710000 out of 1000000 steps (71%)
17:33:27:WU01:FS01:0x22:Completed 720000 out of 1000000 steps (72%)
17:36:49:WU01:FS01:0x22:Completed 730000 out of 1000000 steps (73%)
17:40:10:WU01:FS01:0x22:Completed 740000 out of 1000000 steps (74%)
17:43:38:WU01:FS01:0x22:Completed 750000 out of 1000000 steps (75%)
17:47:26:WU01:FS01:0x22:Completed 760000 out of 1000000 steps (76%)
17:51:04:WU01:FS01:0x22:Completed 770000 out of 1000000 steps (77%)
17:54:48:WU01:FS01:0x22:Completed 780000 out of 1000000 steps (78%)
17:58:26:WU01:FS01:0x22:Completed 790000 out of 1000000 steps (79%)
18:01:57:WU01:FS01:0x22:Completed 800000 out of 1000000 steps (80%)
18:05:28:WU01:FS01:0x22:Completed 810000 out of 1000000 steps (81%)
18:08:59:WU01:FS01:0x22:Completed 820000 out of 1000000 steps (82%)
18:12:32:WU01:FS01:0x22:Completed 830000 out of 1000000 steps (83%)
18:16:07:WU01:FS01:0x22:Completed 840000 out of 1000000 steps (84%)
18:19:45:WU01:FS01:0x22:Completed 850000 out of 1000000 steps (85%)