Page 1 of 2

Project 13414

Posted: Mon Jun 22, 2020 5:28 am
by Curt3g
Following up on a post from this morning that I may not have put in the right forum (leave it to a new guy :D ).

viewtopic.php?f=74&t=35624&start=15#p338142 (includes log extracts)

The final processing time on Project: 13414 (Run 824, Clone 39, Gen 1) was a little over 13 hours (RTX 2060).

Not experienced enough to know if this is an issue or not.

Cheers,
Curt

Re: Project 13414

Posted: Mon Jun 22, 2020 5:59 am
by bruce
The preferred deadline is 1.0 days and the final deadline is 2.0 days for that project. 13 hours isn't unreasonable.

Re: Project 13414

Posted: Mon Jun 22, 2020 2:36 pm
by Curt3g
Project: 13412 (Run 2439, Clone 10, Gen 0) was processed a little bit later last night, with a run time a hair over 4 hrs.

Cheers,

Re: Project 13414

Posted: Mon Jun 22, 2020 2:57 pm
by JohnChodera
Wow, we were very much aiming for ~4h, so 13h sounds like something weird is going on. We're seeing some heterogeneity in our calculation, and are looking into what is causing this.

Thanks so much!

~ John Chodera // MSKCC

Re: Project 13414

Posted: Mon Jun 22, 2020 5:15 pm
by Curt3g
Thanks John. Picked up another 13414 this morning (Run 255, Clone 83, Gen 0). The estimated time between frames is 2 min 18 sec, so looks to be in the 4 hour range also.

Cheers,

Re: Project 13414

Posted: Mon Jun 22, 2020 6:13 pm
by Ichbin3
I got a 13414 recently which was unexpected slow too
13414 (841,39,1)
TPF was 1:31 (other 13414 where 1:08 normaly) on a 2080Ti

Re: Project 13414

Posted: Tue Jun 23, 2020 3:49 am
by kc2lrc
I'm seeing some interesting performance numbers myself:

- 13415 on GTX 970, Debian 10: 1m 21s TPF, 106K points/day - nvidia-smi says GPU usage is around 46%
- 13415 on GTX 960, Windows 10: 1m 38s TPF, 79K points/day, HWMon shows GPU usage of 47%
- 13414 on GTX 1080, Debian 10: 2m 12s TPF, 1,076K points/day, nvidia-smi says GPU usage is 98%

Haven't seen it hit any of my 750Ti's yet.

Cheers -
Sam

Re: Project 13414

Posted: Tue Jun 23, 2020 3:29 pm
by Ichbin3
Just another slow one at my 2080Ti
13414 (1143,7,0)
TPF 1:46 vs 1:08 a normal one

Re: Project 13414

Posted: Tue Jun 23, 2020 11:03 pm
by cine.chris
I`ve seen a couple of very low PPD for WU 13414 today, which is why I`m here. Latest is on a 1660Ti, GPU utilization was at 75%.

Re: Project 13414

Posted: Tue Jun 23, 2020 11:55 pm
by Joe_H
cine.chris wrote:I`ve seen a couple of very low PPD for WU 13414 today, which is why I`m here. Latest is on a 1660Ti, GPU utilization was at 75%.
What were the PRCG numbers for the underperforming WUs? They are trying to track down which Runs seem to have this issue and see if they can figure out why.

Re: Project 13414

Posted: Wed Jun 24, 2020 1:48 am
by kc2lrc
My slugs were:

13415 (R 835, C 51, G 1) - 1m 21s on GTX 970 (Debian 10, Core 2 Duo E8200), reported 210ns/day, GPU usage around 47%
13415 (R 850, C 50, G 0) - 1m 23s on GTX 970, reported 207ns/day, wasn't around to see GPU usage
13415 (R 792, C 51, G 1) - 1m 40s on GTX 960 (Windows 10, i7-4790K), reported 128ns/day, GPU usage around 46%
13413 (R 767, C 3, G 0) - 2m 5s on GTX 960, reported 130ns/day, wasn't around to see GPU usage

No slugs are churning right now, but I'll keep my eyes open when I'm around.

Cheers -
Sam

Re: Project 13414

Posted: Wed Jun 24, 2020 1:52 am
by kc2lrc
Noticed one more currently running:

13415 (1610, 8, 0) - 1m 44s on GTX 750Ti (Debian 10, Core 2 E8200), nvidia-smi reports GPU usage around 58%

Cheers -
Sam

Re: Project 13414

Posted: Wed Jun 24, 2020 3:50 am
by Curt3g
And a couple more today (RTX 2060), first looks as expected, second looks slow. And didn't mention in previous posts, running 24/7.

From earlier today: 13414 (Run 1816, Clone 4, Gen 1), TPF at 2 min 10 sec
Currently running: 13414 (Run 1553, Clone 12, Gen 0), TPF at 7 min, 18 sec

Regards,
Curt

Re: Project 13414

Posted: Wed Jun 24, 2020 8:54 am
by uyaem
Adding to this another Run that seems to be slow (although with sample size = 1, I can't be sure if it's the entire Run):
Project 13414 (Run 879, Clone 26, Gen 0)

I've observed the same behaviour as others 75% GPU load (nVidia 1660 GTX).

Re: Project 13414

Posted: Wed Jun 24, 2020 9:07 am
by ajm
Two slow ones more currently running:
13414 (1209, 20, 0) -> 1080 ti, TPF 3 mins 06 secs
13414 (1586, 20, 0) -> 2070 S, TPF 2 mins 40 secs

EDIT: one more, also currently running:
13414 (1618, 12, 0) -> 1050 ti, TPF 8 mins 51 secs

Three others currently running, but with a normal TPF:
13414 (2227, 14, 1) -> 1050 ti, TPF 6 mins 13 secs
13414 (1275, 23, 0) -> 5700 XT, TPF 2 mins 05 secs
13414 (2737, 22, 0) -> 2080 S, TPF 1 mins 22 secs