Project 13414

[Curt3g]

Following up on a post from this morning that I may not have put in the right forum (leave it to a new guy ).

viewtopic.php?f=74&t=35624&start=15#p338142 (includes log extracts)

The final processing time on Project: 13414 (Run 824, Clone 39, Gen 1) was a little over 13 hours (RTX 2060).

Not experienced enough to know if this is an issue or not.

Cheers,
Curt

[bruce]

The preferred deadline is 1.0 days and the final deadline is 2.0 days for that project. 13 hours isn't unreasonable.

[Curt3g]

Project: 13412 (Run 2439, Clone 10, Gen 0) was processed a little bit later last night, with a run time a hair over 4 hrs.

Cheers,

[JohnChodera]

Wow, we were very much aiming for ~4h, so 13h sounds like something weird is going on. We're seeing some heterogeneity in our calculation, and are looking into what is causing this.

Thanks so much!

~ John Chodera // MSKCC

[Curt3g]

Thanks John. Picked up another 13414 this morning (Run 255, Clone 83, Gen 0). The estimated time between frames is 2 min 18 sec, so looks to be in the 4 hour range also.

Cheers,

[Ichbin3]

I got a 13414 recently which was unexpected slow too
13414 (841,39,1)
TPF was 1:31 (other 13414 where 1:08 normaly) on a 2080Ti

[kc2lrc]

I'm seeing some interesting performance numbers myself:

- 13415 on GTX 970, Debian 10: 1m 21s TPF, 106K points/day - nvidia-smi says GPU usage is around 46%
- 13415 on GTX 960, Windows 10: 1m 38s TPF, 79K points/day, HWMon shows GPU usage of 47%
- 13414 on GTX 1080, Debian 10: 2m 12s TPF, 1,076K points/day, nvidia-smi says GPU usage is 98%

Haven't seen it hit any of my 750Ti's yet.

Cheers -
Sam

[Ichbin3]

Just another slow one at my 2080Ti
13414 (1143,7,0)
TPF 1:46 vs 1:08 a normal one

[cine.chris]

I`ve seen a couple of very low PPD for WU 13414 today, which is why I`m here. Latest is on a 1660Ti, GPU utilization was at 75%.

[Joe_H]

I`ve seen a couple of very low PPD for WU 13414 today, which is why I`m here. Latest is on a 1660Ti, GPU utilization was at 75%.

What were the PRCG numbers for the underperforming WUs? They are trying to track down which Runs seem to have this issue and see if they can figure out why.

[kc2lrc]

My slugs were:

13415 (R 835, C 51, G 1) - 1m 21s on GTX 970 (Debian 10, Core 2 Duo E8200), reported 210ns/day, GPU usage around 47%
13415 (R 850, C 50, G 0) - 1m 23s on GTX 970, reported 207ns/day, wasn't around to see GPU usage
13415 (R 792, C 51, G 1) - 1m 40s on GTX 960 (Windows 10, i7-4790K), reported 128ns/day, GPU usage around 46%
13413 (R 767, C 3, G 0) - 2m 5s on GTX 960, reported 130ns/day, wasn't around to see GPU usage

No slugs are churning right now, but I'll keep my eyes open when I'm around.

Cheers -
Sam

[kc2lrc]

Noticed one more currently running:

13415 (1610, 8, 0) - 1m 44s on GTX 750Ti (Debian 10, Core 2 E8200), nvidia-smi reports GPU usage around 58%

Cheers -
Sam

[Curt3g]

And a couple more today (RTX 2060), first looks as expected, second looks slow. And didn't mention in previous posts, running 24/7.

From earlier today: 13414 (Run 1816, Clone 4, Gen 1), TPF at 2 min 10 sec
Currently running: 13414 (Run 1553, Clone 12, Gen 0), TPF at 7 min, 18 sec

Regards,
Curt

[uyaem]

Adding to this another Run that seems to be slow (although with sample size = 1, I can't be sure if it's the entire Run):
Project 13414 (Run 879, Clone 26, Gen 0)

I've observed the same behaviour as others 75% GPU load (nVidia 1660 GTX).

[ajm]

Two slow ones more currently running:
13414 (1209, 20, 0) -> 1080 ti, TPF 3 mins 06 secs
13414 (1586, 20, 0) -> 2070 S, TPF 2 mins 40 secs

EDIT: one more, also currently running:
13414 (1618, 12, 0) -> 1050 ti, TPF 8 mins 51 secs

Three others currently running, but with a normal TPF:
13414 (2227, 14, 1) -> 1050 ti, TPF 6 mins 13 secs
13414 (1275, 23, 0) -> 5700 XT, TPF 2 mins 05 secs
13414 (2737, 22, 0) -> 2080 S, TPF 1 mins 22 secs