Nvidia vs AMD CPU usage

arisu · Post by **arisu** » Thu Mar 13, 2025 3:09 am

Joe_H wrote: ↑Tue Mar 04, 2025 4:40 am From what I understand, the difference is in how Nvidia and AMD wrote their drivers. Nvidia's driver is doing a spin-wait looking for instructions to be processed and sent to the GPU. AMD from the explanations I have seen implemented this as an interrupt instead. As soon as something is handed off to the driver to process, it wakes up and takes CPU cycles to handle the request and then goes inactive until the next request. So the Nvidia driver process is always active, but the actual amount of work done by the CPU may be a fraction of the cycles available.

I believe I have the definitive answer about why we see this.

It turns out it's not the driver, just a choice that FAH made in their configuration of OpenMM. They overrode the default for UseBlockingSync and set it to false. This increases performance slightly but causes the CPU usage people report.

http://docs.openmm.org/latest/userguide ... a-platform

UseBlockingSync: This is used to control how the CUDA runtime synchronizes between the CPU and GPU. If this is set to “true” (the default), CUDA will allow the calling thread to sleep while the GPU is performing a computation, allowing the CPU to do other work. If it is set to “false”, CUDA will spin-lock while the GPU is working. Setting it to “false” can improve performance slightly, but also prevents the CPU from doing anything else while the GPU is working.

When the CPU sends data to the GPU, it calls cudaDeviceSynchronize() which will wait until the GPU has finished before it returns. The majority of the CPU's time will be spent in that function. That function will either use a spin wait loop or will yield the CPU and wait for an interrupt before returning, depending on if cudaSetDeviceFlags(cudaDeviceScheduleBlockingSync) has been called or not. The UseBlockingSync feature in OpenMM just calls that function.

So why did FAH do this? I don't know. From reports, the performance improvement is very very slight. Maybe it was just set and forget? Maybe there are some systems out there where the performance improvement is big enough to make it worth while? I'll test this out when I get a new Nvidia and see if I can write a simple program that injects the blocking sync flag for a test WU.

Post by **calxalot** » Thu Mar 13, 2025 3:29 am

I remember reading about attempts to reduce cpu usage. Usage could be reduced but also greatly reduced folding speed. I think I read 30% slower.

This was somewhere on the openmm.org site or the github repos for openmm.

arisu · Post by **arisu** » Thu Mar 13, 2025 3:31 am

The site only says "slightly", so I'm guessing it's on the repo. I'll look.

Edit: Yep it's in the repo, probably this one https://github.com/openmm/openmm/issues/2955

The perf decrease is negligible... except on Linux where it is 35% (and Windows if kept in balanced power mode). That explains why it was done. Linux runs on a lot of the most important folding boxes!

Post by **Joe_H** » Thu Mar 13, 2025 3:59 am

The last note on the issue, that it was not possible and to continue supporting OpenCL 1.2 may be a big part. Nvidia was slow to support any version of OpenCL beyond 1.2, and drivers released in the last few years including support for higher versions were not back ported to older generation GPUs.

muziqaz · Post by **muziqaz** » Thu Mar 13, 2025 5:53 am

arisu wrote: ↑Thu Mar 13, 2025 3:31 am The site only says "slightly", so I'm guessing it's on the repo. I'll look.

Edit: Yep it's in the repo, probably this one https://github.com/openmm/openmm/issues/2955

The perf decrease is negligible... except on Linux where it is 35% (and Windows if kept in balanced power mode). That explains why it was done. Linux runs on a lot of the most important folding boxes!

The person who done these configs was nVidia employee at the time and one of the FAH founders. Their knowledge of nVidia hardware preferences beats anyone here or anywhere else.
The choices made could be considered best choices made for performance. Since that person now moved on from nVidia and CUDA fahcore is running great as is, I don't see anything changing too much in near future regarding this

arisu · Post by **arisu** » Thu Mar 13, 2025 6:21 am

Yep it looks like it. I've been reading up on potential improvements (not because I think I know better than the devs, but because I like to learn). I find that the ones that reduce CPU use without reducing GPU folding performance, like estimating the amount of time a certain kernel will take to complete, and sleeping for 90% of that duration so it only spins at the end, come with considerable engineering and testing challenges that wouldn't be worth it just to free up a single CPU core.

muziqaz · Post by **muziqaz** » Thu Mar 13, 2025 6:43 am

arisu wrote: ↑Thu Mar 13, 2025 6:21 am Yep it looks like it. I've been reading up on potential improvements (not because I think I know better than the devs, but because I like to learn). I find that the ones that reduce CPU use without reducing GPU folding performance, like estimating the amount of time a certain kernel will take to complete, and sleeping for 90% of that duration so it only spins at the end, come with considerable engineering and testing challenges that wouldn't be worth it just to free up a single CPU core.

You can try pushing PR to OpenMM GitHub, but remember, people who brought CUDA to the table, did not just do a single test and said: yeah, looks good enough. There was a lot of testing and nVidia's own labs involved in developing this stuff.
There are always compromises available in development (software or hardware).

DarkFoss · Post by **DarkFoss** » Thu Apr 17, 2025 9:01 pm

I can't speak for other platforms but the one in my sig received a bios update on April Fool's Day.

Installed it 2 days ago.I'm seeing a different behavior under Linux now. Ubuntu also released an updated 6.11 hwe kernel (using the Low latency varient).
Bios setting tweaks are enabling DOCP (ram timings) and lowering the default 90c cut off to 88c
Before it would only settle in at 4.2 across all with normal downclocking for temp reg cpu frequency dipping to 4.1 with 1 core dropping to 500 briefly.
Now with all the new changes it still does all of the above but PBO seems to be kicking in 1 core with bump up to 4.3 for additional single thread performance.
Not sure how much impact it's will have on my x570 but those with newer AMD platforms may have that little bit extra. As always ymmv.

arisu · Post by **arisu** » Mon Apr 21, 2025 1:29 pm

I've tested FAH with blocking sync turned on by hooking a CUDA function and changing the flags. It took me longer than expected (it would be easy if the EULA allowed reverse engineering or if it was open source, but I had to whitebox it and play with the CUDA runtime and driver API just to not violate it). There was a lot of messiness involved just to keep within the rules which is annoying because I'm not even doing this on live projects.

Here is a stripped down version of the library I used, which is loaded via LD_PRELOAD, in case anyone wants to play with it (disclaimer: Do not use on any WUs that you are intending to send back to the server):

Code: Select all

#include <dlfcn.h>
#include <assert.h>
#include <cuda.h>

typedef CUresult (*cuEventCreate_type)(CUevent *, unsigned int);
cuEventCreate_type orig_cuEventCreate;

static __attribute__((constructor)) void hook_function_early(void)
{
        void *handle;

        handle = dlopen("libcuda.so.1", RTLD_NOW);
        assert(handle);

        orig_cuEventCreate = (cuEventCreate_type)dlsym(handle, "cuEventCreate");
        assert(orig_cuEventCreate);
}

extern CUresult cuEventCreate(CUevent *phEvent, unsigned int flags)
{
        return orig_cuEventCreate(phEvent, flags | CU_EVENT_BLOCKING_SYNC);
}

Command I used to monitor GPU usage:

Code: Select all

nvidia-smi dmon -i 0 -s pucmt -o T

Command I used to monitor CPU usage:

Code: Select all

pid=$(pgrep -x FahCore_2.)
watch -tp -n1 "top -b -n1 -o -PID -p $pid; lscpu -e=CPU,CORE,CACHE,ONLINE,SCALMHZ%,MHZ,MAXMHZ,MINMHZ"

CPU usage on Linux does decrease, but only slightly on an RTX 4090 Mobile. The CPU core (running on a P-core) only went from 100% to around 90%, and the clock fell from 4 GHz to 3.8 GHz. Some projects seem to have minimal PPD impact, others have more substantial impact. Without looking at TPF, I saw the SM usage go from ~95 to a bit over 85 on one project. I suspect there are some scheduler tweaks that could eliminate the PPD loss.

When I tested this on a GTX 970M, there was no detectable PPD loss and CPU usage went down to 10%. I suspect that faster Nvidia GPUs finish kernels so quickly that the overhead of using non-blocking sync becomes substantial.

arisu · Post by **arisu** » Mon May 19, 2025 6:27 am

I did a test with a GTX 970M to get the duration of each synchronization call over a period of 30 seconds (1122 calls). The first field is the timestamp and the second field is the number of microseconds that the cuEventSynchronize() call took (aka the number of microseconds that it burning CPU in a spin-wait).

Code: Select all

1747633510.000717 21053
1747633510.010863 9843
1747633510.024635 13483
1747633510.045907 20982
1747633510.056172 9965
1747633510.069413 12957
1747633510.090615 20866
1747633510.100990 10068
1747633510.122653 21367
1747633510.132919 9979
1747633510.174018 9733
1747633510.195013 8470
1747633510.205284 9988
1747633510.219139 13557
1747633510.240537 21104
1747633510.250818 9984
1747633510.272161 21028
1747633510.282551 10065
1747633510.295904 13044
1747633510.317228 21025
1747633510.327557 10028
1747633510.349102 21251
1747633510.359413 10042
1747633510.373153 13471
1747633510.394279 20812
1747633510.405464 10904
1747633510.427491 21846
1747633510.437887 10118
1747633510.459909 21760
1747633510.470402 10158
1747633510.491586 20891
1747633510.501972 10097
1747633510.523879 21646
1747633510.534384 10189
1747633510.556214 21546
1747633510.566598 10094
1747633510.580436 13554
1747633510.621821 9609
1747633510.643048 8491
1747633510.653539 10154
1747633510.666917 13102
1747633510.688524 21326
1747633510.698910 10130
1747633510.712379 13185
1747633510.734298 21649
1747633510.744701 10137
1747633510.766654 21688
1747633510.777186 10236
1747633510.790659 13181
1747633510.812554 21613
1747633510.822959 10101
1747633510.836870 13619
1747633510.858736 21560
1747633510.869143 10130
1747633510.891353 21945
1747633510.901881 10254
1747633510.923334 21168
1747633510.933761 10136
1747633510.955864 21843
1747633510.966418 10244
1747633510.979903 13208
1747633511.001562 21357
1747633511.011957 10115
1747633511.034134 21992
1747633511.069551 9873
1747633511.090578 8327
1747633511.101001 10154
1747633511.123023 21739
1747633511.133478 10136
1747633511.154945 21180
1747633511.166081 10844
1747633511.187644 21246
1747633511.198033 10104
1747633511.211644 13285
1747633511.233296 21373
1747633511.244017 10442
1747633511.258091 13781
1747633511.279786 21422
1747633511.290371 10291
1747633511.304135 13435
1747633511.326405 21979
1747633511.337096 10385
1747633511.350779 13358
1747633511.372821 21699
1747633511.383448 10308
1747633511.397869 14118
1747633511.420093 21951
1747633511.430701 10271
1747633511.444443 13472
1747633511.466884 22157
1747633511.477444 10282
1747633511.519403 9939
1747633511.541436 8847
1747633511.551993 10282
1747633511.565531 13234
1747633511.587793 21968
1747633511.598331 10278
1747633511.620485 21880
1747633511.631083 10306
1747633511.653490 22114
1747633511.664108 10316
1747633511.686261 21876
1747633511.696862 10321
1747633511.710750 13612
1747633511.732802 21772
1747633511.743389 10300
1747633511.757597 13923
1747633511.780240 22365
1747633511.790834 10315
1747633511.813327 22178
1747633511.823909 10299
1747633511.838231 14042
1747633511.860769 22260
1747633511.871567 10500
1747633511.894205 22366
1747633511.905132 10645
1747633511.918965 13448
1747633511.942088 22815
1747633511.979071 10262
1747633512.000626 8398
1747633512.011969 11049
1747633512.026148 13861
1747633512.048806 22341
1747633512.059607 10500
1747633512.073530 13651
1747633512.095873 22045
1747633512.106662 10455
1747633512.120495 13567
1747633512.142770 21971
1747633512.155234 12257
1747633512.178250 22744
1747633512.189085 10533
1747633512.211555 22191
1747633512.222513 10679
1747633512.236455 13639
1747633512.258569 21833
1747633512.269841 10981
1747633512.283834 13706
1747633512.306350 22229
1747633512.317217 10578
1747633512.331070 13615
1747633512.353580 22162
1747633512.364395 10519
1747633512.386346 21668
1747633512.398093 11449
1747633512.441735 10511
1747633512.464186 9201
1747633512.475134 10650
1747633512.497927 22526
1747633512.508728 10514
1747633512.522508 13521
1747633512.545142 22336
1747633512.555936 10527
1747633512.569821 13613
1747633512.592391 22271
1747633512.603335 10629
1747633512.625479 21809
1747633512.636290 10510
1747633512.658706 22146
1747633512.669623 10607
1747633512.692772 22874
1747633512.703582 10491
1747633512.726544 22679
1747633512.737480 10630
1747633512.751339 13573
1747633512.774096 22476
1747633512.784976 10610
1747633512.808036 22761
1747633512.818862 10486
1747633512.841102 21954
1747633512.851914 10529
1747633512.874572 22366
1747633512.910905 10089
1747633512.933406 9046
1747633512.944224 10558
1747633512.966655 22135
1747633512.977468 10530
1747633513.000383 22648
1747633513.011254 10592
1747633513.025247 13679
1747633513.047049 21459
1747633513.058897 11569
1747633513.073484 14322
1747633513.095904 22127
1747633513.106740 10500
1747633513.120717 13704
1747633513.144111 23125
1747633513.154939 10527
1747633513.168746 13537
1747633513.191396 22381
1747633513.202200 10512
1747633513.216065 13595
1747633513.238927 22587
1747633513.249896 10657
1747633513.272121 21883
1747633513.282968 10549
1747633513.305263 22041
1747633513.316251 10695
1747633513.338664 22120
1747633513.375180 10534
1747633513.397582 8934
1747633513.408427 10555
1747633513.431285 22586
1747633513.442283 10717
1747633513.456133 13546
1747633513.479503 23091
1747633513.490368 10576
1747633513.504393 13735
1747633513.526191 21510
1747633513.576437 9001
1747633513.587141 10433
1747633513.600887 13444
1747633513.623172 21982
1747633513.633863 10384
1747633513.656076 21940
1747633513.666855 10444
1747633513.689344 22206
1747633513.700025 10392
1747633513.722166 21796
1747633513.732985 10519
1747633513.754799 21528
1747633513.765530 10441
1747633513.779281 13492
1747633513.801721 22166
1747633513.812494 10450
1747633513.826212 13419
1747633513.867898 10052
1747633513.888710 7610
1747633513.901111 12112
1747633513.915582 14180
1747633513.937815 21949
1747633513.948544 10436
1747633513.970336 21522
1747633513.981868 11231
1747633513.995703 13501
1747633514.017258 21275
1747633514.028752 11195
1747633514.043270 14220
1747633514.065842 22299
1747633514.076612 10504
1747633514.090891 14015
1747633514.113852 22689
1747633514.124711 10575
1747633514.139573 14578
1747633514.161903 22039
1747633514.172943 10752
1747633514.195706 22475
1747633514.206610 10617
1747633514.221455 14559
1747633514.244934 23187
1747633514.255821 10589
1747633514.278483 22399
1747633514.289412 10589
1747633514.332270 10188
1747633514.345669 373
1747633514.359717 13820
1747633514.382985 22968
1747633514.393918 10648
1747633514.407911 13739
1747633514.430759 22474
1747633514.441694 10651
1747633514.464697 22729
1747633514.475634 10635
1747633514.498445 22538
1747633514.509337 10583
1747633514.531935 22336
1747633514.542821 10555
1747633514.566565 23444
1747633514.577454 10598
1747633514.601140 23407
1747633514.612069 10622
1747633514.626235 13855
1747633514.649016 22442
1747633514.659918 10618
1747633514.682685 22395
1747633514.693696 10713
1747633514.707751 13757
1747633514.731071 22909
1747633514.742024 10670
1747633514.756146 13849
1747633514.786111 10478
1747633514.808812 9054
1747633514.819729 10641
1747633514.833755 13746
1747633514.855854 21804
1747633514.867215 11036
1747633514.881282 13788
1747633514.904661 23051
1747633514.915599 10630
1747633514.938805 22908
1747633514.949751 10665
1747633514.963726 13659
1747633514.985665 21642
1747633514.998365 12404
1747633515.021244 22616
1747633515.032225 10704
1747633515.054916 22389
1747633515.066000 10781
1747633515.088897 22609
1747633515.099835 10584
1747633515.122346 22211
1747633515.133704 11075
1747633515.148475 14486
1747633515.171515 22754
1747633515.182463 10654
1747633515.205706 22963
1747633515.216673 10668
1747633515.248055 10537
1747633515.270574 8913
1747633515.281554 10637
1747633515.304292 22445
1747633515.315379 10791
1747633515.329468 13811
1747633515.353035 23274
1747633515.363995 10673
1747633515.386973 22682
1747633515.397911 10653
1747633515.411924 13741
1747633515.435772 23545
1747633515.446861 10790
1747633515.460961 13825
1747633515.483585 22322
1747633515.494640 10746
1747633515.517517 22575
1747633515.528505 10705
1747633515.552049 23271
1747633515.563223 10863
1747633515.586627 23113
1747633515.597795 10887
1747633515.621183 23126
1747633515.632347 10874
1747633515.655708 23090
1747633515.666915 10880
1747633515.690461 23261
1747633515.714887 10438
1747633515.737106 8520
1747633515.749223 11845
1747633515.764416 14936
1747633515.788408 23683
1747633515.799615 10912
1747633515.814002 14049
1747633515.837778 23401
1747633515.848966 10886
1747633515.872454 23181
1747633515.883665 10930
1747633515.907639 23636
1747633515.918838 10914
1747633515.942378 23197
1747633515.953560 10893
1747633515.968157 14330
1747633515.991406 22976
1747633516.002601 10861
1747633516.026160 23252
1747633516.037464 11011
1747633516.051843 14101
1747633516.075834 23707
1747633516.087022 10876
1747633516.110967 23651
1747633516.122165 10895
1747633516.137163 14677
1747633516.161343 23834
1747633516.186409 10880
1747633516.210078 9549
1747633516.221641 11280
1747633516.245125 23199
1747633516.256570 11154
1747633516.280550 23712
1747633516.292109 11262
1747633516.306710 14316
1747633516.330389 23348
1747633516.341851 11185
1747633516.356779 14651
1747633516.380394 23317
1747633516.392109 11423
1747633516.406835 14443
1747633516.430851 23734
1747633516.442314 11182
1747633516.466086 23440
1747633516.477534 11162
1747633516.492912 15088
1747633516.516639 23433
1747633516.528109 11188
1747633516.542704 14317
1747633516.566780 23801
1747633516.578241 11181
1747633516.602096 23539
1747633516.613554 11169
1747633516.637432 23596
1747633516.663421 11027
1747633516.687140 9714
1747633516.698602 11147
1747633516.713920 15041
1747633516.737802 23600
1747633516.749398 11290
1747633516.764722 14983
1747633516.788987 23978
1747633516.800485 11154
1747633516.815356 14586
1747633516.839710 24066
1747633516.851123 11138
1747633516.875031 23654
1747633516.886591 11274
1747633516.910821 23964
1747633516.922273 11159
1747633516.946083 23490
1747633516.957644 11269
1747633516.981094 23164
1747633516.992550 11172
1747633517.016460 23624
1747633517.028063 11142
1747633517.051046 22720
1747633517.063483 12165
1747633517.087297 23536
1747633517.098884 11317
1747633517.114168 15132
1747633517.146237 10693
1747633517.170180 9536
1747633517.181653 11191
1747633517.196272 14303
1747633517.219892 23332
1747633517.231325 11088
1747633517.255143 23530
1747633517.266584 11111
1747633517.290543 23672
1747633517.301977 11139
1747633517.316509 14247
1747633517.341122 24323
1747633517.352646 11179
1747633517.376624 23685
1747633517.388115 11202
1747633517.412352 23965
1747633517.423946 11285
1747633517.438523 14294
1747633517.462293 23440
1747633517.473770 11188
1747633517.500127 25969
1747633517.509850 9447
1747633517.533778 23667
1747633517.545240 11117
1747633517.569067 23531
1747633517.580567 11229
1747633517.604387 23517
1747633517.629927 11267
1747633517.653700 9609
1747633517.665200 11214
1747633517.689756 24279
1747633517.701281 11222
1747633517.725732 24183
1747633517.737220 11200
1747633517.760965 23470
1747633517.772451 11209
1747633517.796689 23963
1747633517.808299 11309
1747633517.832110 23526
1747633517.843595 11173
1747633517.858867 14989
1747633517.882577 23410
1747633517.894169 11298
1747633517.908795 14301
1747633517.932747 23677
1747633517.944235 11200
1747633517.958982 14461
1747633517.983075 23823
1747633517.994568 11204
1747633518.009245 14406
1747633518.033167 23611
1747633518.044631 11174
1747633518.059906 15009
1747633518.083477 23241
1747633518.109027 11228
1747633518.132705 9327
1747633518.144313 11314
1747633518.168796 24209
1747633518.180284 11127
1747633518.204055 23479
1747633518.215628 11277
1747633518.239444 23540
1747633518.250935 11200
1747633518.306030 9688
1747633518.317489 11201
1747633518.341240 23481
1747633518.352541 11028
1747633518.375889 22978
1747633518.387285 11077
1747633518.402220 14678
1747633518.426317 23774
1747633518.437677 11074
1747633518.452304 14323
1747633518.475368 22778
1747633518.487036 11396
1747633518.509619 22301
1747633518.523137 13173
1747633518.546686 23276
1747633518.558002 11023
1747633518.582000 23706
1747633518.593454 11142
1747633518.633604 10699
1747633518.656578 9088
1747633518.667905 11062
1747633518.691379 23176
1747633518.702700 11019
1747633518.717512 14516
1747633518.741051 23246
1747633518.752360 10963
1747633518.776446 23804
1747633518.787869 11139
1747633518.810975 22785
1747633518.822305 11064
1747633518.845785 23204
1747633518.857276 11198
1747633518.880703 23132
1747633518.893537 12469
1747633518.917038 23132
1747633518.928520 11197
1747633518.952923 24130
1747633518.964311 11099
1747633518.987819 23230
1747633518.999302 11167
1747633519.022586 22997
1747633519.033935 11050
1747633519.057738 23523
1747633519.069204 11160
1747633519.092655 23150
1747633519.127942 10726
1747633519.151543 9617
1747633519.162893 11047
1747633519.186596 23413
1747633519.198082 11202
1747633519.221976 23583
1747633519.233339 11078
1747633519.257126 23512
1747633519.268630 11211
1747633519.283059 14125
1747633519.306636 23221
1747633519.317970 11046
1747633519.342508 24254
1747633519.353862 11078
1747633519.377558 23423
1747633519.388940 11062
1747633519.412379 23091
1747633519.423762 11097
1747633519.447437 23329
1747633519.458823 11115
1747633519.482295 23156
1747633519.493647 11063
1747633519.508243 14295
1747633519.531878 23353
1747633519.543414 11217
1747633519.558071 14409
1747633519.580730 22395
1747633519.623575 10970
1747633519.646869 9350
1747633519.658273 11153
1747633519.681943 23395
1747633519.693356 11126
1747633519.716635 22998
1747633519.728018 11103
1747633519.751372 23074
1747633519.764982 13282
1747633519.780211 14924
1747633519.804306 23814
1747633519.815833 11241
1747633519.839417 23286
1747633519.850850 11136
1747633519.874758 23610
1747633519.886293 11226
1747633519.910252 23672
1747633519.921696 11167
1747633519.945515 23540
1747633519.957028 11233
1747633519.981377 24082
1747633519.992808 11145
1747633520.016090 23009
1747633520.027643 11233
1747633520.051846 23908
1747633520.063251 11088
1747633520.087088 23521
1747633520.123304 10877
1747633520.146535 9533
1747633520.158025 11229
1747633520.172637 14368
1747633520.196129 23207
1747633520.207519 11089
1747633520.231821 23978
1747633520.243231 11126
1747633520.257958 14448
1747633520.282186 23950
1747633520.293645 11161
1747633520.308763 14842
1747633520.332195 23128
1747633520.343563 11079
1747633520.358146 14298
1747633520.382247 23816
1747633520.393650 11033
1747633520.417877 23947
1747633520.429384 11204
1747633520.452844 23186
1747633520.464208 11093
1747633520.488558 24089
1747633520.500109 11265
1747633520.523418 23027
1747633520.534806 11095
1747633520.558239 23155
1747633520.569737 11096
1747633520.615261 10933
1747633520.638696 9447
1747633520.650082 11122
1747633520.664664 14315
1747633520.688180 23261
1747633520.699674 11207
1747633520.715298 15328
1747633520.739240 23657
1747633520.750647 11111
1747633520.765224 14282
1747633520.789064 23593
1747633520.800439 11100
1747633520.815635 14912
1747633520.839505 23584
1747633520.850943 11138
1747633520.874639 23392
1747633520.886066 11143
1747633520.910314 23976
1747633520.921764 11149
1747633520.945485 23432
1747633520.956912 11136
1747633520.971652 14420
1747633520.995375 23448
1747633521.006778 11089
1747633521.021256 14226
1747633521.044791 23245
1747633521.056213 11127
1747633521.100603 10927
1747633521.124176 9151
1747633521.136431 12062
1747633521.160728 24015
1747633521.172316 11295
1747633521.196411 23810
1747633521.207890 11180
1747633521.231403 23217
1747633521.242970 11267
1747633521.267059 23748
1747633521.278520 11167
1747633521.302179 23384
1747633521.313729 11249
1747633521.337616 23571
1747633521.349101 11160
1747633521.363808 14415
1747633521.387993 23900
1747633521.399565 11271
1747633521.423908 24078
1747633521.435379 11167
1747633521.459178 23485
1747633521.470642 11186
1747633521.494129 23198
1747633521.505600 11195
1747633521.529175 23296
1747633521.541267 11754
1747633521.564977 23433
1747633521.602489 10972
1747633521.625785 9343
1747633521.637235 11139
1747633521.661606 24080
1747633521.673190 11394
1747633521.688397 14956
1747633521.712674 24022
1747633521.724202 11224
1747633521.747938 23476
1747633521.759388 11170
1747633521.773966 14297
1747633521.797828 23549
1747633521.809287 11169
1747633521.832845 23288
1747633521.844646 11531
1747633521.859353 14438
1747633521.883571 23924
1747633521.895025 11172
1747633521.919013 23729
1747633521.930587 11295
1747633521.945924 15074
1747633521.969815 23637
1747633521.981273 11177
1747633522.005026 23467
1747633522.016816 11516
1747633522.040223 23132
1747633522.051716 11216
1747633522.094560 10772
1747633522.108296 407
1747633522.123305 14795
1747633522.138592 15000
1747633522.162186 23317
1747633522.173639 11165
1747633522.198372 24446
1747633522.209855 11158
1747633522.225162 15012
1747633522.249647 24205
1747633522.261175 11249
1747633522.285410 23926
1747633522.297346 11640
1747633522.321903 24269
1747633522.333448 11239
1747633522.357270 23525
1747633522.368748 11107
1747633522.392689 23671
1747633522.404270 11272
1747633522.428186 23574
1747633522.439658 11198
1747633522.463266 23342
1747633522.474858 11304
1747633522.497136 22015
1747633522.510386 12928
1747633522.525102 14430
1747633522.549106 23721
1747633522.574359 11028
1747633522.597715 9344
1747633522.609180 11187
1747633522.633068 23611
1747633522.644534 11181
1747633522.668628 23817
1747633522.680108 11185
1747633522.703395 23022
1747633522.715462 11784
1747633522.739351 23550
1747633522.750828 11171
1747633522.774088 22966
1747633522.786370 11990
1747633522.801277 14643
1747633522.825016 23384
1747633522.836536 11231
1747633522.860294 23462
1747633522.871856 11271
1747633522.896878 24681
1747633522.907830 10649
1747633522.931443 23303
1747633522.943005 11283
1747633522.966813 23528
1747633522.978293 11165
1747633522.993016 14432
1747633523.017312 24031
1747633523.028745 11120
1747633523.059025 11061
1747633523.082708 9418
1747633523.135434 9652
1747633523.146776 11069
1747633523.170313 23265
1747633523.181602 10988
1747633523.205881 24097
1747633523.217218 11077
1747633523.240743 23197
1747633523.253925 12832
1747633523.268620 14369
1747633523.291969 23081
1747633523.303393 11126
1747633523.327691 23999
1747633523.338975 11002
1747633523.362918 23669
1747633523.374378 11194
1747633523.398005 23325
1747633523.409353 11083
1747633523.432974 23335
1747633523.444384 11138
1747633523.468120 23457
1747633523.479411 10980
1747633523.502500 22807
1747633523.514803 11971
1747633523.538961 23863
1747633523.550290 11044
1747633523.594596 10883
1747633523.617869 9261
1747633523.629181 11059
1747633523.644604 15139
1747633523.669003 24098
1747633523.680355 11067
1747633523.694780 14157
1747633523.718307 23253
1747633523.729697 11077
1747633523.753557 23567
1747633523.764924 11098
1747633523.788530 23341
1747633523.799896 11078
1747633523.814345 14152
1747633523.837714 23065
1747633523.849142 11127
1747633523.872759 23329
1747633523.884109 11078
1747633523.898609 14242
1747633523.922597 23715
1747633523.933934 11031
1747633523.957278 23081
1747633523.968620 11017
1747633523.992763 23863
1747633524.004153 11118
1747633524.027742 23308
1747633524.039137 11107
1747633524.083161 10800
1747633524.096609 380
1747633524.111703 14914
1747633524.135069 23085
1747633524.146546 11187
1747633524.169900 23090
1747633524.181486 11296
1747633524.204942 23122
1747633524.216299 11071
1747633524.230883 14311
1747633524.254753 23551
1747633524.266175 11134
1747633524.289930 23483
1747633524.301324 11136
1747633524.324978 23376
1747633524.336401 11150
1747633524.351597 14914
1747633524.375145 23240
1747633524.386660 11191
1747633524.410495 23560
1747633524.421907 11097
1747633524.445475 23293
1747633524.457002 11248
1747633524.480949 23691
1747633524.492357 11112
1747633524.516116 23473
1747633524.527623 11201
1747633524.559097 11097
1747633524.582608 9326
1747633524.593969 11040
1747633524.617429 23148
1747633524.628794 11082
1747633524.643422 14350
1747633524.667038 23347
1747633524.679148 11775
1747633524.694432 14978
1747633524.718383 23624
1747633524.729819 11175
1747633524.754154 24075
1747633524.765581 11111
1747633524.789358 23462
1747633524.800801 11162
1747633524.825371 24307
1747633524.836756 11075
1747633524.860302 23241
1747633524.871701 11102
1747633524.886507 14542
1747633524.909957 23154
1747633524.921501 11247
1747633524.945511 23744
1747633524.956936 11124
1747633524.971511 14283
1747633524.995484 23681
1747633525.006880 11100
1747633525.038432 10494
1747633525.062267 9316
1747633525.073658 11109
1747633525.089161 15188
1747633525.113135 23659
1747633525.124568 11142
1747633525.147994 23146
1747633525.159416 11116
1747633525.174078 14367
1747633525.198346 23987
1747633525.209759 11133
1747633525.233627 23575
1747633525.245163 11248
1747633525.269445 24016
1747633525.280872 11149
1747633525.295441 14290
1747633525.320078 24330
1747633525.331501 11128
1747633525.346691 14904
1747633525.369865 22894
1747633525.381440 11279
1747633525.396662 14899
1747633525.420985 24045
1747633525.432426 11172
1747633525.456345 23640
1747633525.467816 11185
1747633525.482495 14360
1747633525.514438 11008
1747633525.538134 9494
1747633525.549601 11186
1747633525.573535 23543
1747633525.585006 11185
1747633525.599678 14397
1747633525.623332 23374
1747633525.634850 11213
1747633525.658779 23648
1747633525.670241 11156
1747633525.685434 14896
1747633525.709244 23510
1747633525.720687 11172
1747633525.743634 22672
1747633525.755778 11895
1747633525.780443 24370
1747633525.791872 11108
1747633525.815470 23312
1747633525.826878 11123
1747633525.841633 14452
1747633525.865081 23166
1747633525.876527 11158
1747633525.900342 23478
1747633525.911881 11252
1747633525.935884 23609
1747633525.947411 11228
1747633525.971311 23601
1747633525.996741 11254
1747633526.020530 9519
1747633526.032091 11273
1747633526.056272 23895
1747633526.067739 11189
1747633526.083024 15011
1747633526.107638 24330
1747633526.119101 11160
1747633526.134389 15001
1747633526.158048 23374
1747633526.169669 11283
1747633526.193701 23743
1747633526.205107 11021
1747633526.219748 14366
1747633526.244146 24128
1747633526.255582 11181
1747633526.270125 14284
1747633526.293901 23468
1747633526.305535 11335
1747633526.320576 14722
1747633526.344645 23740
1747633526.356157 11185
1747633526.380549 24096
1747633526.392046 11206
1747633526.407578 15238
1747633526.430631 22756
1747633526.443324 12386
1747633526.476028 10937
1747633526.499881 9754
1747633526.511417 11199
1747633526.535222 23482
1747633526.547397 11877
1747633526.563163 15383
1747633526.587069 23637
1747633526.598508 11147
1747633526.613308 14517
1747633526.636972 23372
1747633526.648411 11102
1747633526.672573 23869
1747633526.684053 11179
1747633526.699493 15151
1747633526.723769 23991
1747633526.735206 11161
1747633526.750626 15134
1747633526.774869 23948
1747633526.786320 11161
1747633526.801551 14953
1747633526.825594 23775
1747633526.837132 11234
1747633526.852886 15480
1747633526.878060 24871
1747633526.889558 11189
1747633526.904352 14470
1747633526.928367 23714
1747633526.953768 11208
1747633526.977621 9694
1747633526.989106 11194
1747633527.013091 23695
1747633527.024717 11341
1747633527.039271 14286
1747633527.063314 23769
1747633527.074808 11181
1747633527.098718 23630
1747633527.110179 11178
1747633527.133503 23052
1747633527.144945 11111
1747633527.168915 23692
1747633527.180364 11128
1747633527.204126 23453
1747633527.215589 11163
1747633527.239829 23960
1747633527.251271 11159
1747633527.275273 23729
1747633527.286730 11157
1747633527.302015 14975
1747633527.326647 24289
1747633527.338084 11149
1747633527.362723 24361
1747633527.374265 11245
1747633527.399132 24571
1747633527.410574 11112
1747633527.443028 11022
1747633527.466942 9737
1747633527.478351 11111
1747633527.493008 14362
1747633527.517542 24180
1747633527.528978 11158
1747633527.552718 23467
1747633527.564266 11237
1747633527.588754 24159
1747633527.600196 11151
1747633527.614935 14438
1747633527.638758 23512
1747633527.650233 11188
1747633527.674004 23469
1747633527.685494 11204
1747633527.700442 14653
1747633527.724535 23821
1747633527.735982 11144
1747633527.751194 14915
1747633527.775543 24189
1747633527.787001 11185
1747633527.802291 14995
1747633527.826279 23692
1747633527.837866 11294
1747633527.852439 14260
1747633527.875919 23213
1747633527.927792 9680
1747633527.965215 10671
1747633527.988259 9224
1747633527.999659 11093
1747633528.014069 14119
1747633528.037578 23242
1747633528.049573 11719
1747633528.073290 23381
1747633528.084573 11011
1747633528.107847 22968
1747633528.119142 10998
1747633528.142762 23358
1747633528.154049 10990
1747633528.180127 25729
1747633528.188783 8359
1747633528.204130 15067
1747633528.227744 23319
1747633528.239026 10978
1747633528.262668 23364
1747633528.274119 11134
1747633528.289268 14801
1747633528.313466 23897
1747633528.324767 11011
1747633528.348943 23895
1747633528.360282 11061
1747633528.384777 24179
1747633528.396170 11091
1747633528.420630 24153
1747633528.458029 10785
1747633528.481273 9313
1747633528.492621 11063
1747633528.507196 14313
1747633528.531543 24021
1747633528.543054 11212
1747633528.567535 24178
1747633528.578911 11108
1747633528.593472 14286
1747633528.617081 23347
1747633528.628414 11056
1747633528.643706 14994
1747633528.667106 23066
1747633528.678550 11148
1747633528.703207 24341
1747633528.714593 11092
1747633528.729133 14239
1747633528.752792 23371
1747633528.764207 11126
1747633528.788328 23853
1747633528.799719 11102
1747633528.823095 23098
1747633528.834491 11080
1747633528.858348 23578
1747633528.869762 11119
1747633528.894320 24233
1747633528.905719 11101
1747633528.949363 10362
1747633528.973613 9635
1747633528.985012 11137
1747633528.999589 14285
1747633529.024251 24367
1747633529.035614 11082
1747633529.050655 14759
1747633529.074649 23712
1747633529.086192 11250
1747633529.101226 14708
1747633529.125631 24019
1747633529.136995 11086
1747633529.152402 15135
1747633529.177081 24379
1747633529.188512 11089
1747633529.203784 14997
1747633529.227666 23537
1747633529.239086 11143
1747633529.263492 24036
1747633529.274882 11187
1747633529.298810 23777
1747633529.310212 11117
1747633529.324862 14369
1747633529.349164 24004
1747633529.360675 11209
1747633529.375196 14227
1747633529.399445 23945
1747633529.437215 11073
1747633529.460682 9306
1747633529.472108 11134
1747633529.486757 14319
1747633529.510043 23008
1747633529.521430 11080
1747633529.544915 23206
1747633529.556425 11218
1747633529.571048 14283
1747633529.594781 23457
1747633529.606180 11106
1747633529.629669 23178
1747633529.641110 11148
1747633529.655677 14287
1747633529.679338 23379
1747633529.690823 11181
1747633529.715499 24416
1747633529.726896 11116
1747633529.750522 23340
1747633529.762037 11213
1747633529.786544 24216
1747633529.798003 11170
1747633529.813219 14944
1747633529.838170 24662
1747633529.849612 11118
1747633529.873157 23245
1747633529.884883 11348
1747633529.929419 10916
1747633529.952773 9335
1747633529.964289 11239
1747633529.988438 23867
1747633529.999818 11076

There were only 3 calls that took under 1 ms, about 50 that were a bit under 10 ms, about 650 between 10 and 20 ms, and about 60 between 20 and 26 ms. Of course the calls will take a different amount of time depending on WU and GPU, but it looks like there is not a significant fluctuation.

And here is what about 3 seconds looks like for a core23 project on an RTX 4090 Mobile (which is just a 4080 Super with 512 fewer CUDA cores and a maximum power use of 150 watts):

Code: Select all

1747744050.000385 2360
1747744050.008329 2369
1747744050.014205 35
1747744050.049558 1625
1747744050.051570 1931
1747744050.055284 3645
1747744050.057339 1988
1747744050.061048 3643
1747744050.062938 1825
1747744050.066269 3267
1747744050.068521 2188
1747744050.072195 3614
1747744050.074223 1967
1747744050.077951 3662
1747744050.079984 1966
1747744050.084078 4030
1747744050.086420 2277
1747744050.090603 4120
1747744050.092912 2244
1747744050.097072 4097
1747744050.099184 2048
1747744050.103376 4128
1747744050.105696 2253
1747744050.109855 4094
1747744050.111984 2066
1747744050.116119 4072
1747744050.118296 2106
1747744050.122494 4130
1747744050.138832 1653
1747744050.145798 1505
1747744050.148025 2136
1747744050.151949 3848
1747744050.154214 2195
1747744050.158156 3877
1747744050.160391 2170
1747744050.164166 3713
1747744050.166311 2082
1747744050.170028 3655
1747744050.172162 2064
1747744050.175957 3726
1747744050.178046 2022
1747744050.182049 3939
1747744050.184381 2267
1747744050.188511 4066
1747744050.190905 2329
1747744050.195042 4074
1747744050.197383 2270
1747744050.201584 4136
1747744050.203910 2260
1747744050.208129 4155
1747744050.210424 2229
1747744050.214634 4145
1747744050.216975 2228
1747744050.221146 4105
1747744050.223503 2282
1747744050.231426 2222
1747744050.238785 1610
1747744050.241150 2289
1747744050.245263 4041
1747744050.247323 1988
1747744050.251370 3981
1747744050.253580 2146
1747744050.257695 4050
1747744050.259758 2000
1747744050.263769 3948
1747744050.265807 1974
1747744050.269888 4014
1747744050.271919 1965
1747744050.275973 3990
1747744050.278010 1973
1747744050.282096 4021
1747744050.284412 2250
1747744050.288621 4147
1747744050.290982 2294
1747744050.295293 4245
1747744050.297436 2081
1747744050.301689 4189
1747744050.303849 2095
1747744050.308087 4173
1747744050.310484 2327
1747744050.314737 4190
1747744050.316897 2086
1747744050.323190 610
1747744050.328773 40
1747744050.333185 4338
1747744050.335559 2307
1747744050.339901 4278
1747744050.342171 2202
1747744050.346121 3884
1747744050.348136 1949
1747744050.352124 3922
1747744050.354270 2080
1747744050.358209 3873
1747744050.360270 1998
1747744050.364545 4211
1747744050.366854 2246
1747744050.371106 4189
1747744050.373422 2246
1747744050.377703 4218
1747744050.379987 2222
1747744050.384330 4280
1747744050.386663 2269
1747744050.390923 4193
1747744050.393241 2251
1747744050.397564 4258
1747744050.399742 2113
1747744050.404118 4313
1747744050.406499 2309
1747744050.410151 3583
1747744050.416480 644
1747744050.421514 57
1747744050.424198 2613
1747744050.427989 3727
1747744050.430546 2493
1747744050.434814 4203
1747744050.437166 2287
1747744050.441557 4324
1747744050.443931 2310
1747744050.447935 3939
1747744050.449999 2001
1747744050.454052 3989
1747744050.456188 2072
1747744050.460246 3994
1747744050.462563 2250
1747744050.466918 4294
1747744050.469267 2285
1747744050.473683 4351
1747744050.476041 2292
1747744050.480364 4259
1747744050.482558 2130
1747744050.486907 4278
1747744050.489236 2261
1747744050.493493 4195
1747744050.495826 2270
1747744050.500173 4283
1747744050.502367 2127
1747744050.508635 612
1747744050.514173 42
1747744050.518512 4270
1747744050.520764 2190
1747744050.525232 4405
1747744050.527515 2221
1747744050.531514 3937
1747744050.533619 2041
1747744050.537742 4057
1747744050.539961 2123
1747744050.544242 4220
1747744050.546602 2298
1747744050.550985 4318
1747744050.553184 2137
1747744050.557566 4320
1747744050.559927 2293
1747744050.564215 4224
1747744050.566569 2286
1747744050.570892 4260
1747744050.573233 2277
1747744050.577564 4265
1747744050.579765 2137
1747744050.584196 4360
1747744050.586597 2335
1747744050.591060 4395
1747744050.593465 2341
1747744050.597865 4332
1747744050.603758 649
1747744050.609675 41
1747744050.612369 2621
1747744050.616739 4303
1747744050.618972 2171
1747744050.623179 4144
1747744050.625370 2129
1747744050.629494 4061
1747744050.631666 2103
1747744050.635706 3975
1747744050.637761 1991
1747744050.641824 3998
1747744050.644050 2163
1747744050.648407 4294
1747744050.650755 2284
1747744050.655046 4226
1747744050.657396 2284
1747744050.661680 4219
1747744050.664005 2261
1747744050.668360 4290
1747744050.670700 2278
1747744050.675016 4253
1747744050.677341 2261
1747744050.681676 4266
1747744050.683897 2156
1747744050.688285 4325
1747744050.690464 2112
1747744050.696836 627
1747744050.702973 30
1747744050.707426 4370
1747744050.709777 2285
1747744050.714177 4336
1747744050.716379 2137
1747744050.720798 4354
1747744050.722958 2097
1747744050.726971 3949
1747744050.729133 2096
1747744050.733133 3926
1747744050.735375 2170
1747744050.739406 3967
1747744050.741549 2079
1747744050.745757 4143
1747744050.748095 2276
1747744050.752395 4233
1747744050.754605 2145
1747744050.758920 4248
1747744050.761318 2335
1747744050.765612 4225
1747744050.768001 2325
1747744050.772370 4304
1747744050.774559 2123
1747744050.778853 4227
1747744050.781259 2342
1747744050.785768 4439
1747744050.792490 645
1747744050.798493 43
1747744050.801187 2615
1747744050.805520 4268
1747744050.807837 2253
1747744050.812147 4244
1747744050.814471 2260
1747744050.818786 4251
1747744050.821103 2254
1747744050.825252 4072
1747744050.827473 2156
1747744050.831549 4012
1747744050.833797 2184
1747744050.837968 4108
1747744050.840168 2138
1747744050.844346 4114
1747744050.846709 2291
1747744050.851045 4269
1747744050.853384 2277
1747744050.857718 4270
1747744050.860042 2260
1747744050.864383 4246
1747744050.866723 2277
1747744050.871023 4237
1747744050.873375 2286
1747744050.877760 4319
1747744050.880068 2241
1747744050.887680 2051
1747744050.895623 1771
1747744050.898015 2318
1747744050.902119 4041
1747744050.904670 2479
1747744050.909099 4366
1747744050.911458 2275
1747744050.915766 4245
1747744050.918054 2223
1747744050.922194 4075
1747744050.924432 2174
1747744050.928656 4161
1747744050.931066 2345
1747744050.935375 4245
1747744050.937715 2276
1747744050.942033 4253
1747744050.944288 2188
1747744050.948751 4399
1747744050.951076 2261
1747744050.955532 4393
1747744050.957876 2278
1747744050.962323 4384
1747744050.964713 2327
1747744050.969213 4432
1747744050.971654 2375
1747744050.976099 4381
1747744050.978360 2191
1747744050.986618 2405
1747744050.994533 1781
1747744051.029818 1605
1747744051.031866 1963
1747744051.035544 3612
1747744051.037573 1964
1747744051.041271 3634
1747744051.043302 1957
1747744051.046898 3529
1747744051.048945 1984
1747744051.052604 3596
1747744051.054636 1966
1747744051.058254 3554
1747744051.060159 1843
1747744051.064205 3985
1747744051.066573 2303
1747744051.070709 4070
1747744051.072968 2196
1747744051.077046 4014
1747744051.079348 2239
1747744051.083499 4090
1747744051.085854 2294
1747744051.090086 4168
1747744051.092472 2316
1747744051.096698 4162
1747744051.099055 2293
1747744051.103342 4218
1747744051.121704 1684
1747744051.129403 1537
1747744051.131645 2147
1747744051.135684 3974
1747744051.137873 2124
1747744051.141884 3938
1747744051.144035 2082
1747744051.147769 3669
1747744051.149808 1975
1747744051.153482 3612
1747744051.155597 2049
1747744051.159354 3693
1747744051.161375 1957
1747744051.165489 4049
1747744051.167811 2255
1747744051.171997 4121
1747744051.174273 2212
1747744051.178434 4096
1747744051.180727 2227
1747744051.184954 4163
1747744051.187113 2091
1747744051.191382 4191
1747744051.193701 2252
1747744051.197968 4202
1747744051.200283 2251
1747744051.204484 4136
1747744051.206870 2315
1747744051.213349 380
1747744051.218426 38
1747744051.222579 4083
1747744051.224913 2268
1747744051.229081 4104
1747744051.231365 2214
1747744051.235582 4153
1747744051.237834 2130
1747744051.241943 4046
1747744051.244140 2131
1747744051.248139 3930
1747744051.250323 2111
1747744051.253742 3350
1747744051.256290 2482
1747744051.259805 3451
1747744051.262213 2342
1747744051.265704 3426
1747744051.268295 2526
1747744051.272127 3767
1747744051.274462 2268
1747744051.278195 3669
1747744051.280662 2403
1747744051.284578 3851
1747744051.287039 2389
1747744051.291020 3916
1747744051.293277 2191
1747744051.297124 3778
1747744051.303898 656
1747744051.309235 37
1747744051.311906 2597
1747744051.315565 3595
1747744051.318123 2491
1747744051.321783 3590
1747744051.324180 2334
1747744051.327610 3368
1747744051.330003 2327
1747744051.333479 3410
1747744051.335688 2146
1747744051.338988 3235
1747744051.341394 2341
1747744051.345177 3720
1747744051.347927 2686
1747744051.351759 3769
1747744051.354498 2674
1747744051.358225 3661
1747744051.360860 2570
1747744051.364511 3586
1747744051.367135 2559
1747744051.371272 4067
1747744051.373688 2351
1747744051.377476 3725
1747744051.380062 2519
1747744051.383798 3670
1747744051.386517 2652
1747744051.392745 691
1747744051.398166 56
1747744051.402263 4026
1747744051.405019 2683
1747744051.409143 4060
1747744051.411520 2310
1747744051.415244 3662
1747744051.417906 2598
1747744051.421608 3639
1747744051.424120 2450
1747744051.427790 3606
1747744051.430323 2465
1747744051.434420 4034
1747744051.436744 2260
1747744051.440871 4063
1747744051.443046 2110
1747744051.447339 4229
1747744051.449540 2139
1747744051.453819 4214
1747744051.456221 2336
1747744051.460578 4293
1747744051.462764 2123
1747744051.467104 4278
1747744051.469335 2159
1747744051.473178 3779
1747744051.475808 2564
1747744051.479416 3533
1747744051.485624 661
1747744051.491378 48
1747744051.494083 2632
1747744051.498311 4158
1747744051.500684 2310
1747744051.505074 4326
1747744051.507510 2370
1747744051.511355 3783
1747744051.514121 2704
1747744051.517706 3516
1747744051.520178 2409
1747744051.523901 3662
1747744051.526176 2212
1747744051.530508 4270
1747744051.532913 2342
1747744051.537212 4238
1747744051.539594 2320
1747744051.543900 4244
1747744051.546300 2337
1747744051.550627 4212
1747744051.552992 2302
1747744051.557314 4260
1747744051.559699 2320
1747744051.564015 4254
1747744051.566397 2320
1747744051.570719 4259
1747744051.573114 2324
1747744051.581112 2323
1747744051.588009 1786
1747744051.590399 2318
1747744051.594823 4359
1747744051.597166 2276
1747744051.601587 4351
1747744051.603872 2219
1747744051.607942 4006
1747744051.610152 2146
1747744051.614273 4058
1747744051.616334 1999
1747744051.620358 3960
1747744051.622819 2391
1747744051.627190 4304
1747744051.629572 2318
1747744051.633928 4293
1747744051.636284 2293
1747744051.640688 4337
1747744051.643053 2303
1747744051.647338 4220
1747744051.649589 2178
1747744051.653933 4277
1747744051.656307 2311
1747744051.660662 4291
1747744051.663084 2360
1747744051.667562 4410
1747744051.669948 2316
1747744051.677973 2399
1747744051.683891 41
1747744051.688297 4333
1747744051.690697 2334
1747744051.695137 4372
1747744051.697558 2354
1747744051.701908 4286
1747744051.704322 2350
1747744051.708717 4329
1747744051.711092 2310
1747744051.715391 4233
1747744051.717846 2378
1747744051.722138 4224
1747744051.724249 2040
1747744051.728655 4342
1747744051.731029 2311
1747744051.735506 4412
1747744051.737887 2317
1747744051.741877 3926
1747744051.744448 2507
1747744051.748875 4362
1747744051.751265 2326
1747744051.755629 4300
1747744051.758019 2327
1747744051.762444 4360
1747744051.764703 2195
1747744051.769135 4365
1747744051.775751 620
1747744051.781559 43
1747744051.784184 2555
1747744051.788562 4314
1747744051.790923 2292
1747744051.795252 4260
1747744051.797593 2276
1747744051.801985 4327
1747744051.804202 2154
1747744051.807999 3735
1747744051.810150 2088
1747744051.814226 4011
1747744051.816388 2096
1747744051.820402 3949
1747744051.822783 2318
1747744051.827230 4382
1747744051.829612 2318
1747744051.834026 4351
1747744051.836393 2303
1747744051.840778 4318
1747744051.843160 2318
1747744051.847508 4283
1747744051.849865 2294
1747744051.854293 4363
1747744051.856568 2213
1747744051.860942 4310
1747744051.863320 2308
1747744051.871404 2357
1747744051.878682 1775
1747744051.880927 2173
1747744051.885147 4158
1747744051.887260 2009
1747744051.891455 4130
1747744051.893772 2252
1747744051.897965 4127
1747744051.900240 2210
1747744051.904416 4112
1747744051.906808 2326
1747744051.911087 4215
1747744051.913372 2212
1747744051.917558 4122
1747744051.919869 2245
1747744051.924038 4105
1747744051.926386 2285
1747744051.930725 4275
1747744051.933102 2312
1747744051.937412 4231
1747744051.939737 2261
1747744051.944045 4244
1747744051.946420 2310
1747744051.950744 4260
1747744051.953093 2286
1747744051.957416 4252
1747744051.959787 2299
1747744051.965958 589
1747744051.971517 44
1747744052.007944 1404
1747744052.010017 1995
1747744052.013351 3266
1747744052.015539 2121
1747744052.019195 3592
1747744052.021248 1989
1747744052.024859 3547
1747744052.026801 1877
1747744052.030484 3620
1747744052.032498 1949
1747744052.036165 3604
1747744052.038132 1902
1747744052.041845 3644
1747744052.043956 2047
1747744052.048148 4129
1747744052.050344 2130
1747744052.054539 4132
1747744052.056864 2259
1747744052.061065 4137
1747744052.063448 2318
1747744052.067715 4206
1747744052.070080 2303
1747744052.074355 4210
1747744052.076585 2166
1747744052.080798 4147
1747744052.097419 1671
1747744052.104244 1704
1747744052.106647 2314
1747744052.111041 4325
1747744052.113427 2317
1747744052.117604 4111
1747744052.119863 2195
1747744052.123842 3914
1747744052.126079 2171
1747744052.129971 3824
1747744052.132119 2079
1747744052.135999 3806
1747744052.138121 2055
1747744052.141956 3762
1747744052.143984 1965
1747744052.148210 4161
1747744052.150569 2293
1747744052.154748 4117
1747744052.156894 2083
1747744052.161038 4081
1747744052.163479 2377
1747744052.167662 4121
1747744052.169807 2081
1747744052.174073 4202
1747744052.176272 2133
1747744052.180542 4205
1747744052.182896 2282
1747744052.189213 614
1747744052.194751 31
1747744052.199074 4251
1747744052.201477 2337
1747744052.205624 4078
1747744052.207695 2007
1747744052.211839 4082
1747744052.213888 1985
1747744052.217960 4005
1747744052.220173 2150
1747744052.224235 4000
1747744052.226308 2008
1747744052.230286 3916
1747744052.232332 1982
1747744052.236418 4022
1747744052.238668 2184
1747744052.242900 4135
1747744052.245208 2244
1747744052.249409 4108
1747744052.251823 2342
1747744052.256000 4112
1747744052.258350 2285
1747744052.262567 4153
1747744052.264962 2329
1747744052.269171 4146
1747744052.271494 2260
1747744052.275706 4143
1747744052.283448 2224
1747744052.290448 1677
1747744052.292648 2125
1747744052.296900 4188
1747744052.299076 2098
1747744052.303296 4149
1747744052.305623 2258
1747744052.309817 4129
1747744052.312110 2230
1747744052.316256 4076
1747744052.318301 1979
1747744052.322237 3873
1747744052.324375 2072
1747744052.328624 4185
1747744052.330962 2272
1747744052.335214 4188
1747744052.337553 2275
1747744052.341820 4205
1747744052.344188 2304
1747744052.348457 4206
1747744052.350750 2227
1747744052.355064 4249
1747744052.357380 2252
1747744052.361623 4180
1747744052.364037 2352
1747744052.368362 4264
1747744052.370758 2326
1747744052.378934 2306
1747744052.384404 41
1747744052.388734 4258
1747744052.391077 2279
1747744052.395385 4247
1747744052.397764 2310
1747744052.402129 4303
1747744052.404299 2097
1747744052.408335 3972
1747744052.410522 2121
1747744052.414491 3907
1747744052.416658 2105
1747744052.420674 3951
1747744052.422893 2152
1747744052.427199 4242
1747744052.429517 2254
1747744052.433767 4186
1747744052.436116 2286
1747744052.440400 4213
1747744052.442596 2134
1747744052.446955 4296
1747744052.449348 2327
1747744052.453737 4327
1747744052.456094 2294
1747744052.460475 4310
1747744052.462850 2312
1747744052.467181 4266
1747744052.473722 610
1747744052.479474 30
1747744052.482064 2512
1747744052.486290 4162
1747744052.488617 2262
1747744052.492842 4163
1747744052.495144 2235
1747744052.499383 4170
1747744052.501678 2229
1747744052.505837 4095
1747744052.508252 2351
1747744052.512469 4153
1747744052.514804 2269
1747744052.518777 3906
1747744052.520996 2155
1747744052.524991 3931
1747744052.527333 2277
1747744052.531654 4258
1747744052.533977 2260
1747744052.538211 4170
1747744052.540433 2157
1747744052.544755 4258
1747744052.547087 2269
1747744052.551426 4269
1747744052.553796 2305
1747744052.557988 4129
1747744052.560427 2372
1747744052.568893 2255
1747744052.574773 41
1747744052.579188 4303
1747744052.581566 2313
1747744052.585707 4078
1747744052.587983 2214
1747744052.592151 4104
1747744052.594320 2100
1747744052.598406 4024
1747744052.600737 2269
1747744052.604883 4085
1747744052.607144 2197
1747744052.611200 3989
1747744052.613417 2154
1747744052.617472 3991
1747744052.619773 2238
1747744052.624012 4177
1747744052.626206 2129
1747744052.630505 4235
1747744052.632888 2320
1747744052.637211 4261
1747744052.639565 2277
1747744052.643931 4302
1747744052.646301 2304
1747744052.650615 4250
1747744052.652925 2244
1747744052.657133 4141
1747744052.664024 459
1747744052.669625 42
1747744052.672329 2627
1747744052.676718 4325
1747744052.679098 2318
1747744052.683208 4041
1747744052.685382 2109
1747744052.689338 3891
1747744052.691376 1973
1747744052.695395 3956
1747744052.697589 2130
1747744052.701617 3963
1747744052.703773 2089
1747744052.707802 3968
1747744052.709982 2114
1747744052.713988 3934
1747744052.716231 2179
1747744052.720635 4340
1747744052.722897 2196
1747744052.727144 4183
1747744052.729489 2277
1747744052.733655 4105
1747744052.736118 2393
1747744052.740367 4177
1747744052.742725 2294
1747744052.747132 4344
1747744052.749426 2217
1747744052.757578 2049
1747744052.762772 31
1747744052.767172 4327
1747744052.769538 2302
1747744052.773950 4349
1747744052.776352 2336
1747744052.780724 4308
1747744052.782919 2130
1747744052.787025 4035
1747744052.789269 2178
1747744052.793339 4006
1747744052.795558 2155
1747744052.799644 4021
1747744052.801882 2172
1747744052.806216 4271
1747744052.808559 2274
1747744052.812859 4236
1747744052.815261 2336
1747744052.819559 4235
1747744052.821758 2136
1747744052.826040 4219
1747744052.828459 2354
1747744052.832768 4243
1747744052.835143 2311
1747744052.839547 4332
1747744052.841799 2187
1747744052.846237 4372
1747744052.853923 2021
1747744052.861602 1743
1747744052.863922 2245
1747744052.868236 4251
1747744052.870572 2270
1747744052.874878 4245
1747744052.877219 2279
1747744052.881463 4178
1747744052.883726 2196
1747744052.887837 4047
1747744052.890079 2178
1747744052.894189 4043
1747744052.896469 2215
1747744052.900636 4105
1747744052.902881 2136
1747744052.907066 4121

That's 800 synchronization calls in under 3 seconds. The calls last on average between 1 and 4 ms, but some are as short as 30 us.

For reference here is the code I used to get that information:

Code: Select all

#include <stdio.h>
#include <dlfcn.h>
#include <assert.h>
#include <cuda.h>
#include <sys/time.h>

typedef CUresult (*cuEventSynchronize_type)(CUevent);
cuEventSynchronize_type orig_cuEventSynchronize;

static __attribute__((constructor)) void hook_function_early(void)
{
        void *handle;

        handle = dlopen("libcuda.so.1", RTLD_NOW);
        assert(handle);

        orig_cuEventSynchronize = (cuEventSynchronize_type)dlsym(handle, "cuEventSynchronize");
        assert(orig_cuEventSynchronize);
}

extern CUresult cuEventSynchronize(CUevent hEvent)
{
        CUresult result;
        long before, after;
        struct timeval tv;

        gettimeofday(&tv, NULL);
        before = tv.tv_sec * 1000000 + tv.tv_usec;

        result = orig_cuEventSynchronize(hEvent);

        gettimeofday(&tv, NULL);
        after = tv.tv_sec * 1000000 + tv.tv_usec;

        printf("%s %d %ld.%06ld %ld\n", __func__, result, tv.tv_sec, tv.tv_usec, after - before);
        return result;
}

I could write a different hook that samples the duration for a few thousand calls (storing previous values in a static variable) and then pick the lowest, or one of the lowest, and sleeps for that duration before calling the original synchronize function. Or I could have it start out near the lowest and, if the call returns immediately (indicating that it slept too long), it could decrease the sleep duration by a slight amount. It could then gradually nudge the sleep duration up until that happens again. That should reduce the amount of CPU usage without hurting GPU performance at all. The less variation there is among synchronize duration, the more CPU usage can be reduced.

Maybe there is a pattern we can exploit. For the GTX 970M, most calls look like they come in repeating pairs of about 10 ms then 20 ms, and then every couple dozen calls, a single call that lasts under 10 ms. But the pattern is not perfect, and there could be WUs with no noticible pattern anyway.

The idea is that we can generate at runtime an estimation of how long we can be almost sure that the synchronize call will not return, and then instead of letting the synchronize call spin uselessly, we just sleep which yields the task so another task can run or so the kernel idle task can put the CPU into a low power state. At around the time that we know the event might occur, we stop sleeping and let the original synchronize call do its low-latency spin-wait.

Does anyone have an idea for a better algorithm than the one I suggested? It will have to run before each synchronization call, so it cannot be heavy. It also has to be single-threaded and use little memory.

arisu · Post by **arisu** » Thu May 22, 2025 12:55 pm

I realized that I was misreading some of the OpenMM code, so I managed to reduce CPU usage by an extra 50%. Here is the current library, which has reduced CPU usage by a total of 95% without any detectable performance loss on a GTX 970M. When I tested with my 4090 Mobile, I got at most 3% performance loss at most in exchange for reducing CPU usage by 85%, and with only a few samples so it very well might be no loss. I will continue testing with other projects.

This is the code I am currently using, compiled as a library and invoked with LD_PRELOAD:

Code: Select all

#include <dlfcn.h>
#include <assert.h>
#include <cuda.h>

typedef CUresult (*cuCtxCreate_v2_type)(CUcontext *, unsigned int, CUdevice);
cuCtxCreate_v2_type orig_cuCtxCreate_v2;

typedef CUresult (*cuEventCreate_type)(CUevent *, unsigned int);
cuEventCreate_type orig_cuEventCreate;

static __attribute__((constructor)) void hook_function_early(void)
{
        void *handle;

        handle = dlopen("libcuda.so.1", RTLD_NOW);
        assert(handle);

        orig_cuCtxCreate_v2 = (cuCtxCreate_v2_type)dlsym(handle, "cuCtxCreate_v2");
        assert(orig_cuCtxCreate_v2);

        orig_cuEventCreate = (cuEventCreate_type)dlsym(handle, "cuEventCreate");
        assert(orig_cuEventCreate);
}

extern CUresult cuCtxCreate_v2(CUcontext *pctx, unsigned int flags, CUdevice dev)
{
        flags &= ~(CU_CTX_SCHED_MASK);
        flags |= CU_CTX_SCHED_BLOCKING_SYNC;

        return orig_cuCtxCreate_v2(pctx, flags, dev);
}

extern CUresult cuEventCreate(CUevent *phEvent, unsigned int flags)
{
        return orig_cuEventCreate(phEvent, flags | CU_EVENT_BLOCKING_SYNC);
}

I'm thinking that whatever caused that infamous 35% performance hit on Linux with non-blocking sync is either no longer an issue with newer Nvidia drivers, or it's only an issue with the highest-end GPUs on relatively low-end CPUs.

And an erratum about a previous post: When I said it reduced CPU usage from 100% to 90%, the code I was actually using was not the code in the post. I was hooking cuCtxCreate(), not cuEventCreate(). Lowest CPU usage occurs when hooking both.

Folding Forum

Nvidia vs AMD CPU usage

Re: Nvidia vs AMD CPU usage

Re: Nvidia vs AMD CPU usage

Re: Nvidia vs AMD CPU usage

Re: Nvidia vs AMD CPU usage

Re: Nvidia vs AMD CPU usage

Re: Nvidia vs AMD CPU usage

Re: Nvidia vs AMD CPU usage

Re: Nvidia vs AMD CPU usage

Re: Nvidia vs AMD CPU usage

Re: Nvidia vs AMD CPU usage

Re: Nvidia vs AMD CPU usage