Creating CUDA context takes a long time after adding a 2nd RTX 5070

Moderators: Site Moderators, FAHC Science Team

Post Reply
andan
Posts: 3
Joined: Sat Feb 08, 2025 10:14 pm
Hardware configuration: System01: i9-7940x, 2x RTX 5070, RTX 4070

Creating CUDA context takes a long time after adding a 2nd RTX 5070

Post by andan »

This is a NotAProblemProblem, in that my client runs fine and WUs are worked, but...

I have a rig with three RTX cards: 1x 4070, 2x5070.
I'm running the nvidia-driver-580-open packages.

One of the 5070s is new, replacing a 3080.

When I was running with the 4070, 3080 and 5070, creating a CUDA context on any card was pretty quick. Sub 1-minute. After replacing the 3080 with another 5070 it can take up to 7 minutes or so, but only on the 5070s.

Anyone else seeing this? Any ideas what's going on and how to "fix" it?

thanks!

P.S.: I didn't notice any startup lag on any card before I replaced the 3080, but now I'm paranoid I just didn't notice that the 5070 was being slow, so apologies if this is WAI.

4070 startup

Code: Select all

22:28:01:I1:WU5025: Branch: HEAD
22:28:01:I1:WU5025: Compiler: GNU 9.4.0
22:28:01:I1:WU5025: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:28:01:I1:WU5025: -fdata-sections -O3 -funroll-loops -fno-pie
22:28:01:I1:WU5025: -DOPENMM_VERSION="\"7.7.0\""
22:28:01:I1:WU5025: Platform: linux 5.11.0-1025-azure
22:28:01:I1:WU5025: Bits: 64
22:28:01:I1:WU5025: Mode: Release
22:28:01:I1:WU5025:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
22:28:01:I1:WU5025: <peastman@stanford.edu>
22:28:01:I1:WU5025: Args: -dir 9zV1kMPvvp7MdF2Qj58BDcf5qaVLceQaPefRtDA-5s4 -suffix 01
22:28:01:I1:WU5025: -version 8.4.9 -lifeline 169 -gpu-uuid
22:28:01:I1:WU5025: d65f3411-caf4-41a8-0248-0dcf200dee35 -gpu-platform cuda -gpu-vendor
22:28:01:I1:WU5025: nvidia -opencl-platform 0 -opencl-device 0 -cuda-platform 0
22:28:01:I1:WU5025: -cuda-device 0 -gpu 0
22:28:01:I1:WU5025:************************************ libFAH ************************************
22:28:01:I1:WU5025: Date: Jan 20 2022
22:28:01:I1:WU5025: Time: 00:57:22
22:28:01:I1:WU5025: Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
22:28:01:I1:WU5025: Branch: HEAD
22:28:01:I1:WU5025: Compiler: GNU 9.4.0
22:28:01:I1:WU5025: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:28:01:I1:WU5025: -fdata-sections -O3 -funroll-loops -fno-pie
22:28:01:I1:WU5025: Platform: linux 5.11.0-1025-azure
22:28:01:I1:WU5025: Bits: 64
22:28:01:I1:WU5025: Mode: Release
22:28:01:I1:WU5025:************************************ CBang *************************************
22:28:01:I1:WU5025: Date: Jan 20 2022
22:28:01:I1:WU5025: Time: 00:57:00
22:28:01:I1:WU5025: Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
22:28:01:I1:WU5025: Branch: HEAD
22:28:01:I1:WU5025: Compiler: GNU 9.4.0
22:28:01:I1:WU5025: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:28:01:I1:WU5025: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
22:28:01:I1:WU5025: Platform: linux 5.11.0-1025-azure
22:28:01:I1:WU5025: Bits: 64
22:28:01:I1:WU5025: Mode: Release
22:28:01:I1:WU5025:************************************ System ************************************
22:28:01:I1:WU5025: CPU: Intel(R) Core(TM) i9-7940X CPU @ 3.10GHz
22:28:01:I1:WU5025: CPU ID: GenuineIntel Family 6 Model 85 Stepping 4
22:28:01:I1:WU5025: CPUs: 28
22:28:01:I1:WU5025: Memory: 31.04GiB
22:28:01:I1:WU5025:Free Memory: 11.31GiB
22:28:01:I1:WU5025: Threads: POSIX_THREADS
22:28:01:I1:WU5025: OS Version: 6.8
22:28:01:I1:WU5025:Has Battery: false
22:28:01:I1:WU5025: On Battery: false
22:28:01:I1:WU5025: UTC Offset: -7
22:28:01:I1:WU5025: PID: 6837
22:28:01:I1:WU5025: CWD: /config/work
22:28:01:I1:WU5025:************************************ OpenMM ************************************
22:28:01:I1:WU5025: Version: 7.7.0
22:28:01:I1:WU5025:********************************************************************************
22:28:01:I1:WU5025:Project: 14973 (Run 1, Clone 59, Gen 12)
22:28:01:I1:WU5025:Reading tar file core.xml
22:28:01:I1:WU5025:Reading tar file integrator.xml
22:28:01:I1:WU5025:Reading tar file state.xml
22:28:02:I1:WU5025:Reading tar file system.xml
22:28:04:I1:WU5025:Digital signatures verified
22:28:04:I1:WU5025:Folding@home GPU Core22 Folding@home Core
22:28:04:I1:WU5025:Version 0.0.20
22:28:04:I1:WU5025: Checkpoint write interval: 50000 steps (5%) [20 total]
22:28:04:I1:WU5025: JSON viewer frame write interval: 10000 steps (1%) [100 total]
22:28:04:I1:WU5025: XTC frame write interval: 25000 steps (2.5%) [40 total]
22:28:04:I1:WU5025: Global context and integrator variables write interval: disabled
22:28:04:I1:WU5025:There are 4 platforms available.
22:28:04:I1:WU5025:Platform 0: Reference
22:28:04:I1:WU5025:Platform 1: CPU
22:28:04:I1:WU5025:Platform 2: OpenCL
22:28:04:I1:WU5025: opencl-device 0 specified
22:28:04:I1:WU5025:Platform 3: CUDA
22:28:04:I1:WU5025: cuda-device 0 specified
22:28:33:I1:WU5025:Attempting to create CUDA context:                                                 <
22:28:33:I1:WU5025: Configuring platform CUDA                                                         < 22s startup
22:28:55:I1:WU5025: Using CUDA and gpu 0.                                                             < 
22:28:55:I1:WU5025:Completed 0 out of 1000000 steps (0%)
22:28:57:I1:WU5025:Checkpoint completed at step 0
22:29:52:I1:WU5025:Completed 10000 out of 1000000 steps (1%)
5070 A startup

Code: Select all

22:14:33:I1:WU5023: dd40d876-3e63-1967-9ae5-8227782a482e -gpu-platform cuda -gpu-vendor
22:14:33:I1:WU5023: nvidia -opencl-platform 0 -opencl-device 1 -cuda-platform 0
22:14:33:I1:WU5023: -cuda-device 1 -gpu 1
22:14:33:I1:WU5023:************************************ libFAH ************************************
22:14:33:I1:WU5023: Date: Jan 20 2022
22:14:33:I1:WU5023: Time: 00:57:22
22:14:33:I1:WU5023: Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
22:14:33:I1:WU5023: Branch: HEAD
22:14:33:I1:WU5023: Compiler: GNU 9.4.0
22:14:33:I1:WU5023: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:14:33:I1:WU5023: -fdata-sections -O3 -funroll-loops -fno-pie
22:14:33:I1:WU5023: Platform: linux 5.11.0-1025-azure
22:14:33:I1:WU5023: Bits: 64
22:14:33:I1:WU5023: Mode: Release
22:14:33:I1:WU5023:************************************ CBang *************************************
22:14:33:I1:WU5023: Date: Jan 20 2022
22:14:33:I1:WU5023: Time: 00:57:00
22:14:33:I1:WU5023: Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
22:14:33:I1:WU5023: Branch: HEAD
22:14:33:I1:WU5023: Compiler: GNU 9.4.0
22:14:33:I1:WU5023: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:14:33:I1:WU5023: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
22:14:33:I1:WU5023: Platform: linux 5.11.0-1025-azure
22:14:33:I1:WU5023: Bits: 64
22:14:33:I1:WU5023: Mode: Release
22:14:33:I1:WU5023:************************************ System ************************************
22:14:33:I1:WU5023: CPU: Intel(R) Core(TM) i9-7940X CPU @ 3.10GHz
22:14:33:I1:WU5023: CPU ID: GenuineIntel Family 6 Model 85 Stepping 4
22:14:33:I1:WU5023: CPUs: 28
22:14:33:I1:WU5023: Memory: 31.04GiB
22:14:33:I1:WU5023:Free Memory: 14.61GiB
22:14:33:I1:WU5023: Threads: POSIX_THREADS
22:14:33:I1:WU5023: OS Version: 6.8
22:14:33:I1:WU5023:Has Battery: false
22:14:33:I1:WU5023: On Battery: false
22:14:33:I1:WU5023: UTC Offset: -7
22:14:33:I1:WU5023: PID: 6443
22:14:33:I1:WU5023: CWD: /config/work
22:14:33:I1:WU5023:************************************ OpenMM ************************************
22:14:33:I1:WU5023: Version: 7.7.0
22:14:33:I1:WU5023:********************************************************************************
22:14:33:I1:WU5023:Project: 14973 (Run 2, Clone 67, Gen 2)
22:14:33:I1:WU5023:Reading tar file core.xml
22:14:33:I1:WU5023:Reading tar file integrator.xml
22:14:33:I1:WU5023:Reading tar file state.xml
22:14:34:I1:WU5023:Reading tar file system.xml
22:14:35:I1:WU5023:Digital signatures verified
22:14:35:I1:WU5023:Folding@home GPU Core22 Folding@home Core
22:14:35:I1:WU5023:Version 0.0.20
22:14:35:I1:WU5023: Checkpoint write interval: 50000 steps (5%) [20 total]
22:14:35:I1:WU5023: JSON viewer frame write interval: 10000 steps (1%) [100 total]
22:14:35:I1:WU5023: XTC frame write interval: 25000 steps (2.5%) [40 total]
22:14:35:I1:WU5023: Global context and integrator variables write interval: disabled
22:14:35:I1:WU5023:There are 4 platforms available.
22:14:35:I1:WU5023:Platform 0: Reference
22:14:35:I1:WU5023:Platform 1: CPU
22:14:35:I1:WU5023:Platform 2: OpenCL
22:14:35:I1:WU5023: opencl-device 1 specified
22:14:35:I1:WU5023:Platform 3: CUDA
22:14:35:I1:WU5023: cuda-device 1 specified
22:15:01:I1:WU5023:Attempting to create CUDA context:                                                 <
22:15:01:I1:WU5023: Configuring platform CUDA                                                         < 6m38s startup
22:21:39:I1:WU5023: Using CUDA and gpu                                                                <
22:21:39:I1:WU5023:Completed 0 out of 1000000 steps (0%)
5070 B startup

Code: Select all

22:15:01:I1:WU5024: -cuda-device 2 -gpu 2
22:15:01:I1:WU5024:************************************ libFAH ************************************
22:15:01:I1:WU5024: Date: Jan 20 2022
22:15:01:I1:WU5024: Time: 00:57:22
22:15:01:I1:WU5024: Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
22:15:01:I1:WU5024: Branch: HEAD
22:15:01:I1:WU5024: Compiler: GNU 9.4.0
22:15:01:I1:WU5024: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:15:01:I1:WU5024: -fdata-sections -O3 -funroll-loops -fno-pie
22:15:01:I1:WU5024: Platform: linux 5.11.0-1025-azure
22:15:01:I1:WU5024: Bits: 64
22:15:01:I1:WU5024: Mode: Release
22:15:01:I1:WU5024:************************************ CBang *************************************
22:15:01:I1:WU5024: Date: Jan 20 2022
22:15:01:I1:WU5024: Time: 00:57:00
22:15:01:I1:WU5024: Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
22:15:01:I1:WU5024: Branch: HEAD
22:15:01:I1:WU5024: Compiler: GNU 9.4.0
22:15:01:I1:WU5024: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:15:01:I1:WU5024: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
22:15:01:I1:WU5024: Platform: linux 5.11.0-1025-azure
22:15:01:I1:WU5024: Bits: 64
22:15:01:I1:WU5024: Mode: Release
22:15:01:I1:WU5024:************************************ System ************************************
22:15:01:I1:WU5024: CPU: Intel(R) Core(TM) i9-7940X CPU @ 3.10GHz
22:15:01:I1:WU5024: CPU ID: GenuineIntel Family 6 Model 85 Stepping 4
22:15:01:I1:WU5024: CPUs: 28
22:15:01:I1:WU5024: Memory: 31.04GiB
22:15:01:I1:WU5024:Free Memory: 12.71GiB
22:15:01:I1:WU5024: Threads: POSIX_THREADS
22:15:01:I1:WU5024: OS Version: 6.8
22:15:01:I1:WU5024:Has Battery: false
22:15:01:I1:WU5024: On Battery: false
22:15:01:I1:WU5024: UTC Offset: -7
22:15:01:I1:WU5024: PID: 6504
22:15:01:I1:WU5024: CWD: /config/work
22:15:01:I1:WU5024:************************************ OpenMM ************************************
22:15:01:I1:WU5024: Version: 7.7.0
22:15:01:I1:WU5024:********************************************************************************
22:15:01:I1:WU5024:Project: 14955 (Run 6, Clone 23, Gen 22)
22:15:01:I1:WU5024:Reading tar file core.xml
22:15:01:I1:WU5024:Reading tar file integrator.xml
22:15:01:I1:WU5024:Reading tar file state.xml
22:15:02:I1:WU5024:Reading tar file system.xml
22:15:04:I1:WU5024:Digital signatures verified
22:15:04:I1:WU5024:Folding@home GPU Core22 Folding@home Core
22:15:04:I1:WU5024:Version 0.0.20
22:15:04:I1:WU5024: Checkpoint write interval: 50000 steps (5%) [20 total]
22:15:04:I1:WU5024: JSON viewer frame write interval: 10000 steps (1%) [100 total]
22:15:04:I1:WU5024: XTC frame write interval: 25000 steps (2.5%) [40 total]
22:15:04:I1:WU5024: Global context and integrator variables write interval: disabled
22:15:04:I1:WU5024:There are 4 platforms available.
22:15:04:I1:WU5024:Platform 0: Reference
22:15:04:I1:WU5024:Platform 1: CPU
22:15:04:I1:WU5024:Platform 2: OpenCL
22:15:04:I1:WU5024: opencl-device 2 specified
22:15:04:I1:WU5024:Platform 3: CUDA
22:15:04:I1:WU5024: cuda-device 2 specified
22:15:33:I1:WU5024:Attempting to create CUDA context:                                                <
22:15:33:I1:WU5024: Configuring platform CUDA                                                        <  4m12s startup
22:19:45:I1:WU5024: Using CUDA and gpu 2                                                             < 
22:19:45:I1:WU5024:Completed 0 out of 1000000 steps (0%)
22:19:48:I1:WU5024:Checkpoint completed at step 0
22:20:34:I1:WU5024:Completed 10000 out of 1000000 steps (1%)
Top
andan
Posts: 3
Joined: Sat Feb 08, 2025 10:14 pm
Hardware configuration: System01: i9-7940x, 2x RTX 5070, RTX 4070

Re: Creating CUDA context takes a long time after adding a 2nd RTX 5070

Post by andan »

Hrm. It's actually more random that I thought. Maybe actually a not a problem problem or something to do with the project that is running?

From currently running WU:

5070 A startup: 12s

Code: Select all

16:21:26:I1:WU5038:Requesting WU assignment for user andan team 0
16:21:27:I1:WU5038:Received WU assignment pSKKTZBuyPItFoxDbBJgAzHiCkvtVgyjhpU7uc8J6NI
16:21:27:I1:WU5038:Downloading WU
16:21:44:I1:WU5038:DOWNLOAD 77% 29.52MiB of 38.47MiB
16:21:45:I1:WU5038:Received WU P12129 R39 C93 G12
...
16:21:49:I1:WU5038: cuda-device 1 specified
16:22:06:I1:WU5038:Attempting to create CUDA context:
16:22:06:I1:WU5038: Configuring platform CUDA
16:22:18:I1:WU5038: Using CUDA on CUDA Platform and gpu 1
16:22:18:I1:WU5038: GPU info: Platform: CUDA
16:22:18:I1:WU5038: GPU info: PlatformIndex: 0
16:22:18:I1:WU5038: GPU info: Device: NVIDIA GeForce RTX 5070
16:22:18:I1:WU5038: GPU info: DeviceIndex: 1
16:22:18:I1:WU5038: GPU info: Vendor: 0x10de
16:22:18:I1:WU5038: GPU info: PCI: 101:00:00
16:22:18:I1:WU5038: GPU info: Compute: 12.0
16:22:18:I1:WU5038: GPU info: Driver: 13.0
16:22:18:I1:WU5038: GPU info: GPU: true
16:22:18:I1:WU5038:Completed 0 out of 5000000 steps (0%)
5070 B startup: 11s

Code: Select all

13:22:31:I1:WU5035:Requesting WU assignment for user andan team 0
13:22:32:I1:WU5035:Received WU assignment -jvwdRXA-Qa6MqMt7T1URFpeHL0bwd792lOPxsTYBf4
13:22:32:I1:WU5035:Downloading WU
13:23:50:I1:WU5035:DOWNLOAD 66% 25.46MiB of 38.40MiB
13:23:51:I1:WU5035:DOWNLOAD 100% 38.40MiB of 38.40MiB
13:23:51:I1:WU5035:Received WU P12129 R25 C46 G18
...
13:23:55:I1:WU5035: cuda-device 2 specified
13:24:12:I1:WU5035:Attempting to create CUDA context:
13:24:12:I1:WU5035: Configuring platform CUDA
13:24:29:I1:WU5035: Using CUDA on CUDA Platform and gpu 2
13:24:29:I1:WU5035: GPU info: Platform: CUDA
13:24:29:I1:WU5035: GPU info: PlatformIndex: 0
13:24:29:I1:WU5035: GPU info: Device: NVIDIA GeForce RTX 5070
13:24:29:I1:WU5035: GPU info: DeviceIndex: 2
13:24:29:I1:WU5035: GPU info: Vendor: 0x10de
13:24:29:I1:WU5035: GPU info: PCI: 102:00:00
13:24:29:I1:WU5035: GPU info: Compute: 12.0
13:24:29:I1:WU5035: GPU info: Driver: 13.0
13:24:29:I1:WU5035: GPU info: GPU: true
13:24:29:I1:WU5035:Completed 0 out of 5000000 steps (0%)
4070 startup: 1s

Code: Select all

16:27:13:I1:WU5039:Requesting WU assignment for user andan team 0
16:27:14:I1:WU5039:Received WU assignment SYf5S1zgHRsJfFtkXp93WTBj_87lXPaWxQVaPMXXy70
16:27:14:I1:WU5039:Downloading WU
16:27:35:I1:WU5039:DOWNLOAD 80% 28.81MiB of 35.95MiB
16:27:36:I1:WU5039:Received WU P12129 R10 C43 G17
...
16:27:40:I1:WU5039: cuda-device 0 specified
16:27:57:I1:WU5039:Attempting to create CUDA context:
16:27:57:I1:WU5039: Configuring platform CUDA
16:28:12:I1:WU5039: Using CUDA on CUDA Platform and gpu 0
16:28:12:I1:WU5039: GPU info: Platform: CUDA
16:28:12:I1:WU5039: GPU info: PlatformIndex: 0
16:28:12:I1:WU5039: GPU info: Device: NVIDIA GeForce RTX 4070
16:28:12:I1:WU5039: GPU info: DeviceIndex: 0
16:28:12:I1:WU5039: GPU info: Vendor: 0x10de
16:28:12:I1:WU5039: GPU info: PCI: 23:00:00
16:28:12:I1:WU5039: GPU info: Compute: 8.9
16:28:12:I1:WU5039: GPU info: Driver: 13.0
16:28:12:I1:WU5039: GPU info: GPU: true
Top
muziqaz
Posts: 2101
Joined: Sun Dec 16, 2007 6:22 pm
Hardware configuration: 9950x, 9950x3D, 5950x, 5800x3D
7900xtx, RX9070, Radeon 7, 5700xt, 6900xt, Intel B580
Location: London
Contact:
Contact muziqaz

Re: Creating CUDA context takes a long time after adding a 2nd RTX 5070

Post by muziqaz »

Different projects (depending on their size) will be creating context for different periods of time. That is normal. Some projects can take up to 5-8 minutes on non x3d chips
FAH Omega tester
Image
Top
andan
Posts: 3
Joined: Sat Feb 08, 2025 10:14 pm
Hardware configuration: System01: i9-7940x, 2x RTX 5070, RTX 4070

Re: Creating CUDA context takes a long time after adding a 2nd RTX 5070

Post by andan »

muziqaz wrote: ↑Sat Oct 11, 2025 5:39 pm Different projects (depending on their size) will be creating context for different periods of time. That is normal. Some projects can take up to 5-8 minutes on non x3d chips
Ah! Thanks :)
Top
Post Reply

Return to “v8.4.9 Public Release for Windows / Linux / macOS”