Core22 v0.0.13 return CUDA_ERROR_INVALID_PTX (218)

[vmzy]

when Core22 v0.0.13 run Project: 14484 return Error loading CUDA module: CUDA_ERROR_INVALID_PTX (218)

Code: Select all

*********************** Log Started 2020-09-20T01:35:39Z ***********************
01:35:39:Trying to access database...
01:35:39:Successfully acquired database lock
01:35:42:Downloading GPUs.txt from assign1.foldingathome.org:80
01:35:42:Connecting to assign1.foldingathome.org:80
01:35:46:Read GPUs.txt
01:35:47:Enabled folding slot 01: READY gpu:0:GP108 [GeForce MX150 (GT 1030) Max-Q] 1127
01:35:47:****************************** FAHClient ******************************
01:35:47:        Version: 7.6.13
01:35:47:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
01:35:47:      Copyright: 2020 foldingathome.org
01:35:47:       Homepage: https://foldingathome.org/
01:35:47:           Date: Apr 27 2020
01:35:47:           Time: 21:21:01
01:35:47:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
01:35:47:         Branch: master
01:35:47:       Compiler: Visual C++ 2008
01:35:47:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
01:35:47:       Platform: win32 10
01:35:47:           Bits: 32
01:35:47:           Mode: Release
01:35:47:         Config: D:\Program Files (x86)\FAHData\config.xml
01:35:47:******************************** CBang ********************************
01:35:47:           Date: Apr 24 2020
01:35:47:           Time: 17:07:55
01:35:47:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
01:35:47:         Branch: master
01:35:47:       Compiler: Visual C++ 2008
01:35:47:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
01:35:47:       Platform: win32 10
01:35:47:           Bits: 32
01:35:47:           Mode: Release
01:35:47:******************************* System ********************************
01:35:47:            CPU: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
01:35:47:         CPU ID: GenuineIntel Family 6 Model 142 Stepping 9
01:35:47:           CPUs: 4
01:35:47:         Memory: 7.91GiB
01:35:47:    Free Memory: 2.92GiB
01:35:47:        Threads: WINDOWS_THREADS
01:35:47:     OS Version: 6.2
01:35:47:    Has Battery: true
01:35:47:     On Battery: false
01:35:47:     UTC Offset: 8
01:35:47:            PID: 91232
01:35:47:            CWD: D:\Program Files (x86)\FAHData
01:35:47:  Win32 Service: false
01:35:47:             OS: Windows 10 Home China
01:35:47:        OS Arch: AMD64
01:35:47:           GPUs: 1
01:35:47:          GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:5 GP108 [GeForce MX150 (GT 1030)
01:35:47:                 Max-Q] 1127
01:35:47:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:6.1 Driver:9.1
01:35:47:OpenCL Device 0: Platform:0 Device:0 Bus:NA Slot:NA Compute:2.1 Driver:24.20
01:35:47:OpenCL Device 2: Platform:1 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:391.24
01:35:47:******************************* libFAH ********************************
01:35:47:           Date: Apr 15 2020
01:35:47:           Time: 14:53:14
01:35:47:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
01:35:47:         Branch: master
01:35:47:       Compiler: Visual C++ 2008
01:35:47:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
01:35:47:       Platform: win32 10
01:35:47:           Bits: 32
01:35:47:           Mode: Release
01:35:47:***********************************************************************
01:35:47:<config>
01:35:47:  <!-- Client Control -->
01:35:47:  <disable-sleep-when-active v='false'/>
01:35:47:
01:35:47:  <!-- Folding Core -->
01:35:47:  <checkpoint v='30'/>
01:35:47:
01:35:47:  <!-- Network -->
01:35:47:  <proxy v='210.101.131.231:8080'/>
01:35:47:
01:35:47:  <!-- Slot Control -->
01:35:47:  <power v='full'/>
01:35:47:
01:35:47:  <!-- User Information -->
01:35:47:  <passkey v='*****'/>
01:35:47:  <team v='3213'/>
01:35:47:  <user v='vmzy'/>
01:35:47:
01:35:47:  <!-- Folding Slots -->
01:35:47:  <slot id='1' type='GPU'/>
01:35:47:</config>
...
05:03:24:WU00:FS01:Connecting to assign1.foldingathome.org:80
05:03:25:WU00:FS01:Assigned to work server 128.252.203.9
05:03:25:WU00:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GP108 [GeForce MX150 (GT 1030) Max-Q] 1127 from 128.252.203.9
05:03:25:WU00:FS01:Connecting to 128.252.203.9:8080
05:03:26:WU00:FS01:Downloading 20.60MiB
...
05:30:35:WU00:FS01:Download complete
05:30:35:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:14484 run:0 clone:920 gen:74 core:0x22 unit:0x0000006f80fccb095f171a3d48be36d6
05:30:35:WU00:FS01:Downloading core from http://cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah
05:30:35:WU00:FS01:Connecting to cores.foldingathome.org:80
05:30:36:WU00:FS01:FahCore 22: Downloading 77.67MiB
...
07:23:03:WU00:FS01:FahCore 22: Download complete
07:23:03:WU00:FS01:Valid core signature
07:23:03:WU00:FS01:Unpacked 8.33MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe
07:23:03:WU00:FS01:Unpacked 3.06MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/nvrtc-builtins64_92.dll
07:23:03:WU00:FS01:Unpacked 3.13MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/OpenMM.dll
07:23:03:WU00:FS01:Unpacked 2.07MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/OpenMMOpenCL.dll
07:23:03:WU00:FS01:Unpacked 14.82MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/nvrtc64_92.dll
07:23:03:WU00:FS01:Unpacked 82.99MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/cufft64_92.dll
07:23:03:WU00:FS01:Unpacked 275.00KiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/OpenMMCudaCompiler.dll
07:23:03:WU00:FS01:Unpacked 1.95MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/OpenMMCUDA.dll
07:23:03:WU00:FS01:Unpacked 786.00KiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/OpenMMCPU.dll
07:23:03:WU00:FS01:Starting
07:23:03:WU00:FS01:Running FahCore: "D:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" "D:\Program Files (x86)\FAHData\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe" -dir 00 -suffix 01 -version 706 -lifeline 91232 -checkpoint 30 -gpu-vendor nvidia -opencl-platform 1 -opencl-device 0 -cuda-device 0 -gpu 0
07:23:04:WU00:FS01:Started FahCore on PID 157352
07:23:04:WU00:FS01:Core PID:154580
07:23:04:WU00:FS01:FahCore 0x22 started
07:23:05:WU00:FS01:0x22:*********************** Log Started 2020-09-25T07:23:04Z ***********************
07:23:05:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:23:05:WU00:FS01:0x22:       Core: Core22
07:23:05:WU00:FS01:0x22:       Type: 0x22
07:23:05:WU00:FS01:0x22:    Version: 0.0.13
07:23:05:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
07:23:05:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
07:23:05:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
07:23:05:WU00:FS01:0x22:       Date: Sep 19 2020
07:23:05:WU00:FS01:0x22:       Time: 02:35:58
07:23:05:WU00:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
07:23:05:WU00:FS01:0x22:     Branch: core22-0.0.13
07:23:05:WU00:FS01:0x22:   Compiler: Visual C++ 2015
07:23:05:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:05:WU00:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
07:23:05:WU00:FS01:0x22:   Platform: win32 10
07:23:05:WU00:FS01:0x22:       Bits: 64
07:23:05:WU00:FS01:0x22:       Mode: Release
07:23:05:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
07:23:05:WU00:FS01:0x22:             <peastman@stanford.edu>
07:23:05:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 157352 -checkpoint 30
07:23:05:WU00:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 1 -opencl-device 0 -cuda-device
07:23:05:WU00:FS01:0x22:             0 -gpu 0
07:23:05:WU00:FS01:0x22:************************************ libFAH ************************************
07:23:05:WU00:FS01:0x22:       Date: Sep 7 2020
07:23:05:WU00:FS01:0x22:       Time: 19:09:56
07:23:05:WU00:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
07:23:05:WU00:FS01:0x22:     Branch: HEAD
07:23:05:WU00:FS01:0x22:   Compiler: Visual C++ 2015
07:23:05:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:05:WU00:FS01:0x22:   Platform: win32 10
07:23:05:WU00:FS01:0x22:       Bits: 64
07:23:05:WU00:FS01:0x22:       Mode: Release
07:23:05:WU00:FS01:0x22:************************************ CBang *************************************
07:23:05:WU00:FS01:0x22:       Date: Sep 7 2020
07:23:05:WU00:FS01:0x22:       Time: 19:08:30
07:23:05:WU00:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
07:23:05:WU00:FS01:0x22:     Branch: HEAD
07:23:05:WU00:FS01:0x22:   Compiler: Visual C++ 2015
07:23:05:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:05:WU00:FS01:0x22:   Platform: win32 10
07:23:05:WU00:FS01:0x22:       Bits: 64
07:23:05:WU00:FS01:0x22:       Mode: Release
07:23:05:WU00:FS01:0x22:************************************ System ************************************
07:23:05:WU00:FS01:0x22:        CPU: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
07:23:05:WU00:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 142 Stepping 9
07:23:05:WU00:FS01:0x22:       CPUs: 4
07:23:05:WU00:FS01:0x22:     Memory: 7.91GiB
07:23:05:WU00:FS01:0x22:Free Memory: 2.56GiB
07:23:05:WU00:FS01:0x22:    Threads: WINDOWS_THREADS
07:23:05:WU00:FS01:0x22: OS Version: 6.2
07:23:05:WU00:FS01:0x22:Has Battery: true
07:23:05:WU00:FS01:0x22: On Battery: false
07:23:05:WU00:FS01:0x22: UTC Offset: 8
07:23:05:WU00:FS01:0x22:        PID: 154580
07:23:05:WU00:FS01:0x22:        CWD: D:\Program Files (x86)\FAHData\work
07:23:05:WU00:FS01:0x22:************************************ OpenMM ************************************
07:23:05:WU00:FS01:0x22:   Revision: 189320d0
07:23:05:WU00:FS01:0x22:********************************************************************************
07:23:05:WU00:FS01:0x22:Project: 14484 (Run 0, Clone 920, Gen 74)
07:23:05:WU00:FS01:0x22:Unit: 0x0000006f80fccb095f171a3d48be36d6
07:23:05:WU00:FS01:0x22:Reading tar file core.xml
07:23:05:WU00:FS01:0x22:Reading tar file integrator.xml.bz2
07:23:05:WU00:FS01:0x22:Reading tar file state.xml.bz2
07:23:05:WU00:FS01:0x22:Reading tar file system.xml.bz2
07:23:05:WU00:FS01:0x22:Digital signatures verified
07:23:05:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:23:05:WU00:FS01:0x22:Version 0.0.13
07:23:06:WU00:FS01:0x22:  Checkpoint write interval: 25000 steps (2%) [50 total]
07:23:06:WU00:FS01:0x22:  JSON viewer frame write interval: 12500 steps (1%) [100 total]
07:23:06:WU00:FS01:0x22:  XTC frame write interval: 10000 steps (0.8%) [125 total]
07:23:06:WU00:FS01:0x22:  Global context and integrator variables write interval: disabled
07:23:06:WU00:FS01:0x22:There are 4 platforms available.
07:23:06:WU00:FS01:0x22:Platform 0: Reference
07:23:06:WU00:FS01:0x22:Platform 1: CPU
07:23:06:WU00:FS01:0x22:Platform 2: OpenCL
07:23:06:WU00:FS01:0x22:  opencl-device 0 specified
07:23:06:WU00:FS01:0x22:Platform 3: CUDA
07:23:06:WU00:FS01:0x22:  cuda-device 0 specified
07:23:31:WU00:FS01:0x22:Attempting to create CUDA context:
07:23:31:WU00:FS01:0x22:  Configuring platform CUDA
07:23:33:WU00:FS01:0x22:Failed to create CUDA context:
07:23:33:WU00:FS01:0x22:Error loading CUDA module: CUDA_ERROR_INVALID_PTX (218)
07:23:33:WU00:FS01:0x22:Attempting to create OpenCL context:
07:23:33:WU00:FS01:0x22:  Configuring platform OpenCL
07:23:42:WU00:FS01:0x22:  Using OpenCL on platformId 1 and gpu 0
07:23:42:WU00:FS01:0x22:Completed 0 out of 1250000 steps (0%)

[foldy]

It means it cannot run CUDA so it switches to OpenCL

[Kjetil]

core 22 v0.0.13 is om beta and p 14484 is not v13 beta Project. And did not see beta on in his log.

[gunnarre]

Core 22 v. 0.0.13 is apparently released to non-beta users too. I have folded a WU on project 14486 in CUDA on a GTX 1080. I have client-type = advanced set, which is technically late stage beta, but it does seem like some projects are now using Core 22 v. 0.0.13 in the general non-beta folding. Yay, CUDA?

[aetch]

I'm just a pleb and I have it too. :p

[gunnarre]

Prepare for massive speed boosts, ladies and gentlemen!

[vmzy]

Fix the problem by upgrade nVidia driver from 391.24 to latest 456.38.
But when FAH run CUDA platform successfully, it has ignored the checkpoints(has been ran 18%) generated by opencl platform previous, and restart from beginning(0%).
The TPF has been reduced from 25min to 19min.

[Kjetil]

Core 22 v. 0.0.13 is apparently released to non-beta users too. I have folded a WU on project 14486 in CUDA on a GTX 1080. I have client-type = advanced set, which is technically late stage beta, but it does seem like some projects are now using Core 22 v. 0.0.13 in the general non-beta folding. Yay, CUDA?

Okay sorry all, on beta and only running 17xxx projects.(0013) Now i running p 134xx on c 22 0011

[JohnChodera]

> Core 22 v. 0.0.13 is apparently released to non-beta users too.

Whoops! This made it out of the testing lab accidentally. We're still doing a lot of testing, so nothing official yet...

~ John Chodera // MSKCC

[PantherX]

...when FAH run CUDA platform successfully, it has ignored the checkpoints(has been ran 18%) generated by opencl platform previous, and restart from beginning(0%)...

Currently, that's expected since the checkpoints written in CUDA context can't be read by OpenCL and vice-versa. Thus, the WU will restart from 0% and that will also throw off HFM.NET (if you're using it). It is a very rare edge case that you have discovered and the likelihood of it happening outside of the dev/testers environment is exceptionally low. This won't normally happen even for Beta testers... you can consider yourself really lucky in that aspect

It would be nice to ensure that that the checkpoint written is platform independent but there's no official commitment yet if it will be available in FahCore_22.