17919 - Did one slip through the cracks?

[BobWilliams757]

This project did hit beta and was there for a few days, then was released to FAH.

It's giving very high PPD returns.

https://folding.lar.systems/gpu_ppd/b ... _rtx_3090

About twice (or more) than average on a 3090. The 3070 about normal PPD.

My 1660 Super got crazy high PPD. 12.78m on about 82 watts.

I'm suspecting something is a bit strange on this project. I've only picked up the one, and the others were already on LAR when I looked.

[BobWilliams757]

RTX 4080 hits 133M PPD.

[kiore]

RTX 4080 hits 133M PPD.

Are you actually getting those credits or is this a projection? Really looks like something amiss here.

[BobWilliams757]

All the ones on LAR are people other than me, so I can't speak for that.

But I got the 12.78M on my 1660 Super, and was credited as such, almost 222K points. Less than a half hour, 15 second TPF.

Going back on the Generation from mine, it appears everyone is getting huge numbers or base credit.


I could have contacted the researcher through here, but I didn't know if that was within forum rules.

[Joe_H]

How many steps were reported for the WUs you saw from 17919? There have been bugs in the past where the WS would change the number of steps from the initial setting the WUs were benchmarked on.

The other possibility is that the size in atoms for this project is towards middle sized. Possibly processing on high end GPUs is pushing the bonus points farther up the exponential curve than usual.

[BobWilliams757]

Joe,

250,000 steps. 103,824 atoms.


If we see something like this is the future, what's the preferred way to send up a flare? PM or email the researcher, or one of the admins, Beta members? I just assumed that the researchers don't want emails from regular folders on any frequent basis, and wasn't sure if making comment or note in the Announcements/Consortium section would draw the researchers notice, same for PM as I don't know if most of them check out the forum frequently.

[BobWilliams757]

I got another one, same thing. I'm going to send an email to the researcher.

[muziqaz]

Can we have a snippet of the log with few frames folded of this project. Project RCG would be great too
Thanks

[BobWilliams757]

Here you go....

This was the first one. The second I got performed in a similar manner. I had a different power limit set so the second one "only" returned about 11.7M PPD.


I did email the researcher through the forum here, but haven't gotten any type of response.

And I'm sure the number of GPU's shown to have run it on LARS has grown some since I got the second one. There was no 1660 Super on that site that had run it, and it wasn't one of mine logged on the site.

Code: Select all

08:12:14:WU00:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:TU116 [GeForce GTX 1660 SUPER] from 128.174.73.78
08:12:14:WU00:FS01:Connecting to 128.174.73.78:8080
08:12:14:WU01:FS01:0x22:Checkpoint completed at step 1250000
08:12:20:WU00:FS01:Downloading 13.77MiB
08:12:21:WU00:FS01:Download complete
08:12:21:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:17919 run:2 clone:28 gen:47 core:0x22 unit:0x0000001c0000002f000045ff00000002
08:12:29:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:18213 run:6702 clone:1 gen:29 core:0x22 unit:0x000000010000001d0000472500001a2e
08:12:29:WU01:FS01:Uploading 27.50MiB to 206.223.170.146
08:12:29:WU01:FS01:Connecting to 206.223.170.146:8080
08:12:29:WU00:FS01:Starting08:12:23:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
08:12:23:WU01:FS01:0x22:Saving result file checkpointIntegrator.xml
08:12:23:WU01:FS01:0x22:Saving result file checkpointState.xml
08:12:28:WU01:FS01:0x22:Saving result file positions.xtc
08:12:28:WU01:FS01:0x22:Saving result file science.log
08:12:28:WU01:FS01:0x22:Saving result file xtcAtoms.csv.bz2
08:12:28:WU01:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
08:12:29:WU01:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
08:12:29:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\rober\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.20/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 11000 -checkpoint 20 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
08:12:29:WU00:FS01:Started FahCore on PID 14444
08:12:29:WU00:FS01:Core PID:5256
08:12:29:WU00:FS01:FahCore 0x22 started
08:12:30:WU00:FS01:0x22:*********************** Log Started 2023-03-14T08:12:29Z ***********************
08:12:30:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
08:12:30:WU00:FS01:0x22:       Core: Core22
08:12:30:WU00:FS01:0x22:       Type: 0x22
08:12:30:WU00:FS01:0x22:    Version: 0.0.20
08:12:30:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
08:12:30:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
08:12:30:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
08:12:30:WU00:FS01:0x22:       Date: Jan 20 2022
08:12:30:WU00:FS01:0x22:       Time: 01:15:36
08:12:30:WU00:FS01:0x22:   Revision: 3f211b8a4346514edbff34e3cb1c0e0ec951373c
08:12:30:WU00:FS01:0x22:     Branch: HEAD
08:12:30:WU00:FS01:0x22:   Compiler: Visual C++
08:12:30:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
08:12:30:WU00:FS01:0x22:             -DOPENMM_VERSION="\"7.7.0\""
08:12:30:WU00:FS01:0x22:   Platform: win32 10
08:12:30:WU00:FS01:0x22:       Bits: 64
08:12:30:WU00:FS01:0x22:       Mode: Release
08:12:30:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
08:12:30:WU00:FS01:0x22:             <peastman@stanford.edu>
08:12:30:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 14444 -checkpoint 20
08:12:30:WU00:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
08:12:30:WU00:FS01:0x22:             0 -gpu 0
08:12:30:WU00:FS01:0x22:************************************ libFAH ************************************
08:12:30:WU00:FS01:0x22:       Date: Jan 20 2022
08:12:30:WU00:FS01:0x22:       Time: 01:14:17
08:12:30:WU00:FS01:0x22:   Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
08:12:30:WU00:FS01:0x22:     Branch: HEAD
08:12:30:WU00:FS01:0x22:   Compiler: Visual C++
08:12:30:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
08:12:30:WU00:FS01:0x22:   Platform: win32 10
08:12:30:WU00:FS01:0x22:       Bits: 64
08:12:30:WU00:FS01:0x22:       Mode: Release
08:12:30:WU00:FS01:0x22:************************************ CBang *************************************
08:12:30:WU00:FS01:0x22:       Date: Jan 20 2022
08:12:30:WU00:FS01:0x22:       Time: 01:13:20
08:12:30:WU00:FS01:0x22:   Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
08:12:30:WU00:FS01:0x22:     Branch: HEAD
08:12:30:WU00:FS01:0x22:   Compiler: Visual C++
08:12:30:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
08:12:30:WU00:FS01:0x22:   Platform: win32 10
08:12:30:WU00:FS01:0x22:       Bits: 64
08:12:30:WU00:FS01:0x22:       Mode: Release
08:12:30:WU00:FS01:0x22:************************************ System ************************************
08:12:30:WU00:FS01:0x22:        CPU: AMD Ryzen 5 2400G with Radeon Vega Graphics
08:12:30:WU00:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 17 Stepping 0
08:12:30:WU00:FS01:0x22:       CPUs: 8
08:12:30:WU00:FS01:0x22:     Memory: 15.93GiB
08:12:30:WU00:FS01:0x22:Free Memory: 11.29GiB
08:12:30:WU00:FS01:0x22:    Threads: WINDOWS_THREADS
08:12:30:WU00:FS01:0x22: OS Version: 6.2
08:12:30:WU00:FS01:0x22:Has Battery: false
08:12:30:WU00:FS01:0x22: On Battery: false
08:12:30:WU00:FS01:0x22: UTC Offset: -4
08:12:30:WU00:FS01:0x22:        PID: 5256
08:12:30:WU00:FS01:0x22:        CWD: C:\Users\rober\AppData\Roaming\FAHClient\work
08:12:30:WU00:FS01:0x22:************************************ OpenMM ************************************
08:12:30:WU00:FS01:0x22:    Version: 7.7.0
08:12:30:WU00:FS01:0x22:********************************************************************************
08:12:30:WU00:FS01:0x22:Project: 17919 (Run 2, Clone 28, Gen 47)
08:12:30:WU00:FS01:0x22:Reading tar file core.xml
08:12:30:WU00:FS01:0x22:Reading tar file integrator.xml
08:12:30:WU00:FS01:0x22:Reading tar file state.xml
08:12:30:WU00:FS01:0x22:Reading tar file system.xml
08:12:31:WU00:FS01:0x22:Digital signatures verified
08:12:31:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
08:12:31:WU00:FS01:0x22:Version 0.0.20
08:12:31:WU00:FS01:0x22:  Checkpoint write interval: 12500 steps (5%) [20 total]
08:12:31:WU00:FS01:0x22:  JSON viewer frame write interval: 2500 steps (1%) [100 total]
08:12:31:WU00:FS01:0x22:  XTC frame write interval: 25000 steps (10%) [10 total]
08:12:31:WU00:FS01:0x22:  Global context and integrator variables write interval: disabled
08:12:31:WU00:FS01:0x22:There are 4 platforms available.
08:12:31:WU00:FS01:0x22:Platform 0: Reference
08:12:31:WU00:FS01:0x22:Platform 1: CPU
08:12:31:WU00:FS01:0x22:Platform 2: OpenCL
08:12:31:WU00:FS01:0x22:  opencl-device 0 specified
08:12:31:WU00:FS01:0x22:Platform 3: CUDA
08:12:31:WU00:FS01:0x22:  cuda-device 0 specified
08:12:35:WU01:FS01:Upload 21.59%
08:12:42:WU01:FS01:Upload complete
08:12:42:WU01:FS01:Server responded WORK_ACK (400)
08:12:42:WU01:FS01:Final credit estimate, 289254.00 points
08:12:42:WU01:FS01:Cleaning up
08:12:57:WU00:FS01:0x22:Attempting to create CUDA context:
08:12:57:WU00:FS01:0x22:  Configuring platform CUDA
08:13:09:WU00:FS01:0x22:  Using CUDA and gpu 0
08:13:09:WU00:FS01:0x22:Completed 0 out of 250000 steps (0%)
08:13:10:WU00:FS01:0x22:Checkpoint completed at step 0
08:13:26:WU00:FS01:0x22:Completed 2500 out of 250000 steps (1%)
08:13:41:WU00:FS01:0x22:Completed 5000 out of 250000 steps (2%)
08:13:56:WU00:FS01:0x22:Completed 7500 out of 250000 steps (3%)
08:14:10:WU00:FS01:0x22:Completed 10000 out of 250000 steps (4%)
08:14:25:WU00:FS01:0x22:Completed 12500 out of 250000 steps (5%)
08:14:27:WU00:FS01:0x22:Checkpoint completed at step 12500
08:14:42:WU00:FS01:0x22:Completed 15000 out of 250000 steps (6%)
....................
08:35:55:WU00:FS01:0x22:Completed 225000 out of 250000 steps (90%)
08:35:57:WU00:FS01:0x22:Checkpoint completed at step 225000
08:36:12:WU00:FS01:0x22:Completed 227500 out of 250000 steps (91%)
08:36:26:WU00:FS01:0x22:Completed 230000 out of 250000 steps (92%)
08:36:41:WU00:FS01:0x22:Completed 232500 out of 250000 steps (93%)
08:36:56:WU00:FS01:0x22:Completed 235000 out of 250000 steps (94%)
08:37:11:WU00:FS01:0x22:Completed 237500 out of 250000 steps (95%)
08:37:13:WU00:FS01:0x22:Checkpoint completed at step 237500
08:37:28:WU00:FS01:0x22:Completed 240000 out of 250000 steps (96%)
08:37:42:WU00:FS01:0x22:Completed 242500 out of 250000 steps (97%)
08:37:57:WU00:FS01:0x22:Completed 245000 out of 250000 steps (98%)
08:38:12:WU00:FS01:0x22:Completed 247500 out of 250000 steps (99%)
08:38:27:WU00:FS01:0x22:Completed 250000 out of 250000 steps (100%)
08:38:27:WU00:FS01:0x22:Average performance: 58.3784 ns/day
08:38:28:WU01:FS01:Connecting to 13.59.134.176:80
08:38:28:WU01:FS01:Assigned to work server 129.32.209.202
08:38:28:WU01:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:TU116 [GeForce GTX 1660 SUPER] from 129.32.209.202
08:38:28:WU01:FS01:Connecting to 129.32.209.202:8080
08:38:28:WU00:FS01:0x22:Checkpoint completed at step 250000
08:38:29:WU01:FS01:Downloading 15.85MiB
08:38:30:WU01:FS01:Download complete
08:38:30:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:18453 run:7 clone:28 gen:95 core:0x22 unit:0x0000001c0000005f0000481500000007


08:38:32:WU00:FS01:0x22:Saving result file ..\logfile_01.txt
08:38:32:WU00:FS01:0x22:Saving result file checkpointIntegrator.xml
08:38:32:WU00:FS01:0x22:Saving result file checkpointState.xml
08:38:34:WU00:FS01:0x22:Saving result file positions.xtc
08:38:34:WU00:FS01:0x22:Saving result file science.log
08:38:34:WU00:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
08:38:35:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
08:38:35:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:17919 run:2 clone:28 gen:47 core:0x22 unit:0x0000001c0000002f000045ff00000002
08:38:35:WU00:FS01:Uploading 12.64MiB to 128.174.73.78
4.73.78:8080


08:38:37:WU00:FS01:Upload complete
08:38:37:WU00:FS01:Server responded WORK_ACK (400)
08:38:37:WU00:FS01:Final credit estimate, 221927.00 points

[muziqaz]

From log I can see that WU is 5x smaller than it is supposed to be. Researcher is enjoying a weekend, so it should be fixed on Monday. These things happen from time to time with misconfigured projects. Until then enjoy extra PPD

[BobWilliams757]

Well I figured something was wrong when I got 12+M PPD on my 1660.

I did get another finally and got points in line with normal. And it looks like the averages for many GPUs on LAR are moving down, so hopefully all of them are fixed now. Thanks for the help.


BUT.... as asked above, what's the best way to send up a flare? Here, Discord, PM a beta tester, email the researcher? Just making sure it goes to whoever can get the attention of the proper person the quickest.

[Joe_H]

Post here and include enough of the log to show things like number of steps. If I or the other mods see some pattern we will escalate it to the researchers. Some of the internal and beta testers also watch posts here. In this case the day after you posted with info from your log muziqaz read it and compared it with past runs he had processed and notified the researcher about the wrong number of steps. That is a bug in the project settings on the server. There are other causes of points being out of whack, sometimes it takes a few different examples to point towards the correction.

[muziqaz]

If you are on Discord, you can post there too. I tend to be on Discord more often than here. And push notification nature of the app is easier to react to especially if one is not near the PC.
Emailing the researcher is not the best idea

[BobWilliams757]

Works for me. So few slip through that it's never really an issue these days.