Project:14415 run:0 clone:1273 gen:42 core:0x22

[Swedis]

Tried to upload this for two days now.

18:12:35:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:14415 run:0 clone:1273 gen:42 core:0x22 unit:0x000000460d5262775e839e59cd83524e
18:12:35:WU02:FS01:Uploading 242.52MiB to 13.82.98.119
18:12:35:WU02:FS01:Connecting to 13.82.98.119:8080
18:12:35:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
18:12:35:WU02:FS01:Trying to send results to collection server
18:12:35:WU02:FS01:Uploading 242.52MiB to 52.224.109.74
18:12:35:WU02:FS01:Connecting to 52.224.109.74:8080
18:12:35:ERROR:WU02:FS01:Exception: Transfer failed

[PantherX]

Welcome to the F@H Forum Swedis,

Please note that I am able to reach the landing page of both servers so it could be possible that the issue is at your end. Can you please review this topic to get some pointers: viewtopic.php?f=18&t=17794

[Swedis]

Thank you.

So can i by typing the IP-adresses. Mind that i have successfully downloaded and uploaded other WU:s from both slots during this issue.
It´s only this specific WU that is causing trouble.

Code: Select all

19:41:52:FS00:Unpaused
19:41:52:FS01:Unpaused
19:41:52:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:14415 run:0 clone:1273 gen:42 core:0x22 unit:0x000000460d5262775e839e59cd83524e
19:41:52:WU02:FS01:Uploading 242.52MiB to 13.82.98.119
19:41:52:WU02:FS01:Connecting to 13.82.98.119:8080
19:41:53:WU00:FS00:Starting
19:41:53:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Swedis\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/avx/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 706 -lifeline 2588 -checkpoint 15 -np 6
19:41:53:WU00:FS00:Started FahCore on PID 6084
19:41:53:WU00:FS00:Core PID:10116
19:41:53:WU00:FS00:FahCore 0xa7 started
19:41:53:WU01:FS01:Starting
19:41:53:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Swedis\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 2588 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
19:41:53:WU01:FS01:Started FahCore on PID 10176
19:41:53:WU01:FS01:Core PID:10416
19:41:53:WU01:FS01:FahCore 0x22 started
19:41:53:WU00:FS00:0xa7:*********************** Log Started 2020-05-15T19:41:53Z ***********************
19:41:53:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
19:41:53:WU00:FS00:0xa7:       Type: 0xa7
19:41:53:WU00:FS00:0xa7:       Core: Gromacs
19:41:53:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 6084 -checkpoint 15 -np 6
19:41:53:WU00:FS00:0xa7:************************************ CBang *************************************
19:41:53:WU00:FS00:0xa7:       Date: Oct 26 2019
19:41:53:WU00:FS00:0xa7:       Time: 01:38:25
19:41:53:WU00:FS00:0xa7:   Revision: c46a1a011a24143739ac7218c5a435f66777f62f
19:41:53:WU00:FS00:0xa7:     Branch: master
19:41:53:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
19:41:53:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
19:41:53:WU00:FS00:0xa7:   Platform: win32 10
19:41:53:WU00:FS00:0xa7:       Bits: 64
19:41:53:WU00:FS00:0xa7:       Mode: Release
19:41:53:WU00:FS00:0xa7:************************************ System ************************************
19:41:53:WU00:FS00:0xa7:        CPU: Intel(R) Core(TM) i7-4820K CPU @ 3.70GHz
19:41:53:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
19:41:53:WU00:FS00:0xa7:       CPUs: 8
19:41:53:WU00:FS00:0xa7:     Memory: 15.94GiB
19:41:53:WU00:FS00:0xa7:Free Memory: 10.12GiB
19:41:53:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
19:41:53:WU00:FS00:0xa7: OS Version: 6.2
19:41:53:WU00:FS00:0xa7:Has Battery: false
19:41:53:WU00:FS00:0xa7: On Battery: false
19:41:53:WU00:FS00:0xa7: UTC Offset: 2
19:41:53:WU00:FS00:0xa7:        PID: 10116
19:41:53:WU00:FS00:0xa7:        CWD: C:\Users\Swedis\AppData\Roaming\FAHClient\work
19:41:53:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
19:41:53:WU00:FS00:0xa7:    Version: 0.0.18
19:41:53:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:41:53:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
19:41:53:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
19:41:53:WU00:FS00:0xa7:       Date: Oct 26 2019
19:41:53:WU00:FS00:0xa7:       Time: 01:52:30
19:41:53:WU00:FS00:0xa7:   Revision: c1e3513b1bc0c16013668f2173ee969e5995b38e
19:41:53:WU00:FS00:0xa7:     Branch: master
19:41:53:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
19:41:53:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
19:41:53:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
19:41:53:WU02:FS01:Trying to send results to collection server
19:41:53:WU00:FS00:0xa7:   Platform: win32 10
19:41:53:WU00:FS00:0xa7:       Bits: 64
19:41:53:WU00:FS00:0xa7:       Mode: Release
19:41:53:WU00:FS00:0xa7:************************************ Build *************************************
19:41:53:WU00:FS00:0xa7:       SIMD: avx_256
19:41:53:WU00:FS00:0xa7:********************************************************************************
19:41:53:WU02:FS01:Uploading 242.52MiB to 52.224.109.74
19:41:53:WU00:FS00:0xa7:Project: 14542 (Run 0, Clone 769, Gen 209)
19:41:53:WU02:FS01:Connecting to 52.224.109.74:8080
19:41:53:WU00:FS00:0xa7:Unit: 0x000000d780fccb045e7fbfabe87db18d
19:41:53:WU00:FS00:0xa7:Digital signatures verified
19:41:53:WU00:FS00:0xa7:Calling: mdrun -s frame209.tpr -o frame209.trr -x frame209.xtc -cpi state.cpt -cpt 15 -nt 6
19:41:53:WU00:FS00:0xa7:Steps: first=104500000 total=500000
19:41:53:WU01:FS01:0x22:*********************** Log Started 2020-05-15T19:41:53Z ***********************
19:41:53:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
19:41:53:WU01:FS01:0x22:       Type: 0x22
19:41:53:WU01:FS01:0x22:       Core: Core22
19:41:53:WU01:FS01:0x22:    Website: https://foldingathome.org/
19:41:53:WU01:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
19:41:53:WU01:FS01:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
19:41:53:WU01:FS01:0x22:             <rafal.wiewiora@choderalab.org>
19:41:53:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 10176 -checkpoint 15
19:41:53:WU01:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
19:41:53:WU01:FS01:0x22:     Config: <none>
19:41:53:WU01:FS01:0x22:************************************ Build *************************************
19:41:53:WU01:FS01:0x22:    Version: 0.0.5
19:41:53:WU01:FS01:0x22:       Date: Apr 22 2020
19:41:53:WU01:FS01:0x22:       Time: 04:42:59
19:41:53:WU01:FS01:0x22: Repository: Git
19:41:53:WU01:FS01:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
19:41:53:WU01:FS01:0x22:     Branch: HEAD
19:41:53:WU01:FS01:0x22:   Compiler: Visual C++ 2008
19:41:53:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
19:41:53:WU01:FS01:0x22:   Platform: win32 10
19:41:53:WU01:FS01:0x22:       Bits: 64
19:41:53:WU01:FS01:0x22:       Mode: Release
19:41:53:WU01:FS01:0x22:************************************ System ************************************
19:41:53:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i7-4820K CPU @ 3.70GHz
19:41:53:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
19:41:53:WU01:FS01:0x22:       CPUs: 8
19:41:53:WU01:FS01:0x22:     Memory: 15.94GiB
19:41:53:WU01:FS01:0x22:Free Memory: 10.12GiB
19:41:53:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
19:41:53:WU01:FS01:0x22: OS Version: 6.2
19:41:53:WU01:FS01:0x22:Has Battery: false
19:41:53:WU01:FS01:0x22: On Battery: false
19:41:53:WU01:FS01:0x22: UTC Offset: 2
19:41:53:WU01:FS01:0x22:        PID: 10416
19:41:53:WU01:FS01:0x22:        CWD: C:\Users\Swedis\AppData\Roaming\FAHClient\work
19:41:53:WU01:FS01:0x22:         OS: Windows 10 Pro
19:41:53:WU01:FS01:0x22:    OS Arch: AMD64
19:41:53:WU01:FS01:0x22:********************************************************************************
19:41:53:WU01:FS01:0x22:Project: 16443 (Run 0, Clone 1236, Gen 19)
19:41:53:WU01:FS01:0x22:Unit: 0x0000001880fccb015eaa001956cc6dc0
19:41:53:WU01:FS01:0x22:Digital signatures verified
19:41:53:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
19:41:53:WU01:FS01:0x22:Version 0.0.5
19:41:53:WU01:FS01:0x22:  Found a checkpoint file
19:41:55:WU00:FS00:0xa7:Completed 157462 out of 500000 steps (31%)
19:42:01:ERROR:WU02:FS01:Exception: Transfer failed
19:42:10:WU01:FS01:0x22:Completed 1220000 out of 5000000 steps (24%)
19:42:10:WU01:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
19:43:30:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:14415 run:0 clone:1273 gen:42 core:0x22 unit:0x000000460d5262775e839e59cd83524e
19:43:30:WU02:FS01:Uploading 242.52MiB to 13.82.98.119
19:43:30:WU02:FS01:Connecting to 13.82.98.119:8080
19:43:30:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
19:43:30:WU02:FS01:Trying to send results to collection server
19:43:30:WU02:FS01:Uploading 242.52MiB to 52.224.109.74
19:43:30:WU02:FS01:Connecting to 52.224.109.74:8080
19:43:31:ERROR:WU02:FS01:Exception: Transfer failed

Mod Edit: Added Code Tags - PantherX

[Neil-B]

If you edit your post - select the log text and click on the Code button above the editing window then save it will put the log in a scrollable window which will make it easier to read.

As to your WU upload failures this may well just be a busy/overloaded server ... your client will keep retrying automatically - hopefully it will upload before deadline, if not the client will dispose of it.

[Swedis]

Oh sorry, i used quick reply, not full editor. I will remember that and thanks for your support, hopefully it will upload.

[Neil-B]

It took me quite a while to find out why I couldn't enter logs like everyone else did

[Swedis]

Got another WU with same issue, same project number and WS/CS. Coincidence? They will probably not succeed to upload

Code: Select all

23:48:56:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:14415 run:0 clone:48 gen:59 core:0x22 unit:0x000000570d5262775e839e5d7d117abb
23:48:56:WU03:FS01:Starting
23:48:56:WU03:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Swedis\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 03 -suffix 01 -version 706 -lifeline 11348 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
23:48:56:WU03:FS01:Started FahCore on PID 14196
23:48:56:WU03:FS01:Core PID:6904
23:48:56:WU03:FS01:FahCore 0x22 started
23:48:57:WU03:FS01:0x22:*********************** Log Started 2020-05-15T23:48:56Z ***********************
23:48:57:WU03:FS01:0x22:*************************** Core22 Folding@home Core ***************************
23:48:57:WU03:FS01:0x22:       Type: 0x22
23:48:57:WU03:FS01:0x22:       Core: Core22
23:48:57:WU03:FS01:0x22:    Website: https://foldingathome.org/
23:48:57:WU03:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
23:48:57:WU03:FS01:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
23:48:57:WU03:FS01:0x22:             <rafal.wiewiora@choderalab.org>
23:48:57:WU03:FS01:0x22:       Args: -dir 03 -suffix 01 -version 706 -lifeline 14196 -checkpoint 15
23:48:57:WU03:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
23:48:57:WU03:FS01:0x22:     Config: <none>
23:48:57:WU03:FS01:0x22:************************************ Build *************************************
23:48:57:WU03:FS01:0x22:    Version: 0.0.5
23:48:57:WU03:FS01:0x22:       Date: Apr 22 2020
23:48:57:WU03:FS01:0x22:       Time: 04:42:59
23:48:57:WU03:FS01:0x22: Repository: Git
23:48:57:WU03:FS01:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
23:48:57:WU03:FS01:0x22:     Branch: HEAD
23:48:57:WU03:FS01:0x22:   Compiler: Visual C++ 2008
23:48:57:WU03:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
23:48:57:WU03:FS01:0x22:   Platform: win32 10
23:48:57:WU03:FS01:0x22:       Bits: 64
23:48:57:WU03:FS01:0x22:       Mode: Release
23:48:57:WU03:FS01:0x22:************************************ System ************************************
23:48:57:WU03:FS01:0x22:        CPU: Intel(R) Core(TM) i7-4820K CPU @ 3.70GHz
23:48:57:WU03:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
23:48:57:WU03:FS01:0x22:       CPUs: 8
23:48:57:WU03:FS01:0x22:     Memory: 15.94GiB
23:48:57:WU03:FS01:0x22:Free Memory: 12.44GiB
23:48:57:WU03:FS01:0x22:    Threads: WINDOWS_THREADS
23:48:57:WU03:FS01:0x22: OS Version: 6.2
23:48:57:WU03:FS01:0x22:Has Battery: false
23:48:57:WU03:FS01:0x22: On Battery: false
23:48:57:WU03:FS01:0x22: UTC Offset: 2
23:48:57:WU03:FS01:0x22:        PID: 6904
23:48:57:WU03:FS01:0x22:        CWD: C:\Users\Swedis\AppData\Roaming\FAHClient\work
23:48:57:WU03:FS01:0x22:         OS: Windows 10 Pro
23:48:57:WU03:FS01:0x22:    OS Arch: AMD64
23:48:57:WU03:FS01:0x22:********************************************************************************
23:48:57:WU03:FS01:0x22:Project: 14415 (Run 0, Clone 48, Gen 59)
23:48:57:WU03:FS01:0x22:Unit: 0x000000570d5262775e839e5d7d117abb
23:48:57:WU03:FS01:0x22:Reading tar file core.xml
23:48:57:WU03:FS01:0x22:Reading tar file integrator.xml
23:48:57:WU03:FS01:0x22:Reading tar file state.xml
23:48:59:WU03:FS01:0x22:Reading tar file system.xml
23:49:01:WU03:FS01:0x22:Digital signatures verified
23:49:01:WU03:FS01:0x22:Folding@home GPU Core22 Folding@home Core
23:49:01:WU03:FS01:0x22:Version 0.0.5
23:49:46:WU03:FS01:0x22:Completed 0 out of 1000000 steps (0%)
23:49:46:WU03:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
23:53:27:WU03:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
23:57:04:WU03:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
00:00:41:WU03:FS01:0x22:Completed 30000 out of 1000000 steps (3%)
00:04:18:WU03:FS01:0x22:Completed 40000 out of 1000000 steps (4%)
00:04:54:WU03:FS01:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
00:04:54:WU03:FS01:0x22:Following exception occured: Particle coordinate is nan
00:08:31:WU03:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
00:12:08:WU03:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
00:15:45:WU03:FS01:0x22:Completed 30000 out of 1000000 steps (3%)
00:16:37:WU03:FS01:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
00:16:37:WU03:FS01:0x22:Following exception occured: Particle coordinate is nan
00:20:14:WU03:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
00:23:51:WU03:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
00:27:28:WU03:FS01:0x22:Completed 30000 out of 1000000 steps (3%)
00:31:06:WU03:FS01:0x22:Completed 40000 out of 1000000 steps (4%)
00:31:31:WU03:FS01:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
00:31:31:WU03:FS01:0x22:Following exception occured: Particle coordinate is nan
00:31:31:WU03:FS01:0x22:ERROR:114: Max Retries Reached
00:31:31:WU03:FS01:0x22:Saving result file ..\logfile_01.txt
00:31:31:WU03:FS01:0x22:Saving result file badstate-0.xml
00:31:32:WU03:FS01:0x22:Saving result file badstate-1.xml
00:31:33:WU03:FS01:0x22:Saving result file badstate-2.xml
00:31:34:WU03:FS01:0x22:Saving result file checkpt.crc
00:31:34:WU03:FS01:0x22:Saving result file science.log
00:31:34:WU03:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
00:31:34:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
00:31:34:WU03:FS01:Sending unit results: id:03 state:SEND error:FAULTY project:14415 run:0 clone:48 gen:59 core:0x22 unit:0x000000570d5262775e839e5d7d117abb
00:31:34:WU03:FS01:Uploading 201.57MiB to 13.82.98.119
00:31:34:WU03:FS01:Connecting to 13.82.98.119:8080
00:31:35:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
00:31:35:WU03:FS01:Trying to send results to collection server
00:31:35:WU03:FS01:Uploading 201.57MiB to 52.224.109.74
00:31:35:WU03:FS01:Connecting to 52.224.109.74:8080
00:31:36:ERROR:WU03:FS01:Exception: Transfer failed

[Neil-B]

Obviously the failing uploads are a worry … both are quite large so that may be playing into it … but your latest log is showing you have some form of issue causing the WU to fail (was this the same with the first one?) and Failed WUs can end up being larger iirc.

Someone more technical than I (and who is a GPU guru - I'm really not) will need to advise both on the size of the WUs you are trying to return as they may just be too large (for a variety of reasons) and also on the issue with the "BAD_WORK_UNIT".

I'll ask as I believe it is one thing that can cause this - is your GPU at stock speeds or does it have an OC (factory or otherwise) … If it has, even if it has been stable for other projects, this one may use the core in a way that pushes the GPU further than it can cope and I believe the solution would be to remove the OC.

[ajm]

But the problem here seems to be the folding itself, rather than the upload. Three times the core tried unsuccessfully to crunch the WU. It completed 3 or 4% only than quit with the error:

Code: Select all

00:16:37:WU03:FS01:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?

After three trials, it gave the error:

Code: Select all

00:31:31:WU03:FS01:0x22:ERROR:114: Max Retries Reached

and then proceeded to send the results. And that failed too:

Code: Select all

00:31:36:ERROR:WU03:FS01:Exception: Transfer failed

In the meantime, this WU has been successfully crunched and returned: https://apps.foldingathome.org/wu#proje ... 273&gen=42

[Swedis]

I´m using a stock card, it´s not an OC type of card either. Tuned up the fan to keep the GPU cooler (don´t mind the noise, computer is in the basement ) all turbo-feautures on the CPU is off and that is watercooled so no heat or power-issues there. System pulling 400w by the outlet and power supply is rated for 800W.

I tuned down the GPU-clock 10% anyhow to see if that could help future hickups. Thanks for your support.

[Neil-B]

Hopefully one of the GPU Gurus will spot this (as there may well be other things rather than just clock that might cause this type of issue) and give further advice.

[PantherX]

What driver version are you running on your AMD GPU?

I am aware that the next version of FahCore_22 will provide better messages to help in debugging issues like this. There's no ETA on when the new FahCore_22 version will be released.

[Swedis]

I have version 20.2.2, release date 2/28 -20.

[Swedis]

I stumbled on a blog at AMD.com where they adressed their cooperation with folding@home and they wrote this:
"Please ensure that you have the Radeon Software Adrenalin 2020 Edition 20.4.2 version or later installed if you’re running a system equipped with Radeon graphics."
For todays date that version is an "optional", recommended is 20.2.2. I will update anyway to rule out the drivers in this case.