02:55:33:WU00:FS01:0x22:Completed 830000 out of 1000000 steps (83%)
02:56:15:WU00:FS01:0x22:Completed 840000 out of 1000000 steps (84%)
02:56:58:WU00:FS01:0x22:Completed 850000 out of 1000000 steps (85%)
03:01:13:WU00:FS01:0x22:Completed 910000 out of 1000000 steps (91%)
copy in the project, run, clone, gen on the top line in this fashion:
Project: 13415 (Run 1769, Clone 21, Gen 0)
I just copied and pasted that particular one as it was one of the 4-5 failures I have gotten on this project. Given that those two 750 Ti's are the oldest video cards I am still running and they have had to be down clocked once before for a new folding core (21 I think, but can't remember) I wanted to be sure they were not finally failing.
The page results show that it is most likely not the ancient video cards as there is four other folders who have failed to finish that particular work unit. The same has been true on the other 4 to 5 failed work units.
These two 750 Ti's at 1375 mhz on xubuntu 18.04 were getting 118k to 128k ppd on core 22 before project 13415. When they suddenly started getting 285k to 333k ppd under the same conditions.....I quickly double checked a lot of things in disbelief. I'll even own up to an evil smile or two....but this project does seem to have a pretty high failure rate. I've had at least 5 out of an estimated 30 attempted; probably needs to be looked at.
12:36:59:WU02:FS01:0x22:Completed 570000 out of 600000 steps (95%)
12:37:44:WU00:FS02:0x22:Completed 970000 out of 1000000 steps (97%)
12:38:49:WU00:FS02:0x22:Completed 980000 out of 1000000 steps (98%)
12:39:29:WU02:FS01:0x22:Completed 576000 out of 600000 steps (96%)
12:39:54:WU00:FS02:0x22:Completed 990000 out of 1000000 steps (99%)
12:39:54:WU01:FS02:Connecting to assign1.foldingathome.org:80
12:39:54:WU01:FS02:Assigned to work server 18.188.125.154
12:39:54:WU01:FS02:Requesting new work unit for slot 02: RUNNING gpu:1:Ellesmere XT [Radeon RX 470/480/570/580/590] from 18.188.125.154
12:39:54:WU01:FS02:Connecting to 18.188.125.154:8080
12:40:58:WU00:FS02:0x22:Completed 1000000 out of 1000000 steps (100%)
12:40:58:WU00:FS02:0x22:Average performance: 265.846 ns/day
12:40:58:WU00:FS02:0x22:Saving result file ..\logfile_01.txt
12:40:58:WU00:FS02:0x22:Saving result file checkpointState.xml
12:40:58:WU00:FS02:0x22:Saving result file globals.csv
12:40:59:WU00:FS02:0x22:Saving result file positions.xtc
12:40:59:WU00:FS02:0x22:Saving result file science.log
12:40:59:WU00:FS02:0x22:Folding@home Core Shutdown: FINISHED_UNIT
12:41:00:WU01:FS02:Downloading 437.39KiB
12:41:00:WU01:FS02:Download complete
12:41:00:WU01:FS02:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13415 run:3007 clone:13 gen:0 core:0x22 unit:0x0000000312bc7d9a5ef1ae4498d009ca
12:41:18:WARNING:WU00:FS02:FahCore returned an unknown error code which probably indicates that it crashed
12:41:18:WARNING:WU00:FS02:FahCore returned: UNKNOWN_ENUM (-1073740940 = 0xc0000374)
12:41:18:WU01:FS02:Starting
12:41:18:WU01:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\phil\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 5568 -checkpoint 15 -gpu-vendor amd -opencl-platform 1 -opencl
Date of last Work Unit 2020-08-04 22:17:53
Total score 63,300,083
Total WUs 1,013
Overall rank (if points are combined) 15,440 of 2,735,299
@TPL … This is a tough one … my personal call is if it is folding and will complete within deadline even if it is was slower than anything I usually get from a project I'll let it run … If it completes then the logs are uploaded and the researchers have a chance to see things were slow/awry … If I were t dump it the researchers would have nothing … but that is how I see it - hence why I beta team - I don't care about points as points … Sometimes however with a known issue, and once the researchers are aware I may dump a WU under instruction … If it is a beta project then I definitely flag up in the beta forums - if not then I'll probably flag up in the relevant location (unless already known issue) … and I believe the researchers are watching this and related Projects carefully specifically because of their test nature and the variances in performance being seen - so slow results are useful to help them try and ascertain the cause.
30min per fold (yours appears a bit less) is just over two days to complete … timeout is 2.3 days so if you leave it to complete it will be done before it would be re-released to another folder so even if really slow on your rig it is still fast enough to beat timeout
Last edited by Neil-B on Thu Jun 25, 2020 1:00 pm, edited 2 times in total.
Timeout for 13415 is 1 day only. For some WUs it is longer but for example this morning I got one (other WU) with 1 day to timeout and my ETA was 3.68 days. So I dumped it.
This is interesting and I would be helpful if possible. Maybe I let this recent just run and see what happens. In a week or so...
And I think it is well known that old slow GPUs are not enough for nowaday large WUs. Nothing to show there. Now it is completely different thing as my old and slow have been able before. So I let it run.
Sorry, I'd picked up the PRCG for the CPU core … Problem as I see it is that if it is going to take over the timeout for everyone who gets it and everyone dumps it it will never get done (albeit slowly) and the researchers will never get any results to look at and understand … Timeout is simply the reissue point - deadline is when it ceases to be relevant … if having seen someone else complete before ones own does and the researchers are ok with it then dump it … actually having two completions - one from a non slow fold and the other from a slow fold could well be useful in identifying some of the issues.
Yes I understand and let it go whatever it takes. I have competed about ten of them without any problems except one in the beginning. That I dumped but this one I'm not going to dump.
16:41:39:WU02:FS02:0x22:Completed 300000 out of 1000000 steps (30%)
16:42:44:WU02:FS02:0x22:Completed 310000 out of 1000000 steps (31%)
16:43:50:WU02:FS02:0x22:Completed 320000 out of 1000000 steps (32%)
16:44:56:WU02:FS02:0x22:Completed 330000 out of 1000000 steps (33%)
16:46:02:WU02:FS02:0x22:Completed 340000 out of 1000000 steps (34%)
16:46:44:WU03:FS01:0x22:Completed 740000 out of 2000000 steps (37%)
16:47:08:WU02:FS02:0x22:Completed 350000 out of 1000000 steps (35%)
16:48:13:WU02:FS02:0x22:Completed 360000 out of 1000000 steps (36%)
16:49:19:WU02:FS02:0x22:Completed 370000 out of 1000000 steps (37%)
16:50:25:WU02:FS02:0x22:Completed 380000 out of 1000000 steps (38%)
16:51:31:WU02:FS02:0x22:Completed 390000 out of 1000000 steps (39%)
16:52:38:WU02:FS02:0x22:Completed 400000 out of 1000000 steps (40%)
16:53:10:WU03:FS01:0x22:Completed 760000 out of 2000000 steps (38%)
16:53:44:WU02:FS02:0x22:Completed 410000 out of 1000000 steps (41%)
16:54:50:WU02:FS02:0x22:Completed 420000 out of 1000000 steps (42%)
16:55:56:WU02:FS02:0x22:Completed 430000 out of 1000000 steps (43%)
16:57:02:WU02:FS02:0x22:Completed 440000 out of 1000000 steps (44%)
16:58:08:WU02:FS02:0x22:Completed 450000 out of 1000000 steps (45%)
16:59:16:WU02:FS02:0x22:Completed 460000 out of 1000000 steps (46%)
16:59:36:WU03:FS01:0x22:Completed 780000 out of 2000000 steps (39%)
17:00:23:WU02:FS02:0x22:Completed 470000 out of 1000000 steps (47%)
17:01:32:WU02:FS02:0x22:Completed 480000 out of 1000000 steps (48%)
17:02:42:WU02:FS02:0x22:Completed 490000 out of 1000000 steps (49%)
17:03:49:WU02:FS02:0x22:Completed 500000 out of 1000000 steps (50%)
17:04:55:WU02:FS02:0x22:Completed 510000 out of 1000000 steps (51%)
17:06:01:WU02:FS02:0x22:Completed 520000 out of 1000000 steps (52%)
17:06:09:WU03:FS01:0x22:Completed 800000 out of 2000000 steps (40%)
17:07:08:WU02:FS02:0x22:Completed 530000 out of 1000000 steps (53%)
17:08:15:WU02:FS02:0x22:Completed 540000 out of 1000000 steps (54%)
17:09:22:WU02:FS02:0x22:Completed 550000 out of 1000000 steps (55%)
17:10:28:WU02:FS02:0x22:Completed 560000 out of 1000000 steps (56%)
17:11:34:WU02:FS02:0x22:Completed 570000 out of 1000000 steps (57%)
17:12:40:WU02:FS02:0x22:Completed 580000 out of 1000000 steps (58%)
17:12:44:WU03:FS01:0x22:Completed 820000 out of 2000000 steps (41%)
17:13:46:WU02:FS02:0x22:Completed 590000 out of 1000000 steps (59%)
17:14:51:WU02:FS02:0x22:Completed 600000 out of 1000000 steps (60%)
17:15:57:WU02:FS02:0x22:Completed 610000 out of 1000000 steps (61%)
17:17:02:WU02:FS02:0x22:Completed 620000 out of 1000000 steps (62%)
17:18:07:WU02:FS02:0x22:Completed 630000 out of 1000000 steps (63%)
17:19:12:WU02:FS02:0x22:Completed 640000 out of 1000000 steps (64%)
17:19:12:WU03:FS01:0x22:Completed 840000 out of 2000000 steps (42%)
17:20:17:WU02:FS02:0x22:Completed 650000 out of 1000000 steps (65%)
17:21:24:WU02:FS02:0x22:Completed 660000 out of 1000000 steps (66%)
17:22:30:WU02:FS02:0x22:Completed 670000 out of 1000000 steps (67%)
17:23:36:WU02:FS02:0x22:Completed 680000 out of 1000000 steps (68%)
17:24:43:WU02:FS02:0x22:Completed 690000 out of 1000000 steps (69%)
17:25:47:WU03:FS01:0x22:Completed 860000 out of 2000000 steps (43%)
17:25:49:WU02:FS02:0x22:Completed 700000 out of 1000000 steps (70%)
17:26:55:WU02:FS02:0x22:Completed 710000 out of 1000000 steps (71%)
17:28:00:WU02:FS02:0x22:Completed 720000 out of 1000000 steps (72%)
17:29:07:WU02:FS02:0x22:Completed 730000 out of 1000000 steps (73%)
17:30:11:WU02:FS02:0x22:Completed 740000 out of 1000000 steps (74%)
17:31:16:WU02:FS02:0x22:Completed 750000 out of 1000000 steps (75%)
17:32:11:WU03:FS01:0x22:Completed 880000 out of 2000000 steps (44%)
17:32:23:WU02:FS02:0x22:Completed 760000 out of 1000000 steps (76%)
17:33:30:WU02:FS02:0x22:Completed 770000 out of 1000000 steps (77%)
17:34:37:WU02:FS02:0x22:Completed 780000 out of 1000000 steps (78%)
17:35:44:WU02:FS02:0x22:Completed 790000 out of 1000000 steps (79%)
17:36:51:WU02:FS02:0x22:Completed 800000 out of 1000000 steps (80%)
17:37:58:WU02:FS02:0x22:Completed 810000 out of 1000000 steps (81%)
17:38:40:WU03:FS01:0x22:Completed 900000 out of 2000000 steps (45%)
17:39:06:WU02:FS02:0x22:Completed 820000 out of 1000000 steps (82%)
17:40:14:WU02:FS02:0x22:Completed 830000 out of 1000000 steps (83%)
17:41:22:WU02:FS02:0x22:Completed 840000 out of 1000000 steps (84%)
17:42:30:WU02:FS02:0x22:Completed 850000 out of 1000000 steps (85%)
17:43:38:WU02:FS02:0x22:Completed 860000 out of 1000000 steps (86%)
17:44:45:WU02:FS02:0x22:Completed 870000 out of 1000000 steps (87%)
17:45:15:WU03:FS01:0x22:Completed 920000 out of 2000000 steps (46%)
17:45:53:WU02:FS02:0x22:Completed 880000 out of 1000000 steps (88%)
17:47:00:WU02:FS02:0x22:Completed 890000 out of 1000000 steps (89%)
17:48:09:WU02:FS02:0x22:Completed 900000 out of 1000000 steps (90%)
17:49:17:WU02:FS02:0x22:Completed 910000 out of 1000000 steps (91%)
17:50:25:WU02:FS02:0x22:Completed 920000 out of 1000000 steps (92%)
17:51:34:WU02:FS02:0x22:Completed 930000 out of 1000000 steps (93%)
17:51:45:WU03:FS01:0x22:Completed 940000 out of 2000000 steps (47%)
17:52:43:WU02:FS02:0x22:Completed 940000 out of 1000000 steps (94%)
17:53:52:WU02:FS02:0x22:Completed 950000 out of 1000000 steps (95%)
17:55:02:WU02:FS02:0x22:Completed 960000 out of 1000000 steps (96%)
17:56:10:WU02:FS02:0x22:Completed 970000 out of 1000000 steps (97%)
17:57:18:WU02:FS02:0x22:Completed 980000 out of 1000000 steps (98%)
17:58:17:WU03:FS01:0x22:Completed 960000 out of 2000000 steps (48%)
17:58:27:WU02:FS02:0x22:Completed 990000 out of 1000000 steps (99%)
17:58:28:WU00:FS02:Connecting to assign1.foldingathome.org:80
17:58:28:WU00:FS02:Assigned to work server 128.252.203.10
17:58:28:WU00:FS02:Requesting new work unit for slot 02: RUNNING gpu:1:Ellesmere XT [Radeon RX 470/480/570/580/590] from 128.252.203.10
17:58:28:WU00:FS02:Connecting to 128.252.203.10:8080
17:58:32:WU00:FS02:Downloading 5.82MiB
17:58:32:WU00:FS02:Download complete
17:58:32:WU00:FS02:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:11760 run:0 clone:8696 gen:92 core:0x22 unit:0x000000a780fccb0a5e6f22966deb24ea
17:59:35:WU02:FS02:0x22:Completed 1000000 out of 1000000 steps (100%)
17:59:35:WU02:FS02:0x22:Average performance: 251.895 ns/day
17:59:35:WU02:FS02:0x22:Saving result file ..\logfile_01.txt
17:59:35:WU02:FS02:0x22:Saving result file checkpointState.xml
17:59:35:WU02:FS02:0x22:Saving result file globals.csv
17:59:35:WU02:FS02:0x22:Saving result file positions.xtc
17:59:35:WU02:FS02:0x22:Saving result file science.log
17:59:35:WU02:FS02:0x22:Folding@home Core Shutdown: FINISHED_UNIT
17:59:44:WARNING:WU02:FS02:FahCore returned an unknown error code which probably indicates that it crashed
17:59:44:WARNING:WU02:FS02:FahCore returned: UNKNOWN_ENUM (-1073740940 = 0xc0000374)
17:59:44:WU00:FS02:Starting
17:59:44:WU00:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\phil\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 5568 -checkpoint 15 -gpu-vendor amd -opencl-platform 1 -opencl-device 0 -gpu 0
17:59:44:WU00:FS02:Started FahCore on PID 2908
17:59:44:WU00:FS02:Core PID:11536
17:59:44:WU00:FS02:FahCore 0x22 started
17:59:44:WU00:FS02:0x22:*********************** Log Started 2020-06-25T17:59:44Z ***********************
17:59:44:WU00:FS02:0x22:*************************** Core22 Folding@home Core ***************************
17:59:44:WU00:FS02:0x22: Core: Core22
17:59:44:WU00:FS02:0x22: Type: 0x22
17:59:44:WU00:FS02:0x22: Version: 0.0.10
17:59:44:WU00:FS02:0x22: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
17:59:44:WU00:FS02:0x22: Copyright: 2020 foldingathome.org
17:59:44:WU00:FS02:0x22: Homepage: https://foldingathome.org/
17:59:44:WU00:FS02:0x22: Date: Jun 16 2020
17:59:44:WU00:FS02:0x22: Time: 14:33:22
17:59:44:WU00:FS02:0x22: Revision: 147051aad40bcbec7d4b25105bbedfab425f1dc2
17:59:44:WU00:FS02:0x22: Branch: core22-0.0.10
17:59:44:WU00:FS02:0x22: Compiler: Visual C++ 2015
17:59:44:WU00:FS02:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
17:59:44:WU00:FS02:0x22: Platform: win32 10
17:59:44:WU00:FS02:0x22: Bits: 64
17:59:44:WU00:FS02:0x22: Mode: Release
17:59:44:WU00:FS02:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
17:59:44:WU00:FS02:0x22: <peastman@stanford.edu>
17:59:44:WU00:FS02:0x22: Args: -dir 00 -suffix 01 -version 706 -lifeline 2908 -checkpoint 15
17:59:44:WU00:FS02:0x22: -gpu-vendor amd -opencl-platform 1 -opencl-device 0 -gpu 0
17:59:44:WU00:FS02:0x22:************************************ libFAH ************************************
17:59:44:WU00:FS02:0x22: Date: Jun 15 2020
17:59:44:WU00:FS02:0x22: Time: 18:05:04
17:59:44:WU00:FS02:0x22: Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
17:59:44:WU00:FS02:0x22: Branch: HEAD
17:59:44:WU00:FS02:0x22: Compiler: Visual C++ 2015
17:59:44:WU00:FS02:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
17:59:44:WU00:FS02:0x22: Platform: win32 10
17:59:44:WU00:FS02:0x22: Bits: 64
17:59:44:WU00:FS02:0x22: Mode: Release
17:59:44:WU00:FS02:0x22:************************************ CBang *************************************
17:59:44:WU00:FS02:0x22: Date: Jun 16 2020
17:59:44:WU00:FS02:0x22: Time: 14:31:33
17:59:44:WU00:FS02:0x22: Revision: 75fcee0b8e713cb47f5191a3689d5f4f07244c7f
17:59:44:WU00:FS02:0x22: Branch: HEAD
17:59:44:WU00:FS02:0x22: Compiler: Visual C++ 2015
17:59:44:WU00:FS02:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
17:59:44:WU00:FS02:0x22: Platform: win32 10
17:59:44:WU00:FS02:0x22: Bits: 64
17:59:44:WU00:FS02:0x22: Mode: Release
17:59:44:WU00:FS02:0x22:************************************ System ************************************
17:59:44:WU00:FS02:0x22: CPU: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
17:59:44:WU00:FS02:0x22: CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
17:59:44:WU00:FS02:0x22: CPUs: 8
17:59:44:WU00:FS02:0x22: Memory: 15.95GiB
17:59:44:WU00:FS02:0x22:Free Memory: 11.62GiB
17:59:45:WU00:FS02:0x22: Threads: WINDOWS_THREADS
17:59:45:WU00:FS02:0x22: OS Version: 6.2
17:59:45:WU00:FS02:0x22:Has Battery: false
17:59:45:WU00:FS02:0x22: On Battery: false
17:59:45:WU00:FS02:0x22: UTC Offset: -4
17:59:45:WU00:FS02:0x22: PID: 11536
17:59:45:WU00:FS02:0x22: CWD: C:\Users\phil\AppData\Roaming\FAHClient\work
17:59:45:WU00:FS02:0x22:********************************************************************************
17:59:45:WU00:FS02:0x22:Project: 11760 (Run 0, Clone 8696, Gen 92)
17:59:45:WU00:FS02:0x22:Unit: 0x000000a780fccb0a5e6f22966deb24ea
17:59:45:WU00:FS02:0x22:Reading tar file core.xml
17:59:45:WU00:FS02:0x22:Reading tar file integrator.xml
17:59:45:WU00:FS02:0x22:Reading tar file state.xml
17:59:45:WU00:FS02:0x22:Reading tar file system.xml
17:59:45:WU00:FS02:0x22:Digital signatures verified
17:59:45:WU00:FS02:0x22:Folding@home GPU Core22 Folding@home Core
17:59:45:WU00:FS02:0x22:Version 0.0.10
17:59:45:WU00:FS02:0x22: Checkpoint write interval: 100000 steps (5%) [20 total]
17:59:45:WU00:FS02:0x22: JSON viewer frame write interval: 20000 steps (1%) [100 total]
17:59:45:WU00:FS02:0x22: XTC frame write interval: 50000 steps (2.5%) [40 total]
17:59:45:WU00:FS02:0x22: Global context and integrator variables write interval: disabled
18:00:04:WU00:FS02:0x22:Completed 0 out of 2000000 steps (0%)
18:02:26:WU00:FS02:0x22:Completed 20000 out of 2000000 steps (1%)
18:04:46:WU03:FS01:0x22:Completed 980000 out of 2000000 steps (49%)
18:04:49:WU00:FS02:0x22:Completed 40000 out of 2000000 steps (2%)
18:07:11:WU00:FS02:0x22:Completed 60000 out of 2000000 steps (3%)
18:09:34:WU00:FS02:0x22:Completed 80000 out of 2000000 steps (4%)
18:11:18:WU03:FS01:0x22:Completed 1000000 out of 2000000 steps (50%)
18:11:56:WU00:FS02:0x22:Completed 100000 out of 2000000 steps (5%)
18:14:18:WU00:FS02:0x22:Completed 120000 out of 2000000 steps (6%)
18:16:40:WU00:FS02:0x22:Completed 140000 out of 2000000 steps (7%)
18:17:43:WU03:FS01:0x22:Completed 1020000 out of 2000000 steps (51%)
Date of last Work Unit 2020-08-04 22:17:53
Total score 63,300,083
Total WUs 1,013
Overall rank (if points are combined) 15,440 of 2,735,299