two WUs in « Cleanup »

[Mactin]

I've been folding for 10+ years and never seen this before.
I have two WUs in « Cleanup » Status shown in FAHControl (v7.5.1. OSX 10.13.6).

Code: Select all

12:04:23:WU00:FS02:0x22:Completed 3960000 out of 4000000 steps (99%)
12:04:24:WU01:FS02:Connecting to assign-GPU.stanford.edu:80
12:04:24:WU01:FS02:News: 
12:04:24:WU01:FS02:Assigned to work server 13.82.98.119
12:04:24:WU01:FS02:Requesting new work unit for slot 02: RUNNING gpu:2:Baffin [Polaris11] from 13.82.98.119
12:04:24:WU01:FS02:Connecting to 13.82.98.119:8080
12:04:25:WU01:FS02:Downloading 119.45MiB
12:04:31:WU01:FS02:Download 18.84%
12:04:37:WU01:FS02:Download 37.52%
12:04:43:WU01:FS02:Download 57.71%
12:04:49:WU01:FS02:Download 77.02%
12:04:55:WU01:FS02:Download 95.81%
12:04:56:WU01:FS02:Download complete
12:04:56:WU01:FS02:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14539 run:0 clone:480 gen:2 core:0x22 unit:0x000000020d5262775e7bf1fa8ea17d8e
12:07:28:WU00:FS02:0x22:Completed 4000000 out of 4000000 steps (100%)
12:07:29:WU00:FS02:0x22:Saving result file ..\logfile_01.txt
12:07:29:WU00:FS02:0x22:Saving result file checkpointState.xml
12:07:29:WU00:FS02:0x22:Saving result file checkpt.crc
12:07:29:WU00:FS02:0x22:Saving result file positions.xtc
12:07:29:WU00:FS02:0x22:Saving result file science.log
12:07:29:WU00:FS02:0x22:Folding@home Core Shutdown: FINISHED_UNIT
12:07:29:WU00:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
12:07:29:WU00:FS02:Sending unit results: id:00 state:SEND error:NO_ERROR project:14538 run:0 clone:359 gen:6 core:0x22 unit:0x000000070d5262775e7b9cdf6ab52b24
12:07:29:WU00:FS02:Uploading 14.70MiB to 13.82.98.119
12:07:29:WU00:FS02:Connecting to 13.82.98.119:8080
12:07:29:WU01:FS02:Starting
12:07:29:WU01:FS02:Running FahCore: C:\Users\Folding\FAHClient/FAHCoreWrapper.exe C:/Users/Folding/FAHClient/cores/cores.foldingathome.org/v7/win/64bit/beta/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 703 -lifeline 5008 -checkpoint 5 -gpu 1 -gpu-vendor ati
12:07:29:WU01:FS02:Started FahCore on PID 2332
12:07:29:WU01:FS02:Core PID:6276
12:07:29:WU01:FS02:FahCore 0x22 started
12:07:30:WU01:FS02:0x22:*********************** Log Started 2020-03-28T12:07:30Z ***********************
12:07:30:WU01:FS02:0x22:*************************** Core22 Folding@home Core ***************************
12:07:30:WU01:FS02:0x22:       Type: 0x22
12:07:30:WU01:FS02:0x22:       Core: Core22
12:07:30:WU01:FS02:0x22:    Website: https://foldingathome.org/
12:07:30:WU01:FS02:0x22:  Copyright: (c) 2009-2018 foldingathome.org
12:07:30:WU01:FS02:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
12:07:30:WU01:FS02:0x22:             <rafal.wiewiora@choderalab.org>
12:07:30:WU01:FS02:0x22:       Args: -dir 01 -suffix 01 -version 703 -lifeline 2332 -checkpoint 5 -gpu 1
12:07:30:WU01:FS02:0x22:             -gpu-vendor ati
12:07:30:WU01:FS02:0x22:     Config: <none>
12:07:30:WU01:FS02:0x22:************************************ Build *************************************
12:07:30:WU01:FS02:0x22:    Version: 0.0.2
12:07:30:WU01:FS02:0x22:       Date: Dec 6 2019
12:07:30:WU01:FS02:0x22:       Time: 21:30:31
12:07:30:WU01:FS02:0x22: Repository: Git
12:07:30:WU01:FS02:0x22:   Revision: abeb39247cc72df5af0f63723edafadb23d5dfbe
12:07:30:WU01:FS02:0x22:     Branch: HEAD
12:07:30:WU01:FS02:0x22:   Compiler: Visual C++ 2008
12:07:30:WU01:FS02:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
12:07:30:WU01:FS02:0x22:   Platform: win32 10
12:07:30:WU01:FS02:0x22:       Bits: 64
12:07:30:WU01:FS02:0x22:       Mode: Release
12:07:30:WU01:FS02:0x22:************************************ System ************************************
12:07:30:WU01:FS02:0x22:        CPU: Intel(R) Core(TM) i7 CPU X 980 @ 3.33GHz
12:07:30:WU01:FS02:0x22:     CPU ID: GenuineIntel Family 6 Model 44 Stepping 2
12:07:30:WU01:FS02:0x22:       CPUs: 12
12:07:30:WU01:FS02:0x22:     Memory: 12.00GiB
12:07:30:WU01:FS02:0x22:Free Memory: 8.14GiB
12:07:30:WU01:FS02:0x22:    Threads: WINDOWS_THREADS
12:07:30:WU01:FS02:0x22: OS Version: 6.1
12:07:30:WU01:FS02:0x22:Has Battery: false
12:07:30:WU01:FS02:0x22: On Battery: false
12:07:30:WU01:FS02:0x22: UTC Offset: -4
12:07:30:WU01:FS02:0x22:        PID: 6276
12:07:30:WU01:FS02:0x22:        CWD: C:\Users\Folding\FAHClient\work
12:07:30:WU01:FS02:0x22:         OS: Windows 7 Ultimate Service Pack 1
12:07:30:WU01:FS02:0x22:    OS Arch: AMD64
12:07:30:WU01:FS02:0x22:********************************************************************************
12:07:30:WU01:FS02:0x22:Project: 14539 (Run 0, Clone 480, Gen 2)
12:07:30:WU01:FS02:0x22:Unit: 0x000000020d5262775e7bf1fa8ea17d8e
12:07:30:WU01:FS02:0x22:Reading tar file core.xml
12:07:30:WU01:FS02:0x22:Reading tar file integrator.xml
12:07:30:WU01:FS02:0x22:Reading tar file state.xml
12:07:31:WU01:FS02:0x22:Reading tar file system.xml
12:07:33:WU01:FS02:0x22:Digital signatures verified
12:07:33:WU01:FS02:0x22:Folding@home GPU Core22 Folding@home Core
12:07:33:WU01:FS02:0x22:Version 0.0.2
12:07:35:WU00:FS02:Upload 15.74%
12:07:41:WU00:FS02:Upload 33.60%
12:07:47:WU00:FS02:Upload 45.08%
12:07:52:WU01:FS02:0x22:ERROR:exception: Error invoking kernel sortShortList: clEnqueueNDRangeKernel (-5)
12:07:52:WU01:FS02:0x22:Saving result file ..\logfile_01.txt
12:07:52:WU01:FS02:0x22:Saving result file science.log
12:07:52:WU01:FS02:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
12:07:53:WU00:FS02:Upload 55.71%
12:07:53:WARNING:WU01:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
12:07:53:WU01:FS02:Sending unit results: id:01 state:SEND error:FAULTY project:14539 run:0 clone:480 gen:2 core:0x22 unit:0x000000020d5262775e7bf1fa8ea17d8e
12:07:53:WU01:FS02:Uploading 8.00KiB to 13.82.98.119
12:07:53:WU01:FS02:Connecting to 13.82.98.119:8080
12:07:53:WU01:FS02:Upload complete
12:07:53:WU01:FS02:Server responded WORK_ACK (400)
12:07:53:WU01:FS02:Cleaning up
12:07:53:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:07:53:WU03:FS02:Connecting to assign-GPU.stanford.edu:80
12:07:53:WU01:FS02:Cleaning up
12:07:53:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:07:54:WU03:FS02:News: 
12:07:54:WU03:FS02:Assigned to work server 13.82.98.119
12:07:54:WU03:FS02:Requesting new work unit for slot 02: READY gpu:2:Baffin [Polaris11] from 13.82.98.119
12:07:54:WU03:FS02:Connecting to 13.82.98.119:8080
12:07:54:WU03:FS02:Downloading 13.26MiB
12:07:59:WU00:FS02:Upload 64.22%
12:07:59:WU03:FS02:Download complete
12:07:59:WU03:FS02:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:14538 run:0 clone:318 gen:11 core:0x22 unit:0x0000000b0d5262775e7b9cdf546b235e
12:07:59:WU03:FS02:Starting
12:07:59:WU03:FS02:Running FahCore: C:\Users\Folding\FAHClient/FAHCoreWrapper.exe C:/Users/Folding/FAHClient/cores/cores.foldingathome.org/v7/win/64bit/beta/Core_22.fah/FahCore_22.exe -dir 03 -suffix 01 -version 703 -lifeline 5008 -checkpoint 5 -gpu 1 -gpu-vendor ati
12:07:59:WU03:FS02:Started FahCore on PID 8644
12:07:59:WU03:FS02:Core PID:6460
12:07:59:WU03:FS02:FahCore 0x22 started
12:07:59:WU03:FS02:0x22:*********************** Log Started 2020-03-28T12:07:59Z ***********************
12:07:59:WU03:FS02:0x22:*************************** Core22 Folding@home Core ***************************
12:07:59:WU03:FS02:0x22:       Type: 0x22
12:07:59:WU03:FS02:0x22:       Core: Core22
12:07:59:WU03:FS02:0x22:    Website: https://foldingathome.org/
12:07:59:WU03:FS02:0x22:  Copyright: (c) 2009-2018 foldingathome.org
12:07:59:WU03:FS02:0x22:     Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
12:07:59:WU03:FS02:0x22:             <rafal.wiewiora@choderalab.org>
12:07:59:WU03:FS02:0x22:       Args: -dir 03 -suffix 01 -version 703 -lifeline 8644 -checkpoint 5 -gpu 1
12:07:59:WU03:FS02:0x22:             -gpu-vendor ati
12:07:59:WU03:FS02:0x22:     Config: <none>
12:07:59:WU03:FS02:0x22:************************************ Build *************************************
12:07:59:WU03:FS02:0x22:    Version: 0.0.2
12:07:59:WU03:FS02:0x22:       Date: Dec 6 2019
12:07:59:WU03:FS02:0x22:       Time: 21:30:31
12:07:59:WU03:FS02:0x22: Repository: Git
12:07:59:WU03:FS02:0x22:   Revision: abeb39247cc72df5af0f63723edafadb23d5dfbe
12:07:59:WU03:FS02:0x22:     Branch: HEAD
12:07:59:WU03:FS02:0x22:   Compiler: Visual C++ 2008
12:07:59:WU03:FS02:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
12:07:59:WU03:FS02:0x22:   Platform: win32 10
12:07:59:WU03:FS02:0x22:       Bits: 64
12:07:59:WU03:FS02:0x22:       Mode: Release
12:07:59:WU03:FS02:0x22:************************************ System ************************************
12:07:59:WU03:FS02:0x22:        CPU: Intel(R) Core(TM) i7 CPU X 980 @ 3.33GHz
12:07:59:WU03:FS02:0x22:     CPU ID: GenuineIntel Family 6 Model 44 Stepping 2
12:07:59:WU03:FS02:0x22:       CPUs: 12
12:07:59:WU03:FS02:0x22:     Memory: 12.00GiB
12:07:59:WU03:FS02:0x22:Free Memory: 7.99GiB
12:07:59:WU03:FS02:0x22:    Threads: WINDOWS_THREADS
12:07:59:WU03:FS02:0x22: OS Version: 6.1
12:07:59:WU03:FS02:0x22:Has Battery: false
12:07:59:WU03:FS02:0x22: On Battery: false
12:07:59:WU03:FS02:0x22: UTC Offset: -4
12:07:59:WU03:FS02:0x22:        PID: 6460
12:07:59:WU03:FS02:0x22:        CWD: C:\Users\Folding\FAHClient\work
12:07:59:WU03:FS02:0x22:         OS: Windows 7 Ultimate Service Pack 1
12:07:59:WU03:FS02:0x22:    OS Arch: AMD64
12:07:59:WU03:FS02:0x22:********************************************************************************
12:07:59:WU03:FS02:0x22:Project: 14538 (Run 0, Clone 318, Gen 11)
12:07:59:WU03:FS02:0x22:Unit: 0x0000000b0d5262775e7b9cdf546b235e
12:07:59:WU03:FS02:0x22:Reading tar file core.xml
12:07:59:WU03:FS02:0x22:Reading tar file integrator.xml
12:07:59:WU03:FS02:0x22:Reading tar file state.xml
12:07:59:WU03:FS02:0x22:Reading tar file system.xml
12:07:59:WU03:FS02:0x22:Digital signatures verified
12:07:59:WU03:FS02:0x22:Folding@home GPU Core22 Folding@home Core
12:07:59:WU03:FS02:0x22:Version 0.0.2
12:08:05:WU00:FS02:Upload 75.27%
12:08:06:WU03:FS02:0x22:Completed 0 out of 4000000 steps (0%)
12:08:06:WU03:FS02:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
12:08:11:WU00:FS02:Upload 86.33%
12:08:17:WU00:FS02:Upload 97.39%
12:08:18:WU00:FS02:Upload complete
12:08:18:WU00:FS02:Server responded WORK_ACK (400)
12:08:18:WU00:FS02:Final credit estimate, 27958.00 points
12:08:18:WU00:FS02:Cleaning up
12:08:18:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:08:19:WU00:FS02:Cleaning up
12:08:19:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:08:54:WU01:FS02:Cleaning up
12:08:54:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:09:19:WU00:FS02:Cleaning up
12:09:19:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:10:31:WU01:FS02:Cleaning up
12:10:31:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:10:56:WU00:FS02:Cleaning up
12:10:56:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:11:12:WU03:FS02:0x22:Completed 40000 out of 4000000 steps (1%)
12:13:08:WU01:FS02:Cleaning up
12:13:08:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:13:33:WU00:FS02:Cleaning up
12:13:33:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:14:18:WU03:FS02:0x22:Completed 80000 out of 4000000 steps (2%)
12:17:22:WU01:FS02:Cleaning up
12:17:22:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:17:23:WU03:FS02:0x22:Completed 120000 out of 4000000 steps (3%)
12:17:47:WU00:FS02:Cleaning up
12:17:47:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:20:28:WU03:FS02:0x22:Completed 160000 out of 4000000 steps (4%)
12:23:33:WU03:FS02:0x22:Completed 200000 out of 4000000 steps (5%)
12:24:14:WU01:FS02:Cleaning up
12:24:14:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:24:39:WU00:FS02:Cleaning up
12:24:39:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:26:39:WU03:FS02:0x22:Completed 240000 out of 4000000 steps (6%)
12:29:45:WU03:FS02:0x22:Completed 280000 out of 4000000 steps (7%)
12:32:50:WU03:FS02:0x22:Completed 320000 out of 4000000 steps (8%)
12:35:19:WU01:FS02:Cleaning up
12:35:19:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:35:44:WU00:FS02:Cleaning up
12:35:44:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:35:56:WU03:FS02:0x22:Completed 360000 out of 4000000 steps (9%)
12:39:01:WU03:FS02:0x22:Completed 400000 out of 4000000 steps (10%)
12:42:06:WU03:FS02:0x22:Completed 440000 out of 4000000 steps (11%)
12:45:12:WU03:FS02:0x22:Completed 480000 out of 4000000 steps (12%)
12:48:17:WU03:FS02:0x22:Completed 520000 out of 4000000 steps (13%)
12:51:23:WU03:FS02:0x22:Completed 560000 out of 4000000 steps (14%)
12:53:16:WU01:FS02:Cleaning up
12:53:16:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:53:41:WU00:FS02:Cleaning up
12:53:41:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:54:28:WU03:FS02:0x22:Completed 600000 out of 4000000 steps (15%)
12:57:33:WU03:FS02:0x22:Completed 640000 out of 4000000 steps (16%)
13:00:39:WU03:FS02:0x22:Completed 680000 out of 4000000 steps (17%)
13:03:45:WU03:FS02:0x22:Completed 720000 out of 4000000 steps (18%)
13:06:50:WU03:FS02:0x22:Completed 760000 out of 4000000 steps (19%)
13:09:55:WU03:FS02:0x22:Completed 800000 out of 4000000 steps (20%)
13:13:01:WU03:FS02:0x22:Completed 840000 out of 4000000 steps (21%)
13:16:07:WU03:FS02:0x22:Completed 880000 out of 4000000 steps (22%)
13:19:13:WU03:FS02:0x22:Completed 920000 out of 4000000 steps (23%)
13:22:18:WU03:FS02:0x22:Completed 960000 out of 4000000 steps (24%)
13:22:18:WU01:FS02:Cleaning up
13:22:18:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
13:22:43:WU00:FS02:Cleaning up
13:22:43:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"

To me it looks like :
- It finished and eventually sent in p14538-r0-c239-g6 (WU00).
- It downloaded p14539-r0-c480-g2 and it failed as soon as it started (WU01).
- It downloaded p14538-r0-c318-g11 and now folds successfully (WU03).
- It's trying to cleanup the previous two and i's not making it (WU00 & WU01).

BTW, I have had several p14539 fail on both my (old) GPUs.

[jonault]

I'm not sure what other process has a hold on those log files. Have you tried restarting your Mac?

As for your GPU, GPU folding is not supported under Mac OS X. You might be able to use your GPUs if you have a Mac with a discrete GPU (iMacs, Mac Pros and some MacBook Pros have these) and you booted your Mac into Linux or Windows, but you should tell us more about your specific hardware first. If your Mac is so old that it can't be upgraded to Mojave or Catalina, there's a good chance the GPUs aren't supported anymore.

[Mactin]

I'm sorry, there is a little mixup on my part.
The folding is done a Win7 PC, I'm running it just to fold. I'm monitoring from my Mac.
My original msg stands.

[jonault]

OK, then have you tried restarting your Windows PC? Hopefully that will clear out whatever other process has grabbed onto those log files.

[bruce]

FAHClient should cleanup WUs that are stuck for one reason or another when the WU espires. If some other process has opened one of the files being cleaned up for update, the cleanup process cannot be completed The reboot will allow the cleanup to proceed because whatever other process will have been aborted, no matter why it originally opened the file.

[Mactin]

jonault,
bruce,
Thanks, all cleaned-up with restart. Never saw this before. Before I did this my other slot completed and did the same thing.
Take care and Stay@Home in addition to Folding@Home

[bruce]

(Yep. I'm hunkered down at home here in California.)

[dawoo]

Hi.
Thought I'd add to this thread as I too experience the same issue. My OS is Windows 10. The Cleanup task always says it has 2 days ETA and never decreases and until it is removed no further folding will happen.
My workaround has been to quit the application then relaunch, not ideal, but it's the only way I can get the task to remove itself from the Work Queue.

Thanks, D.

[PantherX]

Welcome to the F@H Forum dawoo,

Can you please post the log file? Ensure you include the first 100 lines which will inform us of what the system configuration is and what the client settings are. If you require guidance, please view this topic: viewtopic.php?f=24&t=26036