Project 16959: multiple WUs not credited (129.32.209.205)

Moderators: Site Moderators, FAHC Science Team

Post Reply
PaulTV
Posts: 208
Joined: Mon Jan 25, 2021 4:53 pm
Location: Netherlands

Project 16959: multiple WUs not credited (129.32.209.205)

Post by PaulTV »

Hi,

Credits for several project 16959 WUs seem to be stuck. Results for all those jobs are uploaded to the same server, may be related.
Those are my currently uncredited jobs (for this project alone, that is):

Code: Select all

Assigned (UTC)       Type  Project  Run      Clone    Gen      Work server      Collection server
-------------------  ----  -------  -------  -------  -------  ---------------  -----------------
2021-04-16 19:43:33  CPU   16959    43       75       22                        
2021-04-16 22:38:52  CPU   16959    20       747      6                         
2021-04-17 05:43:45  CPU   16959    16       670      13       129.32.209.205   129.32.209.207
2021-04-17 08:36:04  CPU   16959    34       135      19       129.32.209.205   129.32.209.200
2021-04-17 11:46:32  CPU   16959    28       639      5        129.32.209.205   129.32.209.200
2021-04-17 16:07:37  CPU   16959    15       169      9        129.32.209.205   129.32.209.202
2021-04-17 19:06:09  CPU   16959    37       996      2        129.32.209.205   129.32.209.202
2021-04-20 04:15:28  CPU   16959    22       982      5        129.32.209.205   129.32.209.201
2021-04-20 23:03:07  CPU   16959    13       555      16       129.32.209.205   129.32.209.206
2021-04-21 01:48:53  CPU   16959    15       910      2        129.32.209.205   129.32.209.201
The relevant logging:

22:40:34:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:16959 run:43 clone:75 gen:22 core:0xa8 unit:0x0000004b000000160000423f0000002b
22:40:34:WU01:FS00:Uploading 5.12MiB to 129.32.209.205
22:40:46:WU01:FS00:Server responded WORK_ACK (400)
01:28:02:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:16959 run:20 clone:747 gen:6 core:0xa8 unit:0x000002eb000000060000423f00000014
01:28:02:WU02:FS00:Uploading 5.12MiB to 129.32.209.205
01:28:14:WU02:FS00:Server responded WORK_ACK (400)
08:37:48:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:16959 run:16 clone:670 gen:13 core:0xa8 unit:0x0000029e0000000d0000423f00000010
08:37:48:WU01:FS00:Uploading 5.10MiB to 129.32.209.205
08:38:04:WU01:FS00:Server responded WORK_ACK (400)
11:30:12:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16959 run:34 clone:135 gen:19 core:0xa8 unit:0x00000087000000130000423f00000022
11:30:12:WU00:FS00:Uploading 5.12MiB to 129.32.209.205
11:30:21:WU00:FS00:Server responded WORK_ACK (400)
14:39:42:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16959 run:28 clone:639 gen:5 core:0xa8 unit:0x0000027f000000050000423f0000001c
14:39:42:WU00:FS00:Uploading 5.12MiB to 129.32.209.205
14:39:54:WU00:FS00:Server responded WORK_ACK (400)
19:07:59:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:16959 run:15 clone:169 gen:9 core:0xa8 unit:0x000000a9000000090000423f0000000f
19:07:59:WU02:FS00:Uploading 5.12MiB to 129.32.209.205
19:08:15:WU02:FS00:Server responded WORK_ACK (400)
22:09:09:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16959 run:37 clone:996 gen:2 core:0xa8 unit:0x000003e4000000020000423f00000025
22:09:09:WU00:FS00:Uploading 5.12MiB to 129.32.209.205
22:09:18:WU00:FS00:Server responded WORK_ACK (400)
16:07:52:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:16959 run:22 clone:982 gen:5 core:0xa8 unit:0x000003d6000000050000423f00000016
16:07:52:WU01:FS00:Uploading 5.13MiB to 129.32.209.205
16:08:05:WU01:FS00:Server responded WORK_ACK (400)
01:50:32:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:16959 run:13 clone:555 gen:16 core:0xa8 unit:0x0000022b000000100000423f0000000d
01:50:32:WU01:FS00:Uploading 5.12MiB to 129.32.209.205
01:50:46:WU01:FS00:Server responded WORK_ACK (400)
04:37:04:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:16959 run:15 clone:910 gen:2 core:0xa8 unit:0x0000038e000000020000423f0000000f
04:37:04:WU02:FS00:Uploading 5.12MiB to 129.32.209.205
04:37:16:WU02:FS00:Server responded WORK_ACK (400)

Thanks,
Paul
Last edited by PaulTV on Wed Apr 21, 2021 4:32 pm, edited 2 times in total.
Image

Ryzen 5800X / RTX 4090 / Windows 11
Ryzen 5600X / RTX 3070 Ti / Ubuntu 22.04
Ryzen 5600 / RTX 3060 Ti / Windows 11
Top
comixgoddess
Posts: 85
Joined: Wed Apr 08, 2020 9:57 pm
Location: Pacific Northwest

Re: Project 16959: multiple WUs not credited

Post by comixgoddess »

I am experiencing the same issue. My work units were uploaded to 129.32.209.205. In all instances, I received the "WORK_ACK" response and a final credit estimate, but the work units couldn't be found on the WU Status page.

2021-04-17 10:41:38 - project:16959 run:5 clone:949 gen:5
2021-04-21 02:03:55 - project:16959 run:28 clone:367 gen:17
2021-04-21 13:51:41 - project:16959 run:30 clone:531 gen:11

UPDATE: Add these work units to the list

2021-04-22 07:20:52 - project:16959 run:16 clone:493 gen:10
2021-04-23 07:02:30 - project:16959 run:28 clone:767 gen:9
2021-04-23 18:29:32 - project:16959 run:6 clone:575 gen:19
Last edited by comixgoddess on Sat Apr 24, 2021 6:24 pm, edited 1 time in total.
Image
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Project 16959: multiple WUs not credited (129.32.209.20)

Post by bruce »

I've added the server's IP address to the title of the first post. Problems like this one tend to be related to any project on the offending server.
Top
goodyca
Posts: 187
Joined: Sun Dec 02, 2007 12:36 pm

Re: Project 16959: multiple WUs not credited (129.32.209.205

Post by goodyca »

I have also seen the same issue. The following WU's were successfully uploaded to the subject server, but no credit was issued.

Completion Time and Project Description

04/20/21 01:48 PM Project: 16959 (Run 33, Clone 658, Gen 5)
04/21/21 06:36 AM Project: 16959 (Run 37, Clone 961, Gen 5)
04/21/21 12:59 PM Project: 16959 (Run 17, Clone 188, Gen 35)
04/21/21 02:03 PM Project: 16959 (Run 16, Clone 702, Gen 7)
Top
bojan
Posts: 1
Joined: Sat Apr 24, 2021 7:42 pm

Re: Project 16959: multiple WUs not credited (129.32.209.205

Post by bojan »

Hi,
I am also seeing this a lot recently. I counted at least 6 WUs shown in the logs as finished and sent successfully with an ack and a credit estimate, all showing as missing and not actually credited.
What's more peculiar is that I can't seem to avoid getting these WUs.. 4 out of my 6 clients are currently folding these units.
Do we know if it's a WS/CS-related issue that will take some time to clear up?
In the meantime, is there any config way to avoid these units?

Thanks!
Top
SilvioMartin
Posts: 30
Joined: Thu Sep 24, 2020 6:06 pm
Hardware configuration: iMac 2017 Intel Quad-Core i5 3,4 GHz, 8 GB RAM, Radeon Pro 560 4 GB, typically with the latest macOS update. 5 Raspberry Pi 4B (2 GB).
Location: Oberhausen, Germany
Contact:
Contact SilvioMartin

Re: Project 16959: multiple WUs not credited (129.32.209.205

Post by SilvioMartin »

Here is some more debugging matter. Logfile of project 16959 (run 39, clone 729, gen 10) up to "Cleaning Up". The WU status says that this WU wasn't finished yet. It should have been finished by SilvioMartin.

https://apps.foldingathome.org/wu?p=169 ... c=729&g=10

I can provide more examples if this is helpful for debugging.

Code: Select all

*********************** Log Started 2021-04-23T08:17:17Z ***********************
08:17:17:******************************* libFAH ********************************
08:17:17:       Date: Oct 20 2020
08:17:17:       Time: 20:36:48
08:17:17:   Revision: 5ca109d295a6245e2a2f590b3d0085ad5e567aeb
08:17:17:     Branch: master
08:17:17:   Compiler: GNU 8.3.0
08:17:17:    Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
08:17:17:             -fdata-sections -O3 -funroll-loops -fno-pie
08:17:17:   Platform: linux2 4.19.0-9-arm64
08:17:17:       Bits: 64
08:17:17:       Mode: Release
08:17:17:****************************** FAHClient ******************************
08:17:17:    Version: 7.6.21
08:17:17:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
08:17:17:  Copyright: 2020 foldingathome.org
08:17:17:   Homepage: https://foldingathome.org/
08:17:17:       Date: Oct 20 2020
08:17:17:       Time: 20:39:10
08:17:17:   Revision: 6efbf0e138e22d3963e6a291f78dcb9c6422a278
08:17:17:     Branch: master
08:17:17:   Compiler: GNU 8.3.0
08:17:17:    Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
08:17:17:             -fdata-sections -O3 -funroll-loops -fno-pie
08:17:17:   Platform: linux2 4.19.0-9-arm64
08:17:17:       Bits: 64
08:17:17:       Mode: Release
08:17:17:       Args: --child /etc/fahclient/config.xml --run-as fahclient
08:17:17:             --pid-file=/var/run/fahclient.pid --daemon
08:17:17:     Config: /etc/fahclient/config.xml
08:17:17:******************************** CBang ********************************
08:17:17:       Date: Oct 20 2020
08:17:17:       Time: 18:38:03
08:17:17:   Revision: 7e4ce85225d7eaeb775e87c31740181ca603de60
08:17:17:     Branch: master
08:17:17:   Compiler: GNU 8.3.0
08:17:17:    Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
08:17:17:             -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
08:17:17:   Platform: linux2 4.19.0-9-arm64
08:17:17:       Bits: 64
08:17:17:       Mode: Release
08:17:17:******************************* System ********************************
08:17:17:        CPU: Unknown
08:17:17:     CPU ID: 
08:17:17:       CPUs: 4
08:17:17:     Memory: 1.81GiB
08:17:17:Free Memory: 1.58GiB
08:17:17:    Threads: POSIX_THREADS
08:17:17: OS Version: 5.4
08:17:17:Has Battery: false
08:17:17: On Battery: false
08:17:17: UTC Offset: 2
08:17:17:        PID: 682
08:17:17:        CWD: /var/lib/fahclient
08:17:17:         OS: Linux 5.4.79-v8+ aarch64
08:17:17:    OS Arch: ARM64
08:17:17:       GPUs: 0
08:17:17:       CUDA: Not detected: Failed to open dynamic library 'libcuda.so':
08:17:17:             libcuda.so: cannot open shared object file: No such file or
08:17:17:             directory
08:17:17:     OpenCL: Not detected: Failed to open dynamic library 'libOpenCL.so':
08:17:17:             libOpenCL.so: cannot open shared object file: No such file or
08:17:17:             directory
08:17:17:***********************************************************************
08:17:17:<config>
08:17:17:  <!-- Client Control -->
08:17:17:  <fold-anon v='true'/>
08:17:17:
08:17:17:  <!-- Folding Slot Configuration -->
08:17:17:  <gpu v='false'/>
08:17:17:
08:17:17:  <!-- Slot Control -->
08:17:17:  <power v='full'/>
08:17:17:
08:17:17:  <!-- User Information -->
08:17:17:  <passkey v='*****'/>
08:17:17:  <user v='SilvioMartin'/>
08:17:17:
08:17:17:  <!-- Folding Slots -->
08:17:17:  <slot id='0' type='CPU'/>
08:17:17:</config>
08:17:17:Trying to access database...
08:17:17:Successfully acquired database lock
08:17:17:FS00:Initialized folding slot 00: cpu:4
08:17:17:WU00:FS00:Starting
08:17:17:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit-aarch64/a8-0.0.12/Core_a8.fah/FahCore_a8 -dir 00 -suffix 01 -version 706 -lifeline 682 -checkpoint 15 -np 4
08:17:17:WU00:FS00:Started FahCore on PID 696
08:17:17:WU00:FS00:Core PID:701
08:17:17:WU00:FS00:FahCore 0xa8 started
08:17:18:WU00:FS00:0xa8:*********************** Log Started 2021-04-23T08:17:18Z ***********************
08:17:18:WU00:FS00:0xa8:************************** Gromacs Folding@home Core ***************************
08:17:18:WU00:FS00:0xa8:       Core: Gromacs
08:17:18:WU00:FS00:0xa8:       Type: 0xa8
08:17:18:WU00:FS00:0xa8:    Version: 0.0.12
08:17:18:WU00:FS00:0xa8:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
08:17:18:WU00:FS00:0xa8:  Copyright: 2020 foldingathome.org
08:17:18:WU00:FS00:0xa8:   Homepage: https://foldingathome.org/
08:17:18:WU00:FS00:0xa8:       Date: Jan 16 2021
08:17:18:WU00:FS00:0xa8:       Time: 19:29:29
08:17:18:WU00:FS00:0xa8:   Compiler: GNU 8.3.0
08:17:18:WU00:FS00:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
08:17:18:WU00:FS00:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie
08:17:18:WU00:FS00:0xa8:   Platform: linux2 4.15.0-128-generic
08:17:18:WU00:FS00:0xa8:       Bits: 64
08:17:18:WU00:FS00:0xa8:       Mode: Release
08:17:18:WU00:FS00:0xa8:       SIMD: arm_neon_asimd
08:17:18:WU00:FS00:0xa8:     OpenMP: ON
08:17:18:WU00:FS00:0xa8:       CUDA: OFF
08:17:18:WU00:FS00:0xa8:       Args: -dir 00 -suffix 01 -version 706 -lifeline 696 -checkpoint 15 -np 4
08:17:18:WU00:FS00:0xa8:************************************ libFAH ************************************
08:17:18:WU00:FS00:0xa8:       Date: Jan 16 2021
08:17:18:WU00:FS00:0xa8:       Time: 19:29:00
08:17:18:WU00:FS00:0xa8:   Compiler: GNU 8.3.0
08:17:18:WU00:FS00:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
08:17:18:WU00:FS00:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie
08:17:18:WU00:FS00:0xa8:   Platform: linux2 4.15.0-128-generic
08:17:18:WU00:FS00:0xa8:       Bits: 64
08:17:18:WU00:FS00:0xa8:       Mode: Release
08:17:18:WU00:FS00:0xa8:************************************ CBang *************************************
08:17:18:WU00:FS00:0xa8:       Date: Jan 16 2021
08:17:18:WU00:FS00:0xa8:       Time: 19:28:44
08:17:18:WU00:FS00:0xa8:   Compiler: GNU 8.3.0
08:17:18:WU00:FS00:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
08:17:18:WU00:FS00:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
08:17:18:WU00:FS00:0xa8:   Platform: linux2 4.15.0-128-generic
08:17:18:WU00:FS00:0xa8:       Bits: 64
08:17:18:WU00:FS00:0xa8:       Mode: Release
08:17:18:WU00:FS00:0xa8:************************************ System ************************************
08:17:18:WU00:FS00:0xa8:        CPU: Cortex-A
08:17:18:WU00:FS00:0xa8:     CPU ID: Arm Family 8 Model 72 Stepping 3
08:17:18:WU00:FS00:0xa8:       CPUs: 4
08:17:18:WU00:FS00:0xa8:     Memory: 1.81GiB
08:17:18:WU00:FS00:0xa8:Free Memory: 1.55GiB
08:17:18:WU00:FS00:0xa8:    Threads: POSIX_THREADS
08:17:18:WU00:FS00:0xa8: OS Version: 5.4
08:17:18:WU00:FS00:0xa8:Has Battery: false
08:17:18:WU00:FS00:0xa8: On Battery: false
08:17:18:WU00:FS00:0xa8: UTC Offset: 2
08:17:18:WU00:FS00:0xa8:        PID: 701
08:17:18:WU00:FS00:0xa8:        CWD: /var/lib/fahclient/work
08:17:18:WU00:FS00:0xa8:********************************************************************************
08:17:18:WU00:FS00:0xa8:Project: 16959 (Run 39, Clone 729, Gen 10)
08:17:18:WU00:FS00:0xa8:Unit: 0x00000000000000000000000000000000
08:17:18:WU00:FS00:0xa8:Digital signatures verified
08:17:18:WU00:FS00:0xa8:Calling: mdrun -c frame10.gro -s frame10.tpr -x frame10.xtc -cpi state.cpt -cpt 15 -nt 4 -ntmpi 1
08:17:19:WU00:FS00:0xa8:Steps: first=25000000 total=27500000
08:17:29:WU00:FS00:0xa8:Completed 965851 out of 2500000 steps (38%)
[93m08:51:59:WARNING:WU00:FS00:Detected clock skew (34 mins 25 secs), I/O delay, laptop hibernation or other slowdown noted, adjusting time estimates[0m
09:09:16:WU00:FS00:0xa8:Completed 975000 out of 2500000 steps (39%)
09:56:45:WU00:FS00:0xa8:Completed 1000000 out of 2500000 steps (40%)
10:44:15:WU00:FS00:0xa8:Completed 1025000 out of 2500000 steps (41%)
11:31:45:WU00:FS00:0xa8:Completed 1050000 out of 2500000 steps (42%)
12:19:07:WU00:FS00:0xa8:Completed 1075000 out of 2500000 steps (43%)
13:06:36:WU00:FS00:0xa8:Completed 1100000 out of 2500000 steps (44%)
13:54:07:WU00:FS00:0xa8:Completed 1125000 out of 2500000 steps (45%)
******************************* Date: 2021-04-23 *******************************
14:41:36:WU00:FS00:0xa8:Completed 1150000 out of 2500000 steps (46%)
15:29:03:WU00:FS00:0xa8:Completed 1175000 out of 2500000 steps (47%)
16:16:23:WU00:FS00:0xa8:Completed 1200000 out of 2500000 steps (48%)
17:03:51:WU00:FS00:0xa8:Completed 1225000 out of 2500000 steps (49%)
17:51:19:WU00:FS00:0xa8:Completed 1250000 out of 2500000 steps (50%)
18:38:48:WU00:FS00:0xa8:Completed 1275000 out of 2500000 steps (51%)
19:26:14:WU00:FS00:0xa8:Completed 1300000 out of 2500000 steps (52%)
20:13:37:WU00:FS00:0xa8:Completed 1325000 out of 2500000 steps (53%)
******************************* Date: 2021-04-23 *******************************
21:01:08:WU00:FS00:0xa8:Completed 1350000 out of 2500000 steps (54%)
21:48:38:WU00:FS00:0xa8:Completed 1375000 out of 2500000 steps (55%)
22:36:07:WU00:FS00:0xa8:Completed 1400000 out of 2500000 steps (56%)
23:23:28:WU00:FS00:0xa8:Completed 1425000 out of 2500000 steps (57%)
00:11:01:WU00:FS00:0xa8:Completed 1450000 out of 2500000 steps (58%)
00:58:29:WU00:FS00:0xa8:Completed 1475000 out of 2500000 steps (59%)
01:45:57:WU00:FS00:0xa8:Completed 1500000 out of 2500000 steps (60%)
02:33:26:WU00:FS00:0xa8:Completed 1525000 out of 2500000 steps (61%)
******************************* Date: 2021-04-24 *******************************
03:20:46:WU00:FS00:0xa8:Completed 1550000 out of 2500000 steps (62%)
04:08:14:WU00:FS00:0xa8:Completed 1575000 out of 2500000 steps (63%)
04:55:47:WU00:FS00:0xa8:Completed 1600000 out of 2500000 steps (64%)
05:43:17:WU00:FS00:0xa8:Completed 1625000 out of 2500000 steps (65%)
06:30:46:WU00:FS00:0xa8:Completed 1650000 out of 2500000 steps (66%)
07:18:10:WU00:FS00:0xa8:Completed 1675000 out of 2500000 steps (67%)
08:05:44:WU00:FS00:0xa8:Completed 1700000 out of 2500000 steps (68%)
08:53:14:WU00:FS00:0xa8:Completed 1725000 out of 2500000 steps (69%)
******************************* Date: 2021-04-24 *******************************
09:40:43:WU00:FS00:0xa8:Completed 1750000 out of 2500000 steps (70%)
10:28:15:WU00:FS00:0xa8:Completed 1775000 out of 2500000 steps (71%)
11:15:38:WU00:FS00:0xa8:Completed 1800000 out of 2500000 steps (72%)
12:03:08:WU00:FS00:0xa8:Completed 1825000 out of 2500000 steps (73%)
12:50:38:WU00:FS00:0xa8:Completed 1850000 out of 2500000 steps (74%)
13:38:06:WU00:FS00:0xa8:Completed 1875000 out of 2500000 steps (75%)
14:25:30:WU00:FS00:0xa8:Completed 1900000 out of 2500000 steps (76%)
15:12:54:WU00:FS00:0xa8:Completed 1925000 out of 2500000 steps (77%)
******************************* Date: 2021-04-24 *******************************
16:00:26:WU00:FS00:0xa8:Completed 1950000 out of 2500000 steps (78%)
16:47:58:WU00:FS00:0xa8:Completed 1975000 out of 2500000 steps (79%)
17:35:26:WU00:FS00:0xa8:Completed 2000000 out of 2500000 steps (80%)
18:22:47:WU00:FS00:0xa8:Completed 2025000 out of 2500000 steps (81%)
19:10:16:WU00:FS00:0xa8:Completed 2050000 out of 2500000 steps (82%)
19:57:46:WU00:FS00:0xa8:Completed 2075000 out of 2500000 steps (83%)
20:45:15:WU00:FS00:0xa8:Completed 2100000 out of 2500000 steps (84%)
21:32:45:WU00:FS00:0xa8:Completed 2125000 out of 2500000 steps (85%)
******************************* Date: 2021-04-24 *******************************
22:20:07:WU00:FS00:0xa8:Completed 2150000 out of 2500000 steps (86%)
23:07:36:WU00:FS00:0xa8:Completed 2175000 out of 2500000 steps (87%)
23:55:05:WU00:FS00:0xa8:Completed 2200000 out of 2500000 steps (88%)
00:42:32:WU00:FS00:0xa8:Completed 2225000 out of 2500000 steps (89%)
01:30:00:WU00:FS00:0xa8:Completed 2250000 out of 2500000 steps (90%)
02:17:23:WU00:FS00:0xa8:Completed 2275000 out of 2500000 steps (91%)
03:04:53:WU00:FS00:0xa8:Completed 2300000 out of 2500000 steps (92%)
03:52:21:WU00:FS00:0xa8:Completed 2325000 out of 2500000 steps (93%)
******************************* Date: 2021-04-25 *******************************
04:39:50:WU00:FS00:0xa8:Completed 2350000 out of 2500000 steps (94%)
05:27:21:WU00:FS00:0xa8:Completed 2375000 out of 2500000 steps (95%)
06:14:42:WU00:FS00:0xa8:Completed 2400000 out of 2500000 steps (96%)
07:02:11:WU00:FS00:0xa8:Completed 2425000 out of 2500000 steps (97%)
07:49:41:WU00:FS00:0xa8:Completed 2450000 out of 2500000 steps (98%)
08:37:09:WU00:FS00:0xa8:Completed 2475000 out of 2500000 steps (99%)
08:37:10:WU01:FS00:Connecting to assign1.foldingathome.org:80
08:37:11:WU01:FS00:Assigned to work server 129.32.209.205
08:37:11:WU01:FS00:Requesting new work unit for slot 00: cpu:4 from 129.32.209.205
08:37:11:WU01:FS00:Connecting to 129.32.209.205:8080
08:37:11:WU01:FS00:Downloading 1.47MiB
08:37:12:WU01:FS00:Download complete
08:37:12:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:16959 run:35 clone:897 gen:6 core:0xa8 unit:0x00000381000000060000423f00000023
09:24:32:WU00:FS00:0xa8:Completed 2500000 out of 2500000 steps (100%)
09:24:34:WU00:FS00:0xa8:Saving result file ../logfile_01.txt
09:24:34:WU00:FS00:0xa8:Saving result file frame10.gro
09:24:34:WU00:FS00:0xa8:Saving result file frame10.xtc
09:24:34:WU00:FS00:0xa8:Saving result file md.log
09:24:34:WU00:FS00:0xa8:Saving result file science.log
09:24:34:WU00:FS00:0xa8:Saving result file state.cpt
09:24:34:WU00:FS00:0xa8:Folding@home Core Shutdown: FINISHED_UNIT
09:24:35:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
09:24:35:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16959 run:39 clone:729 gen:10 core:0xa8 unit:0x000002d90000000a0000423f00000027
09:24:35:WU00:FS00:Uploading 5.14MiB to 129.32.209.205
09:24:35:WU00:FS00:Connecting to 129.32.209.205:8080
09:24:35:WU01:FS00:Starting
09:24:35:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit-aarch64/a8-0.0.12/Core_a8.fah/FahCore_a8 -dir 01 -suffix 01 -version 706 -lifeline 682 -checkpoint 15 -np 4
09:24:35:WU01:FS00:Started FahCore on PID 10991
09:24:35:WU01:FS00:Core PID:10995
09:24:35:WU01:FS00:FahCore 0xa8 started
09:24:35:WU01:FS00:0xa8:*********************** Log Started 2021-04-25T09:24:35Z ***********************
09:24:35:WU01:FS00:0xa8:************************** Gromacs Folding@home Core ***************************
09:24:35:WU01:FS00:0xa8:       Core: Gromacs
09:24:35:WU01:FS00:0xa8:       Type: 0xa8
09:24:35:WU01:FS00:0xa8:    Version: 0.0.12
09:24:35:WU01:FS00:0xa8:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
09:24:35:WU01:FS00:0xa8:  Copyright: 2020 foldingathome.org
09:24:35:WU01:FS00:0xa8:   Homepage: https://foldingathome.org/
09:24:35:WU01:FS00:0xa8:       Date: Jan 16 2021
09:24:35:WU01:FS00:0xa8:       Time: 19:29:29
09:24:35:WU01:FS00:0xa8:   Compiler: GNU 8.3.0
09:24:35:WU01:FS00:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
09:24:35:WU01:FS00:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie
09:24:35:WU01:FS00:0xa8:   Platform: linux2 4.15.0-128-generic
09:24:35:WU01:FS00:0xa8:       Bits: 64
09:24:35:WU01:FS00:0xa8:       Mode: Release
09:24:35:WU01:FS00:0xa8:       SIMD: arm_neon_asimd
09:24:35:WU01:FS00:0xa8:     OpenMP: ON
09:24:35:WU01:FS00:0xa8:       CUDA: OFF
09:24:35:WU01:FS00:0xa8:       Args: -dir 01 -suffix 01 -version 706 -lifeline 10991 -checkpoint 15 -np
09:24:35:WU01:FS00:0xa8:             4
09:24:35:WU01:FS00:0xa8:************************************ libFAH ************************************
09:24:35:WU01:FS00:0xa8:       Date: Jan 16 2021
09:24:35:WU01:FS00:0xa8:       Time: 19:29:00
09:24:35:WU01:FS00:0xa8:   Compiler: GNU 8.3.0
09:24:35:WU01:FS00:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
09:24:35:WU01:FS00:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie
09:24:35:WU01:FS00:0xa8:   Platform: linux2 4.15.0-128-generic
09:24:35:WU01:FS00:0xa8:       Bits: 64
09:24:35:WU01:FS00:0xa8:       Mode: Release
09:24:35:WU01:FS00:0xa8:************************************ CBang *************************************
09:24:35:WU01:FS00:0xa8:       Date: Jan 16 2021
09:24:35:WU01:FS00:0xa8:       Time: 19:28:44
09:24:35:WU01:FS00:0xa8:   Compiler: GNU 8.3.0
09:24:35:WU01:FS00:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
09:24:35:WU01:FS00:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
09:24:35:WU01:FS00:0xa8:   Platform: linux2 4.15.0-128-generic
09:24:35:WU01:FS00:0xa8:       Bits: 64
09:24:35:WU01:FS00:0xa8:       Mode: Release
09:24:35:WU01:FS00:0xa8:************************************ System ************************************
09:24:35:WU01:FS00:0xa8:        CPU: Cortex-A
09:24:35:WU01:FS00:0xa8:     CPU ID: Arm Family 8 Model 72 Stepping 3
09:24:35:WU01:FS00:0xa8:       CPUs: 4
09:24:35:WU01:FS00:0xa8:     Memory: 1.81GiB
09:24:35:WU01:FS00:0xa8:Free Memory: 1.29GiB
09:24:35:WU01:FS00:0xa8:    Threads: POSIX_THREADS
09:24:35:WU01:FS00:0xa8: OS Version: 5.4
09:24:35:WU01:FS00:0xa8:Has Battery: false
09:24:35:WU01:FS00:0xa8: On Battery: false
09:24:35:WU01:FS00:0xa8: UTC Offset: 2
09:24:35:WU01:FS00:0xa8:        PID: 10995
09:24:35:WU01:FS00:0xa8:        CWD: /var/lib/fahclient/work
09:24:35:WU01:FS00:0xa8:********************************************************************************
09:24:35:WU01:FS00:0xa8:Project: 16959 (Run 35, Clone 897, Gen 6)
09:24:35:WU01:FS00:0xa8:Unit: 0x00000000000000000000000000000000
09:24:35:WU01:FS00:0xa8:Reading tar file core.xml
09:24:35:WU01:FS00:0xa8:Reading tar file frame6.tpr
09:24:35:WU01:FS00:0xa8:Digital signatures verified
09:24:35:WU01:FS00:0xa8:Calling: mdrun -c frame6.gro -s frame6.tpr -x frame6.xtc -cpt 15 -nt 4 -ntmpi 1
09:24:35:WU01:FS00:0xa8:Steps: first=15000000 total=17500000
09:24:41:WU00:FS00:Upload 70.54%
09:24:44:WU00:FS00:Upload complete
09:24:44:WU00:FS00:Server responded WORK_ACK (400)
09:24:44:WU00:FS00:Final credit estimate, 8816.00 points
09:24:44:WU00:FS00:Cleaning up
My Raspberry Pi folding rack: http://www.anne-emscher.net/fah/
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Project 16959: multiple WUs not credited (129.32.209.205

Post by bruce »

We don't need more debugging information. The problem is isolated to a couple of Work Servers and they keep sending out assignments and receiving the results from FAHClients but for reasons we don't understand they don't get reported to the stats subsystem until somebody comes along (generally on Monday morning) and fixes something. Then all the missing credits all arrive at the stats subsystem in a big recovery.
Top
csal
Posts: 1
Joined: Mon May 03, 2021 2:06 pm

Re: Project 16959: multiple WUs not credited (129.32.209.205

Post by csal »

I'm also having this issue on same server since starting folding again this week, my total shows 0 points after at least 3 completed work units.

Are you saying my points are being registered and I will get them in a batch every Monday?

Still nothing today (Monday) but maybe my morning finishes before yours starts ;-)
Top
PaulTV
Posts: 208
Joined: Mon Jan 25, 2021 4:53 pm
Location: Netherlands

Re: Project 16959: multiple WUs not credited (129.32.209.205

Post by PaulTV »

Welcome back :) Last Friday the active feed for newly returned results for p16959 was restored, and the backlog from April 22 on was processed. All results I returned for that project after that point were credited within half an hour tops. I myself only see p16959 credits missing for results returned between April 16 and 21. So, if you haven't seen the credits for those results yet, please provide the PRCG numbers. Lemme know if you need help finding which results you returned, and where you can check if results are credited.

If you're still missing resutls from project 16935, that feed is still broken.
Image

Ryzen 5800X / RTX 4090 / Windows 11
Ryzen 5600X / RTX 3070 Ti / Ubuntu 22.04
Ryzen 5600 / RTX 3060 Ti / Windows 11
Top
Post Reply

Return to “Issues with a specific WU”