Skippimg WU numbers

Moderators: Site Moderators, FAHC Science Team

Post Reply
Mactin
Posts: 247
Joined: Sun Dec 02, 2007 1:08 pm
Location: Outremont, Montréal, Québec

Skippimg WU numbers

Post by Mactin »

It's me again. This time reporting a new funny/hard to describe problem.

Every WU I have completed in the past 5 hours (5 times) has skipped WU numbers. This has happened in 3 folding instances on two different machines.

The log entry illustrates what i happening every time.

Code: Select all

...
20:49:46:I1:WU1099:Completed 245000 out of 250000 steps (98%)
20:51:00:I1:WU1099:Completed 247500 out of 250000 steps (99%)
20:52:12:I1:WU1099:Completed 250000 out of 250000 steps (100%)
20:52:14:I1:WU1099:Saving result file ../logfile_01.txt
20:52:14:I1:WU1099:Saving result file frame480.gro
20:52:14:I1:WU1099:Saving result file frame480.xtc
20:52:14:I1:WU1099:Saving result file md.log
20:52:14:I1:WU1099:Saving result file science.log
20:52:14:I1:WU1099:Saving result file state.cpt
20:52:14:I1:WU1099:Folding@home Core Shutdown: FINISHED_UNIT
20:52:14:I1:WU1099:Core returned FINISHED_UNIT (100)
20:52:15:I1:Default:Added new work unit: cpus:10 gpus:
20:52:15:I1:WU1099:Uploading WU results
20:52:15:I1:WU1100:Requesting WU assignment for user Martin_MBA_M4 team 96377
20:52:15:I1:OUT40:> POST https://highland1.seas.upenn.edu/api/results HTTP/1.1
20:52:15:I1:OUT41:> POST https://assign5.foldingathome.org/api/assign HTTP/1.1
20:52:15:I1:OUT41:< HTTP/1.1 500 HTTP_INTERNAL_SERVER_ERROR
[91m20:52:15:E :OUT41:HTTP_INTERNAL_SERVER_ERROR: {"error":{"message":"DB ERROR: IO error: sqllog-v2.leveldb/039830.log: No space left on device"}}[0m
20:52:17:I1:Default:Added new work unit: cpus:10 gpus:
20:52:17:I1:WU1101:Requesting WU assignment for user Martin_MBA_M4 team 96377
20:52:17:I1:OUT42:> POST https://assign6.foldingathome.org/api/assign HTTP/1.1
20:52:18:I1:OUT42:< HTTP/1.1 200 HTTP_OK
20:52:18:I1:WU1101:Received WU assignment m6LuXpFzJiDaqC-wZ7_tl1PK1pCOYMUqq5EQvG1gaJ4
20:52:18:I1:WU1101:Downloading WU
20:52:18:I1:OUT43:> POST https://highland1.seas.upenn.edu/api/assign HTTP/1.1
20:52:19:I1:OUT43:< HTTP/1.1 200 HTTP_OK
20:52:19:I1:WU1101:Received WU P18240 R766 C3 G453
20:52:19:I3:Running FahCore: "/Library/Application Support/FAHClient/cores/fahcore-a8-osx-64bit-aarch64-0.0.12/FahCore_a8" -dir m6LuXpFzJiDaqC-wZ7_tl1PK1pCOYMUqq5EQvG1gaJ4 -suffix 01 -version 8.4.9 -lifeline 495 -np 10
20:52:19:I3:WU1101:Started FahCore on PID 5341
20:52:19:I1:WU1101:*********************** Log Started 2025-08-12T20:52:19Z ***********************
20:52:19:I1:WU1101:************************** Gromacs Folding@home Core ***************************
20:52:19:I1:WU1101:       Core: Gromacs
20:52:19:I1:WU1101:       Type: 0xa8
20:52:19:I1:WU1101:    Version: 0.0.12
20:52:19:I1:WU1101:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
20:52:19:I1:WU1101:  Copyright: 2020 foldingathome.org
20:52:19:I1:WU1101:   Homepage: https://foldingathome.org/
20:52:19:I1:WU1101:       Date: Jan 16 2021
20:52:19:I1:WU1101:       Time: 21:35:50
20:52:19:I1:WU1101:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
20:52:19:I1:WU1101:     Branch: master
20:52:19:I1:WU1101:   Compiler: GNU Apple LLVM 12.0.0 (clang-1200.0.32.28)
20:52:19:I1:WU1101:    Options: -faligned-new -std=c++14 -fsigned-char -stdlib=libc++ -O3
20:52:19:I1:WU1101:             -funroll-loops -fno-pie -mmacosx-version-min=11.0
20:52:19:I1:WU1101:   Platform: darwin 20.2.0
20:52:19:I1:WU1101:       Bits: 64
20:52:19:I1:WU1101:       Mode: Release
20:52:19:I1:WU1101:       SIMD: arm_neon_asimd
20:52:19:I1:WU1101:     OpenMP: ON
20:52:19:I1:WU1101:       CUDA: OFF
20:52:19:I1:WU1101:       Args: -dir m6LuXpFzJiDaqC-wZ7_tl1PK1pCOYMUqq5EQvG1gaJ4 -suffix 01
20:52:19:I1:WU1101:             -version 8.4.9 -lifeline 495 -np 10
20:52:19:I1:WU1101:************************************ libFAH ************************************
20:52:19:I1:WU1101:       Date: Jan 16 2021
20:52:19:I1:WU1101:       Time: 21:35:27
20:52:19:I1:WU1101:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
20:52:19:I1:WU1101:     Branch: master
20:52:19:I1:WU1101:   Compiler: GNU Apple LLVM 12.0.0 (clang-1200.0.32.28)
20:52:19:I1:WU1101:    Options: -faligned-new -std=c++14 -fsigned-char -stdlib=libc++ -O3
20:52:19:I1:WU1101:             -funroll-loops -fno-pie -mmacosx-version-min=11.0
20:52:19:I1:WU1101:   Platform: darwin 20.2.0
20:52:19:I1:WU1101:       Bits: 64
20:52:19:I1:WU1101:       Mode: Release
20:52:19:I1:WU1101:************************************ CBang *************************************
20:52:19:I1:WU1101:       Date: Jan 16 2021
20:52:19:I1:WU1101:       Time: 21:35:02
20:52:19:I1:WU1101:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
20:52:19:I1:WU1101:     Branch: master
20:52:19:I1:WU1101:   Compiler: GNU Apple LLVM 12.0.0 (clang-1200.0.32.28)
20:52:19:I1:WU1101:    Options: -faligned-new -std=c++14 -fsigned-char -stdlib=libc++ -O3
20:52:19:I1:WU1101:             -funroll-loops -fno-pie -mmacosx-version-min=11.0 -fPIC
20:52:19:I1:WU1101:   Platform: darwin 20.2.0
20:52:19:I1:WU1101:       Bits: 64
20:52:19:I1:WU1101:       Mode: Release
20:52:19:I1:WU1101:************************************ System ************************************
20:52:19:I1:WU1101:        CPU: Apple
20:52:19:I1:WU1101:     CPU ID: Apple Family 8 Model 0 Stepping 0
20:52:19:I1:WU1101:       CPUs: 10
20:52:19:I1:WU1101:     Memory: 16.00GiB
20:52:19:I1:WU1101:Free Memory: 111.58MiB
20:52:19:I1:WU1101:    Threads: POSIX_THREADS
20:52:19:I1:WU1101: OS Version: 15.6
20:52:19:I1:WU1101:Has Battery: true
20:52:19:I1:WU1101: On Battery: false
20:52:19:I1:WU1101: UTC Offset: -4
20:52:19:I1:WU1101:        PID: 5341
20:52:19:I1:WU1101:        CWD: /Library/Application Support/FAHClient/work
20:52:19:I1:WU1101:********************************************************************************
20:52:19:I1:WU1101:Project: 18240 (Run 766, Clone 3, Gen 453)
20:52:19:I1:WU1101:Unit: 0x00000000000000000000000000000000
20:52:19:I1:WU1101:Reading tar file core.xml
20:52:19:I1:WU1101:Reading tar file frame453.tpr
20:52:19:I1:WU1101:Digital signatures verified
20:52:19:I1:WU1101:Calling: mdrun -c frame453.gro -s frame453.tpr -x frame453.xtc -cpt 5 -nt 10 -ntmpi 1
20:52:19:I1:WU1101:Steps: first=113250000 total=113500000
20:52:19:I1:WU1101:Completed 1 out of 250000 steps (0%)
20:52:20:I1:OUT40:< HTTP/1.1 200 HTTP_OK
20:52:20:I1:WU1099:Credited
20:53:32:I1:WU1101:Completed 2500 out of 250000 steps (1%)
20:54:44:I1:WU1101:Completed 5000 out of 250000 steps (2%)
20:55:56:I1:WU1101:Completed 7500 out of 250000 steps (3%)
...
You can see WU1099 complete.
It trues to download WU1100 and encounters an "No space left on device" error.
Downloads and starts processing WU1101.

Somwthing is happening, it looks of little or no consequence.

The same in 4 other times. 1 MAC, 4 PC (3 GPU & 1 CPU).
I have one of each completing in the next hour.

Edit (2 hours later)
Phenomena continues ...
Image
Top
Joe_H
Site Admin
Posts: 8180
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Studio M1 Max 32 GB smp6
Mac Hack i7-7700K 48 GB smp4
Location: W. MA

Re: Skippimg WU numbers

Post by Joe_H »

One of the two assignment servers appears to have run out of space for a log file connected with WU assignments. I have notified the person in charge of those. It appears for you that it is properly failing over to the other AS and then successfully assigning a WU. But the client has incremented the request after the failed assignment.
Image
Top
Post Reply

Return to “v8.4.9 Public Release for Windows / Linux / macOS”