20210206 Missing Work?

[cine.chris]

Numbers from yesterday appeared too low.
EOC data & my FAH daily calculated data were within 1WU.
When compared to my HFM logged data compared over the same GMT-6 24 hr period I showed a difference of 21wu 3,877,871 points.
Having seen the aggregate data seldom below 19M PPD, I'd have to say that some of the data wasn't reported in the daily summary.

124 18,567,878
103 14,690,007
21 3,877,871

I then compared the data from 12/1-02/01 and found that my logged data, was much lower.
Having made an effort to log all work since Sept, this was disappointing. The FAH data showed an additional 127M points & 823 wu.
But that is a different problem...

2677157890 17302
1678534736 11772
998,623,154 5530
871,189,252 4707
127,433,902 823

[bollix47]

If this is you then can you explain what the problem is? It looks good to me.

Please supply the project numbers with RCG so we can check them or you can check @ https://apps.foldingathome.org/wu

With those we'll know which server is involved and can report if it's not updating the stats.

[cine.chris]

HFM logged 124wu 18,567,878 points for the same period & this matches my impression of the #s that I was seeing, usually >19M.
Checking 124wu isn't something I'm about to chase. Is the data downloadable?
Python & Pandas can be trained to easily find things like that.

I've checked all the system settings. Scanned the FAH data to see if a system was misconfigured, that's happened.

This is the HFM data: https://fahtech.com/data/WU-20210207-hfm.zip

[cine.chris]

I'm consistently showing 18-19M PPD & EOC is showing 3hr logs of <2M. Something is not connecting.
The one new system appears to be working perfectly, but I'm now shutting down all systems and will have to test them in 3hr increments in an effort to isolate the problem.
All the systems show that completed work is sending w/o errors.
Checked the config on all the systems.

[cine.chris]

NOT FOUND
===============
05:23:02:WU01:FS00:0x22:Saving result file science.log
05:23:02:WU01:FS00:0x22:Folding@home Core Shutdown: FINISHED_UNIT
05:23:02:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
05:23:02:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17431 run:0 clone:1366 gen:90 core:0x22 unit:0x000005560000005a0000441700000000
05:23:02:WU01:FS00:Uploading 17.88MiB to 206.223.170.146
05:23:02:WU01:FS00:Connecting to 206.223.170.146:8080
05:23:02:WU00:FS00:Starting

[cine.chris]

NOT FOUND
==========================
05:54:25:WU00:FS00:0x22:Saving result file science.log
05:54:25:WU00:FS00:0x22:Folding@home Core Shutdown: FINISHED_UNIT
05:54:25:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
05:54:25:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:17431 run:0 clone:1164 gen:101 core:0x22 unit:0x0000048c000000650000441700000000
05:54:26:WU00:FS00:Uploading 17.89MiB to 206.223.170.146
05:54:26:WU00:FS00:Connecting to 206.223.170.146:8080
05:54:26:WU01:FS00:Starting

[cine.chris]

NOT FOUND
=================
06:48:15:WU01:FS01:0x22:Saving result file science.log
06:48:15:WU01:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
06:48:16:WU01:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
06:48:16:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:17702 run:1103 clone:0 gen:11 core:0x22 unit:0x000000000000000b000045260000044f
06:48:16:WU01:FS01:Uploading 14.73MiB to 128.174.73.74
06:48:16:WU01:FS01:Connecting to 128.174.73.74:8080
06:48:16:WU00:FS01:Starting

[cine.chris]

NOT FOUND
===========================
05:57:43:WU01:FS00:0x22:Saving result file science.log
05:57:43:WU01:FS00:0x22:Folding@home Core Shutdown: FINISHED_UNIT
05:57:44:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
05:57:44:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17425 run:0 clone:353 gen:153 core:0x22 unit:0x00000161000000990000441100000000
05:57:44:WU01:FS00:Uploading 17.90MiB to 206.223.170.146
05:57:44:WU01:FS00:Connecting to 206.223.170.146:8080
05:57:44:WU00:FS00:Starting

[ajm]

Confirmed: several of my jobs delivered to 206.223.170.146 haven't made it into the stats.

See also: viewtopic.php?f=18&t=36777&p=348868#p348868

(snip) I think that there is something wrong or delayed with the stats again.
I haven't checked all WUs to isolate the faulty server, but since at least 36 hours, my EOC account gets only some 60-70% of the Total Estimated Points Per Day announced by FAHControl.
And I observe the same gap lately in the EOC aggregate summary https://folding.extremeoverclocking.com ... ary.php?s= as well as in several teams summaries I checked.

[bollix47]

Developer has been notified ... hopefully server(s) will be fixed within the next day or two.

[cine.chris]

Thx bollix47 & ajm for the follow-up.
Will someone make an effort to recover the missing stats... like a Holiday Bonus Check

[mgetz]

So I'm seeing the missing stats on EOC... just not on the official boards. This is weird.

[Joe_H]

Will someone make an effort to recover the missing stats... like a Holiday Bonus Check

The usual case is that the network connection between a WS and the stats database server fails for some reason. The stats reports will be queued up on the WS waiting for the connection to come back. When the connection is reestablished, the reports will be sent and processed into the database.

Less common is some problem processing a stats report. One might have become corrupted or have a bad value that causes the report to either not be processed at all or just partially. In most cases the data can be recovered and processed. Depending on exactly what, this usually takes longer to fix and for the stats to show up. In very rare cases they can not find backup information in the logs.

In this case it looks like the connection has failed, so points should just take a bit to show up once the connection has been fixed. The only thing sent to the stats database is the WU information, PRCG, who folded it, and the points to credit. The WU data remains on the WS.

[bollix47]

The following did show up on 128.174.73.74 so that server may not have a problem or it was fixed but still not seeing results from 206.223.170.146.

Hi cine.chris (team 257944), Your WU (P17702 R1103 C0 G11) was added to the stats database on Mon, 08 Feb 2021 11:48:27 GMT for 109790 points of credit.

[ajm]

Just checked a few (on the present log of a one machine, for the server 206.223.170.146):
Not found:
project:17426 run:0 clone:984 gen:169 core:0x22
project:17434 run:0 clone:561 gen:23 core:0x22
project:17435 run:0 clone:358 gen:19 core:0x22
project:17426 run:0 clone:1647 gen:118 core:0x22
project:17435 run:0 clone:66 gen:21 core:0x22
project:17433 run:0 clone:1701 gen:8 core:0x22
project:17425 run:0 clone:1131 gen:167 core:0x22
project:17435 run:0 clone:732 gen:18 core:0x22
project:17431 run:0 clone:1939 gen:112 core:0x22
project:17434 run:0 clone:305 gen:17 core:0x22
project:17433 run:0 clone:1462 gen:27 core:0x22
project:17435 run:0 clone:136 gen:13 core:0x22

... fast forward to some 6 hours before now ...

Still not found:
project:17424 run:0 clone:247 gen:135 core:0x22
project:17434 run:0 clone:1125 gen:23 core:0x22
and so on