Project: 7085 (Run 0, Clone 695, Gen 16)

[netblazer]

It's not an error more an annoyance. But as a programmer I know this may hide an issue so I'm reporting anyways.

Here's the full log. And then I'll highlight the "issues".

Code: Select all

22:03:12:WU01:FS00:0xa4:Project: 7085 (Run 0, Clone 695, Gen 16)
22:03:12:WU01:FS00:0xa4:
22:03:12:WU01:FS00:0xa4:Assembly optimizations on if available.
22:03:12:WU01:FS00:0xa4:Entering M.D.
22:03:17:WU00:FS00:Upload 59.31%
22:03:18:WU01:FS00:0xa4:Mapping NT from 1 to 1 
22:03:18:WU01:FS00:0xa4:Completed 0 out of 10000000 steps  (0%)
22:03:22:WU00:FS00:Upload complete
22:03:22:WU00:FS00:Server responded WORK_ACK (400)
22:03:22:WU00:FS00:Final credit estimate, 1161.00 points
22:03:22:WU00:FS00:Cleaning up
00:52:10:WU01:FS00:0xa4:Completed 100000 out of 10000000 steps  (1%)
******************************* Date: 2013-12-14 *******************************
02:41:31:WU01:FS00:0xa4:Completed 200000 out of 10000000 steps  (2%)
04:45:27:WU01:FS00:0xa4:Completed 300000 out of 10000000 steps  (3%)
08:13:33:WU01:FS00:0xa4:Completed 400000 out of 10000000 steps  (4%)
******************************* Date: 2013-12-14 *******************************
10:25:38:WU01:FS00:0xa4:Completed 500000 out of 10000000 steps  (5%)
11:51:03:WU01:FS00:0xa4:Completed 600000 out of 10000000 steps  (6%)
13:18:57:WU01:FS00:0xa4:Completed 700000 out of 10000000 steps  (7%)
14:43:56:WU01:FS00:0xa4:Completed 800000 out of 10000000 steps  (8%)
16:08:58:WU01:FS00:0xa4:Completed 900000 out of 10000000 steps  (9%)
******************************* Date: 2013-12-14 *******************************
17:33:45:WU01:FS00:0xa4:Completed 1000000 out of 10000000 steps  (10%)
19:52:41:WU01:FS00:0xa4:Completed 1100000 out of 10000000 steps  (11%)
21:56:31:WU01:FS00:0xa4:Completed 1200000 out of 10000000 steps  (12%)
23:28:45:WU01:FS00:0xa4:Completed 1300000 out of 10000000 steps  (13%)
******************************* Date: 2013-12-15 *******************************
01:01:31:WU01:FS00:0xa4:Completed 1400000 out of 10000000 steps  (14%)
02:37:45:WU01:FS00:0xa4:Completed 1500000 out of 10000000 steps  (15%)
04:50:20:WU01:FS00:0xa4:Completed 1600000 out of 10000000 steps  (16%)
******************************* Date: 2013-12-15 *******************************
07:11:45:WU01:FS00:0xa4:Completed 1700000 out of 10000000 steps  (17%)
09:59:38:WU01:FS00:0xa4:Completed 1800000 out of 10000000 steps  (18%)
12:07:20:WU01:FS00:0xa4:Completed 1900000 out of 10000000 steps  (19%)
******************************* Date: 2013-12-15 *******************************
13:33:29:WU01:FS00:0xa4:Completed 2000000 out of 10000000 steps  (20%)
14:59:05:WU01:FS00:0xa4:Completed 2100000 out of 10000000 steps  (21%)
16:24:19:WU01:FS00:0xa4:Completed 2200000 out of 10000000 steps  (22%)
17:49:49:WU01:FS00:0xa4:Completed 2300000 out of 10000000 steps  (23%)
19:15:04:WU01:FS00:0xa4:Completed 2400000 out of 10000000 steps  (24%)
******************************* Date: 2013-12-15 *******************************
20:57:50:WU01:FS00:0xa4:Completed 2500000 out of 10000000 steps  (25%)
22:35:45:WU01:FS00:0xa4:Completed 2600000 out of 10000000 steps  (26%)
01:19:49:WU01:FS00:0xa4:Completed 2700000 out of 10000000 steps  (27%)
******************************* Date: 2013-12-16 *******************************
03:28:51:WU01:FS00:0xa4:Completed 2800000 out of 10000000 steps  (28%)
05:28:34:WU01:FS00:0xa4:Completed 2900000 out of 10000000 steps  (29%)
******************************* Date: 2013-12-16 *******************************
09:57:22:WU01:FS00:0xa4:Completed 3000000 out of 10000000 steps  (30%)
14:02:50:WU01:FS00:0xa4:Completed 3100000 out of 10000000 steps  (31%)
******************************* Date: 2013-12-16 *******************************
17:02:46:WU01:FS00:0xa4:Completed 3200000 out of 10000000 steps  (32%)
18:47:34:WU01:FS00:0xa4:Completed 3300000 out of 10000000 steps  (33%)
21:08:00:WU01:FS00:0xa4:Completed 3400000 out of 10000000 steps  (34%)
******************************* Date: 2013-12-16 *******************************
23:59:42:WU01:FS00:0xa4:Completed 3500000 out of 10000000 steps  (35%)
03:14:03:WU01:FS00:0xa4:Completed 3600000 out of 10000000 steps  (36%)
******************************* Date: 2013-12-17 *******************************
07:02:14:WU01:FS00:0xa4:Completed 3700000 out of 10000000 steps  (37%)
10:38:07:WU01:FS00:0xa4:Completed 3800000 out of 10000000 steps  (38%)
******************************* Date: 2013-12-17 *******************************
13:24:47:WU01:FS00:0xa4:Completed 3900000 out of 10000000 steps  (39%)
15:16:58:WU01:FS00:0xa4:Completed 4000000 out of 10000000 steps  (40%)
16:54:23:WU01:FS00:0xa4:Completed 4100000 out of 10000000 steps  (41%)
18:25:46:WU01:FS00:0xa4:Completed 4200000 out of 10000000 steps  (42%)
******************************* Date: 2013-12-17 *******************************
20:05:24:WU01:FS00:0xa4:Completed 4300000 out of 10000000 steps  (43%)
22:58:46:WU01:FS00:0xa4:Completed 4400000 out of 10000000 steps  (44%)
******************************* Date: 2013-12-18 *******************************
03:20:35:WU01:FS00:0xa4:Completed 4500000 out of 10000000 steps  (45%)
05:50:21:WU01:FS00:0xa4:Completed 4600000 out of 10000000 steps  (46%)
07:49:51:WU01:FS00:0xa4:Completed 4700000 out of 10000000 steps  (47%)
******************************* Date: 2013-12-18 *******************************
10:24:02:WU01:FS00:0xa4:Completed 4800000 out of 10000000 steps  (48%)
13:17:51:WU01:FS00:0xa4:Completed 4900000 out of 10000000 steps  (49%)
15:00:41:WU01:FS00:0xa4:Completed 5000000 out of 10000000 steps  (50%)
******************************* Date: 2013-12-18 *******************************
16:26:24:WU01:FS00:0xa4:Completed 5100000 out of 10000000 steps  (51%)
17:51:30:WU01:FS00:0xa4:Completed 5200000 out of 10000000 steps  (52%)
19:22:57:WU01:FS00:0xa4:Completed 5300000 out of 10000000 steps  (53%)
20:55:06:WU01:FS00:0xa4:Completed 5400000 out of 10000000 steps  (54%)
******************************* Date: 2013-12-18 *******************************
22:35:37:WU01:FS00:0xa4:Completed 5500000 out of 10000000 steps  (55%)
00:54:26:WU01:FS00:0xa4:Completed 5600000 out of 10000000 steps  (56%)
02:49:50:WU01:FS00:0xa4:Completed 5700000 out of 10000000 steps  (57%)
04:24:15:WU01:FS00:0xa4:Completed 5800000 out of 10000000 steps  (58%)
******************************* Date: 2013-12-19 *******************************
06:32:40:WU01:FS00:0xa4:Completed 5900000 out of 10000000 steps  (59%)
08:39:11:WU01:FS00:0xa4:Completed 6000000 out of 10000000 steps  (60%)
10:14:18:WU01:FS00:0xa4:Completed 6100000 out of 10000000 steps  (61%)
11:46:13:WU01:FS00:0xa4:Completed 6200000 out of 10000000 steps  (62%)
******************************* Date: 2013-12-19 *******************************
13:20:28:WU01:FS00:0xa4:Completed 6300000 out of 10000000 steps  (63%)
15:04:43:WU01:FS00:0xa4:Completed 6400000 out of 10000000 steps  (64%)
16:37:15:WU01:FS00:0xa4:Completed 6500000 out of 10000000 steps  (65%)
18:10:42:WU01:FS00:0xa4:Completed 6600000 out of 10000000 steps  (66%)
******************************* Date: 2013-12-19 *******************************
20:21:01:WU01:FS00:0xa4:Completed 6700000 out of 10000000 steps  (67%)
22:14:54:WU01:FS00:0xa4:Completed 6800000 out of 10000000 steps  (68%)
01:00:39:WU01:FS00:0xa4:Completed 6900000 out of 10000000 steps  (69%)
******************************* Date: 2013-12-20 *******************************
03:18:07:WU01:FS00:0xa4:Completed 7000000 out of 10000000 steps  (70%)
04:58:09:WU01:FS00:0xa4:Completed 7100000 out of 10000000 steps  (71%)
07:26:58:WU01:FS00:0xa4:Completed 7200000 out of 10000000 steps  (72%)
******************************* Date: 2013-12-20 *******************************
09:53:19:WU01:FS00:0xa4:Completed 7300000 out of 10000000 steps  (73%)
12:16:48:WU01:FS00:0xa4:Completed 7400000 out of 10000000 steps  (74%)
13:51:19:WU01:FS00:0xa4:Completed 7500000 out of 10000000 steps  (75%)
15:18:36:WU01:FS00:0xa4:Completed 7600000 out of 10000000 steps  (76%)
******************************* Date: 2013-12-20 *******************************
16:45:59:WU01:FS00:0xa4:Completed 7700000 out of 10000000 steps  (77%)
18:17:34:WU01:FS00:0xa4:Completed 7800000 out of 10000000 steps  (78%)
19:53:16:WU01:FS00:0xa4:Completed 7900000 out of 10000000 steps  (79%)
21:36:49:WU01:FS00:0xa4:Completed 8000000 out of 10000000 steps  (80%)
******************************* Date: 2013-12-21 *******************************
00:49:35:WU01:FS00:0xa4:Completed 8100000 out of 10000000 steps  (81%)
03:44:00:WU01:FS00:0xa4:Completed 8200000 out of 10000000 steps  (82%)
06:08:30:WU01:FS00:0xa4:Completed 8300000 out of 10000000 steps  (83%)
******************************* Date: 2013-12-21 *******************************
09:04:00:WU01:FS00:0xa4:Completed 8400000 out of 10000000 steps  (84%)
11:07:53:WU01:FS00:0xa4:Completed 8500000 out of 10000000 steps  (85%)
12:33:26:WU01:FS00:0xa4:Completed 8600000 out of 10000000 steps  (86%)
13:59:39:WU01:FS00:0xa4:Completed 8700000 out of 10000000 steps  (87%)
******************************* Date: 2013-12-21 *******************************
15:25:00:WU01:FS00:0xa4:Completed 8800000 out of 10000000 steps  (88%)
16:51:05:WU01:FS00:0xa4:Completed 8900000 out of 10000000 steps  (89%)
18:17:54:WU01:FS00:0xa4:Completed 9000000 out of 10000000 steps  (90%)
19:44:55:WU01:FS00:0xa4:Completed 9100000 out of 10000000 steps  (91%)
******************************* Date: 2013-12-21 *******************************
21:47:32:WU01:FS00:0xa4:Completed 9200000 out of 10000000 steps  (92%)
02:17:36:WU01:FS00:0xa4:Completed 9300000 out of 10000000 steps  (93%)
******************************* Date: 2013-12-22 *******************************
05:38:09:WU01:FS00:0xa4:Completed 9400000 out of 10000000 steps  (94%)
08:03:34:WU01:FS00:0xa4:Completed 9500000 out of 10000000 steps  (95%)

As you can see this "monster" usually runs at a TPF of ±90 minutes if doing nothing else and 150 minutes under normal / heavier use.


This is the first time I noticed something odd in the progression but I wasn't in front of the laptop so I couldn't know for sure :

Code: Select all

03:28:51:WU01:FS00:0xa4:Completed 2800000 out of 10000000 steps  (28%)
05:28:34:WU01:FS00:0xa4:Completed 2900000 out of 10000000 steps  (29%)
******************************* Date: 2013-12-16 *******************************
09:57:22:WU01:FS00:0xa4:Completed 3000000 out of 10000000 steps  (30%)
14:02:50:WU01:FS00:0xa4:Completed 3100000 out of 10000000 steps  (31%)
******************************* Date: 2013-12-16 *******************************
17:02:46:WU01:FS00:0xa4:Completed 3200000 out of 10000000 steps  (32%)

As you can see from 29 to 30% it goes from 2 hours to 4.5, then again to 4 and then only 1 (world record by 50% of this system). I was away from the machine but doing some light background tasks (costing ±30 minutes per frame).


Then this happened :

Code: Select all

19:44:55:WU01:FS00:0xa4:Completed 9100000 out of 10000000 steps  (91%)
******************************* Date: 2013-12-21 *******************************
21:47:32:WU01:FS00:0xa4:Completed 9200000 out of 10000000 steps  (92%)
02:17:36:WU01:FS00:0xa4:Completed 9300000 out of 10000000 steps  (93%)
******************************* Date: 2013-12-22 *******************************
05:38:09:WU01:FS00:0xa4:Completed 9400000 out of 10000000 steps  (94%)
08:03:34:WU01:FS00:0xa4:Completed 9500000 out of 10000000 steps  (95%)

Again ± 2 hours, jumps to 3.5, 3.5, 2.5. The problem is that in 2X 3.5 I was using the laptop just reading stuff on the web and the tpf should be around 1.5, and they were. I saw the client hit 94%. I checked the log and no note was made of it. I finally went up for supper at around 94.5%, still nothing logged. And when I came back down, the counter had gone back to 94%.


The project is still running and no errors have been displayed anywhere. But this is still very odd.

Username Remi_Gregoire.
113 units completed over ± 7 weeks.
At least 10-15 came from that laptop.
Never had any errors before.

As you can imagine this is not an oced system !!!

Edit: changed tpf under normal use from 210 minutes to 150.

[PantherX]

Welcome to the F@H Forum netblazer,

Regarding the increase of TPF, CPU folding is a highly synchronous task. Thus, any interruption would have a negative impact. Since you have a single CPU, everything done on the system has to share resources. Moreover, even though the background task is using 3% of the CPU, it will have a non-linear negative impact on the TPF. There are instances where my browser (Cyberfox/Firefox) will use up to a single CPU (12.5%) while I am using it. Thus, something similar could be happening on your system too. The only way to check it is to view the Details Tab in Task Manager and view the processes to see if any tasks are continuously taking up some percentage of your CPU.

Regarding the disparity between what Advanced Control (AKA FAHControl) shows and what is reported in the log, FAHControl predicts the percentage from the past data. It doesn't factor in any slow-downs due to other tasks running. Thus, it will show the updated percentage even though that percentage isn't done by the FahCore. After few frames, it will automatically correct it.

[netblazer]

... couple stupid questions. How do I subscribe to my own thread to get notifications??? I've added to auto-sub in the profile but I can't subscribe to my own thread!?!?!

I have a screenshot to send, where's the upload picture/file function?

[bollix47]

When you post in a thread you are subscribed. As long as you said Yes to "Notify me upon replies by default:" in your profile you will receive an email when new posts arrive in that thread. If you are reading a thread where you have not been involved you should see a Subscribe topic link at the bottom of the page while reading that thread. Once subscribed to a thread the description of that link will change to Unsubscribe. You can also use the Manage Subscriptions link in your User Control Panel. There may be some variations depending on which theme you're using but basically that's it.

For pictures you need to setup an account at one of the many free image hosting websites. Once done you would upload the image to that site and post their provided link here between img tags.

[netblazer]

Wow, let's just say that this site is much LESS than intuitive. I did search up and down the page at least twice and never the darn sub link (with 25 k posts to my credit on other boards). Why the heck isn't that option by default??? It's like asking "why do you want to know that someone's trying to help you & communicate with you?".

I was going to post a picture to show an possible error in the assignment servers (giving me a job that was just impossible to finish on time). But I'll pass.

P.S. That ain't working either : http://fah-web.stanford.edu/cgi-bin/fah ... ned?p=7812

[7im]

The assignment server has no idea how much you use your computer for other tasks. It can only read basic hardware information. If you have x amount of CPU power, but use half that power for other purposes, that's not a problem with the assignment server.

Click the Full Editor button at the bottom of the quick post window. On the full editor screen, the "Notify me when a reply is posted" check box is clearly visible.

p7812 is a brand new work unit. The info for that WU hasn't been posted yet. It's usually one of the last things done on a new WU.

[netblazer]

No but that's a problem with statistical analysis, which you should excel at @ this point.

This was my ±12th unit on this machine so you have access to my averages if you wish to (not my problem if you don't use that). And don't tell me that select avg from table where userid = is a "hard" query to run (same for the update). I have seem single 2 core xeon server run 1M + queries like that every hour without breaking a molecule of sweat on TB size databases.

I folded 161 hours (100% CPU) out 200 wall clock hours. That's over 80% 24/7 folding and I not only missed the timeout by days, but was hours away from missing the expiration too. You were hours, almost minutes away from wasting 10 days of 24/7 processing power.

That's WAY beneath the tolerance you say you set in your various manuals.

[netblazer]

Happened again. Assigned a 7 days 100% workload with 5 days and change to finish. Now I have to actually stop using the laptop almost entirely to hope the make the final deadline.


11:58:59:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:7085 run:0 clone:583 gen:21 core:0xa4 unit:0x000000b20001329c4ffb2abe05498b40

[Napoleon]

How are the temperatures and clocks? Even without overclocking, there are laptops which don't have adequate cooling for 24/7 folding, consequently throttling clocks in order to prevent overheating.

[netblazer]

How are the temperatures and clocks? Even without overclocking, there are laptops which don't have adequate cooling for 24/7 folding, consequently throttling clocks in order to prevent overheating.

I can keep my hand right on the exhaust vent for 30+ seconds before it gets uncomfortable so I really doubt this is the issue. Also I use the laptop most of the day and it's always responsive when I stay within its known op. limits.

I downloaded cpu-z hoping to see temps in there but alas they're not shown.

Here's the relevant info anyways :

Processor 1 (of 1) ID = 0
Number of cores 1 (max 1)
Number of threads 1 (max 1)
Name Intel Celeron M
Codename Conroe-L
Specification Intel(R) Celeron(R) CPU 560 @ 2.13GHz
Package (platform ID) Socket 479 mPGA (0x7)
CPUID 6.6.1
Extended CPUID 6.16
Core Stepping A1
Technology 65 nm
Core Speed 2128.3 MHz
Multiplier x Bus Speed 16.0 x 133.0 MHz
Rated Bus speed 532.1 MHz
Instructions sets MMX, SSE, SSE2, SSE3, SSSE3, EM64T
L1 Data cache 32 KBytes, 8-way set associative, 64-byte line size
L1 Instruction cache 32 KBytes, 8-way set associative, 64-byte line size
L2 cache 1024 KBytes, 4-way set associative, 64-byte line size
FID/VID Control no




As you can see this is not a strong laptop, but this is normally what 60-90% of the market was buying 4 years ago at the Wal-Mart and Staples of this world. Aside from your top 1% producers, this type of hardware should be rather common amongst users (let's say lower tier).


This WU has 1221 base credits. So ±11 days on your benchmarking machine, and yet you only give 5 days and change to complete it. When in the documentation you explain you double that to allow for downtime and normal user usage. So rather than ±22 days expiration and ??29?? final deadline, I only have 5.5 exp and 9.5 deadline.

So either update your documentation or fix the assignment script. I should NOT be getting this WU / deadline combo on this setup, period.

My best guess is that the last 10 days were wasted because the WU was reassigned anyways, and now this is happening again.

If this was a call center, I'd be requesting escalation right about now.

[PantherX]

Thanks for your report. I have informed the appropriate PG Member so let's see what happens.

Please note that currently, there isn't any system in place to track the performance of individual donor's system, that I know off.

[netblazer]

How are the temperatures and clocks? Even without overclocking, there are laptops which don't have adequate cooling for 24/7 folding, consequently throttling clocks in order to prevent overheating.

Just found and installed Core temp.

Even after 83+ hours@ 100%, I only get 71-74 C with apparent max rating of 100 (can't confirm that info tho).

[netblazer]

Thanks for your report. I have informed the appropriate PG Member so let's see what happens.

Please note that currently, there isn't any system in place to track the performance of individual donor's system, that I know off.

Actually all you'd need is a failure / missed deadline log (with minor BI to understand the data). Then just don't reassign project in that list for that user / hardware combo.

Really simple to code and very low maintenance DB wise.

Then when you see a butt load of users missing on the same project, you know you messed it up.

[Napoleon]

Just found and installed Core temp.

Even after 83+ hours@ 100%, I only get 71-74 C with apparent max rating of 100 (can't confirm that info tho).

I'd think the Core Temp reading is reliable enough, and http://ark.intel.com/products/34439/Int ... 33-MHz-FSB suggests max. 100C as well. So I don't think it is a CPU temperature problem, provided that the CPU is actually running at 2.13GHz after those 83+ hours of nonstop folding. However, I still wouldn't rule out some sort of thermal throttling just yet. I've seen a laptop where the CPU had plenty of headroom, temperature wise, but the chipset with integrated GPU was constantly on the verge of meltdown, and the extra heat from CPU folding caused it to throttle drastic performance fluctuations every now and then.

This WU has 1221 base credits. So ±11 days on your benchmarking machine, and yet you only give 5 days and change to complete it. When in the documentation you explain you double that to allow for downtime and normal user usage. So rather than ±22 days expiration and ??29?? final deadline, I only have 5.5 exp and 9.5 deadline.

So either update your documentation or fix the assignment script. I should NOT be getting this WU / deadline combo on this setup, period.

Which piece of documentation are you referring to? Closest one leaping to my mind would be http://folding.stanford.edu/home/faq/faq-points/#ntoc9, and AFAICT it has been updated fairly recently, along the lines of the new unified benchmarking scheme. I can't quite grasp the "This WU has 1221 base credits. So ±11 days on your benchmarking machine..." bit. The benchmarking machine is a quadcore i5 750 at stock clocks, and I'd be very surprised if it actually needed anything close to 11 days to complete a P7085 WU. There was no mention during beta testing that P7085 would be restricted to setups with less than four cores, so I assume it was benchmarked with a SMP:4 setting, business as usual.

The assignment logic isn't perfect, by any means. Take my 2C/4T Atom330 CPU, for example. It's just barely fast enough to complete any of the currently assigned CPU WUs, even when overclocked, and there's no headroom for doing anything else besides folding. Good old core_78 WUs (longer deadlines, uniprocessor only, no QRB) aren't in circulation anymore, nor do I think they'll ever make a comeback.

According to http://folding.stanford.edu/home/happy- ... d-to-2014/, PG expects to release a whole new type of client and backend server in 2014, which just might solve the problem. But the solution might very well be that slower hardware like your unicore or my Atom330 CPU simply won't get assigned work any more, period.

[netblazer]

It's on the main faq ( http://folding.stanford.edu/home/faq/#ntoc37 ). In MS SQL Server, there's a similar reference to an obsolete machine that would take 1 second to return x amount of work and this is now an official "work unit" (just to use the same language). Obviously that machine is like 15 years old now so even the lowest atom processor crushes those results.

Obviously a decent 4 threads i5 will crush that workload in 6-12 hours tops.

I would find it extremely ill-advised to throw away anything. I work with sales department in my job and the costliest mistake ever is to lose a customer no matter how small. You'd be amazed at the lengths they go to (with admin permission) to keep or get someone back.

I know these are mega dwarfed by the top 1% producers on the project but the combined forced of my old crap lying around collecting dust is around 1 M PPY (enough to be in the top 2-3% in the lifetime rankings).

With only 400K people doing that you'd double the entire lifetime FAH production in just 12 months. That's without accounting for the few people in there who will start getting dedicated 2P Xeon and start making 1M PPD (even if it's 1 in a 1000, that helps a lot). That's how you build a community, not by ignoring the "small" helpers.

If the entire projects take 1 year to complete, I see absolutely no issue whatsoever to let some smaller WU complete over 2 weeks rather than 2 days (even if they do only 1 M steps rather than 10 M).