9210 (Run 56, Clone 14, Gen 0) UNKNOWN_ENUM (127 = 0x7f)

[Yakbreeder]

Was looking for a 90,000 point drop, and noticed it hasn't shown up on any updates.

Everything has been running fine for the last month or so until I got the "UNKNOWN_ENUM (127 = 0X7f)

Stock speeds on both CPU and GPU. Temps are/were within normal parameters 45C CPU, 50-52C GPU.

System:

Code: Select all

20:46:51:******************************** Build ********************************
20:46:51:      Version: 7.4.4
20:46:51:         Date: Mar 4 2014
20:46:51:         Time: 20:26:54
20:46:51:      SVN Rev: 4130
20:46:51:       Branch: fah/trunk/client
20:46:51:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
20:46:51:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
20:46:51:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
20:46:51:     Platform: win32 XP
20:46:51:         Bits: 32
20:46:51:         Mode: Release
20:46:51:******************************* System ********************************
20:46:51:          CPU: Intel(R) Core(TM) i5 CPU 750 @ 2.67GHz
20:46:51:       CPU ID: GenuineIntel Family 6 Model 30 Stepping 5
20:46:51:         CPUs: 4
20:46:51:       Memory: 16.00GiB
20:46:51:  Free Memory: 13.80GiB
20:46:51:      Threads: WINDOWS_THREADS
20:46:51:   OS Version: 6.1
20:46:51:  Has Battery: false
20:46:51:   On Battery: false
20:46:51:   UTC Offset: -4
20:46:51:          PID: 4676
20:46:51:          CWD: C:/Users/Yak/AppData/Roaming/FAHClient
20:46:51:           OS: Windows 7 Home Premium
20:46:51:      OS Arch: AMD64
20:46:51:         GPUs: 1
20:46:51:        GPU 0: NVIDIA:4 GM107 [GeForce GTX 750 Ti]
20:46:51:         CUDA: 5.0
20:46:51:  CUDA Driver: 8000
20:46:51:Win32 Service: false

Error:

Code: Select all

11:31:53:WU02:FS01:0x21:*********************** Log Started 2016-03-29T11:31:52Z ***********************
11:31:53:WU02:FS01:0x21:Project: 9210 (Run 56, Clone 14, Gen 0)
11:31:53:WU02:FS01:0x21:Unit: 0x00000003664f2dd056ea08b8dabe5713
11:31:53:WU02:FS01:0x21:CPU: 0x00000000000000000000000000000000
11:31:53:WU02:FS01:0x21:Machine: 1
11:31:53:WU02:FS01:0x21:Reading tar file core.xml
11:31:53:WU02:FS01:0x21:Reading tar file integrator.xml
11:31:53:WU02:FS01:0x21:Reading tar file state.xml
11:31:53:WU02:FS01:0x21:Reading tar file system.xml
11:31:55:WU02:FS01:0x21:Digital signatures verified
11:31:55:WU02:FS01:0x21:Folding@home GPU Core21 Folding@home Core
11:31:55:WU02:FS01:0x21:Version 0.0.17
11:32:34:WU02:FS01:0x21:Completed 0 out of 2500000 steps (0%)
11:32:34:WU02:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
11:47:00:WU02:FS01:0x21:Completed 25000 out of 2500000 steps (1%)
12:01:48:WU02:FS01:0x21:Completed 50000 out of 2500000 steps (2%)
12:16:49:WU02:FS01:0x21:Completed 75000 out of 2500000 steps (3%)

Edited to save space


******************************* Date: 2016-03-30 *******************************
10:51:57:WU02:FS01:0x21:Completed 2225000 out of 2500000 steps (89%)
11:07:42:WU02:FS01:0x21:Completed 2250000 out of 2500000 steps (90%)
11:23:27:WU02:FS01:0x21:Completed 2275000 out of 2500000 steps (91%)
11:39:11:WU02:FS01:0x21:Completed 2300000 out of 2500000 steps (92%)
11:55:02:WU02:FS01:0x21:Completed 2325000 out of 2500000 steps (93%)
12:10:47:WU02:FS01:0x21:Completed 2350000 out of 2500000 steps (94%)
12:26:31:WU02:FS01:0x21:Completed 2375000 out of 2500000 steps (95%)
12:42:16:WU02:FS01:0x21:Completed 2400000 out of 2500000 steps (96%)
12:58:06:WU02:FS01:0x21:Completed 2425000 out of 2500000 steps (97%)
13:13:51:WU02:FS01:0x21:Completed 2450000 out of 2500000 steps (98%)
13:29:36:WU02:FS01:0x21:Completed 2475000 out of 2500000 steps (99%)
13:29:36:WU01:FS01:Connecting to 171.67.108.45:80
13:29:37:WU01:FS01:Assigned to work server 171.67.108.157
13:29:37:WU01:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GM107 [GeForce GTX 750 Ti] from 171.67.108.157
13:29:37:WU01:FS01:Connecting to 171.67.108.157:8080
13:29:38:WU01:FS01:Downloading 29.47MiB
13:29:44:WU01:FS01:Download 12.51%
13:29:50:WU01:FS01:Download 26.09%
13:29:56:WU01:FS01:Download 39.87%
13:30:02:WU01:FS01:Download 53.44%
13:30:08:WU01:FS01:Download 67.23%
13:30:14:WU01:FS01:Download 80.80%
13:30:20:WU01:FS01:Download 94.37%
13:30:22:WU01:FS01:Download complete
13:30:22:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9441 run:3 clone:0 gen:131 core:0x21 unit:0x000000a0ab436c9d56af1b2809b08103
13:45:20:WU02:FS01:0x21:Completed 2500000 out of 2500000 steps (100%)
13:45:27:WU02:FS01:0x21:Saving result file logfile_01.txt
13:45:27:WU02:FS01:0x21:Saving result file checkpointState.xml
13:45:27:WU02:FS01:0x21:Saving result file checkpt.crc
13:45:27:WU02:FS01:0x21:Saving result file log.txt
13:45:27:WU02:FS01:0x21:Saving result file positions.xtc
13:45:27:WU02:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
13:45:28:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
13:45:28:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:9210 run:56 clone:14 gen:0 core:0x21 unit:0x00000003664f2dd056ea08b8dabe5713
13:45:28:WU02:FS01:Uploading 37.95MiB to 171.64.65.104
13:45:28:WU02:FS01:Connecting to 171.64.65.104:8080
13:45:28:WU01:FS01:Starting
13:45:28:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Yak/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 704 -lifeline 3308 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
13:45:28:WU01:FS01:Started FahCore on PID 19592
13:45:28:WU01:FS01:Core PID:18364
13:45:28:WU01:FS01:FahCore 0x21 started
13:45:29:WU01:FS01:0x21:*********************** Log Started 2016-03-30T13:45:28Z ***********************
13:45:29:WU01:FS01:0x21:Project: 9441 (Run 3, Clone 0, Gen 131)
13:45:29:WU01:FS01:0x21:Unit: 0x000000a0ab436c9d56af1b2809b08103
13:45:29:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
13:45:29:WU01:FS01:0x21:Machine: 1
13:45:29:WU01:FS01:0x21:Reading tar file core.xml
13:45:29:WU01:FS01:0x21:Reading tar file integrator.xml
13:45:29:WU01:FS01:0x21:Reading tar file state.xml
13:45:29:WU01:FS01:0x21:Reading tar file system.xml
13:45:30:WU01:FS01:0x21:Digital signatures verified
13:45:30:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
13:45:30:WU01:FS01:0x21:Version 0.0.17
13:45:34:WU02:FS01:Upload 0.82%
13:45:38:WU01:FS01:0x21:Completed 0 out of 2500000 steps (0%)
13:45:39:WU01:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
13:45:40:WU02:FS01:Upload 1.65%
13:45:46:WU02:FS01:Upload 2.64%
13:45:52:WU02:FS01:Upload 3.46%
13:45:58:WU02:FS01:Upload 4.28%
13:46:04:WU02:FS01:Upload 5.27%
13:46:10:WU02:FS01:Upload 6.09%

Edited to save space

13:56:22:WU02:FS01:Upload 96.19%
13:56:26:WU01:FS01:0x21:Completed 50000 out of 2500000 steps (2%)
13:56:28:WU02:FS01:Upload 97.01%
13:56:35:WU02:FS01:Upload 98.00%
13:56:41:WU02:FS01:Upload 98.99%
13:56:47:WU02:FS01:Upload 99.81%
13:56:48:WU02:FS01:Upload complete
13:56:48:WU02:FS01:Server responded WORK_QUIT (404)
13:56:48:WARNING:WU02:FS01:Server did not like results, dumping
13:56:49:WU02:FS01:Cleaning up
14:01:52:WU01:FS01:0x21:Completed 75000 out of 2500000 steps (3%)
14:07:16:WU01:FS01:0x21:Completed 100000 out of 2500000 steps (4%)
14:12:43:WU01:FS01:0x21:Completed 125000 out of 2500000 steps (5%)
14:18:06:WU01:FS01:0x21:Completed 150000 out of 2500000 steps (6%)
14:23:33:WU01:FS01:0x21:Completed 175000 out of 2500000 steps (7%)
14:28:57:WU01:FS01:0x21:Completed 200000 out of 2500000 steps (8%)
14:34:23:WU01:FS01:0x21:Completed 225000 out of 2500000 steps (9%)
14:39:47:WU01:FS01:0x21:Completed 250000 out of 2500000 steps (10%)
14:45:13:WU01:FS01:0x21:Completed 275000 out of 2500000 steps (11%)
14:50:37:WU01:FS01:0x21:Completed 300000 out of 2500000 steps (12%)
14:56:04:WU01:FS01:0x21:Completed 325000 out of 2500000 steps (13%)
15:01:28:WU01:FS01:0x21:Completed 350000 out of 2500000 steps (14%)
15:06:25:WARNING:WU01:FS01:FahCore returned an unknown error code which probably indicates that it crashed
15:06:25:WARNING:WU01:FS01:FahCore returned: UNKNOWN_ENUM (127 = 0x7f)

I've received a couple more WU's and they seem to be fine. Although I did just get a core 18 project 9679 (Run 1, Clone 1, Gen 103) that only took +/- 1 hour and 5 minutes to complete. Never had one take so little time to complete before, weird.

Was 9210 just a bad WU?

[bruce]

Project: 9210 (Run 56, Clone 14, Gen 0) is strange. It's not a bad WU in the traditional sense. Three people attempted it and failed but the server finally assigned it to someone who was able to complete it. I can't explain it.

CPUId: XXXXXXXX5911A069
Days taken to complete WU: 0.02
Your WU (P9210 R56 C14 G0) was added to the stats database on 2016-02-28 02:11:42 for 0 points of credit.

CPUId: XXXXXXXX5911A069
Days taken to complete WU: 0
Your WU (P9210 R56 C14 G0) was added to the stats database on 2016-03-28 05:05:40 for 0 points of credit.

CPUId: XXXXXXXX12C4350D
Days taken to complete WU: 0.02
Your WU (P9210 R56 C14 G0) was added to the stats database on 2016-03-28 18:05:56 for 0 points of credit.

CPUId: XXXXXXXX55EDFA6F
Days taken to complete WU: 0.47
Your WU (P9210 R56 C14 G0) was added to the stats database on 2016-03-29 05:05:11 for 140150 points of credit.

None of these four people are Yakbreeder.
EDIT:
... nor Hypno_Toad

[Yakbreeder]

Thanks for the reply Bruce.

I don't fold under Yakbreeder sorry for the confusion. I fold under a friends name (Hypno_Toad) that got me into folding. He passed back in '11, so I fold in his memory.

Not sure how all this works on the backend, but should it have even been assigned to me? Seems to be a waste of resources if someone else got it, completed it, and then I get it again 5-6 hours later, only to be rejected.

Thank you for your reply, and explanation of this particular WU.

Yak

[toTOW]

What happened after the UNKNOWN_ENUM (127 = 0x7f) error message ?

I'm sometimes getting this error, but the core usually restarts, resume the WU from checkpoint and then complete it without issue.

[Yakbreeder]

ToTOW here's the log from that error onward, starting from the last line in my first post with this error.

It looks like it continued with a new WU (Project: 9441 (Run 3, Clone 0, Gen 131)) and I haven't had any problems since.

Code: Select all

15:06:25:WARNING:WU01:FS01:FahCore returned: UNKNOWN_ENUM (127 = 0x7f)
15:06:25:WU01:FS01:Starting
15:06:25:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Yak/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 704 -lifeline 3308 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
15:06:25:WU01:FS01:Started FahCore on PID 20900
15:06:25:WU01:FS01:Core PID:20912
15:06:25:WU01:FS01:FahCore 0x21 started
15:06:26:WU01:FS01:0x21:*********************** Log Started 2016-03-30T15:06:26Z ***********************
15:06:26:WU01:FS01:0x21:Project: 9441 (Run 3, Clone 0, Gen 131)
15:06:26:WU01:FS01:0x21:Unit: 0x000000a0ab436c9d56af1b2809b08103
15:06:26:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
15:06:26:WU01:FS01:0x21:Machine: 1
15:06:26:WU01:FS01:0x21:Digital signatures verified
15:06:26:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
15:06:26:WU01:FS01:0x21:Version 0.0.17
15:06:26:WU01:FS01:0x21:  Found a checkpoint file
15:06:35:WU01:FS01:0x21:Completed 350000 out of 2500000 steps (14%)
15:06:35:WU01:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
15:12:00:WU01:FS01:0x21:Completed 375000 out of 2500000 steps (15%)
15:17:26:WU01:FS01:0x21:Completed 400000 out of 2500000 steps (16%)

Edited to save space

16:33:29:WU01:FS01:0x21:Completed 750000 out of 2500000 steps (30%)
16:38:55:WU01:FS01:0x21:Completed 775000 out of 2500000 steps (31%)
******************************* Date: 2016-03-30 *******************************
16:44:19:WU01:FS01:0x21:Completed 800000 out of 2500000 steps (32%)
16:49:46:WU01:FS01:0x21:Completed 825000 out of 2500000 steps (33%)


Edited to save space


22:25:58:WU01:FS01:0x21:Completed 2375000 out of 2500000 steps (95%)
22:31:28:WU01:FS01:0x21:Completed 2400000 out of 2500000 steps (96%)
22:37:04:WU01:FS01:0x21:Completed 2425000 out of 2500000 steps (97%)
******************************* Date: 2016-03-30 *******************************
22:43:03:WU01:FS01:0x21:Completed 2450000 out of 2500000 steps (98%)
22:48:57:WU01:FS01:0x21:Completed 2475000 out of 2500000 steps (99%)
22:48:58:WU00:FS01:Connecting to 171.67.108.45:80
22:48:59:WU00:FS01:Assigned to work server 171.67.108.155
22:48:59:WU00:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GM107 [GeForce GTX 750 Ti] from 171.67.108.155
22:48:59:WU00:FS01:Connecting to 171.67.108.155:8080
22:49:00:WU00:FS01:Downloading 388.46KiB
22:49:01:WU00:FS01:Download complete
22:49:01:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9679 run:1 clone:1 gen:103 core:0x18 unit:0x0000006dab436c9b56de69bfb94464e9
22:55:44:WU01:FS01:0x21:Completed 2500000 out of 2500000 steps (100%)
22:55:49:WU01:FS01:0x21:Saving result file logfile_01.txt
22:55:49:WU01:FS01:0x21:Saving result file checkpointState.xml
22:55:50:WU01:FS01:0x21:Saving result file checkpt.crc
22:55:50:WU01:FS01:0x21:Saving result file log.txt
22:55:50:WU01:FS01:0x21:Saving result file positions.xtc
22:55:50:WU01:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
22:55:51:WU01:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
22:55:51:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:9441 run:3 clone:0 gen:131 core:0x21 unit:0x000000a0ab436c9d56af1b2809b08103
22:55:52:WU01:FS01:Uploading 25.29MiB to 171.67.108.157
22:55:52:WU01:FS01:Connecting to 171.67.108.157:8080
22:55:52:WU00:FS01:Starting
22:55:52:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Yak/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 00 -suffix 01 -version 704 -lifeline 3308 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
22:55:52:WU00:FS01:Started FahCore on PID 21264
22:55:52:WU00:FS01:Core PID:20592
22:55:52:WU00:FS01:FahCore 0x18 started
22:55:53:WU00:FS01:0x18:*********************** Log Started 2016-03-30T22:55:52Z ***********************

Here's my warning log going back to the 27th of March. Hasn't missed a beat since.

Code: Select all

******************************* Date: 2016-03-26 *******************************
******************************* Date: 2016-03-27 *******************************
******************************* Date: 2016-03-27 *******************************
******************************* Date: 2016-03-27 *******************************
******************************* Date: 2016-03-27 *******************************
******************************* Date: 2016-03-28 *******************************
******************************* Date: 2016-03-28 *******************************
******************************* Date: 2016-03-28 *******************************
******************************* Date: 2016-03-28 *******************************
******************************* Date: 2016-03-29 *******************************
******************************* Date: 2016-03-29 *******************************
******************************* Date: 2016-03-29 *******************************
******************************* Date: 2016-03-29 *******************************
******************************* Date: 2016-03-30 *******************************
******************************* Date: 2016-03-30 *******************************
13:56:48:WARNING:WU02:FS01:Server did not like results, dumping
15:06:25:WARNING:WU01:FS01:FahCore returned an unknown error code which probably indicates that it crashed
15:06:25:WARNING:WU01:FS01:FahCore returned: UNKNOWN_ENUM (127 = 0x7f)
******************************* Date: 2016-03-30 *******************************
******************************* Date: 2016-03-30 *******************************
******************************* Date: 2016-03-31 *******************************
******************************* Date: 2016-03-31 *******************************

[bruce]

Not sure how all this works on the backend, but should it have even been assigned to me? Seems to be a waste of resources if someone else got it, completed it, and then I get it again 5-6 hours later, only to be rejected.

Good question.

When a new WU is tested in the lab, followed by beta testing, followed by Adv testing ... certain problems are identified. Each round of testing expands the possibility of encountering all possible systems configurations. (Lab testing is quite limited, beta testing exposes the project to a larger but still limited set of volunteers, etc.)

The first questions to be asked are what makes those first three systems different from the fourth? Where is the data to answer that question?

Maybe those questions have answers, and if something can be identified that allows FAH to restrict specific assignments, they'll do it. Obviously the earlier those questions can be answered, the better.

Of course another question is whether we're talking about a single WU out of a single project or to the entire project. From what information I can gather, it appears to be the project. I've asked Jadeshi to look into it.

[jadeshi]

Currently looking into this. Moving 9205-9213 back to Advanced until this is resolved.