Page 1 of 1

project 13414 - "server did not like results, dumping"

Posted: Thu Jul 02, 2020 2:28 am
by emf

Code: Select all

19:03:30:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:13414 run:6691 clone:4 gen:1 core:0x22 unit:0x0000000212bc7d9a5efc2bea944eb351
19:03:30:WU03:FS01:Starting
19:03:30:WU03:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 03 -suffix 01 -version 706 -lifeline 1630 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
19:03:30:WU03:FS01:Started FahCore on PID 13601
19:03:30:WU03:FS01:Core PID:13605
19:03:30:WU03:FS01:FahCore 0x22 started
19:03:31:WU03:FS01:0x22:*********************** Log Started 2020-07-01T19:03:30Z ***********************
19:03:31:WU03:FS01:0x22:*************************** Core22 Folding@home Core ***************************
19:03:31:WU03:FS01:0x22:       Core: Core22
19:03:31:WU03:FS01:0x22:       Type: 0x22
19:03:31:WU03:FS01:0x22:    Version: 0.0.11
19:03:31:WU03:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:03:31:WU03:FS01:0x22:  Copyright: 2020 foldingathome.org
19:03:31:WU03:FS01:0x22:   Homepage: https://foldingathome.org/
19:03:31:WU03:FS01:0x22:       Date: Jun 27 2020
19:03:31:WU03:FS01:0x22:       Time: 22:50:00
19:03:31:WU03:FS01:0x22:   Revision: cfc2940c5dd1aa80f60daa6e28d4a2a417f74edb
19:03:31:WU03:FS01:0x22:     Branch: core22-0.0.11
19:03:31:WU03:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
19:03:31:WU03:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
19:03:31:WU03:FS01:0x22:             -funroll-loops
19:03:31:WU03:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
19:03:31:WU03:FS01:0x22:       Bits: 64
19:03:31:WU03:FS01:0x22:       Mode: Release
19:03:31:WU03:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
19:03:31:WU03:FS01:0x22:             <peastman@stanford.edu>
19:03:31:WU03:FS01:0x22:       Args: -dir 03 -suffix 01 -version 706 -lifeline 13601 -checkpoint 15
19:03:31:WU03:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device
19:03:31:WU03:FS01:0x22:             1 -gpu 1
19:03:31:WU03:FS01:0x22:************************************ libFAH ************************************
19:03:31:WU03:FS01:0x22:       Date: Jun 27 2020
19:03:31:WU03:FS01:0x22:       Time: 22:11:04
19:03:31:WU03:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
19:03:31:WU03:FS01:0x22:     Branch: HEAD
19:03:31:WU03:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
19:03:31:WU03:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
19:03:31:WU03:FS01:0x22:             -funroll-loops
19:03:31:WU03:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
19:03:31:WU03:FS01:0x22:       Bits: 64
19:03:31:WU03:FS01:0x22:       Mode: Release
19:03:31:WU03:FS01:0x22:************************************ CBang *************************************
19:03:31:WU03:FS01:0x22:       Date: Jun 27 2020
19:03:31:WU03:FS01:0x22:       Time: 22:10:11
19:03:31:WU03:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
19:03:31:WU03:FS01:0x22:     Branch: HEAD
19:03:31:WU03:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
19:03:31:WU03:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
19:03:31:WU03:FS01:0x22:             -funroll-loops -fPIC
19:03:31:WU03:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
19:03:31:WU03:FS01:0x22:       Bits: 64
19:03:31:WU03:FS01:0x22:       Mode: Release
19:03:31:WU03:FS01:0x22:************************************ System ************************************
19:03:31:WU03:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5430 @ 2.66GHz
19:03:31:WU03:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 23 Stepping 6
19:03:31:WU03:FS01:0x22:       CPUs: 4
19:03:31:WU03:FS01:0x22:     Memory: 7.79GiB
19:03:31:WU03:FS01:0x22:Free Memory: 5.07GiB
19:03:31:WU03:FS01:0x22:    Threads: POSIX_THREADS
19:03:31:WU03:FS01:0x22: OS Version: 4.15
19:03:31:WU03:FS01:0x22:Has Battery: false
19:03:31:WU03:FS01:0x22: On Battery: false
19:03:31:WU03:FS01:0x22: UTC Offset: 0
19:03:31:WU03:FS01:0x22:        PID: 13605
19:03:31:WU03:FS01:0x22:        CWD: /var/lib/fahclient/work
19:03:31:WU03:FS01:0x22:********************************************************************************
19:03:31:WU03:FS01:0x22:Project: 13414 (Run 6691, Clone 4, Gen 1)
19:03:31:WU03:FS01:0x22:Unit: 0x0000000212bc7d9a5efc2bea944eb351
19:03:31:WU03:FS01:0x22:Reading tar file core.xml
19:03:31:WU03:FS01:0x22:Reading tar file integrator.xml
19:03:31:WU03:FS01:0x22:Reading tar file state.xml
19:03:31:WU03:FS01:0x22:Reading tar file system.xml
19:03:31:WU03:FS01:0x22:Digital signatures verified
19:03:31:WU03:FS01:0x22:Folding@home GPU Core22 Folding@home Core
19:03:31:WU03:FS01:0x22:Version 0.0.11
19:03:31:WU03:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
19:03:31:WU03:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
19:03:31:WU03:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
19:03:31:WU03:FS01:0x22:  Global context and integrator variables write interval: 250 steps (0.025%) [4000 total]
19:03:53:WU03:FS01:0x22:Completed 0 out of 1000000 steps (0%)
19:07:30:WU03:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
19:11:15:WU03:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
19:15:02:WU03:FS01:0x22:Completed 30000 out of 1000000 steps (3%)
19:18:50:WU03:FS01:0x22:Completed 40000 out of 1000000 steps (4%)
19:22:38:WU03:FS01:0x22:Completed 50000 out of 1000000 steps (5%)
19:26:27:WU03:FS01:0x22:Completed 60000 out of 1000000 steps (6%)
19:30:15:WU03:FS01:0x22:Completed 70000 out of 1000000 steps (7%)
19:34:02:WU03:FS01:0x22:Completed 80000 out of 1000000 steps (8%)
19:37:50:WU03:FS01:0x22:Completed 90000 out of 1000000 steps (9%)
19:41:38:WU03:FS01:0x22:Completed 100000 out of 1000000 steps (10%)
19:45:28:WU03:FS01:0x22:Completed 110000 out of 1000000 steps (11%)
19:49:16:WU03:FS01:0x22:Completed 120000 out of 1000000 steps (12%)
19:53:04:WU03:FS01:0x22:Completed 130000 out of 1000000 steps (13%)
19:56:52:WU03:FS01:0x22:Completed 140000 out of 1000000 steps (14%)
20:00:40:WU03:FS01:0x22:Completed 150000 out of 1000000 steps (15%)
20:04:30:WU03:FS01:0x22:Completed 160000 out of 1000000 steps (16%)
20:08:18:WU03:FS01:0x22:Completed 170000 out of 1000000 steps (17%)
20:12:05:WU03:FS01:0x22:Completed 180000 out of 1000000 steps (18%)
20:15:54:WU03:FS01:0x22:Completed 190000 out of 1000000 steps (19%)
20:19:41:WU03:FS01:0x22:Completed 200000 out of 1000000 steps (20%)
20:23:31:WU03:FS01:0x22:Completed 210000 out of 1000000 steps (21%)
20:27:19:WU03:FS01:0x22:Completed 220000 out of 1000000 steps (22%)
20:31:07:WU03:FS01:0x22:Completed 230000 out of 1000000 steps (23%)
20:34:55:WU03:FS01:0x22:Completed 240000 out of 1000000 steps (24%)
20:38:42:WU03:FS01:0x22:Completed 250000 out of 1000000 steps (25%)
20:42:34:WU03:FS01:0x22:Completed 260000 out of 1000000 steps (26%)
20:46:23:WU03:FS01:0x22:Completed 270000 out of 1000000 steps (27%)
20:50:12:WU03:FS01:0x22:Completed 280000 out of 1000000 steps (28%)
20:54:02:WU03:FS01:0x22:Completed 290000 out of 1000000 steps (29%)
20:57:51:WU03:FS01:0x22:Completed 300000 out of 1000000 steps (30%)
21:01:42:WU03:FS01:0x22:Completed 310000 out of 1000000 steps (31%)
21:05:32:WU03:FS01:0x22:Completed 320000 out of 1000000 steps (32%)
21:09:21:WU03:FS01:0x22:Completed 330000 out of 1000000 steps (33%)
21:13:10:WU03:FS01:0x22:Completed 340000 out of 1000000 steps (34%)
21:16:59:WU03:FS01:0x22:Completed 350000 out of 1000000 steps (35%)
21:20:50:WU03:FS01:0x22:Completed 360000 out of 1000000 steps (36%)
21:24:39:WU03:FS01:0x22:Completed 370000 out of 1000000 steps (37%)
21:28:28:WU03:FS01:0x22:Completed 380000 out of 1000000 steps (38%)
21:32:17:WU03:FS01:0x22:Completed 390000 out of 1000000 steps (39%)
21:36:06:WU03:FS01:0x22:Completed 400000 out of 1000000 steps (40%)
21:39:57:WU03:FS01:0x22:Completed 410000 out of 1000000 steps (41%)
21:43:47:WU03:FS01:0x22:Completed 420000 out of 1000000 steps (42%)
21:47:36:WU03:FS01:0x22:Completed 430000 out of 1000000 steps (43%)
21:51:25:WU03:FS01:0x22:Completed 440000 out of 1000000 steps (44%)
21:55:14:WU03:FS01:0x22:Completed 450000 out of 1000000 steps (45%)
21:59:05:WU03:FS01:0x22:Completed 460000 out of 1000000 steps (46%)
22:02:55:WU03:FS01:0x22:Completed 470000 out of 1000000 steps (47%)
22:06:44:WU03:FS01:0x22:Completed 480000 out of 1000000 steps (48%)
22:10:33:WU03:FS01:0x22:Completed 490000 out of 1000000 steps (49%)
22:14:21:WU03:FS01:0x22:Completed 500000 out of 1000000 steps (50%)
22:18:10:WU03:FS01:0x22:Completed 510000 out of 1000000 steps (51%)
22:21:57:WU03:FS01:0x22:Completed 520000 out of 1000000 steps (52%)
22:25:44:WU03:FS01:0x22:Completed 530000 out of 1000000 steps (53%)
22:29:31:WU03:FS01:0x22:Completed 540000 out of 1000000 steps (54%)
22:33:19:WU03:FS01:0x22:Completed 550000 out of 1000000 steps (55%)
22:37:09:WU03:FS01:0x22:Completed 560000 out of 1000000 steps (56%)
22:40:56:WU03:FS01:0x22:Completed 570000 out of 1000000 steps (57%)
22:44:44:WU03:FS01:0x22:Completed 580000 out of 1000000 steps (58%)
22:48:31:WU03:FS01:0x22:Completed 590000 out of 1000000 steps (59%)
22:52:19:WU03:FS01:0x22:Completed 600000 out of 1000000 steps (60%)
22:56:08:WU03:FS01:0x22:Completed 610000 out of 1000000 steps (61%)
22:59:56:WU03:FS01:0x22:Completed 620000 out of 1000000 steps (62%)
23:03:44:WU03:FS01:0x22:Completed 630000 out of 1000000 steps (63%)
23:07:32:WU03:FS01:0x22:Completed 640000 out of 1000000 steps (64%)
23:11:19:WU03:FS01:0x22:Completed 650000 out of 1000000 steps (65%)
23:15:08:WU03:FS01:0x22:Completed 660000 out of 1000000 steps (66%)
23:18:56:WU03:FS01:0x22:Completed 670000 out of 1000000 steps (67%)
23:22:43:WU03:FS01:0x22:Completed 680000 out of 1000000 steps (68%)
23:26:31:WU03:FS01:0x22:Completed 690000 out of 1000000 steps (69%)
23:30:19:WU03:FS01:0x22:Completed 700000 out of 1000000 steps (70%)
23:34:09:WU03:FS01:0x22:Completed 710000 out of 1000000 steps (71%)
23:37:56:WU03:FS01:0x22:Completed 720000 out of 1000000 steps (72%)
23:41:44:WU03:FS01:0x22:Completed 730000 out of 1000000 steps (73%)
23:45:32:WU03:FS01:0x22:Completed 740000 out of 1000000 steps (74%)
23:49:19:WU03:FS01:0x22:Completed 750000 out of 1000000 steps (75%)
23:53:10:WU03:FS01:0x22:Completed 760000 out of 1000000 steps (76%)
23:56:58:WU03:FS01:0x22:Completed 770000 out of 1000000 steps (77%)
00:00:46:WU03:FS01:0x22:Completed 780000 out of 1000000 steps (78%)
00:04:35:WU03:FS01:0x22:Completed 790000 out of 1000000 steps (79%)
00:08:23:WU03:FS01:0x22:Completed 800000 out of 1000000 steps (80%)
00:12:14:WU03:FS01:0x22:Completed 810000 out of 1000000 steps (81%)
00:16:03:WU03:FS01:0x22:Completed 820000 out of 1000000 steps (82%)
00:19:51:WU03:FS01:0x22:Completed 830000 out of 1000000 steps (83%)
00:23:40:WU03:FS01:0x22:Completed 840000 out of 1000000 steps (84%)
00:27:28:WU03:FS01:0x22:Completed 850000 out of 1000000 steps (85%)
00:31:19:WU03:FS01:0x22:Completed 860000 out of 1000000 steps (86%)
00:35:07:WU03:FS01:0x22:Completed 870000 out of 1000000 steps (87%)
00:38:56:WU03:FS01:0x22:Completed 880000 out of 1000000 steps (88%)
00:42:45:WU03:FS01:0x22:Completed 890000 out of 1000000 steps (89%)
00:46:34:WU03:FS01:0x22:Completed 900000 out of 1000000 steps (90%)
00:50:24:WU03:FS01:0x22:Completed 910000 out of 1000000 steps (91%)
00:54:12:WU03:FS01:0x22:Completed 920000 out of 1000000 steps (92%)
00:58:01:WU03:FS01:0x22:Completed 930000 out of 1000000 steps (93%)
01:01:50:WU03:FS01:0x22:Completed 940000 out of 1000000 steps (94%)
01:05:38:WU03:FS01:0x22:Completed 950000 out of 1000000 steps (95%)
01:09:28:WU03:FS01:0x22:Completed 960000 out of 1000000 steps (96%)
01:13:16:WU03:FS01:0x22:Completed 970000 out of 1000000 steps (97%)
01:17:04:WU03:FS01:0x22:Completed 980000 out of 1000000 steps (98%)
01:20:52:WU03:FS01:0x22:Completed 990000 out of 1000000 steps (99%)
01:24:41:WU03:FS01:0x22:Completed 1000000 out of 1000000 steps (100%)
01:24:41:WU03:FS01:0x22:Average performance: 75.7895 ns/day
01:24:45:WU03:FS01:0x22:Saving result file ../logfile_01.txt
01:24:45:WU03:FS01:0x22:Saving result file checkpointState.xml
01:24:47:WU03:FS01:0x22:Saving result file globals.csv
01:24:47:WU03:FS01:0x22:Saving result file positions.xtc
01:24:47:WU03:FS01:0x22:Saving result file science.log
01:24:47:WU03:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
01:24:48:WU03:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
01:24:48:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:13414 run:6691 clone:4 gen:1 core:0x22 unit:0x0000000212bc7d9a5efc2bea944eb351
01:24:48:WU03:FS01:Uploading 7.50MiB to 18.188.125.154
01:24:48:WU03:FS01:Connecting to 18.188.125.154:8080
01:25:19:WU03:FS01:Upload 0.83%
01:28:15:WU03:FS01:Upload 1.67%
01:28:15:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
01:28:15:WU03:FS01:Trying to send results to collection server
01:28:15:WU03:FS01:Uploading 7.50MiB to 3.21.157.11
01:28:15:WU03:FS01:Connecting to 3.21.157.11:8080
01:28:21:WU03:FS01:Upload 9.17%
01:28:27:WU03:FS01:Upload 15.83%
01:28:33:WU03:FS01:Upload 22.50%
01:28:39:WU03:FS01:Upload 30.00%
01:28:45:WU03:FS01:Upload 36.67%
01:28:51:WU03:FS01:Upload 44.17%
01:28:57:WU03:FS01:Upload 49.17%
01:29:03:WU03:FS01:Upload 56.67%
01:29:09:WU03:FS01:Upload 63.34%
01:29:15:WU03:FS01:Upload 70.84%
01:29:21:WU03:FS01:Upload 77.51%
01:29:27:WU03:FS01:Upload 84.17%
01:29:33:WU03:FS01:Upload 91.68%
01:29:39:WU03:FS01:Upload 98.34%
01:29:43:WU03:FS01:Upload complete
01:29:43:WU03:FS01:Server responded WORK_QUIT (404)
01:29:43:WARNING:WU03:FS01:Server did not like results, dumping
01:29:43:WU03:FS01:Cleaning up

Re: project 13414 - "server did not like results, dumping"

Posted: Thu Jul 02, 2020 4:29 am
by aetch
Server doesn't have any free space, it's been like that for a while now.
https://apps.foldingathome.org/serverstats

Re: project 13414 - "server did not like results, dumping"

Posted: Thu Jul 02, 2020 12:58 pm
by Joe_H
Or again. They have been freeing up space on that CS and it has been filling back up as fast as they get free space. Main issue is that Work Servers that these WUs need to go back to are full, and it is taking time expanding their RAID capacity.