13422 failing on RX 5700XT Linux

jaos · Post by **jaos** » Tue Aug 18, 2020 3:38 pm

Getting a lot of these over the last 3 or 4 days:

15:10:42:WU00:FS00:Starting
15:10:42:WU00:FS00:Running FahCore: /home/jason/projects/fah/FAHCoreWrapper /home/jason/projects/fah/cores/cores.foldingathome.org/lin/64bit/22-0.0.11/Core_22.fah/FahCore_22 -dir 00 -suffix 01 -version 706 -lifeline 7094 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
15:10:42:WU00:FS00:Started FahCore on PID 2836921
15:10:42:WU00:FS00:Core PID:2836925
15:10:42:WU00:FS00:FahCore 0x22 started
15:10:42:WU00:FS00:0x22:*********************** Log Started 2020-08-18T15:10:42Z ***********************
15:10:42:WU00:FS00:0x22:*************************** Core22 Folding@home Core ***************************
15:10:42:WU00:FS00:0x22:       Core: Core22
15:10:42:WU00:FS00:0x22:       Type: 0x22
15:10:42:WU00:FS00:0x22:    Version: 0.0.11
15:10:42:WU00:FS00:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
15:10:42:WU00:FS00:0x22:  Copyright: 2020 foldingathome.org
15:10:42:WU00:FS00:0x22:   Homepage: https://foldingathome.org/
15:10:42:WU00:FS00:0x22:       Date: Jun 27 2020
15:10:42:WU00:FS00:0x22:       Time: 22:50:00
15:10:42:WU00:FS00:0x22:   Revision: cfc2940c5dd1aa80f60daa6e28d4a2a417f74edb
15:10:42:WU00:FS00:0x22:     Branch: core22-0.0.11
15:10:42:WU00:FS00:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
15:10:42:WU00:FS00:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
15:10:42:WU00:FS00:0x22:             -funroll-loops
15:10:42:WU00:FS00:0x22:   Platform: linux2 4.19.76-linuxkit
15:10:42:WU00:FS00:0x22:       Bits: 64
15:10:42:WU00:FS00:0x22:       Mode: Release
15:10:42:WU00:FS00:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
15:10:42:WU00:FS00:0x22:             <peastman@stanford.edu>
15:10:42:WU00:FS00:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 2836921 -checkpoint 15
15:10:42:WU00:FS00:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
15:10:42:WU00:FS00:0x22:************************************ libFAH ************************************
15:10:42:WU00:FS00:0x22:       Date: Jun 27 2020
15:10:42:WU00:FS00:0x22:       Time: 22:11:04
15:10:42:WU00:FS00:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
15:10:42:WU00:FS00:0x22:     Branch: HEAD
15:10:42:WU00:FS00:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
15:10:42:WU00:FS00:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
15:10:42:WU00:FS00:0x22:             -funroll-loops
15:10:42:WU00:FS00:0x22:   Platform: linux2 4.19.76-linuxkit
15:10:42:WU00:FS00:0x22:       Bits: 64
15:10:42:WU00:FS00:0x22:       Mode: Release
15:10:42:WU00:FS00:0x22:************************************ CBang *************************************
15:10:42:WU00:FS00:0x22:       Date: Jun 27 2020
15:10:42:WU00:FS00:0x22:       Time: 22:10:11
15:10:42:WU00:FS00:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
15:10:42:WU00:FS00:0x22:     Branch: HEAD
15:10:42:WU00:FS00:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
15:10:42:WU00:FS00:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
15:10:42:WU00:FS00:0x22:             -funroll-loops -fPIC
15:10:42:WU00:FS00:0x22:   Platform: linux2 4.19.76-linuxkit
15:10:42:WU00:FS00:0x22:       Bits: 64
15:10:42:WU00:FS00:0x22:       Mode: Release
15:10:42:WU00:FS00:0x22:************************************ System ************************************
15:10:42:WU00:FS00:0x22:        CPU: AMD Ryzen 9 3900X 12-Core Processor
15:10:42:WU00:FS00:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
15:10:42:WU00:FS00:0x22:       CPUs: 24
15:10:42:WU00:FS00:0x22:     Memory: 31.37GiB
15:10:42:WU00:FS00:0x22:Free Memory: 14.20GiB
15:10:42:WU00:FS00:0x22:    Threads: POSIX_THREADS
15:10:42:WU00:FS00:0x22: OS Version: 5.8
15:10:42:WU00:FS00:0x22:Has Battery: false
15:10:42:WU00:FS00:0x22: On Battery: false
15:10:42:WU00:FS00:0x22: UTC Offset: -4
15:10:42:WU00:FS00:0x22:        PID: 2836925
15:10:42:WU00:FS00:0x22:        CWD: /home/jason/projects/fah/work
15:10:42:WU00:FS00:0x22:********************************************************************************
15:10:42:WU00:FS00:0x22:Project: 13422 (Run 1348, Clone 81, Gen 0)
15:10:42:WU00:FS00:0x22:Unit: 0x0000000012bc7d9a5f3992f28af35bbc
15:10:42:WU00:FS00:0x22:Reading tar file core.xml
15:10:42:WU00:FS00:0x22:Reading tar file integrator.xml
15:10:42:WU00:FS00:0x22:Reading tar file state.xml.bz2
15:10:42:WU00:FS00:0x22:Reading tar file system.xml.bz2
15:10:42:WU00:FS00:0x22:Digital signatures verified
15:10:42:WU00:FS00:0x22:Folding@home GPU Core22 Folding@home Core
15:10:42:WU00:FS00:0x22:Version 0.0.11
15:10:42:WU00:FS00:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
15:10:42:WU00:FS00:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
15:10:42:WU00:FS00:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
15:10:42:WU00:FS00:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
15:10:45:WU02:FS00:Upload complete
15:10:45:WU02:FS00:Server responded WORK_ACK (400)
15:10:45:WU02:FS00:Cleaning up
15:10:51:WU01:FS01:0xa7:Completed 180000 out of 250000 steps (72%)
15:10:56:WU00:FS00:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
15:10:56:WU00:FS00:0x22:Saving result file ../logfile_01.txt
15:10:56:WU00:FS00:0x22:Saving result file science.log
15:10:56:WU00:FS00:0x22:Saving result file state.xml.bz2
15:10:56:WU00:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
15:10:56:WARNING:WU00:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)

Driver info

Code: Select all

Number of platforms:                             1
  Platform Profile:                              FULL_PROFILE
  Platform Version:                              OpenCL 2.1 AMD-APP (3143.9)
  Platform Name:                                 AMD Accelerated Parallel Processing
  Platform Vendor:                               Advanced Micro Devices, Inc.
  Platform Extensions:                           cl_khr_icd cl_amd_event_callback cl_amd_offline_devices
  Platform Name:                                 AMD Accelerated Parallel Processing
Number of devices:                               1
  Device Type:                                   CL_DEVICE_TYPE_GPU
  Vendor ID:                                     1002h
  Board name:                                    AMD Radeon RX 5700 XT
  Device Topology:                               PCI[ B#11, D#0, F#0 ]
  Max compute units:                             20
  Max work items dimensions:                     3
    Max work items[0]:                           1024
    Max work items[1]:                           1024
    Max work items[2]:                           1024
  Max work group size:                           256
  Preferred vector width char:                   4
  Preferred vector width short:                  2
  Preferred vector width int:                    1
  Preferred vector width long:                   1
  Preferred vector width float:                  1
  Preferred vector width double:                 1
  Native vector width char:                      4
  Native vector width short:                     2
  Native vector width int:                       1
  Native vector width long:                      1
  Native vector width float:                     1
  Native vector width double:                    1
  Max clock frequency:                           2030Mhz
  Address bits:                                  64
  Max memory allocation:                         7059013632
  Image support:                                 Yes
  Max number of images read arguments:           128
  Max number of images write arguments:          64
  Max image 2D width:                            16384
  Max image 2D height:                           16384
  Max image 3D width:                            2048
  Max image 3D height:                           2048
  Max image 3D depth:                            2048
  Max samplers within kernel:                    16
  Max size of kernel argument:                   1024
  Alignment (bits) of base address:              2048
  Minimum alignment (bytes) for any datatype:    128
  Single precision floating point capability
    Denorms:                                     Yes
    Quiet NaNs:                                  Yes
    Round to nearest even:                       Yes
    Round to zero:                               Yes
    Round to +ve and infinity:                   Yes
    IEEE754-2008 fused multiply-add:             Yes
  Cache type:                                    Read/Write
  Cache line size:                               64
  Cache size:                                    16384
  Global memory size:                            8573157376
  Constant buffer size:                          7059013632
  Max number of constant args:                   8
  Local memory type:                             Scratchpad
  Local memory size:                             65536
  Max pipe arguments:                            16     
  Max pipe active reservations:                  16
  Max pipe packet size:                          2764046336
  Max global variable size:                      6353112064
  Max global variable preferred total size:      8573157376
  Max read/write image args:                     64 
  Max on device events:                          1024
  Queue on device max size:                      8388608
  Max on device queues:                          1
  Queue on device preferred size:                262144
  SVM capabilities:
    Coarse grain buffer:                         Yes
    Fine grain buffer:                           Yes
    Fine grain system:                           No
    Atomics:                                     No
  Preferred platform atomic alignment:           0
  Preferred global atomic alignment:             0
  Preferred local atomic alignment:              0
  Kernel Preferred work group size multiple:     32
  Error correction support:                      0
  Unified memory for Host and Device:            0
  Profiling timer resolution:                    1
  Device endianess:                              Little
  Available:                                     Yes
  Compiler available:                            Yes  
  Execution capabilities:
    Execute OpenCL kernels:                      Yes
    Execute native function:                     No
  Queue on Host properties:
    Out-of-Order:                                No
    Profiling :                                  Yes
  Queue on Device properties:
    Out-of-Order:                                Yes
    Profiling :                                  Yes
  Platform ID:                                   0x7ff047c05f30
Name:                                          gfx1010
  Vendor:                                        Advanced Micro Devices, Inc.
  Device OpenCL C version:                       OpenCL C 2.0
  Driver version:                                3143.9 (PAL,LC)
  Profile:                                       FULL_PROFILE
  Version:                                       OpenCL 2.0 AMD-APP (3143.9)
  Extensions:                                    cl_khr_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_gl_sharing cl_khr_gl_depth_images cl_amd_device_attribute_query cl_amd_media_ops cl_amd_media_ops2 cl_khr_image2d_from_buffer cl_khr_subgroups cl_khr_gl_event cl_khr_depth_images cl_khr_mipmap_image cl_khr_mipmap_image_writes cl_amd_copy_buffer_p2p

Post by **JohnChodera** » Tue Aug 18, 2020 7:54 pm

Apologies for this. I think the most likely explanation is that some of the compounds are just not able to fit into the active site well, leading to large forces.
If you get *nothing* but these blowups for 13422---let me know!

~ John Chodera // MSKCC

jaos · Post by **jaos** » Tue Aug 18, 2020 8:11 pm

Looks like I only had one succeed out of 72:

Code: Select all

15:31:15:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:722 clone:68 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c32a01a85a18
15:31:33:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:722 clone:96 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c32a5114dafc
15:31:51:WU03:FS00:Sending unit results: id:03 state:SEND error:FAULTY project:13422 run:723 clone:26 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c32afc63c5bf
15:32:08:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:723 clone:55 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c328e189e1f8
15:32:25:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:723 clone:87 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c32885ea228f
15:32:41:WU03:FS00:Sending unit results: id:03 state:SEND error:FAULTY project:13422 run:724 clone:16 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c32acbcb99d0
15:32:58:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:101 clone:15 gen:1 core:0x22 unit:0x0000000112bc7d9a5f3885139b030fe7
15:33:14:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:724 clone:87 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c3283016911e
15:33:31:WU03:FS00:Sending unit results: id:03 state:SEND error:FAULTY project:13422 run:191 clone:4 gen:1 core:0x22 unit:0x0000000112bc7d9a5f38851e6807486c
15:33:49:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:725 clone:51 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c328ba5f3401
15:37:18:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:729 clone:54 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c32a5b4f3d7c
17:27:42:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:854 clone:95 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c32d27245442
17:28:00:WU03:FS00:Sending unit results: id:03 state:SEND error:FAULTY project:13422 run:856 clone:68 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c32e23f4ded9
17:28:17:WU04:FS00:Sending unit results: id:04 state:SEND error:FAULTY project:13422 run:857 clone:8 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c32d6b16bdfc
17:28:32:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:857 clone:47 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c32d1550d175
17:28:46:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:857 clone:79 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c32d7c0700df
17:29:04:WU03:FS00:Sending unit results: id:03 state:SEND error:FAULTY project:13422 run:858 clone:22 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c32e344c64a4
17:29:22:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:858 clone:50 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c32ebf4719b5
17:29:39:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:858 clone:97 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c32fe5d5895c
17:29:57:WU03:FS00:Sending unit results: id:03 state:SEND error:FAULTY project:13422 run:859 clone:39 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c32d54b9f7d9
17:30:14:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:859 clone:77 gen:0 core:0x22 unit:0x0000000012bc7d9a5f38c32db1bb15a1
03:17:16:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:447 clone:87 gen:0 core:0x22 unit:0x0000000112bc7d9a5f38c31c0acb97df
07:04:36:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:557 clone:82 gen:1 core:0x22 unit:0x0000000112bc7d9a5f38c31f04a919e8
07:04:52:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:300 clone:55 gen:0 core:0x22 unit:0x0000000112bc7d9a5f38c31809893395
07:05:11:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:403 clone:83 gen:2 core:0x22 unit:0x0000000312bc7d9a5f38c31938b10a3d
07:05:30:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:502 clone:43 gen:1 core:0x22 unit:0x0000000112bc7d9a5f38c31ed9902e69
07:05:48:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:558 clone:3 gen:0 core:0x22 unit:0x0000000112bc7d9a5f38c31f88d740e4
07:06:05:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:508 clone:5 gen:1 core:0x22 unit:0x0000000212bc7d9a5f38c32145fab313
07:06:21:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:437 clone:46 gen:2 core:0x22 unit:0x0000000212bc7d9a5f38c31e315480a9
07:06:36:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:140 clone:46 gen:1 core:0x22 unit:0x0000000212bc7d9a5f3885183f460714
14:06:15:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:13422 run:676 clone:11 gen:1 core:0x22 unit:0x0000000112bc7d9a5f38c3264c6d47bc
19:13:33:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:913 clone:24 gen:1 core:0x22 unit:0x0000000112bc7d9a5f38c3314f8b4cbf
19:13:50:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:853 clone:28 gen:1 core:0x22 unit:0x0000000212bc7d9a5f38c32d02e91283
19:14:09:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:913 clone:52 gen:1 core:0x22 unit:0x0000000212bc7d9a5f38c3313a76a790
19:14:25:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:914 clone:31 gen:1 core:0x22 unit:0x0000000112bc7d9a5f38c330a00f0253
19:14:44:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:732 clone:98 gen:2 core:0x22 unit:0x0000000212bc7d9a5f38c3290bd5c842
23:00:46:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:915 clone:57 gen:2 core:0x22 unit:0x0000000212bc7d9a5f38c3302b11f237
23:01:03:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:1001 clone:59 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3992e2e5797909
23:01:21:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:910 clone:41 gen:2 core:0x22 unit:0x0000000212bc7d9a5f38c3320836c88e
23:01:39:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:871 clone:13 gen:2 core:0x22 unit:0x0000000412bc7d9a5f38c32f2624fbf9
23:01:57:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:23 clone:49 gen:2 core:0x22 unit:0x0000000412bc7d9a5f3881e98fd2517b
23:02:15:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:431 clone:69 gen:2 core:0x22 unit:0x0000000212bc7d9a5f38c31ed5364a9d
23:02:33:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:731 clone:34 gen:2 core:0x22 unit:0x0000000412bc7d9a5f38c329556d460a
23:02:48:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:995 clone:96 gen:1 core:0x22 unit:0x0000000312bc7d9a5f38c33489981f6c
23:03:06:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:497 clone:69 gen:1 core:0x22 unit:0x0000000212bc7d9a5f38c32179ce4daf
23:03:22:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:899 clone:19 gen:2 core:0x22 unit:0x0000000212bc7d9a5f38c3304967218e
13:15:35:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:704 clone:75 gen:1 core:0x22 unit:0x0000000312bc7d9a5f38c32a7181eb10
13:15:52:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:1224 clone:55 gen:1 core:0x22 unit:0x0000000312bc7d9a5f3992ea84478b68
13:16:09:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:560 clone:88 gen:0 core:0x22 unit:0x0000000212bc7d9a5f38c31f0724ab49
13:16:27:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:603 clone:82 gen:2 core:0x22 unit:0x0000000412bc7d9a5f38c323af056f6c
13:16:43:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:658 clone:39 gen:1 core:0x22 unit:0x0000000312bc7d9a5f38c325636cf289
13:17:00:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:1120 clone:83 gen:2 core:0x22 unit:0x0000000412bc7d9a5f3992e9be56a962
13:17:15:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:1216 clone:94 gen:1 core:0x22 unit:0x0000000112bc7d9a5f3992ebc3247e4c
13:17:34:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:1146 clone:29 gen:1 core:0x22 unit:0x0000000112bc7d9a5f3992e8d77208c0
13:17:49:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:742 clone:42 gen:1 core:0x22 unit:0x0000000212bc7d9a5f38c3290bbc45a1
15:08:18:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:1347 clone:4 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3992f28b93b612
15:08:36:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:1223 clone:2 gen:2 core:0x22 unit:0x0000000212bc7d9a5f3992ead9396665
15:08:52:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:1107 clone:71 gen:1 core:0x22 unit:0x0000000212bc7d9a5f3992e707d97072
15:09:09:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:1348 clone:23 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3992f29852838f
15:09:25:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:1225 clone:14 gen:2 core:0x22 unit:0x0000000212bc7d9a5f3992eabbefed5c
15:09:44:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:1275 clone:22 gen:2 core:0x22 unit:0x0000000212bc7d9a5f3992ef3369b993
15:09:59:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:1323 clone:52 gen:1 core:0x22 unit:0x0000000112bc7d9a5f3992f38ad32a1e
15:10:22:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:1060 clone:96 gen:1 core:0x22 unit:0x0000000112bc7d9a5f3992e3a41b336b
15:10:40:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:1348 clone:62 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3992f214ed2575
15:10:56:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:13422 run:1348 clone:81 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3992f28af35bbc
15:24:38:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:1353 clone:27 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3992f11c13eb0b
15:24:55:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:1319 clone:47 gen:1 core:0x22 unit:0x0000000212bc7d9a5f3992f35281c43b
15:25:11:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:1353 clone:60 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3992f157d1be4a
15:25:28:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:1353 clone:75 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3992f1996dae7c
15:25:47:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:1322 clone:83 gen:1 core:0x22 unit:0x0000000112bc7d9a5f3992f3573d5271
15:26:05:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:1042 clone:70 gen:2 core:0x22 unit:0x0000000212bc7d9a5f3992e451d87770
15:26:23:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:1216 clone:93 gen:1 core:0x22 unit:0x0000000112bc7d9a5f3992eb29e9e39a

jaos · Post by **jaos** » Tue Aug 18, 2020 8:32 pm

Latest one seems to be running so far at 39%

Code: Select all

19:16:43:WU01:FS00:0x22:Project: 13422 (Run 583, Clone 10, Gen 2)
19:16:43:WU01:FS00:0x22:Unit: 0x0000000312bc7d9a5f38c3242ee9bc24

mr_raider · Post by **mr_raider** » Tue Aug 18, 2020 9:15 pm

i have the same issue on an RX 5700, running under linux: forces are blowing up!

Code: Select all

21:11:26:WU00:FS01:Connecting to 18.188.125.154:8080
21:11:26:WU02:FS01:Connecting to assign1.foldingathome.org:80
21:11:26:WU02:FS01:Assigned to work server 18.188.125.154
21:11:26:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] from 18.188.125.154
21:11:26:WU02:FS01:Connecting to 18.188.125.154:8080
21:11:26:WU02:FS01:Downloading 7.03MiB
21:11:27:WU02:FS01:Download complete
21:11:27:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13422 run:1463 clone:45 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3992f7863257a8
21:11:27:WU02:FS01:Starting
21:11:27:WU02:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.11/Core_22.fah/FahCore_22 -dir 02 -suffix 01 -version 706 -lifeline 51328 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
21:11:27:WU02:FS01:Started FahCore on PID 65167
21:11:27:WU02:FS01:Core PID:65171
21:11:27:WU02:FS01:FahCore 0x22 started
21:11:28:WU02:FS01:0x22:*********************** Log Started 2020-08-18T21:11:27Z ***********************
21:11:28:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
21:11:28:WU02:FS01:0x22:       Core: Core22
21:11:28:WU02:FS01:0x22:       Type: 0x22
21:11:28:WU02:FS01:0x22:    Version: 0.0.11
21:11:28:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
21:11:28:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
21:11:28:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
21:11:28:WU02:FS01:0x22:       Date: Jun 27 2020
21:11:28:WU02:FS01:0x22:       Time: 22:50:00
21:11:28:WU02:FS01:0x22:   Revision: cfc2940c5dd1aa80f60daa6e28d4a2a417f74edb
21:11:28:WU02:FS01:0x22:     Branch: core22-0.0.11
21:11:28:WU02:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
21:11:28:WU02:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
21:11:28:WU02:FS01:0x22:             -funroll-loops
21:11:28:WU02:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
21:11:28:WU02:FS01:0x22:       Bits: 64
21:11:28:WU02:FS01:0x22:       Mode: Release
21:11:28:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
21:11:28:WU02:FS01:0x22:             <peastman@stanford.edu>
21:11:28:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 65167 -checkpoint 15
21:11:28:WU02:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
21:11:28:WU02:FS01:0x22:************************************ libFAH ************************************
21:11:28:WU02:FS01:0x22:       Date: Jun 27 2020
21:11:28:WU02:FS01:0x22:       Time: 22:11:04
21:11:28:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
21:11:28:WU02:FS01:0x22:     Branch: HEAD
21:11:28:WU02:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
21:11:28:WU02:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
21:11:28:WU02:FS01:0x22:             -funroll-loops
21:11:28:WU02:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
21:11:28:WU02:FS01:0x22:       Bits: 64
21:11:28:WU02:FS01:0x22:       Mode: Release
21:11:28:WU02:FS01:0x22:************************************ CBang *************************************
21:11:28:WU02:FS01:0x22:       Date: Jun 27 2020
21:11:28:WU02:FS01:0x22:       Time: 22:10:11
21:11:28:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
21:11:28:WU02:FS01:0x22:     Branch: HEAD
21:11:28:WU02:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
21:11:28:WU02:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
21:11:28:WU02:FS01:0x22:             -funroll-loops -fPIC
21:11:28:WU02:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
21:11:28:WU02:FS01:0x22:       Bits: 64
21:11:28:WU02:FS01:0x22:       Mode: Release
21:11:28:WU02:FS01:0x22:************************************ System ************************************
21:11:28:WU02:FS01:0x22:        CPU: AMD Ryzen 7 2700X Eight-Core Processor
21:11:28:WU02:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
21:11:28:WU02:FS01:0x22:       CPUs: 16
21:11:28:WU02:FS01:0x22:     Memory: 15.63GiB
21:11:28:WU02:FS01:0x22:Free Memory: 2.54GiB
21:11:28:WU02:FS01:0x22:    Threads: POSIX_THREADS
21:11:28:WU02:FS01:0x22: OS Version: 5.7
21:11:28:WU02:FS01:0x22:Has Battery: false
21:11:28:WU02:FS01:0x22: On Battery: false
21:11:28:WU02:FS01:0x22: UTC Offset: -4
21:11:28:WU02:FS01:0x22:        PID: 65171
21:11:28:WU02:FS01:0x22:        CWD: /var/lib/fahclient/work
21:11:28:WU02:FS01:0x22:********************************************************************************
21:11:28:WU02:FS01:0x22:Project: 13422 (Run 1463, Clone 45, Gen 0)
21:11:28:WU02:FS01:0x22:Unit: 0x0000000012bc7d9a5f3992f7863257a8
21:11:28:WU02:FS01:0x22:Reading tar file core.xml
21:11:28:WU02:FS01:0x22:Reading tar file integrator.xml
21:11:28:WU02:FS01:0x22:Reading tar file state.xml.bz2
21:11:28:WU02:FS01:0x22:Reading tar file system.xml.bz2
21:11:28:WU02:FS01:0x22:Digital signatures verified
21:11:28:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
21:11:28:WU02:FS01:0x22:Version 0.0.11
21:11:28:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
21:11:28:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
21:11:28:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
21:11:28:WU02:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
21:11:30:WU00:FS01:Upload complete
21:11:30:WU00:FS01:Server responded WORK_ACK (400)
21:11:30:WU00:FS01:Cleaning up
21:11:48:WU02:FS01:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
21:11:48:WU02:FS01:0x22:Saving result file ../logfile_01.txt
21:11:48:WU02:FS01:0x22:Saving result file science.log
21:11:48:WU02:FS01:0x22:Saving result file state.xml.bz2
21:11:48:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
21:11:48:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
21:11:48:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:1463 clone:45 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3992f7863257a8
21:11:48:WU02:FS01:Uploading 5.60MiB to 18.188.125.154
21:11:48:WU02:FS01:Connecting to 18.188.125.154:8080
21:11:52:WU02:FS01:Upload complete
21:11:52:WU02:FS01:Server responded WORK_ACK (400)
21:11:52:WU02:FS01:Cleaning up

ThWuensche · Post by **ThWuensche** » Wed Aug 19, 2020 2:43 pm

On my Radeon VIIs on Linux all of these WUs fail right at the beginning. Errors include particle coordinate nan, potential energy error, RMSE error. 13416 had a lot of fails, 13418 was a little better, 13420 bad again, 13422 doesn't work at all.

Though I really want to support project Moonshot and created my systems to support that project, until these problems are fixed I decided to avoid that type of WUs. I switched the disease selection to cancer. As a result I still get Covid WUs, mostly in the range 1174x, 1175x. These WUs my GPUs crunch away with PPD in the range 1.2 - 1.5 million PPD without any trouble. However I have some doubt about the scientific value of completing these WUs. Question to the researchers: Do these WUs provide additional insight or are they only scheduled to avoid that people complain about missing GPU WUs?

NormalDiffusion · Post by **NormalDiffusion** » Thu Aug 20, 2020 1:07 pm

Only for information:
13422 running without problems on 2x Radeon VII and 1x R9 290x under Win 10
Actually one of the best WUs with over 2 million PPD on the rvii.

Nuitari · Post by **Nuitari** » Thu Aug 20, 2020 3:46 pm

Going through the same with 13423 on Linux with RX570 / RX560 / APU
190 out of 193 WUs came back faulty, today. Woke up to all 7 GPUs being deteceted as faulty by the client.

I'm dropping the rig out of advanced as I'm just tired of baby sitting this.

Code: Select all

14:14:09:WU01:FS07:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1416 clone:42 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993741987e088
14:14:10:WU03:FS06:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:13423 run:1416 clone:44 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937476c85cda
14:14:17:WU02:FS04:0x22:ERROR:Force RMSE error of 13.4173 with threshold of 5
14:14:17:WU00:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:14:19:WU01:FS07:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:14:20:WU05:FS03:Received Unit: id:05 state:DOWNLOAD error:NO_ERROR project:13423 run:1417 clone:2 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399374591f9dbd
14:14:20:WU03:FS06:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:14:21:WU07:FS07:Received Unit: id:07 state:DOWNLOAD error:NO_ERROR project:13423 run:1417 clone:4 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399374bda0d292
14:14:22:WU06:FS04:Received Unit: id:06 state:DOWNLOAD error:NO_ERROR project:13423 run:1417 clone:5 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399374cbf3ad02
14:14:23:WU08:FS06:Received Unit: id:08 state:DOWNLOAD error:NO_ERROR project:13423 run:1417 clone:8 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993744677d844
14:14:29:WU05:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:14:30:WU07:FS07:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:14:31:WU06:FS04:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:14:31:WU09:FS03:Received Unit: id:09 state:DOWNLOAD error:NO_ERROR project:13423 run:1405 clone:13 gen:0 core:0x22 unit:0x0000000112bc7d9a5f39937454b46002
14:14:33:WU08:FS06:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:14:33:WU10:FS07:Received Unit: id:10 state:DOWNLOAD error:NO_ERROR project:13423 run:1417 clone:12 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993749927e5bb
14:14:33:WU12:FS04:Received Unit: id:12 state:DOWNLOAD error:NO_ERROR project:13423 run:1417 clone:13 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399374a8cfbde7
14:14:36:WU13:FS06:Received Unit: id:13 state:DOWNLOAD error:NO_ERROR project:13423 run:1417 clone:17 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399374d5d5078c
14:14:40:WU09:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:14:41:WU14:FS02:Received Unit: id:14 state:DOWNLOAD error:NO_ERROR project:13423 run:1417 clone:23 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399371fcab3364
14:14:43:WU10:FS07:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:14:43:WU12:FS04:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:14:43:WU15:FS03:Received Unit: id:15 state:DOWNLOAD error:NO_ERROR project:13423 run:1383 clone:23 gen:0 core:0x22 unit:0x0000000412bc7d9a5f39936ef536ef26
14:14:45:WU13:FS06:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:14:46:WU16:FS04:Received Unit: id:16 state:DOWNLOAD error:NO_ERROR project:13423 run:1417 clone:22 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399374dcd0882e
14:14:48:WU17:FS07:Received Unit: id:17 state:DOWNLOAD error:NO_ERROR project:13423 run:1417 clone:25 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993744b0a58e2
14:14:50:WU18:FS06:Received Unit: id:18 state:DOWNLOAD error:NO_ERROR project:13423 run:1405 clone:36 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399374f16cb184
14:14:50:WU14:FS02:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:14:52:WU15:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:14:53:WU19:FS02:Received Unit: id:19 state:DOWNLOAD error:NO_ERROR project:13423 run:1417 clone:34 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993747c2f5c64
14:14:55:WU20:FS03:Received Unit: id:20 state:DOWNLOAD error:NO_ERROR project:13423 run:1417 clone:36 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399374cf1d6fb8
14:14:55:WU16:FS04:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:14:58:WU17:FS07:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:14:58:WU21:FS04:Received Unit: id:21 state:DOWNLOAD error:NO_ERROR project:13423 run:1395 clone:14 gen:0 core:0x22 unit:0x0000000212bc7d9a5f39936ed1e8b145
14:14:59:WU18:FS06:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:15:00:WU22:FS07:Received Unit: id:22 state:DOWNLOAD error:NO_ERROR project:13423 run:1417 clone:43 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993711cf7d984
14:15:02:WU23:FS06:Received Unit: id:23 state:DOWNLOAD error:NO_ERROR project:13423 run:1417 clone:33 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993743502cf4a
14:15:05:WU20:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:15:08:WU24:FS03:Received Unit: id:24 state:DOWNLOAD error:NO_ERROR project:13423 run:1417 clone:44 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937430c85ed6
14:15:11:WU22:FS07:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:15:13:WU25:FS07:Received Unit: id:25 state:DOWNLOAD error:NO_ERROR project:13423 run:1417 clone:49 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399374cd46b060
14:15:13:WU23:FS06:0x22:ERROR:Discrepancy: Forces are blowing up! 3 1
14:15:16:WU26:FS06:Received Unit: id:26 state:DOWNLOAD error:NO_ERROR project:13423 run:1418 clone:2 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937498052e40
14:15:18:WU24:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:15:21:WU27:FS03:Received Unit: id:27 state:DOWNLOAD error:NO_ERROR project:13423 run:1405 clone:48 gen:0 core:0x22 unit:0x0000000212bc7d9a5f3993745991460f
14:15:23:WU25:FS07:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:15:26:WU00:FS07:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1418 clone:9 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993748849bba5
14:15:31:WU27:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:15:33:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1418 clone:14 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993749843eaf7
14:15:36:WU00:FS07:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:15:39:WU02:FS07:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13423 run:1418 clone:18 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937428ac2a1a
14:15:44:WU01:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:15:46:WU03:FS03:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:13423 run:1418 clone:30 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937408005992
14:15:50:WU02:FS07:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:15:52:WU05:FS07:Received Unit: id:05 state:DOWNLOAD error:NO_ERROR project:13423 run:1407 clone:3 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993715e5491cf
14:15:57:WU03:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:15:59:WU06:FS03:Received Unit: id:06 state:DOWNLOAD error:NO_ERROR project:13423 run:1406 clone:43 gen:0 core:0x22 unit:0x0000000212bc7d9a5f399371b8b50d2d
14:16:02:WU05:FS07:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:16:05:WU07:FS07:Received Unit: id:07 state:DOWNLOAD error:NO_ERROR project:13423 run:1419 clone:2 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937437aa914a
14:16:10:WU06:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:16:16:WU07:FS07:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:16:29:WU07:FS03:Received Unit: id:07 state:DOWNLOAD error:NO_ERROR project:13423 run:1419 clone:28 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399374be2d8a71
14:16:31:WU08:FS07:Received Unit: id:08 state:DOWNLOAD error:NO_ERROR project:13423 run:1419 clone:32 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399374f589472b
14:16:39:WU07:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:16:42:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1419 clone:43 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937447705ec1
14:16:52:WU00:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:16:55:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1408 clone:10 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399374c144271f
14:17:05:WU01:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:17:08:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1420 clone:26 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993745823c328
14:17:19:WU00:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:17:21:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1408 clone:34 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399374205c7f2c
14:17:32:WU01:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:17:34:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1408 clone:44 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399374bd4ad3c2
14:17:45:WU00:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:17:48:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1421 clone:16 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937411bb8ec7
14:17:58:WU01:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:18:01:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1421 clone:20 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399374cc3103fc
14:18:12:WU00:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:18:14:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1409 clone:24 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993746413db37
14:18:25:WU01:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:18:28:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1399 clone:3 gen:0 core:0x22 unit:0x0000000212bc7d9a5f399371e7522222
14:18:38:WU00:FS03:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0
14:18:54:WU04:FS00:Received Unit: id:04 state:DOWNLOAD error:NO_ERROR project:13423 run:1399 clone:23 gen:0 core:0x22 unit:0x0000000212bc7d9a5f39937187a58b33
14:19:59:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1418 clone:4 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993741f9e47f0
14:20:11:WU00:FS03:0x22:ERROR:Force RMSE error of 12.6225 with threshold of 5
14:20:13:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1423 clone:47 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937112e8ccc7
14:20:26:WU01:FS03:0x22:ERROR:Force RMSE error of 13.2715 with threshold of 5
14:20:28:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1424 clone:14 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993732b07f591
14:20:41:WU00:FS03:0x22:ERROR:Force RMSE error of 15.9349 with threshold of 5
14:20:43:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1408 clone:48 gen:0 core:0x22 unit:0x0000000212bc7d9a5f3993744ade4c52
14:20:55:WU01:FS03:0x22:ERROR:Force RMSE error of 17.0333 with threshold of 5
14:20:58:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:610 clone:12 gen:0 core:0x22 unit:0x0000000112bc7d9a5f38c343b8e88778
14:21:10:WU00:FS03:0x22:ERROR:Force RMSE error of 14.612 with threshold of 5
14:21:13:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1424 clone:39 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993732386e151
14:21:25:WU01:FS03:0x22:ERROR:Force RMSE error of 15.9349 with threshold of 5
14:21:28:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1402 clone:13 gen:0 core:0x22 unit:0x0000000212bc7d9a5f39937411a25533
14:21:40:WU00:FS03:0x22:ERROR:Force RMSE error of 15.6288 with threshold of 5
14:21:43:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1425 clone:25 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399373eb2c3bcd
14:21:55:WU01:FS03:0x22:ERROR:Force RMSE error of 18.6977 with threshold of 5
14:21:58:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1414 clone:38 gen:0 core:0x22 unit:0x0000000212bc7d9a5f3993749d5bb031
14:22:10:WU00:FS03:0x22:ERROR:Force RMSE error of 13.7595 with threshold of 5
14:22:13:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1425 clone:48 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399373a3a3a1c8
14:22:25:WU01:FS03:0x22:ERROR:Force RMSE error of 18.6977 with threshold of 5
14:26:04:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1429 clone:43 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399371870abaab
14:26:17:WU00:FS03:0x22:ERROR:Force RMSE error of 13.9136 with threshold of 5
14:26:19:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1430 clone:21 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399373cfc95725
14:26:32:WU01:FS03:0x22:ERROR:Force RMSE error of 18.1821 with threshold of 5
14:26:34:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1408 clone:21 gen:0 core:0x22 unit:0x0000000212bc7d9a5f399374556b15bd
14:26:47:WU00:FS03:0x22:ERROR:Force RMSE error of 17.0333 with threshold of 5
14:26:49:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1430 clone:44 gen:0 core:0x22 unit:0x0000000112bc7d9a5f39937313446faa
14:27:02:WU01:FS03:0x22:ERROR:Force RMSE error of 18.1821 with threshold of 5
14:27:04:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1431 clone:10 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937366fa5294
14:27:17:WU00:FS03:0x22:ERROR:Force RMSE error of 10.2489 with threshold of 5
14:27:20:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1431 clone:12 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993731b247812
14:27:32:WU01:FS03:0x22:ERROR:Force RMSE error of 10.2489 with threshold of 5
14:27:35:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1426 clone:3 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993714cffe19a
14:27:48:WU00:FS03:0x22:ERROR:Force RMSE error of 10.7144 with threshold of 5
14:27:51:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1421 clone:42 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399373da49713c
14:28:03:WU01:FS03:0x22:ERROR:Force RMSE error of 15.1533 with threshold of 5
14:28:06:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1422 clone:5 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399373d9f869b9
14:28:18:WU00:FS03:0x22:ERROR:Force RMSE error of 13.9645 with threshold of 5
14:28:21:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1420 clone:15 gen:0 core:0x22 unit:0x0000000212bc7d9a5f399371c5059bc8
14:28:33:WU01:FS03:0x22:ERROR:Force RMSE error of 11.6005 with threshold of 5
14:30:10:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1434 clone:28 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937391c8bdeb
14:30:23:WU00:FS03:0x22:ERROR:Force RMSE error of 11.205 with threshold of 5
14:30:25:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:610 clone:6 gen:0 core:0x22 unit:0x0000000212bc7d9a5f38c341751b96b2
14:30:38:WU01:FS03:0x22:ERROR:Force RMSE error of 14.612 with threshold of 5
14:30:40:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1435 clone:1 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399373496c7430
14:30:53:WU00:FS03:0x22:ERROR:Force RMSE error of 13.5835 with threshold of 5
14:30:55:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1435 clone:4 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993734682f270
14:31:07:WU01:FS03:0x22:ERROR:Force RMSE error of 13.5835 with threshold of 5
14:31:10:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1435 clone:27 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399373af71bb7c
14:31:22:WU00:FS03:0x22:ERROR:Force RMSE error of 13.5835 with threshold of 5
14:31:25:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1435 clone:41 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937324353bc5
14:31:37:WU01:FS03:0x22:ERROR:Force RMSE error of 13.5835 with threshold of 5
14:31:40:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1425 clone:41 gen:0 core:0x22 unit:0x0000000212bc7d9a5f399373825f58d4
14:31:52:WU00:FS03:0x22:ERROR:Force RMSE error of 18.6977 with threshold of 5
14:31:55:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1436 clone:11 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993726d056af1
14:32:08:WU01:FS03:0x22:ERROR:Force RMSE error of 10.7933 with threshold of 5
14:32:10:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1403 clone:31 gen:0 core:0x22 unit:0x0000000312bc7d9a5f399371e8f8c982
14:32:23:WU00:FS03:0x22:ERROR:Force RMSE error of 14.4514 with threshold of 5
14:32:25:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1436 clone:27 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937298525ea3
14:32:38:WU01:FS03:0x22:ERROR:Force RMSE error of 10.7933 with threshold of 5
14:33:33:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1437 clone:39 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399372ca86c166
14:33:45:WU00:FS03:0x22:ERROR:Force RMSE error of 11.6252 with threshold of 5
14:33:48:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1438 clone:6 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937325366012
14:34:00:WU01:FS03:0x22:ERROR:Force RMSE error of 15.8425 with threshold of 5
14:34:03:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1438 clone:18 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937318bc73c5
14:34:15:WU00:FS03:0x22:ERROR:Force RMSE error of 15.8425 with threshold of 5
14:34:18:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1438 clone:34 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993731135865b
14:34:30:WU01:FS03:0x22:ERROR:Force RMSE error of 15.8425 with threshold of 5
14:34:33:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1438 clone:38 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993734577ecc8
14:34:45:WU00:FS03:0x22:ERROR:Force RMSE error of 15.8425 with threshold of 5
14:34:48:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1438 clone:49 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399373ef700df4
14:35:01:WU01:FS03:0x22:ERROR:Force RMSE error of 15.8425 with threshold of 5
14:35:03:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1439 clone:21 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399373535be6a4
14:35:16:WU00:FS03:0x22:ERROR:Force RMSE error of 18.6777 with threshold of 5
14:35:19:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1419 clone:12 gen:0 core:0x22 unit:0x0000000212bc7d9a5f39937417e97d49
14:35:31:WU01:FS03:0x22:ERROR:Force RMSE error of 13.5776 with threshold of 5
14:35:34:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1439 clone:42 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937309326da6
14:35:46:WU00:FS03:0x22:ERROR:Force RMSE error of 18.6777 with threshold of 5
14:35:48:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1440 clone:7 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993730db829d8
14:36:01:WU01:FS03:0x22:ERROR:Force RMSE error of 13.5489 with threshold of 5
14:40:27:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1444 clone:49 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993732cb7628b
14:40:39:WU00:FS03:0x22:ERROR:Force RMSE error of 16.2029 with threshold of 5
14:40:41:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1445 clone:11 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399372e9b7d8de
14:40:54:WU01:FS03:0x22:ERROR:Force RMSE error of 14.4903 with threshold of 5
14:40:56:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1445 clone:21 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399373cebffd63
14:41:09:WU00:FS03:0x22:ERROR:Force RMSE error of 14.4903 with threshold of 5
14:41:12:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1436 clone:2 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993734f501d2c
14:41:24:WU01:FS03:0x22:ERROR:Force RMSE error of 10.7933 with threshold of 5
14:41:27:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1436 clone:13 gen:0 core:0x22 unit:0x0000000112bc7d9a5f39937370a83b8a
14:41:40:WU00:FS03:0x22:ERROR:Force RMSE error of 10.7933 with threshold of 5
14:41:42:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1446 clone:4 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993731d1c4260
14:41:55:WU01:FS03:0x22:ERROR:Force RMSE error of 13.4815 with threshold of 5
14:41:57:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1446 clone:35 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399372eec76ae0
14:42:10:WU00:FS03:0x22:ERROR:Force RMSE error of 13.4815 with threshold of 5
14:42:13:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1447 clone:4 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399373d2d5524c
14:42:25:WU01:FS03:0x22:ERROR:Force RMSE error of 15.5684 with threshold of 5
14:42:28:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1447 clone:21 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399373ab53534e
14:42:40:WU00:FS03:0x22:ERROR:Force RMSE error of 15.5684 with threshold of 5
14:42:43:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1427 clone:41 gen:0 core:0x22 unit:0x0000000212bc7d9a5f399373877496a2
14:42:55:WU01:FS03:0x22:ERROR:Force RMSE error of 15.01 with threshold of 5
14:44:13:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1438 clone:45 gen:0 core:0x22 unit:0x0000000212bc7d9a5f399373f620e84e
14:44:26:WU00:FS03:0x22:ERROR:Force RMSE error of 15.8425 with threshold of 5
14:44:29:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1449 clone:14 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399373271d631f
14:44:42:WU01:FS03:0x22:ERROR:Force RMSE error of 13.3277 with threshold of 5
14:44:44:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1449 clone:20 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399373c44954f4
14:44:56:WU00:FS03:0x22:ERROR:Force RMSE error of 13.3277 with threshold of 5
14:44:59:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1449 clone:42 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399373186c2108
14:45:11:WU01:FS03:0x22:ERROR:Force RMSE error of 13.3277 with threshold of 5
14:45:14:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1439 clone:41 gen:0 core:0x22 unit:0x0000000312bc7d9a5f39937329a578ac
14:45:26:WU00:FS03:0x22:ERROR:Force RMSE error of 18.6777 with threshold of 5
14:45:29:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1450 clone:21 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399373fc717958
14:45:41:WU01:FS03:0x22:ERROR:Force RMSE error of 12.9487 with threshold of 5
14:45:44:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1450 clone:31 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993739b0b1797
14:45:57:WU00:FS03:0x22:ERROR:Force RMSE error of 12.9487 with threshold of 5
14:45:59:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1440 clone:27 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399372dde28c09
14:46:12:WU01:FS03:0x22:ERROR:Force RMSE error of 13.5489 with threshold of 5
14:46:14:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1451 clone:9 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937319b78653
14:46:27:WU00:FS03:0x22:ERROR:Force RMSE error of 9.43793 with threshold of 5
14:46:29:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1431 clone:25 gen:0 core:0x22 unit:0x0000000212bc7d9a5f399373a8364b3a
14:46:42:WU01:FS03:0x22:ERROR:Force RMSE error of 10.2489 with threshold of 5
14:47:55:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:776 clone:35 gen:0 core:0x22 unit:0x0000000412bc7d9a5f38c34df6d85333
14:48:08:WU00:FS03:0x22:ERROR:Force RMSE error of 11.2752 with threshold of 5
14:48:10:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1443 clone:3 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399372909b351a
14:48:22:WU01:FS03:0x22:ERROR:Force RMSE error of 12.3551 with threshold of 5
14:48:25:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1453 clone:24 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993727c82618e
14:48:37:WU00:FS03:0x22:ERROR:Force RMSE error of 13.8274 with threshold of 5
14:48:40:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1453 clone:41 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399372466cac7b
14:48:52:WU01:FS03:0x22:ERROR:Force RMSE error of 13.8274 with threshold of 5
14:48:55:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1454 clone:4 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399372209d17ab
14:49:07:WU00:FS03:0x22:ERROR:Force RMSE error of 16.6798 with threshold of 5
14:49:09:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1454 clone:15 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399372fbcc9072
14:49:22:WU01:FS03:0x22:ERROR:Force RMSE error of 16.6798 with threshold of 5
14:49:24:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1454 clone:29 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937216adef7f
14:49:36:WU00:FS03:0x22:ERROR:Force RMSE error of 16.6798 with threshold of 5
14:49:39:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1454 clone:45 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993726b4f39c0
14:49:51:WU01:FS03:0x22:ERROR:Force RMSE error of 16.6798 with threshold of 5
14:49:54:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1455 clone:2 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993721da96dbd
14:50:07:WU00:FS03:0x22:ERROR:Force RMSE error of 12.9964 with threshold of 5
14:50:09:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1455 clone:19 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937207565dcb
14:50:22:WU01:FS03:0x22:ERROR:Force RMSE error of 12.9964 with threshold of 5
14:50:55:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:615 clone:17 gen:0 core:0x22 unit:0x0000000112bc7d9a5f38c34352dcd538
14:51:07:WU00:FS03:0x22:ERROR:Force RMSE error of 11.4579 with threshold of 5
14:51:09:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1456 clone:14 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993721434d92c
14:51:22:WU01:FS03:0x22:ERROR:Force RMSE error of 9.88709 with threshold of 5
14:51:25:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1456 clone:28 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993729329d6a7
14:51:37:WU00:FS03:0x22:ERROR:Force RMSE error of 9.88709 with threshold of 5
14:51:39:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1446 clone:19 gen:0 core:0x22 unit:0x0000000212bc7d9a5f399372aae819cc
14:51:52:WU01:FS03:0x22:ERROR:Force RMSE error of 13.4815 with threshold of 5
14:51:55:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1456 clone:48 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993729ae9b36a
14:52:08:WU00:FS03:0x22:ERROR:Force RMSE error of 9.88709 with threshold of 5
14:52:10:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1427 clone:31 gen:0 core:0x22 unit:0x0000000412bc7d9a5f3993715ffd514e
14:52:22:WU01:FS03:0x22:ERROR:Force RMSE error of 15.01 with threshold of 5
14:52:25:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1457 clone:35 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937288da0f97
14:52:37:WU00:FS03:0x22:ERROR:Force RMSE error of 14.0756 with threshold of 5
14:52:40:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1457 clone:42 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399372e51683a2
14:52:53:WU01:FS03:0x22:ERROR:Force RMSE error of 14.0756 with threshold of 5
14:52:55:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1458 clone:9 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993723d9108b2
14:53:08:WU00:FS03:0x22:ERROR:Force RMSE error of 13.8693 with threshold of 5
14:53:10:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1458 clone:20 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399372c0c91e6d
14:53:23:WU01:FS03:0x22:ERROR:Force RMSE error of 13.8693 with threshold of 5
14:54:43:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1460 clone:5 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993720726f857
14:54:56:WU00:FS03:0x22:ERROR:Force RMSE error of 17.45 with threshold of 5
14:54:58:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1450 clone:23 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399372a894834f
14:55:11:WU01:FS03:0x22:ERROR:Force RMSE error of 12.9487 with threshold of 5
14:55:13:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1460 clone:31 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937285a27c8f
14:55:26:WU00:FS03:0x22:ERROR:Force RMSE error of 17.45 with threshold of 5
14:55:28:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1450 clone:40 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399373ed2980e8
14:55:41:WU01:FS03:0x22:ERROR:Force RMSE error of 12.9487 with threshold of 5
14:55:44:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1450 clone:47 gen:0 core:0x22 unit:0x0000000212bc7d9a5f399373b0c27092
14:55:56:WU00:FS03:0x22:ERROR:Force RMSE error of 12.9487 with threshold of 5
14:55:59:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1461 clone:12 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993721ed6a595
14:56:11:WU01:FS03:0x22:ERROR:Force RMSE error of 11.879 with threshold of 5
14:56:13:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1461 clone:24 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399372cf1c1ff9
14:56:26:WU00:FS03:0x22:ERROR:Force RMSE error of 11.879 with threshold of 5
14:56:28:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1461 clone:38 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993727c9374f6
14:56:41:WU01:FS03:0x22:ERROR:Force RMSE error of 11.879 with threshold of 5
14:56:43:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1452 clone:13 gen:0 core:0x22 unit:0x0000000212bc7d9a5f3993738d707a33
14:56:56:WU00:FS03:0x22:ERROR:Force RMSE error of 11.605 with threshold of 5
14:56:59:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1462 clone:35 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937128890b8d
14:57:11:WU01:FS03:0x22:ERROR:Force RMSE error of 13.4033 with threshold of 5
14:57:39:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1463 clone:27 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399372c72b6ad7
14:57:52:WU00:FS03:0x22:ERROR:Force RMSE error of 11.6829 with threshold of 5
14:57:54:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1463 clone:41 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399372c135d958
14:58:06:WU01:FS03:0x22:ERROR:Force RMSE error of 11.6829 with threshold of 5
14:58:09:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1464 clone:10 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937296ce37dd
14:58:21:WU00:FS03:0x22:ERROR:Force RMSE error of 11.0501 with threshold of 5
14:58:24:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1464 clone:23 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399372d8833490
14:58:36:WU01:FS03:0x22:ERROR:Force RMSE error of 11.0501 with threshold of 5
14:58:39:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1464 clone:37 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937231d2b26c
14:58:51:WU00:FS03:0x22:ERROR:Force RMSE error of 11.0501 with threshold of 5
14:58:54:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1465 clone:2 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937220b40a7c
14:59:06:WU01:FS03:0x22:ERROR:Force RMSE error of 16.4834 with threshold of 5
14:59:09:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1465 clone:24 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993721c349490
14:59:21:WU00:FS03:0x22:ERROR:Force RMSE error of 16.4834 with threshold of 5
14:59:23:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1465 clone:43 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399371df95210d
14:59:36:WU01:FS03:0x22:ERROR:Force RMSE error of 16.4834 with threshold of 5
14:59:38:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1466 clone:9 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937264a39751
14:59:51:WU00:FS03:0x22:ERROR:Force RMSE error of 13.1426 with threshold of 5
14:59:53:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1466 clone:11 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993717d020e36
15:00:06:WU01:FS03:0x22:ERROR:Force RMSE error of 13.1426 with threshold of 5
15:00:54:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1456 clone:24 gen:0 core:0x22 unit:0x0000000112bc7d9a5f39937284728250
15:01:07:WU00:FS03:0x22:ERROR:Force RMSE error of 9.88709 with threshold of 5
15:01:09:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1456 clone:35 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399372b5bd8ccc
15:01:22:WU01:FS03:0x22:ERROR:Force RMSE error of 9.88709 with threshold of 5
15:01:24:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1468 clone:18 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399372f8126c3f
15:01:37:WU00:FS03:0x22:ERROR:Force RMSE error of 12.4322 with threshold of 5
15:01:40:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1468 clone:30 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993724c54562d
15:01:52:WU01:FS03:0x22:ERROR:Force RMSE error of 12.4322 with threshold of 5
15:01:55:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1468 clone:46 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937203e956e7
15:02:08:WU00:FS03:0x22:ERROR:Force RMSE error of 12.4322 with threshold of 5
15:02:10:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1469 clone:1 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399372ed4a7f64
15:02:22:WU01:FS03:0x22:ERROR:Force RMSE error of 14.4379 with threshold of 5
15:02:25:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1469 clone:22 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993720d20d86d
15:02:37:WU00:FS03:0x22:ERROR:Force RMSE error of 14.4379 with threshold of 5
15:02:40:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1458 clone:17 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993726fb98fde
15:02:52:WU01:FS03:0x22:ERROR:Force RMSE error of 13.8693 with threshold of 5
15:02:55:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1469 clone:47 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937176b7a716
15:03:08:WU00:FS03:0x22:ERROR:Force RMSE error of 14.4379 with threshold of 5
15:03:11:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1470 clone:8 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937285410140
15:03:23:WU01:FS03:0x22:ERROR:Force RMSE error of 14.0927 with threshold of 5
15:03:38:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1470 clone:44 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937275177f35
15:03:50:WU00:FS03:0x22:ERROR:Force RMSE error of 14.0927 with threshold of 5
15:03:53:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1471 clone:8 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937205b22184
15:04:05:WU01:FS03:0x22:ERROR:Force RMSE error of 15.454 with threshold of 5
15:04:08:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1471 clone:18 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937253fe6f61
15:04:20:WU00:FS03:0x22:ERROR:Force RMSE error of 15.454 with threshold of 5
15:04:23:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1471 clone:29 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937263cefa49
15:04:35:WU01:FS03:0x22:ERROR:Force RMSE error of 15.454 with threshold of 5
15:04:38:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1440 clone:10 gen:0 core:0x22 unit:0x0000000312bc7d9a5f399373bf4a5147
15:04:50:WU00:FS03:0x22:ERROR:Force RMSE error of 13.5489 with threshold of 5
15:04:53:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1472 clone:8 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399372d659bd68
15:05:05:WU01:FS03:0x22:ERROR:Force RMSE error of 15.2034 with threshold of 5
15:05:08:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1461 clone:1 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993729a794d3e
15:05:20:WU00:FS03:0x22:ERROR:Force RMSE error of 11.879 with threshold of 5
15:05:23:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1472 clone:26 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399372df0471f0
15:05:35:WU01:FS03:0x22:ERROR:Force RMSE error of 15.2034 with threshold of 5
15:05:38:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1472 clone:41 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993723d5ac9c4
15:05:50:WU00:FS03:0x22:ERROR:Force RMSE error of 15.2034 with threshold of 5
15:05:53:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1473 clone:7 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399371f4e22e91
15:06:05:WU01:FS03:0x22:ERROR:Force RMSE error of 12.8863 with threshold of 5
15:06:14:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1473 clone:29 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993722b20d923
15:06:26:WU00:FS03:0x22:ERROR:Force RMSE error of 12.8863 with threshold of 5
15:06:29:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1473 clone:44 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993725507257d
15:06:42:WU01:FS03:0x22:ERROR:Force RMSE error of 12.8863 with threshold of 5
15:06:44:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1463 clone:3 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399371ccd90cc1
15:06:57:WU00:FS03:0x22:ERROR:Force RMSE error of 11.6829 with threshold of 5
15:06:59:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1474 clone:23 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399371701ba82d
15:07:12:WU01:FS03:0x22:ERROR:Force RMSE error of 11.1426 with threshold of 5
15:07:15:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1474 clone:34 gen:0 core:0x22 unit:0x0000000112bc7d9a5f399372ff4b1af7
15:07:27:WU00:FS03:0x22:ERROR:Force RMSE error of 11.1426 with threshold of 5
15:07:30:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1474 clone:45 gen:0 core:0x22 unit:0x0000000112bc7d9a5f39937287d1f4ab
15:07:42:WU01:FS03:0x22:ERROR:Force RMSE error of 11.1426 with threshold of 5
15:07:45:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1475 clone:22 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993725c2358db
15:07:58:WU00:FS03:0x22:ERROR:Force RMSE error of 12.2219 with threshold of 5
15:08:00:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1475 clone:40 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399372e7561373
15:08:13:WU01:FS03:0x22:ERROR:Force RMSE error of 12.2219 with threshold of 5
15:08:16:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1476 clone:3 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937433411d73
15:08:28:WU00:FS03:0x22:ERROR:Force RMSE error of 10.1491 with threshold of 5
15:08:31:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1476 clone:24 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399375fc7e5ed1
15:08:44:WU01:FS03:0x22:ERROR:Force RMSE error of 10.1491 with threshold of 5
15:09:11:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1477 clone:9 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399375ebe3e8ea
15:09:23:WU00:FS03:0x22:ERROR:Force RMSE error of 11.2392 with threshold of 5
15:09:26:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1477 clone:23 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993748694f266
15:09:38:WU01:FS03:0x22:ERROR:Force RMSE error of 11.2392 with threshold of 5
15:09:41:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1477 clone:40 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399375a87b5d30
15:09:53:WU00:FS03:0x22:ERROR:Force RMSE error of 11.2392 with threshold of 5
15:09:56:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1478 clone:2 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399375a806561b
15:10:08:WU01:FS03:0x22:ERROR:Force RMSE error of 14.9984 with threshold of 5
15:10:10:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1478 clone:14 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399375623dfb15
15:10:23:WU00:FS03:0x22:ERROR:Force RMSE error of 14.9984 with threshold of 5
15:10:26:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1478 clone:31 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399375b944995f
15:10:39:WU01:FS03:0x22:ERROR:Force RMSE error of 14.9984 with threshold of 5
15:10:41:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1468 clone:2 gen:0 core:0x22 unit:0x0000000212bc7d9a5f39937221ed9a8b
15:10:54:WU00:FS03:0x22:ERROR:Force RMSE error of 12.4322 with threshold of 5
15:10:56:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1479 clone:9 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993758da16a4a
15:11:09:WU01:FS03:0x22:ERROR:Force RMSE error of 14.7034 with threshold of 5
15:11:11:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1479 clone:20 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993755dd1c6f2
15:11:24:WU00:FS03:0x22:ERROR:Force RMSE error of 14.7034 with threshold of 5
15:11:26:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1479 clone:36 gen:0 core:0x22 unit:0x0000000212bc7d9a5f399375b781e49a
15:11:39:WU01:FS03:0x22:ERROR:Force RMSE error of 14.7034 with threshold of 5
15:39:06:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1478 clone:39 gen:0 core:0x22 unit:0x0000000312bc7d9a5f3993742b9c1775
15:39:18:WU00:FS03:0x22:ERROR:Force RMSE error of 14.9984 with threshold of 5
15:39:21:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1493 clone:39 gen:0 core:0x22 unit:0x0000000212bc7d9a5f399375e9cd305e
15:39:33:WU01:FS03:0x22:ERROR:Force RMSE error of 15.0421 with threshold of 5
15:39:36:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1509 clone:35 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399376087f8731
15:39:48:WU00:FS03:0x22:ERROR:Force RMSE error of 15.7511 with threshold of 5
15:39:51:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1509 clone:48 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399376e6eb0ac0
15:40:03:WU01:FS03:0x22:ERROR:Force RMSE error of 15.7511 with threshold of 5
15:40:06:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1510 clone:3 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399375a0c5018f
15:40:19:WU00:FS03:0x22:ERROR:Force RMSE error of 12.419 with threshold of 5
15:40:21:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1510 clone:22 gen:0 core:0x22 unit:0x0000000012bc7d9a5f3993761804a1c2
15:40:34:WU01:FS03:0x22:ERROR:Force RMSE error of 12.419 with threshold of 5
15:40:36:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1510 clone:37 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937605030536
15:40:49:WU00:FS03:0x22:ERROR:Force RMSE error of 12.419 with threshold of 5
15:40:51:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1510 clone:46 gen:0 core:0x22 unit:0x0000000012bc7d9a5f39937608c7875a
15:41:04:WU01:FS03:0x22:ERROR:Force RMSE error of 12.419 with threshold of 5
15:41:06:WU00:FS03:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13423 run:1511 clone:11 gen:0 core:0x22 unit:0x0000000012bc7d9a5f399376c54a9928
15:41:19:WU00:FS03:0x22:ERROR:Force RMSE error of 12.332 with threshold of 5
15:41:21:WU01:FS03:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13423 run:1501 clone:8 gen:0 core:0x22 unit:0x0000000112bc7d9a5f3993771cfe94ea
15:41:33:WU01:FS03:0x22:ERROR:Force RMSE error of 12.298 with threshold of 5

Neil-B · Post by **Neil-B** » Thu Aug 20, 2020 4:35 pm

... to give your kit the best chance to get away from 134xx for those slots you may need to change the cause away from Covid-19 ... I think the 134xx WUs are handed out to beta, advanced and fah - switching to a different cause that has active projects should put them higher priority for assignment than the 134xx ones.

ThWuensche · Post by **ThWuensche** » Thu Aug 20, 2020 5:56 pm

NormalDiffusion wrote:Only for information:
13422 running without problems on 2x Radeon VII and 1x R9 290x under Win 10
Actually one of the best WUs with over 2 million PPD on the rvii.

Thanks for the information. Probably we can assume that AMD functional code base for openCL under Windows and Linux (for the proprietary drivers) is mostly the same. So the difference is more likely somewhere in the setup than the base functionality. The fact that under Linux the WUs are mostly failing immediately also points to some initialization issues, whether in the core or the openCL implementation. In my observation on 13418/13420 WUs once they did not fail quite at the beginning (up to first checkpoint) they were likely to succeed.

Post by **JohnChodera** » Fri Aug 21, 2020 1:14 am

We're aware of some GPU/driver combinations that seem to produce large discrepancies or issues in forces between GPU and CPU, like is reported here:

Code: Select all

15:10:56:WU00:FS00:0x22:ERROR:Discrepancy: Forces are blowing up! 0 0

We're working on adding instrumentation to the core22 0.0.12 release to drop into debugging mode and send back a lot more information about what is going wrong in these cases. Hopefully we can get this out soon!

Thanks so much for bearing with us!

~ John Chodera // MSKCC

ViTe · Post by **ViTe** » Fri Aug 21, 2020 2:46 am

Just for the info.

13422 WUs running without issues on 2 PCs.
1 - Win7 64bit, MSI RX570 4gb (not overclocked), Adrenalin 20.2.2, Client ver. 7.6.9, OpenCL 2.0 AMD-APP Driver 3004.8
2 - Win7 64bit, Sapphire R9 280X (factory overclocked), Crimson 17.2.1, Client ver. 7.6.9, OpenCL 1.2 AMD-APP Driver 2264.11

Tohya · Post by **Tohya** » Fri Aug 21, 2020 9:15 am

Completed 24 13422 with no errors, on Win10 pro, Adrenalin 20.8.2, Asus rog strix RX 5700 XT

NormalDiffusion · Post by **NormalDiffusion** » Fri Aug 21, 2020 9:50 am

NormalDiffusion wrote:Only for information:
13422 running without problems on 2x Radeon VII and 1x R9 290x under Win 10
Actually one of the best WUs with over 2 million PPD on the rvii.

Ok, have to revise this.
Now 13422 WUs are failing one after this other on the R9 290x.

- 13422 (Run 2316, Clone 11, Gen 0) --> 2x before being dropped
always:

Code: Select all

WU01:FS01:0x22:Completed [color=#FF0000]Y[/color] out of 1000000 steps (0%)
WU01:FS01:0x22:An exception occurred at step [color=#FF0000]XXX[/color]: Particle coordinate is nan

This one managed to put the slot in a "failed" state. GPU not reacting anymore on F@H.

- 13422 (Run 2458, Clone 78, Gen 0) --> 3x before being dropped
always:

Code: Select all

WU00:FS01:0x22:Completed [color=#FF0000]Y[/color] out of 1000000 steps (6%)
WU00:FS01:0x22:An exception occurred at step [color=#FF0000]XXX[/color]: Particle coordinate is nan

This one was at least being partially processed

- 13422 (Run 2471, Clone 29, Gen 0) --> 2x before being dropped
always:

Code: Select all

WU00:FS01:0x22:Completed 0 out of 1000000 steps (0%)
WU00:FS01:0x22:An exception occurred at step 250: Particle coordinate is nan

- 13422 (Run 2471, Clone 42, Gen 0) --> 2x before being dropped
always:

Code: Select all

WU00:FS01:0x22:Completed 0 out of 1000000 steps (0%)
WU00:FS01:0x22:An exception occurred at step 250: Particle coordinate is nan

- 13422 (Run 1766, Clone 90, Gen 1) --> running now, but already restarted...

If more info or needed, just let me know.

PS: but still going strong on the RVIIs!

Post by **PantherX** » Fri Aug 21, 2020 11:40 am

ThWuensche wrote:...Do these WUs provide additional insight or are they only scheduled to avoid that people complain about missing GPU WUs?

The Project series 134XX are generally released in "pairs" (13410/13411 followed by 13412/13413, etc.) and each pair is an improvement over another. The most recent pairs are now sufficiently optimized that they are used for shortlisting potential drug compounds. This kind of simulation is a first for F@H thus, they are considered to be highly experimental and hae a higher rate of error initially but is lower nowadays. This is closely being monitored by John and optimizations are happening as soon as possible.

None of those Projects are "filler" projects to keep the GPUs busy. If there's no suitable WUs, your GPU would be idle.

Folding Forum

13422 failing on RX 5700XT Linux

13422 failing on RX 5700XT Linux

Re: 13422 failing on RX 5700XT Linux

Re: 13422 failing on RX 5700XT Linux

Re: 13422 failing on RX 5700XT Linux

Re: 13422 failing on RX 5700XT Linux

Re: 13422 failing on RX 5700XT Linux

Re: 13422 failing on RX 5700XT Linux

Re: 13422 failing on RX 5700XT Linux

Re: 13422 failing on RX 5700XT Linux

Re: 13422 failing on RX 5700XT Linux

Re: 13422 failing on RX 5700XT Linux

Re: 13422 failing on RX 5700XT Linux

Re: 13422 failing on RX 5700XT Linux

Re: 13422 failing on RX 5700XT Linux

Re: 13422 failing on RX 5700XT Linux