Core A7 RDTSCP problem, Debian on bhyve/iohyve on FreeNAS

[karlinhigh]

I thought I was very smart getting FAH running virtualized on FreeNAS. It completed several WUs until it got one for Core A7. It's unable to run it.

Here's what I've got:
Lenovo ThinkServer TS130, Intel Core i3-4130, 16 GB RAM.
FreeNAS 9.10.2-U4 to run...
bhyve (via iohyve 0.7.7) to run...
Debian 8.8 32-bit

Here's what's happening:

Code: Select all

17:41:07:Saving configuration to /etc/fahclient/config.xml
17:41:07:<config>
17:41:07:  <!-- Client Control -->
17:41:07:  <fold-anon v='true'/>
17:41:07:
17:41:07:  <!-- Folding Slot Configuration -->
17:41:07:  <gpu v='false'/>
17:41:07:
17:41:07:  <!-- HTTP Server -->
17:41:07:  <allow v='127.0.0.1 REDACTED'/>
17:41:07:
17:41:07:  <!-- Network -->
17:41:07:  <proxy v=':8080'/>
17:41:07:
17:41:07:  <!-- Remote Command Server -->
17:41:07:  <password v='********'/>
17:41:07:
17:41:07:  <!-- Slot Control -->
17:41:07:  <pause-on-start v='true'/>
17:41:07:  <power v='full'/>
17:41:07:
17:41:07:  <!-- User Information -->
17:41:07:  <passkey v='********************************'/>
17:41:07:  <team v='REDACTED'/>
17:41:07:  <user v='REDACTED'/>
17:41:07:
17:41:07:  <!-- Folding Slots -->
17:41:07:  <slot id='0' type='CPU'>
17:41:07:    <paused v='true'/>
17:41:07:  </slot>
17:41:07:</config>
17:41:51:FS00:Unpaused
17:41:51:WU01:FS00:Starting
17:41:51:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/AVX/Core_a7.fah/FahCore_a7 -dir 01 -suffix 01 -version 704 -lifeline 4656 -checkpoint 15 -np 4
17:41:51:WU01:FS00:Started FahCore on PID 9931
17:41:51:WU01:FS00:Core PID:9935
17:41:51:WU01:FS00:FahCore 0xa7 started
17:41:52:WU01:FS00:0xa7:*********************** Log Started 2017-06-05T17:41:51Z ***********************
17:41:52:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
17:41:52:WU01:FS00:0xa7:       Type: 0xa7
17:41:52:WU01:FS00:0xa7:       Core: Gromacs
17:41:52:WU01:FS00:0xa7:    Website: http://folding.stanford.edu/
17:41:52:WU01:FS00:0xa7:  Copyright: (c) 2009-2016 Stanford University
17:41:52:WU01:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
17:41:52:WU01:FS00:0xa7:       Args: -dir 01 -suffix 01 -version 704 -lifeline 9931 -checkpoint 15 -np 4
17:41:52:WU01:FS00:0xa7:     Config: <none>
17:41:52:WU01:FS00:0xa7:************************************ Build *************************************
17:41:52:WU01:FS00:0xa7:    Version: 0.0.11
17:41:52:WU01:FS00:0xa7:       Date: Sep 20 2016
17:41:52:WU01:FS00:0xa7:       Time: 06:40:11
17:41:52:WU01:FS00:0xa7: Repository: Git
17:41:52:WU01:FS00:0xa7:   Revision: 957bd90e68d95ddcf1594dc15ff6c64cc4555146
17:41:52:WU01:FS00:0xa7:     Branch: master
17:41:52:WU01:FS00:0xa7:   Compiler: GNU 4.8.5
17:41:52:WU01:FS00:0xa7:    Options: -std=gnu++98 -O3 -funroll-loops -ffast-math -mfpmath=sse
17:41:52:WU01:FS00:0xa7:             -fno-unsafe-math-optimizations -msse2
17:41:52:WU01:FS00:0xa7:   Platform: linux2 4.6.0-1-amd64
17:41:52:WU01:FS00:0xa7:       Bits: 64
17:41:52:WU01:FS00:0xa7:       Mode: Release
17:41:52:WU01:FS00:0xa7:       SIMD: avx_256
17:41:52:WU01:FS00:0xa7:************************************ System ************************************
17:41:52:WU01:FS00:0xa7:        CPU: Intel(R) Core(TM) i3-4130 CPU @ 3.40GHz
17:41:52:WU01:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
17:41:52:WU01:FS00:0xa7:       CPUs: 4
17:41:52:WU01:FS00:0xa7:     Memory: 1.96GiB
17:41:52:WU01:FS00:0xa7:Free Memory: 431.47MiB
17:41:52:WU01:FS00:0xa7:    Threads: POSIX_THREADS
17:41:52:WU01:FS00:0xa7: OS Version: 3.16
17:41:52:WU01:FS00:0xa7:Has Battery: false
17:41:52:WU01:FS00:0xa7: On Battery: false
17:41:52:WU01:FS00:0xa7: UTC Offset: -5
17:41:52:WU01:FS00:0xa7:        PID: 9935
17:41:52:WU01:FS00:0xa7:        CWD: /var/lib/fahclient/work
17:41:52:WU01:FS00:0xa7:         OS: Linux 3.16.0-4-amd64 x86_64
17:41:52:WU01:FS00:0xa7:    OS Arch: AMD64
17:41:52:WU01:FS00:0xa7:********************************************************************************
17:41:52:WU01:FS00:0xa7:Project: 8676 (Run 97, Clone 4, Gen 19)
17:41:52:WU01:FS00:0xa7:Unit: 0x000000160002894b5824aa64011e980b
17:41:52:WU01:FS00:0xa7:Reading tar file core.xml
17:41:52:WU01:FS00:0xa7:Reading tar file frame19.tpr
17:41:52:WU01:FS00:0xa7:Digital signatures verified
17:41:52:WU01:FS00:0xa7:Calling: mdrun -s frame19.tpr -o frame19.trr -cpt 15 -nt 4
17:41:52:WU01:FS00:0xa7:Steps: first=47500000 total=2500000
17:41:52:WU01:FS00:0xa7:ERROR:
17:41:52:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
17:41:52:WU01:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20160919-669094a-unknown
17:41:52:WU01:FS00:0xa7:ERROR:Source code file: /host/debian-testing-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/gmxlib/gmx_detect_hardware.c, line: 212
17:41:52:WU01:FS00:0xa7:ERROR:
17:41:52:WU01:FS00:0xa7:ERROR:Fatal error:
17:41:52:WU01:FS00:0xa7:ERROR:The GROMACS executable was compiled to use the rdtscp CPU instruction. However, this is not supported by the current hardware and continuing would lead to a crash. Please rebuild GROMACS with the GMX_USE_RDTSCP=OFF CMake option.
17:41:52:WU01:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
17:41:52:WU01:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
17:41:52:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
17:41:52:WU01:FS00:0xa7:ERROR:
17:41:52:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
17:41:52:WU01:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20160919-669094a-unknown
17:41:52:WU01:FS00:0xa7:ERROR:Source code file: /host/debian-testing-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/gmxlib/gmx_detect_hardware.c, line: 212
17:41:52:WU01:FS00:0xa7:ERROR:
17:41:52:WU01:FS00:0xa7:ERROR:Fatal error:
17:41:52:WU01:FS00:0xa7:ERROR:The GROMACS executable was compiled to use the rdtscp CPU instruction. However, this is not supported by the current hardware and continuing would lead to a crash. Please rebuild GROMACS with the GMX_USE_RDTSCP=OFF CMake option.
17:41:52:WU01:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
17:41:52:WU01:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
17:41:52:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
17:41:52:WU01:FS00:0xa7:ERROR:
17:41:52:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
17:41:52:WU01:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20160919-669094a-unknown
17:41:52:WU01:FS00:0xa7:ERROR:Source code file: /host/debian-testing-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/gmxlib/gmx_detect_hardware.c, line: 212
17:41:52:WU01:FS00:0xa7:ERROR:
17:41:52:WU01:FS00:0xa7:ERROR:Fatal error:
17:41:52:WU01:FS00:0xa7:ERROR:The GROMACS executable was compiled to use the rdtscp CPU instruction. However, this is not supported by the current hardware and continuing would lead to a crash. Please rebuild GROMACS with the GMX_USE_RDTSCP=OFF CMake option.
17:41:52:WU01:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
17:41:52:WU01:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
17:41:52:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
17:41:52:WU01:FS00:0xa7:ERROR:
17:41:52:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
17:41:52:WU01:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20160919-669094a-unknown
17:41:52:WU01:FS00:0xa7:ERROR:Source code file: /host/debian-testing-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/gmxlib/gmx_detect_hardware.c, line: 212
17:41:52:WU01:FS00:0xa7:ERROR:
17:41:52:WU01:FS00:0xa7:ERROR:Fatal error:
17:41:52:WU01:FS00:0xa7:ERROR:The GROMACS executable was compiled to use the rdtscp CPU instruction. However, this is not supported by the current hardware and continuing would lead to a crash. Please rebuild GROMACS with the GMX_USE_RDTSCP=OFF CMake option.
17:41:52:WU01:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
17:41:52:WU01:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
17:41:52:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
17:42:01:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
17:42:05:FS00:Paused

The client repeats that row over and over again. Options I'm considering:
Find a way to fix the problem with running Core A7
Find a way to not get Core A7 work units
Use the Chrome NACL client instead; that's what the system is running at the moment
Not running FAH at all on this machine

Any suggestions? Thanks in advance!

[bruce]

FAHClient is supposed to be able to test your system to see if the CPU supports AVX or just SSE. The messages indicate that's the problem you're encountering. The FAHCore located at /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/AVX/Core_a7.fah/FahCore_a7 will only run if AVX is actually supported. There's another version of FahCore_a7 that only needs SSE. You can (A) figure out and install whatever drivers for AVX are missing or (B) make the AVX detection code force SSE or (C) temporarily you can probably just get it to run the other version with a little hacking. [Obviously (A) is the best choice, since AVX is faster]

[_r2w_ben]

Have you tried the 7.4.16 beta client? It should be able to detect that AVX is unavailable and instead assign the SSE version of A7.

[JimboPalmer]

The RDTSCP instruction used in A7 is not available in a virtualized environment. (It is a real time instruction time counter, and won't give real times in a VM that only sees some instructions)

It is not an AVX instruction (or SSE) so avoiding AVX will not help

I suspect it is 'left over' profiling code left in A7 for performance testing.

[toTOW]

Debian 8.8 32-bit

But everything in your log refers to a 64 bits OS ... can you clarify what you have ?

[bruce]

You need to post the first page of your log. It looks something like this (except it's going to describe YOUR system):
*********************** Log Started 2017-06-17T07:19:06Z ***********************

Code: Select all

07:19:06:************************* Folding@home Client *************************
07:19:06:        Website: http://folding.stanford.edu/
07:19:06:      Copyright: (c) 2009-2016 Stanford University
07:19:06:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
07:19:06:           Args: 
07:19:06:         Config: C:\Users\Gloria\AppData\Roaming\FAHClient\config.xml
07:19:06:******************************** Build ********************************
07:19:06:        Version: 7.4.16
07:19:06:           Date: Jan 6 2017
07:19:06:           Time: 00:25:14
07:19:06:     Repository: Git
07:19:06:       Revision: a9e9e27dc2ee6ff01398c439677bc27f6cb74032
07:19:06:         Branch: master
07:19:06:       Compiler: Visual C++ 2008
07:19:06:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox -arch:SSE /MT
07:19:06:       Platform: win32 10
07:19:06:           Bits: 32
07:19:06:           Mode: Release
07:19:06:******************************* System ********************************
07:19:06:            CPU: AMD A8-6500 APU with Radeon(tm) HD Graphics
07:19:06:         CPU ID: AuthenticAMD Family 21 Model 19 Stepping 1
07:19:06:           CPUs: 4
07:19:06:         Memory: 15.95GiB
07:19:06:    Free Memory: 14.23GiB
07:19:06:        Threads: WINDOWS_THREADS
07:19:06:     OS Version: 6.2
07:19:06:    Has Battery: false
07:19:06:     On Battery: false
07:19:06:     UTC Offset: -7
07:19:06:            PID: 6260
07:19:06:            CWD: C:\Users\Gloria\AppData\Roaming\FAHClient
07:19:06:             OS: Windows 10 Home
07:19:06:        OS Arch: AMD64
07:19:06:           GPUs: 2
07:19:06:          GPU 0: Bus:2 Slot:0 Func:0 NVIDIA:3 GK107 [GeForce GT 740]
07:19:06:          GPU 1: Bus:1 Slot:0 Func:0 NVIDIA:4 GM107 [GeForce GTX 750 Ti] 1306
07:19:06:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:5.0 Driver:8.0
07:19:06:  CUDA Device 1: Platform:0 Device:1 Bus:2 Slot:0 Compute:3.0 Driver:8.0
07:19:06:OpenCL Device 1: Platform:1 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:382.53
07:19:06:OpenCL Device 2: Platform:1 Device:1 Bus:2 Slot:0 Compute:1.2 Driver:382.53
07:19:06:  Win32 Service: false
07:19:06:***********************************************************************

[bruce]

In my comment above (5 posts above, not 1 post above), I gave three options, I'd like to explore option B, but that'll require us to explore option C first.

Do you have FAH running on a CPU that's old enough to support SSE but not AVX?

Does anybody else reading this have the same problem with a VM?

[jcoffland]

We are testing a beta core that should fix this problem. We need someone with a VM to try it out. Use client-type=beta and project-key=269200501 to get this test WU and core.