Folding Forum

Posted: **Mon Oct 19, 2015 7:09 pm**

I am having issue with the FahCore processes for GPU going into loops on my system during FahCore initialization. My system is the following:

AMD FX-6300, Sapphire HD7850, Mint 17.2, FAH 7.4.4, 15.9 AMD drivers. Every other work unit or so I have the FahCore_17 or _21 process go into a loop during initiatization, it will consume close to 600% CPU and not give the GPU any work to do. To clear it I have to remove the slot which dumps the work unit, and then restart the client and control. The log below appears to be a unit that is working properly, the only difference between this and the ones where FahCore starts looping is you never get any percent completed messages.

I realize troubleshooting from a log of a working unit is probably futile. What data can I provide the next time this occurs that might shed some evidence on what is happening?

Code: Select all

*********************** Log Started 2015-10-19T17:50:43Z ***********************
17:50:43:************************* Folding@home Client *************************
17:50:43:    Website: http://folding.stanford.edu/
17:50:43:  Copyright: (c) 2009-2014 Stanford University
17:50:43:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
17:50:43:       Args: --child --lifeline 3647 /etc/fahclient/config.xml --run-as
17:50:43:             fahclient --pid-file=/var/run/fahclient.pid --daemon
17:50:43:     Config: /etc/fahclient/config.xml
17:50:43:******************************** Build ********************************
17:50:43:    Version: 7.4.4
17:50:43:       Date: Mar 4 2014
17:50:43:       Time: 12:02:38
17:50:43:    SVN Rev: 4130
17:50:43:     Branch: fah/trunk/client
17:50:43:   Compiler: GNU 4.4.7
17:50:43:    Options: -std=gnu++98 -O3 -funroll-loops -mfpmath=sse -ffast-math
17:50:43:             -fno-unsafe-math-optimizations -msse2
17:50:43:   Platform: linux2 3.2.0-1-amd64
17:50:43:       Bits: 64
17:50:43:       Mode: Release
17:50:43:******************************* System ********************************
17:50:43:        CPU: AMD FX(tm)-6300 Six-Core Processor
17:50:43:     CPU ID: AuthenticAMD Family 21 Model 2 Stepping 0
17:50:43:       CPUs: 6
17:50:43:     Memory: 7.70GiB
17:50:43:Free Memory: 5.55GiB
17:50:43:    Threads: POSIX_THREADS
17:50:43: OS Version: 3.13
17:50:43:Has Battery: false
17:50:43: On Battery: false
17:50:43: UTC Offset: -6
17:50:43:        PID: 3649
17:50:43:        CWD: /var/lib/fahclient
17:50:43:         OS: Linux 3.13.0-37-generic x86_64
17:50:43:    OS Arch: AMD64
17:50:43:       GPUs: 1
17:50:43:      GPU 0: ATI:5 Pitcairn PRO [Radeon HD 7800 Series]
17:50:43:       CUDA: Not detected
17:50:43:***********************************************************************
17:50:43:<config>
17:50:43:  <!-- Client Control -->
17:50:43:  <fold-anon v='true'/>
17:50:43:
17:50:43:  <!-- Folding Slot Configuration -->
17:50:43:  <gpu v='false'/>
17:50:43:
17:50:43:  <!-- HTTP Server -->
17:50:43:  <allow v='127.0.0.1 192.168.0.0-192.168.0.255'/>
17:50:43:
17:50:43:  <!-- Network -->
17:50:43:  <proxy v=':8080'/>
17:50:43:
17:50:43:  <!-- Remote Command Server -->
17:50:43:  <password v='********'/>
17:50:43:
17:50:43:  <!-- Slot Control -->
17:50:43:  <power v='full'/>
17:50:43:
17:50:43:  <!-- User Information -->
17:50:43:  <passkey v='********************************'/>
17:50:43:  <team v='37726'/>
17:50:43:  <user v='bfromcolo'/>
17:50:43:
17:50:43:  <!-- Folding Slots -->
17:50:43:  <slot id='1' type='GPU'/>
17:50:43:  <slot id='0' type='CPU'>
17:50:43:    <cpus v='4'/>
17:50:43:  </slot>
17:50:43:</config>
17:50:43:Switching to user fahclient
17:50:43:Trying to access database...
17:51:03:Successfully acquired database lock
17:51:03:Enabled folding slot 01: READY gpu:0:Pitcairn PRO [Radeon HD 7800 Series]
17:51:03:Enabled folding slot 00: READY cpu:4
17:51:03:WU00:FS01:Starting
17:51:03:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/ATI/R600/Core_17.fah/FahCore_17 -dir 00 -suffix 01 -version 704 -lifeline 3649 -checkpoint 15 -gpu 0 -gpu-vendor ati
17:51:03:WU00:FS01:Started FahCore on PID 3666
17:51:03:WU00:FS01:Core PID:3670
17:51:03:WU00:FS01:FahCore 0x17 started
17:51:03:WU01:FS00:Starting
17:51:03:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a4.fah/FahCore_a4 -dir 01 -suffix 01 -version 704 -lifeline 3649 -checkpoint 15 -np 4
17:51:03:WU01:FS00:Started FahCore on PID 3674
17:51:03:WU01:FS00:Core PID:3678
17:51:03:WU01:FS00:FahCore 0xa4 started
17:51:04:WU00:FS01:0x17:*********************** Log Started 2015-10-19T17:51:03Z ***********************
17:51:04:WU00:FS01:0x17:Project: 10466 (Run 1, Clone 8, Gen 148)
17:51:04:WU00:FS01:0x17:Unit: 0x000000f3538b3db95548cebbbaa62d22
17:51:04:WU00:FS01:0x17:CPU: 0x00000000000000000000000000000000
17:51:04:WU00:FS01:0x17:Machine: 1
17:51:04:WU00:FS01:0x17:Digital signatures verified
17:51:04:WU01:FS00:0xa4:
17:51:04:WU01:FS00:0xa4:*------------------------------*
17:51:04:WU01:FS00:0xa4:Folding@Home Gromacs GB Core
17:51:04:WU01:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
17:51:04:WU01:FS00:0xa4:
17:51:04:WU01:FS00:0xa4:Preparing to commence simulation
17:51:04:WU01:FS00:0xa4:- Looking at optimizations...
17:51:04:WU01:FS00:0xa4:- Files status OK
17:51:04:WU01:FS00:0xa4:- Expanded 825111 -> 1401332 (decompressed 169.8 percent)
17:51:04:WU01:FS00:0xa4:Called DecompressByteArray: compressed_data_size=825111 data_size=1401332, decompressed_data_size=1401332 diff=0
17:51:04:WU01:FS00:0xa4:- Digital signature verified
17:51:04:WU01:FS00:0xa4:
17:51:04:WU01:FS00:0xa4:Project: 9023 (Run 729, Clone 5, Gen 26)
17:51:04:WU01:FS00:0xa4:
17:51:04:WU01:FS00:0xa4:Assembly optimizations on if available.
17:51:04:WU01:FS00:0xa4:Entering M.D.
17:51:10:WU01:FS00:0xa4:Using Gromacs checkpoints
17:51:10:WU01:FS00:0xa4:Resuming from checkpoint
17:51:10:WU01:FS00:0xa4:Verified 01/wudata_01.log
17:51:10:WU01:FS00:0xa4:Verified 01/wudata_01.trr
17:51:10:WU01:FS00:0xa4:Verified 01/wudata_01.xtc
17:51:10:WU01:FS00:0xa4:Verified 01/wudata_01.edr
17:51:10:WU01:FS00:0xa4:Completed 20930 out of 250000 steps  (8%)
17:51:31:WU00:FS01:0x17:Completed 0 out of 5000000 steps (0%)
17:53:11:WU01:FS00:0xa4:Completed 22500 out of 250000 steps  (9%)
17:56:52:WU01:FS00:0xa4:Completed 25000 out of 250000 steps  (10%)
18:00:33:WU01:FS00:0xa4:Completed 27500 out of 250000 steps  (11%)
18:04:13:WU01:FS00:0xa4:Completed 30000 out of 250000 steps  (12%)
18:07:52:WU01:FS00:0xa4:Completed 32500 out of 250000 steps  (13%)
18:11:29:WU01:FS00:0xa4:Completed 35000 out of 250000 steps  (14%)
18:15:07:WU01:FS00:0xa4:Completed 37500 out of 250000 steps  (15%)
18:18:46:WU01:FS00:0xa4:Completed 40000 out of 250000 steps  (16%)
18:22:25:WU01:FS00:0xa4:Completed 42500 out of 250000 steps  (17%)
18:26:03:WU01:FS00:0xa4:Completed 45000 out of 250000 steps  (18%)
18:29:41:WU01:FS00:0xa4:Completed 47500 out of 250000 steps  (19%)
18:33:20:WU01:FS00:0xa4:Completed 50000 out of 250000 steps  (20%)
18:37:00:WU01:FS00:0xa4:Completed 52500 out of 250000 steps  (21%)
18:40:38:WU01:FS00:0xa4:Completed 55000 out of 250000 steps  (22%)
18:41:39:Removing old file 'configs/config-20151014-202033.xml'
18:41:39:Saving configuration to /etc/fahclient/config.xml
18:41:39:<config>
18:41:39:  <!-- Client Control -->
18:41:39:  <fold-anon v='true'/>
18:41:39:
18:41:39:  <!-- Folding Slot Configuration -->
18:41:39:  <gpu v='false'/>
18:41:39:
18:41:39:  <!-- HTTP Server -->
18:41:39:  <allow v='127.0.0.1 192.168.0.0-192.168.0.255'/>
18:41:39:
18:41:39:  <!-- Network -->
18:41:39:  <proxy v=':8080'/>
18:41:39:
18:41:39:  <!-- Remote Command Server -->
18:41:39:  <password v='********'/>
18:41:39:
18:41:39:  <!-- Slot Control -->
18:41:39:  <power v='full'/>
18:41:39:
18:41:39:  <!-- User Information -->
18:41:39:  <passkey v='********************************'/>
18:41:39:  <team v='37726'/>
18:41:39:  <user v='bfromcolo'/>
18:41:39:
18:41:39:  <!-- Folding Slots -->
18:41:39:  <slot id='0' type='CPU'>
18:41:39:    <cpus v='4'/>
18:41:39:  </slot>
18:41:39:</config>
18:41:39:FS01:Shutting core down
18:41:39:WU00:FS01:0x17:Caught signal SIGINT(2) on PID 3670
18:41:39:WU00:FS01:0x17:Exiting, please wait. . .
18:41:47:WU00:FS01:0x17:Lost lifeline PID 3666, exiting
18:41:47:WU00:FS01:0x17:ERROR:103: Lost client lifeline
18:41:47:WU00:FS01:0x17:Folding@home Core Shutdown: CLIENT_DIED
18:41:48:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
18:41:48:WARNING:WU00:Slot ID 1 no longer exists and there are no other matching slots, dumping
18:41:48:WU00:Sending unit results: id:00 state:SEND error:DUMPED project:10466 run:1 clone:8 gen:148 core:0x17 unit:0x000000f3538b3db95548cebbbaa62d22
18:41:48:WU00:Connecting to 140.163.4.233:8080
18:41:49:WU00:Server responded WORK_ACK (400)
18:41:49:WU00:Cleaning up
18:41:53:Removing old file 'configs/config-20151014-202051.xml'
18:41:53:Saving configuration to /etc/fahclient/config.xml
18:41:53:<config>
18:41:53:  <!-- Client Control -->
18:41:53:  <fold-anon v='true'/>
18:41:53:
18:41:53:  <!-- Folding Slot Configuration -->
18:41:53:  <gpu v='false'/>
18:41:53:
18:41:53:  <!-- HTTP Server -->
18:41:53:  <allow v='127.0.0.1 192.168.0.0-192.168.0.255'/>
18:41:53:
18:41:53:  <!-- Network -->
18:41:53:  <proxy v=':8080'/>
18:41:53:
18:41:53:  <!-- Remote Command Server -->
18:41:53:  <password v='********'/>
18:41:53:
18:41:53:  <!-- Slot Control -->
18:41:53:  <power v='full'/>
18:41:53:
18:41:53:  <!-- User Information -->
18:41:53:  <passkey v='********************************'/>
18:41:53:  <team v='37726'/>
18:41:53:  <user v='bfromcolo'/>
18:41:53:
18:41:53:  <!-- Folding Slots -->
18:41:53:  <slot id='0' type='CPU'>
18:41:53:    <cpus v='4'/>
18:41:53:  </slot>
18:41:53:</config>
18:42:20:Adding folding slot 01: READY gpu:0:Pitcairn PRO [Radeon HD 7800 Series]
18:42:20:Removing old file 'configs/config-20151015-123853.xml'
18:42:20:Saving configuration to /etc/fahclient/config.xml
18:42:20:<config>
18:42:20:  <!-- Client Control -->
18:42:20:  <fold-anon v='true'/>
18:42:20:
18:42:20:  <!-- Folding Slot Configuration -->
18:42:20:  <gpu v='false'/>
18:42:20:
18:42:20:  <!-- HTTP Server -->
18:42:20:  <allow v='127.0.0.1 192.168.0.0-192.168.0.255'/>
18:42:20:
18:42:20:  <!-- Network -->
18:42:20:  <proxy v=':8080'/>
18:42:20:
18:42:20:  <!-- Remote Command Server -->
18:42:20:  <password v='********'/>
18:42:20:
18:42:20:  <!-- Slot Control -->
18:42:20:  <power v='full'/>
18:42:20:
18:42:20:  <!-- User Information -->
18:42:20:  <passkey v='********************************'/>
18:42:20:  <team v='37726'/>
18:42:20:  <user v='bfromcolo'/>
18:42:20:
18:42:20:  <!-- Folding Slots -->
18:42:20:  <slot id='0' type='CPU'>
18:42:20:    <cpus v='4'/>
18:42:20:  </slot>
18:42:20:  <slot id='1' type='GPU'/>
18:42:20:</config>
18:42:21:WU00:FS01:Connecting to 171.67.108.45:80
18:42:21:WU00:FS01:Assigned to work server 171.64.65.104
18:42:21:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:Pitcairn PRO [Radeon HD 7800 Series] from 171.64.65.104
18:42:21:WU00:FS01:Connecting to 171.64.65.104:8080
18:42:21:WU00:FS01:Downloading 10.00MiB
18:42:27:WU00:FS01:Download 6.25%
18:42:33:WU00:FS01:Download 10.00%
18:42:39:WU00:FS01:Download 15.63%
18:42:45:WU00:FS01:Download 20.63%
18:42:51:WU00:FS01:Download 25.63%
18:42:54:Removing old file 'configs/config-20151015-123906.xml'
18:42:54:Saving configuration to /etc/fahclient/config.xml
18:42:54:<config>
18:42:54:  <!-- Client Control -->
18:42:54:  <fold-anon v='true'/>
18:42:54:
18:42:54:  <!-- Folding Slot Configuration -->
18:42:54:  <gpu v='false'/>
18:42:54:
18:42:54:  <!-- HTTP Server -->
18:42:54:  <allow v='127.0.0.1 192.168.0.0-192.168.0.255'/>
18:42:54:
18:42:54:  <!-- Network -->
18:42:54:  <proxy v=':8080'/>
18:42:54:
18:42:54:  <!-- Remote Command Server -->
18:42:54:  <password v='********'/>
18:42:54:
18:42:54:  <!-- Slot Control -->
18:42:54:  <power v='full'/>
18:42:54:
18:42:54:  <!-- User Information -->
18:42:54:  <passkey v='********************************'/>
18:42:54:  <team v='37726'/>
18:42:54:  <user v='bfromcolo'/>
18:42:54:
18:42:54:  <!-- Folding Slots -->
18:42:54:  <slot id='0' type='CPU'>
18:42:54:    <cpus v='4'/>
18:42:54:  </slot>
18:42:54:  <slot id='1' type='GPU'/>
18:42:54:</config>
18:42:57:WU00:FS01:Download 28.76%
18:43:03:WU00:FS01:Download 33.76%
18:43:10:WU00:FS01:Download 37.51%
18:43:13:WU01:FS00:0xa4:Completed 57500 out of 250000 steps  (23%)
18:43:17:WU00:FS01:Download 41.27%
18:43:23:WU00:FS01:Download 43.77%
18:43:29:WU00:FS01:Download 47.52%
18:43:35:WU00:FS01:Download 50.64%
18:43:41:WU00:FS01:Download 53.77%
18:43:47:WU00:FS01:Download 58.15%
18:43:54:WU00:FS01:Download 61.90%
18:44:00:WU00:FS01:Download 68.15%
18:44:07:WU00:FS01:Download 72.53%
18:44:13:WU00:FS01:Download 76.28%
18:44:19:WU00:FS01:Download 79.40%
18:44:25:WU00:FS01:Download 84.41%
18:44:31:WU00:FS01:Download 87.53%
18:44:37:WU00:FS01:Download 90.66%
18:44:43:WU00:FS01:Download 94.41%
18:44:49:WU00:FS01:Download 97.54%
18:44:53:WU00:FS01:Download complete
18:44:53:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9205 run:12 clone:13 gen:5 core:0x21 unit:0x00000028664f2dd055d4cdf34fa99a08
18:44:53:WU00:FS01:Starting
18:44:53:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/ATI/R600/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 3649 -checkpoint 15 -gpu 0 -gpu-vendor ati
18:44:53:WU00:FS01:Started FahCore on PID 3966
18:44:53:WU00:FS01:Core PID:3970
18:44:53:WU00:FS01:FahCore 0x21 started
18:44:53:WU00:FS01:0x21:*********************** Log Started 2015-10-19T18:44:53Z ***********************
18:44:53:WU00:FS01:0x21:Project: 9205 (Run 12, Clone 13, Gen 5)
18:44:53:WU00:FS01:0x21:Unit: 0x00000028664f2dd055d4cdf34fa99a08
18:44:53:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
18:44:53:WU00:FS01:0x21:Machine: 1
18:44:53:WU00:FS01:0x21:Reading tar file core.xml
18:44:53:WU00:FS01:0x21:Reading tar file system.xml
18:44:54:WU00:FS01:0x21:Reading tar file integrator.xml
18:44:54:WU00:FS01:0x21:Reading tar file state.xml
18:44:56:WU00:FS01:0x21:Digital signatures verified
18:44:56:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
18:44:56:WU00:FS01:0x21:Version 0.0.12
18:45:20:WU01:FS00:0xa4:Completed 60000 out of 250000 steps  (24%)
18:45:51:WU00:FS01:0x21:Completed 0 out of 2500000 steps (0%)
18:45:51:WU00:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
18:49:01:WU01:FS00:0xa4:Completed 62500 out of 250000 steps  (25%)
18:50:18:FS00:Paused
18:50:18:FS01:Paused
18:50:18:FS00:Shutting core down
18:50:18:FS01:Shutting core down
18:50:18:WU00:FS01:0x21:Caught signal SIGINT(2) on PID 3970
18:50:18:WU00:FS01:0x21:Exiting, please wait. . .
18:50:20:WU01:FS00:0xa4:Client no longer detected. Shutting down core.
18:50:20:WU01:FS00:0xa4:
18:50:20:WU01:FS00:0xa4:Folding@home Core Shutdown: CLIENT_DIED
18:50:20:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
18:50:26:WU00:FS01:0x21:Folding@home Core Shutdown: INTERRUPTED
18:50:27:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
18:50:35:FS00:Unpaused
18:50:35:FS01:Unpaused
18:50:35:WU01:FS00:Starting
18:50:35:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a4.fah/FahCore_a4 -dir 01 -suffix 01 -version 704 -lifeline 3649 -checkpoint 15 -np 4
18:50:35:WU01:FS00:Started FahCore on PID 4171
18:50:35:WU01:FS00:Core PID:4175
18:50:35:WU01:FS00:FahCore 0xa4 started
18:50:35:WU00:FS01:Starting
18:50:35:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/ATI/R600/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 3649 -checkpoint 15 -gpu 0 -gpu-vendor ati
18:50:35:WU00:FS01:Started FahCore on PID 4176
18:50:35:WU00:FS01:Core PID:4180
18:50:35:WU00:FS01:FahCore 0x21 started
18:50:36:WU01:FS00:0xa4:
18:50:36:WU01:FS00:0xa4:*------------------------------*
18:50:36:WU01:FS00:0xa4:Folding@Home Gromacs GB Core
18:50:36:WU01:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
18:50:36:WU01:FS00:0xa4:
18:50:36:WU01:FS00:0xa4:Preparing to commence simulation
18:50:36:WU01:FS00:0xa4:- Looking at optimizations...
18:50:36:WU01:FS00:0xa4:- Files status OK
18:50:36:WU01:FS00:0xa4:- Expanded 825111 -> 1401332 (decompressed 169.8 percent)
18:50:36:WU01:FS00:0xa4:Called DecompressByteArray: compressed_data_size=825111 data_size=1401332, decompressed_data_size=1401332 diff=0
18:50:36:WU01:FS00:0xa4:- Digital signature verified
18:50:36:WU01:FS00:0xa4:
18:50:36:WU01:FS00:0xa4:Project: 9023 (Run 729, Clone 5, Gen 26)
18:50:36:WU01:FS00:0xa4:
18:50:36:WU01:FS00:0xa4:Assembly optimizations on if available.
18:50:36:WU01:FS00:0xa4:Entering M.D.
18:50:36:WU00:FS01:0x21:*********************** Log Started 2015-10-19T18:50:35Z ***********************
18:50:36:WU00:FS01:0x21:Project: 9205 (Run 12, Clone 13, Gen 5)
18:50:36:WU00:FS01:0x21:Unit: 0x00000028664f2dd055d4cdf34fa99a08
18:50:36:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
18:50:36:WU00:FS01:0x21:Machine: 1
18:50:36:WU00:FS01:0x21:Digital signatures verified
18:50:36:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
18:50:36:WU00:FS01:0x21:Version 0.0.12
18:50:41:WU01:FS00:0xa4:Using Gromacs checkpoints
18:50:42:WU01:FS00:0xa4:Resuming from checkpoint
18:50:42:WU01:FS00:0xa4:Verified 01/wudata_01.log
18:50:42:WU01:FS00:0xa4:Verified 01/wudata_01.trr
18:50:42:WU01:FS00:0xa4:Verified 01/wudata_01.xtc
18:50:42:WU01:FS00:0xa4:Verified 01/wudata_01.edr
18:50:42:WU01:FS00:0xa4:Completed 51940 out of 250000 steps  (20%)
18:51:11:WU01:FS00:0xa4:Completed 52500 out of 250000 steps  (21%)
18:51:29:WU00:FS01:0x21:Completed 0 out of 2500000 steps (0%)
18:51:29:WU00:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
18:51:56:FS00:Paused
18:51:56:FS00:Shutting core down
18:52:00:WU01:FS00:0xa4:Client no longer detected. Shutting down core.
18:52:00:WU01:FS00:0xa4:
18:52:00:WU01:FS00:0xa4:Folding@home Core Shutdown: CLIENT_DIED
18:52:01:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
18:52:03:Removing old file 'configs/config-20151015-123954.xml'
18:52:03:Saving configuration to /etc/fahclient/config.xml
18:52:03:<config>
18:52:03:  <!-- Client Control -->
18:52:03:  <fold-anon v='true'/>
18:52:03:
18:52:03:  <!-- Folding Slot Configuration -->
18:52:03:  <gpu v='false'/>
18:52:03:
18:52:03:  <!-- HTTP Server -->
18:52:03:  <allow v='127.0.0.1 192.168.0.0-192.168.0.255'/>
18:52:03:
18:52:03:  <!-- Network -->
18:52:03:  <proxy v=':8080'/>
18:52:03:
18:52:03:  <!-- Remote Command Server -->
18:52:03:  <password v='********'/>
18:52:03:
18:52:03:  <!-- Slot Control -->
18:52:03:  <power v='full'/>
18:52:03:
18:52:03:  <!-- User Information -->
18:52:03:  <passkey v='********************************'/>
18:52:03:  <team v='37726'/>
18:52:03:  <user v='bfromcolo'/>
18:52:03:
18:52:03:  <!-- Folding Slots -->
18:52:03:  <slot id='0' type='CPU'>
18:52:03:    <cpus v='4'/>
18:52:03:    <paused v='true'/>
18:52:03:  </slot>
18:52:03:  <slot id='1' type='GPU'/>
18:52:03:</config>
18:54:13:Removing old file 'configs/config-20151019-171103.xml'
18:54:13:Saving configuration to /etc/fahclient/config.xml
18:54:13:<config>
18:54:13:  <!-- Client Control -->
18:54:13:  <fold-anon v='true'/>
18:54:13:
18:54:13:  <!-- Folding Slot Configuration -->
18:54:13:  <gpu v='false'/>
18:54:13:
18:54:13:  <!-- HTTP Server -->
18:54:13:  <allow v='127.0.0.1 192.168.0.0-192.168.0.255'/>
18:54:13:
18:54:13:  <!-- Network -->
18:54:13:  <proxy v=':8080'/>
18:54:13:
18:54:13:  <!-- Remote Command Server -->
18:54:13:  <password v='********'/>
18:54:13:
18:54:13:  <!-- Slot Control -->
18:54:13:  <power v='full'/>
18:54:13:
18:54:13:  <!-- User Information -->
18:54:13:  <passkey v='********************************'/>
18:54:13:  <team v='37726'/>
18:54:13:  <user v='bfromcolo'/>
18:54:13:
18:54:13:  <!-- Folding Slots -->
18:54:13:  <slot id='1' type='GPU'/>
18:54:13:</config>
18:54:13:WARNING:WU01:Slot ID 0 no longer exists and there are no other matching slots, dumping
18:54:13:WU01:Sending unit results: id:01 state:SEND error:DUMPED project:9023 run:729 clone:5 gen:26 core:0xa4 unit:0x00000022ab40417c55f70f2720453cc6
18:54:13:WU01:Connecting to 171.64.65.124:8080
18:54:29:Removing old file 'configs/config-20151019-171117.xml'
18:54:29:Saving configuration to /etc/fahclient/config.xml
18:54:29:<config>
18:54:29:  <!-- Client Control -->
18:54:29:  <fold-anon v='true'/>
18:54:29:
18:54:29:  <!-- Folding Slot Configuration -->
18:54:29:  <gpu v='false'/>
18:54:29:
18:54:29:  <!-- HTTP Server -->
18:54:29:  <allow v='127.0.0.1 192.168.0.0-192.168.0.255'/>
18:54:29:
18:54:29:  <!-- Network -->
18:54:29:  <proxy v=':8080'/>
18:54:29:
18:54:29:  <!-- Remote Command Server -->
18:54:29:  <password v='********'/>
18:54:29:
18:54:29:  <!-- Slot Control -->
18:54:29:  <power v='full'/>
18:54:29:
18:54:29:  <!-- User Information -->
18:54:29:  <passkey v='********************************'/>
18:54:29:  <team v='37726'/>
18:54:29:  <user v='bfromcolo'/>
18:54:29:
18:54:29:  <!-- Folding Slots -->
18:54:29:</config>
18:54:29:FS01:Shutting core down
18:54:30:WU00:FS01:0x21:Caught signal SIGINT(2) on PID 4180
18:54:30:WU00:FS01:0x21:Exiting, please wait. . .
18:54:32:WU01:Server responded WORK_ACK (400)
18:54:32:WU01:Cleaning up
18:55:03:WU00:FS01:0x21:Folding@home Core Shutdown: INTERRUPTED
18:55:05:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
18:55:05:WARNING:WU00:Slot ID 1 no longer exists and there are no other matching slots, dumping
18:55:05:WU00:Sending unit results: id:00 state:SEND error:DUMPED project:9205 run:12 clone:13 gen:5 core:0x21 unit:0x00000028664f2dd055d4cdf34fa99a08
18:55:05:WU00:Connecting to 171.64.65.104:8080
18:55:05:WU00:Server responded WORK_ACK (400)
18:55:05:WU00:Cleaning up
18:55:06:Removing old file 'configs/config-20151019-171204.xml'
18:55:06:Saving configuration to /etc/fahclient/config.xml
18:55:06:<config>
18:55:06:  <!-- Client Control -->
18:55:06:  <fold-anon v='true'/>
18:55:06:
18:55:06:  <!-- Folding Slot Configuration -->
18:55:06:  <gpu v='false'/>
18:55:06:
18:55:06:  <!-- HTTP Server -->
18:55:06:  <allow v='127.0.0.1 192.168.0.0-192.168.0.255'/>
18:55:06:
18:55:06:  <!-- Network -->
18:55:06:  <proxy v=':8080'/>
18:55:06:
18:55:06:  <!-- Remote Command Server -->
18:55:06:  <password v='********'/>
18:55:06:
18:55:06:  <!-- Slot Control -->
18:55:06:  <power v='full'/>
18:55:06:
18:55:06:  <!-- User Information -->
18:55:06:  <passkey v='********************************'/>
18:55:06:  <team v='37726'/>
18:55:06:  <user v='bfromcolo'/>
18:55:06:
18:55:06:  <!-- Folding Slots -->
18:55:06:</config>
18:56:21:Adding folding slot 00: READY gpu:0:Pitcairn PRO [Radeon HD 7800 Series]
18:56:21:Removing old file 'configs/config-20151019-171556.xml'
18:56:21:Saving configuration to /etc/fahclient/config.xml
18:56:21:<config>
18:56:21:  <!-- Client Control -->
18:56:21:  <fold-anon v='true'/>
18:56:21:
18:56:21:  <!-- Folding Slot Configuration -->
18:56:21:  <gpu v='false'/>
18:56:21:
18:56:21:  <!-- HTTP Server -->
18:56:21:  <allow v='127.0.0.1 192.168.0.0-192.168.0.255'/>
18:56:21:
18:56:21:  <!-- Network -->
18:56:21:  <proxy v=':8080'/>
18:56:21:
18:56:21:  <!-- Remote Command Server -->
18:56:21:  <password v='********'/>
18:56:21:
18:56:21:  <!-- Slot Control -->
18:56:21:  <power v='full'/>
18:56:21:
18:56:21:  <!-- User Information -->
18:56:21:  <passkey v='********************************'/>
18:56:21:  <team v='37726'/>
18:56:21:  <user v='bfromcolo'/>
18:56:21:
18:56:21:  <!-- Folding Slots -->
18:56:21:  <slot id='0' type='GPU'/>
18:56:21:</config>
18:56:22:WU00:FS00:Connecting to 171.67.108.45:80
18:56:22:WU00:FS00:Assigned to work server 171.64.65.98
18:56:22:WU00:FS00:Requesting new work unit for slot 00: READY gpu:0:Pitcairn PRO [Radeon HD 7800 Series] from 171.64.65.98
18:56:22:WU00:FS00:Connecting to 171.64.65.98:8080
18:56:22:WU00:FS00:Downloading 7.48MiB
18:56:28:WU00:FS00:Download 15.87%
18:56:34:WU00:FS00:Download 25.06%
18:56:40:WU00:FS00:Download 33.41%
18:56:46:WU00:FS00:Download 40.92%
18:56:52:WU00:FS00:Download 47.61%
18:56:58:WU00:FS00:Download 55.96%
18:57:04:WU00:FS00:Download 64.31%
18:57:08:Removing old file 'configs/config-20151019-171608.xml'
18:57:08:Saving configuration to /etc/fahclient/config.xml
18:57:08:<config>
18:57:08:  <!-- Client Control -->
18:57:08:  <fold-anon v='true'/>
18:57:08:
18:57:08:  <!-- Folding Slot Configuration -->
18:57:08:  <gpu v='false'/>
18:57:08:
18:57:08:  <!-- HTTP Server -->
18:57:08:  <allow v='127.0.0.1 192.168.0.0-192.168.0.255'/>
18:57:08:
18:57:08:  <!-- Network -->
18:57:08:  <proxy v=':8080'/>
18:57:08:
18:57:08:  <!-- Remote Command Server -->
18:57:08:  <password v='********'/>
18:57:08:
18:57:08:  <!-- Slot Control -->
18:57:08:  <power v='full'/>
18:57:08:
18:57:08:  <!-- User Information -->
18:57:08:  <passkey v='********************************'/>
18:57:08:  <team v='37726'/>
18:57:08:  <user v='bfromcolo'/>
18:57:08:
18:57:08:  <!-- Folding Slots -->
18:57:08:  <slot id='0' type='GPU'/>
18:57:08:</config>
18:57:10:WU00:FS00:Download 71.83%
18:57:16:WU00:FS00:Download 77.67%
18:57:22:WU00:FS00:Download 82.68%
18:57:28:WU00:FS00:Download 89.37%
18:57:34:WU00:FS00:Download 96.05%
18:57:37:WU00:FS00:Download complete
18:57:37:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9712 run:10 clone:14 gen:69 core:0x21 unit:0x0000010cab40416255b9a7bfbd12f345
18:57:37:WU00:FS00:Starting
18:57:37:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/ATI/R600/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 3649 -checkpoint 15 -gpu 0 -gpu-vendor ati
18:57:37:WU00:FS00:Started FahCore on PID 4516
18:57:37:WU00:FS00:Core PID:4520
18:57:37:WU00:FS00:FahCore 0x21 started
18:57:37:WU00:FS00:0x21:*********************** Log Started 2015-10-19T18:57:37Z ***********************
18:57:37:WU00:FS00:0x21:Project: 9712 (Run 10, Clone 14, Gen 69)
18:57:37:WU00:FS00:0x21:Unit: 0x0000010cab40416255b9a7bfbd12f345
18:57:37:WU00:FS00:0x21:CPU: 0x00000000000000000000000000000000
18:57:37:WU00:FS00:0x21:Machine: 0
18:57:37:WU00:FS00:0x21:Reading tar file core.xml
18:57:37:WU00:FS00:0x21:Reading tar file integrator.xml
18:57:37:WU00:FS00:0x21:Reading tar file system.xml
18:57:38:WU00:FS00:0x21:Reading tar file state.xml
18:57:39:WU00:FS00:0x21:Digital signatures verified
18:57:39:WU00:FS00:0x21:Folding@home GPU Core21 Folding@home Core
18:57:39:WU00:FS00:0x21:Version 0.0.12
18:58:14:WU00:FS00:0x21:Completed 0 out of 1280000 steps (0%)
18:58:14:WU00:FS00:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
19:03:30:WU00:FS00:0x21:Completed 12800 out of 1280000 steps (1%)

Posted: **Mon Oct 19, 2015 8:26 pm**

What drivers are you running? (You might need to update the drivers from the AND site.)

Posted: **Mon Oct 19, 2015 8:31 pm**

Please show a log where it loops so we can see the last messages from the core.

Your present log shows only that you did a lot of modifying your settings and pausing the WU.

Posted: **Mon Oct 19, 2015 8:33 pm**

I am on the latest AMD driver, 15.9.

Posted: **Mon Oct 19, 2015 11:26 pm**

Core 21 can take up to a few minutes of CPU time to start up before GPU utilization begins. In my log below, Core_21 was spawned at 21:45:01 and began GPU work at 21:49:53.

Code: Select all

21:45:01:WU03:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/[snip]/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/ATI/R600/Core_21.fah/FahCore_21.exe -dir 03 -suffix 01 -version 704 -lifeline 3220 -checkpoint 15 -gpu 0 -gpu-vendor ati
21:45:01:WU03:FS01:Started FahCore on PID 2312
21:45:01:WU03:FS01:Core PID:4944
21:45:01:WU03:FS01:FahCore 0x21 started
21:45:02:WU03:FS01:0x21:*********************** Log Started 2015-10-19T21:45:01Z ***********************
21:45:02:WU03:FS01:0x21:Project: 9704 (Run 17, Clone 19, Gen 119)
21:45:02:WU03:FS01:0x21:Unit: 0x0000009eab404162553ebe72f6de8f23
21:45:02:WU03:FS01:0x21:CPU: 0x00000000000000000000000000000000
21:45:02:WU03:FS01:0x21:Machine: 1
21:45:02:WU03:FS01:0x21:Reading tar file core.xml
21:45:02:WU03:FS01:0x21:Reading tar file system.xml
21:45:07:WU03:FS01:0x21:Reading tar file integrator.xml
21:45:07:WU03:FS01:0x21:Reading tar file state.xml
21:45:11:WU03:FS01:0x21:Digital signatures verified
21:45:11:WU03:FS01:0x21:Folding@home GPU Core21 Folding@home Core
21:45:11:WU03:FS01:0x21:Version 0.0.11
21:49:53:WU03:FS01:0x21:Completed 0 out of 640000 steps (0%)

Posted: **Tue Oct 20, 2015 12:39 am**

Yes I was trying to make a "clean" log of the restart and failure and it started properly, the current work unit seems to be working properly and has about 5 hours to complete. When it fails again I will post the log, but there was nothing to see in previous failures, the process starts consuming max CPU and posts nothing in the log. I gave it plenty of time in previous failures to initialize, over an hour in some cases and the CPU stayed pegged.

Posted: **Tue Oct 20, 2015 2:29 am**

Project: 10466 has deadlines of 4.66 and 6.03 days. 4.66*24 = 111+ hours so you're being impatient if you expect waiting "over an hour" for it to get to 1%.
Project: 9205's deadlines are almost double those.

FAH isn't something that you can watch. To see progress, just let it run at least over night and see what happens (assuming your computer doesn't go to sleep).

Posted: **Sat Oct 24, 2015 7:43 pm**

When this happens the FAHCore process uses all 6 cores of my CPU continuously while not doing anything with the GPU. When it runs correctly it will use 1 core for a few minutes and then go to a very low CPU usage while the work unit crunches on the video card.

Posted: **Tue Nov 17, 2015 4:28 pm**

Well it's happened again. Core 21, its been using 100% of my CPU for the past 15 minutes and has not utilized the GPU at all. No errors, just a looping process, Other than time no other process resource utilization numbers are changing, so it really has to be looping. I tried pausing and unpausing the slot, and also restarting the client, but in both cases after a few minutes the process would resume looping.

I also tried removing and re-adding the slot, it downloaded another Core 21 and eventually started looping again.

I am at a loss, I have rebuilt my Linux install, reinstalled the AMD drivers, reinstalled FAH client and control. This does not happen all the time but often enough.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4479 fahclie+ 39 19 2924544 1.819g 23680 S 584.2 23.6 110:35.29 FahCore_21

Code: Select all

*********************** Log Started 2015-11-17T15:44:45Z ***********************
15:44:45:************************* Folding@home Client *************************
15:44:45:    Website: http://folding.stanford.edu/
15:44:45:  Copyright: (c) 2009-2014 Stanford University
15:44:45:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
15:44:45:       Args: --child --lifeline 4054 /etc/fahclient/config.xml --run-as
15:44:45:             fahclient --pid-file=/var/run/fahclient.pid --daemon
15:44:45:     Config: /etc/fahclient/config.xml
15:44:45:******************************** Build ********************************
15:44:45:    Version: 7.4.4
15:44:45:       Date: Mar 4 2014
15:44:45:       Time: 12:02:38
15:44:45:    SVN Rev: 4130
15:44:45:     Branch: fah/trunk/client
15:44:45:   Compiler: GNU 4.4.7
15:44:45:    Options: -std=gnu++98 -O3 -funroll-loops -mfpmath=sse -ffast-math
15:44:45:             -fno-unsafe-math-optimizations -msse2
15:44:45:   Platform: linux2 3.2.0-1-amd64
15:44:45:       Bits: 64
15:44:45:       Mode: Release
15:44:45:******************************* System ********************************
15:44:45:        CPU: AMD FX(tm)-6300 Six-Core Processor
15:44:45:     CPU ID: AuthenticAMD Family 21 Model 2 Stepping 0
15:44:45:       CPUs: 6
15:44:45:     Memory: 7.70GiB
15:44:45:Free Memory: 6.65GiB
15:44:45:    Threads: POSIX_THREADS
15:44:45: OS Version: 3.13
15:44:45:Has Battery: false
15:44:45: On Battery: false
15:44:45: UTC Offset: -7
15:44:45:        PID: 4056
15:44:45:        CWD: /var/lib/fahclient
15:44:45:         OS: Linux 3.13.0-37-generic x86_64
15:44:45:    OS Arch: AMD64
15:44:45:       GPUs: 1
15:44:45:      GPU 0: ATI:5 Pitcairn PRO [Radeon HD 7800 Series]
15:44:45:       CUDA: Not detected
15:44:45:***********************************************************************
15:44:45:<config>
15:44:45:  <!-- Client Control -->
15:44:45:  <fold-anon v='true'/>
15:44:45:
15:44:45:  <!-- Folding Slot Configuration -->
15:44:45:  <gpu v='false'/>
15:44:45:
15:44:45:  <!-- Network -->
15:44:45:  <proxy v=':8080'/>
15:44:45:
15:44:45:  <!-- Slot Control -->
15:44:45:  <power v='full'/>
15:44:45:
15:44:45:  <!-- User Information -->
15:44:45:  <passkey v='********************************'/>
15:44:45:  <team v='37726'/>
15:44:45:  <user v='bfromcolo'/>
15:44:45:
15:44:45:  <!-- Folding Slots -->
15:44:45:  <slot id='1' type='GPU'>
15:44:45:    <paused v='true'/>
15:44:45:  </slot>
15:44:45:</config>
15:44:45:Switching to user fahclient
15:44:45:Trying to access database...
15:44:45:Successfully acquired database lock
15:44:45:Enabled folding slot 01: PAUSED gpu:0:Pitcairn PRO [Radeon HD 7800 Series] (by user)
15:45:14:FS01:Unpaused
15:45:14:WU00:FS01:Connecting to 171.67.108.45:80
15:45:15:WU00:FS01:Assigned to work server 171.64.65.98
15:45:15:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:Pitcairn PRO [Radeon HD 7800 Series] from 171.64.65.98
15:45:15:WU00:FS01:Connecting to 171.64.65.98:8080
15:45:15:WU00:FS01:Downloading 15.70MiB
15:45:21:WU00:FS01:Download 11.94%
15:45:27:WU00:FS01:Download 18.71%
15:45:33:WU00:FS01:Download 29.46%
15:45:39:WU00:FS01:Download 37.02%
15:45:45:WU00:FS01:Download 43.39%
15:45:46:Saving configuration to /etc/fahclient/config.xml
15:45:46:<config>
15:45:46:  <!-- Client Control -->
15:45:46:  <fold-anon v='true'/>
15:45:46:
15:45:46:  <!-- Folding Slot Configuration -->
15:45:46:  <gpu v='false'/>
15:45:46:
15:45:46:  <!-- Network -->
15:45:46:  <proxy v=':8080'/>
15:45:46:
15:45:46:  <!-- Slot Control -->
15:45:46:  <power v='full'/>
15:45:46:
15:45:46:  <!-- User Information -->
15:45:46:  <passkey v='********************************'/>
15:45:46:  <team v='37726'/>
15:45:46:  <user v='bfromcolo'/>
15:45:46:
15:45:46:  <!-- Folding Slots -->
15:45:46:  <slot id='1' type='GPU'/>
15:45:46:</config>
15:45:51:WU00:FS01:Download 48.17%
15:45:57:WU00:FS01:Download 53.74%
15:46:03:WU00:FS01:Download 58.92%
15:46:09:WU00:FS01:Download 61.31%
15:46:15:WU00:FS01:Download 66.08%
15:46:21:WU00:FS01:Download 73.65%
15:46:27:WU00:FS01:Download 79.22%
15:46:33:WU00:FS01:Download 84.79%
15:46:39:WU00:FS01:Download 89.57%
15:46:45:WU00:FS01:Download 94.75%
15:46:49:WU00:FS01:Download complete
15:46:49:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9704 run:17 clone:3 gen:112 core:0x21 unit:0x000000a7ab404162553ebe483982313d
15:46:49:WU00:FS01:Starting
15:46:49:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/ATI/R600/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 4056 -checkpoint 15 -gpu 0 -gpu-vendor ati
15:46:49:WU00:FS01:Started FahCore on PID 4475
15:46:49:WU00:FS01:Core PID:4479
15:46:49:WU00:FS01:FahCore 0x21 started
15:46:50:WU00:FS01:0x21:*********************** Log Started 2015-11-17T15:46:49Z ***********************
15:46:50:WU00:FS01:0x21:Project: 9704 (Run 17, Clone 3, Gen 112)
15:46:50:WU00:FS01:0x21:Unit: 0x000000a7ab404162553ebe483982313d
15:46:50:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
15:46:50:WU00:FS01:0x21:Machine: 1
15:46:50:WU00:FS01:0x21:Reading tar file core.xml
15:46:50:WU00:FS01:0x21:Reading tar file system.xml
15:46:52:WU00:FS01:0x21:Reading tar file integrator.xml
15:46:52:WU00:FS01:0x21:Reading tar file state.xml
15:46:53:WU00:FS01:0x21:Digital signatures verified
15:46:53:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
15:46:53:WU00:FS01:0x21:Version 0.0.12
15:48:32:WU00:FS01:0x21:Completed 0 out of 640000 steps (0%)
15:48:32:WU00:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900

Posted: **Tue Nov 17, 2015 7:13 pm**

Send me the Wu and I'll try it on my system. YGPM.

You're saying that FahCore_21 is using all 6 CPUs, or some other FAH component is?

Posted: **Tue Nov 17, 2015 7:54 pm**

Yes FahCore_21 is using all 6 cores. Does FAH keep WUs after the slot is removed, or do they go to the recycle bin, or do they just go away? If I don't still have it I will recreate it.

Thanks

Posted: **Tue Nov 17, 2015 8:41 pm**

You got mail!

Posted: **Tue Nov 17, 2015 10:29 pm**

If you have more than one slot for the same hardware, deleting one of them will transfer the WU (enqueuing it on a different GPU with the same characteristics. Otherwise it will just go away.

Posted: **Thu Nov 19, 2015 6:36 pm**

Were you able to recreate the issue I am having with the work unit I sent?

Folding Forum

FahCore_17 and _21 looping

FahCore_17 and _21 looping

Re: FahCore_17 and _21 looping

Re: FahCore_17 and _21 looping

Re: FahCore_17 and _21 looping

Re: FahCore_17 and _21 looping

Re: FahCore_17 and _21 looping

Re: FahCore_17 and _21 looping

Re: FahCore_17 and _21 looping

Re: FahCore_17 and _21 looping

Re: FahCore_17 and _21 looping

Re: FahCore_17 and _21 looping

Re: FahCore_17 and _21 looping

Re: FahCore_17 and _21 looping

Re: FahCore_17 and _21 looping