Page 1 of 2
Core 21 causing severe GUI lag on Linux
Posted: Mon Oct 19, 2015 1:01 pm
by weirddan455
For some reason, Core 21 is causing my Xorg process to spike to 15-25% CPU usage (25% being 1 full core.) I even paused my CPU slot to test and even with only the GPU slot running it was still causing severe GUI lag. I have not noticed this with Core 17 or 18. The lag is so bad with Core 21 that I have to pause it just to be able to use my desktop. I can really only fold while I'm not at my computer whenever I get a Core 21 WU. Has anyone else noticed this?
Re: Core 21 causing severe GUI lag on Linux
Posted: Mon Oct 19, 2015 5:10 pm
by bruce
Those who notice GPU lag will depend on BOTH the particular WU running and the speed of your GPU. The recommended fix is to reconfigure your GPU slot to run "on idle" which will effectively start/stop GPU folding based on whether your OS believes a screensaver should be running or not.
One of the advantages of Core_21 is that it's capable of running larger proteins than Core_17/_18 and if you look at psummary, you can see how many atoms are in each protein.
Re: Core 21 causing severe GUI lag on Linux
Posted: Tue Oct 20, 2015 2:34 pm
by JohnChodera
Thanks for the report, @weirddan455! We've heard some of these reports before.
Does this happen with every core 21 WU, or just some of them (and if so, which project numbers)?
Could it be that FAHClient or the core is consuming all available memory, or is it really a pure GPU lag phenomena?
Re: Core 21 causing severe GUI lag on Linux
Posted: Thu Oct 22, 2015 6:26 pm
by weirddan455
It's actually worse than I initially thought. I had to stop GPU folding all together because I can't complete a Core 21 WU. It will go normally for the first couple of % (typically less than 5 minutes per %) then it will take an hour for the next % to come in the logs (and that's actually when the GUI lag will start.)
I even tried booting my computer to console so Xorg isn't even loaded and I still have the same issue.
EDIT: BTW, RAM isn't an issue. I have 16GB of RAM and I don't recall ever seeing the FAH core/client go above 1GB.
EDIT 2: My specs are:
Intel Core i5 2500K
Nvidia Geforce GTX 770 2GB VRAM
Nvidia 355.11 drivers
16GB RAM
Arch Linux kernel 4.2.3
Here's a log from the last Core 21 WU I tried:
Code: Select all
05:47:21:WU02:FS01:Starting
05:47:21:WU02:FS01:Running FahCore: /opt/fah/FAHCoreWrapper /var/opt/fah/cores/web.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 02 -suffix 01 -version 704 -lifeline 486 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
05:47:21:WU02:FS01:Started FahCore on PID 1073
05:47:21:WU02:FS01:Core PID:1077
05:47:21:WU02:FS01:FahCore 0x21 started
05:47:21:WU02:FS01:0x21:*********************** Log Started 2015-10-20T05:47:21Z ***********************
05:47:21:WU02:FS01:0x21:Project: 9206 (Run 0, Clone 1418, Gen 3)
05:47:21:WU02:FS01:0x21:Unit: 0x00000016664f2dd056202b17ebfb4c65
05:47:21:WU02:FS01:0x21:CPU: 0x00000000000000000000000000000000
05:47:21:WU02:FS01:0x21:Machine: 1
05:47:21:WU02:FS01:0x21:Reading tar file core.xml
05:47:21:WU02:FS01:0x21:Reading tar file system.xml
05:47:22:WU02:FS01:0x21:Reading tar file integrator.xml
05:47:22:WU02:FS01:0x21:Reading tar file state.xml
05:47:23:WU02:FS01:0x21:Digital signatures verified
05:47:23:WU02:FS01:0x21:Folding@home GPU Core21 Folding@home Core
05:47:23:WU02:FS01:0x21:Version 0.0.12
05:48:03:WU02:FS01:0x21:Completed 0 out of 2500000 steps (0%)
05:48:03:WU02:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
05:59:01:WU02:FS01:0x21:Completed 25000 out of 2500000 steps (1%)
06:09:48:WU02:FS01:0x21:Completed 50000 out of 2500000 steps (2%)
06:20:35:WU02:FS01:0x21:Completed 75000 out of 2500000 steps (3%)
06:31:20:WU02:FS01:0x21:Completed 100000 out of 2500000 steps (4%)
06:42:17:WU02:FS01:0x21:Completed 125000 out of 2500000 steps (5%)
09:42:36:WU02:FS01:0x21:Completed 150000 out of 2500000 steps (6%)
14:05:46:Caught signal SIGTERM(15) on PID 486
14:05:46:Exiting, please wait. . .
14:05:46:WU02:FS01:0x21:Caught signal SIGTERM(15) on PID 1077
14:05:46:WU02:FS01:0x21:Exiting, please wait. . .
14:05:47:FS00:Shutting core down
14:05:47:FS01:Shutting core down
14:06:05:WU02:FS01:0x21:Folding@home Core Shutdown: INTERRUPTED
14:06:06:Clean exit
As you can see, the first 5% goes normally then it takes 3 hours to get to 6% then another 4h 20m with no progress (but CPU usage still maxed out as if it's working.) At first I thought bad WU so I deleted the work directory but this is I think the 3rd Core 21 WU I've tried and they've all failed like this so I stopped GPU folding all together (which is a shame because Core 17 and 18 WUs work no problem but I can't selectively turn off Core 21's.)
Re: Core 21 causing severe GUI lag on Linux
Posted: Thu Oct 22, 2015 9:48 pm
by bruce
This is a known problem. Development is still working on figuring out a fix can can be delivered in a future version of FahCore_21.
Your post did not include the system description panel (at the top of the log or on the System tab of FAHControl).
Reverting to the default setting of "client-type" should minimize your exposure to this issue.
Re: Core 21 causing severe GUI lag on Linux
Posted: Thu Oct 22, 2015 11:35 pm
by Grandpa_01
Change your drivers to 346.96 that will alleviate a lot of the slowdown issues and you will get better performance from them. If you are OCed you may need to drop the OC a little or raise the v a little see how it runs first then adjust as needed.
Re: Core 21 causing severe GUI lag on Linux
Posted: Fri Oct 23, 2015 1:51 am
by weirddan455
Here's the top of the log. Is this what you were looking for?
Code: Select all
*********************** Log Started 2015-10-19T23:55:11Z ***********************
23:55:11:************************* Folding@home Client *************************
23:55:11: Website: http://folding.stanford.edu/
23:55:11: Copyright: (c) 2009-2014 Stanford University
23:55:11: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
23:55:11: Args: --config /var/opt/fah/config.xml --exec-directory=/opt/fah
23:55:11: --data-directory=/var/opt/fah
23:55:11: Config: /var/opt/fah/config.xml
23:55:11:******************************** Build ********************************
23:55:11: Version: 7.4.4
23:55:11: Date: Mar 4 2014
23:55:11: Time: 12:02:38
23:55:11: SVN Rev: 4130
23:55:11: Branch: fah/trunk/client
23:55:11: Compiler: GNU 4.4.7
23:55:11: Options: -std=gnu++98 -O3 -funroll-loops -mfpmath=sse -ffast-math
23:55:11: -fno-unsafe-math-optimizations -msse2
23:55:11: Platform: linux2 3.2.0-1-amd64
23:55:11: Bits: 64
23:55:11: Mode: Release
23:55:11:******************************* System ********************************
23:55:11: CPU: Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz
23:55:11: CPU ID: GenuineIntel Family 6 Model 42 Stepping 7
23:55:11: CPUs: 4
23:55:11: Memory: 15.63GiB
23:55:11:Free Memory: 15.42GiB
23:55:11: Threads: POSIX_THREADS
23:55:11: OS Version: 4.2
23:55:11:Has Battery: false
23:55:11: On Battery: false
23:55:11: UTC Offset: -5
23:55:11: PID: 486
23:55:11: CWD: /var/opt/fah
23:55:11: OS: Linux 4.2.3-1-ARCH x86_64
23:55:11: OS Arch: AMD64
23:55:11: GPUs: 1
23:55:11: GPU 0: NVIDIA:3 GK104 [GeForce GTX 770]
23:55:11: CUDA: 3.0
23:55:11:CUDA Driver: 7050
23:55:11:***********************************************************************
23:55:11:<config>
23:55:11: <!-- Slot Control -->
23:55:11: <power v='full'/>
23:55:11:
23:55:11: <!-- User Information -->
23:55:11: <passkey v='********************************'/>
23:55:11: <team v='45032'/>
23:55:11: <user v='weirddan455'/>
23:55:11:
23:55:11: <!-- Folding Slots -->
23:55:11: <slot id='0' type='CPU'/>
23:55:11: <slot id='1' type='GPU'/>
23:55:11:</config>
23:55:11:Trying to access database...
23:55:11:Successfully acquired database lock
23:55:11:Enabled folding slot 00: READY cpu:3
23:55:11:Enabled folding slot 01: READY gpu:0:GK104 [GeForce GTX 770]
Reverting to the default setting of "client-type" should minimize your exposure to this issue.
How do I do this? I believe all my settings are default. The only thing I changed was I moved the slider to "full" so the CPU slot will use 3 cores instead of 2 (leaving 1 core for the GPU slot.)
Re: Core 21 causing severe GUI lag on Linux
Posted: Fri Oct 23, 2015 1:01 pm
by weirddan455
I turned my GPU slot back on to see if I could gather some more info and I found the problem. It's eating up all my VRAM. This WU actually made it all the way to 74% before doing it. I let it fold while I was sleeping and when I woke up all was fine. I checked on my VRAM usage and it was ~220MB. I stepped away from my computer for about 5-10 minutes after that, came back, and my VRAM use went form 220MB all the way up to 2GB. Maybe it was coincidence VRAM use started spiking right after I interacted with my computer, maybe not.
EDIT: I restarted the client and the WU ended early at 87%. I made a new thread to ask if it was a bad WU here: viewtopic.php?f=19&t=28220
Re: Core 21 causing severe GUI lag on Linux
Posted: Fri Oct 23, 2015 7:25 pm
by Calcii
I confirm a bug x21 core, and that's statistics:
1) 21x core versions 0.0.11 and 0.0.12
2) Observed on Windows x64, Windows 10 x64 and Linux x64
3) Affected cards: ASUS AMD 290х, ASUS Nvidia 980, MSI and EVGA Nvidia 970
4) Looks like a freeze sometime after the beginning of the fold, It stops everything so that you need patience and skill to click mouse somewhere
5) At statement of the client on a pause and removal of its brake vanish.
6) After you restore the client generates an error and rolls back calculations, sometimes for several hours. Sometimes swears an error and download a new job
The conclusion from what has been said: x21 core need finalization, and i was surprised then core 21 go to advanced so fast
Re: Core 21 causing severe GUI lag on Linux
Posted: Fri Oct 23, 2015 8:48 pm
by codysluder
Calcii wrote:The conclusion from what has been said: x21 core need finalization, and i was surprised then core 21 go to advanced so fast
Removing the Advanced client-type setting should allow you to opt-out if corex21 until it gets fixed.
Re: Core 21 causing severe GUI lag on Linux
Posted: Fri Oct 23, 2015 9:54 pm
by weirddan455
codysluder wrote:Calcii wrote:The conclusion from what has been said: x21 core need finalization, and i was surprised then core 21 go to advanced so fast
Removing the Advanced client-type setting should allow you to opt-out if corex21 until it gets fixed.
I don't have Advanced set as far as I know and I'm still getting Core 21 WUs.
Re: Core 21 causing severe GUI lag on Linux
Posted: Sat Oct 24, 2015 7:00 am
by Grandpa_01
Several of the core 21 projects are running in the general population so there is no flag that will prevent a rig from getting them.
Re: Core 21 causing severe GUI lag on Linux
Posted: Sat Oct 24, 2015 2:49 pm
by 7im
Lag does depend on the WU size. Not all core 21 project are laggy. Please be more specific which projects you have noted this behavior. If it lags on all projects on your hardware, please describe your hw in detail, including os and gpu driver version.
Re: Core 21 causing severe GUI lag on Linux
Posted: Sat Oct 24, 2015 4:32 pm
by weirddan455
7im wrote:Lag does depend on the WU size. Not all core 21 project are laggy. Please be more specific which projects you have noted this behavior. If it lags on all projects on your hardware, please describe your hw in detail, including os and gpu driver version.
This has happened on every Core 21 WU I've had. I posted my specs on the 4th post. The problem is a VRAM leak. Usually it will eat up all the VRAM by ~10% but the last WU I had made it to 74% before the leak started. The lag only starts once all the VRAM has been used and it also drastically slows down the folding speed (going from < 10 minutes per % to multiple hours.) Restarting the client will free up the VRAM and speed things back up but it will often eat up the VRAM again some time later.
Re: Core 21 causing severe GUI lag on Linux
Posted: Mon Oct 26, 2015 8:59 pm
by JohnChodera
@weirddan: Thanks for pointing this out! Can you confirm this was with core 21 version 0.0.12?