Gather data: GPU Projects with 20000+ atoms,

If you think it might be a driver problem, see viewforum.php?f=79

Moderators: Site Moderators, FAHC Science Team

bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Gather data: GPU Projects with 20000+ atoms,

Post by bruce »

People have observed reduced PPD and reduced GPU utilization on the new projects with high atom-counts.
viewtopic.php?f=74&t=21180
viewtopic.php?f=74&t=21208

I've alluded to suspicions that the PCIe bus may be the limiting factor for these projects rather than the compute speed of the GPU. See Subject: TPF is WAY long -- P7641 for one such example.

Please report your project number, GPU model, and the characteristics of the bus interconnecting it with Main RAM, and any observed changes in PPD or GPU utilization from what you've seen on smaller projects.
*hondo*
Posts: 102
Joined: Sat Mar 08, 2008 9:50 am
Hardware configuration: Geoforce 250 GTS Active for F@H

4 Core Intel not used at all for F@H
Location: England UK

Re: Gather data: GPU Projects with 20000+ atoms,

Post by *hondo* »

I can tell you I'm Folding two WUs with an i5 & a 580GTX
Sorry I know nothing about the following
and the characteristics of the bus interconnecting it with Main RAM,
Edit & updated information
System Specification
Case: Fractal Design Define R3 Black
Power Supply: OCZ ZS 750w PSU
CPU: Intel Core i5 2500K 3.30GHz Sandybridge overclocked to 4.40GHz
Motherboard: Intel Z68 (Socket 1155) DDR3 Motherboard ** B3 REVISION ** = Motherboard Gigabyte GA-Z68AP-D3
Cooler: Xigmatek Dark Knight
RAM: 4GB DDR3 1600MHz Dual Channel Kit
Hard Drive: Western Digital Caviar Blue 500GB SATA 6GB/s
Graphics Card: Gainward 586 PCI-e
Sound: Realtek 7.1 Channel Sound (On-Board) NOT USED Audio thru Creative Labs PCI
Optical Drive: LG DVD+/-RW SATA Drive

Other than both WUs came off the same PC / GPU
Image

Edit More info that doesn't mean too much to me
Image


Image

Well this is what I go off, with a difference of plus 5k PPD and more to the point these Wus ran consecutively from the same PC

In the case of P 7200 fahmon doesn’t in both cases take into account any score from them, because there is no information available in the WU summary
Last edited by *hondo* on Sun Apr 08, 2012 3:22 pm, edited 4 times in total.
Ripper36
Posts: 60
Joined: Sun Sep 18, 2011 8:55 am

Re: Gather data: GPU Projects with 20000+ atoms,

Post by Ripper36 »

I can post data on forty project 803x (1832 atoms) results, and on thirteen 764x (21433 atoms) projects. There is a dramatic difference.

Image

Image

I am folding on four NVIDIA GTX570OC cards

- two on JOHN-PC

Image
Image

- one on TOLKEIN

Image

- one on MEDIACENTRE

Image

The thing that stands out, supporting your hypothesis, is that TOLKEIN which a Z68 chipset on x16 PCI-E slot is so much faster than JOHN-PC-1 and JOHN-PC-2 running x8, when they were both equivalent on 8031 units. Mediacentre is slower again, on a PCI-E 1.1 interface.

There is also a difference in drivers though I'm not sure that's rerlevant to the results.

Interested in your comments
Last edited by Ripper36 on Mon Apr 09, 2012 3:50 am, edited 2 times in total.
Image
Nathan_P
Posts: 1164
Joined: Wed Apr 01, 2009 9:22 pm
Hardware configuration: Asus Z8NA D6C, 2 x5670@3.2 Ghz, , 12gb Ram, GTX 980ti, AX650 PSU, win 10 (daily use)

Asus Z87 WS, Xeon E3-1230L v3, 8gb ram, KFA GTX 1080, EVGA 750ti , AX760 PSU, Mint 18.2 OS

Not currently folding
Asus Z9PE- D8 WS, 2 E5-2665@2.3 Ghz, 16Gb 1.35v Ram, Ubuntu (Fold only)
Asus Z9PA, 2 Ivy 12 core, 16gb Ram, H folding appliance (fold only)
Location: Jersey, Channel islands

Re: Gather data: GPU Projects with 20000+ atoms,

Post by Nathan_P »

I've taken this from a thread over at [H] where a teammate has the same problem

But jeez stanford..... PPD is a mear 25% of what my cards could get on other units. 25%! seriously?! I know you think GPU is over-valued but no warning? just cut PPD to 1/4?

anyhow, here are some details
core 15
Project: 7643 (Run 426, Clone 0, Gen 1)
Project: 7642 (Run 455, Clone 0, Gen 0)
Project: 7641 (Run 151, Clone 0, Gen 2)

each worth 7718 points

570 getting TPF of: ~21 minutes, about 5250 PPD
460 getting TPF of: ~53 minutes, about 2075 PPD

The 570's are in a: EVGA x58 Classified SLI, Intel i7 970 @ 4.15 GHz, Corsair H70, 12GB DDR3-1600 CAS 8, EVGA GTX570 SLI @ 825/1650/1950 and the 460 is in a: ASUS ROG GENE-Z, Intel i7 2600K @ 4.0, Antek Khuler 920, 4GB DDR3-2133 CAS 9, Galaxy GTX460.

He's confirmed that all the cards are running at x16 speed.

Not sure if it helps any but i'm interested in further thoughts on this
Image
GreyWhiskers
Posts: 660
Joined: Mon Oct 25, 2010 5:57 am
Hardware configuration: a) Main unit
Sandybridge in HAF922 w/200 mm side fan
--i7 2600K@4.2 GHz
--ASUS P8P67 DeluxeB3
--4GB ADATA 1600 RAM
--750W Corsair PS
--2Seagate Hyb 750&500 GB--WD Caviar Black 1TB
--EVGA 660GTX-Ti FTW - Signature 2 GPU@ 1241 Boost
--MSI GTX560Ti @900MHz
--Win7Home64; FAH V7.3.2; 327.23 drivers

b) 2004 HP a475c desktop, 1 core Pent 4 HT@3.2 GHz; Mem 2GB;HDD 160 GB;Zotac GT430PCI@900 MHz
WinXP SP3-32 FAH v7.3.6 301.42 drivers - GPU slot only

c) 2005 Toshiba M45-S551 laptop w/2 GB mem, 160GB HDD;Pent M 740 CPU @ 1.73 GHz
WinXP SP3-32 FAH v7.3.6 [Receiving Core A4 work units]
d) 2011 lappy-15.6"-1920x1080;i7-2860QM,2.5;IC Diamond Thermal Compound;GTX 560M 1,536MB u/c@700;16GB-1333MHz RAM;HDD:500GBHyb w/ 4GB SSD;Win7HomePrem64;320.18 drivers FAH 7.4.2ß
Location: Saratoga, California USA

Re: Gather data: GPU Projects with 20000+ atoms,

Post by GreyWhiskers »

Image

From my other post, here's the pivot table that goes along with it. The last section has the large, >20K, projects.

The GPU-Z pic above shows my current memory bandwidth at the O/C I'm running (947 MHz)

Image

[edit]corrected clock rate
Last edited by GreyWhiskers on Mon Apr 09, 2012 2:22 am, edited 1 time in total.
Ripper36
Posts: 60
Joined: Sun Sep 18, 2011 8:55 am

Re: Gather data: GPU Projects with 20000+ atoms,

Post by Ripper36 »

Wow - I must get that app GPU-Z and update my post.
Image
GreyWhiskers
Posts: 660
Joined: Mon Oct 25, 2010 5:57 am
Hardware configuration: a) Main unit
Sandybridge in HAF922 w/200 mm side fan
--i7 2600K@4.2 GHz
--ASUS P8P67 DeluxeB3
--4GB ADATA 1600 RAM
--750W Corsair PS
--2Seagate Hyb 750&500 GB--WD Caviar Black 1TB
--EVGA 660GTX-Ti FTW - Signature 2 GPU@ 1241 Boost
--MSI GTX560Ti @900MHz
--Win7Home64; FAH V7.3.2; 327.23 drivers

b) 2004 HP a475c desktop, 1 core Pent 4 HT@3.2 GHz; Mem 2GB;HDD 160 GB;Zotac GT430PCI@900 MHz
WinXP SP3-32 FAH v7.3.6 301.42 drivers - GPU slot only

c) 2005 Toshiba M45-S551 laptop w/2 GB mem, 160GB HDD;Pent M 740 CPU @ 1.73 GHz
WinXP SP3-32 FAH v7.3.6 [Receiving Core A4 work units]
d) 2011 lappy-15.6"-1920x1080;i7-2860QM,2.5;IC Diamond Thermal Compound;GTX 560M 1,536MB u/c@700;16GB-1333MHz RAM;HDD:500GBHyb w/ 4GB SSD;Win7HomePrem64;320.18 drivers FAH 7.4.2ß
Location: Saratoga, California USA

Re: Gather data: GPU Projects with 20000+ atoms,

Post by GreyWhiskers »

Once you get it, the little camera icon just under the red X close button will take a screenshot of the GPU-z screen and upload it to GPU-z's servers and giving you a link to the image. One stop shopping.
Ripper36
Posts: 60
Joined: Sun Sep 18, 2011 8:55 am

Re: Gather data: GPU Projects with 20000+ atoms,

Post by Ripper36 »

Thanks - my post has now been edited and upgraded!
Image
Jesse_V
Site Moderator
Posts: 2850
Joined: Mon Jul 18, 2011 4:44 am
Hardware configuration: OS: Windows 10, Kubuntu 19.04
CPU: i7-6700k
GPU: GTX 970, GTX 1080 TI
RAM: 24 GB DDR4
Location: Western Washington

Re: Gather data: GPU Projects with 20000+ atoms,

Post by Jesse_V »

My 560TI is working away on project 7641, which has 21,443 atoms and according to GPU-Z is at 89-90% GPU load. V7 tells me that its pulling in 11k PPD, which is below the normal of 14k PPD. As I'm working on this post it finished it off and picked up a 562-atom 7612, and its working away at 99% usage with 14k PPD again, just as I'd expect. So there is a difference. I have a Gigabyte P67X-UD3-B3 (Socket 1155) motherboard, and my PCI has a bus width of 32-bits according to Piriform Speccy, and I only have a single GPU. I have 8 GB of RAM and they're pretty fast.
GPU-Z gives more information about the GPU's Bus Interface: "PCI-E 2.0 x16 @ x8 2.0" which of course matches GreyWhiskers's above.
Let me know if there's any further system specs you need.
F@h is now the top computing platform on the planet and nothing unites people like a dedicated fight against a common enemy. This virus affects all of us. Lets end it together.
GreyWhiskers
Posts: 660
Joined: Mon Oct 25, 2010 5:57 am
Hardware configuration: a) Main unit
Sandybridge in HAF922 w/200 mm side fan
--i7 2600K@4.2 GHz
--ASUS P8P67 DeluxeB3
--4GB ADATA 1600 RAM
--750W Corsair PS
--2Seagate Hyb 750&500 GB--WD Caviar Black 1TB
--EVGA 660GTX-Ti FTW - Signature 2 GPU@ 1241 Boost
--MSI GTX560Ti @900MHz
--Win7Home64; FAH V7.3.2; 327.23 drivers

b) 2004 HP a475c desktop, 1 core Pent 4 HT@3.2 GHz; Mem 2GB;HDD 160 GB;Zotac GT430PCI@900 MHz
WinXP SP3-32 FAH v7.3.6 301.42 drivers - GPU slot only

c) 2005 Toshiba M45-S551 laptop w/2 GB mem, 160GB HDD;Pent M 740 CPU @ 1.73 GHz
WinXP SP3-32 FAH v7.3.6 [Receiving Core A4 work units]
d) 2011 lappy-15.6"-1920x1080;i7-2860QM,2.5;IC Diamond Thermal Compound;GTX 560M 1,536MB u/c@700;16GB-1333MHz RAM;HDD:500GBHyb w/ 4GB SSD;Win7HomePrem64;320.18 drivers FAH 7.4.2ß
Location: Saratoga, California USA

Re: Gather data: GPU Projects with 20000+ atoms,

Post by GreyWhiskers »

Jesse_V wrote: "PCI-E 2.0 x16 @ x8 2.0" which of course matches GreyWhiskers's above.
Let me know if there's any further system specs you need.
Check your GPU-Z again -- you said, "PCI-E 2.0 x16 @ x8 2.0", while my screenshot above shows "PCI-E 2.0 x16 @ x16 2.0"

Tom's Hardware forum has a similar situation from a couple of years ago -- PCIe Bandwidth issues.. Showing x8 width on a single x16 slot. It may or may not apply...Which PCI-E slot the board is installed in, etc.
Switching from x8 to x16 on that GTX 260 probably won't make any noticeable difference. However, you might look into your BIOS settings and make sure there is not a setting that's forcing your PCI-E slot to run x8. You could also try updating to the latest driver versions for your GPU and motherboard chipset to be sure there's no updates needed there. (could also check for BIOS updates)

Do you have another graphics card installed on a second GPU slot? (For instance, a second card for PhysX?)

Is this your motherboard?

GA-MA490X-UD4
http://www.gigabyte.us/Products/Mo [...] uctID=3003

This motherboard has two PCI-E GPU lanes. One runs at 16x, and the other runs at x8. So if you have your GPU in the second lane, it will only run at 8x.

The x16 lane is the one closest to the CPU.
I've pulled out a few lines from the Gigabyte spec sheet for your MOBO.
0.1 x PCI Express x16 slot, running at x16 (PCIEX16)
* For optimum performance, if only one PCI Express graphics card is to be installed, be sure to install it in the PCIEX16 slot.
0.1 x PCI Express x16 slot, running at x8 (PCIEX8)
* The PCIEX8 slot shares bandwidth with the PCIEX16 slot. When the PCIEX8 slot is populated, the PCIEX16 slot will operate at up to x8 mode.
0.3 x PCI Express x1 slots
(All PCI Express slots conform to PCI Express 2.0 standard.)
0.2 x PCI slots
Jesse_V
Site Moderator
Posts: 2850
Joined: Mon Jul 18, 2011 4:44 am
Hardware configuration: OS: Windows 10, Kubuntu 19.04
CPU: i7-6700k
GPU: GTX 970, GTX 1080 TI
RAM: 24 GB DDR4
Location: Western Washington

Re: Gather data: GPU Projects with 20000+ atoms,

Post by Jesse_V »

GreyWhiskers wrote:
Jesse_V wrote: "PCI-E 2.0 x16 @ x8 2.0" which of course matches GreyWhiskers's above.
Let me know if there's any further system specs you need.
Check your GPU-Z again -- you said, "PCI-E 2.0 x16 @ x8 2.0", while my screenshot above shows "PCI-E 2.0 x16 @ x16 2.0"

Tom's Hardware forum has a similar situation from a couple of years ago -- PCIe Bandwidth issues.. Showing x8 width on a single x16 slot. It may or may not apply...Which PCI-E slot the board is installed in, etc.
...
Don't tell me it's been at a reduced performance all this time for the last two months! :( This is my first build, I've still got a lot to learn...
Thanks for the valuable information. The GPU is plugged into the PCI slot that is closest to the CPU, so it should be the x16 one. I'll check out the BIOS to see what's up. I checked for updates and things seemed fine; I'll keep poking around.

EDIT: Confirmed: it is seated in the x16 slot.
EDIT2: Under "Advanced BIOS Features" the "Init Display First" setting is set to "PCIE x16". I'll try to clean/reseat the GPU and see if that helps.
EDIT3: Reseating did nothing. I did however disable the "Turbo SATA3 / USB 3.0" setting in the BIOS as apparently it uses PCI bandwidth. I'm back to x16! :)
Last edited by Jesse_V on Mon Apr 09, 2012 8:40 pm, edited 1 time in total.
sswilson
Posts: 90
Joined: Mon Dec 17, 2007 12:34 am
Hardware configuration: ASUS Crosshair IV Formula / AMD 1090T / 4X2 Gig GSkill Pi PC3-12800 / Corsair TX750W PSU / Sparkle GTX275 Plus / CoolerMaster Cosmos S / MCP655 WC Pump / MCR320 Rad / 6X Yate Loons / PA120.1 / 2X Scythe Ultra Kaze / Enzotech Luna WB / Dell Ultrasharp 2209WA

Gigabyte P35-DQ6 / Q6600 / 2X 1G 1066 Firestix / "Baked" XFX GTX 280 (RIP again :( ) / MSI GTS 450 Cyclone OC /PC P&C 750W Silencer / MCR220-QP-Res / DD DDCPX-Pro / Apogee GT / Highspeed PC Tech Station / Samsung 931BF / BenQ Q9T4
Location: Moncton, New Brunswick, Canada

Re: Gather data: GPU Projects with 20000+ atoms,

Post by sswilson »

I'll update with snippets from the code later this evening, but so far I've isolated out CPU clockspeed, and system memory speed with virtually no difference to TPF. Next up was going to be PCIe bus speed, but my system grabbed two 8XXX GPU WUs so that will have to wait.

As it stands....

Asus 990FX chipset
1090t
8G DDR3
2X 560 ti @ X16 PCIe bus (both verified via gpu-z)

CPU @ 3970 / Mem 1600 = TPF 9 min 29 seconds
CPU @ 3010 / Mem 1600 = TPF 9 Min 32 seconds
CPU @ 3970 / Mem 1K = TPF 9 Min 44 seconds


While the memory TPFs might seem to point to memory being the current bottleneck, I think the difference we're seeing is pretty typical to what we'd see on any other WU. If it were the memory interface causing low PPD on these WUs, I'd expect to see a much larger discrepancy when lowering mem speed by over 1/3 (timings remained the same).

Next possible culprits that I'll try out when I get another high atom WU will be NB speed (running @ 2400 for all above readings), and PCIe bus speed.

I was surprised to see little difference on the CPU front... I guess my initial impression that these were offloading a lot of work to the CPU was false.
{RaW}Eagle1
Posts: 25
Joined: Wed Mar 21, 2012 10:11 pm
Hardware configuration: AMD Phenom II 1090T hex core, over-clocked 3.85GHz (folding on 3 cores).
2 x ASUS Nvidia GTX480s, under-clocked by 8% for stability (both folding at 100%)
Intel Pentium Dual Core 1.86GHz, factory clock (folding on both cores, 100%)
Playstation 3 (folding all the time)
Location: Bristol, UK
Contact:

Re: Gather data: GPU Projects with 20000+ atoms,

Post by {RaW}Eagle1 »

Nathan_P wrote:I've taken this from a thread over at [H] where a teammate has the same problem

But jeez stanford..... PPD is a mear 25% of what my cards could get on other units. 25%! seriously?! I know you think GPU is over-valued but no warning? just cut PPD to 1/4? . . .

570 getting TPF of: ~21 minutes, about 5250 PPD
460 getting TPF of: ~53 minutes, about 2075 PPD

The 570's are in a: EVGA x58 Classified SLI, . . .

. . . He's confirmed that all the cards are running at x16 speed.
There are 3 models iof that mobo. Only one of them runs SLI at 16/16/16. (The E759). The E760 and E761 have split bandwidth on the second and third cards. Link:

http://www.evga.com/articles/00447/

Please can you confirm which one is being used in your example.

Thanks
Image
{RaW}Eagle1
Posts: 25
Joined: Wed Mar 21, 2012 10:11 pm
Hardware configuration: AMD Phenom II 1090T hex core, over-clocked 3.85GHz (folding on 3 cores).
2 x ASUS Nvidia GTX480s, under-clocked by 8% for stability (both folding at 100%)
Intel Pentium Dual Core 1.86GHz, factory clock (folding on both cores, 100%)
Playstation 3 (folding all the time)
Location: Bristol, UK
Contact:

Re: Gather data: GPU Projects with 20000+ atoms,

Post by {RaW}Eagle1 »

86% load
Graphics: ASUS GTX480 SLI (underclocked) 650|1300|1900 @ 16x/16x,
Mainboard: ASUS ROG Crosshair Formula V 990FX/SB950,
CPU: AMD Phenom II 1090T @ 3850MHz,
RAM: G.Skill 12800CL8 8GB @ 1920MHz 9|9|9|24

Graphics driver version: 296.10
FAH client versions: SMP 6.34 (folding on 4 of 6 cores), GPU 6.41r2 (both cards)

Image
Image
HaloJones
Posts: 906
Joined: Thu Jul 24, 2008 10:16 am

Re: Gather data: GPU Projects with 20000+ atoms,

Post by HaloJones »

I would suggest that cpu folding has no impact on this. I have a 460GTX running along with cpu folding and a 450GTS without cpu folding and both are impacted in the same way.
single 1070

Image
Post Reply