Power requirements - GPUs cutting out

[Ripper36]

I have an annoying problem with my folding rig i72600k @3.70 GHz, with two Gigabyte GTX570OC graphics cards. Using a Coolermaster SilentPro Gold 1000W power supply.

At present I am only folding on the two gpus. Mainly getting Core17 units.

I get recurring crashes which leave no trace in logs or windows events manager - makes me think a power overload is stopping the whole system.

A log of the GPU sensors shows no apparent change and both are operating at about 69 degrees.

If I only fold with one card there are no issues.

Any suggestions about power requirements for the system welcomed. Otherwise it must be a random memory failure on the second card that the various test routines fail to show up.

Thanks

[Zagen30]

I find it hard to believe that your PSU wouldn't be providing enough power. The system as a whole can't be drawing much more than 600W (even that's probably a sizable overestimate), and Cooler Master is hardly some cheap no-name brand. If you wanted to be sure it wasn't a power overload, you could use a Kill-a-Watt to measure the draw at the wall.

When you say that if you only fold with one card there are no issues, are you always pausing the same card? If you are, you could try pausing the other one and running the one that you normally pause. That would help determine if it is a problem with the second card.

[Joe_H]

One other thing to check is if there is sufficient air flow to cool the components on the card away from the temperature sensors, especially on the bottom of the PCB that would be close to the second card.

[jrweiss]

Is the 12V voltage from the PSU stable? Use CPUid's HWMonitor or similar to ensure you're not getting voltage drops when the second card is loaded; many current GPUs are sensitive to voltage fluctuations. If both GPUs are running from a single cable, try separate cables, since total draw could exceed 36 amps.

Also ensure you have enough airflow from your case intake fans to handle the exhaust from the 6 GPU fans plus the PSU fan plus the CPU exhaust. It's possible the PSU is overheating if it's starved for air, even if the GPUs are OK.

[Ripper36]

Thanks for all these great suggestions.

There's plenty of airflow with a big Coolermaster case, multiple fans - though the air circulation is possibly not so good around PSU so I'll check that.

The GPU-Z sensors show the voltage for the cards completely stable, but CPUID - HW monitor shows 12v voltage varies sometimes from 12.025 to 12.074

- I will investigate further.

[jrweiss]

A 0.05V fluctuation is nothing to worry about. However, HWMon probably takes that at the motherboard supply, which has a dedicated cable. That shows the PSU itself is stable.

The actual GPU voltage is not the only concern; the 12V supply to the card must be stable. That's why I asked whether both GPUs are powered by the same cable -- 36 amps is a lot for a single skinny cable...

[Ripper36]

No, each card has its own cable back to the PSU.

[jrweiss]

Swap the cables and see if the affected card changes; or simply replace the cable on the affected card if it's modular. It COULD be as simple as a bad cable...

[PS3EdOlkkola]

The Gigabyte GTX570OC has one 6 pin PCIe connector and one 8 pin PCIe connector (version 1.0), and two 6pin PCIe connectors for version 2.0. The Coolermaster SilentPro Gold 1000W is a semi-modular design with PCIe 6+2 pins x 4 and PCIe 6 pins x 4 connectors. The modular connectors for PCIe support 2 x 6 pin connectors, and the rest of the PCIe connectors are directly wired to the power supply. You said that each card has its own cable back to the power supply, so I'd assume that you're using an 8 pin connector to connect each GPU with the main cable (for version 1, 6 pin for version 2) to the PS and the modular 6 pin connector to each GPU. I would suggest removing the modular 6 pin connectors and use the 6 pin PCIe connectors on the main cable to connect to the GPUs. The PS is a single-rail design and can push 82 amps. Each GTX570OC when not over-clocked pulls 35 amps at full load (benched with FurMark) with two cards, or 85% of the PS capacity, so your PS should easily be able to handle the load of both. If you're overclocking at 15% over stock clocks, power draw jumps to 41 amps, so you're right at the PS capacity. I'd suggest switching the cables as described, and if that does not solve the problem, try substantially underclocking your GPUs by 400MHz to start and slowly increase the clock rate 50 MHz at a time until you find the point of instability. Back off 50 MHz from your unstable setting and all should work.

[Napoleon]

Each GTX570OC when not over-clocked pulls 35 amps at full load (benched with FurMark) with two cards

12V*(35A + 35A) = 420W + 420W = 840W... seriously? FurMark is vicious, but 840W total? I thought 75W + 2x150W (mobo + 2xPCIE 8pin cables) was the spec'd max for a single card. Well... dual GPU cards might draw a bit more than 375W at full load, but last I checked, GTX570 isn't one of those.

http://www.geforce.com/hardware/desktop ... ifications
http://www.geforce.com/hardware/desktop ... ifications

732MHz vs 607MHz, 219W vs 365W.

[jrweiss]

Yes, TDP of the GTX570 is 219W. A Tom's Hardware review indicated power draw was in line with the TDP. An OC version might draw a bit more, but still in the range of 20 amps vs the 35 amps claimed above.

Still, a bad cable could be a problem with a 20 amp draw...

Have you isolated the problem to a specific slot or card? Have you swapped the cards to opposite slots, still using the same slot to drive the monitor? Have you tried running the monitor from the integrated gfx rather than one of the discrete GPUs? Have you tried Folding only on the card in the second slot?

Also, have you tried backing off to stock clocks on the CPU?

There are a lot of things to try before deciding on a "random memory failure" on one of the cards...

[Ripper36]

Thanks for all this valuable information. I will work through it over the next week!

[jrweiss]

One other thought... I've been reading in various reviews and forums that these GPUs may have throttling protections that may cut in at 70 or 80C, depending on BIOS, driver, and the phase of the moon (or detection of a particular load pattern). If both GPUs level at 69C, you may be running into one. After the other troubleshooting, you might want to research your particular GPU/driver combination and settings.