Chip shortage will affect PCs, and GPUs

[MeeLee]

The current chip shortage, caused by COVID layoffs, resulted in panic buying among car manufacturers and other companies relying on chips.
Affected CPUs are those dependent on TSMC, MU, Samsung, and other outsourced chip manufacturers in Asia.
Companies that will depend on Intel CPUs, and TI chips, will not be much affected, as these CPUs are built in-house in USA, and manufacturing is still going on.
GPU manufacturers like AMD and Nvidia, who use TSMC and Samsung, and finally would have caught up with RTX 3000 demand, may now again run into a stall.

While some say the shortage could last 1 year, Intel believes it could last up to 2 years.

We'll see a rise in price of things like cars, TVs, but also PCs, GPUs, laptops, etc...
This will certainly affect those who are looking for newer hardware.

Nvidia might also delay the RTX 4000 series by one year (to mid-end of 2022).

My advice?
Buy now, while the hardware is still available.

[aetch]

Chip shortages don't really affect me, I'm more of a mashed potato person

[JimF]

The prices on the Ryzen 5000 series are beginning to ease, and I expect will be down to more normal levels by the end of the year.
As for GPUs, they are a different ballgame affected by a lot of factors, but I am not looking to buy until the 5 nm generation anyway.

[gunnarre]

If the cryptocurrency market collapses, which it tends to do from time to time, there might be a glut of used GPUs hitting the second hand market. Some popular cryptocurrencies are also moving away from using the waste of electricity as their core lottery function, which might have the same effect on the GPU market.

[iero]

PoS in Ether got pushed back to late 2022, so no luck on that front. As for a crypto market crash, if and when it happens, the GPU prices will lag some months behind, before the price normalises to below MSRP for used parts.

[debs3759]

I've not seen an RTX 3000 series card for sale new from a UK retailer at all (OK, I don't check every day). Only cards I have seen are on eBay, probably burned out from crypto, and often at least double the RRP. Even the new cards on ebay I'm skeptical about, if the big retailers can only get enough for pre-built systems.

[iero]

I've not seen an RTX 3000 series card for sale new from a UK retailer at all (OK, I don't check every day). Only cards I have seen are on eBay, probably burned out from crypto, and often at least double the RRP. Even the new cards on ebay I'm skeptical about, if the big retailers can only get enough for pre-built systems.

Given that I am currently mining instead of folding, I'll give my 2cents:

My 3060 is currently running its core at 48~51°C, depending on the time of day. Nvidia doesn't have memory temp data for the end user. Even so, obvious signs of throttling due to excess memory temperatures, appear only if i let the core exceed 70-71°C. So, i would assume, that, if GDDR6's operating temperature celling is 100°C, when the core is at 50°C ---> memory is around 80°C. So everything seems fine. If anything, folding seems to be more harmful for the gpu as a whole, since the core is running at higher temps and voltages.

[MeeLee]

I did hear that in China, a large mining farm (literally an airplane hangar filled with racks of RTX GPUs), got flooded.
They literally had thousands of GPUs running the crypto markets.
I bet they probably cashed in their money, since they'll be needing it for upgraded hardware.
https://fortune.com/2021/04/20/bitcoin- ... pollution/

[aetch]

If anything, folding seems to be more harmful for the gpu as a whole, since the core is running at higher temps and voltages.

The overclocking tools don't just help you overclock your card(s), they can help them run cooler as well by lowering the power limit and/or setting a more aggressive fan profile from the default one. They can also help stop the fan from doing annoying things like whining or periodically spinning up.

In Windows I use MSI Afterburner, it's simple and straightforward.
Power limit can be set with a drag bar.
The fan speed can also be set manually with a drag bar or a custom fan curve.

In Ubuntu I use nvidia-smi which is baked into the geforce drivers.
Setting the power limit was simple enough.
The real pig was setting the fan speeds. The only options here are between the factory set fan curve or a manually set fan speed which is not easily set.
Anyway, I now have a script which gives my GPUs a custom configuration after a reboot.

I did hear that in China, a large mining farm (literally an airplane hangar filled with racks of RTX GPUs), got flooded.

That's so meta, the cards got flooded, the market is about to get flooded with those cards.

[MeeLee]

The real pig was setting the fan speeds. The only options here are between the factory set fan curve or a manually set fan speed which is not easily set.
Anyway, I now have a script which gives my GPUs a custom configuration after a reboot.

Usually you'll need a gui to enable the fans.
Thing is, as soon as you disable the gui (to use minimal resources, and basically run Ubuntu via terminal), for some reason the Nvidia GPU settings also get reset.

How do you overclock and set fan speed in Ubuntu?
Do you use the GUI, and then copy a settings file to boot up with these settings?

[aetch]

How do you overclock and set fan speed in Ubuntu?
Do you use the GUI, and then copy a settings file to boot up with these settings?

Caveat - I am not a linux pro/expert, I'm barely a linux user. What follows is what I did, caused me a lot of frustration and probably has a lot of bad practice in it.

That machine was in hibernation for summer and it did not want to be awoken, it was comfy where it was.

I'm not overclocking per se, I'm underclocking but it's the same techniques.
This only applies to NVidia gpus, I have neither the suitable GPUs nor the inclination to figure this out for AMD.
I run advanced control on a machine separate from the ones I run the client on.

Notes on the Machine
It's Ubuntu desktop I'm running. 18.04 was originally installed, it's now upgraded to 20.04.
I'm folding exclusively on NVidia GPUs, those are the most powerful GPUs I have.
I'm running it headless on a wireless connection. The desktop install gives me the desktop session needed to automatically connect the wireless and run the GPUs.
I didn't disable the gui or anything else to reduce resources, it's a very capable machine I'm running.
That said, I don't do cpu folding on it, the cpu was impeding the gpu folding more than it was adding to cpu folding. My Ryzen handles the cpu folding.

Code: Select all

*********************** Log Started 2021-02-08T23:10:06Z ***********************
23:10:06:Trying to access database...
23:10:06:Successfully acquired database lock
23:10:06:Read GPUs.txt
23:10:08:Enabled folding slot 00: PAUSED gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 (by user)
23:10:08:Enabled folding slot 01: PAUSED gpu:1:TU104 [GeForce RTX 2070 SUPER] 8218 (by user)
23:10:08:****************************** FAHClient ******************************
23:10:08:        Version: 7.6.13
23:10:08:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
23:10:08:      Copyright: 2020 foldingathome.org
23:10:08:       Homepage: https://foldingathome.org/
23:10:08:           Date: Apr 28 2020
23:10:08:           Time: 04:20:16
23:10:08:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
23:10:08:         Branch: master
23:10:08:       Compiler: GNU 8.3.0
23:10:08:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
23:10:08:                 -funroll-loops -fno-pie
23:10:08:       Platform: linux2 4.19.0-5-amd64
23:10:08:           Bits: 64
23:10:08:           Mode: Release
23:10:08:           Args: --child /etc/fahclient/config.xml --run-as fahclient
23:10:08:                 --pid-file=/var/run/fahclient.pid --daemon
23:10:08:         Config: /etc/fahclient/config.xml
23:10:08:******************************** CBang ********************************
23:10:08:           Date: Apr 25 2020
23:10:08:           Time: 00:07:53
23:10:08:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
23:10:08:         Branch: master
23:10:08:       Compiler: GNU 8.3.0
23:10:08:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
23:10:08:                 -funroll-loops -fno-pie -fPIC
23:10:08:       Platform: linux2 4.19.0-5-amd64
23:10:08:           Bits: 64
23:10:08:           Mode: Release
23:10:08:******************************* System ********************************
23:10:08:            CPU: Intel(R) Core(TM) i7-5930K CPU @ 3.50GHz
23:10:08:         CPU ID: GenuineIntel Family 6 Model 63 Stepping 2
23:10:08:           CPUs: 8
23:10:08:         Memory: 31.28GiB
23:10:08:    Free Memory: 30.49GiB
23:10:08:        Threads: POSIX_THREADS
23:10:08:     OS Version: 5.4
23:10:08:    Has Battery: false
23:10:08:     On Battery: false
23:10:08:     UTC Offset: 0
23:10:08:            PID: 1080
23:10:08:            CWD: /var/lib/fahclient
23:10:08:             OS: Linux 5.4.0-65-generic x86_64
23:10:08:        OS Arch: AMD64
23:10:08:           GPUs: 2
23:10:08:          GPU 0: Bus:2 Slot:0 Func:0 NVIDIA:8 GP102 [GeForce GTX 1080 Ti] 11380
23:10:08:          GPU 1: Bus:1 Slot:0 Func:0 NVIDIA:8 TU104 [GeForce RTX 2070 SUPER]
23:10:08:                 8218
23:10:08:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:7.5 Driver:11.1
23:10:08:  CUDA Device 1: Platform:0 Device:1 Bus:2 Slot:0 Compute:6.1 Driver:11.1
23:10:08:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:455.38
23:10:08:OpenCL Device 1: Platform:0 Device:1 Bus:2 Slot:0 Compute:1.2 Driver:455.38
23:10:08:******************************* libFAH ********************************

Setup the machine

The easy bit first.
Install desktop linux OS, configure wireless working if you plan to use it.
Download the Linux 64-bit Geforce driver from NVidia, I found using the browser easier that wget.
Update/upgrade ubuntu as far as you can through the package manager and the console.
In the console if I know I'm going to be doing stuff which needs root privileges I don't waste time prepending everything with sudo, I just elevate the environment to root using sudo -i
Download and install the FAHClient.
You may need to enable SSH for remote access, I can't remember just now.

WARNING - I have found that ubuntu desktop silently installs updates without waiting for your input and can break the loading of the driver, making the following a little sketchy. I'm assuming it's breaking the loader, I'm re-installing the same driver that was installed before the silent updates and it works perfectly fine after. My lazy ass workaround is to simply re-install the drivers while I am applying less important updates/upgrades.

The not-so-easy bit?
Purge the neuveau.
You'll need exit to the console, terminate the xterm and remove/purge the nvidia neuveau drivers. Google for guidelines.
The neuveau drivers need to be completely purged or NVidia's proprietary driver will fail to install.
It took me several rounds of purging the neuveau drivers before the proprietary drivers would install, I think I ended up just disconnecting the monitor and doing this remotely through putty.

Install the the proprietary drivers
bash the NVidia%%%.run file
These NVidia drivers contain everything we need - OpenCL, Cuda, NVidia-SMI, NVidia-settings. No need to separately install opencl.
For the umpteenth time reboot the machine, make sure advanced control can communicate with, and control, the client.
At this point you should be ready if you just want to fold.

But no, that's way too easy. We want to remotely control the GPU performance and fan speeds. Now the fun starts.
Below is my own custom rc.local file. It applies the power limits at startup but fails to apply the fan speeds, so I have to run it again once I've logged in.
Let's pull apart one of the nvidia-smi commands.
"/usr/bin/nvidia-smi -i 0 -pl 110"
"-i 0" - this is the index of the gpu you are affecting, running the command "nvidia-smi" will give a report of the nvidia gpus in your system, the index is the the left of the gpu name. If you leave this out the command will affect all gpus.
"-pl 110" - this is the power limit in watts you want to apply to your card(s), this must be set within the range set by nvidia.
Just punching "nvidia-smi" into the console gives you a summary of the cards in your system, "nvidia-smi -h" go knock yourself out.

The fan settings part
"/usr/bin/nvidia-settings -h" - this simply gives you the commands available to nvidia-settings
"DISPLAY=:0 XAUTHORITY=/run/user/1000/gdm/Xauthority" - this is the context in which nvidia-settings needs to run if you actually want to query or set anything on the gpus. It's important you locate your Xauthority file, this is mine, online guides gave a different location. If you're using the wrong Xauthority file the following nvidia-settings command just throws an error.

"DISPLAY=:0 XAUTHORITY=/run/user/1000/gdm/Xauthority /usr/bin/nvidia-settings -q fans" - will give a report of the fans available to control

In the command to set the speed of the fans "[fan:0]/" is an optional entry, with it your specifying a single fan to control, without it you're doing them all. My dual card setup has 3 fans between them, nvidia-settings exposed all 3 of them.

Code: Select all

#!/bin/bash


# scripts/commands to execute at boot time using root privileges

# persistence mode does not apply to geforce but may work on other nvidia cards
nvidia-smi -pm 1

# lower the power target of the GPUs, GTX 1080 Ti should be id 1 with RTX 2070 Super id 0

# RTX 2070 SUPER and GTX 1080Ti combo
/usr/bin/nvidia-smi -i 0 -pl 110
/usr/bin/nvidia-smi -i 1 -pl 150

# take control of GPU fans
DISPLAY=:0 XAUTHORITY=/run/user/1000/gdm/Xauthority /usr/bin/nvidia-settings -a GPUFanControlState=1

# set speed of gpu fans individually, the "[fan:0]/" part is important, change the number to suit
# NOTE - The fans may be in a different order to the GPUs, test to determine which fan is which
# DISPLAY=:0 XAUTHORITY=/run/user/1000/gdm/Xauthority nvidia-settings -a [Fan:0]/GPUTargetFanSpeed=65

# set speed of all gpu fans
DISPLAY=:0 XAUTHORITY=/run/user/1000/gdm/Xauthority /usr/bin/nvidia-settings -a GPUTargetFanSpeed=55

exit 0

I've just spent the last 3-4 hours pulling all of this together, I hope it makes sense and is useful.
I would love to hear some feedback.

[gunnarre]

Usually you'll need a gui to enable the fans.
Thing is, as soon as you disable the gui (to use minimal resources, and basically run Ubuntu via terminal), for some reason the Nvidia GPU settings also get reset.

This is because when no processes are using the GPU, the GPU driver will un-initialize the GPU even though it is still loaded into the kernel, so the GPU returns to its boot state and forgets the settings. If you're folding in headless mode, you might experience this when one WU finishes before the next one starts. One way to get past this is by leaving the GUI active, while the more elegant soltuion is to use the Persistence Daemon: https://docs.nvidia.com/deploy/driver-p ... nce-daemon The Persistence Daemon should keep GPU settings active in the GPU even without any active client applications using the GPU.

[MeeLee]

Thanks!
I wonder if you can just enter the command "DISPLAY=:0 XAUTHORITY=/run/user/1000/gdm/Xauthority /usr/bin/nvidia-settings -a GPUFanControlState=1" in terminal (perhaps via an .sh script), like save the line to fans.sh, and run it?

I never got what Persistence mode was about.
I guess I'll have to try it someday.
Basically it would mean to do:

sudo ./fans.sh (in case the fan speed is scripted; or see above for full command)
sudo nvidia-smi -pm 1 (to enable persistence mode)
sudo init 3 (I believe, to close the GUI)

I don't have access right now to a Ubuntu system, as they're all running, and I don't want to mess them up.
If the fan speed can be adjusted via a simple terminal line like that, perhaps the next thing would be to set the overclocking of GPU and memory parameters via terminal!

[aetch]

I went through all that pain just so I could control fans speeds from a terminal.

Those are terminal command, they're the ones that work for me, your mileage may vary. They work perfectly fine outside of a script but a script means you don't have to remember exact syntax.
rc.local is a .sh file, it just has the added bonus of being the one ran by root at boot time. You can run it from a terminal whenever you want, I use putty.

Although I setup that system with a keyboard, monitor and mouse, in production it only has a power cable and wifi dongle, nothing else, not even ethernet.

init 3, that rings a bell with me as the one to kill the GUI so you can purge the neuveau drivers.

Glossary
Persistence mode - I suspect this is to preserve settings in the card between reboots, which geforce cards don't do anyway. It doesn't apply to geforce cards, yet, but nvidia-smi complains if you change the power limit without enabling persistence mode first. It doesn't stop you changing the power limit either way.

GPUFanControlState - this merely sets who has control of the fans (0) the card or (1) you. If this is not set to 1 the subsequent GPUTargetFanSpeed fails.

sudo, SUDO, sUdO, SuDo.
I find it easier to just enter the root environment using "sudo -i" then you don't need to prepend anything with sudo.

[MeeLee]

Sorry, I missed that 'fan speed' line.

If you don't like sudo everything, my way of doing would be:

Code: Select all

sudo su
DISPLAY=:0 XAUTHORITY=/run/user/1000/gdm/Xauthority /usr/bin/nvidia-settings -a GPUFanControlState=1
DISPLAY=:0 XAUTHORITY=/run/user/1000/gdm/Xauthority /usr/bin/nvidia-settings -a GPUTargetFanSpeed=70
init 3

Here's a walkthrough site, just in case it won't work on other systems (haven't followed it yet): http://bailiwick.io/2019/09/21/controll ... tu-system/

The question is, once coolbits have been enabled, one should be able to adjust fan size at any time, no?

The advantage with writing the fan speed in a script, is that you can supposedly change the fan speed with %1, and do an 'if' to ask you the %1 value (2 numbers would work).
That way you can run the script via a simple command (f.sh, or fans.sh) , and the script asks you the desired speed, or continues the last known speed setting.
The benefit for putting it in the boot script, is that it'll set a fixed value upon each reboot; good if your system is regularly crashing or rebooting (due to power outages or what not).

Now the overclock....
site here noted the following:
https://gist.github.com/bsodmike/369f8a ... 1264d549e9

Code: Select all

sudo DISPLAY=:0 XAUTHORITY=/var/run/lightdm/root/:0 nvidia-settings --assign "[gpu:0]/GPUGraphicsClockOffset[3]=0" --assign "[gpu:0]/GPUMemoryTransferRateOffset[3]=1400"

If someone could test it, would be nice.