Hardware reviews with FAH in mind

[toTOW]

What would you like to see in a Hadware review (for instance, a review of a CPU, a graphics board, ... ), with FAH in mind (that you might not find in regular reviews) ?

Please post any suggestions you might have and that are not included in the poll.

[kiore]

Great idea, although you do see F@H sometimes run along side games in hardware reviews, reviews aimed at people buying hardware with only folding or mainly folding in mind would be excellent. Also things like the cost effectiveness of say a 275 gtx at 199 euros vs a 260gt at 149 euros in output and points per watt etc,( I'm looking at this right now).
I know discussions like this already go on but specific reviews would be a real bonus.
kiore.

[Flathead74]

Personally, I don't see the need to include "Other distributed projects benchmarks",
when reviewing hardware with FAH in mind.
For one thing, there are so many "Others" that you would never be able to please anyone.
It seems the time would be better spent on FAH, only.

I think this is a great idea, overall.

Points per watt would be a good addition.
Everyone could then deduce there own cost, relative to their own location.

[whynot]

(sorry, couldn't stay this rant) Operating systems -- I'm not about benchmark comparision, just hint bottom lines would be enough.

[MtM]

Personally, I don't see the need to include "Other distributed projects benchmarks",
when reviewing hardware with FAH in mind.
For one thing, there are so many "Others" that you would never be able to please anyone.
It seems the time would be better spent on FAH, only.

I think this is a great idea, overall.

Points per watt would be a good addition.
Everyone could then deduce there own cost, relative to their own location.

+1

Power consumption is a need to know component. Wall draw preferably so it's easier for people to calculate their running costs, draw iddle and load with varieus components, and when you give performance numbers don't forget to include driver version and os, and preferably more then one comparison ( latest whql and latest from cuda_get maybe? Or just the two most popular folding drivers at that moment though I would like seeing atleast whql drivers and certifeid cuda drivers only ). Give accurate numbers not only to one client per system but write reviews with the total system in mind as well. One client reviews could target the occasional folder, but dedicated folders will appriciate a review giving numbers on as many possible combinations which can be run on a particular system.

What would be incrementally valuable as the datasets grows is an searchable database with all previous test results ( as fahinfo.org, but with better search options, links to the actual reviews ect ).

(sorry, couldn't stay this rant) Operating systems -- I'm not about benchmark comparision, just hint bottom lines would be enough.

You're missing the point, it's about the benchmarking when you want to give clear and accurate numbers. For the 'bottom line' there is enough info on this forum to answer many if not all possible questions regarding a bottum line. People like numbers even more then a bottom line, why you might ask well I think people like to think for themselfs, seeing numbers allows us to deduct the bottum line for ourselfs and also allow us to see how the bottom line is determined thus having no doubts about the accuracy. This is imho and maybe you disagree, nothing wrong with that but I thought I'd give the reasoning behind my opposite standpoint.

[mdk777]

Re: Hardware reviews with FAH in mind

Postby whynot » Sat Aug 08, 2009 2:14 pm
(sorry, couldn't stay this rant) Operating systems -- I'm not about benchmark comparision, just hint bottom lines would be enough.

I think he means, if you are say; reviewing a high end QUAD CORE CPU, or a two socket server, a benchmark comparison of WIN VS. VISTA. VS. LINUX on that hardware configuration would be helpful.

I agree #'s tell the story, but I disagree that you can get an accurate bottom line by reading the forums. No one will tell you, "LINUX will get you 2X the efficiency of WIN on a SMP running on a QUAD 9950."

Quite the opposite, with changing optimizations, cores and WU, the official response is always, YOUR MILEAGE MAY VARY. and OFFER SUBJECT TO CHANGE AND CANCELLATION AT ANY TIME.

While there are very good reasons for these responses, it does make getting a quick "bottom-line" answer difficult.

[MtM]

Re: Hardware reviews with FAH in mind

Postby whynot » Sat Aug 08, 2009 2:14 pm
(sorry, couldn't stay this rant) Operating systems -- I'm not about benchmark comparision, just hint bottom lines would be enough.

I think he means, if you are say; reviewing a high end QUAD CORE CPU, or a two socket server, a benchmark comparison of WIN VS. VISTA. VS. LINUX on that hardware configuration would be helpful.

I agree #'s tell the story, but I disagree that you can get an accurate bottom line by reading the forums. No one will tell you, "LINUX will get you 2X the efficiency of WIN on a SMP running on a QUAD 9950."

Quite the opposite, with changing optimizations, cores and WU, the official response is always, YOUR MILEAGE MAY VARY. and OFFER SUBJECT TO CHANGE AND CANCELLATION AT ANY TIME.

While there are very good reasons for these responses, it does make getting a quick "bottom-line" answer difficult.

Let's take out this single part..

No one will tell you, "LINUX will get you 2X the efficiency of WIN on a SMP running on a QUAD 9950."

They won't tell you because it's flawed, but if you look around you will find direct comparisons in posts, with the numbers gotten with a given project/driver/os/hw.

The bottum line is already here I still say, maybe it's not in bite size pieces, or easy to find, in contrary it's not, which is why I applaud toTow's initiatives and hope he will get everything from the ground.

That people add YMMV is imho only because dumb people who don't take into account variables will come back in x months and claim those statements are false because their own numbers are diffrent at that time. Millage with folding doesn't varry, a c2d with x ram and a given os, driver ect combination will always perform the same on the same wu's, you're mistakenly taking statements trying to lower the number of uninformed people raising questions/statements with the possibility that performance really fluctuates depending on mhmmmm let's say the 'weather'


Know what I mean? Two diffrent things alltogether.

Edit: Just to add, I posted performance numbers for certain cases in the past, but after the first few times when I got some people questioning my numbers because their hw got less then mine I just added the ymmv because I didn't want to go over 100 dependancies/inluences ect with everyone who had a number which was x amount off my own numbers. So instead I gave as much hw and software info as I could and the ÝMMV' statements.

Edit2: this is very hard to do, but since folding is one of the most stressfull enviroments for hardware it might be very usefull take into account in reviews even more then normal the power distribution components of the hardware in question. Not only how many lanes ect but more the 'feel' of it. This is a subject where real life testing is impossible and the reviewer will have to show their expertise and build up confidence from the readers to form well founded conclusions.

[mdk777]

I'm not disagreeing with anything you've said.
Let me put it another way.
If you have to benchmark a particular piece of hardware with every combination and permutation of OS, CLIENT, WU and LOAD (CPU, W/GPU, W/out GPU) etc. etc. etc.
; The work required to do one review would be crushing.

My point was, that the OP wanted the OS factored into the review, but was willing to have a "representative sample" IE. "bottom line" comparison rather than requiring an exhaustive compilation of all possible variables.

I know it will not be easy, that's why it hasn't been done before.

[MtM]

I'm not disagreeing with anything you've said.
Let me put it another way.
If you have to benchmark a particular piece of hardware with every combination and permutation of OS, CLIENT, WU and LOAD (CPU, W/GPU, W/out GPU) etc. etc. etc.
; The work required to do one review would be crushing.

My point was, that the OP wanted the OS factored into the review, but was willing to have a "representative sample" IE. "bottom line" comparison rather than requiring an exhaustive compilation of all possible variables.

I know it will not be easy, that's why it hasn't been done before.

Ahh yeah ok I didn't get your point. Offcourse the workload will be crushing, and the fact that it has not been done before just strengthens my believe toTow is putting things in motion which will be very very valuable assets to the folding community in the long run, if they are carried out in the meaning I'm deducting from the proposals/threads.

I'm again just afraid that we already have reviews which offer bottom line statements, but my problem with almost all of them is the lack of info given. Most of the time os,settings and wu details are omitted or otherwise lacking ect.

I thought the overall forums consensis was that the current reviews are all lacking, and while the info we would like does represent a very heavy work load it is the only route to satisfy those who want either or both be able to form their own minds based on the numbers or to be able to check the numbers by duplicating all settings. Maybe that's also it, hw reviewers who touched on f@h are almost allways not real f@h addicts/enthousiasts and therefore don't always know all the aspects which would be important to an (aspiring) folder to be able to make buying decissions. So, I very much applaud this incentive.

Though, maybe, since the trust this community has in it's own members, the crushing load doesn't have to fall in one place. As I said, allot of info is already obtainable through the forums/members, and maybe the only real thing needed is web space and some people who can weed this info out from the misinformation sometimes accompaning it and then correlate all already know info into a comphrehensive comparison/summary. After that's done, any existing holes in that info can be filled by requests from the community to the community, and then combined.

But that's sounding allot diffrent then toTow's idea so I wonder how he sees this?

[jrweiss]

Benchmark data may be hard to come by, unless the Pande Group releases a WU specifically for that purpose. Otherwise, there is too much variation and evolution among WUs for that to be a meaningful figure. Still, since those buying hardware specifically for Folding are primarily interested in PPD, it would be nice to have.

Power consumption under full load, OTOH, should be in EVERY review! All the "Energy Star" numbers make too many assumptions regarding sleep/standby cycles, etc. A minimum power consumption set would include MoBo+CPU, MoBo+GPU, and MoBo+CPU+GPU at full Folding loads.

[MtM]

Benchmark data may be hard to come by, unless the Pande Group releases a WU specifically for that purpose.

This was requested and considerd in the past, maybe it's time to ask the official channels again? Though, with the variations in core's this is complicated and the effort required would be quite big which is if I remember correct the reason it was denied/deemed not having a high enough return value to be really worth it.

[toTOW]

I think both jrweiss and MtM understood the idea I had in mind.

I don't plan to replace regular hardware reviewers : there are plenty of them, and they're all doing a good job, but in most reviews, we don't have any performance indicators that we could use to fold with the reviewed hardware. Synthetic benchmarks are a bit useless in my opinion, and game benchmarks, although more realistic, are only useful to gamers.

So I'd like to give some indication on how a given piece of hardware would do on FAH at a given time (but that's also true in regular reviews, as drivers and games get updated and their performance varies as the time passes) in real folding environment. I know that's not perfect, but that's a good place to start before we find better procedures.

It might not be possible to start with comparisons as it require a lot of pieces to review at the same time, but I might be able to compile some when I had more basic reviews.

I also plan to include power consumption values as I already started to build my machine with the best folding efficiency in mind, and because many people are also interested to know that to save power and money.

I don't know yet when the review will be released, but it will be considered as a "beta version" and all comments will be welcome

[whynot]

(sorry, couldn't stay this rant) Operating systems -- I'm not about benchmark comparision, just hint bottom lines would be enough.

You're missing the point, it's about the benchmarking when you want to give clear and accurate numbers.

As you clearly had stated later want and give are orthogonal.

For the 'bottom line' there is enough info on this forum to answer many if not all possible questions regarding a bottum line. People like numbers even more then a bottom line, why you might ask well I think people like to think for themselfs, seeing numbers allows us to deduct the bottum line for ourselfs and also allow us to see how the bottom line is determined thus having no doubts about the accuracy.

Yeah, sure -- "get latest bubuntu". That has nothing to do with a bottom-line.

Look, When I'd upgraded MB, I'd found that 'mkinitramfs' of 'etch' can't build initrd. I've discovered, that 'lspci' of 'etch' knows 4 of 21 devices found on bus. Surprisingly, the old initrd (built on a previous MB) booted happily, and immediate upgrade to 'lenny' fixed that.

Those MB's seemed to be same, for me. I just didn't know the bottom-line.

This is imho and maybe you disagree, nothing wrong with that but I thought I'd give the reasoning behind my opposite standpoint.

And I didn't give reasoning for mine. I don't reject your point (I think that people wants benchmark, people wants to believe (and then step into reality)). I don't fight your reasons (because I don't reject your points). I'd just thought that some year before that (mentioning bottom-lines) couldn't be possible. May be it's just time to begin?

[toTOW]

We set up a poll on FAH-Addict to choose which card will be used to set up the reviews structure ... (French version)

[toTOW]

It's finally here, the first FAH-Addict review : nVidia 9800 GTX+

Feel free to post any comment, positive or negative, to help us improve future reviews