Real World Benchmarking

FreddyD

I feel that benchmarks are an important part of PC gaming, because gamers routinely make $1000+ decisions based on them. In order to properly compare similar hardware, these benchmarks have to be objective and strictly controlled. But this can make the benchmarks somewhat artificial.

So I would like to make a thread with benchmarks made from real gaming situations, coupled with opinions on the experience. You can use whatever benchmarking software you prefer (I use Fraps and Excel) and post a brief description of your hardware, the performance, and most importantly, your experience to go along with the data. Screenshots can be used to replace or supplement descriptions of the graphical settings.

There will be some e-peen, but that is not the purpose of this thread. I would like to see the widest array of hardware possible. Don't worry about how high the framerates are. Just use the settings that you normally play with.

So I'll start with an example:





I made this benchmark from about five minutes of gameplay during a 16 player online game of F.E.A.R Combat. I felt that it was worth taking the resolution down to keep the framerates high. Overall, I felt that the game looked and ran great on my current hardware, whereas it used to chug badly on my Geforce 6800. The pink line is the average framerate.

Xenocide Geek

i don't have the patience right now to perform an actual benchmark with fraps or something, but i'd say that real world benchmarking is by far the most beneficial type of benchmarking, as opposed to something like 3DMark

don't get me wrong, 3dmark is a fun little tool, and i'll probably make a thread about benchmarking on that whenever i reformat my comp next, but it's hardly the most practical application for testing performance

Monaro

That's awesome.

I'd really like to benchmark my 8800GTX, as I'm having trouble with it, and before I try an RMA would like to reinstall Windows and compare synthetic and real-world benchmarks pre- and post-install.

So how did you get that graph? Hit the FRAPS benchmark hotkey and it records it? Or is that graph an Excel job?

FreddyD

Monaro wrote: »

That's awesome.

I'd really like to benchmark my 8800GTX, as I'm having trouble with it, and before I try an RMA would like to reinstall Windows and compare synthetic and real-world benchmarks pre- and post-install.

So how did you get that graph? Hit the FRAPS benchmark hotkey and it records it? Or is that graph an Excel job?

Both. I hit the benchmark button in fraps, opened the file in Excel, and copied the results into a graph. The whole process took me about 10 minutes, and half of that was gameplay.

I'm going to bench Battlefield 2 next, because the replay benchmarks in that game tell terrible lies. 32 player online matches are really resource intensive.

FreddyD

I'm updating this thread with my Bioshock benchmarks:







Exploration felt really smooth but combat felt somewhat jerky and disconnected. I don't think this was solely a framerate issue. I had the game running at the highest settings, and I was not disappointed with the visuals. I was satasified with the experience but I may turn down the AF to keep framerates as high as possible during combat. I don't know if AA works in this game, but I'll take a look at the config files.

cfgauss

You do realize that without controlling your data, and in fact, having no idea where or how or under what situation it came from, it will mean even less than those fancy graphs you see in gaming magazines and websites (which already mean next to nothing)?

The_Scarab

Make sure you guys post basic system specs with your benchmarks. That includes OS, ram, gpu, cpu and anything else. Shit, just post the dxdiag main window.

dispatch.o

FRAPS also rapes performance in a lot of situations, I wouldn't go so far as to call what you're doing benchmarking. In fact the very nature of benchmarks means they aren't real world.

FreddyD

cfgauss wrote: »

You do realize that without controlling your data, and in fact, having no idea where or how or under what situation it came from, it will mean even less than those fancy graphs you see in gaming magazines and websites (which already mean next to nothing)?

Timedemo benchmarks are pretty much worthless for determining how a game will actually run, and most benchmarking websites don't even give you a description of how they got their data. I just want to give people an idea of the kind of real world performance they can expect on this type of hardware.

So I guess I should give a disclaimer: This isn't a thread to celebrate graphs and numbers and compare them to other graphs and numbers. This thread is really meant for people with older CPUs who are on the fence about an upgrade. Is Bioshock enjoyable on this hardware? The answer is yes.

So for the curious, the first benchmark was taken during the last battle of the demo, right after you exit the vita chamber. The second one was taken after the scene with the big daddy, as you explore the narrow corridors.

urahonky

Freddy, how do you make those graphs? I probably missed it in the OP...

Iroh

FreddyD wrote: »

cfgauss wrote: »

You do realize that without controlling your data, and in fact, having no idea where or how or under what situation it came from, it will mean even less than those fancy graphs you see in gaming magazines and websites (which already mean next to nothing)?

Timedemo benchmarks are pretty much worthless for determining how a game will actually run, and most benchmarking websites don't even give you a description of how they got their data. I just want to give people an idea of the kind of real world performance they can expect on this type of hardware.

So I guess I should give a disclaimer: This isn't a thread to celebrate graphs and numbers and compare them to other graphs and numbers. This thread is really meant for people with older CPUs who are on the fence about an upgrade. Is Bioshock enjoyable on this hardware? The answer is yes.

So for the curious, the first benchmark was taken during the last battle of the demo, right after you exit the vita chamber. The second one was taken after the scene with the big daddy, as you explore the narrow corridors.

This is exactly the line of thinking that people should have when choosing new hardware if they are on a budget.

Not being in control of the data is what makes this testing more relevant than time demos. The idea is to throw all of the unexpected shit at the computer you possibly can, in order to stress it out a little and see if you experience any unplayable conditions.

Where and how frequently you experience a framerate below 60/s or so is way more important than what your max or average is. Overall smoothness makes the game far more enjoyable than being able to beat off to your peak framerate.

FreddyD

urahonky wrote: »

Freddy, how do you make those graphs? I probably missed it in the OP...

You hit the benchmarking key on FRAPs, and hit it again when you're done. You can find the data in the benchmark folder in your fraps install. Then you open the fps table in Excel, and click insert->chart->line..., copy the graph into a picture editor, and save it in your favorite format.

TxdoHawk

FPS benchmarks and 3dmark and so on get slammed pretty hard these days, but IMO real world benchmarks should absolutely not straight-up replace them outright. They primarily are used in hardware reviews that compares the hardware to other pieces in its class, and while real world benchmarks are great to get an actual idea of what the hardware can comfortably run, it's much easier to decipher a simple bar graph when you are looking at who reigns supreme in performance.