As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/
Options

Real World Benchmarking

FreddyDFreddyD Registered User regular
edited August 2007 in Games and Technology
I feel that benchmarks are an important part of PC gaming, because gamers routinely make $1000+ decisions based on them. In order to properly compare similar hardware, these benchmarks have to be objective and strictly controlled. But this can make the benchmarks somewhat artificial.

So I would like to make a thread with benchmarks made from real gaming situations, coupled with opinions on the experience. You can use whatever benchmarking software you prefer (I use Fraps and Excel) and post a brief description of your hardware, the performance, and most importantly, your experience to go along with the data. Screenshots can be used to replace or supplement descriptions of the graphical settings.

There will be some e-peen, but that is not the purpose of this thread. I would like to see the widest array of hardware possible. Don't worry about how high the framerates are. Just use the settings that you normally play with.

So I'll start with an example:

FearBenchmark.jpg

FEAR_Benchmark.png

I made this benchmark from about five minutes of gameplay during a 16 player online game of F.E.A.R Combat. I felt that it was worth taking the resolution down to keep the framerates high. Overall, I felt that the game looked and ran great on my current hardware, whereas it used to chug badly on my Geforce 6800. The pink line is the average framerate.

FreddyD on

Posts

  • Options
    Xenocide GeekXenocide Geek Registered User regular
    edited August 2007
    i don't have the patience right now to perform an actual benchmark with fraps or something, but i'd say that real world benchmarking is by far the most beneficial type of benchmarking, as opposed to something like 3DMark

    don't get me wrong, 3dmark is a fun little tool, and i'll probably make a thread about benchmarking on that whenever i reformat my comp next, but it's hardly the most practical application for testing performance

    Xenocide Geek on
    i wanted love, i needed love
    most of all, most of all
    someone said true love was dead
    but i'm bound to fall
    bound to fall for you
    oh what can i do
  • Options
    MonaroMonaro Registered User regular
    edited August 2007
    That's awesome.

    I'd really like to benchmark my 8800GTX, as I'm having trouble with it, and before I try an RMA would like to reinstall Windows and compare synthetic and real-world benchmarks pre- and post-install.

    So how did you get that graph? Hit the FRAPS benchmark hotkey and it records it? Or is that graph an Excel job?

    Monaro on
    steam_sig.png
  • Options
    FreddyDFreddyD Registered User regular
    edited August 2007
    Monaro wrote: »
    That's awesome.

    I'd really like to benchmark my 8800GTX, as I'm having trouble with it, and before I try an RMA would like to reinstall Windows and compare synthetic and real-world benchmarks pre- and post-install.

    So how did you get that graph? Hit the FRAPS benchmark hotkey and it records it? Or is that graph an Excel job?
    Both. I hit the benchmark button in fraps, opened the file in Excel, and copied the results into a graph. The whole process took me about 10 minutes, and half of that was gameplay.

    I'm going to bench Battlefield 2 next, because the replay benchmarks in that game tell terrible lies. 32 player online matches are really resource intensive.

    FreddyD on
  • Options
    FreddyDFreddyD Registered User regular
    edited August 2007
    I'm updating this thread with my Bioshock benchmarks:

    BioshockBenchmark.png

    BioshockBenchmark2.png

    BioshockBenchmarkPic.jpg

    Exploration felt really smooth but combat felt somewhat jerky and disconnected. I don't think this was solely a framerate issue. I had the game running at the highest settings, and I was not disappointed with the visuals. I was satasified with the experience but I may turn down the AF to keep framerates as high as possible during combat. I don't know if AA works in this game, but I'll take a look at the config files.

    FreddyD on
  • Options
    cfgausscfgauss Registered User regular
    edited August 2007
    You do realize that without controlling your data, and in fact, having no idea where or how or under what situation it came from, it will mean even less than those fancy graphs you see in gaming magazines and websites (which already mean next to nothing)?

    cfgauss on
    The hero and protagonist, whose story the book follows, is the aptly-named Hiro Protagonist: "Last of the freelance hackers and Greatest sword fighter in the world." When Hiro loses his job as a pizza delivery driver for the Mafia, he meets a streetwise young girl nicknamed Y.T. (short for Yours Truly), who works as a skateboard "Kourier", and they decide to become partners in the intelligence business.
  • Options
    The_ScarabThe_Scarab Registered User regular
    edited August 2007
    Make sure you guys post basic system specs with your benchmarks. That includes OS, ram, gpu, cpu and anything else. Shit, just post the dxdiag main window.

    The_Scarab on
  • Options
    dispatch.odispatch.o Registered User regular
    edited August 2007
    FRAPS also rapes performance in a lot of situations, I wouldn't go so far as to call what you're doing benchmarking. In fact the very nature of benchmarks means they aren't real world.

    dispatch.o on
  • Options
    FreddyDFreddyD Registered User regular
    edited August 2007
    cfgauss wrote: »
    You do realize that without controlling your data, and in fact, having no idea where or how or under what situation it came from, it will mean even less than those fancy graphs you see in gaming magazines and websites (which already mean next to nothing)?
    Timedemo benchmarks are pretty much worthless for determining how a game will actually run, and most benchmarking websites don't even give you a description of how they got their data. I just want to give people an idea of the kind of real world performance they can expect on this type of hardware.

    So I guess I should give a disclaimer: This isn't a thread to celebrate graphs and numbers and compare them to other graphs and numbers. This thread is really meant for people with older CPUs who are on the fence about an upgrade. Is Bioshock enjoyable on this hardware? The answer is yes.

    So for the curious, the first benchmark was taken during the last battle of the demo, right after you exit the vita chamber. The second one was taken after the scene with the big daddy, as you explore the narrow corridors.

    FreddyD on
  • Options
    urahonkyurahonky Registered User regular
    edited August 2007
    Freddy, how do you make those graphs? I probably missed it in the OP...

    urahonky on
  • Options
    IrohIroh Registered User regular
    edited August 2007
    FreddyD wrote: »
    cfgauss wrote: »
    You do realize that without controlling your data, and in fact, having no idea where or how or under what situation it came from, it will mean even less than those fancy graphs you see in gaming magazines and websites (which already mean next to nothing)?
    Timedemo benchmarks are pretty much worthless for determining how a game will actually run, and most benchmarking websites don't even give you a description of how they got their data. I just want to give people an idea of the kind of real world performance they can expect on this type of hardware.

    So I guess I should give a disclaimer: This isn't a thread to celebrate graphs and numbers and compare them to other graphs and numbers. This thread is really meant for people with older CPUs who are on the fence about an upgrade. Is Bioshock enjoyable on this hardware? The answer is yes.

    So for the curious, the first benchmark was taken during the last battle of the demo, right after you exit the vita chamber. The second one was taken after the scene with the big daddy, as you explore the narrow corridors.

    This is exactly the line of thinking that people should have when choosing new hardware if they are on a budget.

    Not being in control of the data is what makes this testing more relevant than time demos. The idea is to throw all of the unexpected shit at the computer you possibly can, in order to stress it out a little and see if you experience any unplayable conditions.

    Where and how frequently you experience a framerate below 60/s or so is way more important than what your max or average is. Overall smoothness makes the game far more enjoyable than being able to beat off to your peak framerate.

    Iroh on
    [SIGPIC][/SIGPIC]
  • Options
    FreddyDFreddyD Registered User regular
    edited August 2007
    urahonky wrote: »
    Freddy, how do you make those graphs? I probably missed it in the OP...
    You hit the benchmarking key on FRAPs, and hit it again when you're done. You can find the data in the benchmark folder in your fraps install. Then you open the fps table in Excel, and click insert->chart->line..., copy the graph into a picture editor, and save it in your favorite format.

    FreddyD on
  • Options
    TxdoHawkTxdoHawk Registered User regular
    edited August 2007
    FPS benchmarks and 3dmark and so on get slammed pretty hard these days, but IMO real world benchmarks should absolutely not straight-up replace them outright. They primarily are used in hardware reviews that compares the hardware to other pieces in its class, and while real world benchmarks are great to get an actual idea of what the hardware can comfortably run, it's much easier to decipher a simple bar graph when you are looking at who reigns supreme in performance.

    TxdoHawk on
    TuxedoHawk.png
Sign In or Register to comment.