The image size limit has been raised to 1mb! Anything larger than that should be linked to. This is a HARD limit, please do not abuse it.
Our new Indie Games subforum is now open for business in G&T. Go and check it out, you might land a code for a free game. If you're developing an indie game and want to post about it, follow these directions. If you don't, he'll break your legs! Hahaha! Seriously though.
Our rules have been updated and given their own forum. Go and look at them! They are nice, and there may be new ones that you didn't know about! Hooray for rules! Hooray for The System! Hooray for Conforming!

I need your opinions... FOR SCIENCE [I have enough responses now, thanks!]

Pure DinPure Din Boston-areaRegistered User regular
edited March 2012 in Help / Advice Forum
[posted with mod permission]

This is for a project my friend and I are working on in grad school, comparing two different algorithms for summarizing documents. We're trying to find out which kind of algorithm will help us make better systems, so maybe one day in the future if you don't want to read something, you can just have one of our programs do all the work for you. :) You can also get entered to win a $25 gift certificate.

Here is the link: [study over, sorry!]

Like I wrote on the first page of the survey, you can definitely skip some of the summary pairs if you feel like doing so. The entire survey takes at least an hour to do, but each set of summary evaluations only takes about five minutes and even if you just do one or two it would be a big help.

Also I'd appreciate any ideas on how to better ask people how to compare these types of things, like are there better questions we should be asking, or how to display the summaries so that it's easier to read and compare them quickly. My sense of design is absolutely terrible, so I'm always interested in hearing from other people how to make things easier to use and look at.

Pure Din on


  • k-mapsk-maps I wish I could find the Karnaugh map for love. 2^<3Registered User regular
    I think it might be better if you limit it to 1-2 summaries per person, make the passages about half the size, and maybe let the user choose the topic she/he is more interested in. This is more work than I put in into the Verbal/Writing sections of the GRE...It's actually harder because it involves critically comparing three different sets of text. I know you say so in your post, but if you're giving this survey to a bunch of people it would be better if it was implied by the design. It would also be better if the passages were shown side-by-side so you wouldn't have to scroll up and down to compare them. Better yet, color code the passages with respect to content if there is a straight-forward way of doing so.

    At any rate, I'm also going to grad school for NLP, so I'd be interested to know what your results turn out to be!

  • Pure DinPure Din Boston-areaRegistered User regular
    We can't really change the length of the summaries, we're using a dataset and a task that is fairly standard. I can see what you're saying about limiting the summaries per person though -- we certainly don't want to push people away by the length of the survey but from our perspective it would be better to get a smaller number of people to do a larger number of comparisons because it would make the evaluations more consistent.

    I can tell you more about the task and the experiment via PM if you want... I'm interested in hearing more about your grad school / research plans.

  • Fuzzy Cumulonimbus CloudFuzzy Cumulonimbus Cloud Registered User regular
    TBH you are going to get confounding results because people will get bored and just randomly clicked. I made it through about three or four before it got so boring that I just started clicking based on which paragraph looked more pleasing on the whole. Survey fatigue ruins surveys and as a researcher, you need to account for that. I would also include a view timer of some sort to monitor whether people are actually reading them or not.

Sign In or Register to comment.