As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/
Options

[Sysadmin] Nightmare fuel

19394959698

Posts

  • Options
    CogCog What'd you expect? Registered User regular
    Sales sends me a ticket:

    "Hey, <client> is interested in adding some data loss prevention and peripheral filtering technology to their system. Could you give them a call and discuss some options with them?"




    Isn't that...

    Isn't that your fucking job?

  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    a5ehren wrote: »
    The good mesh solutions all cost more than I want to spend on this.

    I used to think this, but when I actually sat down and thought about it, wifi is easily the most important infrastructure I have where I live (after the *actual* essentials, like power and running water).

    I decided that spending the money on a good mesh router was worth it, did it almost a year ago, and literally haven't thought about it since.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • Options
    chamberlainchamberlain Registered User regular
    Cog wrote: »
    Sales sends me a ticket:

    "Hey, <client> is interested in adding some data loss prevention and peripheral filtering technology to their system. Could you give them a call and discuss some options with them?"




    Isn't that...

    Isn't that your fucking job?

    Answer: Do I get the commission?

  • Options
    ThawmusThawmus +Jackface Registered User regular
    If you build yourself a cheap RADIUS server, and your AP's are capable, you could just be using Enterprise WiFi at home and your handoff issue would probably disappear on you. Use the same SSID on all AP's but use non-interfering frequencies, and you just stay authenticated via RADIUS during the transitions.

    Just take a shitty computer and put freeRADIUS on it, build a small text file with some usernames for you, your wife, and guests, point your access points to it, done. If it doesn't work, all it's cost you is time.

    Twitch: Thawmus83
  • Options
    bowenbowen How you doin'? Registered User regular
    That doesn't seem like it'd solve the problem where one access point is still technically "in range" but performs poorly compared to others that are higher power using the same SSID.

    That's where mesh-based systems shine.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    Oh my god bowen what happened to your face?

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • Options
    CogCog What'd you expect? Registered User regular
    wunderbar wrote: »
    Oh my god bowen what happened to your face?

    This is some forum in-joke that we are not in on. I assume it spawned in SE.

  • Options
    bowenbowen How you doin'? Registered User regular
    what're you talking about my face has always been like this

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    ThawmusThawmus +Jackface Registered User regular
    bowen wrote: »
    That doesn't seem like it'd solve the problem where one access point is still technically "in range" but performs poorly compared to others that are higher power using the same SSID.

    That's where mesh-based systems shine.

    WPA2 Enterprise does indeed make handoffs on modern client devices nigh-invisible. I spent months testing different methodologies before doing my mass deployment of UniFi gear, and WPA2 Enterprise made a huge difference in exactly the scenario you are describing.

    And I do emphasis that "modern" because if you're using old client devices it doesn't matter what you put up. Whether a client disconnects from an AP is largely dependent upon the client and what its threshold is set to (yes, this is an actual thing, and in rare cases you can even fiddle with this thing!). You can put up the nicest AP's in the world but if your client device sucks and thinks a -81 dbm is workable, you're still fucked.

    I also don't think people understand what "mesh" is actually designed for. It's not for fast roaming, it's for quick deployment to a wide area without the need for more than one Point of Presence (PoP). For example, if I wanted to deploy WiFi to a small neighborhood, I'd get on a tower overlooking that neighborhood, and then deploy the mesh CPE's to each home. The CPE's would talk to each other, and a few would talk directly to my tower. This saves me from having to point them all to my tower (where many won't have line of sight because everyone fucking loves trees).

    Some people do use mesh to distribute WiFI to a fairground or other large facility/area where they offer general WiFi, and they do use fast roaming, but this is not unique to mesh devices, it's just a part of 802.11r that's come along for the ride, because you spent money on something that was made after 2007 (which is not something most people on your street can say!). Mesh is nice in a home because you can have multiple repeaters talking to each other, which strengthens the signal that those repeaters get (because as you keep repeating the signal, you get degradation). But again, if your issue is roaming from one AP to another, it's not the solution. It's a good solution, but it's not the solution.

    1. Are your client devices super old?
    2. Use WPA2 Enterprise
    3. Use better access points
    4. If you have the money and a big enough home, get mesh access points

    In that order.

    Twitch: Thawmus83
  • Options
    bowenbowen How you doin'? Registered User regular
    WPA2 enterprise is just not feasible, sorry.

    No one's going to set up VMs and radius servers in their home. We might as tech guys, but even I don't want to fucking maintain that garbage, and I've done it before. And yeah, super old devices or just shitty unconfigurable devices are stupidly common (like TVs or gaming devices). The reason why, though mesh is at the bottom of your list, but it's at the top of mine, is because I literally have to put about 0 thought into it.

    And when I'm at home, the less thought I have to put into shit the better. That's why I have money.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    bowenbowen How you doin'? Registered User regular
    I mean I sat down one day and was like "yeah I should run some cat6 and... like.. or I could just get a powerline adapter" so that's what I did instead.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    twmjrtwmjr Registered User regular
    I think it's important to throw out that WPA2 Enterprise isn't in and of itself a solution to faster roaming. The faster roaming is predicated on your access points being centrally controlled and/or communicating with one another. If you have two stand-alone access points, the client is still going to have to go through the entire reauthentication process when it switches APs. I'm assuming things like UniFi support various protocols that speed up roaming, but a lot of range extenders/etc. aren't going to (if they even support enterprise to begin with).

  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    twmjr wrote: »
    I think it's important to throw out that WPA2 Enterprise isn't in and of itself a solution to faster roaming. The faster roaming is predicated on your access points being centrally controlled and/or communicating with one another. If you have two stand-alone access points, the client is still going to have to go through the entire reauthentication process when it switches APs. I'm assuming things like UniFi support various protocols that speed up roaming, but a lot of range extenders/etc. aren't going to (if they even support enterprise to begin with).

    I've never done an analysis but my understanding is that WPA2-Personal is faster for roaming than WPA2-Enterprise (without 802.11r) because the entire authentication handshake for WPA2-Enterprise takes longer.

    Enabling 802.11r on your WPA2-Enterprise network allows the auth to be passed from AP to AP which makes the roam times roughly the same as WPA2-Personal.

    But I'm happy to be corrected if I'm wrong.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    bowen wrote: »
    No one's going to set up VMs and radius servers in their home.

    I have a VM host and a RADIUS server in my home.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    bowenbowen How you doin'? Registered User regular
    yeah but you're weird feral

    change my mind

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    I don't use the RADIUS server for the household wifi though

    it's just in my test lab

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    twmjrtwmjr Registered User regular
    Feral wrote: »
    twmjr wrote: »
    I think it's important to throw out that WPA2 Enterprise isn't in and of itself a solution to faster roaming. The faster roaming is predicated on your access points being centrally controlled and/or communicating with one another. If you have two stand-alone access points, the client is still going to have to go through the entire reauthentication process when it switches APs. I'm assuming things like UniFi support various protocols that speed up roaming, but a lot of range extenders/etc. aren't going to (if they even support enterprise to begin with).

    I've never done an analysis but my understanding is that WPA2-Personal is faster for roaming than WPA2-Enterprise (without 802.11r) because the entire authentication handshake for WPA2-Enterprise takes longer.

    Enabling 802.11r on your WPA2-Enterprise network allows the auth to be passed from AP to AP which makes the roam times roughly the same as WPA2-Personal.

    But I'm happy to be corrected if I'm wrong.

    No, I'm pretty sure you're right and why I brought it up. The advantage of WPA2-Ent is it enables things like 802.11r (or proprietary solutions ala CCKM). Without them, the full RADIUS handshake should take longer (though possibly be relatively imperceptible to the user).

  • Options
    ThawmusThawmus +Jackface Registered User regular
    twmjr wrote: »
    Feral wrote: »
    twmjr wrote: »
    I think it's important to throw out that WPA2 Enterprise isn't in and of itself a solution to faster roaming. The faster roaming is predicated on your access points being centrally controlled and/or communicating with one another. If you have two stand-alone access points, the client is still going to have to go through the entire reauthentication process when it switches APs. I'm assuming things like UniFi support various protocols that speed up roaming, but a lot of range extenders/etc. aren't going to (if they even support enterprise to begin with).

    I've never done an analysis but my understanding is that WPA2-Personal is faster for roaming than WPA2-Enterprise (without 802.11r) because the entire authentication handshake for WPA2-Enterprise takes longer.

    Enabling 802.11r on your WPA2-Enterprise network allows the auth to be passed from AP to AP which makes the roam times roughly the same as WPA2-Personal.

    But I'm happy to be corrected if I'm wrong.

    No, I'm pretty sure you're right and why I brought it up. The advantage of WPA2-Ent is it enables things like 802.11r (or proprietary solutions ala CCKM). Without them, the full RADIUS handshake should take longer (though possibly be relatively imperceptible to the user).

    It's kinda dependant upon the hardware you're using (both the AP and the RADIUS server), how much latency there is between the AP and RADIUS server, etc., but generally, yes.

    Most radios have very little CPU or memory and some can be downright terrible at the auth process, so handing that shit off can improve that process in some rare cases. Like I said, I spent months testing WPA2 Personal vs. WPA2 Enterprise with several devices before I did my Unifi rollout, and the handoffs were faster for me with Enterprise (the sole exception being some handheld barcode scanners who are using a wpa_supplicant from the fucking stone age and couldn't be updated. It wouldn't roam on Enterprise, period. So I had to add an SSID and VLAN just to cater to them. Still annoys me.). But I'll grant that was probably because I was using Unifi and their version of 802.11r.

    Kinda the same reason I use queues on my Mikrotik routers and not use Traffic Shaping on my UBNT CPE's. Eventually asking radios to do anything but "be a bridge" tends to fuck with your airtime (and I don't like it when things fuck with my airtime), but I'll grant that I get pretty unnecessarily neurotic about that shit, and sometimes the gains aren't appreciable at all. I still spend a lot of time re-configuring firewall rulesets because they're not....100%....efficient....this rule....has.......zero.....packets....motherfucker!

    Twitch: Thawmus83
  • Options
    twmjrtwmjr Registered User regular
    handheld scanners are the printers of the wireless world

  • Options
    bowenbowen How you doin'? Registered User regular
    edited June 2018
    I remember when UPS upgraded their scanners from this old piece of early 90s tech to a new weird "BLUETOOTH DONGLE" madness.

    Like they had little finger scanners that used to be hard wired to the actual scanner device themselves and decided "no that's dumb let's make it bluetooth instead" and in a building with thousands of these little things moving around and 2-3 people in a truck you can imagine how well that went. (it didn't go well at all)

    It's weird that they didn't do the obvious things that would have made the situation better entirely. The entire point of the loaders was to make sure packages didn't get loaded wrong. So you log into a truck and scan packages... why not have a set list of accepted zip codes and have the scanner beep and warn the loader when an invalid package made it through instead? This was 2006ish btw, so that kind of tech wouldn't have been new at all.

    Optimizing and modernizing all the wrong things, so dumb. It wouldn't surprise me if they still haven't improved that.

    bowen on
    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    ThawmusThawmus +Jackface Registered User regular
    I've never had a lot of success with bluetooth. It flatly doesn't work in my home at all.

    Twitch: Thawmus83
  • Options
    SeidkonaSeidkona Had an upgrade Registered User regular
    Thawmus wrote: »
    I've never had a lot of success with bluetooth. It flatly doesn't work in my home at all.

    You've also been around so many radio waves you probably emit interference as a mutant super power.

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
  • Options
    ThawmusThawmus +Jackface Registered User regular
    Entaru wrote: »
    Thawmus wrote: »
    I've never had a lot of success with bluetooth. It flatly doesn't work in my home at all.

    You've also been around so many radio waves you probably emit interference as a mutant super power.

    I mean, I'm probably just super fucking sterile at this point. I think that's the only super power I'm getting.

    Ladies.

    Twitch: Thawmus83
  • Options
    SeidkonaSeidkona Had an upgrade Registered User regular
    Until the AI hive mind that is using low level radio to communicate internally in a mesh network comes to destroy us.

    Then you will be our only hope.

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
  • Options
    bowenbowen How you doin'? Registered User regular
    Entaru wrote: »
    Until the AI hive mind that is using low level radio to communicate internally in a mesh network comes to destroy us.

    Then you will be our only hope.

    Yes, I, too, saw The Quiet Place

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    LD50LD50 Registered User regular
    I assume the hive mind using low level radio is just a bunch of IoT devices running compromised firmware.

  • Options
    ThawmusThawmus +Jackface Registered User regular
    edited June 2018
    LD50 wrote: »
    I assume the hive mind using low level radio is just a bunch of IoT devices running compromised firmware.

    I mean, is there another kind of IoT device?

    Thawmus on
    Twitch: Thawmus83
  • Options
    LD50LD50 Registered User regular
    Thawmus wrote: »
    LD50 wrote: »
    I assume the hive mind using low level radio is just a bunch of IoT devices running compromised firmware.

    I mean, is there another kind of IoT device?

    Can we even know?

  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Thawmus wrote: »
    twmjr wrote: »
    Feral wrote: »
    twmjr wrote: »
    I think it's important to throw out that WPA2 Enterprise isn't in and of itself a solution to faster roaming. The faster roaming is predicated on your access points being centrally controlled and/or communicating with one another. If you have two stand-alone access points, the client is still going to have to go through the entire reauthentication process when it switches APs. I'm assuming things like UniFi support various protocols that speed up roaming, but a lot of range extenders/etc. aren't going to (if they even support enterprise to begin with).

    I've never done an analysis but my understanding is that WPA2-Personal is faster for roaming than WPA2-Enterprise (without 802.11r) because the entire authentication handshake for WPA2-Enterprise takes longer.

    Enabling 802.11r on your WPA2-Enterprise network allows the auth to be passed from AP to AP which makes the roam times roughly the same as WPA2-Personal.

    But I'm happy to be corrected if I'm wrong.

    No, I'm pretty sure you're right and why I brought it up. The advantage of WPA2-Ent is it enables things like 802.11r (or proprietary solutions ala CCKM). Without them, the full RADIUS handshake should take longer (though possibly be relatively imperceptible to the user).

    It's kinda dependant upon the hardware you're using (both the AP and the RADIUS server), how much latency there is between the AP and RADIUS server, etc., but generally, yes.

    Most radios have very little CPU or memory and some can be downright terrible at the auth process, so handing that shit off can improve that process in some rare cases. Like I said, I spent months testing WPA2 Personal vs. WPA2 Enterprise with several devices before I did my Unifi rollout, and the handoffs were faster for me with Enterprise (the sole exception being some handheld barcode scanners who are using a wpa_supplicant from the fucking stone age and couldn't be updated. It wouldn't roam on Enterprise, period. So I had to add an SSID and VLAN just to cater to them. Still annoys me.). But I'll grant that was probably because I was using Unifi and their version of 802.11r.

    Kinda the same reason I use queues on my Mikrotik routers and not use Traffic Shaping on my UBNT CPE's. Eventually asking radios to do anything but "be a bridge" tends to fuck with your airtime (and I don't like it when things fuck with my airtime), but I'll grant that I get pretty unnecessarily neurotic about that shit, and sometimes the gains aren't appreciable at all. I still spend a lot of time re-configuring firewall rulesets because they're not....100%....efficient....this rule....has.......zero.....packets....motherfucker!

    I sincerely appreciate that you backed this up with real world experiments. :+1:

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    MyiagrosMyiagros Registered User regular
    Any Veeam experts among us? I took over a site with Veeam configured and of course one of the VMs hasn't completed a backup in 2.5 months. It is an SQL server running 3 instances and about 10 databases.

    I am getting this error: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. --tr:Failed to verify writers state. --tr:Failed to perform pre-backup tasks.)

    - Replication is scheduled to run between 6am-7pm. VM backup runs at 10pm. I was able to get replication to complete by disabling application processing but that's not ideal, previously it was set to "Perform copy only". The VM backup is set to "Process transaction logs".
    - Resetting the VSS writers or rebooting the server doesn't help.
    - I can manually run an SQL backup from the server no problem.

    One thing I noticed which might be the problem is the Recovery Model, some are set to Full, some are set to Simple, any chance this might prevent the job from running?

    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
  • Options
    LD50LD50 Registered User regular
    edited June 2018
    It would be a lot easier to diagnose the problem if the error message was helpful enough to specify which pre-backup tasks failed.

    You might want to try to check the event viewer logs and see if there are any leads there.

    Edit: Oh, if you run "vssadmin list writers", does it return any results?

    LD50 on
  • Options
    MyiagrosMyiagros Registered User regular
    About 30% of the writers go into time out, another server had this issue as well and a reboot of the host fixed it, not on this one though.

    Event Log errors:
    BACKUP failed to complete the command BACKUP DATABASE master. Check the backup application log for detailed messages.
    and
    Error message: A nonrecoverable I/O error occurred on file "{9BFFE498-817D-432B-A856-4B3593A8EF29}1:" 995(The I/O operation has been aborted because of either a thread exit or an application request.).

    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
  • Options
    LD50LD50 Registered User regular
    edited June 2018
    It's been a long time since I've dealt with VSS, but IIRC via 'application aware processing', a problem with the SQL server itself can prevent the backup from executing. That would be my first guess from the "or an application request" portion of the error message. Doubly so if that file is a transaction log.

    I don't think I know enough to help beyond that. VSS was very finicky when I was using it, and commonly the only fix we could find was to create a brand new VM and migrate whatever we had running over to it (and I understand that may not be a viable option for you).

    LD50 on
  • Options
    taliosfalcontaliosfalcon Registered User regular
    It's astounding to me that backups are still the bane of my existance, as they were when i started working in IT ~12 years ago. It's like no company has improved them at all and no matter what the solution is I still have to spend one or two entire fucking days a month troubleshooting. "What do you mean you can't quiesce virtual machine machine? it's fucking fine, I CAN DO IT FROM VSPHERE
    WHICH YOU CONNECT TO AND USE ! ARHGUIHDFISUHDFIOSUHDFSIUFGH FFFFUUUU"



    giphy.gif

    steam xbox - adeptpenguin
  • Options
    LD50LD50 Registered User regular
    It's astounding to me that backups are still the bane of my existance, as they were when i started working in IT ~12 years ago. It's like no company has improved them at all and no matter what the solution is I still have to spend one or two entire fucking days a month troubleshooting. "What do you mean you can't quiesce virtual machine machine? it's fucking fine, I CAN DO IT FROM VSPHERE
    WHICH YOU CONNECT TO AND USE ! ARHGUIHDFISUHDFIOSUHDFSIUFGH FFFFUUUU"



    giphy.gif

    Yeah, how shit backup software is is a regular thread topic here.

    I have actually written my own backup software because off the shelf solutions are just that fucking bad. That's insane. In no universe am I qualified to be writing a backup application from scratch, but my ruby script+task scheduler was more capable and reliable than anything else I could find at the time.

  • Options
    LD50LD50 Registered User regular
    edited June 2018
    Also, as bad as VSS is, at least it's not the shitshow that was ntbackup!

    LD50 on
  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    I've literally been fighting with my backups *all week*

    Just this morning I had to create a copy of one of my backup jobs from scratch because the existing one just.... doesn't work anymore. And errors when I try to do anything in the job. seriously I can go into the job properties. If I do not change anything, and just click ok to go out, it errors. If I click cancel to go out it doesn't error. And the job just gets stuck to the point where I can't even restart the services and had to reboot the box.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • Options
    MyiagrosMyiagros Registered User regular
    This error I have, I actually had on another server up until last Friday. Exact same error except there was no SQL on that system. I ran all updates and rebooted both servers three nights in a row last week, the backups never worked. Friday night I finally restarted the host, the one server now backs up properly, the SQL one still has the problem. Veeam support is basically useless as well because they just point me to things I've already gone through multiple times.

    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
  • Options
    LD50LD50 Registered User regular
    Myiagros wrote: »
    This error I have, I actually had on another server up until last Friday. Exact same error except there was no SQL on that system. I ran all updates and rebooted both servers three nights in a row last week, the backups never worked. Friday night I finally restarted the host, the one server now backs up properly, the SQL one still has the problem. Veeam support is basically useless as well because they just point me to things I've already gone through multiple times.

    Part of the problem is that the best way to figure out why VSS isn't working is a seance or a ritual involving 12 virgins and a goat.

  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    LD50 wrote: »
    Myiagros wrote: »
    This error I have, I actually had on another server up until last Friday. Exact same error except there was no SQL on that system. I ran all updates and rebooted both servers three nights in a row last week, the backups never worked. Friday night I finally restarted the host, the one server now backs up properly, the SQL one still has the problem. Veeam support is basically useless as well because they just point me to things I've already gone through multiple times.

    Part of the problem is that the best way to figure out why VSS isn't working is a seance or a ritual involving 12 virgins and a goat.

    A goat!

    That's what I've been missing. I had been using a sheep.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
This discussion has been closed.