Awful Forum Load Times

13

Posts

  • godmodegodmode Southeast JapanRegistered User regular
    Also East coast here, and strangely that 17-second delay is right on the money. The loading indicator will spin on the tab for exactly 17 seconds and then the page loads all at once.

  • skeldareskeldare Gresham, ORRegistered User regular
    godmode wrote: »
    Also East coast here, and strangely that 17-second delay is right on the money. The loading indicator will spin on the tab for exactly 17 seconds and then the page loads all at once.

    Sounds pretty OCD to me.

    Nintendo Console Codes
    Switch (JeffConser): SW-3353-5433-5137 Wii U: Skeldare - 3DS: 1848-1663-9345
    PM Me if you add me!
    HAIL HYDRA
  • Sweeney TomSweeney Tom Registered User regular
    This is the worst it's been since it's started happening. Normally things have been slow for me for an hour every 8 PM to 9 PM on the clock (East Coast), but today it's been going on for a solid hour and a half.

  • DelphinidaesDelphinidaes FFXIV: Delphi Kisaragi Registered User regular
    Seemed to start a bit after 5pm PST today and still going as of this post. It's been this way every day around the same time the past few days noticeably worse today than yesterday although that could just be me not paying as close attention yesterday :(

    I counted about 19 seconds each time give or take a second or two.

    NNID: delphinidaes
    Official PA Forums FFXIV:ARR Free Company <GHOST> gitl.enjin.com Join us on Sargatanas!
    delphinidaes.png
  • abotkinabotkin Registered User regular
    I'm still experiencing slowness as of 10:08 CST, which is later than it had been lasting.

    Apologies if this ends up multi-posting, the Post Reply has been finicky during these periods of slowness.

    steam_sig.png
    3DS: 0963-0539-4405
  • DelphinidaesDelphinidaes FFXIV: Delphi Kisaragi Registered User regular
    Speed is now back to normal, seemed to last about 3 hours or so.

    NNID: delphinidaes
    Official PA Forums FFXIV:ARR Free Company <GHOST> gitl.enjin.com Join us on Sargatanas!
    delphinidaes.png
  • IcyLiquidIcyLiquid Two Steaks Montreal, QuebecAdministrator, Vanilla Staff vanilla
    taking-a-bow.jpg

    Banzai5150PaperLuigi44Sweeney TomEsseeAuralynxDasUberEdwardIrukaZilla360spool32Bucketman
  • kimekime Queen of Blades Registered User regular
    edited June 2013
    Guess what time it is!

    (8, aka slow forum times :P )

    Edit: huh. It only lasted like 3 minutes today. I'm assuming it was still the forums since it happened right at 8, but maybe it was actually something on my end?

    Either way, seems good now!

    kime on
    Battle.net ID: kime#1822
    3DS Friend Code: 3110-5393-4113
    Steam profile
    Auralynx
  • skeldareskeldare Gresham, ORRegistered User regular
    Seems to be occurring again today.

    Nintendo Console Codes
    Switch (JeffConser): SW-3353-5433-5137 Wii U: Skeldare - 3DS: 1848-1663-9345
    PM Me if you add me!
    HAIL HYDRA
    AeytherSpoitCambiata_J_curly haired boyEsseeJaysonFourDhalphirkimeOneAngryPossum
  • _J__J_ Pedant Registered User, __BANNED USERS regular
    Getting many 'Service Unavailable' messages.

    It seems to happen as I delve deeper into the forums. I can load forums.penny-arcade.com alright. Then G&T takes a long time to load. Then the Animal Crossing thread gives me Service Unavailable.

    chr1sh4ll3ttb3
  • AkilaeAkilae Registered User regular
    Still happening...

  • PailryderPailryder Registered User regular
    also i can see "quote but not the other options on posts unless i hover over the bottom part of a post

  • PailryderPailryder Registered User regular
    also also the bookmark ribbon is invisible to me? using chrome if that matters.

  • SmasherSmasher Starting to get dizzy Registered User regular
    Pailryder wrote: »
    also i can see "quote but not the other options on posts unless i hover over the bottom part of a post
    It's been show on hover for a while, though it should be (and is for me) the entire post rather than just the bottom.

    Geth
  • PailryderPailryder Registered User regular
    ah thanks, it probably just stood out when i was noticing the other issues, the bookmark ribbon is still a problem though.

  • AntimatterAntimatter Devo Was Right Gates of SteelRegistered User regular
    something just now was up

    Ruby RhodMarth
  • Ruby RhodRuby Rhod Multipass!Registered User regular
    Yeah, got some weird service unavailable when trying to load the forums.

    Tayrun
  • OneAngryPossumOneAngryPossum Registered User regular
    Not as bad as usual, but still doing some chugging for me just now. Not sure when it started, but it's 9:50 EST.

  • kimekime Queen of Blades Registered User regular
    Around 8:20 or so is when I first noticed.

    Battle.net ID: kime#1822
    3DS Friend Code: 3110-5393-4113
    Steam profile
  • SpoitSpoit *twitch twitch* Registered User regular
    I guess we'll see f it happens again I an hour or so

    steam_sig.png
  • I needed a gnome to post.I needed a gnome to post. i'm the bat guy (bluh)Registered User regular
    Yeah at this point I consider it not happening to be worthy of posting - I think it's pretty much 5 PM PST on schedule erryday right now.

    liEt3nH.png
    Essee
  • SpoitSpoit *twitch twitch* Registered User regular
    Well so far so good, it seems

    steam_sig.png
    QuidkimeEssee
  • IcyLiquidIcyLiquid Two Steaks Montreal, QuebecAdministrator, Vanilla Staff vanilla
    edited June 2013
    Yeah I fixed the issue earlier today. I feel kind of dumb about it, but also a bit like "wtf sphinx you crazy fucks". So a bit of a mix.
    I'm really sorry for the poor performance we've been delivering as a result of this debacle. It's even more annoying that this happened because we tried extremely hard to make it seamless and downtime free by deploying the infrastructure in segments. Anyways, most sincere of apologies.

    IcyLiquid on
  • kimekime Queen of Blades Registered User regular
    Glad it's fixed! Thanks, you're awesome!

    Is the problem something you could explain? I'm curious :P

    Battle.net ID: kime#1822
    3DS Friend Code: 3110-5393-4113
    Steam profile
  • IcyLiquidIcyLiquid Two Steaks Montreal, QuebecAdministrator, Vanilla Staff vanilla
    How technical are you? :p

  • SmasherSmasher Starting to get dizzy Registered User regular
    I'm sure I'm not the only one watching the thread who would be interested in as much detail/techspeak as you feel like going into.

    kimeEssee
  • IcyLiquidIcyLiquid Two Steaks Montreal, QuebecAdministrator, Vanilla Staff vanilla
    edited June 2013
    Basically when you perform a search, you're not really hitting the actual database. Once you have a certain amount of data in your DB you no longer want to be directly searching it. So we do periodic indexes of the data and searches hit that index instead, which lives on its own server. Once it returns some results, we join in the actual data from the database with a vastly more efficient query, much like rendering a single page of a discussion. The actual indexing operation got a lot more intense when we added Advanced Search though, so we had to make some changes to how we manage those indexes and how often we rebuild them. All of the search servers also had to be rebuilt to provide a newer version of indexer and search service because we were leveraging more cutting edge features. This is where the troubles began.

    We pretty much never want to be rebuilding from scratch anymore (absolutely murders the DB), so we start with a full index, then do progressive partial "delta" indexes of anything that's happened since that full index was run. Much less hurtey on the DB. Problem is that this delta index starts to become gradually slower and more expensive itself as the time-since-full-index begins to lengthen. So we periodically "merge" the delta onto the end of the full index and reset the time-we-last-full-indexed to now. This was working awesome, and gets us the benefit of a full index, minus the pain of actually doing it.

    So I wrote a daemon that runs in the background on all our search servers and does these indexing jobs as and when they're needed. No need to stuff around with crontab each time we add a customer, it all just happens on its own.

    Problem is, the indexing service itself apparently decided that every night at 8pm EST was an absolutely perfect time to quietly reindex everything. All indexes. Rebuild from scratch. Go. No mention of this automatic provision anywhere either. It was hidden behind a user on the machine with NO SHELL DEFINED and thus no LOGIN PERMISSION. So when I went trolling around to find out why this index job was happening mysteriously, I DID try to log into that user but was rebuffed, so I figured it was just a daemon user for the service itself, no need to screw around with it any further. WRONG.

    So that's that. Indexing service decided that a good non configurable install default would be: "seems like you should reindex everything every night at 8. yep."

    IcyLiquid on
    kimeAntimatterSeriouslyObiFettzerzhulSpoitZilla360EsseeEndT4CTRingoSmasherCCSSnicketysnickSporkAndrewAegisübergeekB_RDhalphirJoJoHoraHora38thDoeAeytherCambiataspool32bowenTofystedeth
  • StericaSterica Yes Registered User, Moderator mod
    I'm interpreting that as "We have staff wizards that conjure data whenever they get search requests."

    YL9WnCY.png
    kimeT4CTCasualB_RDhalphirCambiata
  • TubeTube Administrator, ClubPA admin
    I actually understood that. What the hell is wrong with me?

    Hobnail wrote: »
    This forum has taken everything from me
    This hurts but I deserve it

    zerzhulkimeJaysonFourDhalphirspool32Bucketman
  • IcyLiquidIcyLiquid Two Steaks Montreal, QuebecAdministrator, Vanilla Staff vanilla
    More like awesome explaining right?

    T4CTSnicketysnickagilemaniaübergeekDhalphirCambiataBucketmanTofystedeth
  • AthenorAthenor Battle Hardened Optimist The Skies of HiigaraRegistered User regular
    It really was a good explanation. And makes me wonder about how some of the databases we oversee could be improved.

    (I hate the fact that we don't really have a good DB admin on staff, but there's no reason for us to have one at our size.)

    pTWAqPb.png
    Looking to get rid of some 2005-2010 era Star Wars vehicles/toys. If interested, please PM me.
    He/Him
  • StericaSterica Yes Registered User, Moderator mod
    Tube wrote: »
    I actually understood that. What the hell is wrong with me?
    Years of Alphamonkey exposure.

    YL9WnCY.png
    zerzhulSpoitEsseeagilemaniaübergeekJaysonFourDhalphirbowenBucketmanTofystedeth
  • IcyLiquidIcyLiquid Two Steaks Montreal, QuebecAdministrator, Vanilla Staff vanilla
    Rorus Raz wrote: »
    Tube wrote: »
    I actually understood that. What the hell is wrong with me?
    Years of Alphamonkey exposure.

    There are creams to help with that these days.

    EsseeJaysonFourgodmodeDhalphirbowenBucketman
  • zerzhulzerzhul Registered User, Moderator mod
    Yeah but they cost a fortune and then we also need to get him an IcyLiquid exposure cream too. Where does the madness end?

  • übergeekübergeek Registered User regular
    zerzhul wrote: »
    Yeah but they cost a fortune and then we also need to get him an IcyLiquid exposure cream too. Where does the madness end?

    Is that like some reverse Icy Hot?

    camo_sig.png
  • Mr. ButtonsMr. Buttons Registered User regular
    edited August 2013
    Forum load times have gotten worse (for me at least) over the last couple days. Yesterday I got the "over capacity" page a few times (it was probably 3-4 times over the afternoon), and today it's just been taking a while for pages to load (I haven't had the "over capacity" page today).

    Wasn't paying enough attention to it yesterday to note the time that it happened, but today it's probably been going on for the last half hour or so.

    edit: and it's kind of hit or miss... now the load times seem to be back to normal

    Mr. Buttons on
    chr1sh4ll3ttb3CambiataRingoUEAKCrash
  • UEAKCrashUEAKCrash heh Registered User regular
    I seem to be bouncing back and forth between near-instant load times when clicking the forums button from the main page to 20+ second loads. Occasionally an "over capacity" page, even at off-peak hours.

    bowenTTODewback
  • BucketmanBucketman Call me SkraggRegistered User regular
    I've been getting slow loads and a lot of over capicity errors yesterday and today

  • BucketmanBucketman Call me SkraggRegistered User regular
    I've been getting slow loads and a lot of over capicity errors yesterday and today.
    Also "The "PostController" object does not have a "xMonitoring" method.|PostController|xMonitoring|"

    Mr. Buttons
  • BucketmanBucketman Call me SkraggRegistered User regular
    I've been getting slow loads and a lot of over capicity errors yesterday and today.
    Also "The "PostController" object does not have a "xMonitoring" method.|PostController|xMonitoring|"

    FyndirCambiata
This discussion has been closed.