As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/

[Programming] Kafkaesque rabbits in the queue at the pub

16364666869100

Posts

  • LD50LD50 Registered User regular
    I'm not sure I'm answering the right question, but the insertion is still happening because that is what is causing the integrity error in the first place. (The only reason why there is something to catch is because the insertion happened). The rollback happens as part of the transaction because the insertion failed (the 'transaction' guarantees that an action either 100% succeeds or doesn't happen at all; when the insert fails the rollback is triggered automatically to undo anything that might have only been partially applied).

    You might want to look into limiting your api queries. How many queries are you allowed inside of 15 minutes?

  • OghulkOghulk Tinychat Janitor TinychatRegistered User regular
    LD50 wrote: »
    I'm not sure I'm answering the right question, but the insertion is still happening because that is what is causing the integrity error in the first place. (The only reason why there is something to catch is because the insertion happened). The rollback happens as part of the transaction because the insertion failed (the 'transaction' guarantees that an action either 100% succeeds or doesn't happen at all; when the insert fails the rollback is triggered automatically to undo anything that might have only been partially applied).

    You might want to look into limiting your api queries. How many queries are you allowed inside of 15 minutes?

    180 I think. What's something I can use to check the 15 minutes?

  • LD50LD50 Registered User regular
    Wait 5 seconds between each request :razz:

  • LD50LD50 Registered User regular
    Maybe a better question to ask is "is there a better way to request the data you're asking for?". I don't know what information you're requesting and I don't know the twitter api, but there might be a way to get everything you want with a single request.

  • KetBraKetBra Dressed Ridiculously Registered User regular
    If you want to check time differences you can use datetime, specifically the timedelta class

    But yeah, maybe structuring your api queries better might yield better results overall.

    KGMvDLc.jpg?1
  • OghulkOghulk Tinychat Janitor TinychatRegistered User regular
    I'm requesting a twitter user's metadata and inputting that into a MySQL database. The twitter api doesn't really do anything Yeah I figured out a better way

  • OrcaOrca Also known as Espressosaurus WrexRegistered User regular
    Or...start with a simulator that plays back a recording of a previous transaction? Testing on a live site that rate-limits seems like an exercise in frustration for debugging.

  • OghulkOghulk Tinychat Janitor TinychatRegistered User regular
    Orca wrote: »
    Or...start with a simulator that plays back a recording of a previous transaction? Testing on a live site that rate-limits seems like an exercise in frustration for debugging.

    How would I go about doing this?

    Also turns out that the old way I was doing rate limited it to 900 requests in a 15 minute time span.

    The new one does 15 requests in a 15 minute time span.

    Old one was 900 users/15 minutes, whereas new one is 3000 users/15 minute time span.

    Which would be fine but I'm looking at user counts far above that 3000 limit so here I am at the same problem.

  • OrcaOrca Also known as Espressosaurus WrexRegistered User regular
    Oghulk wrote: »
    Orca wrote: »
    Or...start with a simulator that plays back a recording of a previous transaction? Testing on a live site that rate-limits seems like an exercise in frustration for debugging.

    How would I go about doing this?

    Also turns out that the old way I was doing rate limited it to 900 requests in a 15 minute time span.

    The new one does 15 requests in a 15 minute time span.

    Old one was 900 users/15 minutes, whereas new one is 3000 users/15 minute time span.

    Which would be fine but I'm looking at user counts far above that 3000 limit so here I am at the same problem.

    Wireshark, and then implement the other side of what you see. I'm assuming it's all just HTTP requests, so easy peasy to implement a server.

  • YoshuaYoshua Registered User regular
    Thanks for the sql pointers -- I'll give them a look. The code is talking to MS SQL server, with bits of entity framework in some places -- there's nothing very complicated going on, I think, but I want to make sure I have the basics of what "join" (etc) means clear in my head, rather than trying to form a mental picture just by looking at existing queries and thinking about what they do.

    Maybe this diagram will help you visualize it better: SQL Join Diagram

    Basically it is a way to combine columns from one or more tables. Since this is done with one query, it is more efficient than running multiple queries to get the same information (also it is a lot easier to read).

  • PhyphorPhyphor Building Planet Busters Tasting FruitRegistered User regular
    Orca wrote: »
    Oghulk wrote: »
    Orca wrote: »
    Or...start with a simulator that plays back a recording of a previous transaction? Testing on a live site that rate-limits seems like an exercise in frustration for debugging.

    How would I go about doing this?

    Also turns out that the old way I was doing rate limited it to 900 requests in a 15 minute time span.

    The new one does 15 requests in a 15 minute time span.

    Old one was 900 users/15 minutes, whereas new one is 3000 users/15 minute time span.

    Which would be fine but I'm looking at user counts far above that 3000 limit so here I am at the same problem.

    Wireshark, and then implement the other side of what you see. I'm assuming it's all just HTTP requests, so easy peasy to implement a server.

    Twitter is entirely HTTPS so simple replay is impossible

  • OrcaOrca Also known as Espressosaurus WrexRegistered User regular
    Then it's time to MITM yourself! But I guess that's a bit more complicated.

  • SporkAndrewSporkAndrew Registered User, ClubPA regular
    Orca wrote: »
    Then it's time to MITM yourself! But I guess that's a bit more complicated.

    You can use something like fiddler to intercept the calls to the Twitter API and then return something that you've created yourself.

    Usually what I'd do is be lazy and take a print_r of the return from the API and then turn that into a variable. Then the function to read from Twitter would just return that

    The one about the fucking space hairdresser and the cowboy. He's got a tinfoil pal and a pedal bin
  • ecco the dolphinecco the dolphin Registered User regular
    edited March 2017
    Ecco Bitch Time

    If you don't know about atomic accesses, nor about memory barriers, then you are not qualified to write a lock free class. If you can't map in your mind how two threads can interact with your class in separate ways, then you are very much not qualified to write a lock free class.

    Especially when you've designed it so that n seperate threads can access the class without even a single mutex in sight.

    Ecco rage rising. Urge to swear increasing.

    Flipping off mode engaged.

    Edit: I may have over estimated them. Perhaps without knowledge of mutexes, and their use of global variables instead of member variables, they are indicating that they are unsuited to be writing multi threaded code in the first place, let alone lock free multi threaded code.

    ecco the dolphin on
    Penny Arcade Developers at PADev.net.
  • DelzhandDelzhand Hard to miss. Registered User regular
    Orca wrote: »
    Then it's time to MITM yourself! But I guess that's a bit more complicated.

    You can use something like fiddler to intercept the calls to the Twitter API and then return something that you've created yourself.

    Usually what I'd do is be lazy and take a print_r of the return from the API and then turn that into a variable. Then the function to read from Twitter would just return that

    yeah my general practice when engaging with APIs is to structure my functions to take an interface, then write an abstraction layer that converts actual API results to that interface. It gets harder the more complex the responses are, but the upside is that even if the third party changes the way they return the data, all you ever need to do is change your abstraction.

    If you've done that, then you can make a mock that implements your interface, and you're all set to test, because you've divorced your logic from the API.

  • KakodaimonosKakodaimonos Code fondler Helping the 1% get richerRegistered User regular
    Ecco Bitch Time

    If you don't know about atomic accesses, nor about memory barriers, then you are not qualified to write a lock free class. If you can't map in your mind how two threads can interact with your class in separate ways, then you are very much not qualified to write a lock free class.

    Especially when you've designed it so that n seperate threads can access the class without even a single mutex in sight.

    Ecco rage rising. Urge to swear increasing.

    Flipping off mode engaged.

    Edit: I may have over estimated them. Perhaps without knowledge of mutexes, and their use of global variables instead of member variables, they are indicating that they are unsuited to be writing multi threaded code in the first place, let alone lock free multi threaded code.

    I always like to refer to code like that as "optimistic multithreading".

  • CampyCampy Registered User regular
    edited March 2017
    While I've never written any professional threaded code, I at least know enough about it that the concepts make sense to me. I also know that I definitely don't know it well enough to write a safe class without some help.

    Sounds like this guy/gal has never even read about threading!

    Campy on
  • KakodaimonosKakodaimonos Code fondler Helping the 1% get richerRegistered User regular
    Bah, Ecco, you're bad luck. Right after I reply to your post I walk in this morning to a pile of crap that had to fixed and pushed asap because someone put too many locks in our market feed handlers.

  • EchoEcho ski-bap ba-dapModerator mod
    My IRC server experiment started as an exercise to learn some TCP communication stuff and quickly escalated to messing with threading. Fun stuff.

  • bowenbowen How you doin'? Registered User regular
    Once you start with C# it's hard to use anything else ever again.

    Did you install VS2017? It's great. Much snappier than 2015.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • EchoEcho ski-bap ba-dapModerator mod
    bowen wrote: »
    Did you install VS2017? It's great. Much snappier than 2015.

    On that note, new jorb today!

    Doing the "set up work environment on brand new system" routine right now and trying to figure out what's preventing me from installing VS. :rotate:

    Maybe I need more updates. Git working, Windows Update.

  • EchoEcho ski-bap ba-dapModerator mod
    edited March 2017
    Return code: -2147024864
    Return code details: The process cannot access the file 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Automation\dte90a.olb' because it is being used by another process.

    Brand new computer. What the heck is that about?

    Echo on
  • OghulkOghulk Tinychat Janitor TinychatRegistered User regular
    Echo wrote: »
    Return code: -2147024864
    Return code details: The process cannot access the file 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Automation\dte90a.olb' because it is being used by another process.

    Brand new computer. What the heck is that about?

    There's already a copy of VS17 on the computer? It's still in the process of installing and bugged up? Is it a VM or a isolated environment?

  • EchoEcho ski-bap ba-dapModerator mod
    Oghulk wrote: »
    There's already a copy of VS17 on the computer? It's still in the process of installing and bugged up? Is it a VM or a isolated environment?

    Nope, fresh Windows install. Failed like that on the first installation attempt.

    Just got done with another batch of Windows updates and deleted the whole target folder (though it was empty), let's see what happens this time.

  • EtheaEthea Registered User regular
    Echo wrote: »
    bowen wrote: »
    Did you install VS2017? It's great. Much snappier than 2015.

    On that note, new jorb today!

    Doing the "set up work environment on brand new system" routine right now and trying to figure out what's preventing me from installing VS. :rotate:

    Maybe I need more updates. Git working, Windows Update.

    IIRC the VS2017 RC installer required the VS2015 runtime to be already installed. Was surprising to me when setting a clean machine a couple of months ago.

  • EchoEcho ski-bap ba-dapModerator mod
    edited March 2017
    Ethea wrote: »
    IIRC the VS2017 RC installer required the VS2015 runtime to be already installed. Was surprising to me when setting a clean machine a couple of months ago.

    This is the released version, so I dunno.

    Also, same message, different file.
    Return code: -2147024864
    Return code details: The process cannot access the file 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Automation\msenv140p.dll' because it is being used by another process.

    Target folder didn't exist before starting the installer so :rotate:

    Echo on
  • CampyCampy Registered User regular
    Does the sysadmin there have applications setup to install automatically on user accounts or anything like that?

  • urahonkyurahonky Registered User regular
    Oh Microsoft. You'd think they'd be able to get their own installers together on their own OS without running into issues like that.

  • EchoEcho ski-bap ba-dapModerator mod
    Some googling blames virus protection and it did come with McAfee malware pre-installed...

  • EchoEcho ski-bap ba-dapModerator mod
    Turned the real-time scanning off and now it seems to be installing.

  • OghulkOghulk Tinychat Janitor TinychatRegistered User regular
    Echo wrote: »
    Some googling blames virus protection and it did come with McAfee malware pre-installed...

    But you did a fresh install of Windows? Or was it not a clean reformat?

    Not having VS2015 might be the likeliest culprit since Micro$oft likes to be dumb with the way installations work.

  • InfidelInfidel Heretic Registered User regular
    Yeah, was gonna say, scanner messing it up?

    OrokosPA.png
  • EchoEcho ski-bap ba-dapModerator mod
    Yeah, I meant "factory install" when I said "fresh install". My bad. it's a Thinkpad with a bunch of junk.

  • ecco the dolphinecco the dolphin Registered User regular
    Bah, Ecco, you're bad luck. Right after I reply to your post I walk in this morning to a pile of crap that had to fixed and pushed asap because someone put too many locks in our market feed handlers.

    I am so sorry! =P

    Penny Arcade Developers at PADev.net.
  • SpawnbrokerSpawnbroker Registered User regular
    Echo wrote: »
    Yeah, I meant "factory install" when I said "fresh install". My bad. it's a Thinkpad with a bunch of junk.

    Try running the installer as administrator if turning off the McAfee virus doesn't work.

    Steam: Spawnbroker
  • KakodaimonosKakodaimonos Code fondler Helping the 1% get richerRegistered User regular
    Whomever wrote this JSON structure was salty as hell.

    One property: "XML". I bet you can guess what's in there.

  • OghulkOghulk Tinychat Janitor TinychatRegistered User regular
    Someone explain the purpose of JSON to me. Like I get it's a convenient way to store object metadata, but it's also kind of a hassle to navigate at times?

    I like it don't get me wrong, it just seems kinda silly if you don't know the keys for the data and have to sift through everything.

  • bowenbowen How you doin'? Registered User regular
    JSON is human readable data, similar to XML, but I stress the human readable part of that

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • YoshuaYoshua Registered User regular
    I can read XML just fine.

  • bowenbowen How you doin'? Registered User regular
    I can read assembly too, but that doesn't mean it's not a chore.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
This discussion has been closed.