As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/
We're funding a new Acquisitions Incorporated series on Kickstarter right now! Check it out at https://www.kickstarter.com/projects/pennyarcade/acquisitions-incorporated-the-series-2

[Programming] djmitchella travelling through snow to rfind duplicate dates for singletons

11718202223100

Posts

  • DrezDrez Registered User regular
    Cool, thanks @bowen.

    And I'm familiar with DBs (well...Access lol) and SQL too. But I would assume I'd just use a data file of some kind that can be interpreted by my game rather then incorporating an actual DB into my game? Or is that what you are suggesting?

    Switch: SW-7690-2320-9238Steam/PSN/Xbox: Drezdar
  • bowenbowen How you doin'? Registered User regular
    flat files are similar to access. Most of the common ones use SQL (access does on some level).

    The most common is SQLite.

    If you're doing swift on an iOS, I think they have a port of that:
    https://github.com/nerdyc/Squeal

    That's a library I pulled up for dealing with it.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • EtheaEthea Registered User regular
    Or just write out and read a json flat file. Swift should have built in json parsing.

    bowen
  • bowenbowen How you doin'? Registered User regular
    That could get massive, though, if you've got more than a few hundred things. Well, not massively different, but harder to get the data you want and you'll probably switch to a flat file DB like sqlite anyways.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • EtheaEthea Registered User regular
    bowen wrote: »
    That could get massive, though, if you've got more than a few hundred things. Well, not massively different, but harder to get the data you want and you'll probably switch to a flat file DB like sqlite anyways.

    This is what I like to avoid. The I/O performance of a json file is not going to kill anything even when you have a couple thousand enemy records. Now at that scale adding new enemies by hand will be difficult, but adding new enemies to an sqlite db will be harder.

    Start with simple I/O and work on the things that matter ( unless you are writing an I/O library ).

  • bowenbowen How you doin'? Registered User regular
    Is it harder though? I dunno, first thing I do is work on an admin interface to simplify that stuff. In a place like iOS where memory is at a premium, seems like you'd not want to cache in an entire set of things. I guess you could get around that by having tons of files. Then you get into the position where you don't want tons of files so you find a way to package them...

    And then we're back at square one.

    But yeah, json seems like it'd be just fine!

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • EtheaEthea Registered User regular
    bowen wrote: »
    Is it harder though? I dunno, first thing I do is work on an admin interface to simplify that stuff. In a place like iOS where memory is at a premium, seems like you'd not want to cache in an entire set of things. I guess you could get around that by having tons of files. Then you get into the position where you don't want tons of files so you find a way to package them...

    And then we're back at square one.

    But yeah, json seems like it'd be just fine!

    You are a true programmer.

    Step 1 to making a game: lets write an admin interface to add elements to a flat file db.

    bowengavindelEchoMahnmutVegemyte
  • JasconiusJasconius sword criminal mad onlineRegistered User regular
    i was just talking about this a dozen or so pages ago

    I started with a JSON flat file just to sketch out what kind of data I would need, but eventually you do actually need a tool

    so.. not Step 1, for me it was more like step 80000, but it is required

    i wonder if MySQL workbench has a JSON export... that would be pretty easy

  • bowenbowen How you doin'? Registered User regular
    Fuck if I want to do it by hand man!

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
    gavindel
  • bowenbowen How you doin'? Registered User regular
    That gets old after like... three times!

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • EtheaEthea Registered User regular
    I am going to totally ignore the statement that iOS is a place where memory is at a premium. Mobile user face application development is nearly the last place where memory is a concern.

    The thing has :bleeping: virtual memory, something that tons of developers still don't have.

  • JasconiusJasconius sword criminal mad onlineRegistered User regular
    simultaneously, the application itself is usually given upwards of 30MB of RAM just for doing stuff on the main thread for application-exclusive code

    which is a lot. and if you have 10 megs of JSON data for items and monsters, what are you even doing making an iPhone game

    ios is not that starved unless you do some patently stupid things, like load up multiple mega-pixels worth of texture data and hold it in memory without handing it off to OpenGL or the UI libraries

  • lazegamerlazegamer Registered User regular
    For prototyping, it's pretty nice to just set up a nosql database like mongo and just use a document store. Gives you a good outline of what your schema should look like in the end, and you can transition to an embeddable database like sqlite.

    Are there any decent embeddable nosql options out there? I'm sure there are lots of attempts (came across http://unqlite.org/), but not familiar with anything that I'd actually trust.

  • PhyphorPhyphor Building Planet Busters Tasting FruitRegistered User regular
    Ethea wrote: »
    bowen wrote: »
    Is it harder though? I dunno, first thing I do is work on an admin interface to simplify that stuff. In a place like iOS where memory is at a premium, seems like you'd not want to cache in an entire set of things. I guess you could get around that by having tons of files. Then you get into the position where you don't want tons of files so you find a way to package them...

    And then we're back at square one.

    But yeah, json seems like it'd be just fine!

    You are a true programmer.

    Step 1 to making a game: lets write an admin interface to add elements to a flat file db.

    Pfft, I wrote a filesystem

  • bowenbowen How you doin'? Registered User regular
    Jasconius wrote: »
    simultaneously, the application itself is usually given upwards of 30MB of RAM just for doing stuff on the main thread for application-exclusive code

    which is a lot. and if you have 10 megs of JSON data for items and monsters, what are you even doing making an iPhone game

    ios is not that starved unless you do some patently stupid things, like load up multiple mega-pixels worth of texture data and hold it in memory without handing it off to OpenGL or the UI libraries

    DON'T JUDGE ME

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • EtheaEthea Registered User regular
    Phyphor wrote: »
    Ethea wrote: »
    bowen wrote: »
    Is it harder though? I dunno, first thing I do is work on an admin interface to simplify that stuff. In a place like iOS where memory is at a premium, seems like you'd not want to cache in an entire set of things. I guess you could get around that by having tons of files. Then you get into the position where you don't want tons of files so you find a way to package them...

    And then we're back at square one.

    But yeah, json seems like it'd be just fine!

    You are a true programmer.

    Step 1 to making a game: lets write an admin interface to add elements to a flat file db.

    Pfft, I wrote a filesystem

    Do tell me more, that does sound fun.

  • JasconiusJasconius sword criminal mad onlineRegistered User regular
    i found this very handy page on twitter last week

    https://cpp.zeef.com/faraz.fallahi

    I can't wait until I get some spare time to check some of these out for viability in my game

  • EtheaEthea Registered User regular
    On a side note, "Pride and Prejudice" is less than 850kb uncompressed, and "The Adventures of Sherlock Holmes" is less than 650kb.

    source: http://www.gutenberg.org/ebooks/

    bowen
  • bowenbowen How you doin'? Registered User regular
    Ethea wrote: »
    On a side note, "Pride and Prejudice" is less than 850kb uncompressed, and "The Adventures of Sherlock Holmes" is less than 650kb.

    source: http://www.gutenberg.org/ebooks/

    Does the entire library of congress still fit on a CD or whatnot? Or are they doing PDFs and all that instead of raw text?

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • EtheaEthea Registered User regular
    edited October 2014
    bowen wrote: »
    Ethea wrote: »
    On a side note, "Pride and Prejudice" is less than 850kb uncompressed, and "The Adventures of Sherlock Holmes" is less than 650kb.

    source: http://www.gutenberg.org/ebooks/

    Does the entire library of congress still fit on a CD or whatnot? Or are they doing PDFs and all that instead of raw text?

    Library of congress is pretty large. I believe they have never released official numbers but it is over 3 petabytes.

    Edit:

    Remembering saving for a library does not mean just the text. I would expect they are taking high resolution digitial photo's of each page. They need to capture all Marginalia ( margin notes ), etc. Plus I expect they run OCR on it all to be able to do cross referencing.

    I expect the library of congress is a great place for a programmer to geek out.

    Ethea on
    bowen
  • JasconiusJasconius sword criminal mad onlineRegistered User regular
    the library of congress also has a lot of movies, so that's basically a blu-ray a piece

  • DrezDrez Registered User regular
    Okay thanks for the advice. I already have my tables created in Excel (I was screwing around balancing out the game weeks ago) so short term it might be easiest just to write a program to parse it into json format and use that...just so I can get up and running, perhaps. But then later I can look into an embedded DB - I really like that idea.

    Switch: SW-7690-2320-9238Steam/PSN/Xbox: Drezdar
  • DrezDrez Registered User regular
    They should convert to SSD.

    Switch: SW-7690-2320-9238Steam/PSN/Xbox: Drezdar
  • SmasherSmasher Starting to get dizzy Registered User regular
    Drez wrote: »
    They should convert to SSD.
    Who, the Library of Congress?

  • DrezDrez Registered User regular
    edited October 2014
    Smasher wrote: »
    Drez wrote: »
    They should convert to SSD.
    Who, the Library of Congress?

    Yeah I'm just being stupid.

    Drez on
    Switch: SW-7690-2320-9238Steam/PSN/Xbox: Drezdar
  • PhyphorPhyphor Building Planet Busters Tasting FruitRegistered User regular
    edited October 2014
    Ethea wrote: »
    Phyphor wrote: »
    Ethea wrote: »
    bowen wrote: »
    Is it harder though? I dunno, first thing I do is work on an admin interface to simplify that stuff. In a place like iOS where memory is at a premium, seems like you'd not want to cache in an entire set of things. I guess you could get around that by having tons of files. Then you get into the position where you don't want tons of files so you find a way to package them...

    And then we're back at square one.

    But yeah, json seems like it'd be just fine!

    You are a true programmer.

    Step 1 to making a game: lets write an admin interface to add elements to a flat file db.

    Pfft, I wrote a filesystem

    Do tell me more, that does sound fun.

    Well, we needed a way to access lots of resource files - models, textures, shaders, configuration, etc, and we had level editing built right into the game, so we needed to potentially save a bunch of files with the levels and so needed some way to pack them, not just the baked assets file to be shipped

    The classic choice is a store-only zip file, used by things like Quake 3. Another choice is Blizzard's MPQ (which they finally dropped for something more like what I built actually!). The problem with those is they are static, if you change something you have to re-write the entire file to disk. You can see this with older WoW updates, where they download the update and then have to write the entire massive merged file to disk

    So, I wrote a filesystem. Extents-based with inodes, noting special really, and since it was designed to be file-based, only 8kB of metadata (4 for the superblock and 4 for the root freelist) and insufficient space writes would try to extend the file

    And then I tied it in with the rest of the I/O system. To load a map that was on someone else's machine, we "opened" a file that routed requests to the target machine and copied it locally. But because it was just another file, we could run the level setup at the same time and the requests (cached locally) would be mixed in. I/O requests would be split across extents as needed, I'd made sure the entire I/O backend was asynchronous anyway

    Phyphor on
    bowenASimPerson
  • MvrckMvrck Dwarven MountainhomeRegistered User regular
    The Library of Congress has archival masters of all three original Star Wars films. It makes me so fucking sad they may never see the light of day.

    gavindel
  • GnomeTankGnomeTank What the what? Portland, OregonRegistered User regular
    Drez wrote: »
    Okay thanks for the advice. I already have my tables created in Excel (I was screwing around balancing out the game weeks ago) so short term it might be easiest just to write a program to parse it into json format and use that...just so I can get up and running, perhaps. But then later I can look into an embedded DB - I really like that idea.

    As a side note, World of WarCraft managed it's entire data stack with Excel files through TBC. It was part of the huge infrastructure upgrade with WoTLK that they finally went to editing the production databases through an interface specifically designed for the task.

    Sagroth wrote: »
    Oh c'mon FyreWulff, no one's gonna pay to visit Uranus.
    Steam: Brainling, XBL / PSN: GnomeTank, NintendoID: Brainling, FF14: Zillius Rosh SFV: Brainling
  • LD50LD50 Registered User regular
    Drez wrote: »
    Okay thanks for the advice. I already have my tables created in Excel (I was screwing around balancing out the game weeks ago) so short term it might be easiest just to write a program to parse it into json format and use that...just so I can get up and running, perhaps. But then later I can look into an embedded DB - I really like that idea.

    http://www.convertcsv.com/csv-to-json.htm

  • schussschuss Registered User regular
    So one of the reporting tools I will be using uses SAP Business Objects to die into the database for our EHR. This thing writes terrible SQL. I say this as a guy who has pretty written SQL for about a week (not counting a class in college about 10 years ago).
    I opened up this report someone sent me, and went to the SQL view. The WHERE clause is about 25 lines. About 10 of it is this set of lines where it's checking some code values against some other code or something which is then ANDed to check against an encounter or person active indicator.
    And then, after all of those, it ANDs against a check for the active indicators all by themselves, further increasing the redundancy.

    Basically this section looks like this
    WHERE
    (C AND A)
    AND (D AND A)
    AND (E AND A)
    AND (F AND A)
    AND (G AND A)
    AND (H AND A)
    AND (I AND A)
    AND (A)
    AND (B)
    
    When it could be
    A AND B AND C AND D AND E AND F AND G AND H AND I
    

    Edit: When the query optimizer runs does it go "WTF is this shit?" and fix it?

    Yeah.... As someone who owns an app on the business side that relies on BO - don't use BO.

  • GnomeTankGnomeTank What the what? Portland, OregonRegistered User regular
    In general, don't use SAP. It does nothing that you can can't do yourself with a modern software stack and some web services, over charges you for that privilege, and is built on 1970's technology.

    Sagroth wrote: »
    Oh c'mon FyreWulff, no one's gonna pay to visit Uranus.
    Steam: Brainling, XBL / PSN: GnomeTank, NintendoID: Brainling, FF14: Zillius Rosh SFV: Brainling
  • DrezDrez Registered User regular
    LD50 wrote: »
    Drez wrote: »
    Okay thanks for the advice. I already have my tables created in Excel (I was screwing around balancing out the game weeks ago) so short term it might be easiest just to write a program to parse it into json format and use that...just so I can get up and running, perhaps. But then later I can look into an embedded DB - I really like that idea.

    http://www.convertcsv.com/csv-to-json.htm

    Cool thanks.

    Switch: SW-7690-2320-9238Steam/PSN/Xbox: Drezdar
  • electricitylikesmeelectricitylikesme Registered User regular
    Drez wrote: »
    LD50 wrote: »
    Drez wrote: »
    Okay thanks for the advice. I already have my tables created in Excel (I was screwing around balancing out the game weeks ago) so short term it might be easiest just to write a program to parse it into json format and use that...just so I can get up and running, perhaps. But then later I can look into an embedded DB - I really like that idea.

    http://www.convertcsv.com/csv-to-json.htm

    Cool thanks.

    There's a lot of resources out there for using SQLite for this type of thing I'm finding - it's a really common use case, and a really well developed DB for it.

  • Alistair HuttonAlistair Hutton Dr EdinburghRegistered User regular
    LD50 wrote: »
    bowen wrote: »
    admanb wrote: »
    XML that doesn't quite follow XML standards but still parses? Sounds like all XML to me.

    I feel like there's an XHTML joke in here.



    bowen wrote: »
    LD50 wrote: »
    Echo wrote: »


    Mere words aren't enough for me to express my level of WTF.

    This requires a WTF Professional.

    fI4TPyq.jpg

    Why?

    JSON was supposed to be the alternative to XML. Most of what you do in JSON has a pretty much 1:1 correlation with XML, there's no reason to translate it. Not to mention, this XML is really ... stupid I don't know there's tons of better ways to do that.

    But this JSONx is basically XML... but JSON.

    So we've come full circle basically.

    XML->JSON->JSONx (which is XML++, from the looks of it, doesn't quite follow XML standards, but would probably parse for the most part)

    Yeah, but I want to know why.

    It's an interopability layer.

    I have a thoughtful and infrequently updated blog about games http://whatithinkaboutwhenithinkaboutgames.wordpress.com/

    I made a game, it has penguins in it. It's pay what you like on Gumroad.

    Currently Ebaying Nothing at all but I might do in the future.
  • bowenbowen How you doin'? Registered User regular
    Phyphor wrote: »
    Ethea wrote: »
    Phyphor wrote: »
    Ethea wrote: »
    bowen wrote: »
    Is it harder though? I dunno, first thing I do is work on an admin interface to simplify that stuff. In a place like iOS where memory is at a premium, seems like you'd not want to cache in an entire set of things. I guess you could get around that by having tons of files. Then you get into the position where you don't want tons of files so you find a way to package them...

    And then we're back at square one.

    But yeah, json seems like it'd be just fine!

    You are a true programmer.

    Step 1 to making a game: lets write an admin interface to add elements to a flat file db.

    Pfft, I wrote a filesystem

    Do tell me more, that does sound fun.

    Well, we needed a way to access lots of resource files - models, textures, shaders, configuration, etc, and we had level editing built right into the game, so we needed to potentially save a bunch of files with the levels and so needed some way to pack them, not just the baked assets file to be shipped

    The classic choice is a store-only zip file, used by things like Quake 3. Another choice is Blizzard's MPQ (which they finally dropped for something more like what I built actually!). The problem with those is they are static, if you change something you have to re-write the entire file to disk. You can see this with older WoW updates, where they download the update and then have to write the entire massive merged file to disk

    So, I wrote a filesystem. Extents-based with inodes, noting special really, and since it was designed to be file-based, only 8kB of metadata (4 for the superblock and 4 for the root freelist) and insufficient space writes would try to extend the file

    And then I tied it in with the rest of the I/O system. To load a map that was on someone else's machine, we "opened" a file that routed requests to the target machine and copied it locally. But because it was just another file, we could run the level setup at the same time and the requests (cached locally) would be mixed in. I/O requests would be split across extents as needed, I'd made sure the entire I/O backend was asynchronous anyway

    You should throw that shit up on github, bro.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
    Evigilant
  • TofystedethTofystedeth Registered User regular
    GnomeTank wrote: »
    In general, don't use SAP. It does nothing that you can can't do yourself with a modern software stack and some web services, over charges you for that privilege, and is built on 1970's technology.

    Yeah, it's entirely out of my hands and not going anywhere. It's one of the two provided means of getting reports out of the DB for our EHR and it's the better of the two. When using our own stuff we use SSRS.

    steam_sig.png
  • schussschuss Registered User regular
    Question for developers:
    Is it reasonable for me to rely on my developers to identify technology change risks (ex. iOS 7-8 transition), or should that be on the business team to identify and prioritize?

    Honest question.

  • bowenbowen How you doin'? Registered User regular
    edited October 2014
    schuss wrote: »
    Question for developers:
    Is it reasonable for me to rely on my developers to identify technology change risks (ex. iOS 7-8 transition), or should that be on the business team to identify and prioritize?

    Honest question.

    Depends on the developer. Most of my peers are dumb and lazy.

    ... let me rephrase that... almost everyone I know is dumb and lazy and can't be ass-ed to figure this shit out unless their job depended on it. (ie fear of getting fired)

    So if you've got someone that knows their shit, they can probably tell you. If you've got someone that comes in, clocks in, zones out while they code, then clocks out... probably not.

    bowen on
    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • bowenbowen How you doin'? Registered User regular
    Example: Guy at the hospital I was working with. Zip files are failing to send to their email, they're not getting anything, it fails silently, no one is notified. I ask him "this has been working, has anything changed on your end in the last month?"

    The answer: "No it's on your end."

    So I spend a week looking into it, do all sorts of tests, get google involved (they host our email), nope, it appears it leaves our server just find.

    What was happening? They installed a barracuda and encrypted zip files were getting silently tossed because it couldn't ensure it was virus free.

    Now, if I was head of the department that did that, you're sure as shit I had to know what was going on. (Dumb)

    What's worse, if someone had the problem, I'd at least look into it and figure out why it was failing. (Lazy)

    He responded with what was essentially a professional "meh" when I questioned him why he couldn't tell me that a week ago. (no fear of getting fired)

    The only reason I was able to tell him how his infrastructure worked, was because of a coworker of his I was working with at the time that handled VPNs, he could log into the barracuda and saw the test message I sent him just sitting there in an "unknown" state. This is a person that knows their shit.

    There is nothing worse that grinds my gears than lazy people I have to work with. Lazy is okay, for the most part, but dumb and lazy pisses me the fuck off.

    Dumb and lazy, and then you tell me I'm wrong? Oh I am going to motherfucking gut you bitch. If you can't give me evidence, fuck you.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
    ironsizide
  • urahonkyurahonky Registered User regular
    So one of the reporting tools I will be using uses SAP Business Objects to die into the database for our EHR. This thing writes terrible SQL. I say this as a guy who has pretty written SQL for about a week (not counting a class in college about 10 years ago).
    I opened up this report someone sent me, and went to the SQL view. The WHERE clause is about 25 lines. About 10 of it is this set of lines where it's checking some code values against some other code or something which is then ANDed to check against an encounter or person active indicator.
    And then, after all of those, it ANDs against a check for the active indicators all by themselves, further increasing the redundancy.

    Basically this section looks like this
    WHERE
    (C AND A)
    AND (D AND A)
    AND (E AND A)
    AND (F AND A)
    AND (G AND A)
    AND (H AND A)
    AND (I AND A)
    AND (A)
    AND (B)
    
    When it could be
    A AND B AND C AND D AND E AND F AND G AND H AND I
    

    Edit: When the query optimizer runs does it go "WTF is this shit?" and fix it?

    Hey @Tofystedeth‌! I'm using SAP Business Objects too! It's absolutely terrible at everything it does.

Sign In or Register to comment.