As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/
Options

[Programming] djmitchella travelling through snow to rfind duplicate dates for singletons

12425272930100

Posts

  • Options
    DrezDrez Registered User regular
    Ethea wrote: »
    Drez wrote: »
    bowen wrote: »
    Drez wrote: »
    bowen wrote: »
    Ethea wrote: »
    I am really curious in what you are representing that you require values larger than 2.1B.

    Money would be my first guess.

    More or less, yeah.

    You can get around this by using int16s and doing 4-5 subdivisions.

    Like you'd see in RPGs : copper, silver, gold, platinum, unobtanium

    That is, if you're not making a "real world dollars" program.

    Actually, money is only one aspect. I've decided to go the crazy Disgaea route with stats. So my EXP table effectively reaches toward 100B.

    I'm currently trying to tackle how best to generate random numbers with a min/max range beyond 4.3B, since arc4random_uniform() seems to be the preferred method, even in Swift, but only accepts UInt32.


    custom class of two UInt32 that represent the high / low of a UInt64.

    Of course! (And now I understand @Infidel's post too.) lol

    Thanks to both of you!

    Switch: SW-7690-2320-9238Steam/PSN/Xbox: Drezdar
  • Options
    DrezDrez Registered User regular
    bowen wrote: »
    No you're right. But, if you compiled that as a 32 bit binary, integerDude64 is still a 64 bit integer. But, integerDude becomes 32 bit. Int is basically shorthand for whatever method the compiler is using at the time.

    It's all well and fine to use that if you don't care about the range and are likely to never hit it. If you know you're going to need billions of numbers, specify it as Int64 like your second one.

    Fair point.

    Switch: SW-7690-2320-9238Steam/PSN/Xbox: Drezdar
  • Options
    bowenbowen How you doin'? Registered User regular
    while (!reader.EOF)
    {                                
    	try
    	{
    		reader.MoveToContent();
    		while (reader.Read())
    		{
    			if (reader.IsStartElement())
    			{
    				switch (reader.Name)
    				{
    					case "ROW":
    						records++;
    						Console.WriteLine("Got a row");
    						break;
    					default:
    						Console.WriteLine("Got {0}", reader.Name);
    						break;
    				}
    			}
    		}
    	}
    	catch(Exception Ex)
    	{
    		Console.WriteLine("Could not read record.\r\n{0}", Ex);
    		reader.ReadToNextSibling("ROW");
    	}
    }
    

    Maybe someone else can figure this shit out because I'm starting to get foggy with anger over this stupid XML garbage.

    When it hits one of those garbage records, my exception catches it, but doesn't do anything, read just instantly fails forever and ever after this. How would I "skip the garbage record" basically? The EOF was my latest attempt to keep cycling over and over after a fail, but it doesn't seem to keep going after the fail, does the read() method have some sort of in built protection that I need to circumvent?

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    bowenbowen How you doin'? Registered User regular
    Looks like once the XML reader hits an exception that's the end of the line for it... hm.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    RendRend Registered User regular
    Why is it throwing an exception for the garbage record?

  • Options
    PolaritiePolaritie Sleepy Registered User regular
    Because the record has < and >. The parser breaks, it throws an exception.

    I could swear CDATA handles that.

    Steam: Polaritie
    3DS: 0473-8507-2652
    Switch: SW-5185-4991-5118
    PSN: AbEntropy
  • Options
    bowenbowen How you doin'? Registered User regular
    CDATA might handle it, but I'd have to edit the files by hand.

    It's throwing an exception because it doesn't know what to do. I just wish I could make it skip that line and move on. Is there a way to do that with xmlreader or am I legitimately going to have to write my own?

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    PolaritiePolaritie Sleepy Registered User regular
    You cant run the data through another block of code for preprocessing?

    And yeah, you're kind of stuck if your reader breaks entirely on that instead of just aborting the current method. That seems sloppy.

    Steam: Polaritie
    3DS: 0473-8507-2652
    Switch: SW-5185-4991-5118
    PSN: AbEntropy
  • Options
    bowenbowen How you doin'? Registered User regular
    Polaritie wrote: »
    You cant run the data through another block of code for preprocessing?

    And yeah, you're kind of stuck if your reader breaks entirely on that instead of just aborting the current method. That seems sloppy.

    You're suggesting I modify that text inline... hmm good idea!

    I'll try that.

    And yeah the XMLReader is inbuilt to C#, I'm surprised it just barfs and has no option to skip it.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    IncindiumIncindium Registered User regular
    edited October 2014
    Wouldn't your program flow need to be different to catch the error and proceed? Like while (1){ try {reader.Read();} catch(){ //handle and keep going}}

    Incindium on
    steam_sig.png
    Nintendo ID: Incindium
    PSN: IncindiumX
  • Options
    bowenbowen How you doin'? Registered User regular
    Problem is as soon as the reader hits a parsing error, that's it. I tried inline editing, that turned out to be a pain in the ass.

    I found a way to get this data into access, somehow, with one of the other foxpro exports... this is working okay.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    LD50LD50 Registered User regular
    bowen wrote: »
    Problem is as soon as the reader hits a parsing error, that's it. I tried inline editing, that turned out to be a pain in the ass.

    I found a way to get this data into access, somehow, with one of the other foxpro exports... this is working okay.

    Just remove whichever < or > is closest to the center of the line and try again. That should work for like 85% of cases, right?

  • Options
    bowenbowen How you doin'? Registered User regular
    I just honestly wish I could say fuck off to this database, but he is absolutely adamant about all data being there when switched.

    I wish he had thought of that 20 years ago and not used foxpro.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    gavindelgavindel The reason all your software is brokenRegistered User regular
    Hmm. Buffer overflows. Cool haxxoring, but they made me feel like an idiot today. Here's a basic program for the overflowing:
    #define VALUE 0
    #define VALUE2 0
    
    int function(int a, int b, int c){
      char buf1[5];
      char buf2[10];
      int *ret, i;
    
      ret = buf1 + VALUE;
      (*ret) += VALUE2;
    
    }
    
    int main() {
      int x;
      x = 0;
      function(1,2,3);
      x = 1;
      printf("%d\n", x);
    
    }
    

    This is your basic introductory overflow. Make Value equal a number, make Value2 equal a number, and bam. The goal is to make the program print 0 instead of 1. You have a helper program to examine the stack so you can do your address arithmetic and find VALUE. You can load the program into gdb to examine the assembler code to find VALUE2.

    Of course, being the genius I am, I made VALUE and VALUE2 ints within the program without mirroring that change in the stack inspector. All my numbers failed, the world was doomed...because I added 8 more bytes to the stack in between my buffer and the return pointer.

    Book - Royal road - Free! Seraphim === TTRPG - Wuxia - Free! Seln Alora
  • Options
    PhyphorPhyphor Building Planet Busters Tasting FruitRegistered User regular
    With sufficient compiler optimization (really any optimization at all) it may be impossible for that program to ever print anything except 1

  • Options
    ASimPersonASimPerson Cold... and hard.Registered User regular
    What I learned from my hacking lab in college is that hacking things requires an attention to detail and willingness to put up with really tedious poking at bits that I don't really posses.

  • Options
    PhyphorPhyphor Building Planet Busters Tasting FruitRegistered User regular
    edited October 2014
    In fact, the compiler can legally elide that function entirely - it can determine that no valid code path introduces a side-effect, since pointer arithmetic past the end of the arrays is undefined. Modern compilers are fun!

    Phyphor on
  • Options
    gavindelgavindel The reason all your software is brokenRegistered User regular
    Yep. But isn't half of a stack overflow attack going "Nice legacy system you have there. Be a shame if..."

    Book - Royal road - Free! Seraphim === TTRPG - Wuxia - Free! Seln Alora
  • Options
    RendRend Registered User regular
    gavindel wrote: »
    Yep. But isn't half of a stack overflow attack going "Nice legacy system you have there. Be a shame if..."

    You would be shocked and appalled at how untrue that comment is.

  • Options
    SaerisSaeris Borb Enthusiast flapflapflapflapRegistered User regular
    @bowen Your XML looks mostly like this, right?
    <row>some data</row>
    <row>some bad > data</row>
    <row>more < bad data</row>
    

    If that's the case, why can't you just do something like this?
    // pseudocode
    var input = openFileToRead('bad.xml')
    var output = openFileToWrite('good.xml')
    var openingPlaceholder = '{' + Math.random() + '}'
    var closingPlaceholder = '{' + Math.random() + '}'
    for line in input.lines() do
        line = line.replace('<row>', openingPlaceholder)
        line = line.replace('</row>', closingPlaceholder)
        line = line.replace('<', '&lt;')
        line = line.replace('>', '&gt;')
        line = line.replace(openingPlaceholder, '<row>')
        line = line.replace(closingPlaceholder, '</row>')
        output.writeLine(line)
    end
    

    borb_sig.png
  • Options
    bowenbowen How you doin'? Registered User regular
    A bit more complex than that, but yeah, pretty close.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    PhyphorPhyphor Building Planet Busters Tasting FruitRegistered User regular
    edited October 2014
    Phyphor wrote: »
    bowen wrote: »
    Phyphor wrote: »
    Ethea wrote: »
    Phyphor wrote: »
    Ethea wrote: »
    bowen wrote: »
    Is it harder though? I dunno, first thing I do is work on an admin interface to simplify that stuff. In a place like iOS where memory is at a premium, seems like you'd not want to cache in an entire set of things. I guess you could get around that by having tons of files. Then you get into the position where you don't want tons of files so you find a way to package them...

    And then we're back at square one.

    But yeah, json seems like it'd be just fine!

    You are a true programmer.

    Step 1 to making a game: lets write an admin interface to add elements to a flat file db.

    Pfft, I wrote a filesystem

    Do tell me more, that does sound fun.

    Well, we needed a way to access lots of resource files - models, textures, shaders, configuration, etc, and we had level editing built right into the game, so we needed to potentially save a bunch of files with the levels and so needed some way to pack them, not just the baked assets file to be shipped

    The classic choice is a store-only zip file, used by things like Quake 3. Another choice is Blizzard's MPQ (which they finally dropped for something more like what I built actually!). The problem with those is they are static, if you change something you have to re-write the entire file to disk. You can see this with older WoW updates, where they download the update and then have to write the entire massive merged file to disk

    So, I wrote a filesystem. Extents-based with inodes, noting special really, and since it was designed to be file-based, only 8kB of metadata (4 for the superblock and 4 for the root freelist) and insufficient space writes would try to extend the file

    And then I tied it in with the rest of the I/O system. To load a map that was on someone else's machine, we "opened" a file that routed requests to the target machine and copied it locally. But because it was just another file, we could run the level setup at the same time and the requests (cached locally) would be mixed in. I/O requests would be split across extents as needed, I'd made sure the entire I/O backend was asynchronous anyway

    You should throw that shit up on github, bro.

    Didn't think anyone would be interested. I'll see if I can find it

    https://bitbucket.org/phyphor/packfile

    @bowen, @Ethea‌

    Needs a lot (more) cleanup, but it works. If you go install the dokan usermode fs* one of the projects will mount it in windows. Presumably you could hook it into FUSE too, but I'm too lazy for that right now

    *It can be... crashy sometimes. Of the bluescreen variety

    Phyphor on
  • Options
    PhyphorPhyphor Building Planet Busters Tasting FruitRegistered User regular
    Oh and I guess I didn't bother to finish some features, or I grabbed an old file. Hmm

  • Options
    CokebotleCokebotle 穴掘りの 電車内Registered User regular
    Since Thursday, I've gone from "Goddamn bullshit clouds, breaking my Matlab code" to "Neat! I can now detect image shifts and approximate scaling before stitching these together". This stuff is pretty fascinating!

    Is all programming like this in the real world?

    工事中
  • Options
    Jimmy KingJimmy King Registered User regular
    Cokebotle wrote: »
    Since Thursday, I've gone from "Goddamn bullshit clouds, breaking my Matlab code" to "Neat! I can now detect image shifts and approximate scaling before stitching these together". This stuff is pretty fascinating!

    Is all programming like this in the real world?

    Sometimes. There are dev jobs which are basically always terrible, but most enjoyable dev jobs I think have a lot of what you just described... alternating between "Oh wtf, why doesn't anything fucking work? Why am I too stupid to get this!?!?!" and then hours or days later "Oh, wait, this is fucking awesome. I'm so badass and making totally awesome things happen."

  • Options
    urahonkyurahonky Resident FF7R hater Registered User regular
    Cokebotle wrote: »
    Since Thursday, I've gone from "Goddamn bullshit clouds, breaking my Matlab code" to "Neat! I can now detect image shifts and approximate scaling before stitching these together". This stuff is pretty fascinating!

    Is all programming like this in the real world?

    Not all programming is the same in the real world.

    But the feeling of getting stuff done always exists. When you're stuck on something for a few hours and then you finally figure it out there is no better feeling in this world.

  • Options
    JasconiusJasconius sword criminal mad onlineRegistered User regular
    got Awesomium fully up and running and I'm back to 60FPS in release mode thanks to some SDL hacking

    Being able to do HTML in a defined aspect ratio gives you all kinds of fun opportunities like being able to do 100% liquid layouts in CSS

  • Options
    urahonkyurahonky Resident FF7R hater Registered User regular
    Anyone know how to convert a Teradata date integer into a Flash/Java date?

    Number I'm given: 41517
    Date it represents: 31-Aug-2013

    According to a coworker they hold their dates and integers like this:
    Teradata stores dates as integers in the following manner:

    (year - 1900) * 10000 + (month * 100) + day

  • Options
    bowenbowen How you doin'? Registered User regular
    that's fucking stupid

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    bowenbowen How you doin'? Registered User regular
    Also, 2013-08-31 should be 1130831 in that integer format, yeah?

    41517 is like... not going to be any valid date no matter how you slice it, even if you had significant 0s where they were supposed to be.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    RendRend Registered User regular
    urahonky wrote: »
    Anyone know how to convert a Teradata date integer into a Flash/Java date?

    Number I'm given: 41517
    Date it represents: 31-Aug-2013

    According to a coworker they hold their dates and integers like this:
    Teradata stores dates as integers in the following manner:

    (year - 1900) * 10000 + (month * 100) + day

    Where are you getting that date integer / actual date conversion?

    Those two values do not match in any meaningful way. And it looks like your coworker was correct. august 31 2013 should be encoded as 1130831 as bowen mentioned.
    or, CYYMMDD (where c = century, and c=0 indicates the 20th century, c=1 indicates the 21st century, etc)

    Are you sure that integer is accurate? I agree with bowen: it doesn't actually represent a date in that format.

  • Options
    HounHoun Registered User regular
    https://forums.teradata.com/forum/database/integer7-to-date

    The formula appears correct, but there's no way that number came from that formula.

  • Options
    LuvTheMonkeyLuvTheMonkey High Sierra Serenade Registered User regular
    urahonky wrote: »
    Anyone know how to convert a Teradata date integer into a Flash/Java date?

    Number I'm given: 41517
    Date it represents: 31-Aug-2013

    According to a coworker they hold their dates and integers like this:
    Teradata stores dates as integers in the following manner:

    (year - 1900) * 10000 + (month * 100) + day

    Looks like that date integer is done the same way Excel does. It's indexed against January 0th, 1900 - such that 1 comes out as Jan 1, 1900, etc.

    Molten variables hiss and roar. On my mind-forge, I hammer them into the greatsword Epistemology. Many are my foes this night.
    STEAM | GW2: Thalys
  • Options
    bowenbowen How you doin'? Registered User regular
    41517 could theoretically be : January 7th, 2315, that's about the only valid date, if we inserted 0's after the 5 and 1. Anything else puts you in crazy land.

    7th day of the 51st month of 1941.

    17th day of the 15th month of 1904

    There is no way that number is valid!

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    bowenbowen How you doin'? Registered User regular
    urahonky wrote: »
    Anyone know how to convert a Teradata date integer into a Flash/Java date?

    Number I'm given: 41517
    Date it represents: 31-Aug-2013

    According to a coworker they hold their dates and integers like this:
    Teradata stores dates as integers in the following manner:

    (year - 1900) * 10000 + (month * 100) + day

    Looks like that date integer is done the same way Excel does. It's indexed against January 0th, 1900 - such that 1 comes out as Jan 1, 1900, etc.

    That gets you pretty close, still off by a bit I think. I think that puts you somewhere around August 16th 2013.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    HounHoun Registered User regular
    urahonky wrote: »
    Anyone know how to convert a Teradata date integer into a Flash/Java date?

    Number I'm given: 41517
    Date it represents: 31-Aug-2013

    According to a coworker they hold their dates and integers like this:
    Teradata stores dates as integers in the following manner:

    (year - 1900) * 10000 + (month * 100) + day

    Looks like that date integer is done the same way Excel does. It's indexed against January 0th, 1900 - such that 1 comes out as Jan 1, 1900, etc.

    Two different online calculators are giving me answers NEAR Aug 31, 2013, but not exactly:

    http://www.calculator.net/date-calculator.html?today2=01/01/1900&amp;c2op=+&amp;c2year=0&amp;c2month=0&amp;c2week=0&amp;c2day=41517&amp;calctype=op&amp;x=93&amp;y=9
    Sept 3, 2013 (or 2nd, as you can't start at Jan 0, but have to start at Jan 1)

    http://www.timeanddate.com/date/dateadded.html?m1=1&d1=1&y1=1900&type=add&ay=&am=&aw=&ad=41517
    Sept 2, 2013 (or 1st, once again no Jan 0).

    Not sure the underlaying math on either of these, but I think this is the right concept; just need to find the right formula it's using.

  • Options
    bowenbowen How you doin'? Registered User regular
    My rough bowen styled math gave me 7th month, 45 days, 113 years based off that integer.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    urahonkyurahonky Resident FF7R hater Registered User regular
    edited October 2014
    Sorry, ran to lunch. Yes it's a valid date. It's what is put into this combobox that's on this dashboard and what is actually displayed is that integer number.

    Sorry I'm on Vicodin and this didn't make any sense. The data behind the Dashboard has that value but the displayed value on the Dashboard is 31-Aug-2013

    urahonky on
  • Options
    urahonkyurahonky Resident FF7R hater Registered User regular
    You might be onto something about the Excel thing. Since Dashboard uses Excel for it's data bindings.

    When I put 41517 into an Excel cell and change it to date it comes out correctly!

  • Options
    urahonkyurahonky Resident FF7R hater Registered User regular
    edited October 2014
    public static function dateFromExcel(date:Number):Date {
        var sec_ind_day:Number = 86400;
        var secs:Number = sec_ind_day*date%1;
        var _d:Date = new Date(1970, 0, 1 + (date - 25569));
        return new Date(_d.fullYear, _d.month, _d.date, _d.hours, _d.minutes, secs);
    }
    

    Seriously?

    e: Ah, nevermind:
    public static function dateFromExcel(date:Number):Date {
        var _d:Date = new Date(1970, 0, 1 + (date - 25569));
    }
    

    Works supposedly.

    urahonky on
Sign In or Register to comment.