As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/
Options

[Programming] Reinventing equality, one language at a time

18485878990100

Posts

  • Options
    electricitylikesmeelectricitylikesme Registered User regular
    It's early days but someone might've written a good library to walk to WCF with Python - https://python-zeep.readthedocs.io/en/master/.

  • Options
    EchoEcho ski-bap ba-dapModerator mod
    *grumbles over Amazon's Go AWS SDK*
    You can easily mock out the SDK service clients by taking advantage of Go’s interfaces. By using the methods of an interface, your code can use that interface instead of using the concrete service client directly.

    Ah yes, I'll just mock the

    *checks notes*

    282 functions in the S3 interface.

    Thank science for auto-generated mocks.

  • Options
    CampyCampy Registered User regular
    It's early days but someone might've written a good library to walk to WCF with Python - https://python-zeep.readthedocs.io/en/master/.

    Having implemented an awful version of a WCF SOAP client in Dotnet Core 2.1 for a specific endpoint, I wish these fellows the very best of luck (in keeping their sanity).

  • Options
    tyrannustyrannus i am not fat Registered User regular
    VsCode is an actually great editor. I'm just really happy with this thing.

  • Options
    zeenyzeeny Registered User regular
    I was surprised how much better than Atom it is.

  • Options
    a5ehrena5ehren AtlantaRegistered User regular
    Yeah, the first few versions of VS Code were awful, like Atom. It is much, much better now. I'm using it in place of Eclipse.

  • Options
    templewulftemplewulf The Team Chump USARegistered User regular
    a5ehren wrote: »
    Yeah, the first few versions of VS Code were awful, like Atom. It is much, much better now. I'm using it in place of Eclipse.

    Does it have Atom / Sublime equivalents like Command Palette and fuzzy match search?

    Twitch.tv/FiercePunchStudios | PSN | Steam | Discord | SFV CFN: templewulf
  • Options
    SpoitSpoit *twitch twitch* Registered User regular
    templewulf wrote: »
    a5ehren wrote: »
    Yeah, the first few versions of VS Code were awful, like Atom. It is much, much better now. I'm using it in place of Eclipse.

    Does it have Atom / Sublime equivalents like Command Palette and fuzzy match search?
    I'm not 100% sure what fuzzy match search is, but it does support regex searches

    steam_sig.png
  • Options
    templewulftemplewulf The Team Chump USARegistered User regular
    Spoit wrote: »
    templewulf wrote: »
    a5ehren wrote: »
    Yeah, the first few versions of VS Code were awful, like Atom. It is much, much better now. I'm using it in place of Eclipse.

    Does it have Atom / Sublime equivalents like Command Palette and fuzzy match search?
    I'm not 100% sure what fuzzy match search is, but it does support regex searches

    Fuzzy match would be like

    '#fms' will find a file called 'fuzzy_match_strategy.rb'

    http://docs.sublimetext.info/en/latest/file_management/file_navigation.html#goto-anything

    Twitch.tv/FiercePunchStudios | PSN | Steam | Discord | SFV CFN: templewulf
  • Options
    SaerisSaeris Borb Enthusiast flapflapflapflapRegistered User regular
    I'm pretty sure fuzzy matching just takes whatever input you have and inserts a regex like \w* after every character.

    borb_sig.png
  • Options
    halkunhalkun Registered User regular
    edited December 2018
    Hoo boy, I've run into a real sticky wicky and I don't know how to code my way out of this one... This may be a bit verbose as I don't know what the call the problem other than. "I can't save a pointer to disk" Which doesn't make any sense, so I'll have to elaborate... I'll spoiler the code so it doesn't clog up the forum.

    To start out, here's the basics you should know..

    The game I'm porting has data saved in structs.
    The game was written when an int was 16 bits

    Since I'm having issues with the missiles I'll paste the relevant code here. This is the struct.
    typedef struct MissStru *  MissPtr;
    typedef char *             Ptr;
    
    struct MissStru {	 /* Missile/EBW Data Structure */
    	MissPtr prev;	 /* Previous missile */
    	MissPtr link;	 /* Link to next missile */
    	int      type;	 /* Type >=0 is missile, < 0 is EBW */
    	long    x, y;	 /* Current position */
    	long    dx, dy;	 /* Velocity vectors */
    	long    range;	 /* Remaining range */
    	int      lockid;	 /* ID of target */
    	Ptr     lockto;	 /* Pointer to target */
    	int      locktype;	 /* Type of target (HOTP,HFWS,HENS) */
    	int      size;	 /* Beam size for EBW's */
    	int      firedfrom;	 /* <0 = UDP >0 = FW */
    	int      power;	 /* If EBW, power level */
    	int      radius;	 /* Proximity fuse detonation radius */
    	int      pinpoint;	 /* System number pinpointed */
    };
    
    Here's the problem. In the original game, it would save the game data by just writing the whole struct as a binary blob to disk like so.
       z = NumLinks(Game->missiles);  //how many missiles
       fwrite(&z,2,1,df);  /write that many
       ptri = Game->missiles;  //point to top of struct and dump to disk
       for(y = 0; y < z; y++) {
          fwrite(ptri,(int)sizeof(struct MissStru),1,df);
          ptri = ptri->link;
          }
    
    The code to load the data is the exact reverse.
       fread(&num,2,1,df);  //read how many missiles
       for(x = 0; x < num; x++) {  //read in that many
          tempi = Game->missiles;
          Game->missiles = (MissPtr)MemBlock((int)sizeof(struct MissStru));
          fread(Game->missiles,(int)sizeof(struct MissStru),1,df);
          Game->missiles->link = tempi;
          Game->missiles->prev = NULL;
          if(tempi != NULL) tempi->prev = Game->missiles;
          }
    
    The issue is that if I were to save the game on a system with different endian-ness or type lengths, the data files would be incompatible. The big problem is that these files were written in DOS and I need to read them in 64 bit windows, so the type size has changed, and the game is not portable. My solution was to manually serialize the data, but there is a struct member I obviously serialized wrong, and I don't know what is supped to go there. Here is how I manually saved the data.
    	//note ---> write 16le == Write 16 bit little endian number
    
    	//write Missle data
    	z = NumLinks((B2Ptr)Game->missiles);
    	al_fwrite16le(df, z);
    	ptri = Game->missiles;
    	for (y = 0; y < z; y++) {
    		al_fwrite32le(df, 0xffffffff);		//skip MissPtr prev
    		al_fwrite32le(df, 0xffffffff);		//skip MissPtr link
    		al_fwrite16le(df, Game->missiles->type);
    		al_fwrite32le(df, Game->missiles->x);
    		al_fwrite32le(df, Game->missiles->y);
    		al_fwrite32le(df, Game->missiles->dx);
    		al_fwrite32le(df, Game->missiles->dx);
    		al_fwrite32le(df, Game->missiles->range);
    		al_fwrite16le(df, Game->missiles->lockid);
    		al_fwrite32le(df, 0xffffffff);                    //lockto;
    		al_fwrite16le(df, Game->missiles->locktype);
    		al_fwrite16le(df, Game->missiles->size);
    		al_fwrite16le(df, Game->missiles->firedfrom);
    		al_fwrite16le(df, Game->missiles->power);
    		al_fwrite16le(df, Game->missiles->radius);
    		al_fwrite16le(df, Game->missiles->pinpoint);
    		ptri = ptri->link;
    	}
    
    Here is the issue. When the game loads it tends to fill out the pointer data on it's own so I put dummy data in there. However, I don't know what to put in the "lockto" spot, and I don't know how to save it. I know it's needed because when I load a game that has a missile lock, it will crash with a null pointer. Here's code that uses "lockto"
    	m = Game->missiles;
    	while (m != NULL) {
    		head = 0;
    		switch (m->locktype) {
    		case B303_OUT:   //if outpost
    			x = ((PostPtr)m->lockto)->x;    //<--- crashes here with a null pointer
    			y = ((PostPtr)m->lockto)->y;
    			break;
    		case B303_FWS:  //if good guy ship
    			x = ((FwgPtr)m->lockto)->pos.x;
    			y = ((FwgPtr)m->lockto)->pos.y;
    			x += ((FwgPtr)m->lockto)->pos.dx * elapsed / 200;
    			y += ((FwgPtr)m->lockto)->pos.dy * elapsed / 200;
    			break;
    		case B303_ENS:  //if enemy ship
    			x = ((EngPtr)m->lockto)->pos.x;
    			y = ((EngPtr)m->lockto)->pos.y;
    			x += ((EngPtr)m->lockto)->pos.dx * elapsed / 200;
    			y += ((EngPtr)m->lockto)->pos.dy * elapsed / 200;
    			break;
    //...
    
    So it's a pointer into other pointers? I'm not quite understanding what I'm looking at here. I tried to break it down as best I could. How would I serialize such a beast?





    halkun on
  • Options
    bowenbowen How you doin'? Registered User regular
    Pointer to pointers, now that's fun. I haven't seen that in a decade.

    Here's the real question, do you need to keep it the same? Working with files sucks, but it sucks even more when you're using a serializing system built in the 80s and 90s. I'd say rewrite it using yaml or json or something instead. Keep the old code in place and build a whole new serialization system and plug it in. This way you don't need to worry about pointers to pointers or struct packing.

    With a rewrite you make it easier on yourself going forward, too. Without all the rest of the code, it's going to be hard for us to tell you what's going on. The old way of doing this used IDs to pin point what exactly was being serialized via pointers/pointer-pointers, but I couldn't tell you because we've only got about 70% of the code.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    InfidelInfidel Heretic Registered User regular
    Is the lock pointer always to the same collection of objects? If it’s just one collection then you could store an index in the field instead into that collection during writing and restitch it back up (like the linked list) during load.

    If you have multiple types of objects in different lists then you’d need to write both a list identifier of some type and the index into that list.

    If you have identifiers that uniquely identify all objects you can lock to then you could also store those and search for the memory pointer of the referenced object.

    There are quite a few techniques without even changing your file strategy.

    OrokosPA.png
  • Options
    electricitylikesmeelectricitylikesme Registered User regular
    Assuming everything else is bounded to 16-bit memory and you're deadset on parsing the original serialization format, what you basically need to do is figure out a consistent translation scheme for the pointers.

    Based on what you've shown, an original save file has all the information needed to reconstruct what was where in the linked lists - i.e. when reading the original files, you would end up with a map of every object's original pointer values via the link and prev fields.

    So if you modify the load to build a map of "saved ptr values -> real ptr locations", then you can walk the linked lists after they're loaded and update the pointers with their real memory values.

    Then in your save logic, for consistency, you'd implement the back translation - convert everything to 16-bit pointer addresses (with the linked lists you'd walk, determine an array size, and then write everything out as though it was all offsets into 1 big 16-bit array).

  • Options
    OrcaOrca Also known as Espressosaurus WrexRegistered User regular
    The core problem is that at minimum you need to traverse the entire structure and re-populate the pointers with valid data. Your serialization loses information; lockto is serialized with invalid data! Given 5 seconds of thought, I I think you need to bite the bullet and make some changes here, because there isn't enough information in the struct to reconstitute it once you wipe the pointer data.

    You've been given a few ways to solve that problem. Here's one more.

    Just spitballing, you could add some metadata to each serialized entry of MissStruct that includes data that allows you to restore lockto. Maybe that means an index representing the type and a second index indicating the offset into the array of those types. I dunno. Not enough code has been posted to say. But something along those lines.

  • Options
    electricitylikesmeelectricitylikesme Registered User regular
    edited December 2018
    Basically for all your objects you need to do something like this on load to handle the legacy format (this is far from efficient but it should be more readable)...
    fread(&num,2,1,df);  //read how many missiles
    
    // Store the 16-bit pointers here
    uint16* diskLookups[num];
    
    // Store the concrete pointers here
    MissPtr* memLookups[num];
    
    // Holds the previous entry's 16-bit "link" ptr so we can find out what the current
    // entry ptr value was.
    uint16 diskPrevNext = NULL;
    
    for(x = 0; x < num; x++) {  //read in that many
      tempi = Game->missiles;
      Game->missiles = (MissPtr)MemBlock((int)sizeof(struct MissStru));
      
      // At this point we have a common ID (x) to build a translation with.
    
      // Store the *real* ptr to this
      memLookups[x] = Game->missiles;
      
      // Load original format object
      fread(Game->missiles,(int)sizeof(struct MissStru),1,df);
    
      // Current object must have originally had the value of the serialized objects
      // link field
      diskLookups[x] = diskPrevNext;
      
      // Update the stored previous value (16 bit ptr == 16 bit uint)
      diskPrevNext = Game->missiles->link
      
      // Handle the case where this might be the second object, and we need to
      // get the ptr value for the first in the diskLookup table.
      if (x == 1) {
        diskLookups[0] = Game->missiles->prev;
      }
      
      Game->missiles->link = tempi;
      Game->missiles->prev = NULL;
      if(tempi != NULL) tempi->prev = Game->missiles;
    }
    
    // Somewhere after you do this, you walk the linked-list again and fix up all the lockedto pointers by finding their values in
    // diskLookups and replacing with the value in the same array position in memLookups
    // Also better code would use a hash table, but I doubt you will ever have a problem with speed from this implementation
    
    

    The missing element here is you should probably define a LegacyMissStru as the original format which is sized correctly, rather then what I do here which is just assume that the pointer casts will work out.

    You also would need to do this when saving - which would be quite similar, but you'd just invent a 16-bit array offset format, since in this code it's just a lookup anyway.

    EDIT: Personally if I were debugging this I'd start by writing myself a command line client which did test loads and then cross-checked I got everything right, but it's eminently doable to handle this and support the original format IMO.

    EDIT 2: Also, probably you should only implement the loading logic to import old saves. and define a "new" format that you can extend for new saves.

    electricitylikesme on
  • Options
    halkunhalkun Registered User regular
    edited December 2018
    Heh, months ago when I manually serialized the data, I was doing it for the backward compatibility for the user-editable data. Missions, Solar system, ships, captains, and enemies. Turns out I don't need to maintain backward compatibility the the save files as they are not items you would share. (Actually on Android, they are tucked away from the user).

    What I did was just save the missile struct the "old skool" way by just copying the data out
    	z = NumLinks((B2Ptr)Game->missiles);
    	al_fwrite16le(df, z);
    	ptri = Game->missiles;
    	for (y = 0; y < z; y++) {
    		al_fwrite(df, ptri, (int)sizeof(struct MissStru));
    		ptri = ptri->link;
    	}
    
    and load
    //read Missle data
    	num = al_fread16le(df);
    	for (x = 0; x < num; x++) {
    		tempi = Game->missiles;
    		Game->missiles = (MissPtr)MemBlock((int)sizeof(struct MissStru));
    		al_fread(df,Game->missiles, (int)sizeof(struct MissStru));
    		Game->missiles->link = tempi;
    		Game->missiles->prev = NULL;
    		if (tempi != NULL) tempi->prev = Game->missiles;
    	}
    
    It didn't work and still crashed :(

    When stepping though the code I realized it's now saving the "lockto" pointer in the blob... but... it's just saving the address, right? When I load it, it's the same address that was saved, but what good is that if it's a new game and all the data has been moved around?

    Now keep in mind, how lockto works is somewhat a mystery for me. The variable is used deep in code that I didn't write. I guess I just don't understand what lockto even *is* . It appears to pull data from different structs based on how it was cast. I'm not understanding this black magic. (See my above post under final spoiler to see how it's used in game.) Sadly, your solutions are not very helpful because I don't know how it works or what's it's doing or even how I'm supposed to use it.

    I was thinking about just not saving missile data at all, but then you can cheese the game by saving when a missile is on you and then loading again and the missile would be gone.

    halkun on
  • Options
    PhyphorPhyphor Building Planet Busters Tasting FruitRegistered User regular
    Presumably, since you have locktype and lockid, you could walk your list of stuff and set the pointer appropriately

  • Options
    OrcaOrca Also known as Espressosaurus WrexRegistered User regular
    edited December 2018
    halkun wrote: »
    Now keep in mind, how lockto works is somewhat a mystery for me. The variable is used deep in code that I didn't write. I guess I just don't understand what lockto even *is* . It appears to pull data from different structs based on how it was cast. I'm not understanding this black magic. (See my above post under final spoiler to see how it's used in game.) Sadly, your solutions are not very helpful because I don't know how it works or what's it's doing or even how I'm supposed to use it.

    I was thinking about just not saving missile data at all, but then you can cheese the game by saving when a missile is on you and then loading again and the missile would be gone.

    I found your problem! :D

    Orca on
  • Options
    halkunhalkun Registered User regular
    Wait.. I think I see what you mean.. point the lockto to the beginning of the struct of whatever locktype(enemy, outpost, or friendly) and lockid(the number in that list) are.
    I think I have it.

  • Options
    bowenbowen How you doin'? Registered User regular
    Yeah that's why you have prev/next pointers. That's a doubly linked list and they're there so other functions can walk up and down and find the component they're looking for (and then serialize it if necessary).

    Though that's just a guess.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    tyrannustyrannus i am not fat Registered User regular
    is that a linked list of missiles? that's so neat

  • Options
    halkunhalkun Registered User regular
    The whole universe is linked lists. The game came out before vectors were a thing. I know where the linked list access functions are. I'm implementing the seek code now.

  • Options
    tyrannustyrannus i am not fat Registered User regular
    i just finished my data structures course in C so it was all about implementing linked lists , dynamic arrays, hashmaps, etc. it was real fun. fun to see stuff like that.

  • Options
    bowenbowen How you doin'? Registered User regular
    Linked lists are by far the most common data structure I see, hashmaps/dictionaries closely behind it.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    tyrannustyrannus i am not fat Registered User regular
    edited December 2018
    i'm gonna make a stretch here but linked lists are also kinda gross nowadays because even though they have constant time additions the scattering across memory could cause cache misses in modern CPUs which you wouldn't get in a structure like a vector/dynamic array as those are typically stored in contiguous memory

    but that's just a guess

    tyrannus on
  • Options
    PhyphorPhyphor Building Planet Busters Tasting FruitRegistered User regular
    Yes linked lists are about as bad for performance on a modern CPU as you can get

  • Options
    zeenyzeeny Registered User regular
    Mate mate mate mate mate.....

    Half the underlying data structures one uses have a linked list in their implementation.
    ...so does the other half.

  • Options
    LD50LD50 Registered User regular
    tyrannus wrote: »
    i'm gonna make a stretch here but linked lists are also kinda gross nowadays because even though they have constant time additions the scattering across memory could cause cache misses in modern CPUs which you wouldn't get in a structure like a vector/dynamic array as those are typically stored in contiguous memory

    but that's just a guess

    Most languages with built in linked lists have them backed by dynamic arrays under the hood.

  • Options
    bowenbowen How you doin'? Registered User regular
    Yes if you're storing hundreds of millions of records in a linked list you should probably think about what you're doing for more than 10 seconds.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    tyrannustyrannus i am not fat Registered User regular
    bowen wrote: »
    Yes if you're storing hundreds of millions of records in a linked list you should probably think about what you're doing for more than 10 seconds.

    what data structure would be best? AVL tree? because what if you wanted to sort the records, or something. a hashmap is good but it's awful to sort. Like, I'm still new to programming but like when you have a huge database like that do you just like, chunk up the database to bring it into memory?

  • Options
    hippofanthippofant ティンク Registered User regular
    tyrannus wrote: »
    bowen wrote: »
    Yes if you're storing hundreds of millions of records in a linked list you should probably think about what you're doing for more than 10 seconds.

    what data structure would be best? AVL tree? because what if you wanted to sort the records, or something. a hashmap is good but it's awful to sort. Like, I'm still new to programming but like when you have a huge database like that do you just like, chunk up the database to bring it into memory?

    B-trees.

  • Options
    LD50LD50 Registered User regular
    The real answer is an actual database.

  • Options
    EchoEcho ski-bap ba-dapModerator mod
    LD50 wrote: »
    The real answer is an actual database.

    So... trees for the indexes. :rotate:

  • Options
    tyrannustyrannus i am not fat Registered User regular
    edited December 2018
    I think I am gonna try and make a B-Tree implementation in python to kinda grok the whole concept. It looks pretty sweet. Winter break schminter break

    tyrannus on
  • Options
    bowenbowen How you doin'? Registered User regular
    tyrannus wrote: »
    I think I am gonna try and make a B-Tree implementation in python to kinda grok the whole concept. It looks pretty sweet. Winter break schminter break

    feel free to dump in on us with some code and a github or something I always enjoy reading people doing these kinds of things

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    EtheaEthea Registered User regular
    You can make a really well performing linked list as long as you use a custom allocation strategy (that holds state) that basically converts it to a vector with a bit mask.

  • Options
    Monkey Ball WarriorMonkey Ball Warrior A collection of mediocre hats Seattle, WARegistered User regular
    edited December 2018
    Echo wrote: »
    *grumbles over Amazon's Go AWS SDK*
    You can easily mock out the SDK service clients by taking advantage of Go’s interfaces. By using the methods of an interface, your code can use that interface instead of using the concrete service client directly.

    Ah yes, I'll just mock the

    *checks notes*

    282 functions in the S3 interface.

    Thank science for auto-generated mocks.

    Man that's horrible advice. Nobody is going to do that.

    There are a few local versions of aws services out there, s3 => minio, dynamodb => dynamodb local, I was looking at one the other day Twitch wrote for kinesis that I don't remember the name of.

    Using them will make your tests slow and fragile, but at least you won't have to implement your own, crappy version of SomeComplexServiceClient just to test your code.

    Here's a travis config for a kindof silly ddb util library I wrote that uses dynamodb local to test: https://github.com/shawnsmithdev/ddbmap/blob/master/.travis.yml

    Monkey Ball Warrior on
    "I resent the entire notion of a body as an ante and then raise you a generalized dissatisfaction with physicality itself" -- Tycho
  • Options
    EchoEcho ski-bap ba-dapModerator mod
    Yeah, we use the Minio Go package for S3 stuff. We seriously don't need the entire gargantuan kitchen sink.

  • Options
    halkunhalkun Registered User regular
    Oh God, after 12 hours of hunting and coding and I had to revert the whole mess. I'm so frustrated I want pound my keyboard against my display so bad I can taste it.

    After writing code to rebuild the lockto pointers, I run the game and find all the pointers are all nulled out after the game loads. Turns out, there is code in the game that rebuilds the lockto pointers and I didn't have to write anything! Sadly, this code is broken because to fill out the pointer data it does some validation checks on the ship positions and other garbage I don't feel like chasing down right now.

    At least I found the broken function. I'll drill down on it tomorrow.

    This blows!

This discussion has been closed.