You can easily mock out the SDK service clients by taking advantage of Go’s interfaces. By using the methods of an interface, your code can use that interface instead of using the concrete service client directly.
Having implemented an awful version of a WCF SOAP client in Dotnet Core 2.1 for a specific endpoint, I wish these fellows the very best of luck (in keeping their sanity).
Hoo boy, I've run into a real sticky wicky and I don't know how to code my way out of this one... This may be a bit verbose as I don't know what the call the problem other than. "I can't save a pointer to disk" Which doesn't make any sense, so I'll have to elaborate... I'll spoiler the code so it doesn't clog up the forum.
To start out, here's the basics you should know..
The game I'm porting has data saved in structs.
The game was written when an int was 16 bits
Since I'm having issues with the missiles I'll paste the relevant code here. This is the struct.
typedef struct MissStru * MissPtr;
typedef char * Ptr;
struct MissStru { /* Missile/EBW Data Structure */
MissPtr prev; /* Previous missile */
MissPtr link; /* Link to next missile */
int type; /* Type >=0 is missile, < 0 is EBW */
long x, y; /* Current position */
long dx, dy; /* Velocity vectors */
long range; /* Remaining range */
int lockid; /* ID of target */
Ptr lockto; /* Pointer to target */
int locktype; /* Type of target (HOTP,HFWS,HENS) */
int size; /* Beam size for EBW's */
int firedfrom; /* <0 = UDP >0 = FW */
int power; /* If EBW, power level */
int radius; /* Proximity fuse detonation radius */
int pinpoint; /* System number pinpointed */
};
Here's the problem. In the original game, it would save the game data by just writing the whole struct as a binary blob to disk like so.
z = NumLinks(Game->missiles); //how many missiles
fwrite(&z,2,1,df); /write that many
ptri = Game->missiles; //point to top of struct and dump to disk
for(y = 0; y < z; y++) {
fwrite(ptri,(int)sizeof(struct MissStru),1,df);
ptri = ptri->link;
}
The code to load the data is the exact reverse.
fread(&num,2,1,df); //read how many missiles
for(x = 0; x < num; x++) { //read in that many
tempi = Game->missiles;
Game->missiles = (MissPtr)MemBlock((int)sizeof(struct MissStru));
fread(Game->missiles,(int)sizeof(struct MissStru),1,df);
Game->missiles->link = tempi;
Game->missiles->prev = NULL;
if(tempi != NULL) tempi->prev = Game->missiles;
}
The issue is that if I were to save the game on a system with different endian-ness or type lengths, the data files would be incompatible. The big problem is that these files were written in DOS and I need to read them in 64 bit windows, so the type size has changed, and the game is not portable. My solution was to manually serialize the data, but there is a struct member I obviously serialized wrong, and I don't know what is supped to go there. Here is how I manually saved the data.
//note ---> write 16le == Write 16 bit little endian number
//write Missle data
z = NumLinks((B2Ptr)Game->missiles);
al_fwrite16le(df, z);
ptri = Game->missiles;
for (y = 0; y < z; y++) {
al_fwrite32le(df, 0xffffffff); //skip MissPtr prev
al_fwrite32le(df, 0xffffffff); //skip MissPtr link
al_fwrite16le(df, Game->missiles->type);
al_fwrite32le(df, Game->missiles->x);
al_fwrite32le(df, Game->missiles->y);
al_fwrite32le(df, Game->missiles->dx);
al_fwrite32le(df, Game->missiles->dx);
al_fwrite32le(df, Game->missiles->range);
al_fwrite16le(df, Game->missiles->lockid);
al_fwrite32le(df, 0xffffffff); //lockto;
al_fwrite16le(df, Game->missiles->locktype);
al_fwrite16le(df, Game->missiles->size);
al_fwrite16le(df, Game->missiles->firedfrom);
al_fwrite16le(df, Game->missiles->power);
al_fwrite16le(df, Game->missiles->radius);
al_fwrite16le(df, Game->missiles->pinpoint);
ptri = ptri->link;
}
Here is the issue. When the game loads it tends to fill out the pointer data on it's own so I put dummy data in there. However, I don't know what to put in the "lockto" spot, and I don't know how to save it. I know it's needed because when I load a game that has a missile lock, it will crash with a null pointer. Here's code that uses "lockto"
m = Game->missiles;
while (m != NULL) {
head = 0;
switch (m->locktype) {
case B303_OUT: //if outpost
x = ((PostPtr)m->lockto)->x; //<--- crashes here with a null pointer
y = ((PostPtr)m->lockto)->y;
break;
case B303_FWS: //if good guy ship
x = ((FwgPtr)m->lockto)->pos.x;
y = ((FwgPtr)m->lockto)->pos.y;
x += ((FwgPtr)m->lockto)->pos.dx * elapsed / 200;
y += ((FwgPtr)m->lockto)->pos.dy * elapsed / 200;
break;
case B303_ENS: //if enemy ship
x = ((EngPtr)m->lockto)->pos.x;
y = ((EngPtr)m->lockto)->pos.y;
x += ((EngPtr)m->lockto)->pos.dx * elapsed / 200;
y += ((EngPtr)m->lockto)->pos.dy * elapsed / 200;
break;
//...
So it's a pointer into other pointers? I'm not quite understanding what I'm looking at here. I tried to break it down as best I could. How would I serialize such a beast?
Pointer to pointers, now that's fun. I haven't seen that in a decade.
Here's the real question, do you need to keep it the same? Working with files sucks, but it sucks even more when you're using a serializing system built in the 80s and 90s. I'd say rewrite it using yaml or json or something instead. Keep the old code in place and build a whole new serialization system and plug it in. This way you don't need to worry about pointers to pointers or struct packing.
With a rewrite you make it easier on yourself going forward, too. Without all the rest of the code, it's going to be hard for us to tell you what's going on. The old way of doing this used IDs to pin point what exactly was being serialized via pointers/pointer-pointers, but I couldn't tell you because we've only got about 70% of the code.
not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
Is the lock pointer always to the same collection of objects? If it’s just one collection then you could store an index in the field instead into that collection during writing and restitch it back up (like the linked list) during load.
If you have multiple types of objects in different lists then you’d need to write both a list identifier of some type and the index into that list.
If you have identifiers that uniquely identify all objects you can lock to then you could also store those and search for the memory pointer of the referenced object.
There are quite a few techniques without even changing your file strategy.
Assuming everything else is bounded to 16-bit memory and you're deadset on parsing the original serialization format, what you basically need to do is figure out a consistent translation scheme for the pointers.
Based on what you've shown, an original save file has all the information needed to reconstruct what was where in the linked lists - i.e. when reading the original files, you would end up with a map of every object's original pointer values via the link and prev fields.
So if you modify the load to build a map of "saved ptr values -> real ptr locations", then you can walk the linked lists after they're loaded and update the pointers with their real memory values.
Then in your save logic, for consistency, you'd implement the back translation - convert everything to 16-bit pointer addresses (with the linked lists you'd walk, determine an array size, and then write everything out as though it was all offsets into 1 big 16-bit array).
+1
Options
OrcaAlso known as EspressosaurusWrexRegistered Userregular
The core problem is that at minimum you need to traverse the entire structure and re-populate the pointers with valid data. Your serialization loses information; lockto is serialized with invalid data! Given 5 seconds of thought, I I think you need to bite the bullet and make some changes here, because there isn't enough information in the struct to reconstitute it once you wipe the pointer data.
You've been given a few ways to solve that problem. Here's one more.
Just spitballing, you could add some metadata to each serialized entry of MissStruct that includes data that allows you to restore lockto. Maybe that means an index representing the type and a second index indicating the offset into the array of those types. I dunno. Not enough code has been posted to say. But something along those lines.
Basically for all your objects you need to do something like this on load to handle the legacy format (this is far from efficient but it should be more readable)...
fread(&num,2,1,df); //read how many missiles
// Store the 16-bit pointers here
uint16* diskLookups[num];
// Store the concrete pointers here
MissPtr* memLookups[num];
// Holds the previous entry's 16-bit "link" ptr so we can find out what the current
// entry ptr value was.
uint16 diskPrevNext = NULL;
for(x = 0; x < num; x++) { //read in that many
tempi = Game->missiles;
Game->missiles = (MissPtr)MemBlock((int)sizeof(struct MissStru));
// At this point we have a common ID (x) to build a translation with.
// Store the *real* ptr to this
memLookups[x] = Game->missiles;
// Load original format object
fread(Game->missiles,(int)sizeof(struct MissStru),1,df);
// Current object must have originally had the value of the serialized objects
// link field
diskLookups[x] = diskPrevNext;
// Update the stored previous value (16 bit ptr == 16 bit uint)
diskPrevNext = Game->missiles->link
// Handle the case where this might be the second object, and we need to
// get the ptr value for the first in the diskLookup table.
if (x == 1) {
diskLookups[0] = Game->missiles->prev;
}
Game->missiles->link = tempi;
Game->missiles->prev = NULL;
if(tempi != NULL) tempi->prev = Game->missiles;
}
// Somewhere after you do this, you walk the linked-list again and fix up all the lockedto pointers by finding their values in
// diskLookups and replacing with the value in the same array position in memLookups
// Also better code would use a hash table, but I doubt you will ever have a problem with speed from this implementation
The missing element here is you should probably define a LegacyMissStru as the original format which is sized correctly, rather then what I do here which is just assume that the pointer casts will work out.
You also would need to do this when saving - which would be quite similar, but you'd just invent a 16-bit array offset format, since in this code it's just a lookup anyway.
EDIT: Personally if I were debugging this I'd start by writing myself a command line client which did test loads and then cross-checked I got everything right, but it's eminently doable to handle this and support the original format IMO.
EDIT 2: Also, probably you should only implement the loading logic to import old saves. and define a "new" format that you can extend for new saves.
Heh, months ago when I manually serialized the data, I was doing it for the backward compatibility for the user-editable data. Missions, Solar system, ships, captains, and enemies. Turns out I don't need to maintain backward compatibility the the save files as they are not items you would share. (Actually on Android, they are tucked away from the user).
What I did was just save the missile struct the "old skool" way by just copying the data out
z = NumLinks((B2Ptr)Game->missiles);
al_fwrite16le(df, z);
ptri = Game->missiles;
for (y = 0; y < z; y++) {
al_fwrite(df, ptri, (int)sizeof(struct MissStru));
ptri = ptri->link;
}
and load
//read Missle data
num = al_fread16le(df);
for (x = 0; x < num; x++) {
tempi = Game->missiles;
Game->missiles = (MissPtr)MemBlock((int)sizeof(struct MissStru));
al_fread(df,Game->missiles, (int)sizeof(struct MissStru));
Game->missiles->link = tempi;
Game->missiles->prev = NULL;
if (tempi != NULL) tempi->prev = Game->missiles;
}
It didn't work and still crashed
When stepping though the code I realized it's now saving the "lockto" pointer in the blob... but... it's just saving the address, right? When I load it, it's the same address that was saved, but what good is that if it's a new game and all the data has been moved around?
Now keep in mind, how lockto works is somewhat a mystery for me. The variable is used deep in code that I didn't write. I guess I just don't understand what lockto even *is* . It appears to pull data from different structs based on how it was cast. I'm not understanding this black magic. (See my above post under final spoiler to see how it's used in game.) Sadly, your solutions are not very helpful because I don't know how it works or what's it's doing or even how I'm supposed to use it.
I was thinking about just not saving missile data at all, but then you can cheese the game by saving when a missile is on you and then loading again and the missile would be gone.
Now keep in mind, how lockto works is somewhat a mystery for me. The variable is used deep in code that I didn't write. I guess I just don't understand what lockto even *is* . It appears to pull data from different structs based on how it was cast. I'm not understanding this black magic. (See my above post under final spoiler to see how it's used in game.) Sadly, your solutions are not very helpful because I don't know how it works or what's it's doing or even how I'm supposed to use it.
I was thinking about just not saving missile data at all, but then you can cheese the game by saving when a missile is on you and then loading again and the missile would be gone.
Wait.. I think I see what you mean.. point the lockto to the beginning of the struct of whatever locktype(enemy, outpost, or friendly) and lockid(the number in that list) are.
I think I have it.
Yeah that's why you have prev/next pointers. That's a doubly linked list and they're there so other functions can walk up and down and find the component they're looking for (and then serialize it if necessary).
Though that's just a guess.
not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
The whole universe is linked lists. The game came out before vectors were a thing. I know where the linked list access functions are. I'm implementing the seek code now.
i just finished my data structures course in C so it was all about implementing linked lists , dynamic arrays, hashmaps, etc. it was real fun. fun to see stuff like that.
i'm gonna make a stretch here but linked lists are also kinda gross nowadays because even though they have constant time additions the scattering across memory could cause cache misses in modern CPUs which you wouldn't get in a structure like a vector/dynamic array as those are typically stored in contiguous memory
i'm gonna make a stretch here but linked lists are also kinda gross nowadays because even though they have constant time additions the scattering across memory could cause cache misses in modern CPUs which you wouldn't get in a structure like a vector/dynamic array as those are typically stored in contiguous memory
but that's just a guess
Most languages with built in linked lists have them backed by dynamic arrays under the hood.
Yes if you're storing hundreds of millions of records in a linked list you should probably think about what you're doing for more than 10 seconds.
what data structure would be best? AVL tree? because what if you wanted to sort the records, or something. a hashmap is good but it's awful to sort. Like, I'm still new to programming but like when you have a huge database like that do you just like, chunk up the database to bring it into memory?
Yes if you're storing hundreds of millions of records in a linked list you should probably think about what you're doing for more than 10 seconds.
what data structure would be best? AVL tree? because what if you wanted to sort the records, or something. a hashmap is good but it's awful to sort. Like, I'm still new to programming but like when you have a huge database like that do you just like, chunk up the database to bring it into memory?
You can make a really well performing linked list as long as you use a custom allocation strategy (that holds state) that basically converts it to a vector with a bit mask.
+1
Options
Monkey Ball WarriorA collection of mediocre hatsSeattle, WARegistered Userregular
You can easily mock out the SDK service clients by taking advantage of Go’s interfaces. By using the methods of an interface, your code can use that interface instead of using the concrete service client directly.
Ah yes, I'll just mock the
*checks notes*
282 functions in the S3 interface.
Thank science for auto-generated mocks.
Man that's horrible advice. Nobody is going to do that.
There are a few local versions of aws services out there, s3 => minio, dynamodb => dynamodb local, I was looking at one the other day Twitch wrote for kinesis that I don't remember the name of.
Using them will make your tests slow and fragile, but at least you won't have to implement your own, crappy version of SomeComplexServiceClient just to test your code.
Oh God, after 12 hours of hunting and coding and I had to revert the whole mess. I'm so frustrated I want pound my keyboard against my display so bad I can taste it.
After writing code to rebuild the lockto pointers, I run the game and find all the pointers are all nulled out after the game loads. Turns out, there is code in the game that rebuilds the lockto pointers and I didn't have to write anything! Sadly, this code is broken because to fill out the pointer data it does some validation checks on the ship positions and other garbage I don't feel like chasing down right now.
At least I found the broken function. I'll drill down on it tomorrow.
Posts
Ah yes, I'll just mock the
*checks notes*
282 functions in the S3 interface.
Thank science for auto-generated mocks.
Having implemented an awful version of a WCF SOAP client in Dotnet Core 2.1 for a specific endpoint, I wish these fellows the very best of luck (in keeping their sanity).
http://steamcommunity.com/id/pablocampy
Does it have Atom / Sublime equivalents like Command Palette and fuzzy match search?
Fuzzy match would be like
'#fms' will find a file called 'fuzzy_match_strategy.rb'
http://docs.sublimetext.info/en/latest/file_management/file_navigation.html#goto-anything
To start out, here's the basics you should know..
The game I'm porting has data saved in structs.
The game was written when an int was 16 bits
Since I'm having issues with the missiles I'll paste the relevant code here. This is the struct.
Here's the real question, do you need to keep it the same? Working with files sucks, but it sucks even more when you're using a serializing system built in the 80s and 90s. I'd say rewrite it using yaml or json or something instead. Keep the old code in place and build a whole new serialization system and plug it in. This way you don't need to worry about pointers to pointers or struct packing.
With a rewrite you make it easier on yourself going forward, too. Without all the rest of the code, it's going to be hard for us to tell you what's going on. The old way of doing this used IDs to pin point what exactly was being serialized via pointers/pointer-pointers, but I couldn't tell you because we've only got about 70% of the code.
If you have multiple types of objects in different lists then you’d need to write both a list identifier of some type and the index into that list.
If you have identifiers that uniquely identify all objects you can lock to then you could also store those and search for the memory pointer of the referenced object.
There are quite a few techniques without even changing your file strategy.
Based on what you've shown, an original save file has all the information needed to reconstruct what was where in the linked lists - i.e. when reading the original files, you would end up with a map of every object's original pointer values via the link and prev fields.
So if you modify the load to build a map of "saved ptr values -> real ptr locations", then you can walk the linked lists after they're loaded and update the pointers with their real memory values.
Then in your save logic, for consistency, you'd implement the back translation - convert everything to 16-bit pointer addresses (with the linked lists you'd walk, determine an array size, and then write everything out as though it was all offsets into 1 big 16-bit array).
You've been given a few ways to solve that problem. Here's one more.
Just spitballing, you could add some metadata to each serialized entry of MissStruct that includes data that allows you to restore lockto. Maybe that means an index representing the type and a second index indicating the offset into the array of those types. I dunno. Not enough code has been posted to say. But something along those lines.
The missing element here is you should probably define a LegacyMissStru as the original format which is sized correctly, rather then what I do here which is just assume that the pointer casts will work out.
You also would need to do this when saving - which would be quite similar, but you'd just invent a 16-bit array offset format, since in this code it's just a lookup anyway.
EDIT: Personally if I were debugging this I'd start by writing myself a command line client which did test loads and then cross-checked I got everything right, but it's eminently doable to handle this and support the original format IMO.
EDIT 2: Also, probably you should only implement the loading logic to import old saves. and define a "new" format that you can extend for new saves.
What I did was just save the missile struct the "old skool" way by just copying the data out and load It didn't work and still crashed
When stepping though the code I realized it's now saving the "lockto" pointer in the blob... but... it's just saving the address, right? When I load it, it's the same address that was saved, but what good is that if it's a new game and all the data has been moved around?
Now keep in mind, how lockto works is somewhat a mystery for me. The variable is used deep in code that I didn't write. I guess I just don't understand what lockto even *is* . It appears to pull data from different structs based on how it was cast. I'm not understanding this black magic. (See my above post under final spoiler to see how it's used in game.) Sadly, your solutions are not very helpful because I don't know how it works or what's it's doing or even how I'm supposed to use it.
I was thinking about just not saving missile data at all, but then you can cheese the game by saving when a missile is on you and then loading again and the missile would be gone.
I found your problem!
I think I have it.
Though that's just a guess.
but that's just a guess
Half the underlying data structures one uses have a linked list in their implementation.
Most languages with built in linked lists have them backed by dynamic arrays under the hood.
what data structure would be best? AVL tree? because what if you wanted to sort the records, or something. a hashmap is good but it's awful to sort. Like, I'm still new to programming but like when you have a huge database like that do you just like, chunk up the database to bring it into memory?
B-trees.
So... trees for the indexes. :rotate:
feel free to dump in on us with some code and a github or something I always enjoy reading people doing these kinds of things
Man that's horrible advice. Nobody is going to do that.
There are a few local versions of aws services out there, s3 => minio, dynamodb => dynamodb local, I was looking at one the other day Twitch wrote for kinesis that I don't remember the name of.
Using them will make your tests slow and fragile, but at least you won't have to implement your own, crappy version of SomeComplexServiceClient just to test your code.
Here's a travis config for a kindof silly ddb util library I wrote that uses dynamodb local to test: https://github.com/shawnsmithdev/ddbmap/blob/master/.travis.yml
After writing code to rebuild the lockto pointers, I run the game and find all the pointers are all nulled out after the game loads. Turns out, there is code in the game that rebuilds the lockto pointers and I didn't have to write anything! Sadly, this code is broken because to fill out the pointer data it does some validation checks on the ship positions and other garbage I don't feel like chasing down right now.
At least I found the broken function. I'll drill down on it tomorrow.
This blows!