The new forums will be named Coin Return (based on the most recent vote)! You can check on the status and timeline of the transition to the new forums here.
Please vote in the Forum Structure Poll. Polling will close at 2PM EST on January 21, 2025.
Did somebody hack my frigging map?! A [chat] thread.
Donkey Kong
Putting Nintendo out of business with AI nipsRegistered User regular
Putting Nintendo out of business with AI nipsRegistered User regular
I have a little Raspberry Pi embedded computer that I use to drive a bunch of LED arrays mounted behind a map. Nothing special, it's just a 3D printed relief map of the earth with combined elevation and bathymetry data, so you can visualize the size of mountains relative to ocean trenches. It is a project that my husband and I have worked on together intermittently for over a year.
The LED arrays + Raspberry Pi give the map realtime, animated coloration for things like sunlight, moon position, date, etc. It's pretty neat and a lot more fun than a wall clock. If you are not a computer IRL, just enjoy the pretty pictures of the make process. If you are a computer, then enjoy the make photos and this ridiculous troubleshooting mystery.
The 3D printer that started it all. Look at them custom noctua fans on the print head, mmmm
So I complete work on the map on March 31st this year. It runs great as a piece of interactive wall art for a little over a month until a few weeks ago, when the map breaks down. I turn it on to show some guests and the date and time are wrong on the sunlight map overlay. The sun isn't moving. Which is weird because it's supposed to set its clock using internet time. I check my wifi router and the Raspberry Pi is on my network and responding to pings. I open my laptop and try to SSH in (remotely log in using a text terminal) and it refuses my password! No matter what I put for a password, the one I set, variants, the original default, it refuses SSH connections. At this point I am worried it was hacked and had its password changed somehow, so I unplug it to deal with later. It was my wedding day (!) so I had to go. Imagine having to drop an amazing technical mystery for some dumb ceremony.
A few days later, I come back, married, and pick this thing back up. I look online for "Raspberry Pi refuses SSH credentials" and other similar queries and every response was a condescending "ok grandma, so you forgot your password..." I mean, fair, but no, fuck you! I didn't! This is definitely the right password it's just not working!
First test print: A tile containing India with combined elevation/bathymetry data
I take the Pi and plug it into a keyboard and monitor and boot it. It starts up, presents a login prompt, and proceeds to refuse my credentials LOCALLY. Ok that's really not good. Maybe it was hacked. I look online for password reset options and people suggest putting the pi into "single user mode", which is like a safe boot mode where you get access to the command line. You don't have to log in, and you get full access to fix things, reset passwords, etc.
If you're not familiar, the Raspberry Pi's primary storage is a micro sd card that's split into two partitions. One is called "boot" and is FAT formatted (readable and writable from a Mac or Windows PC). This partition is tiny and contains only a handful of configuration files meant to make setting up the raspberry pi easy. From "boot" you can specify wifi info, CPU clock speed, HDMI format, etc.
Several map tiles coming together
The other partition is called "rootfs" and is where the linux OS lives. It is ext4 formatted, which requires custom file system drivers for read-only access under Mac and Windows, and read/write access is discouraged from anything but a Linux machine. Of which the Raspberry Pi was my only linux machine.
Luckily putting the Pi into single user mode can be done through changes to configuration files on the "boot" partition, the one I can easily read and write. So I stuff the SD card into my laptop, and edit the config file. I boot the Pi into single user mode and uh... well at first it won't connect to my keyboard. It says unknown USB device on the command line and hangs. Nobody online has seen this issue before. I hunt around for the keyboard I'd used when I first set things up. I find it, connect it, tap a few keys, and it works! Ok. I can use the command line! I punch in a command, press return and... nothing. The computer does nothing. It doesn't respond and it stops accepting new input.
I try several different flavors of single user mode suggested online. One says "cannot attach to TTY device". Another asks for my root password then tells me it's wrong (it's not).
I sigh and restore all the configuration files back to default, plug the Pi back into the map and power it on. Now it doesn't even come up with the wrong time. It's blank. It's dead. It feels hot. I unplug it. Defeated, I push to another day.
A frame made from 80-20, and mounting the LED panels
I start day 2 with a new plan. I need to dig through the Linux filesystem of the raspberry pi and see if I can spot anything wrong. I need another Linux machine to do this, so I install Mint Linux on a Virtual Machine. I spend an hour troubleshooting VM configuration issues (a hearty fuck you to both virtualbox and mint). I track down a USB SD card reader that I can forward to the VM. I spend another 20 minutes getting that to work. (another hearty fuck you to virtual box).
Finally, I pull the SD card from the raspberry pi, insert it into my computer, and forward it to the VM. After a moment, it mounts and I can see the raspberry pi's entire filesystem, not just the "boot" half. I dig in.
I'm no Linux guru but everything seems normal. The typical folders you'd expect for a linux system are there. My software is all built and configured to run. The config files in /etc seem ok. Eventually I find a file I want to look at that needs system-level access, so I use "sudo nano /etc/passwd" to open it. Right away, Nano, the text editor I'm using, complains that it can't make a backup file because 'there is no available disk space.
Wait a minute. WAAAAAIIIIIT A MINUTE.
I check the rootfs partition. Zero bytes available. Zero bytes of a 32GB SD card that was nearly empty when I set it up! I get a disk space utility and look where all the space has gone.
Frame assembled and map complete
/var/log contains 29GB of data. daemon.log is 12 GB alone. The system logs and log backups make up the rest. I try to read a log file by dumbly double clicking it. The VM freezes hard, but I can see that something is spitting out repeated messages:
Received unknown command ""
Received unknown command ""
Received unknown command ""
(repeating all the way down the screen)
I recognize these messages. They're from the software I wrote to run the visuals for the map. It accepts text commands by UDP packet or by standard input (if running in the foreground on the command line). So this message repeating means either I fucked up or something is on my network spewing blank UDP packets on the port used by my software. I pray it's not the latter because that just creates a fresh new hell.
I force reboot the frozen VM, delete all the logs, and put the SD card back into the Pi. My LED array service happily boots and shows the pretty animated map graphics and the correct date/time! Hooray! I try to log in with SSH. Password refused! Hooray!
An LED systems test. I'd later tune it to be a lot brighter. This is just a real color earth map.
Power map off, SD card back in the laptop. Over the next hour, I discover that while running the system with free disk space at 0, Linux managed to, entirely on its own, trash configuration file at "/etc/shadow". The shadow file is where linux stores your password (encrypted ofc). I couldn't log in not because my password was wrong, but because my account didn't HAVE a password. No accounts had a password anymore! I learn enough linux admin to restore "/etc/shadow" from a backup (thank fuck for this linux distribution automatically backing it up to /var/backup!).
Put card back into pi. Boot pi. Try SSH again. I CAN LOG IN! FINALLY! I check the log files and yes they are growing steadily as long as my map service is running. A megabyte or so a minute. It can run for about 20 days straight and then the card will be full again. Time to fix this thing. I try to stop my service to debug it, which requires sudo (how to tell linux you're permitted to edit system files and end services). Sudo, not my command but sudo itself fails with the message "cannot resolve hostname". What.
Another hour passes. Did you know that if your /etc/hosts file is missing you cannot use sudo over SSH? The system tries to look up if the user "pi@hostname" is permitted to use sudo, but doesn't know that hostname is ITS OWN NAME. So you cannot use sudo. Jesus why did all these files spontaneously delete with low disk space?
A debug overlay, similar to a normal map, testing my code for projecting latitude/longitude data on to the map. All the projection is done in shader language in the Rasbperry Pi's GPU to free the CPU to drive the LED panels
After more linux admin learning, I go off and recreate the "hosts" and "group" configuration files by hand (no backups this time). They must have been lost the same way shadow was. After this, I reboot the pi and behold, I can SSH and sudo while SSH'd.
I finally get to the part where I can debug my app and discover that, mercifully, the blank commands are not coming from some network device spewing UDP packets (as I feared). The mysteries finally start decreasing in number. My network is secure, no Internet of Things device is being dumb, nobody hacked anything.
My app is logging thousands of messages per minute because to process command input from the command line, I'm using std::getline(). In a background process, one not attached to a terminal, getline() returns "false" right away, indicating there is no input device, and provides a blank string to the app. I wasn't checking what getline() returned at all. I hadn't noticed because I did all my testing from a regular terminal that didn't have this behavior.
I add a case to make sure that if getline returns false, I stop trying to call it. The log file stops growing. I build my changes, reboot, and the map is running happily, not to mention using way less CPU now that I don't have a dedicated thread continuously filling my disk with trash!
My map is back! Hopefully it won't break again. All is well.
The map in "day mode" where I simplify the graphics to darkest black and brightest white so you can still see it with lots of ambient light
This was all pretty interesting to me because the symptom presentation was particularly bizarre and it's apparently never happened to anyone else on the internet ever before. All googling told me was "you forgot your password lol". I started doubting my sanity until I noted the full disk and started following the breadcrumbs.
And when single user mode didn't work, google just said "you are doing it wrong also just reflash the pi, you noob". Glad I didn't do that. It would have worked... for about 20 days before the disk filled again. I'd have repeated this again in a month before probably throwing the entire thing in the trash.
Anyway that's my map troubleshooting saga. It better run forever now.
The map in "night mode" where you get a full color earth and light pollution visualization
Oh yeah, it has a voice interface too:
https://www.youtube.com/watch?v=zD00phfiT5c
The LED arrays + Raspberry Pi give the map realtime, animated coloration for things like sunlight, moon position, date, etc. It's pretty neat and a lot more fun than a wall clock. If you are not a computer IRL, just enjoy the pretty pictures of the make process. If you are a computer, then enjoy the make photos and this ridiculous troubleshooting mystery.
The 3D printer that started it all. Look at them custom noctua fans on the print head, mmmm
So I complete work on the map on March 31st this year. It runs great as a piece of interactive wall art for a little over a month until a few weeks ago, when the map breaks down. I turn it on to show some guests and the date and time are wrong on the sunlight map overlay. The sun isn't moving. Which is weird because it's supposed to set its clock using internet time. I check my wifi router and the Raspberry Pi is on my network and responding to pings. I open my laptop and try to SSH in (remotely log in using a text terminal) and it refuses my password! No matter what I put for a password, the one I set, variants, the original default, it refuses SSH connections. At this point I am worried it was hacked and had its password changed somehow, so I unplug it to deal with later. It was my wedding day (!) so I had to go. Imagine having to drop an amazing technical mystery for some dumb ceremony.
Secret Wedding Zone
I don't have many photos from the wedding until the photographer finishes her work, so here's basically everything good that I've got:
My sister pinning on my boutonnière
My sister's fiancee pinning on my husband's boutonnière
Duffy trying on his tux
Duffy refused to pose for any photos so I scooped him up like a baby
Gay wedding selfie with my sister and her fiancee
I am Bowser obviously. Duffy is the piranha plant.
I don't have many photos from the wedding until the photographer finishes her work, so here's basically everything good that I've got:
My sister pinning on my boutonnière
My sister's fiancee pinning on my husband's boutonnière
Duffy trying on his tux
Duffy refused to pose for any photos so I scooped him up like a baby
Gay wedding selfie with my sister and her fiancee
I am Bowser obviously. Duffy is the piranha plant.
A few days later, I come back, married, and pick this thing back up. I look online for "Raspberry Pi refuses SSH credentials" and other similar queries and every response was a condescending "ok grandma, so you forgot your password..." I mean, fair, but no, fuck you! I didn't! This is definitely the right password it's just not working!
First test print: A tile containing India with combined elevation/bathymetry data
I take the Pi and plug it into a keyboard and monitor and boot it. It starts up, presents a login prompt, and proceeds to refuse my credentials LOCALLY. Ok that's really not good. Maybe it was hacked. I look online for password reset options and people suggest putting the pi into "single user mode", which is like a safe boot mode where you get access to the command line. You don't have to log in, and you get full access to fix things, reset passwords, etc.
If you're not familiar, the Raspberry Pi's primary storage is a micro sd card that's split into two partitions. One is called "boot" and is FAT formatted (readable and writable from a Mac or Windows PC). This partition is tiny and contains only a handful of configuration files meant to make setting up the raspberry pi easy. From "boot" you can specify wifi info, CPU clock speed, HDMI format, etc.
Several map tiles coming together
The other partition is called "rootfs" and is where the linux OS lives. It is ext4 formatted, which requires custom file system drivers for read-only access under Mac and Windows, and read/write access is discouraged from anything but a Linux machine. Of which the Raspberry Pi was my only linux machine.
Luckily putting the Pi into single user mode can be done through changes to configuration files on the "boot" partition, the one I can easily read and write. So I stuff the SD card into my laptop, and edit the config file. I boot the Pi into single user mode and uh... well at first it won't connect to my keyboard. It says unknown USB device on the command line and hangs. Nobody online has seen this issue before. I hunt around for the keyboard I'd used when I first set things up. I find it, connect it, tap a few keys, and it works! Ok. I can use the command line! I punch in a command, press return and... nothing. The computer does nothing. It doesn't respond and it stops accepting new input.
I try several different flavors of single user mode suggested online. One says "cannot attach to TTY device". Another asks for my root password then tells me it's wrong (it's not).
I sigh and restore all the configuration files back to default, plug the Pi back into the map and power it on. Now it doesn't even come up with the wrong time. It's blank. It's dead. It feels hot. I unplug it. Defeated, I push to another day.
A frame made from 80-20, and mounting the LED panels
I start day 2 with a new plan. I need to dig through the Linux filesystem of the raspberry pi and see if I can spot anything wrong. I need another Linux machine to do this, so I install Mint Linux on a Virtual Machine. I spend an hour troubleshooting VM configuration issues (a hearty fuck you to both virtualbox and mint). I track down a USB SD card reader that I can forward to the VM. I spend another 20 minutes getting that to work. (another hearty fuck you to virtual box).
Finally, I pull the SD card from the raspberry pi, insert it into my computer, and forward it to the VM. After a moment, it mounts and I can see the raspberry pi's entire filesystem, not just the "boot" half. I dig in.
I'm no Linux guru but everything seems normal. The typical folders you'd expect for a linux system are there. My software is all built and configured to run. The config files in /etc seem ok. Eventually I find a file I want to look at that needs system-level access, so I use "sudo nano /etc/passwd" to open it. Right away, Nano, the text editor I'm using, complains that it can't make a backup file because 'there is no available disk space.
Wait a minute. WAAAAAIIIIIT A MINUTE.
I check the rootfs partition. Zero bytes available. Zero bytes of a 32GB SD card that was nearly empty when I set it up! I get a disk space utility and look where all the space has gone.
Frame assembled and map complete
/var/log contains 29GB of data. daemon.log is 12 GB alone. The system logs and log backups make up the rest. I try to read a log file by dumbly double clicking it. The VM freezes hard, but I can see that something is spitting out repeated messages:
Received unknown command ""
Received unknown command ""
Received unknown command ""
(repeating all the way down the screen)
I recognize these messages. They're from the software I wrote to run the visuals for the map. It accepts text commands by UDP packet or by standard input (if running in the foreground on the command line). So this message repeating means either I fucked up or something is on my network spewing blank UDP packets on the port used by my software. I pray it's not the latter because that just creates a fresh new hell.
I force reboot the frozen VM, delete all the logs, and put the SD card back into the Pi. My LED array service happily boots and shows the pretty animated map graphics and the correct date/time! Hooray! I try to log in with SSH. Password refused! Hooray!
An LED systems test. I'd later tune it to be a lot brighter. This is just a real color earth map.
Power map off, SD card back in the laptop. Over the next hour, I discover that while running the system with free disk space at 0, Linux managed to, entirely on its own, trash configuration file at "/etc/shadow". The shadow file is where linux stores your password (encrypted ofc). I couldn't log in not because my password was wrong, but because my account didn't HAVE a password. No accounts had a password anymore! I learn enough linux admin to restore "/etc/shadow" from a backup (thank fuck for this linux distribution automatically backing it up to /var/backup!).
Put card back into pi. Boot pi. Try SSH again. I CAN LOG IN! FINALLY! I check the log files and yes they are growing steadily as long as my map service is running. A megabyte or so a minute. It can run for about 20 days straight and then the card will be full again. Time to fix this thing. I try to stop my service to debug it, which requires sudo (how to tell linux you're permitted to edit system files and end services). Sudo, not my command but sudo itself fails with the message "cannot resolve hostname". What.
Another hour passes. Did you know that if your /etc/hosts file is missing you cannot use sudo over SSH? The system tries to look up if the user "pi@hostname" is permitted to use sudo, but doesn't know that hostname is ITS OWN NAME. So you cannot use sudo. Jesus why did all these files spontaneously delete with low disk space?
A debug overlay, similar to a normal map, testing my code for projecting latitude/longitude data on to the map. All the projection is done in shader language in the Rasbperry Pi's GPU to free the CPU to drive the LED panels
After more linux admin learning, I go off and recreate the "hosts" and "group" configuration files by hand (no backups this time). They must have been lost the same way shadow was. After this, I reboot the pi and behold, I can SSH and sudo while SSH'd.
I finally get to the part where I can debug my app and discover that, mercifully, the blank commands are not coming from some network device spewing UDP packets (as I feared). The mysteries finally start decreasing in number. My network is secure, no Internet of Things device is being dumb, nobody hacked anything.
My app is logging thousands of messages per minute because to process command input from the command line, I'm using std::getline(). In a background process, one not attached to a terminal, getline() returns "false" right away, indicating there is no input device, and provides a blank string to the app. I wasn't checking what getline() returned at all. I hadn't noticed because I did all my testing from a regular terminal that didn't have this behavior.
I add a case to make sure that if getline returns false, I stop trying to call it. The log file stops growing. I build my changes, reboot, and the map is running happily, not to mention using way less CPU now that I don't have a dedicated thread continuously filling my disk with trash!
My map is back! Hopefully it won't break again. All is well.
The map in "day mode" where I simplify the graphics to darkest black and brightest white so you can still see it with lots of ambient light
This was all pretty interesting to me because the symptom presentation was particularly bizarre and it's apparently never happened to anyone else on the internet ever before. All googling told me was "you forgot your password lol". I started doubting my sanity until I noted the full disk and started following the breadcrumbs.
And when single user mode didn't work, google just said "you are doing it wrong also just reflash the pi, you noob". Glad I didn't do that. It would have worked... for about 20 days before the disk filled again. I'd have repeated this again in a month before probably throwing the entire thing in the trash.
Anyway that's my map troubleshooting saga. It better run forever now.
The map in "night mode" where you get a full color earth and light pollution visualization
Oh yeah, it has a voice interface too:
https://www.youtube.com/watch?v=zD00phfiT5c Thousands of hot, local singles are waiting to play at bubbulon.com.
Donkey Kong on
+96
This discussion has been closed.
Posts
Choose Your Own Chat 1 Choose Your Own Chat 2 Choose Your Own Chat 3
It refused to allow me to submit a post containing the names of the Linux password and shadow files! It straight up blocked me with a security alert!
Also that map is next level rad
Told them what the guy said and thankfully the supervisor agreed we should be doing a lot more work and testing before deciding that somehow they don’t want my money
In this case its my work filter being stupid, but yes.
I thought the photos were what was causing Vanilla to poop the bed at first, so I hosted them at bubbulon.com. Are you at work and your workplace blocks bubbulon? The home page of Easy Tetris?
It is a surprising fickle and arbitrary set of things it decides to block.
touching your own boob
the worst kind of harassment
Realizing you forgot your deodorant
i physically cannot
Cable companies: the worst
I'm not saying I haven't done exactly that gesture...
Is she laughing at top right?
Yeah but what are you gonna do for the next time you get to start a [chat]?
Warframe/Steam: NFyt
It didn’t go over well
Wearing an ill-fitting Lisa Simpsons wig in public.
They have to get the porn on there somehow.
I will resist the temptation to make stupid UDP jokes for now
the "no true scotch man" fallacy.
It's a really weird idea. I think its supposed to be like patreon only streams that were popular with like youtube streamers. But like without the non subs to watch your stream you're cutting out views and ad dollars and potential subs for no real benefit?
pleasepaypreacher.net
Well the idea is to encourage people to sub so they can get in on the occasional sub only stream, right? Presumably if someone's doing one and you try to tune in, it'll prompt you to subscribe first. If you're doing it all the time sure it won't go well, but if you throw one in occasionally where you do some special event, I can see that driving some numbers.
Won't work for small streamers though.
Hollow Knight, or Bloodstained
Or maybe... both?
*froths*
*twitches*
There is sub only chat and sub only vod viewing yeah. Just not sub only live viewing heh.