I am really going back and forth on whether I love or hate Node. On one hand, trying to do realtime game state logic in PHP would probably literally drive me insane.
On the other hand, Promises and the goddamn labyrinth of code they create to do what should be basic step by step calls can go get fucked.
I wouldn't want to do game state logic in either language.
Well to be fair, I'm prototyping out a tabletop game, so I don't have to do a proper game loop. But I'm mostly using it as an excuse to learn the ins and outs of all this nonsense. But then I regret it when I get situations where it feels like my program is violating the laws of causality.
gavindelThe reason all your softwareis brokenRegistered Userregular
Life in a legacy product:
"Hmm, that's weird. Probably shouldn't touch it though."
"No, actually, that's wrong. When did this happen?"
"Hmm, our history literally doesn't go back far enough and it hasn't been changed since Bush was in office."
"It would be super easy if I could just remove it."
*Four days later*
"After exhaustive review with our PM, scenario archaeology, and manual debugging, we have confirmed that I can remove that line of code!"
Code review: "What about Russian localization?"
"....bugger"
I am really going back and forth on whether I love or hate Node. On one hand, trying to do realtime game state logic in PHP would probably literally drive me insane.
On the other hand, Promises and the goddamn labyrinth of code they create to do what should be basic step by step calls can go get fucked.
"Hmm, that's weird. Probably shouldn't touch it though."
"No, actually, that's wrong. When did this happen?"
"Hmm, our history literally doesn't go back far enough and it hasn't been changed since Bush was in office."
"It would be super easy if I could just remove it."
*Four days later*
"After exhaustive review with our PM, scenario archaeology, and manual debugging, we have confirmed that I can remove that line of code!"
Code review: "What about Russian localization?"
"....bugger"
I am really going back and forth on whether I love or hate Node. On one hand, trying to do realtime game state logic in PHP would probably literally drive me insane.
On the other hand, Promises and the goddamn labyrinth of code they create to do what should be basic step by step calls can go get fucked.
In other node.js news, fucking timezones are not making my life any easier, and moment isn’t really helping.
I’m working with a database that stores all its timestamps in UTC. Fine, that’s pretty standard. When I log an activity at 5pm it saves as 2018-04-18 07:00:00. I’m in GMT+10 so all good so far.
How in the hell am I supposed to get this timestamp, coax it into my local timezone, and display it in a properly formatted HH:mm string?
If I use moment-timezone to run moment.tz(“2018etc”, “UTC”).clone().tz(“Australia/Melbourne”).format() I get the same string back with a +10:00 at the end, which doesn’t help one bit.
Am I wrong in thinking this should be a solved problem? Am I missing some really obvious step here? From what I can work out of the documentation, I’m doing everything I need to convert a timestamp from UTC to my local. It feels like such a blindingly obvious use-case but all I can seem to do is convert from my local timezone back to UTC.
[edit: it works if I create another moment object with the MySQL-formatted date string, parse it into a YYYY-MM-DD HH:mm:ss format, and then feed that into the tz call. I'm not sure what the difference is since the MySQL string doesn't include a timestamp either, but I can live with this for now I guess.
Sorry, was phone-posting so was probably unclear. I can get a formatted string out fine, the issue was that the moment object passed to that function didn't update the timestamp to reflect its new timezone.
Hmm. I have this Prometheus query to show network usage for services:
sort_desc(sum by (name) (rate(container_network_transmit_bytes_total[1m] ) ))
I have two of these, one for input, one for output. I thought I'd be clever and show them in the same Grafana panel, so I did this which works in Prometheus:
-(sort_desc(sum by (name) (rate(container_network_transmit_bytes_total[1m] ) )))
However, that doesn't work in Grafana, apparently. Anyone know some other way to invert the values in a prometheus query in Grafana?
Sorry, was phone-posting so was probably unclear. I can get a formatted string out fine, the issue was that the moment object passed to that function didn't update the timestamp to reflect its new timezone.
... that said, that 01:46:54 should be 11:46:54, if it's 10 hours off my current time of 9:46pm. But I'm still getting the result I expect, so...
You know, this strikes me as an oddly familiar issue. I dealt with this or something similar a few months back. I’ll try to dig up the project and see if I can remember how I solved it tomorrow (going to sleep now)
Quick Angular (5) question: is it OK (as in: A Good Practice™) to have a component that's basically only called for is HTML template? Or should I look into something different?
( < . . .
0
Options
gavindelThe reason all your softwareis brokenRegistered Userregular
More seriously, an open balance was left on a vendor because of incorrect calculations. I work on financial software, so our customers take our calculations quite serious.
GnomeTankWhat the what?Portland, OregonRegistered Userregular
edited April 2018
Re: Promises and async/await.
Keep in mind that async/await are effectively sugar over promises, so you aren't suddenly out of the promise business because of async/await. They certainly put some syntactic sugar around things but they also have language semantics that cause awaits to be serial. Take the following code:
function foo() {
return new Promise((resolve, reject) =>
setTimeout(() => {
console.log('foo()');
resolve('Hello');
}, 1000));
}
function bar() {
console.log('bar()');
return Promise.resolve('World');
}
async function doStuff1() {
// Serial
const x = await foo();
const y = await bar();
console.log(`${x} ${y}`);
}
async function doStuff2() {
// Concurrent
const [x, y] = await Promise.all([foo(), bar()]);
console.log(`${x} ${y}`);
}
If you call doStuff1(), you will get a pause, foo(), bar(), Hello World.
If you call doStuff2(), you will get bar(), pause, foo(), Hello World.
In addition async/await puts no syntactic wrappers around things like Promise.map, Promise.reduce, Promise.each, Promise.props, etc. In "real world" applications these are heavily used when dealing with fan out style concurrency, which is common. Async/await are nice and you should use them but understand their limitations and that they are just sugar over promises (technically they are sugar over promise generators but the distinction is mostly semantic). Learning to write promise code correctly, so that you're not over-nesting promise chains and so they are readable can only help you even if you do use async/await when it makes sense. Promises are not just some "JS thing" anymore. Most modern languages, from Rust, to C#, to Go have popular "Futures" libraries or extensions and some are considering it for core language inclusion (see C#'s ContinueWith in the TPL. Look familiar?). Learning how to program in the paradigm is a good skill to learn as we move to a world where concurrency is the default operating mode of software you write rather than an after thought.
Learning parallelism with javascript is like getting your driver's license on a carriage.
Fixed that for you. Node is perfectly fine at being concurrent. Basically all modern languages are starting to expose safe concurrent paradigms. Node does it with a single thread, an event loop and promises/futures. Go uses channels and green threads. Rust has ownership/borrowing and enforces no data races at compile time with green threads libraries that do futures. Yeah if you're wanting to learn real parallelism with fun things like spin locks than Node is not the thing to do that. If you just want to learn how to think about concurrent code, Node is perfectly fine and the lessons you learn will immediately translate to something like Rust or Go.
Thanks for the fix, but no.
The languages lacks basic constructs to enable concurrency(e.g. channels) despite it being obvious for 3 years now that the model is incomplete. Node is not fine at all as a platform to learn concurrency on. The lessons will translate to an actual concurrent platform as much as arm wrestling would translate to greco-roman.
Nobody should ever learn concurrency on node and believe that what they are doing is anything different than slinging poop at walls and checking what sticks.
I cannot stress this enough, but THANK YOU for pointing out the Async/Await and the explanations. You just cleared up shit that's been hounding me for days. That was the last little bit I needed, because I just could NOT understand the whole .then() chain and I think for what I was doing it was needlessly complex. An hour later and I have the causality loop that was throwing me cleared up.
TL;DR: Is there an easy way to determine if a media file stored in an S3 bucket is audio or video?
Long version: We currently have a process where users can upload media files, which we then do some additional processing on. When a file hits our bucket, we run a celery task that pulls the file down, creates an mp3 of the audio, and then checks if the file is video and if so starts an elastic transcode job to transcode it to a consistent size/format. So for every file a user uploads we create a standardized mp3 file, and possibly a standardized video file. This works fine for our current setup where it's mostly audio uploads with the occasional video, as even the largest current uploads it usually only takes 8-10 seconds to pull it down locally to do the mp3 conversion. However, we have a new customer interested in using our portal that wants to upload 30-40GB files. Now were talking like 10 minutes just to pull the file down, and then suffering that same transfer time again when it is submitted to elastic transcode.
So to avoid the local transfer time we're just going to start doing both processes through ET. However when I create the job, I only want to specify a video output if the original file was actually video. Right now we determine this using ffmpg, but as far as I can tell it doesn't support getting info for a remote file. If I'm wrong and this is possible let me know! I thought about doing something like using file extension but that seems kind of....hokey? Plus it seems like every week we get some kind of weird extension that "just works" with the current ffmpg method, but if anyone knows of a python library that would be able to do this let me know.
Why can't the job be "do the transcode" and you determine if the video output is required after you pull the file down? You need to pull the file down either way and it should be identical up to that point?
Why can't the job be "do the transcode" and you determine if the video output is required after you pull the file down? You need to pull the file down either way and it should be identical up to that point?
This was the way we started out, pulling the file down locally and generating both the audio/video. However as people started uploading more/larger files, we offloaded the video processing to AWS Elastic Transcode, since the speed they could transcode the video with more then offset the fact that the file needed to also be pulled down by their service. This is the setup we're currently using.
However with the new potential customer it's just not really feasible from a throughput perspective for the size of files they want to use. It takes ~10 minutes to pull the file down locally, then if it's a video another ~10 minutes for AWS to pull down their copy for Elastic Transcode, plus the time to do the actual transcoding. Most of the files we process currently take less then 5 minutes to do everything, so it's hard to sell a 5x increase in processing time for files that are the same duration, although much larger size, then what we do now.
I'm trying to move all of the file generation out of our local system entirely and do it all in AWS, so that only one copy of the file needs to be pulled down by them. When you submit the file to Elastic Transcode you specify the output formats you want, but in order to know the output formats I need to know if the original file is audio or video, and I can't seem to find a way to do that without needing to pull the entire file down locally, which defeats the purpose of what i'm trying to do.
Well, my most recent adventures with Promises were with Ionic 2, that is mostly Angular 2 in Typescript. Turns out that loading.dismiss() returns a promise to ensure that whatever comes next is executed after the dismiss animation is finished. So, had this bug when if I switched views without a .then() (like, let's say, after getting data from the server for the view), then the animation didn't finish (because the event was running on the controller of the previous view) and the app got "stuck" with the loading popup over it.
Why wouldn't extension checking work, though? If someone uploads a video with the filename "music.mp3", then is your app really expected to handle that?
Turns out that it's super easy to not spot a missing "rows.Close()" in a Go code review and suddenly you have something hogging all the database connections.
Turns out that it's super easy to not spot a missing "rows.Close()" in a Go code review and suddenly you have something hogging all the database connections.
I love when languages have features like
with(database.open()){
//something
}
that automatically closes the connection when you leave the with block.
Yeah, Go has a sorta-equivalent with the "defer" keyword that runs when the function exits. Normal routine would be:
func DoStuff(db *sql.Db) {
rows, err := db.Query(...)
if err != nil {
panic("DB shitting the bed")
}
// Closes "rows" connection on function return
defer rows.Close()
// Do whatever with result rows
...
// rows.Close() executes here
}
Super easy to not spot that it's missing when doing reviews since it's not a block like in other languages.
Echo on
0
Options
OrcaAlso known as EspressosaurusWrexRegistered Userregular
Languages without deterministic destructors make baby Orca cry.
+3
Options
gavindelThe reason all your softwareis brokenRegistered Userregular
doing the first major test of my new product tomorrow
first test a few months ago was 500 accounts
this test.. 40,000 (!!!)
so far, no real disasters have occurred under the new data load
as an aside, I've been messing around with tools/side projects on Heroku, and Heroku is like.. really a lot better than I found it 6-8 years ago. As long as its weaknesses are understood I feel like it's kinda good now if you're just wanting to get a hobby online (apart from the fact that they've raised the price substantially to match the improved service)
i got a django/postgres/S3 project online in less than 2 hours completely blind
Yeah, Go has a sorta-equivalent with the "defer" keyword that runs when the function exits. Normal routine would be:
func DoStuff(db *sql.Db) {
rows, err := db.Query(...)
if err != nil {
panic("DB shitting the bed")
}
// Closes "rows" connection on function return
defer rows.Close()
// Do whatever with result rows
...
// rows.Close() executes here
}
Super easy to not spot that it's missing when doing reviews since it's not a block like in other languages.
This is the sort of thing where you really want a type system powerful enough to check that resources are being appropriately closed. Making it easier to do the right thing helps a little, but making it easier to be aware you're not doing the right thing is so much more valuable.
0
Options
The Escape Goatincorrigible ruminantthey/themRegistered Userregular
Is Firefox's debugger garbage and slow for anyone else? Like, even IE's performs way better for me.
So in the game I'm porting I had a bit of a conundrum. The game originally ran in a resolution of 320x200 at an aspect ratio of 4:3 (The pixels were not square back then). Today's displays do have square pixels so 320x200 works out to a 16:10 aspect ratio. I have long since quadrupled the working resolution to 1280x800, but as monitors are largely 16:9 now and support higher resolutions, there was a question of what to do about the border.
When I was at GDC, I met someone demoing a retro game he wrote that he intentionally displayed in 4:3. To fit the retro late 80s aesthetic, he had moving neon paint splatter in the non-rendered area of the 16:9 display. It looked really cool, so I told him that I was going to completely steal his idea for my own game. He said it was cool.
In my game, I wanted a moving starfield in my border. I tried first with a simple repeating space texture that moved x+1 and y+1 every frame. It looked kind of bland and you could see it tile. I was looking for something a little more rotate-y and less tile-ish.
I decided that I would use a 3D skybox that rolled and panned though a full panoramic view of a nebula. It surprisingly only took two days to implement, but I feel like I overkilled a simple problem. The upshot is that my borders look effing sweet!
The actual GUI theme is using the old graphics. (The game itself is theme-able) I'm going to give those a facelift before I enter beta. Also there exists a windowed mode and a stretch to fullscreen option too.
I looks even cooler when it's moving (and in 1920x1080) because the bright stars "winkle" in luminosity. I'm not using any filtering on the space texture so you kind of get that pixel scale/rotate effect, but ever so slightly as the resolution is so large. There are pictures on the blog in my sig of course.
Posts
I mean, it works, but also c'mon brain.
On the other hand, Promises and the goddamn labyrinth of code they create to do what should be basic step by step calls can go get fucked.
Well to be fair, I'm prototyping out a tabletop game, so I don't have to do a proper game loop. But I'm mostly using it as an excuse to learn the ins and outs of all this nonsense. But then I regret it when I get situations where it feels like my program is violating the laws of causality.
"Hmm, that's weird. Probably shouldn't touch it though."
"No, actually, that's wrong. When did this happen?"
"Hmm, our history literally doesn't go back far enough and it hasn't been changed since Bush was in office."
"It would be super easy if I could just remove it."
*Four days later*
"After exhaustive review with our PM, scenario archaeology, and manual debugging, we have confirmed that I can remove that line of code!"
Code review: "What about Russian localization?"
"....bugger"
async / await, my dude.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/await
And what was the value in removing the code?
I’m working with a database that stores all its timestamps in UTC. Fine, that’s pretty standard. When I log an activity at 5pm it saves as 2018-04-18 07:00:00. I’m in GMT+10 so all good so far.
How in the hell am I supposed to get this timestamp, coax it into my local timezone, and display it in a properly formatted HH:mm string?
If I use moment-timezone to run moment.tz(“2018etc”, “UTC”).clone().tz(“Australia/Melbourne”).format() I get the same string back with a +10:00 at the end, which doesn’t help one bit.
Am I wrong in thinking this should be a solved problem? Am I missing some really obvious step here? From what I can work out of the documentation, I’m doing everything I need to convert a timestamp from UTC to my local. It feels like such a blindingly obvious use-case but all I can seem to do is convert from my local timezone back to UTC.
[edit: it works if I create another moment object with the MySQL-formatted date string, parse it into a YYYY-MM-DD HH:mm:ss format, and then feed that into the tz call. I'm not sure what the difference is since the MySQL string doesn't include a timestamp either, but I can live with this for now I guess.
What I had before:
moment.tz(date_from_db, "UTC").clone().tz("Australia/Melbourne").format("YYYY-MM-DD HH:mm")
where date_from_db is "2018-04-18T01:46:54.000Z"
What ended up working:
moment.tz(moment(date_from_db).format("YYYY-MM-DD HH:mm"), "UTC").tz("Australia/Melbourne").format("YYYY-MM-DD HH:mm")
... that said, that 01:46:54 should be 11:46:54, if it's 10 hours off my current time of 9:46pm. But I'm still getting the result I expect, so...
I have two of these, one for input, one for output. I thought I'd be clever and show them in the same Grafana panel, so I did this which works in Prometheus:
However, that doesn't work in Grafana, apparently. Anyone know some other way to invert the values in a prometheus query in Grafana?
You know, this strikes me as an oddly familiar issue. I dealt with this or something similar a few months back. I’ll try to dig up the project and see if I can remember how I solved it tomorrow (going to sleep now)
It was wrong.
More seriously, an open balance was left on a vendor because of incorrect calculations. I work on financial software, so our customers take our calculations quite serious.
Keep in mind that async/await are effectively sugar over promises, so you aren't suddenly out of the promise business because of async/await. They certainly put some syntactic sugar around things but they also have language semantics that cause awaits to be serial. Take the following code:
function foo() { return new Promise((resolve, reject) => setTimeout(() => { console.log('foo()'); resolve('Hello'); }, 1000)); } function bar() { console.log('bar()'); return Promise.resolve('World'); } async function doStuff1() { // Serial const x = await foo(); const y = await bar(); console.log(`${x} ${y}`); } async function doStuff2() { // Concurrent const [x, y] = await Promise.all([foo(), bar()]); console.log(`${x} ${y}`); }If you call doStuff1(), you will get a pause, foo(), bar(), Hello World.
If you call doStuff2(), you will get bar(), pause, foo(), Hello World.
In addition async/await puts no syntactic wrappers around things like Promise.map, Promise.reduce, Promise.each, Promise.props, etc. In "real world" applications these are heavily used when dealing with fan out style concurrency, which is common. Async/await are nice and you should use them but understand their limitations and that they are just sugar over promises (technically they are sugar over promise generators but the distinction is mostly semantic). Learning to write promise code correctly, so that you're not over-nesting promise chains and so they are readable can only help you even if you do use async/await when it makes sense. Promises are not just some "JS thing" anymore. Most modern languages, from Rust, to C#, to Go have popular "Futures" libraries or extensions and some are considering it for core language inclusion (see C#'s ContinueWith in the TPL. Look familiar?). Learning how to program in the paradigm is a good skill to learn as we move to a world where concurrency is the default operating mode of software you write rather than an after thought.
Fixed that for you. Node is perfectly fine at being concurrent. Basically all modern languages are starting to expose safe concurrent paradigms. Node does it with a single thread, an event loop and promises/futures. Go uses channels and green threads. Rust has ownership/borrowing and enforces no data races at compile time with green threads libraries that do futures. Yeah if you're wanting to learn real parallelism with fun things like spin locks than Node is not the thing to do that. If you just want to learn how to think about concurrent code, Node is perfectly fine and the lessons you learn will immediately translate to something like Rust or Go.
The languages lacks basic constructs to enable concurrency(e.g. channels) despite it being obvious for 3 years now that the model is incomplete. Node is not fine at all as a platform to learn concurrency on. The lessons will translate to an actual concurrent platform as much as arm wrestling would translate to greco-roman.
Nobody should ever learn concurrency on node and believe that what they are doing is anything different than slinging poop at walls and checking what sticks.
TL;DR: Is there an easy way to determine if a media file stored in an S3 bucket is audio or video?
Long version: We currently have a process where users can upload media files, which we then do some additional processing on. When a file hits our bucket, we run a celery task that pulls the file down, creates an mp3 of the audio, and then checks if the file is video and if so starts an elastic transcode job to transcode it to a consistent size/format. So for every file a user uploads we create a standardized mp3 file, and possibly a standardized video file. This works fine for our current setup where it's mostly audio uploads with the occasional video, as even the largest current uploads it usually only takes 8-10 seconds to pull it down locally to do the mp3 conversion. However, we have a new customer interested in using our portal that wants to upload 30-40GB files. Now were talking like 10 minutes just to pull the file down, and then suffering that same transfer time again when it is submitted to elastic transcode.
So to avoid the local transfer time we're just going to start doing both processes through ET. However when I create the job, I only want to specify a video output if the original file was actually video. Right now we determine this using ffmpg, but as far as I can tell it doesn't support getting info for a remote file. If I'm wrong and this is possible let me know! I thought about doing something like using file extension but that seems kind of....hokey? Plus it seems like every week we get some kind of weird extension that "just works" with the current ffmpg method, but if anyone knows of a python library that would be able to do this let me know.
I found this (https://aws.amazon.com/blogs/compute/extracting-video-metadata-using-lambda-and-mediainfo/) which seems along the lines of what I want to do, however just looking at the mediainfo page I can see some of the formats that I know people are already using are not listed, so I'm going to give it a shot but I don't think it'll end up working.
Right now i'll probably just end up going down the file extension path, but I'd welcome other suggestions.
This was the way we started out, pulling the file down locally and generating both the audio/video. However as people started uploading more/larger files, we offloaded the video processing to AWS Elastic Transcode, since the speed they could transcode the video with more then offset the fact that the file needed to also be pulled down by their service. This is the setup we're currently using.
However with the new potential customer it's just not really feasible from a throughput perspective for the size of files they want to use. It takes ~10 minutes to pull the file down locally, then if it's a video another ~10 minutes for AWS to pull down their copy for Elastic Transcode, plus the time to do the actual transcoding. Most of the files we process currently take less then 5 minutes to do everything, so it's hard to sell a 5x increase in processing time for files that are the same duration, although much larger size, then what we do now.
I'm trying to move all of the file generation out of our local system entirely and do it all in AWS, so that only one copy of the file needs to be pulled down by them. When you submit the file to Elastic Transcode you specify the output formats you want, but in order to know the output formats I need to know if the original file is audio or video, and I can't seem to find a way to do that without needing to pull the entire file down locally, which defeats the purpose of what i'm trying to do.
That was a fun one to solve.
Why wouldn't extension checking work, though? If someone uploads a video with the filename "music.mp3", then is your app really expected to handle that?
I love when languages have features like
with(database.open()){ //something }that automatically closes the connection when you leave the with block.
func DoStuff(db *sql.Db) { rows, err := db.Query(...) if err != nil { panic("DB shitting the bed") } // Closes "rows" connection on function return defer rows.Close() // Do whatever with result rows ... // rows.Close() executes here }Super easy to not spot that it's missing when doing reviews since it's not a block like in other languages.
How much garbage data are you going to read at the end of this sentence
first test a few months ago was 500 accounts
this test.. 40,000 (!!!)
so far, no real disasters have occurred under the new data load
as an aside, I've been messing around with tools/side projects on Heroku, and Heroku is like.. really a lot better than I found it 6-8 years ago. As long as its weaknesses are understood I feel like it's kinda good now if you're just wanting to get a hobby online (apart from the fact that they've raised the price substantially to match the improved service)
i got a django/postgres/S3 project online in less than 2 hours completely blind
When I was at GDC, I met someone demoing a retro game he wrote that he intentionally displayed in 4:3. To fit the retro late 80s aesthetic, he had moving neon paint splatter in the non-rendered area of the 16:9 display. It looked really cool, so I told him that I was going to completely steal his idea for my own game. He said it was cool.
In my game, I wanted a moving starfield in my border. I tried first with a simple repeating space texture that moved x+1 and y+1 every frame. It looked kind of bland and you could see it tile. I was looking for something a little more rotate-y and less tile-ish.
I decided that I would use a 3D skybox that rolled and panned though a full panoramic view of a nebula. It surprisingly only took two days to implement, but I feel like I overkilled a simple problem. The upshot is that my borders look effing sweet!
The actual GUI theme is using the old graphics. (The game itself is theme-able) I'm going to give those a facelift before I enter beta. Also there exists a windowed mode and a stretch to fullscreen option too.