Dynamic content systems used by large websites..

Recoil42

.. and various things like that. Basically, I'm wondering how these things work. I'm learning PHP right now, and it's cool, but still limited to execution only when specifically called, and you're restricted to .php files... and I just wonder how I can take it further...

Some examples:

- What powers Digg? Like, the addresses don't end in .php or .asp, right? (ie: http://digg.com/programming/Dot_Clock_A_dot_for_every_second_in_the_day ) so what powers it?

- Hell, penny-arcade's comic section, for example? I'm sure they don't manually create an entire htm file and redirect www.penny-arcade.com/comic to it every time they up a new comic..

- How does YTMND recompile itself, automatically by itself, every single night?

- How does myGamerCard work, where when you link what seems to be a static file ( ie, http://card.mygamercard.net/sig/jdarksun.png )


Just to be clear... I know how PHP, ASP, etc. work. I know how dynamic sites generate a whole page. What I'm asking about is specific things, like how to trigger an event not when someone accesses a file, but rather at a certain time of day.

devoir

I'm not a web developer, but I keep an eye on stuff like this as much as I can.

Databases, PHP, Perl, Python.

Digg: http://computer.howstuffworks.com/digg.htm

Penny Arcade: They'll have an interface where they upload an image file, set the date, tags, and when they want it to appear. Whenever you go to /comic it makes a call to see when is the most up to date comic it is allowed to show, and shows you the appropriate one. Most likely PHP.

YTMND: I have no idea about this, I've only seen a couple of them and haven't paid attention to the backend of the site.

myGamerCard: In the same way that when you request a .php file you don't get the source code, when you request that particular file it generates the image on the fly via database calls and whatnot.

misbehavin

Files don't have to end in .php to be executed by the PHP engine. That's just the default. You can use anything. In Apache, you just set:

AddType application/x-httpd-php .{WHAT YOU WANT HERE}

I won't go into IIS instructions, because I'm just showing this for effect.

Also, Penny Arcade just had a database, and the /comic page just pulls the latest comic from the database and displays it.

MySpace, Google and other HUGE distributed systems use a much different approach to managing data, and I won't go into depth here, but there is enormous info on the Google File System (as one example) out there. Like this: http://labs.google.com/papers/gfs.html

Jasconius

Just because it doesn't have a file extension doesn't mean it's not a regular old tech.

Dynamic content is basically run by pages written in PHP, ASP, or other scripting languages. They request literal content from a database and output it in preset patterns.

So, you have a database with the field "content" that you submit "blah blah blah" to

So, instead of your HTML page printing

<p>blah blah blah</p>

You have

<p>(call the variable)</p>

Where the variable is assigned earlier by calling the field in the database using SQL and assigning the value of that field to the variable.

Hope that makes sense.

misbehavin

Example:

<?

mysql_select_db("database",$db);

$query = "SELECT first_name,last_name FROM users WHERE id = {$user_id};";
$get_names_q = mysql_query($query,$db);
$get_names_r = mysql_fetch_array($get_names_q);
$first_name = $get_names_r["first_name"];
$first_name = $get_names_r["first_name"];
mysql_free_result($get_names_q);

print "Welcome to my site, {$first_name} {$last_name}!";

?>

Recoil42

Just to be clear... I know how PHP, ASP, etc. work. I know how dynamic sites generate a whole page. Hell, my sig that you see right now is self-coded PHP. What I'm asking about is specific things, like how to trigger an event not when someone accesses a file, but rather at a certain time of day, or when a certain non-php/asp file is opened.

RoundBoy

PA runs ruby on rails now (the front page anyway)

Trigger an event? The web browser is always used to trigger things on page views, and you look up a time ot some other criteria.

To do things sans web browser interface, you need to have a backend script that runs via CRON job or some other schedule interface.

I have a perl script that periodically runs mySQL backups and archives data off, as well as deletes old inactive accounts. Your backend script is entirely dependent on yopur hosting, your access to your web box, and your os.

misbehavin

Recoil42 wrote: »

Just to be clear... I know how PHP, ASP, etc. work. I know how dynamic sites generate a whole page. Hell, my sig that you see right now is self-coded PHP. What I'm asking about is specific things, like how to trigger an event not when someone accesses a file, but rather at a certain time of day, or when a certain non-php/asp file is opened.

That's more than likely not PHP code, but rather scripts/apps that run on the server-side that are executed at certain times of day, or constantly monitoring the database for changes. At least, that's how I do it. I have Perl scripts that execute from a scheduler on my LAMP machine at specific times.

Also, PHP has the ability to execute programs on the server machine. That can be used to run event-specific programs.

http://us2.php.net/exec

Jasconius

Recoil42 wrote: »

Just to be clear... I know how PHP, ASP, etc. work. I know how dynamic sites generate a whole page. Hell, my sig that you see right now is self-coded PHP. What I'm asking about is specific things, like how to trigger an event not when someone accesses a file, but rather at a certain time of day, or when a certain non-php/asp file is opened.

Well, if it used ASP or such it would still have to be *.asp or whatever.

Alternatively, JavaScripts can be used to perform small, basically insignificant functions without having any special language or file extension.

An example would be using JavaScript to get the time of day, and form a conditional so that if its breakfast time is says "go eat breakfast" or lunchtime, go eat lunch, etc....

That's not very powerful stuff, though, not sure what you really want based on your posts.

Æthelred

The URLs you're wondering about are done through Mod Rewrite. Basically you just set a rule so that a certain URL directs to the real page. For example, digg, if they use php, would go like:

http://digg.com/programming/Linux_is_cool >>> http://digg.com/story.php?s=Linux_is_cool

The first is what you see in your browser, the second is where you actually are.

misbehavin

Jasconius wrote: »

Well, if it used ASP or such it would still have to be *.asp or whatever.

Once again, no it doesn't.

Recoil42

RoundBoy wrote:

To do things sans web browser interface, you need to have a backend script that runs via CRON job or some other schedule interface.

I have a perl script that periodically runs mySQL backups and archives data off, as well as deletes old inactive accounts. Your backend script is entirely dependent on yopur hosting, your access to your web box, and your os.

Ok, that sounds a lot more like what I'm looking for and want to learn about. So yeah... where can I learn about something like that?

Jimmy King

Recoil42 wrote: »

RoundBoy wrote:

To do things sans web browser interface, you need to have a backend script that runs via CRON job or some other schedule interface.

I have a perl script that periodically runs mySQL backups and archives data off, as well as deletes old inactive accounts. Your backend script is entirely dependent on yopur hosting, your access to your web box, and your os.

Ok, that sounds a lot more like what I'm looking for and want to learn about. So yeah... where can I learn about something like that?

The programming is just plain old programming, nothing special about it. Then you just configure cron (I'm assuming this is linux/bsd since you're dealing with php).

Recoil42

Awesome, looking at it and messing around with it in cpanel, this looks exactly like what I'm looking for.

So the commands are executed command-line-like at the shell? IE, it's all standard unix/linux commands?

edit: oh fuck me awesome, yeah, that's exactly what I was trying to learn.

http://www.tech-geeks.org/contrib/mdrone/cron&crontab-howto.htm

Recoil42

misbehavin wrote: »

Files don't have to end in .php to be executed by the PHP engine. That's just the default. You can use anything. In Apache, you just set:

AddType application/x-httpd-php .{WHAT YOU WANT HERE}

Is there any way to only push this rule for a specific folder/subdomain?

also, what happens if the file is not php, as expected?


IE, let's say I wanted to have some jpgs that are static on my server, and some that are dynamic, changed by PHP. So I add that type rule into Apache.

How does the server now handle the jpg's that AREN'T php files "in disguise"?

RoundBoy

php has a whole host of functions to manipulate images.. rather then reconfigure apache to serve jpeg files as php .. i would suggest coding a php page to take your input and output the proper jpeg..

something like :

<img src='myimage.php?id=1'>

You could probably do what you suggest in a .htaccess file .. that will allow per directory control.. but without knowing what exactly you are trying to accomplish.. i hesitate to push in one direction or another

Recoil42

I'm actually already doing that... my current sig?

http://recoil42.ath.cx/images/image.php

<?php
header ("Content-type: image/png");
$im = @imagecreatetruecolor(500,80)
     or die("Cannot Initialize new GD image stream");
$text_color_white = imagecolorallocate($im, 255, 255, 255);
$text_color_grey = imagecolorallocate($im, 30, 30, 30);

$daysleft = (int)((1175202000 - time()) / 86400);
$hoursleft = (int)(((1175202000 - time()) - ($daysleft*86400))/3600);
$texttoprint = $daysleft." days, ".$hoursleft." hours left";

$logo = imagecreatefrompng("gta4.png"); /* Attempt to open */
  if (!$logo) { /* See if it failed */
       $logo = imagecreatetruecolor(100,100); /* Create a blank image */
  }

imagecopy($im, $logo, 20, 0, 0, 0, 107, 80);

$fontt = 'arounded.TTF';
imagettftext($im, 20, 0, 215, 40+10, $text_color_grey, $fontt, $texttoprint);
imagettftext($im, 12, 0, 115+200, 40+6, $text_color_white, $fontt, $texttoprint);

imagepng($im);
imagedestroy($logo);
imagedestroy($im);
?>

Here's the problem I'm currently running into, while messing around with this..


so I've got cron running this on a timed basis:

#!/usr/local/bin/php -q

<?php
$im = @imagecreatetruecolor(500,80)
     or die("Cannot Initialize new GD image stream");
$text_color_white = imagecolorallocate($im, 255, 255, 255);
$text_color_grey = imagecolorallocate($im, 30, 30, 30);

$daysleft = (int)((1175202000 - time()) / 86400);

$hoursleft = (int)(((1175202000 - time()) - ($daysleft*86400))/3600);

$texttoprint = $daysleft." days, ".$hoursleft." hours left";

$logo = imagecreatefrompng("gta4.png"); /* Attempt to open */
  if (!$logo) { /* See if it failed */
       $logo = imagecreatetruecolor(100,100); /* Create a blank image */
  }

imagecopy($im, $logo, 20, 0, 0, 0, 107, 80);

$fontt = 'arounded.TTF';
imagettftext($im, 20, 0, 215, 40+10, $text_color_grey, $fontt, $texttoprint);
imagettftext($im, 12, 0, 115+200, 40+6, $text_color_white, $fontt, $texttoprint);

imagepng($im, "countdown.png");
imagedestroy($logo);
imagedestroy($im);
?>

Basically, it's supposed to create an image, and output it to countdown.png. This works, I tested it non-cron, by just putting it in a regular .php file, and hitting that up in firefox... countdown.png was created fine.

But what happens when I do it via cron is that it tells me that it cannot open or create any of the filenames inside of the script, WITHIN the .php file. Ie, it's not trying to create the files inside of the folder... it's trying to create them inside of the php file itself. how can I fix this? throwing /../ before the filenames did nothing.

Recoil42

After a looong google hunt, I fixed that issue, but it's honestly kind of a stopgap....

Turns out:

http://www.modwest.com/help/kb5-125.html

When you have a PHP script that manipulates file structure (say deleting
certain files at a certain time) and that script is activated by a cron
job, make sure that you explicitly specify the file system structure in
your script -- that is, write out '/htdocs/www/deleteThisFile.txt'
instead of $_SERVER[DOCUMENT_ROOT].'/deleteThisFile.txt'

For some reason the cron server doesn't activate the
$_SERVER[DOCUMENT_ROOT] variable.

In other words, all your file paths must be absolute. ugh. anyone know a better fix for this?

RoundBoy

You could try basename but that will only set the path for you to use later.

Check out THIS page for usefull varialbes to use. I would imagine _FILE_ or $_SERVER would do it.

Lets take a step back though. You are generating files, images really, with a date stamp on them?
How often is this cron job running?

Presumably these files are not useful once they are 'seen', so why keep a stock of files around? Unless you have some other reason for doing it, why not have the file generated on the fly while people are actually looking at it ? What am I missing in your applicaton ?

Jimmy King

Recoil: Cron doesn't know any $_SERVER variables because it's not part of the httpd environment. If you've got several places that are going to use the full path but all use the same one (or a same set of directories), just create a variable to hold the paths and stick it at the top of the script.

RoundBoy: Creating the image ahead of time actually makes sense. This is for his sig, which presumably will be loaded several times per day, possibly hundreds or thousands. Generating it on the fly each time would create pretty unnecessary load on the server. Personally, I'd probably go with something in between what he's doing and on the fly... check to see if the file for the correct date exists, display it if it does, and create it if it doesn't (the first time the script is loaded that day).

misbehavin

Recoil42 wrote: »

misbehavin wrote: »

Files don't have to end in .php to be executed by the PHP engine. That's just the default. You can use anything. In Apache, you just set:

AddType application/x-httpd-php .{WHAT YOU WANT HERE}

Is there any way to only push this rule for a specific folder/subdomain?

also, what happens if the file is not php, as expected?


IE, let's say I wanted to have some jpgs that are static on my server, and some that are dynamic, changed by PHP. So I add that type rule into Apache.

How does the server now handle the jpg's that AREN'T php files "in disguise"?

If the PHP engine encounters a file with the pre-set PHP extension, it scans the file for PHP code and works with that. Anything not between <? ?> brackets is simply displayed untouched. So, even if the extension is .php or whatever you set the PHP extension to be and the file does not contain any code, it will just display the file as-is.

RoundBoy

Jimmy King wrote: »

Recoil: Cron doesn't know any $_SERVER variables because it's not part of the httpd environment. If you've got several places that are going to use the full path but all use the same one (or a same set of directories), just create a variable to hold the paths and stick it at the top of the script.

RoundBoy: Creating the image ahead of time actually makes sense. This is for his sig, which presumably will be loaded several times per day, possibly hundreds or thousands. Generating it on the fly each time would create pretty unnecessary load on the server. Personally, I'd probably go with something in between what he's doing and on the fly... check to see if the file for the correct date exists, display it if it does, and create it if it doesn't (the first time the script is loaded that day).

Yeah, I think basename was the way to go.

As for the images... the intended application really makes the case. In the case of a sig image... yes, I completely agree to create an image every hour, overwriting the previous hours file. That is a very intelligent way to use this.

Any level lower then that (minute, more information specific to user, etc) it really is better on an as needed basis.

This is probably one of the only cases where I can see running php scripts locally, as opposed to my normal process of using perl...

Jimmy King

Oh yeah, definitely agreed. Anything updating by the minute or so is going to need to be regenerated almost every load anyway on all but the busiest of sites. User specific info is again going to be dependent on the actual usage, if it's user specific info, but it's going to be seen for that one user multiple times with the same data, might as well create and cache it to disk.

And :^: on the choice of Perl. Probably my favorite language to work with for most back end and web based stuff and what I do full time for my job (although I'm far from an expert).

misbehavin

Jimmy King wrote: »

Oh yeah, definitely agreed. Anything updating by the minute or so is going to need to be regenerated almost every load anyway on all but the busiest of sites. User specific info is again going to be dependent on the actual usage, if it's user specific info, but it's going to be seen for that one user multiple times with the same data, might as well create and cache it to disk.

And :^: on the choice of Perl. Probably my favorite language to work with for most back end and web based stuff and what I do full time for my job (although I'm far from an expert).

I, too, like Perl for most server-side work, but find any image manipulation to be an excellent job for PHP. The functions available in PHP for image work is astoundingly large and powerful. Just sayin'.

RoundBoy

Actually... excepting for very simple resizing and image generation, the php method is pretty bland. The most complex things I have ever done with it are dynamically creating a barcode for scanning , or to replace a few colors in a B&W image to user selected colors. CPU usage also skyrockets.

image magic integration is the way to go

I wish I could get off my ass and use perl for website design as opposed to backend only stuff... php is just too cludgy in specific (but common) cases.. Looking into ruby on rails maybe ...

We now return you to the original topic.