Small Scripting Project - Editing contents of text file

Ponge

Hey guys, first off I'm running Linux Ubuntu so I'm looking for some help writing a shell script or python script that can run through a series of sequential text files, edit one or two lines and re-save them.

The files are generated by a 3D programme called Houdini, and are basically text versions of scene files that are used to render from the command line to save on RAM (Houdini doesn't need to be open to render these files so can substantially reduce RAM requirements).

They would be named something like "test_0001.ifd" and can be anything from 10kb-100mb in size.

I have created a very basic scene to show what the contents of such a file would look like:

ray_property object velocityscale 0.0208333333333
# IFD created by Houdini Version: 10.0.430
# Generation Time: Apr 22, 2010 at 15:33:22
#   Render Target: mantra10.0
#     Render Defs: /opt/hfs10.0.430/houdini/soho/IFDmantra10.0.py
#        HIP File: /nccamade/i7977054/IFD_TEST/IFD_Test_01.hipnc, $T=0, $FPS=24
#   Output driver: /out/mantra1
ray_version VEX10.0.430

setenv HIP = "/nccamade/i7977054/IFD_TEST"
# Retained geometry
# SOP /obj/box_object1/box1
# Save geometry for /obj/box_object1/box1 at time 0
ray_start geo	# {
    ray_detail /obj/box_object1/box1 stdin
BgeoV                                 &#65533;   &#65533;   &#65533;   ?&#65533;  ?   &#65533;   &#65533;   ?&#65533;  ?   &#65533;   ?   ?&#65533;  &#65533;   &#65533;   ?   ?&#65533;  &#65533;   ?   &#65533;   ?&#65533;  ?   ?   &#65533;   ?&#65533;  ?   ?   ?   ?&#65533;  &#65533;   ?   ?   ?&#65533;  &#65533;&#65533;&#65533;&#65533;       <        <       <       <        <        <     &#65533;ray_end	# }

# Main image from /obj/cam1
ray_time 0	# {
    ray_declare plane string IPlay.s3dleftplane ""
    ray_declare plane string IPlay.s3drightplane ""
    ray_image "/nccamade/i7977054/IFD_TEST/IFD_test_0001.jpg"
	ray_declare plane string IPlay.rendermode "append"
	ray_declare plane string IPlay.framerange "1 1"
	ray_declare plane float IPlay.currentframe 1
	ray_declare plane string IPlay.rendersource "/out/mantra1"
	ray_declare plane int IPlay.houdiniportnum 23883
	ray_start plane	# {
	    ray_property plane variable "Cf+Af"
	    ray_property plane vextype "vector4"
	    ray_property plane channel "C"
	ray_end	# }
    
    ray_property image resolution 640 480
    ray_property image pixelaspect 1
    ray_property image samples 3 3
    ray_property photon photongfile "/nccamade/i7977054/IFD_TEST/global.pmap"
    ray_property photon photoncfile "/nccamade/i7977054/IFD_TEST/caustic.pmap"
    ray_property camera projection "perspective"
    ray_property camera zoom 1.21319175727
    ray_property camera clip 0.00999999977648 1000
    ray_property image window 0 1 0 1
    ray_property image crop 0 1 0 1
    ray_transform 0.707106781187 -0.353553410979 0.612372423926 0 5.24313241839e-18 0.86602538714 0.50000002883 0 -0.707106781187 -0.353553410979 0.612372423926 0 1.34869915205e-06 -3.57974763692e-06 -5.06176299331 1
    ray_start light	# {
	ray_transform 0.611159338728 -2.9802306883e-08 0.791507588521 0 0.417195908483 0.849809344052 -0.322136077935 0 -0.672630535012 0.527090199836 0.519368929188 0 -3.44786643982 2.70183157921 2.66225743294 1
	ray_property light name "/obj/spotlight1"
	ray_property light shader opdef:/Shop/v_asadlight lightcolor 1 1 1 docone 1 coneangle 45 conedelta 10 conerolloff 1 doatten 1 atten 1e+06
	ray_property light projection "perspective"
	ray_property light zoom 0.784842742499 0.784842742499
    ray_end	# }

    ray_start object	# {
	ray_transform 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1
	ray_geometry /obj/box_object1/box1
	ray_property object name "/obj/box_object1"
    ray_end	# }

    ray_raytrace
ray_quit	# }

There is some gibberish code in there which I'm not quite sure what it does, hopefully it's not important.

These files can sometimes take a minute or two to generate each, and what has generally happened in the past is that after I have generated 1000 of these files, I realise I have set them up to use the wrong camera, or to save in the wrong directory. So basically what I'm looking for is some sort of shell-script that I can copy into the directory containing the sequence of IFD's, give it the line I want to replace and the string that i want to replace it with.

An example might be:
" ray_property image resolution 640 480"
Which indicates that the output resolution is to be 640x480px. If I wanted to change this to 1080x720px for a series of files, how would I achieve this?

Any help is appreciated guys, I have a little previous scripting experience and I can generally parse other peoples examples to understand how they achieve things, but I'm useless at coming up with my own code.

Cheers,

mrcheesypants

You'll probably want to use regexes. http://txt2re.com/ will be a big help.

vonPoonBurGer

You'll definitely use regular expressions for this, but I'd argue against doing file reads/writes in your own script. You're reinventing the wheel if you do that, since someone's already written a utility that will do exactly what you want. It's called sed (short for "stream editor"):

C:\temp>sed -e "s/ray_property image resolution 640 480/ray_property image resolution 1080 720/" -i.bak test0001.ifd

C:\temp>dir test*
 Volume in drive C has no label.
 Volume Serial Number is 70EE-AB19

 Directory of C:\temp

04/22/2010  11:16 AM             3,059 test0001.ifd
04/22/2010  11:13 AM             3,117 test0001.ifd.bak
               2 File(s)          6,176 bytes
               0 Dir(s)   2,310,889,472 bytes free

C:\temp>diff test0001.ifd test0001.ifd.bak
34c34
<     ray_property image resolution 1080 720
---
>     ray_property image resolution 640 480

I'm using a Windows version of sed, but the syntax should be almost identical under Ubuntu. I think the only difference is that you'd want to use single instead of double quotes in a bash shell. Let me break down what each part of that command does:

• -e: following this switch you specify the regular expression to apply to the input
• "s/: this denotes a substitution expression
• ray_property image resolution 640 480/: this is the pattern we're looking for
• ray_property image resolution 1080 720/": this is what we'll substitute with when a match is found
• -i: modify the specified file(s) in place
• .bak: make a backup of the file if it's modified, giving it a .bak extension

You could specify multiple files (e.g. *.ifd or project_name*.ifd) and it would work the same. If the pattern you're looking for is going to appear multiple times, you can use the /g modifier to perform the substitution globally (e.g. "s/foo/bar/g" substitutes all instances of "foo" with "bar"). You can do case insensitivity with /i, or combine the two with /ig. Don't want a backup made? Use the -i switch but don't specify an extension. Want to delete lines where a certain keyword appears? It can do that too:

sed -e "/foo/ d" -i blah.txt

That deletes any line that contains the text "foo" from blah.txt. It's a ridiculously versatile utility, it should be able to do just about any kind of batch modification you want.

Ponge

Hey Von,

Thanks for the awesome write-up on SED, I've tried giving it a run and it seems to work, however several lines in my IFD file contain quotation marks and slashes, both of which seem to be used in the syntax of SED.

If I wanted to modify the current line:

ray_property photon photongfile "/nccamade/i7977054/IFD_TEST/global.pmap"

How would I achieve this?

Many thanks,

*edit* Oh wait, I just realised you said to use single instead of double quotation marks. Let me try that out!

vonPoonBurGer

Yeah, under Linux you should be able to have double quotes in your regex without any trouble, as long as your expression string is enclosed in single quotes. Issues with quotes aren't so much a regex issue as a shell issue. If you've got the whole expression string escaped in double quotes, plus some double quotes inside the expression string itself, it's going to give you problems because your shell is going to interpret all of those double quotes and mangle your command. You can use an escape character (usually backslash) to force your shell to interpret it the way you want, e.g.:

sed -e "s/ray_property photon photongfile \"/nccamade/i7977054/IFD_TEST/global.pmap\"/ray_property photon photongfile \"/some/other/path/global.pmap\"/" -i.bak test0001.ifd

It's not a bad idea to throw an escape character in front of any quotes (single or double) inside your expression string. That way you don't have to worry about how the shell will interpret them, with the escape character in front you know the character will get passed through to your command (sed in this case) unmangled.

Beyond issues with the way your shell interprets quotes, you may also run into issues with regex special characters getting interpreted by the regex engine when you actually want them treated as literal characters. For example, say one of your IFD files has an asterisk (*) on a line. That character is used for wildcard matching in regex, and having one in the match portion of a substitution string could cause it to match a whole lot more lines then you intended. In those situations, you can use the escape character to force sed (or grep, or perl, or whatever) to treat any of those regex special characters as literals too.