Tools for Creating, Automating And Replicating (CAAR)

Ok, as promised, here is a break down of the desktop tools I use daily.

The underlying theme is that these tools are great for Creating, Automating And Replicating (CAAR).

Ubuntu

My OS of choice for desktop is Ubuntu. I use it because it is the best supported FOSS software, its the easiest OS to install and it is very very stable and no less powerful than Fbsd which was OS for about 4-5 years.

I strongly feel that if you are going to work with Linux servers for hosting your websites you should also be using a linux distro as your desktop. You will be greatly more productive.

If you are running windows, buy another hard drive, and install Ubuntu on it, you will be creating and replicating in no time.

If you are still not willing to do that install cygwin.

At any rate you need a Apache, MySQL and PHP on your home box anyways, and linux makes that trivial to install.

zsh

Zsh is my shell of choice, oddly enough its both the most user friendly and powerful of all the commonly available shells.

Its ability to cycle through tab completes is my favorite function.

Emacs

Not only a text editor, it is also a religion. Emacs has my day planner. I use it to read mail via gnus. And basically emacs is always open when my computer is on.

I can use it to edit any kind of source code, php-mode, sql-mode, all my custom PHP coding hacks. I just couldn’t live without emacs.

It has tramp mode, which means i can edit files on other servers just like they were on my box, minimizing bandwidth usage and having to keep uploading and downloading the file.

The syntax to open the remote file is trivial to /victory@dfhu.org:~/www/blog/index.php. It is seamless.

I also know basic vim and use it for really quick edits, but it doesn't compare to the power of emacs.

Subversion

My version control system of choice is SVN, its quick and easy and keeps my source code well organized.

I am even starting to use svn propset svn:keywords 'Date Revision' now to keep the date/version in the individual files. The boiler plate is inserted via a emacs customization.

scp

I only get hosts which allow ssh access, this is almost all hosts so it is not much of a restriction. I don't know why people are still using FTP(S) when life is so much easier with scp. scp is a copy command using the SSH protocol.

I have all my SSH accounts wired up set up to use public key authentication (PGP). This is more secure and doesn't require me to have to key in password every time. You can do this manually from the commandline but ubuntu comes with Seahorse which can set up the keys in three easy steps (choose secure key, put in host, put in password).

rsync

rsync is king among men when it comes to replication. It uses ssh and some powerful magic i don't quite understand to remotely copy files to servers, but only sending the data that is really needed, it does all the permissions, stuff, it has an easy syntax for what not to replicate.

It makes replicating a one command deal, not screwing around with ftp. Please use rsync, if you are not already, you are hurting. I will write a post on rsync later.

screen

Gnu screen is a way to keep your shell scripts running on remote servers even when you are not logged in. So basically i can login to any one of my servers, run `screen whatever-script` and then detach it.

When i logout my script is still running. I can then reattach it later when i need it.

Please use screen if you are not already, it will save you plenty of time.

Xvfb

Xvfb is a tool to run a virtual X display. I use it when i need to run a program which requires a GUI window, but on a headless server.

Its not a huge tool, but i don't see that many people in IM talking about it so i thought i would mention it.

PHP5.2

I try to keep up to date with my programming language/OS's but not be on the latest or experimental branch. I am shamless about not coding for PHP4, its been dead for over a year. Who among you would code in PHP4 and bitch that your users are using IE6?

I would rather use Python (pylons) to code my websites, but pylons sites are not as easy to replicate and you need better webhosting.

PHP does have some really ugly bits. The two that annoy me the most are nasty looking named parameters and prior to PHP5, insanely ugly lamda functions.

My database of Choice: SQLite3

I love SQLite, its so much easier to replicate (just copy a file) than MySQL. MySQL which requires a deamon and setting up usernames and passwords and such.

You can have one sqlite file/db which you replicate globally (like bad words, good domains, spider ips, etc...) and then have a local database for each install of your website, which changes on a local basis. You rsync all your global databases to all the sites in your campaign and you are good to good (a one word one line command).

Similarly you can download all your sqlite databases and merg them together for one global piece of uber awesome data. It is trivial, with scp/rsync.

If you know MySQL it will only take a day or two to learn SQLite3. They are just different dialects of SQL, with many of the same features.

Note that in PHP Sqlite3 is the default of the PDO class, sqlite_* is SQLite2 which I don't like.

Firefox uses SQLite.

My database by Necessity is MySQL

Most of the time MySQL is huge over kill and often doesn't actually introduce any appreciable performance enhancements. Most problems with databases are with sloppy database design, not a the database software.

I learned MySQL before SQLite, because it _was_ more widely supported, now that PDO comes bundled standard with PHP binary builds, SQLite3 is extremely common and most webhosts are very cool about installing it with minutes if you ask.

Because everyone else uses it, i learned it, but i won't use it if i can use SQLite3.

Don't get me wrong MySQL is great, but you can't replicate sites that use it with the ease that you can with SQLite3.

jQuery Javascript

Javascript is a wonderful language if you learn to discipline yourself enough to use good coding style.

jQuery, I bow down the developers of jQuery who are able to find, nurture and grow the beauty of Javascript.

If you are coding javascript and not using jQuery, i don't really know what your argument is, other than not wanting to spend the five hours needed to learn jQuery.

The plugin system is great too.

Python

Python is such a beautiful and powerful language that is always a pleasure to code in it. If i need to really rip out some serious scrapping or random website/css design python is the boy. The NLTK has the key to doing super powerful article spinning which gets the most human readable content possible.

BeatifulSoup is super handy for scraping.

The downside of Python is that it doesn't run super fast, but i feel that my time is much more important than CPU time.

The other language type things i Use

Tools that i use that come to mind right now, there are probably more.

- sed
- awk
- lynx -dump (i.e the poor mans scraper)
- grep
- matlab/octave
- LaTeX
- eLisp
- C++
- find
- The Actual Bricks and Mortar Library

Summary

If you are using linux servers, go out and buy another hard drive (screw messing around with duel boot) and install Ubuntu.

Now that you have Ubuntu, install Apache, PHP5, MySQL, Emacs, rsync, sqlite3, screen, zsh, scp and learn these tools.

It will take a little time, for sure, but once you got it, your competition (who is still messing around with the alternatives) wont be able to touch you.

Are you in this game for only a year or two until you go back to pumping gas or sucking recycled air in a cubical or are you in this game for the long run? If you are in it for the long term, use the big boy tools.

Get to the library, get out the BOOKS (again don't try to google-as-you-go learn, IT IS A TRAP).

I am willing to take suggestions on topics in the comments, but until then, I am currently working on a release of my PHP script to dynamically optimize titles over time, which I think you will really like. Think PolyPageTitle (but not just for Wordpress) and Eli's blue hat technique number 19 (keyword spinning)