Thu, 18 Feb 2010
Tripping Myself Up
Caught with My Pants Down
At about 4:15 yesterday afternoon I received an unusal phone call. The guy at the other end didn’t identify himself at first. He just asked me if I was the site admin for eldoradotech.org, which I am. He then somewhat murkily explained that he’d received a phishing email from a third party. Further, the email linked to my web site. No big deal, except that my web site was in fact serving up a page that looked just like a JPMorgan Chase login page. Not good.
After determining that there was a problem, I immediately deleted the
unauthorized files from the server and then shut it down. Unfortunately, this
resulted in all of my web sites, email, and other services being unavailable,
which is a hassle for my millions thousands legions hundreds scores
dozens of two friends and fans. However, it was necessary because
whoever had put those files on my server the first time could always put them
back a second, and could further exploit not only that server, but the other
servers in my gleaming, high-tech ghetto basement datacenter
as well as the desktops and laptops around the house.
Fortunately, my workday was largely over; so I rushed out to my car at 5:05 to get home and check things out. All the way there I considered the many possible vectors an attacker might have used to break in to my server. Some of them are difficult and unlikely, while others would simply require access to my password. I generally keep close tabs on my password, but you can never rule out some kind of a slip-up. This is why changing your password regularly is a good idea, even if it is a pain in the ass.
During transit I also thought about the varying consequences of the attack: if it was just the one system, damage could be limited. However, if the attacker had been in the system a long time before using it for nefarious purposes, he or she might have logged other passwords, confidential business information, financial records, or other valuable data. This worried me.
When I arrived at home, I first turned off all the other computers in the house. That’s three other servers, three desktops, and two laptops, currently. This was to prevent the attacker from using one of them to re-infect the first system if he or she was already loose on my network.
After this first step, I booted up the infected server from a clean Knoppix CD image and analyzed the logs. It looks like two IPs were running a dictionary attack against a weakly-passworded mythtv user account:
58.177.188.213
172.173.83.246
Neither IP is responding to ping, now.
The attacker appears to have gained access to the brand-spanking-new mythtv account (no this server wasn’t being used for MythTV, but I keep accounts synchronized across my hosts to keep things simple) and then used a privilege escalation exploit to create a new user, ‘ftpd’. Then the attacker gave the new ftpd account a UID of ‘0’ (essentially, the same access level as root). From there, it was all down hill.
Logs don’t always tell the truth, because they can be edited, deleted, or corrupted. Having something to track back through was nice, but it’s not sufficient. Because for all I know the attacker was leaving a false trail, I elected to nuke the site from orbit. It’s the only way to be sure. So, for remediation, I wiped the system and reinstalled the OS and applications from known clean sources, removed the unauthorized ftpd account, changed passwords left-and-right, then restored user data from my latest backup.
I’m lucky that this was all it took. If my not-quite-anonymous caller hadn’t clued me in, it might’ve been several hours, or possibly several days, before I noticed a problem. And if the attacker had been more sophisticated about covering tracks, I might still not know what vector had been used to break in to my system. In other words, relatively little damage was done (at least to me; I can’t speak for people who may have been phished) and this was a relatively easy system to get back up and running. Now I just need to be more conscientious about my passwords.
posted at: 13:48 | permanent link to this entryTue, 19 Jan 2010
Blog Coding Updated
Although I've based the code for this blog on the Blosxom framework, and use the Tiny MCE JavaScript library to handle editing chores, the fundamentals are significantly modified from the original.
Among other things, I've custom-coded the blog to support picture uploads, automagically create thumbnails and link to the full-sized images; added an authentication mechanism; and configured custom blogs for each member of the family.
Unfortunately, due to a misconfiguration on my part, Tiny MCE was substituting a relative path for the absolute path on each of the image includes and links. This worked just fine, until I added the ability to browse the blog by category or by date. When these features are active, it causes the blog script to generate temporary subdirectories in the URL, and in conjunction with Apache, redirects requests for the category- or calendar-based pages, breaking the images.
When I discovered this problem, I did a little research and determined that I could simply tweak the Tiny MCE configuration to eliminate the issue on all new blog posts. However, this did not fix any existing entries.
Because Blosxom generates web pages based on the datestamp on the individual files created when a user writes a post, simply running a search-and-replace against all files to change the relative paths to absolute paths would result in all existing blog entries showing up as being new as of the time I executed the change.
Obviously, this is not good.
Further, although I could individually modify the files one at a time to restore their original datestamps, the volume of files involved made that a non-starter.
To resolve the issue, I have written a Perl script that parses through all of the existing blog entries, corrects the paths, and then saves the file with the original datestamp. This wasn't rocket surgery; but it was a new endeavor for me. On the off-chance that you might encounter a similar problem, I am making this script available under the GPL. Feel free to use it, but be sure to make a backup copy of your data before executing it.
Sat, 02 May 2009
Social Networks
The latest rage on Facebook apparently is apps that ask you to list five of your favorite things in a category. I don't think I can even name five wrestlers, so it's probably just as well I'm not on Facebook. Although my wife is, and she thinks it's wonderful. I did once, long ago, join Classmates.com. None of the people I would have considered renewing relationships with ever seem to have joined. Before I reached my current viewpoints on Internet-enabled social networking, I also joined Friendster.com; but it has been nearly a decade since then.
Facebook, though, for me, is just not a draw. Yes, I do have a blog, which I even occasionally update; but as you can see, it's on my personal website, where I have total control of the context and copyrights. But when it comes to interpersonal relationships, I like to keep my one-on-one communications confidential. The idea of a "wall" where I get drive-by comments from acquaintances, or having the entire subscriber base (or even just the people already on my friends list) know who is in my social network, or having people tag pictures with my name, with or without me actually being in them — even with the "privacy" controls Facebook provides — just leaves me cold.
Further, having to maintain personas on multiple networks (Linked-in, Facebook, Classmates.com, MySpace, Friendster, Orkut, etc.) to maintain separation between professional and personal lives, as well as hit the "right" sites so that the "right" people see I'm a member of the same communities… well, it's burdensome. The MySpace people want to use MySpace; the Facebook people want to use Facebook. It's just too much work for me. The single option of Geocities in the '90s was easier.
Finally, I've had a website since 1996. My Google PageRank is excellent. If people I already know want to find me online, it's a piece of cake. If people I don't already know want to find someone with my skills (e.g., recruiters), my resume is at or near the top of Google's results in all the markets I care about.
That being said, for many, Facebook and its ilk are pretty darn good. They have reasonably featureful interfaces, a critical mass of users, and the backing of major corporations to ensure they stay available. And that's fine; it's just not for me.
Thu, 19 Mar 2009
Unlocking Cryptography
Generating SSL Keys on Debian Linux
As reported in my last entry, I recently created updated SSL keys for my server. This is a somewhat arcane process, involving wizardly incantations on the command line. As a service to the community, I will now describe this process and provide a simple script to stramline the process.
First, the reason it was necessary to generate these keys is that the default Debian install creates keys that are only good for one year. Further, these keys are “snakeoil”; that is literally what the configuration calls them, which serves as a reminder to sys-admins that they are the default configuration (generally more exploitable), they are not part of a chain-of-trust (nobody else is vouching that you are who you say you are), and they potentially do not uniquely identify your server (setting up a series of servers with the same configuration can cause confusion among various connecting hosts).
These instructions apply to generating a self-signed key: just as with the default Debian key, nobody else is vouching that you are who you say you are. If you want to get an “official” key, you have several options, of varying expense:
Unless you are going to be selling something to the general public, or will be accepting payments from people you don’t personally know, however, these are all overkill. A self-signed cert will work just fine for you if you are using this server inside an organization where you have control over browser deployments, or if you are working with a technical audience of people you already know.
On to the steps!
Remember, these are specific to Debian GNU/Linux default installs. If your system is on another version of Linux, you’ve customized your install in some unusual way, or you’re using another OS, you will have to modify these instructions to match your environment.
Login as root:
sudo su -
Change directories to your SSL configuration directory:
cd /etc/ssl;
Create the seed for your private key:
openssl genrsa -out example.com.key 1024;
Use the seed to generate a public/private key pair request:
openssl req -new -key example.com.key -out example.com.csr;
Generate and sign the keys:
openssl x509 -req -days 365 -in example.com.csr -signkey example.com.key -out example.com.crt;
copy the old/default key to a timestamped file:
mv /etc/ssl/example.com.csr "/etc/ssl/example.com.csr.`/bin/date +%Y%m%d`";
Copy the old/default apache certificate to a timestamped file:
mv /etc/apache-ssl/apache.pem "/etc/apache-ssl/apache.pem.`/bin/date +%Y%m%d`";
Copy the new private key to the apache-ssl certificate:
cp -p example.com.key /etc/apache-ssl/apache.pem;
Sign the new apache-ssl certificate:
cat example.com.crt >> /etc/apache-ssl/apache.pem;
Change permissions on the certificate to avoid security issues:
chmod 600 /etc/apache-ssl/apache.pem;
Delete the originals:
rm /etc/apache2/apache.pem;
link the apache-ssl certificate to apache2’s, so you don’t deal with multiple certs when you don’t need to:
ln /etc/apache-ssl/apache.pem /etc/apache2/apache.pem;
copy the apache cert to the generic ssl cert library:
cp -p /etc/apache-ssl/apache.pem /etc/ssl/certs/ssl-cert-example.com.pem;
copy the private key to a restricted area:
mv ./example.com.key /etc/ssl/private/;
Change permissions on the private keys to ensure they remain private:
chmod 600 /etc/ssl/private/*;
change ownership on the private keys, as well:
chown root.ssl-cert /etc/ssl/private/example.com.key;
Move the public key into the certificate directory:
mv example.com.crt /etc/ssl/certs/;
Change permissions on the public keys, also:
chmod 600 /etc/ssl/hall*;
chmod 600 /etc/ssl/certs/example.com.crt;
chmod go+r /etc/ssl/certs/example.com.pem;
Restart Apache and your mailserver (I use Postfix rather than Exim) so that they reload their keys:
etc/init.d/./apache2 restart;
/etc/init.d/./postfix reload;
All done!
I’ve also written a script to automate this process. Feel free to use it, but remember I’m not responsible if it breaks anything.
Comments, criticisms, and corrections are welcome.
posted at: 01:00 | permanent link to this entryMon, 16 Mar 2009
Marvelously Modified Mailserver
Geeky Fun!
After spending a whole week of classroom time in a “System p LPAR and Virtualization I: Planning and Configuration” training session, this weekend I was feeling motivated to make a few changes. As I’d been deferring the (completely unrelated) migration of my email and SSH server to a new platform, it was time to take action!
This is a relatively large change for my small environment. Currently, I’m running a web server (Apache), a mail server (Postfix with SpamAssassin), a remote access server (SSH), Windows (Samba) and Unix (NFS) networking servers, some monitoring utilities (Monit), and various smaller functional programs.
Fortunately, the migration process was to be relatively painless. As I had planned for this, I already had mirrored the configuration from my “Old and Busted” system (based on an Intel Pentium III running at 800 MHz, and although rock solid, dreadfully slow), to the “New and Kewl” system (based on an Intel Xeon dual core, dual processor running at 2.3 GHz). All that needed to be done, then, was:
- At the router, stop accepting inbound email for the duration of the migration.
- Disable the Postfix daemon on OldAndBusted.
- Copy the user mailboxes from OldAndBusted to NewAndKewl.
- At the router, set inbound email connections to be directed to NewAndKewl.
And that should do it!
Except for the small item of ensuring that my users’ individual email clients are all configured to talk to NewAndKewl instead of OldAndBusted. Not a problem! I use DNS for my internal network, so I updated the DNS configuration to point mail.hallmarc.net at NewAndKewl, and everything was good.
Except the email clients were using the IP address rather than the fully-qualified domain name for the mail server. Uh. Dumb. Ah! but I can modify the configuration from the command line for all the kids accounts by logging in remotely and changing all of Thunderbird’s instances of OldAndBusted’s IP to NewAndKewl’s IP. Done and done. (Yes, I did have to use the GUI on my wife’s Windows XP PC to do this. One more reason not to support Windows.)
About using that GUI… Apparently my wife for months had been clicking through a dialog box every time she collected email. The dialog indicated that the mail server’s SSL/TLS certificates had expired. I only learned this because, yup, I used the GUI to change her server setting. So now I needed to update my server certs. Which will be the subject of my next blog entry.
posted at: 19:33 | permanent link to this entryThu, 12 Feb 2009
Time’s up!
Unix time is fun! As I noted in a previous entry about the nature of timekeeping in the computer world, Unix tracks time by counting seconds since January 1, 1970.
The latest milestone for Unix time is at 23:31:30 UTC on February 13, 2009. People who like patterns in their numbers will rejoice as Unix time reaches 1234567890 seconds since the beginning of the Epoch. By coincidence, this day falls on Friday the 13th on the Gregorian calendar.
If you're using a Unix or Unix-like system, you can see how this works:
[on GNU]
>$ date -ud@1234567890
[on BSD]
>$ date -ur 1234567890
That will look like this:
For more information see the time_t entry on Wikipedia, the free encyclopedia.
Marc Elliot Hall St. Peters, Missouri
Page created: 21 January 2002
Page modified: 31 December 2009
