mtr…the love child of ping and traceroute

Just a quick note to tell you to check out mtr if you haven’t already.  It combines the features of ping and traceroute in kind of an interpreted mode that runs until you quit.  There are a few command line switches as well to make life easier for automation.  Here’s a generic example of it’s output going out to Google:

My traceroute  [v0.75]
centos.mydomain.com (0.0.0.0)             Mon Jun 29 10:10:25 2009
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                            Packets              Pings
Host              Loss%   Snt  Last   Avg  Best  Wrst StDev
1. 192.168.1.1    0.0%     6    0.3   0.4   0.3   0.6   0.1
2. 66.53.18.65    0.0%     6   37.7  21.2   7.4  37.7  12.8
3. ???

I’ve changed the IP addresses above to protect the innocent, but it should give you a general idea of what to expect.  For years as a network engineer I used “ping –t” to watch for downed gear to come back online.  This tool gives you that ability and much more as it keeps track of numerous interesting and useful statistics.

Filetransfer Appliance

Our Firm constantly gets requests to send large data files back and forth between clients.  Up until now, we’ve been using a mixture of email attachments and a locked down FTP site. For a number of reasons that are too many to mention we needed a better way.  Lance from ILTA (thanks Lance) turned me on to a device from Allardsoft that takes this pain away.   The device isn’t completely Open Source, but it is based on Open Source tools.  The Filetransfer appliance contains a web server which can be used to post a file/files and address it to a user.  External users can only send files to internal users, but internal users can send files to anyone.  The recipient  of the email gets a single link that they use to retrieve the file.  All files are deleted automatically after a set time and user’s can’t see each others files.  The whole process is done behind SSL, secured by a certificate and is logged.  Very cool.  It looks like they are even working on a plug-in for Outlook.  That alone is worth the price of admission.  The appliance isn’t free, but even at it’s maximum unlimited users configuration it’s only $499.  If your email server if busting at the seems be sure to take a look.

Cool and Different Presentations

Having worked in the IT sector now for about 20 years I have a finely tuned bullshit monitor.  This is especially true of presentations.  Most hour long presentations I sit in on have 5 valuable minutes of new or interesting material. We all know “cool” when we see it and we all know what the other 55 minutes of the presentation are for….to fill time.  So when I saw my first Google I/O Ignite presentation I about fell out of my chair.  The presentations are 5 minutes long each with 20 PowerPoint slides.  Oh yeah and they are rotated every 15 seconds so the speaker isn’t in control…he/she has to keep up.  I love this format.  It forces the speaker to cut the the raunchy essence of the topic and eliminate the bs.  My hat is off the folks at Google on this.  I’m presenting a small piece of an open source firewall talk at ILTA this year (pfsense…yeah baby!) and I may very well steal this concept for that presentation.

As an avid viewer of TED presentations, I’m always intrigued by new and different presentation styles.  This presentation is done in a style that I’ve copied a few times and had much success with.  It’s based on the “Lessig Method” (example here) which I find drives the listener to pay attention just to keep up.  At last years ILTA, I did a presentation on Wireshark in this format and it was well received.  This is my presentation style of choice if my presentation is a tutorial or training session.

What presentations styles have knocked your socks off?  Hit me up in the comments and let me know. I’m all ears.

Linux System Maintenance and Setup

There are a few things I like to do on all my Linux servers when the box is set up and from time to time to verify all is well.  Here is a short list of some of these items:

• Remove rights for root to login via ssh.
  In /etc/ssh/ssh_config change "PermitRootLogin yes" to "PermitRootLogin no".  Then restart ssh
• Boot into command line mode instead of the Gui (runlevel 3 instead of 5)
  In /etc/inittab change the line
  "id:5:initdefault:" to "id:3:initdefault:"
• Setup logwatch to email you daily logs of what's happening each day
  In /usr/share/logwatch/default.conf/logwatch.conf change
  "MailTo = root" to
  "MailTo = yourname@youremail.com"
• Better yet...send all mail for root to your email email account.
  Edit /etc/mail/aliases and change
  "#root:   marc" to "root: yourname@youremail.com".  Now you need to run /usr/bin/newaliases to recreate the aliases.db file.
• Update your box nightly at midnight...but skip kernel updates as they may break stuff
  Edit your cron jobs file (crontab -e) and add the line:
  0 0 * * * yum --exclude=kernel* -y update
• Reboot your machine weekly (Reboot every Sunday at 1am)
  Edit your cron jobs files (crontab -e) and add the line:
  0 1 * * 0 /sbin/shutdown -r now
• Adjust the time for forced disk checks to once a quarter because it can take a long time to boot with large drives.
  Run something like
  "tune2fs -c 12 -C 0 /dev/VolGroup00/LogVol00"
  Assuming you reboot once a week this will force a check once a quarter.
• Get a good look at the processes and what started them on your system
  Run "ps auxwww"
• Get a good baseline of your hard drives performance and age before you go live. Note that this wont' work on a VM and you will need check /dev/hda1 to machine your machines config.
  Run "smartctl --all /dev/hda1"
• Determine what ports are open and listening
  Run "netstat -anp --tcp --udp | grep LISTEN"

What things to do you do?

I’m working on authoring a system maintenance document outlining things that should be done for maintenance on a daily, weekly, monthly, quarterly and annual schedule.  Let me know what you are doing and I’ll email you a copy of my document when it’s done.

More on SELinux

Yesterday I posted about some issues I had with SELinux after a kernel update to CentOS.  My post was commented on by Dan Walsh, a top notch security guy from Red Hat. I clicked on his name and found his blog which turned out to be a goldmine for me.  In reading his blog I found a link to the best resource I’ve seen on SELinux for a system admin.  It’s the Security-Enhanced Linux User Guide.  I read the whole thing in about 90 minutes and it provided some insight into SELinux that I’ve found nowhere else.  This is why I love Open Source and the Internet.  I’m sure if I posted a note about some feature in Windows I’d never hear back from anyone in the developer community at Microsoft about how to fix my problem.  Heck, I didn’t even post my note to a newsgroup…just my humble blog.  It’s great to see people so involved and interested in what they do that they go looking for issues just to help people and keep up to date on what others are saying.

Thanks Dan. I truly appreciate it.

CentOS Kernel upgrade breaks SELinux

So last night I did a yum update on one of our web servers which included a kernel update.  All went well until the reboot at which time SELinux was preventing httpd from starting.  Dropping SELinux into permissive mode (setenforce permissive) allowed httpd to start and things went well except for the banter of SELinux messages in my logs bitching about one thing after another.  At first I thought about a system-wide relabel of the drive…but I’m truthfully a bit concerned that the hammer approach might break too many things.  After some research on the web I took this approach instead:

• Grep out the line items from /var/log/messages that seem to be creating a problem. I ran something like: tail /var/log/messages | grep avc > fix1
• Use the audit2allow script to build a file of fixes that could be applied to SELinux (audit2allow –M fix1a < fix1).  This creates a file called fix1a.bb.
• Run fix1a.bb against the semodule command (semodule –i fix1a.bb)

That’s it.  I had to do this a few times as errors popped up, but it seems to have fixed the problem.   Be sure to read through the offending lines in the messages log to verify that things that are being denied should actually be working.

Here is an example of some of the errors I was getting:

Jun  4 16:15:45 MY-WEB01 kernel: type=1400 audit(1244197545.492:6268): avc:  denied  { append } for  pid=10103 comm="httpd" path="/var/log/httpd/access_log" dev=dm-0 ino=851362 scontext=root:system_r:httpd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:file_t:s0 tclass=file

Jun  4 16:12:29 MY-WEB01 kernel: type=1400 audit(1244112349.412:6262): avc:  denied  { search } for  pid=9737 comm="httpd" name="mysql" dev=dm-0 ino=851641 scontext=root:system_r:httpd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:file_t:s0 tclass=dir

I’m still in favor of SELinux although I think I’ll be taking more precautions (like snapshoting  a machine) before I update kernels.  Once I got past the errors I restarted SELinux (setenforce enforcing).