The Admin's Archive

Even the most seasoned Unix admin may forget some of the most powerful commands from time to time. I can say there have been several times while administering various flavors of unix and linux that I could have solved a problem with much greater ease had I just remembered that particular god-mode command. These commands may vary slightly between distributions, but their function remains the same.

In the interest of preserving god-mode, I hereby present this list of oft-forgotten commands that may help ease a large variety of administrative & troubleshooting tasks. So, without further ado, I present you this:

10 Commands a’la god-mode:

• apropos: This command searches the names & descriptions of man pages for keywords. While it may not be all to oft-forgotten, it is a very useful & important command which I felt was necessary to include.
• col: Have you ever wanted to save man pages as formatted text files? This command can do it! Try “man <somebinary> | col -b > somebinary_manpage.txt“.
• file: So, you found something sitting in some directory, and it has the execute bit set… Just what is it? Well, “file” will tell you!
• fuser: fuser is to PID’s as lsof is to files. This command does the reverse of lsof, by listing PID’s for whoever/whatever is using a particular file. Need to kill an entire process group, such as apache and all it’s spawned threads? try “fuser -k /var/log/apache2/access.log”.
• ldd: Displays info about shared libraries. This can come in handy when you have a binary that just doesn’t function, or doesn’t function as expected. The solution could be as simple as restoring a particular library file from one of your backups.
• script: Not everyone in my company is a *nix guru. So, sometimes I have to make a short tutorial for common tasks. That’s where script comes in! I can just record my terminal session, then print it!
• stat: Need info about a file or directory with a highly customizable format? This is the command for you! Pay close attention to the –printf option and formats. (notice the similarity to netstat and lstat ?)
• strings: This utility is great for investigating binary files. Ever wonder what programmers put in their programs? This tool can help you find out!
• time: This command displays the amount of time spent on processing a particular command. This is great for guaging how long a particular operation takes, which can then be used for properly timing automated tasks & such.
• xargs: Again, this isn’t such an uncommon command. But it’s very useful and I can’t help but present it here. Not all commands accept a piped input. This command will let you build a set of arguments for use with other commands.

Putting it all together:

So, one sunny morning you awake to find your web server is acting a little slower than usual. Since you forgot most of these commands, but still somehow remembered apropos, you are able to figure out the rest, right? Now, suppose you find some strange file in /var/www while poking around. So, you decide to investigate:

linux:~# apropos “file type”
[ (1) - check file types and compare values
file (1) - determine file type
test (1) - check file types and compare values

Ok, so I'll run file /var/www/a.out

linux:~# file /var/www/a.out
/var/www/a.out: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.6.0, dynamically linked (uses shared libs), for GNU/Linux 2.6.0, not stripped

Ok, so it's an executable using shared libraries... hmm... so let's run ldd /var/www/a.out and see what we find...

linux:~# ldd /var/www/a.out
libc.so.6 => /lib/libc.so.6 (0x00002ae9c0211000)
/lib64/ld-linux-x86-64.so.2 (0x00002ae9c00f9000)

Well, it looks like it was possibly written in the C or C++ programming languages. And it's using just a couple common libraries for those types of programs.

So, what's this file's deal? Let's run stat /var/www/a.out

linux:~# stat /var/www/a.out
File: `/var/www/a.out'
Size: 8597            Blocks: 24         IO Block: 4096   regular file
Device: fd03h/64771d    Inode: 913307      Links: 1
Access: (4777/-rwsrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2008-11-13 01:46:20.000000000 -0800
Modify: 2008-11-13 01:44:12.000000000 -0800
Change: 2008-11-13 01:53:01.000000000 -0800

Oh. My. God. This binary has the suid bit set! And it's owned by root! This could be bad...

Let's run strings /var/www/a.out and see what kind of info we can dig out of this binary:

linux:~# strings /var/www/a.out
/lib64/ld-linux-x86-64.so.2
__gmon_start__
libc.so.6
puts
__libc_start_main
GLIBC_2.2.5
ATUS
[]A\
hello world!

Whew! So some hacker just put a simple “hello world” program in your web server root. I guess you can relax now… or can you? I wouldn’t just yet…

I’d start by searching for other files with the suid bit set, and then either change their permissions if needed, or delete them. I think it’s safe to say that binaries located outside of your cgi-enabled directories can be safely removed–unless you’re hosting downloads. You might use the following command (from within /var/www) to find suid-enabled files:

find -type f -perm /u+s

Or this variant will work if you’re sure you don’t need suid capabilities.

find -type f -perm /u+s | xargs chmod 0644

Generally, unless there’s a *very* specific reason for it, no binary should be suid root. Many Unix and Linux distributions will have a small handful of suid files, however. So don’t go deleting and chmod’ing things unless you’re sure it’s not needed.

On a final note, I’d like to mention that there are a great many commands available to the linux/unix sysadmin. Don’t worry about mastering every one of them, since this could very well drive you mad. Just try them as you go, and develop a style of your own. Consider this: While it may be more elegant to use 3 more advanced commands to perform a task, if 5 commands come faster to you, then you will probably have greater efficiency executing those 5 commands. With experience, you’ll modify your style and adopt new variations of commands as needed.