The adventures of MykoSpark!

initrd woes: RAMDISK: incomplete write (165 != 16397)

I recently rebuilt my initrd and added more kernel modules.  Upon boot of my system, I received the following error:

RAMDISK: gzip image found at block 0
RAMDISK: incomplete write (165 != 16397)

I realized that my initrd image was 18MB, and my RAM disk was only 16MB. After removing some unnecessary files from the ramdisk, I rebuilt it and things worked fine. If you still have an issue with the kernel not being able to find the root device, ensure that your cpio command contains the “-H newc” option. It ensures that the correct cpio format is used for building the image.

Python: Build RPMs from setuptools

There isn’t a ton of documentation on doing this, but I was able to get an RPM build using setuptools and python 2.6. This example is for beanstalkc, a beanstalk client for Python:

# note: you probably need the following packages installed:
# rpm-build
# python
# python-devel
# python-setuptools
python26 ./setup.py bdist_rpm --fix-python

That was much easier than you were expecting, right? I did this build on a CentOS 5 machine. I wanted to build this package for python26 instead of the default python24. If you don’t specify the ‘–fix-python’ flag, then the package will get built for python24 regardless of how you execute setup.py. You’ll see a bunch of output (mostly from rpmbuild) on your screen, and it should tell you where the RPMs ended up.

One thing I was unsure about is changing the package name to ‘python26-beanstalkc’ instead of the default ‘beanstalkc’. I *could* go modify the SPEC file and re-compile, but it’d be nice if I could do everything with setuptools.

Python: Response headers as dictionary with urllib2

I spent hours trying to figure this out via documentation and Google, but I ended up getting my answer from the source code. Hopefully I’ve learned my lesson and will check the source code first in the future.

As the title states, I wanted to get the response headers from a urllib2 request. I also wanted the headers in dictionary-form because I didn’t feel like parsing them myself. I found a really simple way to do it, but since it is undocumented, it COULD go away in a future version. For reference, I’m on Python 2.7.2. Here’s the code snippet:

#!/bin/env python
import urllib2
from pprint import pprint
 
request = urllib2.Request("http://www.google.com/")
response = urllib2.urlopen(request)
pprint(response.info().items())

And that’s it! Happy coding!

Create loopback filesystem in Linux

I overheard a co-worker complaining about how long it takes to delete 3 million files on his test server. He was running a job to collect data, store it in these small files, then delete the files when complete. Aside from using a database or keyvalue store for his data output, one recommendation I gave him was to use a loopback filesystem. After his files are written, he can easily remove it, or compress it and send it to a colleague. Here’s what I sent him:

First, you need to create a dummy file on-disk to store the filesystem. The following snippet will create a 100MB file of zeros at /tmp/loopback.img:

# dd if=/dev/zero of=/tmp/loopback.img bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.0835574 s, 1.3 GB/s

You should be able to see that the file exists, and is indeed 100MB:

# ls -hs /tmp/loopback.img 
101M /tmp/loopback.img

Next, you’ll need to format the loopback file with your favorite filesystem. You may get asked if you want to format the file because it isn’t a block device. Just say ‘y’:

# mkfs.ext2 /tmp/loopback.img 
mke2fs 1.41.14 (22-Dec-2010)
/tmp/loopback.img is not a block special device.
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
25688 inodes, 102400 blocks
5120 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
13 block groups
8192 blocks per group, 8192 fragments per group
1976 inodes per group
Superblock backups stored on blocks: 
        8193, 24577, 40961, 57345, 73729
 
Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done
 
This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Create a place to mount your loopback file:

# mkdir /mnt/loopback

Now you can mount it:

# mount -o loop /tmp/loopback.img /mnt/loopback/

If you get a strange error like ‘mount: could not find any device /dev/loop#’, you may need to insert the module. On some systems, the ‘loop’ module doesn’t get loaded automatically:

# modprobe loop

You should see your loopback filesystem available with (almost) 100MB of space:

# df -h | grep loopback
/dev/loop0             97M  1.6M   91M   2% /mnt/loopback

If you want to write anything to your loopback filesystem, do it in /mnt/loopback:

# touch /mnt/loopback/my_file

When you’re done, you can unmount it, gzip it, and send it to a friend. Your friend can uncompress it, and mount it locally to read the data you put inside:

# umount /mnt/loopback/
# gzip /tmp/loopback.img
# ls -hs /tmp/loopback.img.gz 
108K /tmp/loopback.img.gz

BASH CGI: Writing a CGI script with shell

I while back, I built a clustered storage system based on GlusterFS for my company. I had no issues with the system until recently when it unexpectedly died (it’s now back up and running just fine). I decided it would be a good time to implement some basic monitoring which would ensure that everything was working.

In my company, fetching a document via HTTP is the easiest way for our monitoring system to detect failures. I could have just created a static file for our monitoring system to fetch, but I wanted the test to be more involved. I decided on having a script perform a write/read/verify test that would ensure the storage system was available for reads and writes. I also didn’t want the baggage of installing PHP/Python/etc to perform this test, so I decided to see if I could do it with a shell script. It’s actually very easy! Here’s my script:

#!/bin/bash
# Define some canary data, and a place to test write/read/verify
DATA="$(date +%s)"
TARGET="/storage/test/.test"
 
# put data in the target, then read the target back
echo "${DATA}" > "${TARGET}"
exec < "${TARGET}"
read TEST
 
# check to make sure they're equal, then write the appropriate response
if [ "${DATA}" == "${TEST}" ]; then
  echo "Status: 200 OK"
  echo "Content-Type: test/plain"
  echo
  echo "OK"
  exit 0
fi
 
# anything else would be considered a failure
echo "Status: 500 Internal Server Error"
echo "Content-Type: text/plain"
echo
echo "FAIL: Test failed"

If you ‘chmod 755′ the script, you should be able to run it at the command prompt and get an appropriate response back. You’ll notice the extra ‘Status’ and ‘Content-Type’ headers passed back in the response. This is to conform to the CGI specification which is what Apache will use to talk to your script. Also, please note that you MUST have an extra newline after your response headers for Apache to execute it correctly. In order to execute the above script through Apache, you’ll need the following configuration directive:

ScriptAlias /test/ /location/to/your/test/script.sh

If you get ’403 Forbidden’ errors, make sure you have the appropriate Allow/Deny directives so that Apache can read/execute the script.

Here’s an example of running the script through Apache on my own server:

# curl -v -s http://localhost/test/
* About to connect() to localhost port 80
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 80
> GET /test/ HTTP/1.1
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: localhost
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Tue, 05 Apr 2011 23:50:08 GMT
< Server: Apache/2.2.3 (CentOS)
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: test/plain
OK

That should be it! You can write whatever you want in your shell script as long as you pass back proper response headers. If you need extra information about writing CGI scripts for Apache, you can see their documentation.

Finding yesterday’s date in Linux / Unix

I recently wrote a set of scripts to backup a production database on a nightly basis. The ultimate goal is to setup a DR site which automatically restores day-old backups. In order to accomplish this, I need to be able to figure out the date for ‘yesterday’, and I didn’t want to resort to using any other code (perl, PHP, python, etc). It turns out that reasonably modern versions of the ‘date’ command can do this for you:

Today’s Date:

# date +'%Y-%m-%d'
2011-04-02

Yesterday’s Date:

# date +'%Y-%m-%d' --date='1 day ago'
2011-04-01

A month ago:

# date +'%Y-%m-%d' --date='1 month ago'
2011-03-02

You can also convert between timestamps:

# date +'%s'
1301786252
# date --date='@1301786252' +'%Y-%m-%d'
2011-04-02

Doing math with timestamps could be useful too:

# TS=$(date +'%s'); let TS="${TS} - 604800"; date --date="@${TS}" +'%Y-%m-%d'
2011-03-26

Advanced mod_rewrite: Using RewriteMap & MD5 hashes

I’ve run into several situations where I had to intelligently store and serve a large number of static files from a local filesystem. In a recent project, I was tasked with creating a static copy of a website with millions of URIs. Based on prior experience, I had become quite accustomed to hashing the entire URI, and using the hash to create an on-disk directory tree for the data. You never want to store too many files in a single directory, so some form of hashing strategy is a must. Here’s an example of the hashing strategy I will be using:

If the URI is:

http://mykospark.net/2011/03/advanced-mod_rewrite-using-rewritemap/

Then the hash is:
(-n is used to suppress the automatic newline which would alter the hash)

$ echo -n "http://mykospark.net/2011/03/advanced-mod_rewrite-using-rewritemap/" | md5
b269b0888e800c4d4e691dd7c619179f

And here’s how the file might look on disk:

/var/www/data/b2/69/b269b0888e800c4d4e691dd7c619179f.html

See what I did there? I took the first 2 characters of the MD5 hash, and used them to create the first level of directories. I then used the next 2 characters of the hash to form the second level of directories. I then put an extension on the file so Apache knows what Content-Type to put in the response headers. If you do a little math on my directory structure, you’ll find that you have 256 possible directories at the first and second level EACH. This means 65,536 directories total. If you have 10 million URIs to deal with, then you’ll have about 153 files per second level directory on average. You might be wondering about MD5′s ability to evenly spread, and I will tell you that in most situations, the spread is very even. You may also wonder about potential collisions with MD5, but I will tell you that it’s HIGHLY unlikely.

Ok, so now you’ve got this wickedly awesome generic hashing scheme for just about any URI. The next question is how do you get Apache to go from a URI to a hashed file on disk? If we were using existing components of the URI to create the directory structure, then you could probably get away with a few RewriteRules, but mod_rewrite has no way to actually get the MD5 hash of a URI. RewriteMap comes to the rescue!

RewriteMap is an awesome feature of mod_rewrite that greatly expands its feature set. Apache’s mod_rewrite has a few built-in functions for RewriteMap that allow you do things such as:

• Use a plain text file for Key->Value mappings
• Use a DBM-type file for Key->Value mappings
• Uppercase/Lowercase URI elements
• Escape/Unescape URI elements
• Use an external binary to provide a mapping

The feature I will talk about is using an external binary to provide mappings. The Apache documentation on this feature is located at:

http://httpd.apache.org/docs/current/mod/mod_rewrite.html#rewritemap

In order to satisfy the above hashing algorithm, I have used the following directives in my Apache config:

# setup md5-hash map
RewriteEngine On
RewriteMap hashed_path prg:/usr/local/bin/apache_md5.py
 
# translate the URI to an on-disk path
RewriteRule ^(.*)$ /var/www/data/${hashed_path:%{ENV:SCRIPT_URI}} [NC,L]

In the example above, we have initialized the RewriteEngine, setup a RewriteMap called ‘hashed_path’, assigned ‘hashed_path’ to the external program /usr/local/bin/apache_md5.py, and used ‘hashed_path’ to translate SCRIPT_URI to our on-disk path. The following document explain the environment variable SCRIPT_URI:

http://httpd.apache.org/docs/current/mod/mod_rewrite.html#EnvVar

You can use any language to create the translation script for RewriteMap. The only requirement is that it accepts queries on stdin, and outputs responses on stdout. I chose Python, and here’s my script:

#!/usr/bin/python2.6 -u
import urlparse
import hashlib
import sys
 
if __name__ == "__main__":
  while True:
    # grab line from stdin and sanitize it
    line = sys.stdin.readline().strip()
    if not line:
      break
 
    # parse it for the extension, html is a default
    urlparts = urlparse.urlparse(line)
    extension = urlparts[2].rpartition('.')[2]
    if extension == urlparts[2]:
      extension = "html"
 
    # hash it
    m = hashlib.md5()
    m.update(line)
    md5_hash = m.hexdigest()
 
    # build path
    result = "%s/%s/%s.%s" % (md5_hash[0:2], md5_hash[2:4], md5_hash, extension)
 
    # send it back
    print "%s" % result

As you can see, the script is very simple. Here’s a summary on what it does:

• Grab a line from stdin, this will be SCRIPT_URI from Apache
• Use ‘urlparse’ to get the path portion of the URI
• Attempt to get the extension of the file
• If we didn’t seem to get the extension, then come up with a default
• Use ‘hashlib’ to hash the entire URI from SCRIPT_URI
• Output the entire path on stdout

And that’s it! Apache should now be able to translate between your public URI space and your custom hashed on-disk solution.

Central LDAP authentication with OpenDS

I recently started a new project to introduce central authentication to my environment. Based on my own experience, and some additional research, I decided to go with OpenDS for my directory server. My goal is to create a simple and stable system.

The first thing you need to do is ensure that you have Java-1.5 or better installed on your system. I used the java-1.6.0-openjdk package that came with my CentOS 5.5 system:

# yum install java-1.6.0-openjdk

Next, you need to download the OpenDS core package onto your directory server. At the time of this writing, OpenDS-2.2.1 is the latest version:

# cd /opt/
# wget "http://www.opends.org/promoted-builds/2.2.1/OpenDS-2.2.1.zip"
# unzip OpenDS-2.2.1.zip
# mv OpenDS-2.2.1 opends
# rm OpenDS-2.2.1.zip

The OpenDS package comes with a nifty setup tool that helps you get the ball rolling:

# cd /opt/opends/
# ./setup --cli

You should be able to use the defaults for almost everything. Here’s what I picked:

• Initial root user DN: cn=Directory Manager
• LDAP port: 389
• Administration Port: 4444
• Base DN: dc=example,dc=com
• Database population: Only create the base entry
• SSL/TLS: No
• Start Server: No

The OpenDS package doesn’t come with an init script for RedHat/CentOS, but you can download one from this article:

# cd /etc/init.d
# wget "https://www.opends.org/wiki/attach/ManagingTheDsAsARedhatService/opends"
# chmod 755 opends
# chkconfig --add opends
# chkconfig opends on
# /etc/init.d/opends start

At this point, you should be able to use the OpenDS control panel to configure your users and groups. You’ll want to download the OpenDS distribution on your workstation and run bin/control-panel:

You’ll notice on the right side of the control panel that you can manage your organization with a very simple-to-use GUI.  Let’s walk through creating a very basic structure with a single user:

• Right-click on “dc=example,dc=com” -> New Organization
  • Name: My Company
• Right-click on “My Company” -> New Organizational Unit
  • Name: Sales
• Right-click on “Sales” -> New User
  • First Name: John
  • Last Name: Smith
  • Common Name: John Smith
  • User ID: jsmith
• For the user “John Smith”, click on the Edit button for “Object Class”
  • Add “posixAccount”
• Update “John Smith” with posix attributes
  • gidNumber: 1000
  • homeDirectory: /home/jsmith
  • uidNumber: 1000
• Click on “Save Changes”

Your screen should now look like the following:

Now we’ll create a posix group:

• Right-click on “Sales” -> New Group
  • Name: Users
  • Members: John Smith
• For the group “Users”, click on the Edit button for “Object Class”
  • Add “posixGroup”
• Update “Users” with posix attributes
  • gidNumber: 1000

And here’s what your control panel should look like now:

Now that we have our first user and group in the system, I’ll show you how to setup your clients to authenticate against this directory server via LDAP.  On RedHat-based systems, you can use the authconfig-tui interface, but I will be working directly with the relevant config files.

The first step on the client is to see if you’re able to communicate with the directory server. We’ll be using the ldapsearch tool to demonstrate this. If it isn’t installed, do so now:

# yum install openldap-clients

Next, go ahead and try to do an anonymous search. We aren’t using SASL authentication (yet), so you’ll want to pass the -x option:

# ldapsearch -H ldap://opends02/ -x
# extended LDIF
#
# LDAPv3
# base <> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
 
# example.com
dn: dc=example,dc=com
dc: example
objectClass: domain
objectClass: top
 
# My Company, example.com
dn: o=My Company,dc=example,dc=com
objectClass: top
objectClass: organization
o: My Company
 
# Sales, My Company, example.com
dn: ou=Sales,o=My Company,dc=example,dc=com
ou: Sales
objectClass: organizationalUnit
objectClass: top
 
# John Smith, Sales, My Company, example.com
dn: cn=John Smith,ou=Sales,o=My Company,dc=example,dc=com
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: top
givenName: John
uid: jsmith
cn: John Smith
sn: Smith
homeDirectory: /home/jsmith
uidNumber: 1000
gidNumber: 1000
 
# Users, Sales, My Company, example.com
dn: cn=Users,ou=Sales,o=My Company,dc=example,dc=com
objectClass: groupOfUniqueNames
objectClass: posixGroup
objectClass: top
gidNumber: 1000
cn: Users
uniqueMember: cn=John Smith,ou=Sales,o=My Company,dc=example,dc=com
 
# search result
search: 2
result: 0 Success
 
# numResponses: 6
# numEntries: 5

The above output should look very familiar. It’s what we just setup in OpenDS. You can update /etc/openldap/ldap.conf to specify some defaults for ldapsearch:

# cat /etc/openldap/ldap.conf
URI ldap://opends02/
BASE dc=example,dc=com
# ldapsearch -x

Now that we can talk to the directory server, let’s setup login authentication. You’ll need the nss_ldap package to start:

# yum install nss_ldap

The first config file to update is /etc/pam.d/system-auth. In addition to querying LDAP for logins, we can also use pam_mkhomedir.so to automatically create our home directories if they don’t exist:

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 500 quiet
auth        sufficient    pam_ldap.so use_first_pass config=/etc/ldap.conf
auth        required      pam_deny.so
 
account     required      pam_unix.so broken_shadow
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 500 quiet
account     [default=bad success=ok user_unknown=ignore] pam_ldap.so config=/etc/ldap.conf
account     required      pam_permit.so
 
password    requisite     pam_cracklib.so try_first_pass retry=3
password    sufficient    pam_unix.so md5 shadow nullok try_first_pass use_authtok
password    sufficient    pam_ldap.so use_authtok config=/etc/ldap.conf
password    required      pam_deny.so
 
session     required      pam_mkhomedir.so skel=/etc/skel/ umask=0022
session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so
session     optional      pam_ldap.so config=/etc/ldap.conf

You’ll also need to configure the Name Service Switch to use ldap. It’s at /etc/nsswitch.conf:

passwd:     files ldap
shadow:     files ldap
group:      files ldap
hosts:      files dns
bootparams: nisplus [NOTFOUND=return] files
ethers:     files
netmasks:   files
networks:   files
protocols:  files
rpc:        files
services:   files
netgroup:   files ldap
publickey:  nisplus
automount:  files ldap
aliases:    files nisplus

The last config file to edit is /etc/ldap.conf. This tells the PAM LDAP module how to talk to our directory server:

# this is what we're connecting to
uri ldap://opends02/
 
# The distinguished name of the search base
base ou=sales,o=my company,dc=example,dc=com
 
# The port.
port 389
 
# timelimits
bind_timelimit 5
timelimit 10
idle_timelimit 3600
 
# report errors back to the app right away
bind_policy soft
 
# The user ID attribute (defaults to uid)
pam_login_attribute uid
 
# some sensible defaults for attributes not defined in the directory
nss_default_attribute_value homeDirectory /tmp
nss_default_attribute_value loginShell /bin/bash
 
# ignore users that we know are local
nss_initgroups_ignoreusers root,ldap,named,avahi,haldaemon,dbus,radvd,tomcat,radiusd,news,mailman,nscd,gdm
 
# password change method (OpenLDAP extended operation)
pam_password exop

Now the fun part! We’re going to login as “jsmith” for the first time:

$ ssh jsmith@ldapclient02
jsmith@ldapclient02's password:
Creating directory '/home/jsmith'.
[jsmith@ldapclient02 ~]$ id
uid=1000(jsmith) gid=1000(Users) groups=1000(Users)
[jsmith@ldapclient02 ~]$

You’ll notice that the home directory was automatically created, the bash profile was copied into place, and he’s already a member of the ‘Users’ group which we created in OpenDS. We’re not done yet though, we’re still allowing anonymous access and sending passwords in plain-text over the network. Let’s tackle the anonymous access issue by requiring authentication.

Go back to the OpenDS control panel and make sure you’re looking at the “Manage Entries” window:

• For “Base DN” near the top of the window, select “All Base DN’s”
• Expand “cn=config”
• Select “Access Control Handler”
• In the menu, select View -> Attribute View
• Look for the ds-cfg-global-aci attribute with (targetattr!=”userPassword||authPassword”)
• At the end of the attribute text, the “userdn” is set to “ldap:///anyone”
• Change the “userdn” to “ldap:///all”
• Save Changes

The default setting of “ldap:///anyone” allows anonymous access.  If we change this to “ldap:///all”, then the directory data is only visible to authenticated users.  Let’s verify this by running our ldapsearch tool again:

# ldapsearch -x
# extended LDIF
#
# LDAPv3
# base <> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
 
# search result
search: 2
result: 0 Success
 
# numResponses: 1

Excellent. But now we have a problem. Our user can no longer login. That’s because /etc/ldap.conf is telling the PAM LDAP module to try and bind to the directory without any credentials. Using the instructions previously mentioned in this article, create a user called “Bind Service” under “dc=example,dc=com”. This user does NOT need the posixAccount object class:

First, we’ll test this with the ldapsearch tool:

# ldapsearch -D "cn=Bind Service,dc=example,dc=com" -x -W
Enter LDAP Password: 
# extended LDIF
#
# LDAPv3
# base <> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
 
# example.com
dn: dc=example,dc=com
dc: example
objectClass: domain
objectClass: top
 
# My Company, example.com
dn: o=My Company,dc=example,dc=com
objectClass: top
objectClass: organization
o: My Company
 
# Sales, My Company, example.com
dn: ou=Sales,o=My Company,dc=example,dc=com
ou: Sales
objectClass: organizationalUnit
objectClass: top
 
# John Smith, Sales, My Company, example.com
dn: cn=John Smith,ou=Sales,o=My Company,dc=example,dc=com
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: top
givenName: John
uid: jsmith
cn: John Smith
sn: Smith
homeDirectory: /home/jsmith
uidNumber: 1000
gidNumber: 1000
 
# Users, Sales, My Company, example.com
dn: cn=Users,ou=Sales,o=My Company,dc=example,dc=com
objectClass: groupOfUniqueNames
objectClass: posixGroup
objectClass: top
gidNumber: 1000
cn: Users
uniqueMember: cn=John Smith,ou=Sales,o=My Company,dc=example,dc=com
 
# Bind Service, example.com
dn: cn=Bind Service,dc=example,dc=com
userPassword:: e1NTSEF9VE50WEEyYUN5Vk5jOUZMM3VOb1FwVSsyWVdsTmxUdDRPNE0wMmc9PQ=
 =
objectClass: person
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: top
givenName: Bind
cn: Bind Service
sn: Service
 
# search result
search: 2
result: 0 Success
 
# numResponses: 7
# numEntries: 6

Looks like we’re back in business. Next, we’ll update /etc/ldap.conf with the service account info:

binddn cn=bind service,dc=example,dc=com
bindpw password

And now you can login again! But we’re still not done. All authentication is happening in plain-text right now. We’re going to setup authentication via SASL DIGEST-MD5. First, install the cyrus-sasl-md5 package:

yum install cyrus-sasl-md5

Next, add the SASL directive to /etc/ldap.conf:

pam_sasl_mech DIGEST-MD5

Next, simulate a login using ldapsearch:

# ldapsearch -D "cn=John Smith,ou=sales,o=my company,dc=example,dc=com" -Y DIGEST-MD5 -W -U jsmith
Enter LDAP Password: 
SASL/DIGEST-MD5 authentication started
ldap_sasl_interactive_bind_s: Invalid credentials (49)

Hmm, it’s still broken. After reviewing the access logs on the directory server (/opt/opends/logs/access), we find that SASL authentication requires a password stored in a reversible format. By default, OpenDS uses SHA1 encryption which is a one-way non-reversible hash. Let’s use AES encryption instead. Instead of going through the control panel, we’ll use the command-line interface on the directory server to update the default password policy:

# cd /opt/opends/
# bin/dsconfig

• Select “Password Policy” (28)
• Select “View and edit an existing Password Policy” (3)
• Select “Default Password Policy” (1)
• Select “default-password-storage-scheme” (4)
• Select “Add one or more values” (2)
• Select “AES” (2)
• Select “Use these values” (1)
• Select “finish” (f)

You’ll also need to reset the users’ password in the control panel so you can store the AES version of the password.  At this point, your login should be working.  If you are seeing the error “DIGEST-MD5: digest response format violation. Mismatched URI”, then make sure the FQDN of your directory server and the FQDN sent by the client match up.

Congratulations! You now have a working directory server.

Apache HTTPD CustomLog to Syslog via UDP

If you’re reading this post, then you’ve probably discovered that Apache’s mod_log_config module does not support logging to syslog. There are numerous articles on the internet showing you how to write a simple perl script to read from stdin, and send to syslog. I wanted to use the tools that came with my distribution and not resort to writing custom scripts, so I decided to go with netcat.

Netcat is a very simple tool that comes with almost every Linux distribution to send or receive data over the network. You can take advantage of log piping to use netcat just like you would use a simple perl script. The best part is – you don’t have to write anything!

If you want the quick ‘n dirty version, you can use the following in your Apache config:

# obviously, replace 'localhost' with your syslog host
CustomLog "| nc -u localhost 514" combined

The above directive tells mod_log_config to create a piped logger to netcat. The ‘-u’ option on netcat means we want UDP, then you can specify the host/port to send to.

Now, here is where things get interesting. I began noticing that logs with a payload longer than 1024 characters were being cut off. After a little bit of testing, I noticed that netcat (on RHEL5) would only read 1024 bytes from stdin before sending the UDP packet. If there was still additional data to be read on stdin, then netcat would continue reading it (1024 bytes at a time), and send additional UDP packets as necessary. Since UDP is a message-oriented protocol, those additional UDP packets are interpreted as separate log entries which is probably NOT what you wanted. Here’s proof from the perspective of strace:

# dd if=/dev/zero bs=2048 count=1 2>/dev/null | tr '\0' 'X' | strace -e read,write,connect nc -u localhost 514
...
connect(3, {sa_family=AF_INET, sin_port=htons(514), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
read(0, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1024) = 1024
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1024) = 1024
read(0, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1024) = 1024
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 1024) = 1024
...

As you can see from the above, I’ve sent a string of 2048 ‘X’ characters to netcat. The strace output tells us that netcat reads from stdin using a 1024-byte buffer, then immediately sends data to the UDP socket. If you want further proof, of this being a netcat issue, see the following code snippet:

/*
 * readwrite()
 * Loop that polls on the network file descriptor and stdin.
 */
void
readwrite(int nfd)
{
  struct pollfd pfd[2];
  unsigned char buf[8192];
  int n, wfd = fileno(stdin);
  int lfd = fileno(stdout);
  int plen;
 
  plen = jflag ? 8192 : 1024;
...

Well that’s interesting. The variable ‘plen’ defaults to 1024 unless ‘jflag’ is set? What is jflag?

/* Command Line Options */
int dflag;          /* detached, no stdin */
int iflag;          /* Interval Flag */
int jflag;          /* use jumbo frames if we can */

Ok, that makes sense. There’s even a switch/case to handle ‘-j’ and set ‘jflag’. But why isn’t it documented?

void
help(void)
{
  usage(0);
  fprintf(stderr, "\tCommand Summary:\n\
  \t-4    Use IPv4\n\
  \t-6    Use IPv6\n\
  \t-D    Enable the debug socket option\n\
  \t-d    Detach from stdin\n\
  \t-h    This help text\n\
  \t-i secs\t Delay interval for lines sent, ports scanned\n\
  \t-k    Keep inbound sockets open for multiple connects\n\
  \t-l    Listen mode, for inbound connects\n\
  \t-n    Suppress name/port resolutions\n\
  \t-p port\t Specify local port for remote connects\n\
  \t-r    Randomize remote ports\n\
  \t-S    Enable the TCP MD5 signature option\n\
  \t-s addr\t Local source address\n\
  \t-T ToS\t  Set IP Type of Service\n\
  \t-t    Answer TELNET negotiation\n\
  \t-U    Use UNIX domain socket\n\
  \t-u    UDP mode\n\
  \t-v    Verbose\n\
  \t-w secs\t Timeout for connects and final net reads\n\
  \t-X proto  Proxy protocol: \"4\", \"5\" (SOCKS) or \"connect\"\n\
  \t-x addr[:port]\tSpecify proxy address and port\n\
  \t-z    Zero-I/O mode [used for scanning]\n\
  Port numbers can be individual or ranges: lo-hi [inclusive]\n");
  exit(1);
}

Well that’s funny. Why would they want to hide that feature? I don’t know the answer, but it sure solves my issue:

# dd if=/dev/zero bs=2048 count=1 2>/dev/null | tr '\0' 'X' | strace -e read,write,connect nc -j -u localhost 514
...
connect(3, {sa_family=AF_INET, sin_port=htons(514), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
read(0, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 8192) = 2048
write(3, "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"..., 2048) = 2048

So, if you’re having issues with long log lines showing up in syslog, then you may want to try the following CustomLog directive:

# obviously, replace 'localhost' with your syslog host
CustomLog "| nc -j -u localhost 514" combined

As always, I hope someone finds this useful!

Where did InnoDB go?

Have you ever run into an issue where InnoDB gets disabled, or just doesn’t appear to exist? Your issue is likely corrupted log files. On a server I recently worked on, I saw the following output from ‘show engines’:

mysql> show engines;
+------------+---------+----------------------------------------------------------------+--------------+------+------------+
| Engine     | Support | Comment                                                        | Transactions | XA   | Savepoints |
+------------+---------+----------------------------------------------------------------+--------------+------+------------+
| MyISAM     | DEFAULT | Default engine as of MySQL 3.23 with great performance         | NO           | NO   | NO         |
| MRG_MYISAM | YES     | Collection of identical MyISAM tables                          | NO           | NO   | NO         |
| BLACKHOLE  | YES     | /dev/null storage engine (anything you write to it disappears) | NO           | NO   | NO         |
| CSV        | YES     | CSV storage engine                                             | NO           | NO   | NO         |
| MEMORY     | YES     | Hash based, stored in memory, useful for temporary tables      | NO           | NO   | NO         |
| FEDERATED  | NO      | Federated MySQL storage engine                                 | NULL         | NULL | NULL       |
| ARCHIVE    | YES     | Archive storage engine                                         | NO           | NO   | NO         |
+------------+---------+----------------------------------------------------------------+--------------+------+------------+
7 rows in set (0.00 sec)

As you can see, InnoDB isn’t even an option! You may also get the following error:

mysql> show engine innodb status;
ERROR 1286 (42000): Unknown table engine 'innodb'

Fortunately for you, the fix could be very simple. Sometimes InnoDB’s log files get corrupted. These log files track changes to InnoDB structures similar to how binlogs track changes to actual data. You can easily fix the problem like so:

# /etc/init.d/mysql stop
Shutting down MySQL.                                       [  OK  ]
# rm -f ib_logfile0 ib_logfile1
# /etc/init.d/mysql start
Starting MySQL...............................              [  OK  ]
mysql> show engines;
+------------+---------+----------------------------------------------------------------+--------------+------+------------+
| Engine     | Support | Comment                                                        | Transactions | XA   | Savepoints |
+------------+---------+----------------------------------------------------------------+--------------+------+------------+
| InnoDB     | YES     | Supports transactions, row-level locking, and foreign keys     | YES          | YES  | YES        |
| MRG_MYISAM | YES     | Collection of identical MyISAM tables                          | NO           | NO   | NO         |
| BLACKHOLE  | YES     | /dev/null storage engine (anything you write to it disappears) | NO           | NO   | NO         |
| CSV        | YES     | CSV storage engine                                             | NO           | NO   | NO         |
| MEMORY     | YES     | Hash based, stored in memory, useful for temporary tables      | NO           | NO   | NO         |
| FEDERATED  | NO      | Federated MySQL storage engine                                 | NULL         | NULL | NULL       |
| ARCHIVE    | YES     | Archive storage engine                                         | NO           | NO   | NO         |
| MyISAM     | DEFAULT | Default engine as of MySQL 3.23 with great performance         | NO           | NO   | NO         |
+------------+---------+----------------------------------------------------------------+--------------+------+------------+
8 rows in set (0.01 sec)

If you think you might need the InnoDB logs (there’s a good chance you don’t), then you can simply move them out of the way.