December 17, 2013

What Does It Take for Security to Get Management Attention?

What does it take to get management's attention, to get the higher ups to be serious about getting something done with securing systems?

I really enjoy discussing and implementing systems security. However, most of my work is focused on performance issues. Somehow performance and/or availability get all the attention of management who assign administrators time to work on something. Management doesn’t feel any "pain" in the security area—except if you fail an audit, at which time the solution is to fix the failure points enough that a "pass," is given and then everyone gets back to correcting the real (i.e., performance, availability – NOT security) issue.

For me, personally, this is very frustrating. Why? Because a company that has finally realized that something needs to be done (suffered a "security breach") will likely not be in business long enough to get anything fixed. Read any security report or attend any security conference and you will hear a common message: The odds are against the average company, which has no security policy or no real implementation of one, relying on an IT audit to spot everything in time to prevent disaster.

So, I ask you – am I crazy – or is it time security was given some love?

Your opinion counts  so tell me what you think will help to get management's attention! Are your systems secured they way they should be? Would you like to do more but there’s no time—or priority—to secure systems as part of best practice? What are your security best practices missing? Does management ignore this issue? What has gotten their attention in the past? Anything? 


October 15, 2013

Are You Being Naughty?

I have an AIX server—on the Internet—and I have been naughty! Shame on me!

My intent is that this server  is  “open” just enough so "random" activity looking for servers to breach does not take it down. I say "random" because I doubt my ISP would be happy if I were the target of directed or sustained attacks. So, I try not to be too inviting.

So, how have I been naughty? By being complacent about an excessive amount of processor usage by the named9 process. What was my bad behavior? Noticing, but choosing to ignore this “cry for help.” Naughty me.

What I should have done?

In an open environment, by definition, an eye-catching activity should activate my attention (curiosity) sufficiently that I look for a root cause. Finally, after weeks of complacency a different unusual activity prodded me into action.

At the start of a system maintenance cycle I changed the default route from the external interface to the internal interface, effectively removing its active or open connection to the Internet. However, after this change I continued to see a high amount of incoming packets on my Internet-facing interface. My expectation was that the interface would quickly go idle, but it did not. Finally, alarm bells went off in my head.

How could this be happening?!

The interface to the “open” (NAT router) interface was still active so incoming packets were still coming in. What was lacking from my side was a route to respond to the incoming packets. Complacency aside—awake finally—I started tcpdump and saw numerous incoming UDP packets directed at my named service.

00:00:00.000000 IP XXX. > XXX.  10754+ [1au] A (QM)? (41)
00:00:00.000584 IP XXX. > XXX.  10754- 0/13/16 (529)
00:00:00.020531 IP XXX. > XXX.  26644+ [1au] A (QM)? (41)
00:00:00.000385 IP XXX. > XXX.  26644- 0/13/16 (529)
00:00:00.005970 IP XXX. > XXX.  11736+ [1au] A (QM)? (41)
00:00:00.000361 IP XXX. > XXX.  11736- 0/13/16 (529)
00:00:00.027968 IP XXX. > XXX.  22637+ [1au] A (QM)? (41)
00:00:00.000422 IP XXX. > XXX.  22637- 0/13/16 (529)
00:00:00.000650 IP XXX. > XXX.  48240+ [1au] A (QM)? (41)
00:00:00.000361 IP XXX. > XXX.  48240- 0/13/16 (529)
00:00:00.088836 IP XXX. > XXX.  50151+ [1au] A (QM)? (41)
00:00:00.000381 IP XXX. > XXX.  50151- 0/13/16 (529)
00:00:00.003490 IP XXX. > XXX.  8824+ [1au] A (QM)? (41)
00:00:00.000142 IP XXX. > XXX.  20359+ [1au] A (QM)? (41)

Feeling Guilty …

I realized I should have suspected something long ago as I had wondered why the named process was frequently the largest consumer of CPU cycles. Now I had a possible explanation for this unexpected load. Fortunately, I did not have to pay dearly for my lack of attention: The result was neither really extreme nor damaging to what was supposed to be happening on the server. However, I realize this could have been traumatic.

What was the root cause of failure here?

I made an assumption and accepted the unusual behavior rather than investigating it. Another way to describe this: I was being lazy. This is my wake-up call—“Do as I preach!” (FYI: I had accepted the high CPU usage as a result of queries being made by tests I was running on other servers. Now, in hindsight, I realize I was being naïve.)

Taking Action

As this is a well-defined situation: protocol UDP, port number 53, I used smit to create an ipsec filter rule that would "shun any host" that tried to query my server using UDP. In other words, dynamically block any host (IP address) that sends a packet to port 53 using UDP incoming on en0 interface (with address XXX.168.2.1). For now, I’m leaving TCP port open. I may need to add a rule to block TCP requests that start “outside” while permitting TCP queries originating on my server to continue.

The effective command created is:

/usr/sbin/genfilt -v 4  -a 'H’ \
  -s '' -m '' -d '' -M '' \
  -c 'udp' -o 'any' -O 'eq' -P '53' -r 'B' -w 'I' -l 'N' -t '0' -i 'en0'





generate filter

-v 4

IP Version 4

-a 'H'

Action shun host

-s '' -m ''

Any Source Addresses -m will match any source address because the match pattern is 0 bits

-d '' ‑M ''

Only this destination address because -M is 32-bit match

-c udp

Protocol udp

-o 'any'

Any source port number

-O 'eq' -P '53'

Destination port number equals 53

-r 'L'

Packets destined/starting from local host

-w 'I'

Direction Incoming

-l 'N'

No logginh

-t 0

Not a tunnel (t0 is not a tunnel)

-i 'en0'

Interface en0

Immediate Results

About 15 minutes after first activating the packet filter, I had 60 dynamically generated deny hosts rules similar to below. When I checked a day later, the count was down to about 30. When I started writing this blog post (after about three days running) the count was down to one.

Since I had not saved any data, I was afraid I would need some time to get some new data for my examples. I need not have feared. The moment I turned ipsec off, activity jumped as if someone was watching. One positive denial (a response) was enough to wake up others. Result: immediately I had the data I needed to write this blog post. Remember, there was only one IP address being actively blocked but within seconds there were three systems making DNS requests (see results). And as I finish the post, the number of systems continued to grow even though no more replies were being sent—currently holding steady between five and nine dynamic deny rules. My guess is that these systems share with one another, and once one reports success more start trying.

The initial collection was direct to screen (CLI) using:

michael@x054:[/home/michael]tcpdump -i en0

When I saw the predominate activity on UDP port 53, I started collecting data to /tmp/named.tcpdump with the following command:

michael@x054:[/home/michael]tcpdump -U -w /tmp/named.tcpdump -i en0 -n -ttt port 53 &

With the data being stored I could experiment with different flags to get a better understanding of my system behavior. I took a liking to the tcpdump –ttt option.

Word to the wise: Do not forget to kill tcpdump -w after you have collected enough data.

michael@x054:[/home/michael]tcpdump -r /tmp/named.tcpdump -i en0 -n -ttt port 53 |\
   sed -e "s/ [0-9]*\./ XXX./g" | head -15

reading from file /tmp/named.tcpdump, link-type 1
00:00:00.000000 IP XXX. > XXX.  10754+ [1au] A (QM)? (41)
00:00:00.000584 IP XXX. > XXX.  10754- 0/13/16 (529)
00:00:00.020531 IP XXX. > XXX.  26644+ [1au] A (QM)? (41)
00:00:00.000385 IP XXX. > XXX.  26644- 0/13/16 (529)
00:00:00.005970 IP XXX. > XXX.  11736+ [1au] A (QM)? (41)
00:00:00.000361 IP XXX. > XXX.  11736- 0/13/16 (529)
00:00:00.027968 IP XXX. > XXX.  22637+ [1au] A (QM)? (41)
00:00:00.000422 IP XXX. > XXX.  22637- 0/13/16 (529)
00:00:00.000650 IP XXX. > XXX.  48240+ [1au] A (QM)? (41)
00:00:00.000361 IP XXX. > XXX.  48240- 0/13/16 (529)
00:00:00.088836 IP XXX. > XXX.  50151+ [1au] A (QM)? (41)
00:00:00.000381 IP XXX. > XXX.  50151- 0/13/16 (529)
00:00:00.003490 IP XXX. > XXX.  8824+ [1au] A (QM)? (41)
00:00:00.000142 IP XXX. > XXX.  20359+ [1au] A (QM)? (41)

I wish I could give an accurate description of what this output means. Unfortunately, I’m not certain. From  what I could find I became concerned that these queries are meant to poison my DNS process. What jumps out is the repeated queries on the domain What I am not showing (for brevity) is the sudden jump in activity from several locations within seconds of deactivating the filter!

Monitoring Continuing Activity

The command below gives me an indication of how busy the outside is by counting the number of deny filters the shun-host action is creating dynamically. Notice that the -a flag needed for the lsfilt command to see the dynamic, active filters:

michael@x054:[/home/michael]lsfilt -v4 -O -a | grep deny

2|deny|XXX.112.51.63||||yes|all|any|0|any|0|both|inbound|no|all packets|0|en0|~314|||
3|deny|XXX.4.173.222||||yes|all|any|0|any|0|both|inbound|no|all packets|0|en0|~184|||
4|deny|XXX.210.112.24||||yes|all|any|0|any|0|both|inbound|no|all packets|0|en0|~184|||

Although my prompt (michael@x054) may not look like it, I have been doing all of these commands with an euid of 0 (aka superuser): tcpdump -U -w ... genfilt ... and (not shown here)  mkfilt -v4 -u - to activate the changes to the rules in the ODM. If you want to clear all active rules, the simplest way is to deactivate then reactivate the ipsec_v4 device using these two commands:

rmdev -l ipsec_v4

mkdev -l ipsec_v4

If you know enough to explain the meaning of the tcpdump example above, please share that knowledge in a comment, and also, maybe suggest a book or a site for people like me to learn more!

September 24, 2013

Improve Your Password-Hashing Algorithms

Recently, I was at a customer site where I heard a conversation between an AIX admin and a Linux admin who were discussing their concerns regarding password-hash compatibility with Linux, active directory (AD) lightweight directory access protocol (LDAP) passwords and AIX.

The focus, from the Linux admin was on the need to have SMD5 support in AD continued. The question they still have is which LDAP server to use: AD, which they use today, or switch to open LDAP, which is supported on the Linux distribution they’re using. Apparently the security department, which is pushing a move to LDAP for all user authentication on all platforms, thinks request for comment (RFC) 2307 schema is enough for all *NIX platforms. 

My opinion is different, but that will come in a different blog post. Today, I want to discuss how compatible AIX is with Linux’s old (SMD5) and new (SHA512) default algorithms. For a quick reference on Linux architecture choices and changes in defaults in password hashing, I recommend this Linux wiki.

AIX supports several password algorithms. These are specified in /etc/security/pwdalg.cfg, which I’ll quote:


* /usr/lib/security/smd5 is a password hashing load module using

* the MD5 algorithm.


* It supports password length up to 255 characters.

* To generate smd5 password hash compatible to standard salted MD5,

* add the following option line for smd5 stanza.

*       lpa_options = std_hash=true

* Note : password hash generated with this option won't be compatible with

* hash generated without this option.



* smd5:

*        lpa_module = /usr/lib/security/smd5


If you would want the AIX password hash to be compatible with the older SMD5 Linux standard (this changed in 2011 to SHA512), you would need to execute the following command:

# set default password algorithm

chsec -f /etc/security/login.cfg -s usw -a pwd_algorithm="smd5"

However, the better choice, which is supported in AIX 5.3, 6.1 and 7.1 (as is SMD5), is to use the improved SHA512 algorithm.

The stanza in /etc/security/pwdalg.cfg is:


        lpa_module = /usr/lib/security/ssha

        lpa_options = algorithm=sha512

So, the command needed to change the default (which is very weak) password hashing algorithm is:

# set default password algorithm

chsec -f /etc/security/login.cfg -s usw -a pwd_algorithm="ssha512"

If you read the end of the Linux architecture wiki entry you’ll notice that the old SMD5 password hashes began with $5 and the new passwords begin with $6. AIX does this differently. The crypt hash has no prefix—it’s just 14 characters exactly as it would have appeared in /etc/passwd 30-plus years ago. The new algorithms have the label of the present working directory (PWD) algorithm stanza in {} brackets. The following are three examples for the guest account . All three passwords are different to make it more fun for someone trying to hack the algorithm. They are similar, though, to make it possible—and I don’t use this as a password anywhere, not even in a demo.

The following are AIX password-hashing examples as found in the “shadow” file, /etc/password/passwd.


        password = Mx2ijZaNMkdF6

        lastupdate = 1379626278

        flags = ADMCHG


        password = {smd5}WjvcMCtD$7p6dLZ4B.zk6rRURpLOk4.

        lastupdate = 1379626360

        flags = ADMCHG


      password = {ssha512}06$/PVgNw6bUHj0sK2l$uE7hd/HHTy3lLEQwPGSSC0VLDYnOo7ARdkLVkuoj9jMkSlY55IFFJW44Q2koQD.MBQ463/KRkcfGOuJX6CC9..

        lastupdate = 1379626451

        flags = ADMCHG

In short, AIX is very versatile in the password-hashing algorithms it supports, and a high degree of additional customization is possible (read /etc/security/pwdalg.cfg for examples/instructions).

Finally, If you haven’t changed your password-hashing algorithm, do it today! To check, use: 

# list current password hashing algorithm

lssec -f /etc/security/login.cfg -s usw -a pwd_algorithm

If the answer is:

usw pwd_algorithm= 

You have not set it, so it’s still 56-bit crypt. If the answer is not “ssha512,” you aren’t using the best (standard) solution available.

July 23, 2013

Exploring OpenLDAP for AIX

Too much work and then vacations have delayed this post, leaving me feeling guilty when there’s really nothing to feel guilty about. I guess, somewhere inside me, there’s a desire to have some element of "BLING" whenever I write something for SecuringAIX. But there’s no need. I am not a professional writer or journalist, and blogs aren’t meant to always BLING.

Anyway, I have been busy with many things, not all directly related to security in the classic sense of system administration—looking for exposures, etc. Mainly I have been focused on topics that could be called "business resilience." 

But I am not going to write about that now. Instead, both in response to comments on SecuringAIX and conversations with customers about configuring Lightweight Directory Access Protocol (LDAP) for AIX, I am researching OpenLDAP.

While between planes, I made my first attempts at packaging OpenLDAP for AIX. Basically, it tried two packages: Berkeley DB (now a product of Oracle) for the database and OpenLDAP itself.

The initial attempts seem to be working, read compiling. The next step will be the “make test” to see what that gives, and then, learn more.

So, the other good thing—for me—about this, is I should have less difficulty getting my next blog post ready for publication. 

I hope you are having, or had, a great summer vacation.

June 04, 2013

Implementing LDAP on AIX Offers a Few Surprises

 This past week was a surprise—and fun! Rather than a cut-and-dried performance review the customer I visited was more interested in getting Lightweight Directory Access Protocol (LDAP) working on AIX. With the repackaging of the DB2 component for IBM Tivoli Directory Server (TDS) LDAP, I have been a bit worried about whether LDAP on AIX (server and client) was still included in the base—a.k.a. no additional charges.

I’ve been playing with TDS LDAP since before it was tdsldap.* filesets (ldap.client.* and ldap.server.*) AND, importantly, the “special install” version of DB2 for the database. Since seeing the AIX 7.1 expansion DVD, I have been concerned that LDAP might not be included. However, that was just a pointless worry—about cost. The short answer to my concern:

Yes—LDAP is still included at no additional charge WHEN it is only used for AIX administration. 

As a footnote, you still must find a download of DB2. The file you’re looking for is something like tds63-aix-ppc64-db2.tar. This is a “restricted-license” version of DB2. That means it may only be used for supporting TDS. With that clarified, the customer said, “OK, let's learn how to install LDAP on AIX.”

I thought, “I have a presentation and scripts I have been using at IBM Power Technical Universities in Europe—so this should work well.” As a change from the Tech U labs, I first created a bundle to do the installation, rather than using the X11 GUI-based install used when you download everything from the IBM try and buy site. If you go that route, be sure and add the file set idsldap.ent63.* from the AIX DVD (base or expansion). We did have two surprises:

1. When using the bundle, you must load the DB2 (ismp/ppc/db2_09_07_00_04) first or one of the idsldap file sets won’t load. If you forget, just load the bundle a second time, because DB2 loads automatically on the first try).

2. When using the sudo Red Hat Package Manager (RPM)—which is an open LDAP RPM—as a prerequisite, the idsldap client file sets do not install completely because openldap has already set a symbolic link to an LDAP library, called .a library.

Other than that, it’s very simple to install and get started. And, yes, I’m not including a lot of commands here—yet. That’s because, I’m late in posting and, speaking honestly, I’m hoping to get some requests for more information. This will give me much better feedback on where to focus my next blog post—installing LDAP or integrating sudo into role-based access control (RBAC).