July 14, 2014

My Role-Based Security Questions for You

This year has been difficult for me to be "security minded." Besides security, I also do performance trouble-shooting and consultancy and this year that has kept me very busy. So, I do not feel like I have anything "new and improved" to share with you about how to improve the AIX security layer.

Instead, I have some questions for you -- and I thank you in advance (and later again, afterwards) for your replies.

My questions are:

1. Have you tested and rejected any role-based access methodology for securing your systems and/or applications?
2. Are you using AIX RBAC?
3. Are you using LINUX RBAC, in either active or passive/monitoring mode?
4. If you have tested, and rejected any role-based mechanism, why are you not adopting it?
5. If you are using role-based mechanisms, are they meeting or exceeding your expectations or needs?

Again - thank you for your reply and feedback!



June 10, 2014

What Good Are Intentions?

Time just keeps slipping by. This feels like a New Years best-intention gone bad. Three times before I have started a SecuringAIX blog and three times I have not finished.

So today must be different. I must finish this blog entry – anything will be better than nothing.

Today feels a lot like many disscusions I have about security on AIX. Again, great intentions, and yes – aixpert looks great, no we have never used it before and we shall look at it in more detail. For now we shall continue to rely on our firewall for security.

Security is like an onion – it has, or should have, many layers. The simple logic is, if they – anyone who should not be somewhere in your systems, regardless of intent – should have yet another layer to watch them, catch them, contain them.

Worldwide, IT marketing expects security to be big business. I guess this means more people are beginning to secure the application. But a secure suitcase on the back seat of your car is not really safe if the doors to the back seat are open, or if your car is a convertable.

And now to myself – and to you if the shoe fits: Security is not having good intentions. Security is secure when both diligent and consistent. Although good intentions may help some too. And maybe, though if I am honest, not likely - I will have another SecuringAIX blog before I go offline (read: on vacation) later this month.


December 17, 2013

What Does It Take for Security to Get Management Attention?

What does it take to get management's attention, to get the higher ups to be serious about getting something done with securing systems?

I really enjoy discussing and implementing systems security. However, most of my work is focused on performance issues. Somehow performance and/or availability get all the attention of management who assign administrators time to work on something. Management doesn’t feel any "pain" in the security area—except if you fail an audit, at which time the solution is to fix the failure points enough that a "pass," is given and then everyone gets back to correcting the real (i.e., performance, availability – NOT security) issue.

For me, personally, this is very frustrating. Why? Because a company that has finally realized that something needs to be done (suffered a "security breach") will likely not be in business long enough to get anything fixed. Read any security report or attend any security conference and you will hear a common message: The odds are against the average company, which has no security policy or no real implementation of one, relying on an IT audit to spot everything in time to prevent disaster.

So, I ask you – am I crazy – or is it time security was given some love?

Your opinion counts  so tell me what you think will help to get management's attention! Are your systems secured they way they should be? Would you like to do more but there’s no time—or priority—to secure systems as part of best practice? What are your security best practices missing? Does management ignore this issue? What has gotten their attention in the past? Anything? 


October 15, 2013

Are You Being Naughty?

I have an AIX server—on the Internet—and I have been naughty! Shame on me!

My intent is that this server  is  “open” just enough so "random" activity looking for servers to breach does not take it down. I say "random" because I doubt my ISP would be happy if I were the target of directed or sustained attacks. So, I try not to be too inviting.

So, how have I been naughty? By being complacent about an excessive amount of processor usage by the named9 process. What was my bad behavior? Noticing, but choosing to ignore this “cry for help.” Naughty me.

What I should have done?

In an open environment, by definition, an eye-catching activity should activate my attention (curiosity) sufficiently that I look for a root cause. Finally, after weeks of complacency a different unusual activity prodded me into action.

At the start of a system maintenance cycle I changed the default route from the external interface to the internal interface, effectively removing its active or open connection to the Internet. However, after this change I continued to see a high amount of incoming packets on my Internet-facing interface. My expectation was that the interface would quickly go idle, but it did not. Finally, alarm bells went off in my head.

How could this be happening?!

The interface to the “open” (NAT router) interface was still active so incoming packets were still coming in. What was lacking from my side was a route to respond to the incoming packets. Complacency aside—awake finally—I started tcpdump and saw numerous incoming UDP packets directed at my named service.

00:00:00.000000 IP XXX. > XXX.  10754+ [1au] A (QM)? (41)
00:00:00.000584 IP XXX. > XXX.  10754- 0/13/16 (529)
00:00:00.020531 IP XXX. > XXX.  26644+ [1au] A (QM)? (41)
00:00:00.000385 IP XXX. > XXX.  26644- 0/13/16 (529)
00:00:00.005970 IP XXX. > XXX.  11736+ [1au] A (QM)? (41)
00:00:00.000361 IP XXX. > XXX.  11736- 0/13/16 (529)
00:00:00.027968 IP XXX. > XXX.  22637+ [1au] A (QM)? (41)
00:00:00.000422 IP XXX. > XXX.  22637- 0/13/16 (529)
00:00:00.000650 IP XXX. > XXX.  48240+ [1au] A (QM)? (41)
00:00:00.000361 IP XXX. > XXX.  48240- 0/13/16 (529)
00:00:00.088836 IP XXX. > XXX.  50151+ [1au] A (QM)? (41)
00:00:00.000381 IP XXX. > XXX.  50151- 0/13/16 (529)
00:00:00.003490 IP XXX. > XXX.  8824+ [1au] A (QM)? (41)
00:00:00.000142 IP XXX. > XXX.  20359+ [1au] A (QM)? (41)

Feeling Guilty …

I realized I should have suspected something long ago as I had wondered why the named process was frequently the largest consumer of CPU cycles. Now I had a possible explanation for this unexpected load. Fortunately, I did not have to pay dearly for my lack of attention: The result was neither really extreme nor damaging to what was supposed to be happening on the server. However, I realize this could have been traumatic.

What was the root cause of failure here?

I made an assumption and accepted the unusual behavior rather than investigating it. Another way to describe this: I was being lazy. This is my wake-up call—“Do as I preach!” (FYI: I had accepted the high CPU usage as a result of queries being made by tests I was running on other servers. Now, in hindsight, I realize I was being naïve.)

Taking Action

As this is a well-defined situation: protocol UDP, port number 53, I used smit to create an ipsec filter rule that would "shun any host" that tried to query my server using UDP. In other words, dynamically block any host (IP address) that sends a packet to port 53 using UDP incoming on en0 interface (with address XXX.168.2.1). For now, I’m leaving TCP port open. I may need to add a rule to block TCP requests that start “outside” while permitting TCP queries originating on my server to continue.

The effective command created is:

/usr/sbin/genfilt -v 4  -a 'H’ \
  -s '' -m '' -d '' -M '' \
  -c 'udp' -o 'any' -O 'eq' -P '53' -r 'B' -w 'I' -l 'N' -t '0' -i 'en0'





generate filter

-v 4

IP Version 4

-a 'H'

Action shun host

-s '' -m ''

Any Source Addresses -m will match any source address because the match pattern is 0 bits

-d '' ‑M ''

Only this destination address because -M is 32-bit match

-c udp

Protocol udp

-o 'any'

Any source port number

-O 'eq' -P '53'

Destination port number equals 53

-r 'L'

Packets destined/starting from local host

-w 'I'

Direction Incoming

-l 'N'

No logginh

-t 0

Not a tunnel (t0 is not a tunnel)

-i 'en0'

Interface en0

Immediate Results

About 15 minutes after first activating the packet filter, I had 60 dynamically generated deny hosts rules similar to below. When I checked a day later, the count was down to about 30. When I started writing this blog post (after about three days running) the count was down to one.

Since I had not saved any data, I was afraid I would need some time to get some new data for my examples. I need not have feared. The moment I turned ipsec off, activity jumped as if someone was watching. One positive denial (a response) was enough to wake up others. Result: immediately I had the data I needed to write this blog post. Remember, there was only one IP address being actively blocked but within seconds there were three systems making DNS requests (see results). And as I finish the post, the number of systems continued to grow even though no more replies were being sent—currently holding steady between five and nine dynamic deny rules. My guess is that these systems share with one another, and once one reports success more start trying.

The initial collection was direct to screen (CLI) using:

michael@x054:[/home/michael]tcpdump -i en0

When I saw the predominate activity on UDP port 53, I started collecting data to /tmp/named.tcpdump with the following command:

michael@x054:[/home/michael]tcpdump -U -w /tmp/named.tcpdump -i en0 -n -ttt port 53 &

With the data being stored I could experiment with different flags to get a better understanding of my system behavior. I took a liking to the tcpdump –ttt option.

Word to the wise: Do not forget to kill tcpdump -w after you have collected enough data.

michael@x054:[/home/michael]tcpdump -r /tmp/named.tcpdump -i en0 -n -ttt port 53 |\
   sed -e "s/ [0-9]*\./ XXX./g" | head -15

reading from file /tmp/named.tcpdump, link-type 1
00:00:00.000000 IP XXX. > XXX.  10754+ [1au] A (QM)? (41)
00:00:00.000584 IP XXX. > XXX.  10754- 0/13/16 (529)
00:00:00.020531 IP XXX. > XXX.  26644+ [1au] A (QM)? (41)
00:00:00.000385 IP XXX. > XXX.  26644- 0/13/16 (529)
00:00:00.005970 IP XXX. > XXX.  11736+ [1au] A (QM)? (41)
00:00:00.000361 IP XXX. > XXX.  11736- 0/13/16 (529)
00:00:00.027968 IP XXX. > XXX.  22637+ [1au] A (QM)? (41)
00:00:00.000422 IP XXX. > XXX.  22637- 0/13/16 (529)
00:00:00.000650 IP XXX. > XXX.  48240+ [1au] A (QM)? (41)
00:00:00.000361 IP XXX. > XXX.  48240- 0/13/16 (529)
00:00:00.088836 IP XXX. > XXX.  50151+ [1au] A (QM)? (41)
00:00:00.000381 IP XXX. > XXX.  50151- 0/13/16 (529)
00:00:00.003490 IP XXX. > XXX.  8824+ [1au] A (QM)? (41)
00:00:00.000142 IP XXX. > XXX.  20359+ [1au] A (QM)? (41)

I wish I could give an accurate description of what this output means. Unfortunately, I’m not certain. From  what I could find I became concerned that these queries are meant to poison my DNS process. What jumps out is the repeated queries on the domain What I am not showing (for brevity) is the sudden jump in activity from several locations within seconds of deactivating the filter!

Monitoring Continuing Activity

The command below gives me an indication of how busy the outside is by counting the number of deny filters the shun-host action is creating dynamically. Notice that the -a flag needed for the lsfilt command to see the dynamic, active filters:

michael@x054:[/home/michael]lsfilt -v4 -O -a | grep deny

2|deny|XXX.112.51.63||||yes|all|any|0|any|0|both|inbound|no|all packets|0|en0|~314|||
3|deny|XXX.4.173.222||||yes|all|any|0|any|0|both|inbound|no|all packets|0|en0|~184|||
4|deny|XXX.210.112.24||||yes|all|any|0|any|0|both|inbound|no|all packets|0|en0|~184|||

Although my prompt (michael@x054) may not look like it, I have been doing all of these commands with an euid of 0 (aka superuser): tcpdump -U -w ... genfilt ... and (not shown here)  mkfilt -v4 -u - to activate the changes to the rules in the ODM. If you want to clear all active rules, the simplest way is to deactivate then reactivate the ipsec_v4 device using these two commands:

rmdev -l ipsec_v4

mkdev -l ipsec_v4

If you know enough to explain the meaning of the tcpdump example above, please share that knowledge in a comment, and also, maybe suggest a book or a site for people like me to learn more!

September 24, 2013

Improve Your Password-Hashing Algorithms

Recently, I was at a customer site where I heard a conversation between an AIX admin and a Linux admin who were discussing their concerns regarding password-hash compatibility with Linux, active directory (AD) lightweight directory access protocol (LDAP) passwords and AIX.

The focus, from the Linux admin was on the need to have SMD5 support in AD continued. The question they still have is which LDAP server to use: AD, which they use today, or switch to open LDAP, which is supported on the Linux distribution they’re using. Apparently the security department, which is pushing a move to LDAP for all user authentication on all platforms, thinks request for comment (RFC) 2307 schema is enough for all *NIX platforms. 

My opinion is different, but that will come in a different blog post. Today, I want to discuss how compatible AIX is with Linux’s old (SMD5) and new (SHA512) default algorithms. For a quick reference on Linux architecture choices and changes in defaults in password hashing, I recommend this Linux wiki.

AIX supports several password algorithms. These are specified in /etc/security/pwdalg.cfg, which I’ll quote:


* /usr/lib/security/smd5 is a password hashing load module using

* the MD5 algorithm.


* It supports password length up to 255 characters.

* To generate smd5 password hash compatible to standard salted MD5,

* add the following option line for smd5 stanza.

*       lpa_options = std_hash=true

* Note : password hash generated with this option won't be compatible with

* hash generated without this option.



* smd5:

*        lpa_module = /usr/lib/security/smd5


If you would want the AIX password hash to be compatible with the older SMD5 Linux standard (this changed in 2011 to SHA512), you would need to execute the following command:

# set default password algorithm

chsec -f /etc/security/login.cfg -s usw -a pwd_algorithm="smd5"

However, the better choice, which is supported in AIX 5.3, 6.1 and 7.1 (as is SMD5), is to use the improved SHA512 algorithm.

The stanza in /etc/security/pwdalg.cfg is:


        lpa_module = /usr/lib/security/ssha

        lpa_options = algorithm=sha512

So, the command needed to change the default (which is very weak) password hashing algorithm is:

# set default password algorithm

chsec -f /etc/security/login.cfg -s usw -a pwd_algorithm="ssha512"

If you read the end of the Linux architecture wiki entry you’ll notice that the old SMD5 password hashes began with $5 and the new passwords begin with $6. AIX does this differently. The crypt hash has no prefix—it’s just 14 characters exactly as it would have appeared in /etc/passwd 30-plus years ago. The new algorithms have the label of the present working directory (PWD) algorithm stanza in {} brackets. The following are three examples for the guest account . All three passwords are different to make it more fun for someone trying to hack the algorithm. They are similar, though, to make it possible—and I don’t use this as a password anywhere, not even in a demo.

The following are AIX password-hashing examples as found in the “shadow” file, /etc/password/passwd.


        password = Mx2ijZaNMkdF6

        lastupdate = 1379626278

        flags = ADMCHG


        password = {smd5}WjvcMCtD$7p6dLZ4B.zk6rRURpLOk4.

        lastupdate = 1379626360

        flags = ADMCHG


      password = {ssha512}06$/PVgNw6bUHj0sK2l$uE7hd/HHTy3lLEQwPGSSC0VLDYnOo7ARdkLVkuoj9jMkSlY55IFFJW44Q2koQD.MBQ463/KRkcfGOuJX6CC9..

        lastupdate = 1379626451

        flags = ADMCHG

In short, AIX is very versatile in the password-hashing algorithms it supports, and a high degree of additional customization is possible (read /etc/security/pwdalg.cfg for examples/instructions).

Finally, If you haven’t changed your password-hashing algorithm, do it today! To check, use: 

# list current password hashing algorithm

lssec -f /etc/security/login.cfg -s usw -a pwd_algorithm

If the answer is:

usw pwd_algorithm= 

You have not set it, so it’s still 56-bit crypt. If the answer is not “ssha512,” you aren’t using the best (standard) solution available.