Linux Admin Reference – Configuring Auditd in RedHat Enterprise Linux

Guidelines to Implement Audit Rules:

  • – Consolidate your rules where possible.
  • – The system call rules are loaded into a matching engine that intercepts each syscall that all programs on the system makes.
  • – It is very important to only use syscall rules when you have to since these affect performance.
  • – The more rules, the bigger the performance hit. You can help the performance, though, by combining syscalls into one rule whenever possible.-

1. Audit all commands run in the system

The audit package ties in to the Linux kernel audit subsystem.  The audit system audits system calls and other kernel level events, not
user-space events, so we need to audit the execve() system call which is what starts executing new programs.

To log all commands:

a) First, auditd needs to be running.  It should be running by default, but if it’s not, start it with

        #chkconfig auditd on
        #service auditd start

b) If it is a 64 bit architecure then you need to add two rules to catch both 32-bit and  64-bit system calls.

As root, run:

        #auditctl -a exit,always -F arch=b32 -S execve
        #auditctl -a exit,always -F arch=b64 -S execve

c) Run ‘ls /tmp’, these 3 events appears in  /var/log/audit/audit.log.

       lines are wrapped to make it easier to read here, but it’s normally a single long line in the real file.

    type=SYSCALL msg=audit(1296773801.756:35241): arch=c000003e syscall=59
        success=yes exit=0 a0=cf2d10 a1=9da530 a2=cd4e20 a3=8 items=2
        ppid=4146 pid=11827 auid=500 uid=500 gid=500 euid=500
        suid=500 fsuid=500 egid=500 sgid=500 fsgid=500 tty=pts7
        ses=3 comm=”ls” exe=”/bin/ls”
        subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
    type=EXECVE msg=audit(1296773801.756:35241): argc=3 a0=”ls”
        a1=”–color=auto” a2=”/tmp” type=CWD msg=audit(1296773801.756:35241):
        cwd=”/home/username”
    type=PATH msg=audit(1296773801.756:35241): item=0 name=”/bin/ls”
        inode=14418043 dev=fd:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
        obj=system_u:object_r:bin_t:s0 type=PATH
        msg=audit(1296773801.756:35241): item=1 name=(null) inode=20447259
        dev=fd:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
        obj=system_u:object_r:ld_so_t:s0

The audit logs are quite verbose, but they contain the full command, including the fact that ‘ls’ was de-aliased by the shell to  ‘ls –color=auto’!

The first number in parentheses is a timestamp in seconds after the epoch. This can be converted to a human-readable time with a little help from perl:

$ perl -e ‘print scalar(localtime(1296773801.756)) . “\n”;’
    Thu Feb  3 16:56:41 2011

   The ‘exit=0’ in the SYSCALL line above is regarding the execve() system call, not the overall program.

2. Audit all commands for specific user

a) Start auditd service:

        #chkconfig auditd on
        #service auditd start

b)Add audit rule.

For 64-bit architecture:

#auditctl -a exit,always -F arch=b64 -F uid=500 -S execve

For 32-bit architecture:

#auditctl -a exit,always -F arch=b32 -F uid=500 -S execve

Here, uid is of the user for whom auditing is enabled of all commands. Run the auditctl command as root user to add the rules.

c) Verify the logs are being generated by checking /var/log/audit/audit.log file. Once user starts executing commands the logs will appear as below:

type=SYSCALL msg=audit(1393407885.099:8614): arch=c000003e syscall=59 success=yes exit=0 a0=158c9b0 a1=1588f10 a2=15a7ae0 a3=18 items=2 ppid=3123 pid=3307 auid=0 uid=500 gid=500 euid=500 suid=500 fsuid=500 egid=500 sgid=500 fsgid=500 tty=pts1 ses=1 comm=”ls” exe=”/bin/ls” subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
        type=EXECVE msg=audit(1393407885.099:8614): argc=2 a0=”ls” a1=”–color=auto”
        type=CWD msg=audit(1393407885.099:8614):  cwd=”/home/sadaf”
        type=PATH msg=audit(1393407885.099:8614): item=0 name=”/bin/ls” inode=408600 dev=fc:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0
        type=PATH msg=audit(1393407885.099:8614): item=1 name=(null) inode=429303 dev=fc:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s0

        type=SYSCALL msg=audit(1393407910.242:8615): arch=c000003e syscall=59 success=yes exit=0 a0=15971e0 a1=158c9b0 a2=15a7ae0 a3=18 items=2 ppid=3123 pid=3310 auid=0 uid=500 gid=500 euid=500 suid=500 fsuid=500 egid=500 sgid=500 fsgid=500 tty=pts1 ses=1 comm=”rmdir” exe=”/bin/rmdir” subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
        type=EXECVE msg=audit(1393407910.242:8615): argc=2 a0=”rmdir” a1=”data”

NOTE: Audit would generate extensive logs since every command will be recorded which can have a bad impact on system performance.

3. Rotating audit logs on daily basis regardless of size limit rotation

a) Command used to rotate the logs of audit by manually

# service audit rotate        

b) To rotate the logs on daily basis regardless of size,

1. First of all make changes in /etc/audit/auditd.conf file as max_log_file_action = ignore

 max_log_file_action = ignore

max_log_file and max_log_file_action, by default the vaule of max_log_file is 6 MB, and max_log_file_action as ROTATE, which rotates the logs when the size reaches 6 MB.

c) Create script in /etc/cron.daily/auditd.cron

    #!/bin/sh

    ##########
    # This script can be installed to get a daily log rotation
    # based on a cron job.
    ##########

    /sbin/service auditd rotate
    EXITVALUE=$?
    if [ $EXITVALUE != 0 ]; then
        /usr/bin/logger -t auditd “ALERT exited abnormally with [$EXITVALUE]”
    fi
    exit 0

The above Script template can be found at /usr/share/doc/audit-2.2/auditd.cron

 # chmod +x /etc/cron.daily/auditd.cron

Audit logs will now be rotated on the daily basis regardless of size limit of audit logs.

4.  audit system time change?

First, confirm that auditd service is running.

   # service auditd status

Define a new audit rule “adjtime” to audit system time change (system call “adjtimex” and “settimeofday” are called ).

 # auditctl -a entry,always -S adjtimex -S settimeofday -k adjtime

To test the audit rule, search the key name which defined an audit log after changing system time.
 
        # hwclock –hctosys
        # ausearch -k adjtime
        time->Sat Jan 19 22:37:33 2008
        type=SYSCALL msg=audit(1200753453.330:86): arch=40000003 syscall=79 success=yes exit=0 a0=bffeb53c a1=bffeb534 a2=0 a3=0

5. audit SUDO/SU activity for a specific user

This can be achieved using pam_tty_audit.so.  Add the following line in /etc/pam.d/{su,sudo,sudo-i,su-l} files.

session    required     pam_tty_audit.so enable=*

The above config will enable logging of commands run as root.   Logs can listed by running the following command

 aureport –tty

6. Audit the programs that calls Shutdown commands [ RHEL4/5/6]

  • Auditing /sbin/shutdown binary can show what is calling the shutdown command. For this, edit /etc/audit/audit.rules and add the following rule at the end of the file,

    -w /sbin/shutdown

  • Restart auditd using following command.

    $ chkconfig auditd on
    $ service auditd restart

  •  If all execve systemcalls are to be logged to track caller of shutdown closely, use the following rule:

    -a entry,always -S 11

Note: You can ignore “entry rules deprecated, changing to exit rule” warning for the rule.
Caution: The above rule may result in high system load.

  • If you only want to log commands executed as a root, you can use the below rule which is lighter than the previous rule.

        -a entry,always -S execve -F uid=0

  •  If ‘shutdown -r now’ is executed as root, messages similar to this will be logged in /var/log/audit/audit.log:

 

    > type=SYSCALL msg=audit(1273498560.908:29482): arch=c000003e syscall=59 success=yes exit=0 a0=268f9b0 a1=25fafb0 a2=25d  
    > b580 a3=8 items=2 ppid=11742 pid=17741 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 ses=12  
    > comm=”shutdown” exe=”/sbin/shutdown” subj=root:system\_r:unconfined\_t:s0-s0:c0.c1023 key=(null)  
    > type=EXECVE msg=audit(1273498560.908:29482): argc=3 a0=”shutdown” a1=”-r” a2=”now”  
    > type=CWD msg=audit(1273498560.908:29482):  cwd=”/root”  
    > type=PATH msg=audit(1273498560.908:29482): item=0 name=”/sbin/shutdown” inode=3059185 dev=fd:01 mode=0100755 ouid=0 og  
    > id=0 rdev=00:00 obj=system\_u:object\_r:sbin_t:s0

7. Audit a specific SYSCALL [ RHEL5]

  •  Create an audit rule with the following information:

    # vim /etc/audit/audit.rules
    -a entry,always -F arch=b64 -S kill -k teste_kill

Note: “arch” is the CPU architecture of the syscall. If the system is 32 bit OS, you need to set it with “arch=b32”.

  • Restart audit service:

# service auditd restart

  • Test the rule running a kill against some process. In this example we will create and destroy the sleep proccess:

    # vim /etc/audit/audit.rules
    -a entry,always -F arch=b64 -S kill -k teste_kill

    # service auditd restart

    # sleep 100

    # ps aux | grep sleep

    root      1968  0.0  0.0  58876   504 pts/3    S+   16:01   0:00 sleep 100
    root      1975  0.0  0.0  61136   736 pts/4    S+   16:01   0:00 grep sleep

    # kill 1968

    # tail -f /var/log/audit/audit.log
    type=SYSCALL msg=audit(1279134100.434:193): arch=c000003e syscall=62 success=yes exit=0 a0=7b0 a1=f a2=0 a3=0 items=0 ppid=1602 pid=1605 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts4 ses=4294967295 comm=”bash” exe=”/bin/bash” key=”teste_kill”
    type=OBJ_PID msg=audit(1279134100.434:193): opid=1968 oauid=-1 ouid=0 oses=-1 obj=<NULL> ocomm=”sleep”

  • Use audit search to find out the piece of log related to the kill command based on the key we specified in the first step:

    # ausearch -k teste_kill
    —-
    time->Wed Jul 14 16:00:17 2010
    type=CONFIG_CHANGE msg=audit(1279134017.731:186): auid=4294967295 op=add rule key=”teste_kill” list=2 res=1
    —-
    time->Wed Jul 14 16:01:40 2010
    type=OBJ_PID msg=audit(1279134100.434:193): opid=1968 oauid=-1 ouid=0 oses=-1 obj=<NULL> ocomm=”sleep”
    type=SYSCALL msg=audit(1279134100.434:193): arch=c000003e syscall=62 success=yes exit=0 a0=7b0 a1=f a2=0 a3=0 items=0

8. Audit Permission Change Command for a particular directory of file

Set a watch on the required file to be monitored by using the auditctl command:

# auditctl -w /etc/hosts -p war -k monitor-hosts

Alternately, you may place the watch rule in the /etc/audit/audit.rules file:

    # Feel free to add below this line. See auditctl man page
    -w /etc/hosts -p war -k monitor-hosts

And then start (or restart) the audit service with service auditd start (or restart) as required.

In this example, a watch is placed on the /etc/hosts file for any syscalls which perform a write, read, or attribute change (-p war). This is logged with the key monitor-hosts. This key can be used to search through the audit logs to find these actions, using the ausearch command:

   # ausearch -ts today -k monitor-hosts
    —-
    time->Sat Feb  3 07:32:20 2007
    type=PATH msg=audit(1170451940.872:34): item=0 name=”/etc/hosts” inode=1308742 dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:etc_t:s0
    type=CWD msg=audit(1170451940.872:34): cwd=”/root”
    type=SYSCALL msg=audit(1170451940.872:34): arch=40000003 syscall=226 success=yes exit=0 a0=867c4b8 a1=458bcc4f a2=8686800 a3=1c items=1 ppid=3544 pid=3558 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 comm=”vim” exe=”/usr/bin/vim” subj=root:system_r:unconfined_t:s0-s0:c0.c1023 key=”monitor-hosts”
    From this trace, it can be seen that the file /etc/hosts was edited using the /usr/bin/vim command. The user that ran the command was running with the root:system_r:unconfined_t:s0-s0:c0.c1023 SELinux context. And, the timestamp can be converted into readable form.

    # date -d @1170451940

    Sat Feb  3 05:32:20 CST 2007

9. Audit mounting of removable media in RHEL

Define rules using auditctl

    # auditctl -a always,exit -F arch=b64 -S mount -S umount2 -F dir=/media -k media

Or for rules to remain persistent across reboots update file – /etc/audit/audit.rules

   # cat /etc/audit/audit.rules
    -a always,exit -F arch=b64 -S mount -S umount2 -F dir=/media -k media

To view the summary reports for specific key in audit logs using aureport –

    # aureport -k -i | more

    Key Report
    ===============================================
    # date time key success exe auid event
    ===============================================
    # aureport -k -i | grep media
    3. 01/21/2014 11:38:45 media yes ? vineet 21432
    4. 01/21/2014 11:40:01 media no /bin/mount unset 21443
    5. 01/21/2014 11:40:01 media yes /bin/mount unset 21446

10.  audit to monitor file deletion [rhel4/rhel5/rhel6]

Red Hat Enterprise Linux provides audit rules feature to log the file activities done by users or processes. This can be achieved by configuring audit rules.

To configure audit rules, add the following line in /etc/audit.rules file

For Red Hat Enterprise Linux 4:

    # tail -2 /etc/audit.rules  
    auditctl -a exit,always -F arch=b32 -S unlink
    auditctl -a exit,always -F arch=b64 -S unlink

For Red Hat Enterprise Linux 5 and 6:

    # tail -2 /etc/audit/audit.rules
    -a always,exit -F arch=b32 -S unlink -S unlinkat -S rename -S renameat -k delete
    -a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -k delete
After writing the rules, restart the service auditd and make it on in to retain across reboot.

 # /etc/init.d/auditd restart
  # chkconfig auditd on

This will log the file deletion operations.

11. Audit /etc/shadow changes

To monitor /etc/shadow file with Auditd do the following:

Edit /etc/audit/audit.rules and paste the line below

-w /etc/shadow -p wa -k shadow

12. Exclude a Directory/File from the audit rule

Add below Format to exclude a directory from audit rule.

-a exit,never -F dir=/path/to/directory/

Add below Format to exclude a file from audit rule.

-a exit,never -F path=/etc/yum.repos.d/rhds.repo

Add below format to exclude all operations from a uid.

-a exit,never -F auid=<UID number>

Note : never rule must be before the always rule on the directory.

13. Audit the source of SIGKILL Process [ RHEL4/RHEL5/RHEL6]

It’s possible to configure an audit rule for capturing kill signals.
The rule should be something like:

    # auditctl  -a exit,always -F arch=b64 -S kill
    # auditctl  -a exit,always -F arch=b32 -S kill
    # auditctl  -a exit,always -F arch=b64 -S tkill
    # auditctl  -a exit,always -F arch=b32 -S tkill
    # auditctl  -a exit,always -F arch=b64 -S tgkill
    # auditctl  -a exit,always -F arch=b32 -S tgkill

Now try to kill a process with:

 # killall -9 <procname>

This produces an output in /var/log/audit/audit.log like the following:

    type=SYSCALL msg=audit(1342619362.183:8): arch=40000003 syscall=37 success=yes exit=0 a0=f5d a1=9 a2=f5d a3=0 items=0 pid=3936 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 comm=”killall” exe=”/usr/bin/killall”

Where,  A list of the available syscalls and their numbers on RHEL4 are contained respectively in:

    /usr/src/kernels/<version>/include/asm-i386/unistd.h for i386 systems
    /usr/src/kernels/<version>/include/asm-x86_64/unistd.h for x86_64 systems

The above files are included into the kernel-devel rpm package.

Troubleshooting  Common Auditd Issues

Issue 1 :  Error :  Audit: backlog limit exceeded

The audit subsystem in kernel was reporting backlog exceeded errors because the auditd daemon was unable to write the audit data to a file system and as a result the incoming stream of audit data overflowed.

Case 1: If you are using a ext4 filesystem, it might have frozen. In this case we can unfreeze the ext4 mount point with

# fsfreeze -u <mount point>

we can temporarily disable auditd to prevent the message while fixing the filesystem issue

Case 2:  If the failure flag in audit.rules set to 2, system will kernel panic incase of auditd issues with below error.

Kernel panic – not syncing: audit: backlog limit exceeded

Failure flag setting in /etc/audit.rules file (RHEL-4) or /etc/audit/audit.rules file (RHEL-5).

    # Failure of auditd causes a kernel panic
    -f 2  <<<—-  The failure flag value “2” is used to panic the kernel when audit backlog limit exceeded on the system.

Change the value to 1 by using the command :

Way1 : # auditctl -f 1
    
    Way 2: modifying the  /etc/audit.rules file (RHEL-4) or /etc/audit/audit.rules file (RHEL-5) , to change “-f” value to “1”.
    
    -f 1

Case 3: If the auditd.conf has configured with the minimum space requirements and the system doesn’t find the required space during system reboot, it will cause a kernel panic with below error:

audit: audit_backlog=16385 > audit_backlog_limit=16384
audit: audit_lost=1 audit_rate_limit=0 audit_backlog_limit=16384
Kernel panic – not syncing: audit: backlog limit exceeded

Following parameters in /etc/audit/auditd.conf also needs to check if enough space to flush audit buffer

space_left
space_left_action
admin_space_left
admin_space_left_action

Issue2 :  RHEL 6 hangs when start/shutdown auditd with the below log message:

BUG: soft lockup – CPU#N stuck for 67s! [<process>:NNNN]

This is known bug which can be avoided by updating the kernel versions:

RHEL6.2.z(EUS): Update the kernel to 2.6.32-220.45.1.el6  or later to fix the issue.
RHEL6.3.z(EUS): Update the kernel to 2.6.32-279.39.1.el6 or later to fix the issue.
RHEL6.4.z(EUS): Update the kernel to 2.6.32-358.28.1.el6 or later to fix the issue.
RHEL6.5: Update the kernel to 2.6.32-431.el6 or later to fix the issue. This fix is already included in RHEL6.5.

As a immediate workaround:

Start the servers with audit disabled. by adding “audit=0” to the kernel command line (in /boot/grub/grub.conf) and try to boot.

Issue 3 :  Auditd stops server to boot, with below error

Stopping auditd :  Error deleting rule (Operation not permitted)

The Reason could be because of /var filesystem space issue  To diagnose and fix the issue

Option 1:  Boot into single user mode as below and delete the old audit.log files under /var/log/audit/ directory if not required and try booting the system.

Procedure to Boot into Single User Mode:

  •         Single user mode can be accessed by appending an “S”, “s”, or “single” to the kernel command line in GRUB. Likewise, a “3” can be used to boot to runlevel 3. To do this, restart the system and when the GRUB splash screen presents itself:
  •         Select/highlight the desired kernel using the up/down arrow keys.
  •         Press the letter a to modify the kernel line.
  •         On the new screen, press the spacebar to add a space, then type the letter s and press Enter.
  •         This will boot the system into single user mode, i.e. the boot process will stop immediately after execution of rc.sysinit and present a root BASH shell.

Option 2:  If not able to boot into single user mode then boot from rescue mode and delete the old audit.log. This will free up the space under /var partition and  do_space_left_action will pass without any issues.

If old audit.log files are required and cannot delete the same, then consider to take a backup of that files to some other partition on the system having enough space or take a backup to external resource. After successful backup, delete the audit.log files from /var/log/audit/ directory to free up the space.

Issue 4:  Auditd keeps on sending below message to system log, in RHEL5

name_count maxed, losing inode data:

As this messages comes from the audit system, they can probably be safely ignored.   This means that if have an audit watch rule it might not be working right for that task at the time.

Issue 5:  Auditd reports below warning while starting, in RHEL 6.4

Warning – entry rules deprecated, changing to exit rule in line xx

‘entry’ keyword is deprecated in RHEL6.4, So we have to replace ‘entry’ to ‘exit’ as it has the same meaning

Ramdev

Ramdev

I have started unixadminschool.com ( aka gurkulindia.com) in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' unixadminschool.com. You can connect me at - https://www.linkedin.com/in/unixadminschool/

2 Responses

  1. Vikrant says:

    Very Impressive article.

  2. krishna says:

    its really impressive ;)

What is in your mind, about this post ? Leave a Reply

Close
  Our next learning article is ready, subscribe it in your email

What is your Learning Goal for Next Six Months ? Talk to us