DISK IO Impact of Linux Audit Service

Symptom:

    We record that the local DISK IO is high, there are 2 majority consumers.

• One is index operation.On Jira APPS, the apps update, delete index files quite intensively . Same apply to ELK apps.   They both use same lucent index technology. 
• The other is Linux Audit Service

Use iostat 2 10 to find top used volumes
Use lsblk to find out which volume is related to APPS or Audit

You can see massive writes on /var/log/audit/audit.log , and it rotates very quickly
Audit entries like
type=CWD msg=audit(1529554416.727:219411617):  cwd="/data/atlassian/product/jira/bin"
type=PATH msg=audit(1529554416.727:219411617): item=0 name="/u01/app/atlassian/admin/jira/caches/indexes/changes/" inode=11288602 dev=ca:71 mode=040755 ouid=1000 ogid=1001 rdev=00:00
type=PATH msg=audit(1529554416.727:219411617): item=1 name="/data/atlassian/admin/jira/caches/indexes/changes/_13469.fdt" inode=11288601 dev=ca:71 mode=0100644 ouid=1000 ogid=1001 rdev=00:00
type=SYSCALL msg=audit(1529554416.727:219411618): arch=c000003e syscall=87 success=yes exit=0 a0=7fbbef337d60 a1=7f9e9c009ec8 a2=7fbbef337d60 a3=61632f6172696a2f items=2 ppid=1 pid=7146 auid=501742 uid=1000 gid=1001 euid=1000 suid=1000 fsuid=1000 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=38734 comm="java" exe="/u01/app/atlassian/product/jira/jdk/bin/java" key="delete

Jira index operation is expected. We can reduce IO via reduce unnecessary audit entries.

Solution:

• Remove Linux Audit Configuration Immutable . comment "-e2"  in /etc/audit/audit.rules
• Reboot OS
• Remove SYSCALL CWD and PATH from audit. Add below into audit.rules

-a always,exclude -F msgtype=SYSCALL

-a always,exclude -F msgtype=CWD

-a always,exclude -F msgtype=PATH

• Bounce the audit service
• Unfortunately audit service only can exclude via msgtype , it can't exclude based on exe file or directory or key.