Compromised Linux Server Investigation

Project Name: Compromised Linux Server Investigation
Description: Compromised Linux Server Investigation in India is a very precise level of sophisticated job. In this blog we will try yo understand step by step guide to Linux server forensics investigation & incident response. In term of Compromised Linux Server Investigation we will cover website hacking investigation, web application hacking investigation & many more aspects of similar issues. This blog will help all computer forensics investigator, system administrator to analyze server hacking or compromised incident.

Author: Rohit D Sadgune

Summary of Contents

• Disk Acquisition
• Forensics Analysis of Linux System
  • View IP connection count
  • List Open Files
  • Investigate Process Activity
  • Analysis Binary Files
  • Check for unauthorized services.
  • Look everywhere for unusual or hidden files.
  • Log Audit

UNIX and Linux systems are vulnerable to various types of web application level & network attacks. Depending on the assessment of the people administering these systems they can either be extremely easy or extremely difficult to gain unauthorized access to.

Disk Acquisition of Linux system

The linux acqisition can be done in

• Three formats

1. Raw format
2. Proprietary formats
3. Advanced Forensics Format (AFF)

There are different types of acquisitions

• Static acquisitions and live acquisitions
  • Four methods
    1. Bit-stream disk-to-image file
    2. Bit-stream disk-to-disk
    3. Logical disk-to-disk or disk-to-disk data
    4. Sparse data copy of a file or folder

In live acquisition you can use following command

# dd if=/dev/sda of=/path/to/external/medium/file.dd

Or

You can use forensics linux boot CD/DVD e.g DEFT (Digital Evidance Forensics Toolkit), CAIN,

Kali Linux, SPADA (System Preview and Data Acquisition).

In Static acquisition invertigator can use ProDiscover Basic, ProDiscover Forensics or ProDiscover Incident Response, Encase, FTK Imager

• Forensics Image with ProDiscover

• Connecting the suspect’s drive to your system on which prodiscover is installed.
  1. Document the chain of evidence for the suspect drive
  2. Remove the suspect drive drive from the computer & keep in safe antistatic bags for further use.
  3. Connect the suspect drive along with write blocker.

4. Verify Data Acquisitions most important aspect of computer forensics Is has verification, widely used validation techniques are CRC-32, MD5 and SHA-1 to SHA-512

RAID Acquisition

Redundant array of independent disks (RAID)

• • Server configuration which involving two or more physical or logical disks
  • It was Originally developed as a data-redundancy measure

To check RAID is installed or not please use following

commands

Check the array configuration

#cat /proc/mdstat

Forensics Acquiring RAID Disks

• Major factors are

1. How much physical data storage is needed?
2. What type of RAID is used?
3. Do you have the right acquisition tool?
4. Wheather tool has enough capabilities of forensically copied RAID image?
5. Can the software read split data saves of each RAID as speprate disk?

RAID forensics acquisition

Step 1: Acquire and investigate complete RAID volume as a single volume

1. Needs big target device.
2. Use device drivers such as those contained on Linux distributions

Step 2: Acquire individual disks and look for hidden data in possible areas that the RAID volume

did not use.

• Keyword searches can also be performed on the individual disks.

Forensics Analysis of Linux System or Server

1. View IP connection count

#netstat –anp

Number of open connections per ip

        #netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n

        #netstat -an | grep 80 | wc -l

Instead of 80 you can put any respective service port to filter out.

2. List Open Files

lsof stands for List Open Files. It is a command line program which is widely used to list the information about the files that are opened by various processes. In linux or unix file system everything is a file, ( pipes, sockets, directories, devices, etc.). So by using lsof, you can invetigate the information about any opened files.

 # lsof

List processes which opened a specific file

 # lsof /var/log/syslog

List opened files under a directory

 # lsof +D /var/log/

List opened files based on process names starting with

# lsof -c ssh -c init

 List files opened by a specific user

# lsof -u rohit

List all open files by a specific process

# lsof -p 3389

List all network connections

# lsof –i

List all TCP or UDP connections

# lsof -i tcp ; lsof -i udp;

List all Network File System ( NFS ) files

# lsof -N -u rohit -a

3. Investigate Process Activity

To investigate process activity in Linux there are multiple commands

#ps -aux

#ps -ef -f

Display process by user

#ps -f -u www-data

Show process by name or process id

#ps -C apache2

Sort process by cpu or memory usage

#ps aux –sort=-pcpu,+pmem

#ps aux –sort=-pcpu | head -5

Display process hierarchy in a tree style

#ps -f –forest -C apache2

Display threads of a process

# ps -p 3150 -L

4. Analysis Binary Files

Attacker or Intruders may change executable programs on Linux Server or UNIX Server such as login, su, telnet, netstat, ifconfilg, ls, find du, df, libc, sync, any binaries referenced in /etc/inetd.conf, and other critical network and system programs and shared object libraries. Differentiate the current versions on your server with known good copies, such as those from your initial installation media. Trojan horse programs is having capabilities to produce the same standard checksum and timestamp as the legitimate version. The use of prodiscover or EnCase, Tripwire, and other cryptographic checksum tools is sufficient to detect these Trojan horse programs.

5. Check for unauthorized services.

Investigate /etc/inetd.conf or /etc/xinetd.conf for unauthorized additions or changes.

In forensics search for entries that execute a shell program (for example, /bin/sh or /bin/csh) and check all others programs that are specified & available in /etc/inetd.conf to forensically verify that they are in original format and haven’t been replaced by Trojan horse programs. Also check for legitimate services that you have commented out in your /etc/inetd.conf.

6. Look everywhere for unusual or hidden files.

# find / -name “.. “ –print –xdev

# find / -name “.*’ –print –xdev | cat –v

You can also use prodisover incident response kit for Linux unseen file investigation

• Load a forensics image or drive in prodisover incident response

• Clik on IR option

• Now Click on Useen Files

7. Log Audit

 Part -1

1. #cat /var/log/messages – Contains global system related messages, which includes the messages that are processed during system or server startup. There are so many other things that are logged in /var/log/messages includes different mail, scheduled cron, all daemon, kernal related log, authentication log, etc.
2. #cat /var/log/dmesg – This file show information related kernal & kernal related module.     When the system boots up, it prints significant information on the screen that displays information about the hardware devices that the kernel detects during boot process. These significant information are available in kernel ring buffer and whenever the new bulletin comes the old message gets overwritten. You can also view the content of this file using the dmesg command.
3. #cat /var/log/auth.log – This file contains system authorization messages, including user logins activity and authentication machinsm that were used.
4. #cat /var/log/boot.log – This file is useful forensics analysis as it stores information that are logged when the system boots
5. #cat /var/log/daemon.log – This file is useful for forensics analysis as it stores information logged by the various background daemons that r’uns on the system
6. #cat /var/log/dpkg.log – – This file is useful for forensics invetigation as it stores the information that are logged when a package is installed or removed using dpkg command
7. #cat /var/log/kern.log – This file is useful for forensics analysis as it stores the information logged by the kernel. Helpful for you to troubleshoot a custom-built kernel.
8. #cat /var/log/lastlog – Displays the recent login information for all the users.
9. #cat /var/log/maillog /var/log/mail.log – This file is useful for forensics analysis as it stores the information from the mail server that is running on the respective system.
10. #cat /var/log/user.log – Contains information about all user level logs
11. #cat /var/log/Xorg.x.log – Log messages from the X
12. #cat /var/log/btmp – This file is very much useful for digital forensics aspects as it stores information about failed login attemps.
13. #cat /var/log/cups – All printer and printing related log messages
14. #cat /var/log/cron – Whenever cron daemon (or anacron) it generates cron job, it logs the messages about the cron job in this file
15. #cat /var/log/secure – This file is very much useful for digital forensics aspects as this file holds the information related to authentication and authorization privileges.
16. #cat /var/log/wtmp or /var/log/utmp – This file stores login record’s os linux server.
17. #cat /var/log/faillog – This file is very much useful for digital forensics as this files stores crucial information of user failed login attemps.

Part -2

In Addition to the above log files, /var/log directory may also contain the following sub-directories depending on the application that is running on your system.

1. #cat /var/log/httpd/ (or) /var/log/apache2 – This file is very useful for compromised Linux server investigation as it stores the apache web server access_log and error_log
2. #cat /var/log/lighttpd/ – Contains light HTTPD access_log and error_log
3. #cat /var/log/prelink/ – prelink program modifies shared libraries and linked binaries to speed up the startup process. /var/log/prelink/prelink.log contains the information about the .so file that was modified by the prelink.
4. #cat /var/log/audit/ – – This file is very useful for compromised Linux server investigation as it having logs information stored by the Linux audit daemon (auditd).
5. #cat /var/log/setroubleshoot/ – SELinux uses setroubleshootd (SE Trouble Shoot Daemon) to notify about issues in the security context of files, and logs those information in this log file.
6. #cat /var/log/samba/ – – This file is very useful for compromised Linux server investigation as it having log information stored by samba, which is used to connect Windows to Linux.
7. #cat /var/log/sssd/ – Use by system security services daemon that manage access to remote directories and authentication mechanisms.