Detecting Malicious Actors By Observing Commands in Shell History

Among the myriad techniques and tools at the disposal of cybersecurity experts, one subtle yet powerful method often goes unnoticed: the analysis of shell history to detect malicious actors.

At the heart of Unix and Linux systems, shell serves as a fundamental command-line interface that serves as a user’s gateway to the operating system. It allows users to execute a wide range of commands, from basic file manipulation to complex program execution and system administration. The CLI is probably the most commonly used way to interact with Unix based VMs, which means most of the actions on a VM could be executed through the CLI.

In these Unix-based systems, the extensive utility of shell commands, crucial for system administration, also renders them susceptible to exploitation. Originally designed for legitimate purposes, these commands can be repurposed for various malicious activities, including attacks, data exfiltration, and unauthorized system control. Consequently, the CLI emerges as a vital instrument in the detection and investigation of unauthorized activities within systems.

In this blog, we will discuss various shell commands, shedding light on the patterns and anomalies that can signal unauthorized access or malicious intent. This is not just about understanding what commands were executed; it’s about discerning the story they tell. 

Common Suspicious Bash Commands

Let’s delve into common Bash commands that, while standard in Unix/Linux environments, might signal suspicious or malicious activities. Recognizing these commands is crucial for identifying potential security breaches, as attackers frequently use them for unauthorized system access, data manipulation, and other nefarious purposes.

1. Reconnaissance commands
   
   Reconnaissance commands are any commands that reveal information aiding an attacker in their assault – such as their location, accessed user privileges, potential vulnerabilities, and more. For instance, an attacker might search for the system’s version, users with sudo permissions, passwd content, cron jobs, the current user’s profile, or the classic ‘whoami’. In cloud environments, suspicious reconnaissance commands could include accessing metadata services (like 169.254.169.254 in AWS) or credential files.  Identifying reconnaissance attempts, as with other types of suspicious commands, are problematic as the commands used can be extremely common.
   
   Examples for reconnaissance commands:
   • whoami
   • uname
   • find 
   • grep
   • reading sensitive files (passwd, sudoers, shadow)
   • cron -l
   • get-caller-identity
   • describe-instances
   • ls (both s3 and regular)

2. Privilege escalation commands
   
   These commands relate to elevating a user’s permissions, ranging from basic ones like sudo and chmod, to manipulating passwd or shadow files, and in cloud environments, updating credentials or assuming roles. Attackers often escalate privileges to gain a strong foothold and access as much of the system as possible.
   
   Examples for privilege escalation commands:
   • chmod
   • sudo
   • setfacl
   • Writing into relevant files (sudoers, passwd, shadow)
   • aws iam add-user-to-group
   • aws iam attach-user-policy

3. Data theft
   
   Often, an attacker who is infiltrating a system aims to collect data, be it sensitive personal information (ID numbers, credit card details), industrial secrets, credentials, or secret keys. These can further compromise other assets or maintain control over the current system. For example, with the CLI, an attacker can scan for specific file names or types, or directly access files containing desired information. They can use commands like find and ls to search for these files.
   
   Examples for possible data theft commands:
   • grep/find/cp for sensitive file names, such as password, credentials, secret, token, personal information, etc

4. Suspicious Networking
   
   Attackers often have prepared tools for spreading viruses, ransomware, etc., or they may want to maintain a remote command server connection. They perform these actions using CLI commands, which can be detected. Actions might involve using commands like curl or wget to download files, or netcat for shell communication with a C&C server, or even creation of proxies. Monitoring for payload downloads or the execution of downloaded files is also crucial.
   
   Examples for possible commands that may indicate suspicious networking:
   • Netcat
   • nmap
   • http_proxy=
   • -i >& /dev/tcp|nc -l*
   • wget
   • curl
   • tftp

5. Trail erasing 
   
   Attackers often try their best to cover their tracks to avoid detection by defensive tools or investigators. This can involve deleting logs, disabling antivirus agents, or stopping logging processes. The commands for these actions are typically uncommon, so their detection on our system is a strong indicator of malicious activity.
   
   Examples for commands indicating trail erasing:
   • HISTFILESIZE=0
   • HISTSIZE=0
   • >.bash_history/.zsh_history/auth.log
   • service auditd stop
   • history -c
   • systemctl stop auditd

Example: Infiltrating a System Using Shell Commands

Once a malicious actor infiltrates a system, perhaps through brute-force or stolen credentials, their immediate goal is to assess their position: Who is the user I’ve accessed, and what can this user do? What is the nature of this asset?

To gather this intelligence, we would use commands that provide maximum information about the current scenario:

Question number one – whoami?:

Now let’s see if there is any cloud account connectivity:

Ok, so no cloud identity. But what is this system?

hmm, interesting, we have a kali machine.

We want to try and get to sudo, so we check who has sudo – 

What other users are on this system?

We have a user named kali! Sounds promising. But how can we log into it?

Maybe we can look for .pem files on the system.

Ok! Let’s try to use it to connect to the machine using the .pem we found.

And we’re in! 

Next, we’ll want to make sure we’re not documenting any of our actions from now on, using histfile commands.

(HISTFILESIZE is how many commands can be stored in the . bash_history file. HISTSIZE is the number of cached commands)

We should mention here that if at any stage of the attack the attacker would use these commands (or other commands meant to erase the history log files) it would be almost impossible to detect the attack without using elaborate forensics tools. 

Just making sure it works – 

Awesome. Sometimes, we would want to completely empty the history file, in some operating systems the shell file is named differently. We can empty it using the echo command, like this –  

And now, the fun begins.

Our main goal here is to install a cryptominer, so the first thing we would want to do is to download the designated tool – 

And run it – 

In order to make sure it will keep running in case the process stops, we can create a cronjob in charge of running the miner once every while. 

But what if the user changes the password and we are kicked out and can’t get back in?

So let’s add a user for our own future use – 

And one last thing just to make sure nothing was recorded – 

And that’s it!

Keep in mind that there are many other ways to attack using shell commands. This was just a small example of how powerful the shell can be, and how an attacker can use it for their advantage.

Challenges in Shell History Analysis

Analyzing Bash history for security purposes presents unique challenges with a primary obstacle being the potential lack of comprehensive logs; skilled attackers frequently cover their tracks, deleting or altering history files to erase evidence of their presence.

Furthermore, even when scripts are used within the shell, the individual commands executed by these scripts don’t typically get recorded in the history file, making it difficult to reconstruct the sequence of actions from the history log alone. 

Finally, it’s difficult to distinguish between malicious use of these commands, and  legitimate use – for example, an attacker might use chmod to elevate permissions on a recently downloaded payload and run it, and it can also be David from IT who downloaded a package and is trying to run it.

We do not recommend relying solely on shell history to detect malicious factors, as it is not a thorough solution and that can be easily altered. With correct analyzing and detection we can use shell history to be alerted to attackers who slipped up and left a clue for us to find.

Case Studies

In our Honeypotting in the Cloud report we saw many examples of the commands we mentioned above.  

Over 330k reconnaissance attempts, with commands like ‘which ls’, ‘ifconfig’, ‘crontab -l’, ‘whoami’, ‘uname’,  ‘cat /proc/cpuinfo’ with grep of specific lines, and more. 

Over 2300 suspicious network commands, such as  downloading of packages with apt, wget, curl and others

More than 30k commands related to privilege escalation, mostly simple commands such as chmod and sudo

More than 70k commands indicating attempts at persistence, like useradd, writing users into passwd and authorized_keys files.

Additionally, there were many obfuscated commands, creation of files, using known vulnerabilities, search of existing miners, killing and deleting important processes and files (kill -r %%1, rm -rf .file .z .x, rm .s, and more)

These results make sense – most attackers will first try to understand where they are, what type of user they have access to (root, simple user, etc) and what vulnerabilities the asset is exposed to,  before trying to elevate their permissions and create backdoors for themselves, and only then will they start the attacks themselves. 

Conclusion

In our exploration of detecting adversaries through shell history in Unix and Linux systems, we have uncovered the dual nature of the command line interface – a powerful tool for both system administration and potential exploitation. Our journey through various suspicious command categories, from reconnaissance to privilege escalation and beyond, highlights the CLI’s critical role in cybersecurity.

The insights gained from case studies and real-world scenarios in our honeypot study emphasizes the prevalence and sophistication of these tactics. However, the challenges in analyzing shell history, such as incomplete logs and the difficulty in differentiating between legitimate and malicious activities, underline the need for comprehensive monitoring strategies. This exploration affirms that while shell history analysis is a valuable tool in the cybersecurity arsenal, it must be part of a broader, multi-layered approach to effectively detect and identify adversaries.

Wherever possible, infrastructure should be managed as cattle and not pets using Infrastructure as Code.  If that was the case then any commands run in an interactive shell would automatically be anomalous and easy to investigate as there would be so few instances.  This would also allow security scanning for misconfigurations to occur long before they enter the production estate through our Shift Left security.

How the Orca Platform can help

The Orca Cloud Security Platform notifies when suspicious shell commands are detected and provides the necessary context for investigations and possible remediation. Orca leverages a specifically curated scoring system to differentiate between malicious and legitimate behavior. The scores vary from 1-10, with 10 indicating the highest level of certainty that malicious activity has occurred. Orca also stores data for forensic analysis if an actual breach occurs.

Let’s take a look at an Orca alert – 

Orca detects suspicious activity in shell history

And the findings – what commands were identified as suspicious?

The Orca alert specifies which commands it detected

It is also interesting to look at the score breakdown of the alert – 

Breakdown of the different data points that the alert score considers

The commands shown here are the same commands we used in the example above. We divided our commands into three categories – depending on how certain we are, based on the command, that they indicate malicious activity. So a simple chmod would not raise our suspicion that much – but trying to read sudoers or looking for .pem files, would. This division into categories allows us to cover most scenarios, while keeping false positives to a minimum.

Each category has its own weight, which is used to calculate the overall alert score. The more commands detected in the category, the higher the percentage will be. So in our example there were a lot of medium certainty commands, which brought the percentage to 100, meaning the entire weight of the category was added to the final score.

If you’d like to learn more about the Orca Platform and how we can detect malicious shell commands, schedule a personalized demo with one of our experts.