Child pages
  • Determine Your System's Status
Skip to end of metadata
Go to start of metadata

We last updated this page on:

Overview

cPanel, Inc's Support department detected the following security issues:

  • Compromised RPMs in the OpenSSH binaries. 
  • Compromised libkeyutils directories.

In both cases, trojan horses (trojans) affected files that these directories and binaries contain. We strongly recommend that hosting providers and system administrators use this document to determine the status of their systems.

Note:

The affected OpenSSH Binaries and libkeyutils directories produce similar network traffic.

If you experience any issues while you perform these checks, or you suspect that your server is compromised, contact your hosting provider for assistance.

Note:

To check your system for compromises unrelated to the OpenSSH Binaries and libkeyutils directories, use WHM's Security Advisor interface (Home >> Security Center >> Security Advisor).

Compromised systems

Click the tab below that corresponds to the item you wish to check.

On CentOS and Red Hat® Enterprise Linux® systems, we determined that the following OpenSSH binaries contain the Ebury trojan:

  • The sshd binary.
  • The ssh binary.
  • The ssh-keygen binary.
  • The ssh-askpass binary.

This trojan's code collects authentication credentials for inbound and outbound network connections, as well as the SSH keys that these binaries generate.

Use the following checks to determine whether the trojan compromised your system's OpenSSH binaries.

 


 

Check for malicious processes

To check for malicious processes on your server, run the following command:

netstat -plan | grep atd

This command does not return output on non-compromised systems. On compromised systems, this command returns output that resembles the following example:

unix 2 [ ACC ] STREAM LISTENING 103713 8119/atd @/tmp/dbus-ZP7tFO4xsL

Check the RPM's release numbers.

To check your system's OpenSSH RPM versions, run the following command:

rpm -qa | grep -i ssh

Non-compromised OpenSSH RPMs contain two-digit release numbers, and will resemble the following example:

openssh-6.6.1p1-35.el7_3.x86_64
libssh2-1.4.3-10.el7_2.1.x86_64
openssh-server-6.6.1p1-35.el7_3.x86_64
cpanel-perl-524-Net-OpenSSH-0.74-1.cp1162.noarch
openssh-clients-6.6.1p1-35.el7_3.x86_64

Compromised OpenSSH RPMs contain three-digit release numbers. These three-digit release numbers ensure that updates to not overwrite the versions.

A compromised OpenSSH RPM will resemble the following example:

Note:

In this example, 721 represents the release number.

openssh-4.3p2-721.el5_61.65
openssh-clients-4.3p2-721.el5_61.65
openssh-server-4.3p2-721.el5_61.65

 


 

Check the RPMs' change logs.

To check an RPM version's change log, run the following command:

Note:

In this example, openssh-5.3p1-209.el6 represents the OpenSSH RPM binary that you wish to check.

Non-compromised RPMs contain a signature

rpm -q --changelog openssh-5.3p1-209.el6 | head -10

A compromised RPM's change log contains no entries.

 


 

Check the RPM's signature.

A non-compromised RPM constains a signature, while a compromised RPM does not contain a signature.

To check an RPM's signature, run the following command:

rpm -q -i openssh-clients

A non-compromised RPM's output resembles the following example:

Name        : openssh-clients
Version     : 6.6.1p1
Release     : 35.el7_3
Architecture: x86_64
Install Date: Mon 22 May 2017 09:09:56 AM CDT
Group       : Applications/Internet
Size        : 2299340
License     : BSD
Signature   : RSA/SHA256, Wed 12 Apr 2017 08:31:18 PM CDT, Key ID 24c6a8a7f4a80eb5

A compromised RPM's output resembles the following example:

Name        : openssh-clients
...
Signature   : (none)
...

Note:

Several RPMs reside on a typical installation of cPanel & WHM without signatures and that do not contain complete change logs. With the exception of OpenSSH, an unsigned RPM does not indicate a compromised system.

 


 

Check your system's yum logs.

Open your system's /var/log/yum.log file with a text editor and search for the OpenSSH binaries. The file's contents resemble the following example:

Jun 22 10:33:00 Installed: geronimo-jms-1.1.1-19.el7.noarch
Jun 22 10:33:00 Installed: xml-commons-apis-1.4.01-16.el7.noarch
Jun 22 10:33:00 Installed: xml-commons-resolver-1.2-15.el7.noarch
Jun 22 10:33:00 Installed: xalan-j2-2.7.1-23.el7.noarch
Jun 22 10:33:00 Installed: xerces-j2-2.11.0-17.el7_0.noarch
Jun 22 10:33:01 Installed: avalon-framework-4.3-10.el7.noarch
Jun 22 10:33:01 Installed: tomcat-el-2.2-api-7.0.69-11.el7_3.noarch
Jun 22 10:33:01 Installed: tomcat-lib-7.0.69-11.el7_3.noarch
Jun 22 10:33:02 Installed: tomcat-7.0.69-11.el7_3.noarch
Jun 22 10:33:02 Installed: openssh-6.6.1p1-35.el7_3.x86_64
Jun 22 10:33:02 Installed: libssh2-1.4.3-10.el7_2.1.x86_64
Jun 22 10:33:02 Installed: openssh-server-6.6.1p1-35.el7_3.x86_64
Jun 22 10:33:02 Installed: cpanel-perl-524-Net-OpenSSH-0.74-1.cp1162.noarch
Jun 22 10:33:02 Installed: openssh-clients-6.6.1p1-35.el7_3.x86_64

If your system's /var/log/yum.log file does not contain the OpenSSH RPMs, the server may exist in a compromised state.

 

 

To check your libkeysutils utility for the Ebury trojan, perform the following steps:

Important:

  • We recommend that you perform each of the following steps to fully determine your server's status.
  • Your server's architecture determines the output that you receive when you run the commands in this section.
    • 32-bit systems use lib. 64-bit systems use /lib64/lib64 may also contain a lib directory with 32-bit libraries for compatibility purposes. To find your server's architecture, run the arch command:

      [user@host ~]$ arch

      The output will resemble one of the following examples:

      64-bit server
      x86_64 
      32-bit server
      i386

 


 

Verify the keyutils-libs package.

Changes should not occur on your system without your consent. To check for changes on your system, run either of the following commands:

rpm -V keyutils-libs
netstat -plan | grep atd 

This command does not return output on non-compromised systems. On compromised systems, this command returns the following output:

....L...    /lib64/libkeyutils.so.1 

 


 

Check the files' installation source.

If the command in Step 1 returns output, verify whether an RPM provided the files. To do this, run the following command:

rpm -qf /lib64/libkeyutils.so.1

Note:

In this example, /lib64/libkeyutils.so.1 represents the file to verify.

On non-compromised systems, the output resembles the following example:

keyutils-libs-1.4-5.el6.i686

On compromised systems, the output resembles the following example:

file /lib64/tls/libkeyutils.so.1.5 is not owned by any package

 


 

Verify the file that links to the libkeyutils.so.1 file.

To see which file links to the libkeyutils.so.1 file, run the following command:

ls -l /lib64/libkeyutils.so.1

On non-compromised systems, the output will resemble the following example:

lrwxrwxrwx 1 root root 18 Feb 20 12:15 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.3*

On compromised systems, the output will return one or more of the following files:

libkeyutils.so.1.9
libkeyutils.so.1.3.2
libkeyutils-1.2.so.2

 

 

 


 

Check the strings for all libkeyutils.* libraries for items that relate to networking.

The default libkeyutils.so.1.3 file should not contain the following strings:

connect
socket
inet_ntoa
gethostbyname

To check your system for these strings, run the following command:

strings /lib64/libkeyutils.so.1.3 | egrep 'connect|socket|inet_ntoa|gethostbyname'

This command does not return output on non-compromised systems. On compromised systems, this command returns the following output:

connect
socket
inet_ntoa
gethostbyname

 

 


 

 

Check sshd processes.

On compromised servers, sshd processes use shared memory segments.

To check whether any sshd processes currently use shared memory segments, run the following command:

ipcs -mp

On a non-compromised server, the output will resemble the following example:

------ Shared Memory Creator/Last-op --------
shmid      owner      cpid       lpid

On a compromised server, the output will resemble the following example:

------ Shared Memory Creator/Last-op --------
shmid      owner      cpid       lpid
1769472    root       1975       1975
2129921    root       2931       2940
1736706    root       1965       1965
2162691    root       2931       2940
2195460    root       2931       2940
2228229    postgres   4011       6813

If any programs use shared memory segments, run the ps command, for example:

ps aux | grep 1975

This command checks whether any of the items in the cpid and lpid columns correspond to the sshd processes. The output will resemble the following example:

root     1975  0.0  0.0  64080  1172 ?        Ss   Feb17
0:00 /usr/sbin/sshd

 


 

Monitor outbound UDP port 53 traffic.

We recommend that you use a network data capture tool, such as the tcpdump utility, to monitor outbound User Datagram Protocol (UDP) on port 53. This utility returns DNS traffic between your server and the local resolvers that reside in your /etc/resolv.conf file.

To monitor outbound UDP on port 53, run the following command:

[root@host ~]# tcpdump -Annvvs 1500 -i any udp and dst port 53

On a non-compromised server, the output will resemble the following example:

22:18:22.038264 ARP, Request who-has 10.147.0.62 tell 10.147.0.64, length 46
22:18:22.244856 ARP, Request who-has 10.147.0.64 tell 10.147.0.61, length 46
22:18:22.245149 ARP, Request who-has 10.147.0.61 tell 10.147.0.64, length 46
22:18:22.888851 b4:99:ba:02:18:66 > Broadcast, ethertype Unknown    (0xcafe), length 90:
        0x0000:  0500 0100 0900 0000 0100 ffff 0c00 0002  ..... ...........
        0x0010:  4c00 0000 0000 0000 8300 0080 0000 0000  L...............

On a compromised server, the output will resemble the following example:

07:54:48.233159 IP (tos 0x0, ttl  64, id 31281, offset 0, flags [DF],
proto: UDP (17), length: 91) 1.2.3.4.43089 > 72.156.139.154.53:
[bad udp cksum d7a9!]  4619+ A?
6196g8f43a4facd3561de4gec736fb.5.5.5.5. (63)

Important:

The example of the packet above shows that the cPanel server at 1.2.3.4 sends a UDP packet on port 53 to the host at 72.156.139.154. You should notice the following false query from the example above:

6196g8f43a4facd3561de4gec736fb.5.5.5.5

 The string 6196g8f43a4facd3561de4gec736fb represents an encoded password, and 5.5.5.5 represents the IP address that you used to log in to the server. This output indicates that the malicious host at 72.156.139.154 captured the following information:

  • The IP address that you used to log in to the server.
  • The login password.

 


 

Verify that each library that sshd links against belongs to a known package.

To check the libraries the sshd daemon links against, run the following command

ldd /usr/sbin/sshd

The output will resemble the following example:

	libnspr4.so => /lib64/libnspr4.so (0x00007fcb04a30000)
	libfreebl3.so => /lib64/libfreebl3.so (0x00007fcb0482d000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fcb0461d000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fcb04419000)
	libattr.so.1 => /lib64/libattr.so.1 (0x00007fcb04213000)
	libelf.so.1 => /lib64/libelf.so.1 (0x00007fcb03ffb000)
	libbz2.so.1 => /lib64/libbz2.so.1 (0x00007fcb03dea000)

To verify that these libraries belong to a valid package, run the following command:

rpm -qf /lib64/libfipscheck.so.1

The output will resemble the following example:

fipscheck-lib-1.2.0-7.el6.x86_64


Additional documentation