Child pages
  • Determine Your System's Status
Skip to end of metadata
Go to start of metadata

Note:

The technical information in this document was last updated on February 24, 2014.

Overview

Regretfully, cPanel's support department has detected two types of security issues:

  • Compromised RPMs in the OpenSSH binaries. 
  • Compromised libkeyutils

In both cases, files that these directories and binaries contain were "trojaned." We highly encourage system administrators to read this document to determine the status of their systems. If you experience any issues while you perform these checks, contact Tech Support for assistance.

 

Compromised RPMs

We have determined that some systems were compromised with "trojaned" OpenSSH binaries. The OpenSSH binaries appears to contain the Ebury trojan

Note:

The network traffic that the "trojaned" OpenSSH binaries and the libkeyutils create are similar.

In regards to CentOS and Red Hat® systems, we have determined that the sshdsshssh-keygen, and ssh-askpassbinaries all appear to contain trojan code. This code is used to collect authentication credentials for both inbound and outbound connections. We believe that the SSH keys that these binaries generate were also captured.

OpenSSH RPMs Trojans Characteristics

The most distinctive features of the "trojaned" OpenSSH RPMs are:

The RPMs contain 3-digit release numbers to ensure that updates do not overwrite them

The following example indicates a desired output:

[user@host ~]$ rpm -qa | grep -i ssh
[...]
openssh-clients-5.3p1-81.el6_3.x86_64
openssh-server-5.3p1-81.el6_3.x86_64
openssh-5.3p1-81.el6_3.x86_64

The following three examples indicate a compromised system:

  • In the following example, 730 is the release number:

    openssh-askpass-4.3p2-730.el5_67.3
  • In the following example, 721 is the release number:

    user@host ~]$ rpm -qa | grep -i ssh
    [...]
    openssh-4.3p2-721.el5_61.65
    openssh-clients-4.3p2-721.el5_61.65
    openssh-server-4.3p2-721.el5_61.65
  • In the following example, 209 is the release number:

    openssh-5.3p1-209.el6_10.41.x86_64
    openssh-clients-5.3p1-209.el6_10.41.x86_64
    openssh-server-5.3p1-209.el6_10.41.x86_64

The change logs for RPMs do not contain any entries for the installed release

jd@goat:~/$ rpm -q --changelog openssh-5.3p1-209.el6 | head -10
* Thu Aug 12 2010 Jan F. Chadima <jchadima@redhat.com> - 5.5p1-20
- set correct socket name length in abstract socket (#621691)

The RPMs may not be signed

The following output shows an empty signature, which may indicate that the RPM is not signed:

jd@goat:~/$ rpm -q -i openssh-clients
Name        : openssh-clients
...
Signature   : (none)
...

Warning:

New variations of this trojan will appear to be signed, and therefore this check alone is not sufficient to determine whether a system is compromised. 

The system YUM logs have no record of the RPMs' installation

Note:

On a typical installation of cPanel & WHM, there are several RPMs that are not signed and do not contain complete change logs. With the exception of OpenSSH, an unsigned RPM does not indicate a compromised system.

How to Check Your System for the trojan in libkeyutils

Important:

Before you run these commands, note that you will receive different information based on the architecture of your server.

32-bit systems use lib. 64-bit systems use /lib64/lib64 may also have a lib directory that contains 32-bit libraries for compatibility purposes. If you are not sure of your server's architecture, you can use the arch command:

[user@host ~]$ arch

An output of x86_64 means you use a 64-bit server. An output of i386i686, or something similar means you use a 32-bit server.

 

We recommend that you run each of the following commands to fully determine the status of your server.

Important:

 Even if you receive the desired output, we recommend that you continue to run the other commands.

Command 1

Verify the keyutils-libs package.

Ideally, a change did not take place in your system without your consent. If a change did not take place, you will receive a blank output.

The following output indicates that your system was not compromised:

[user@host ~]$ rpm -V keyutils-libs
[user@host ~]$ 

However, the following output indicates something changed. If you receive this output, you should investigate further:

 [user@host ~]$ rpm -V keyutils-libs
....L...    /lib64/libkeyutils.so.1 

Command 2

Verify which file is linked to the libkeyutils.so.1 file.

Run the following command to see which file is linked to the libkeyutils.so.1 file:

[user@host]$ ls -l /lib64/libkeyutils.so.1

The following example shows the desired output:

lrwxrwxrwx 1 root root 18 Feb 20 12:15 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.3*

If the /lib64/libkeyutils.so.1 file points to a file with one of the following names, then the server may be compromised:

  • libkeyutils.so.1.9
  • libkeyutils.so.1.3.2
  • libkeyutils-1.2.so.2

Command 3

Check the strings for all libkeyutils.* libraries for items that relate to networking.

The default libkeyutils.so.1.3 file should not contain the following strings:

  • connect
  • socket
  • inet_ntoa
  • gethostbyname

The following output indicates that you server was not compromised:

[user@host ~]$ strings /lib64/libkeyutils.so.1.3 | egrep 'connect|socket|inet_ntoa|gethostbyname'
[user@host ~]$

The following output indicates that your server may be compromised:

[user@host ~]$ strings /lib64/libkeyutils.so.1.9 | egrep 'connect|socket|inet_ntoa|gethostbyname'
gethostbyname
socket
inet_ntoa
connect

Command 4

Check whether any sshd processes currently use shared memory segments.

Run the following command:

[root@host ~]# ipcs -mp

The following output indicates that there are no programs that use shared memory segments:

[root@host ~]# ipcs -mp

------ Shared Memory Creator/Last-op --------
shmid      owner      cpid       lpid   

The following output indicates that there are programs that use shared memory segments:

------ Shared Memory Creator/Last-op --------
shmid      owner      cpid       lpid
1769472    root       1975       1975
2129921    root       2931       2940
1736706    root       1965       1965
2162691    root       2931       2940
2195460    root       2931       2940
2228229    postgres   4011       6813

If there are programs that use shared memory segments, you can use the ps command to check whether any of the items in thecpid and lpid columns correspond to the sshd processes:

[user@host ~]$ ps aux | grep 1975
root     1975  0.0  0.0  64080  1172 ?        Ss   Feb17
0:00 /usr/sbin/sshd

Note:

While is it not typical of sshd to use shared memory segments, you should investigate further to see if this is an indication of a compromise.

Command 5

You should use a network sniffer, such as tcpdump, to monitor outbound UDP port 53 traffic while you log in to the shell.

Run the following command:

[root@host ~]# tcpdump -Annvvs 1500 -i any udp and dst port 53

It is normal to see DNS traffic between your server and your local resolvers from the /etc/resolv.conf file (and possibly to other servers). However, it is not normal to see traffic that contains odd, alphanumeric strings, followed by the IP address from which you logged in.

The following output indicates a compromise. 

Note:

The asterisks indicate important items to notice.

07:54:48.233159 IP (tos 0x0, ttl  64, id 31281, offset 0, flags [DF],
proto: UDP (17), length: 91) **1.2.3.4.43089** > **72.156.139.154.53**:
[bad udp cksum d7a9!]  4619+ A?
**6196g8f43a4facd3561de4gec736fb.5.5.5.5**. (63)

 

The example of the packet above shows that the cPanel server at 1.2.3.4 sends a UDP packet on port 53 to the host at 72.156.139.154. You should notice the following false query from the example above:

6196g8f43a4facd3561de4gec736fb.5.5.5.5

The string 6196g8f43a4facd3561de4gec736fb is an encoded password, and 5.5.5.5 is the IP address that was used to log in to the cPanel server. This indicates that the malicious host at 72.156.139.154 has learned the following information:

  • The IP address that was used to log in to the server, and
  • The password that was used to log in.

Note:

You may see two of these packets in quick succession: one packet contains the username that was used to log in, while the other packet contains the password. The content of both packets will be extremely similar.

Command 6

Verify that each library that sshd is linked against belongs to a known package.

You can use the ldd command to see which libraries sshd links against:

[user@host ~]$ ldd /usr/sbin/sshd
   linux-vdso.so.1 =>  (0x00007fffb5dfd000)
   libfipscheck.so.1 => /lib64/libfipscheck.so.1 (0x00002aaabc550000)
   libwrap.so.0 => /lib64/libwrap.so.0 (0x00002aaabc753000)
[...]

 

The items to the right of the arrow are the libraries to which sshd links. In this example, the libraries are:

  • libfipscheck.so.1
  • libwrap.so.0

To verify that these libraries belong to a valid package, run the following command:

[user@host ~]$ rpm -qf /lib64/libfipscheck.so.1
fipscheck-lib-1.2.0-7.el6.x86_64

[user@host ~]$ rpm -qf /lib64/libwrap.so.0
tcp_wrappers-libs-7.6-57.el6.x86_64

Additional documentation