Wednesday, November 14, 2018

Cloud Security - Prevent DDOS attacks through Memcached

Recently, I provisioned a new server and went home for 2 days did not login, once I was back, I could see around 2.5 lakh failed login attempts from various IPs from China. So first step was change the SSH default port and disable root login to the server. Chinese attempts did not end there, the next was attempt to use my memcached vanilla setup, which was secured as explained below, hope this helps somebody.

Memcached is an open-source distributed memory object caching system which is generic in nature but often used for speeding up dynamic web applications. In the default configuration, memcached by default listens on ports 11211/tcp and 11211/udp. 

Memcached servers openly accessible from anywhere on the Internet via UDP are abused for DDoS reflection attacks against third parties on a regular basis. This way, extremely high amplification factors can be achieved which poses a serious security threat.

 If a memcached server is openly accessible from the Internet via TCP or UDP and no SASL authentification has been configured, anyone who can connect to the server has unrestricted access to the data stored with it. This allows attackers to modify or delete any dataor potentially steal sensitive information like login credentials for web applications or customer data from online shops.

Now how do we enable SASL 

1) Verify default Memcached configuration

[root@OMeghaCloud ~(omegha_eu)]# cat /etc/sysconfig/memcached
PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS=""
[root@OMeghaCloud ~(omegha_eu)]# 

2) Check if UDP protocol is enabled (We have to close UDP protocol access)

[root@OMeghaCloud ~(omegha_eu)]# sudo netstat -plunt|grep memcache
tcp        0      0 0.0.0.0:11211           0.0.0.0:*               LISTEN      1982/memcached      
tcp6       0      0 :::11211                :::*                    LISTEN      1982/memcached      
udp        0      0 0.0.0.0:11211           0.0.0.0:*                           1982/memcached      
udp6       0      0 :::11211                :::*                                1982/memcached      
[root@OMeghaCloud ~(omegha_eu)]# 

3) Change it as show below to restrict Memcached access only through tcp and only to localhost. -S and -vv parameters, -vv provides verbose output to /var/log/memcached, which will help us as we debug. -S enables SASL. -l will restrict listening to loopback IP and -U 0 will switch off UDP protocol.

[root@OMeghaCloud ~(omegha_eu)]# cat /etc/sysconfig/memcached
PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS="-l 127.0.0.1 -U 0 -S -vv"
[root@OMeghaCloud ~(omegha_eu)]# 

4) Restart the Memcached service and verify if UDP listening is disabled.

[root@OMeghaCloud ~(omegha_eu)]# systemctl restart memcached
[root@OMeghaCloud ~(omegha_eu)]# sudo netstat -plunt|grep memcache
tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN      29056/memcached     
[root@OMeghaCloud ~(omegha_eu)]# 

5) Verify if the Memcached service has enabled SASL this time

[root@OMeghaCloud ~(omegha_eu)]#journalctl -u memcached
-- Logs begin at Wed 2018-11-14 05:49:06 CET, end at Wed 2018-11-14 18:02:02 CET. --
Nov 14 05:49:22 OMeghaCloud systemd[1]: Started Memcached.
Nov 14 05:49:22 OMeghaCloud systemd[1]: Starting Memcached...
Nov 14 18:01:54 OMeghaCloud systemd[1]: Stopping Memcached...
Nov 14 18:01:54 OMeghaCloud systemd[1]: Started Memcached.
Nov 14 18:01:54 OMeghaCloud systemd[1]: Starting Memcached...
Nov 14 18:01:54 OMeghaCloud memcached[29056]: Initialized SASL.
Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   1: chunk size        96 perslab   10922
Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   2: chunk size       120 perslab    8738
Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   3: chunk size       152 perslab    6898
Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   4: chunk size       192 perslab    5461
Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   5: chunk size       240 perslab    4369
Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   6: chunk size       304 perslab    3449
Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   7: chunk size       384 perslab    2730
Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   8: chunk size       480 perslab    2184
Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   9: chunk size       600 perslab    1747
Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class  10: chunk size       752 perslab    1394
Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class  11: chunk size       944 perslab    1110
Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class  12: chunk size      1184 perslab     885
Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class  13: chunk size      1480 perslab     708
Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class  14: chunk size      1856 perslab     564
Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class  15: chunk size      2320 perslab     451
Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class  16: chunk size      2904 perslab     361
[root@OMeghaCloud ~(omegha_eu)]#

6)  Verify if your are able to connect to Memcached locally.

[root@OMeghaCloud ~(omegha_eu)]# memstat --servers="127.0.0.1"
[root@OMeghaCloud ~(omegha_eu)]# echo $?
1
[root@OMeghaCloud ~(omegha_eu)]# 

 We should see an exit status of 1, which tells us that the memstat command failed.

7) Create a user with password (Which will be stored in the SASL database which we will create now)

[root@OMeghaCloud ~(omegha_eu)]# sudo yum install cyrus-sasl-devel cyrus-sasl-plain
..
..
..


[root@OMeghaCloud ~(omegha_eu)]# sudo mkdir -p /etc/sasl2
[root@OMeghaCloud ~(omegha_eu)]# vi /etc/sasl2/memcached.conf 
[root@OMeghaCloud ~(omegha_eu)]# saslpasswd2 -a memcached -c -f /etc/sasl2/memcached-sasldb2
[root@OMeghaCloud ~(omegha_eu)]# chown memcached:memcached /etc/sasl2/memcached-sasldb2
[root@OMeghaCloud ~(omegha_eu)]# systemctl restart memcached
[root@OMeghaCloud ~(omegha_eu)]# memstat --servers="127.0.0.1" --username= —password=<your password>
Server: 127.0.0.1 (11211) pid: 3831
uptime: 9 time: 1520028517 version: 1.4.25

Happy Learning!