Cloud Security - Prevent DDOS attacks through Memcached

Recently, I provisioned a new server and went home for 2 days did not login, once I was back, I could see around 2.5 lakh failed login attempts from various IPs from China. So first step was change the SSH default port and disable root login to the server. Chinese attempts did not end there, the next was attempt to use my memcached vanilla setup, which was secured as explained below, hope this helps somebody.

Memcached is an open-source distributed memory object caching system which is generic in nature but often used for speeding up dynamic web applications. In the default configuration, memcached by default listens on ports 11211/tcp and 11211/udp. 

Memcached servers openly accessible from anywhere on the Internet via UDP are abused for DDoS reflection attacks against third parties on a regular basis. This way, extremely high amplification factors can be achieved which poses a serious security threat.

 If a memcached server is openly accessible from the Internet via TCP or UDP and no SASL authentification has been configured, anyone who can connect to the server has unrestricted access to the data stored with it. This allows attackers to modify or delete any dataor potentially steal sensitive information like login credentials for web applications or customer data from online shops.

Now how do we enable SASL 

1) Verify default Memcached configuration

[root@OMeghaCloud ~(omegha_eu)]# cat /etc/sysconfig/memcached

PORT="11211"

USER="memcached"

MAXCONN="1024"

CACHESIZE="64"

OPTIONS=""

[root@OMeghaCloud ~(omegha_eu)]# 

2) Check if UDP protocol is enabled (We have to close UDP protocol access)

[root@OMeghaCloud ~(omegha_eu)]# sudo netstat -plunt|grep memcache

tcp        0      0 0.0.0.0:11211           0.0.0.0:*               LISTEN      1982/memcached      

tcp6       0      0 :::11211                :::*                    LISTEN      1982/memcached      

udp        0      0 0.0.0.0:11211           0.0.0.0:*                           1982/memcached      

udp6       0      0 :::11211                :::*                                1982/memcached      

[root@OMeghaCloud ~(omegha_eu)]# 

3) Change it as show below to restrict Memcached access only through tcp and only to localhost. -S and -vv parameters, -vv provides verbose output to /var/log/memcached, which will help us as we debug. -S enables SASL. -l will restrict listening to loopback IP and -U 0 will switch off UDP protocol.

[root@OMeghaCloud ~(omegha_eu)]# cat /etc/sysconfig/memcached

PORT="11211"

USER="memcached"

MAXCONN="1024"

CACHESIZE="64"

OPTIONS="-l 127.0.0.1 -U 0 -S -vv"

[root@OMeghaCloud ~(omegha_eu)]# 

4) Restart the Memcached service and verify if UDP listening is disabled.

[root@OMeghaCloud ~(omegha_eu)]# systemctl restart memcached

[root@OMeghaCloud ~(omegha_eu)]# sudo netstat -plunt|grep memcache

tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN      29056/memcached     

[root@OMeghaCloud ~(omegha_eu)]# 

5) Verify if the Memcached service has enabled SASL this time

[root@OMeghaCloud ~(omegha_eu)]#journalctl -u memcached

-- Logs begin at Wed 2018-11-14 05:49:06 CET, end at Wed 2018-11-14 18:02:02 CET. --

Nov 14 05:49:22 OMeghaCloud systemd[1]: Started Memcached.

Nov 14 05:49:22 OMeghaCloud systemd[1]: Starting Memcached...

Nov 14 18:01:54 OMeghaCloud systemd[1]: Stopping Memcached...

Nov 14 18:01:54 OMeghaCloud systemd[1]: Started Memcached.

Nov 14 18:01:54 OMeghaCloud systemd[1]: Starting Memcached...

Nov 14 18:01:54 OMeghaCloud memcached[29056]: Initialized SASL.

Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   1: chunk size        96 perslab   10922

Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   2: chunk size       120 perslab    8738

Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   3: chunk size       152 perslab    6898

Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   4: chunk size       192 perslab    5461

Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   5: chunk size       240 perslab    4369

Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   6: chunk size       304 perslab    3449

Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   7: chunk size       384 perslab    2730

Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   8: chunk size       480 perslab    2184

Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class   9: chunk size       600 perslab    1747

Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class  10: chunk size       752 perslab    1394

Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class  11: chunk size       944 perslab    1110

Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class  12: chunk size      1184 perslab     885

Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class  13: chunk size      1480 perslab     708

Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class  14: chunk size      1856 perslab     564

Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class  15: chunk size      2320 perslab     451

Nov 14 18:01:54 OMeghaCloud memcached[29056]: slab class  16: chunk size      2904 perslab     361

[root@OMeghaCloud ~(omegha_eu)]#

6)  Verify if your are able to connect to Memcached locally.

[root@OMeghaCloud ~(omegha_eu)]# memstat --servers="127.0.0.1"

[root@OMeghaCloud ~(omegha_eu)]# echo $?

1

[root@OMeghaCloud ~(omegha_eu)]# 

 We should see an exit status of 1, which tells us that the memstat command failed.

7) Create a user with password (Which will be stored in the SASL database which we will create now)

[root@OMeghaCloud ~(omegha_eu)]# sudo yum install cyrus-sasl-devel cyrus-sasl-plain

..

..

..

[root@OMeghaCloud ~(omegha_eu)]# sudo mkdir -p /etc/sasl2

[root@OMeghaCloud ~(omegha_eu)]# vi /etc/sasl2/memcached.conf 

[root@OMeghaCloud ~(omegha_eu)]# saslpasswd2 -a memcached -c -f /etc/sasl2/memcached-sasldb2

[root@OMeghaCloud ~(omegha_eu)]# chown memcached:memcached /etc/sasl2/memcached-sasldb2

[root@OMeghaCloud ~(omegha_eu)]# systemctl restart memcached

[root@OMeghaCloud ~(omegha_eu)]# memstat --servers="127.0.0.1" --username= —password=<your password>

Server: 127.0.0.1 (11211) pid: 3831

uptime: 9 time: 1520028517 version: 1.4.25

Happy Learning!

N.R. Narayanamurthy, Ajith Narayanan, Bulgaria, Sofia, Istanbul, India - What's the link?

This blogpost was again long pending one, but I was not sure on how to write this non technical real life experience as a blogpost, as I used to write only technical blogposts. Recently was searching for something in google and landed up on this link. 

https://www.cntraveller.in/story/how-hitchhiking-through-europe-inspired-narayana-murthy-to-set-up-infosys/#s-cust0 

This experience of mine had a striking similarity to NRN’s nomad journey during his late twenties and thought, if I don't blog it now, I am not going to blog this forever.

It was about the horrific experience Mr. NRN had during his early days of travelling across Europe (Well before he became an entrepreneur and started Infosys). The story is written very well and almost giving the chill through the spine when we read the strange and horrifying experience in a different country.

After reading this story, I though I should also write one of my travelling experience to an east European country which had a striking resemblance to NRN’s story. I was a regular Oracle conference goer as a speaker, wherever I used to get opportunity to speak, I used to grab the opportunity and go and share my real time work experiences with the attendees in the conference. In the year 2012, I had two such conferences in which my abstracts were selected and invited to speak. The conferences were in two different countries and two different cities, the first one was in Nuremberg, Germany and followed by another  conference in Sofia, Bulgaria.

I did plan my travel itinerary according to the conferences and speaking slots given to me in both these conferences, and my travel plan was something like this

Bangalore -> Berlin -> Nurenberg -> Berlin -> Sofia -> Berlin -> Bangalore. 

I had the schengen visa applied and got visa from German consulate (thats why there was multiple hops through Berlin as it was my first port of disembarkment into EU).

I was in German city of Nurenberg, met many experts in Oracle world and had a great conference there, I also made some new friends there, some of them were very famous and established speakers compared to a novice like me. 

Then came the worst part of the journey, I with some of the new friends started enroute to Bulgaria(The next conference location), took flight to Berlin and from there taking a flight to Sofia, Bulgaria. It was a good flight with new friends whom I met in the first conference. Happily landed in Sofia, all of them who landed with me crossed the border control except me and to my surprise, the border control informed me that, The schepgen visa I had was no more valid, they've tentatively stoped accepting people with Schengen visa since last couple of weeks citing some diplomatic issue with EU (Bulgaria is and was not part of EU always).

All the people who were with me from Germany to Bulgaria in flight crossed the border control and I was sitting at the border control officers cabin, it was a nightmare, I could not believe what had really happened to me, I did not know what to  do, the officer was a lady and she did not know a single word of english and I did not know anything other than English, then more flights kept landing and more people from various countries where passing through the border control, but saw few of them being stopped at border check and where brought inside the same officer's cabin with me

 Some of them looked like as shown in the below pic, believe me that was the scariest part of this story.

I started praying, I started imagining about the Bulgarian jails. 

After around 3-4 Hrs, the border control officers had their shift change, then another officer came for his change of duty and handover, he was briefed about my case by the lady officer before leaving for the day. The new officer was again a six  plus feet tall heavy built, tough looking officer, he was looking at my face throughout his discussion with his colleague, after sometime he saw me sweating and came near me and asked “oh indien, from India, I know, I know india, bombay’

I had a great relief when I heard the officer speaking in English, to my surprise the officer, though looked a tough guy by looks, was very jovial, he started asking about me, my job, about Bangalore etc etc, for around an half an hour, during the conversation he even asked me about job opportunities in Bangalore and said, they pay them very less there and was thinking of leaving Bulgaria, meanwhile, I forgot the tough situation I was in and was happily talking to the officer.

The officer told me unfortunately there is no direct flights from Sofia to India, So I will have to take another route either through Dubai or Qatar way back to home, and they will help me reaching home safely. I was happy to hear that, thanked god and my family, The first flight available was a Turkish airways to Istanbul, the officer came to me and asked if I could check if I can get a visa to turkey quickly and take a flight from Istanbul to back home in Mumbai.

I connected to the Sofia airport wi-fi, again to my surprise, I was able to get a e-visa to turkey in next 15 to 20 mins, quickly, the officer arranged a ticket in that Turkish airways for Istanbul. The officer helped me a lot, but he had only one request for me while escorting me to the Istanbul flight gate, he asked me to buy a big pack of cigarettes from the duty paid shop for him and said, he cannot buy that from the duty paid shop while in duty and CCTV’s might trouble him later if he did so.

I was very happy for the officer, happily bought the big pack of cigarretes for him, he helped me a lot and escorted me until the Turkish airways gate. I reached Istanbul with a heavy heart, ataturk airport was huge and beautiful. From getting down at Istanbul airport, I was sure about reaching back home safely, I called up my family to inform the horrible experience I had until then, I had the courage to manage it by myself, but did not have the courage to call up my home and tell them about this episode.

My flight to Mumbai was in next 20 hrs, though I had e-visa valid for 30 days and could have stayed in Istanbul and seen the places like Blue mosque before returning to Mumbai, but , I was really exhausted, I wanted to be back home as early as possible, booked my ticket to Mumbai, waited in the Ata`turk airport for next 20 Hrs, did not feel like stepping out. Finally after 20 hrs of wait at Ataturk airport, the flight from to Mumbai arrived. I boarded the flight and after almost 6+ hrs I finally reached Mumbai.

The moment I landed in Mumbai, I cannot describe the happiness in words, no matter which country you are in, it is always felt safe when you are back to your home country.

After a month after reaching home, I started receiving emails from the friends I made in Germany, who were helpless in the Bulgarian border, but had to see me being denied entry from the other side of the border. It was even the first time they where travelling with someone who throughout the flight were sitting next and talking and all of the sudden vanish. It took atleast a week's time to come out of this trauma.

My original itinerary was 

 Bangalore -> Berlin -> Nurenberg -> Berlin -> Sofia -> Berlin -> Bangalore

Both NRNs and my itinerary got changed from Sofia, Not sure where NRN went from Istanbul.

Bangalore -> Berlin -> Nurenberg -> Berlin -> Sofia -> Istanbul -> Mumbai -> Bangalore

What I learnt from this whole experience ?

- Somebody said, fact is stranger than fiction, I don’t know who said that, but it was very very true.

- The more you travel, meet more people, you get to know the big world we live in and how small we as humans are.

- Travel more, gain more real-world experience, explore the world you live-in and life is short, our world will teach more lessons than any of the books available in library.

Note:- This is a real story, if somebody wants to make a film out of this thread, please remember to take my written permission, else I may sue you for stealing the real story of mine.

For a change -> HAPPY READING!  

Why InfraStack-Labs as a startup is doing what it is doing?

Happy to back with blogpost again after a very long time, currently going through a very tough phase, on May 4th, I broke my right leg ankle with multiple fractures and underwent a surgery. Since then, I am immobile which has affected many of the plans like travel, meetings and discussions. All plans took a hard hit. Hope to hit the road again by end of August/Early September.

Though many topics keeps hopping in mind for blogging, there are many factors like(My current situation), which are not under anybody's control. I was thinking of blogging about these very common and frequently asked questions since the day one of InfraStack-Labs (27-Oct-2015)

I hope, this blog can help in realising India's position in the technology context and to some extent will help in avoiding the misconceptions and a prejudice mindset on our country's capabilities on technology front.

We in our country have still not learned how to lay a proper road (which is one the main pillar of infrastructure that leads to many other developments in the country), that is the reason we are taking about filling potholes every monsoon and not thinking/talking about how to built a road that is robust enough to withstand any severe monsoon.

RESULT -------------------------------------------->

Hint - We keep tarring the roads every year on top of the previous layer, without even thinking if we have a proper water drainage on both sides of the roads and few technical aspects that will determine the life of the new road infrastructure, Main design factors include:

1. location
2. terrain and soil properties,
3. drainage capabilities,
4. traffic volume,
5. the ratio of cars to trucks and buses,
6. possible future development in the area,
7. effects on the environment or  nearby residents.

I think, its too much of analogies and examples to set the context of this blogpost. Let's goto some FAQ's we have encountered for almost last 3 years.


1) Why InfraStack-Labs as a startup is doing what it is doing?

We were not a group of lucky fresh graduates of today who come straight out from a college with many brilliant ideas and did not have any compulsion to send back a part of the salary to home. Youngsters of today have the luxury of taking risks and exploring things and lot of resources. We were a bunch of infrastructure people (Mainly database) who wanted to put our expertise and build on a technology which really had a vacuum in India. That technology happened to be Cloud Technology. (Literally we did not want to give lip-service to all buzzwords in technology, we wanted it to be practically implemented)

Now how is our choice important in Indian startup ecosystem context

•   India is grooming itself as a preferred startup ecosystem
•   Indian government is promoting startups and progressing towards its   digital India dream
•   Indian startup ecosystem revolves around below categories

            Aggregator websites
            Marketplace
            e-Commerce websites
            Web/Mobile Apps
            System Integrators
            Fintech
            Buzzword bees (AI, ML, Analytics, Blockchain, Future etc)

•   One common factor among all these categories of startups is the underlying cloud technology that is used to host their products and reach out to a larger audience.

2) How do you guys compete with other big cloud companies?

If some cloud vendor is the market leader, no disputes in that, but that cannot be a valid reason for anybody to avoid trying into getting into cloud market (Regardless the market cap of any market leader in cloud computing, it does not give any significant impact on others who wish to compete in same space)

We don't need to compete with anybody in this world, this world is too big for anybody to have 100% control, I've travelled length and breadth of India in train and buses and know how big this country is, If you always take flights, you may have a wrong impression of the very earth which you live in, you may feel that earth is very small.


3) Why will somebody trust you instead of other big cloud companies?

Be it the market leader or any other high tech companies in Cloudcomputing space , our Indians have significant contribution in all of those companies, Why should the same talent pool be unhappy If India gives them same opportunity to show their skills and get rewarded for their skills.

So if the same talent pool is created in India to work on similar exciting projects, there can be more popular softwares, solutions and platforms that will eventually be trusted in the days to come.


4) What is the advantage of going with OMegha Cloud instead of other big cloud companies?

CHALLENGES CONVERTED TO SOLUTION

• High cost & Complex pricing model
• Dedicated support & training only at extra cost
• Overbuilding & recurring charges Monitoring considered as additional event
• Fixed infrastructure options Reactive process
• No guidance

• Hourly Billing
• Dedicated support at ZERO additional cost
• Transparent & cost centric platform
• Default live performance tracker On demand scalable infrastructure
• Proactive automated process
• Handheld cloud enablement 

5) Do you think, India is a good place to start a Cloud company?

Absolutely, we have already tested the waters, we have enough resources to even operate tier-1 Datacenters and serve the cloud computing needs of the whole world. We have been operating the OMegha™ cloud for more than 30 months without any issues.


Now, let me compare India with some European countries who are very much worried about their citizens and the data privacy. 

Note:- The below data has both web hosting companies & cloud companies listed. Both are different.

India is 108 times bigger than Belgium & Belgium has approximately 43 (Cloud & Web hosting) companies providing (IaaS and/or PaaS)

India is 76 times bigger than Denmark & Denmark has approximately 55 (Cloud & Web hosting) companies providing (IaaS and/or PaaS)

India is 10 times bigger than Finland & Finland has approximately 43 (Cloud & Web hosting) companies providing (IaaS and/or PaaS)

India is 10 times bigger than Norway & Norway has approximately 43 (Cloud & Web hosting) companies providing (IaaS and/or PaaS)

India is 7 times bigger than Sweden & Sweden has approximately 47 (Cloud & Web hosting) companies providing (IaaS and/or PaaS)

India is 79 times bigger than Netherlands & Netherlands has approximately 33 (Cloud & Web hosting) companies providing (IaaS and/or PaaS)

India is 7 times bigger than Spain & Spain has approximately 61 (Cloud & Web hosting) companies providing (IaaS and/or PaaS)

India is 9 times bigger than Germany & Germany has approximately 62 (Cloud & Web hosting) companies providing (IaaS and/or PaaS)

India is 13 times bigger than UK & UK has approximately 66 (Cloud & Web hosting) companies providing (IaaS and/or PaaS)

Finally, how InfraStack-Labs & OMegha Public Cloud is  positioning itself in the cloud computing vendor space currently and future roadmap.

	InfraStack-Labs (OMegha™ Public Cloud)	2nd Largest Global Cloud Company
Year of inception	2015	2012
Headquarters	Bangalore, India	NewYork, USA
Headcount	~10	~300+
Accelerators	None	TechStars
Was AWS leading the pack then?	Yes	Yes
Total Funding	Bootstrapping	$305+ Million
Compute	Yes	Yes
Block Storage	Yes	Yes
Object Storage	Yes	Yes
Loadbalancer-as-a-Service	Yes	Yes
Database-as-a-Service	Yes	No
Converged Analytics Engine	Yes	No
Vinayak - Techstack Dignostics Robot	Yes	No
Network Function Virtualization	Yes	No
High Performance Computing Appliance	Yes	No
Private Cloud Appliance	Yes	No
Serverless Platform	Yes - In Progress	No
IoT Platform	Yes - In Progress	No
BareMetal Service	Yes - 2019 Roadmap	No

HAPPY LEARNING !