For the first time, an LRob IP has been blacklisted

Seeing an IP blacklisted is part of life as a web host, a daily occurrence for the biggest or most permissive ones. Nevertheless, this is a first for LRob in its 10 years of existence!

The more positive among you will say that this is the price of glory... Inevitably, as the volume increases, so does the risk of unauthorised activity on a site. The most critical will cause a scandal. Whatever the case, at LRob we're committed to transparency. So we'll answer all your questions.

What happened and what solutions are we implementing? Here are the answers.

Discovering blacklisting on AbuseIPDB

On Sunday 13 October, I was alerted by my client Weather Centre that its site was blocked by the paid version of MalwareBytes antivirus (thanks to them for this alert).

Faced with this anomaly, LRob was in a hurry. Whether it was Sunday or Christmas Day, I immediately checked his site, which was fine, and then investigated with MalwareBytes. The cause of the blockage was given in just 1 hour (well done MalwareBytes support): the IPv4 of the server hosting this site is blacklisted on AbuseIPDB for malicious requests. But it seems that the problem has come from another site.

Favourite AbuseIPDB

Not all blacklists are created equal (CF: SPFBL), nonetheless, AbuseIPDB is a real project, well done and serious.

In the midst of all this bad news, the discovery of this community blacklist was a stroke of genius: not only had it enabled me to detect an anomaly on my server, but it was also easy to contribute. And that's why LRob immediately joined AbuseIPDB to help report attacking IPs and help the global sysadmin community in return.

But the most important thing was to stop the source of the malicious requests. So where were they coming from?

An unexpected cause

Of course, I immediately looked for the cause of these requests coming from one of my servers. If it was a customer, this would not be tolerated. If it's a hacked site, it has to be shut down and repaired immediately... If it's a server intrusion, it's extremely serious. So I started investigating, and what I found was both harmless and unusual.

It will not have escaped your attention that LRob carries out the repairing and securing hacked WordPress sites for several years now. This has led many customers to choosing LRob web hosting to benefit from discounted repairs and rock-solid support.

But recently, a new repaired site played a nasty trick on me: a repeat offence. A repaired site was hacked again, which has never happened to me before on any of my hostings.

As this was covered by the 90-day guarantee, I repaired the site free of charge. But the attacks, although less frequent, continued to arrive on AbuseIPDB.

After 3 days of fruitless log searches during the day and evening, I finally analysed all the network traffic on the server (a mammoth task), until I traced one of the abnormal processes (programmes) launched as the system user of this site. Eureka! I've found the root cause of the problem.

How is this possible?

I now know that, as soon as it arrived, this site actually contained malicious programs (not just PHP scripts, but actually programs), which managed to run as soon as the site was imported. This meant that even once the site had been deleted, and even with it deactivated, the malicious programmes continued to run. The hackers were therefore able to use this as an attack vector even after the site had been repaired. They were also able to rewrite malicious files, generating visible recurrence.

As these programs were not running directly via the usual web server, but independently, it was impossible to detect their activity via the web server logs, making detection more difficult (which is why network traffic had to be analysed as a last resort, which is unusual). What's more, this type of programme isn't even supposed to be able to run, and for good reason: the «PHP exec()» function that launches them is supposed to be disabled... But it wasn't for this client because of a specific configuration (since corrected). I'd already seen this problem on other servers and if it hadn't been mine, I'd have thought to look directly at the processes launched and I'd have found the problem in 30 minutes... But here, as it wasn't supposed to happen, it took me 3 days. The joys of the job!

The immediate solution

All dubious processes (programs running) were of course saved for investigation, and then stopped. A complete checkup of the site was redone and a manual check 3x a day was set up to ensure that there were no recurrences.

Current situation

At the time of writing, no further attacks have been reported on AbuseIPDB, but the blacklist has not yet been lifted.

Delisting

The delisting request was made on the day the problem was discovered, and can take between 7 and 10 days to be lifted. It's slow, and that would be AbuseIPDB's only flaw. Fortunately, the impact of this blacklisting is fairly low until the delisting is lifted.

No other sites affected

Thanks to the excellent website isolation applied to my hosting, the hacked site was confined to itself and had no impact on the other hosted sites.

Additional long-term solutions

The impact of this blacklisting is very minimal, as shown by the fact that only Weather Centre alerted me to this problem, as this is probably the most popular site I host, which means they can quickly be notified of this type of information. Thanks again to them and to MalwareBytes for their responsiveness in reporting relevant information.

Nevertheless, some customers may one day want to be able to avoid their site being hosted on a blacklisted IP because of a third-party website. The solution is a dedicated IP address.

That's why I'll shortly be launching a dedicated IP option available to all LRob web hosting.

This requires some preparation because adding multiple IPs to a dedicated server is not trivial. I also need to estimate and define the price of the option. When the option is launched, each LRob customer will be able to benefit from a dedicated IPv4 and IPv6 if they wish.

With a little imagination, this could also allow server migrations without IP changes, thanks to «IP Failover» (more expensive but more flexible, they can be transported from one server to another, as long as the server provider remains unchanged). This should please my resellers who don't have control over all their customers' DNS zones...!

The importance of transparency

There's no such thing as a faultless service. Of course, I could have hidden this minor problem and only one customer would have noticed. But at LRob, we choose to show our commitment to managing all anomalies with rigour and devotion.

Transparency is the basis of trust. And I thank you for your trust.