Monday, August 07, 2017

How Did I Recover a Hacked Wordpress Website

It was a very interesting and exciting journey. This Sunday evening, one of my friend called me and asked me for an emergency/urgent favor, he is running a startup company and currently in the middle of second round of fund rising, and the company website is not working (Hacked). Great! Isn't it? Having the company website hacked during the fund rising. Well he didn't realize the website was hacked until I logged in, did some investigation and told him :)

I was able to recover the whole website from the evil hands within hours and saved his company a bit chunk of time and money, I guess I was lucky but also skills and experience are count. Not trying to show off here, but just want to share this real-life and exciting story (Based on a true story), lol. Maybe one day you will get "lucky" :)

Background:
Friend's wordpress website is not working, by typing "compnay.domain.com", nothing shows up in the web browser.

Environment:
OS: Ubuntu 12.04.5 LTS
Web server: Apache2

Investigating part:
1. Get the server admin access, remeber to do the password thing over the phone :)

2. Everyone off the server, only the investigator should be logging in.

3. First thing I would to is to go to the Apache log folder, check error*.log and access*.log

4. As soon as I open the "error.log", I am seeing lots of repeating log entries like:
PHP Notice:  Undefined index: my_id in /var/www/wordpress/wp-content/uploads/ad.php
PHP Notice:  Undefined index: web_root in /var/www/wordpress/wp-content/uploads/ad.php on line 3
PHP Warning:  preg_replace_callback(): Requires argument 2, '', to be a valid callback in /var/www/wordpress/wp-content/uploads/ad.php on line 3

5. My first reaction was: Oh no, looks like someone hacked into the system and doing bad stuff, turned the whole website into a advertisement bot or sth.

6. Then I confirmed with my friend, there is no "Ads" stuff going on in their website, also confirmed with him the IP range is not familiar.

7. Then I set up a iptable rule by blocking the suspicious IP subnet, cut off the connections first.

8. Then I did a IP trace and checking, these IPs are actually listed by Google as "Network Abuse". This confirms my guessing that the whole website gets hacked.

9. Shutdown the web server, so no in and out traffic, no potential file updates.

10. Make a copy of the current wordpress folder, before you start changing anything.

11. Start from the log file entires, check the content of the "/var/www/wordpress/wp-content/uploads/ad.php", start from there. In my case, there are suspicious code in this file, for example, calling "/var/www/wordpress/wp-content/uploads/ad.js", also embedded Javascript calling Googlebot.

12. Once you identified the suspected files, move them all into a isolated place, for further investigations.

13. Pay attention to strange file names, such as "imdex.php", also check important/frequent called files such as "index.php", "wp-config.php" files. In my case, the original "index.php" was injected the following code:
<?php require('./wp-content/uploads/ad.php')?>

14. A good practice would download the same version of wordpress source file from official website, and compare the main folder structure, identity any suspicious.

15. I was lucky to identify some big changes the hacker made and recovered the changes, after undo these changes, the website was functional again.

16. Since you will never sure, what are the infected files, you should create a completely new website on a completely new server and (unfortunately) manually restore the current website to the new one (The safest way). Unless you guarantee the backedn database is not infected, you can do a DB dump and restore.

17. Suggest my friend's company to do a post mortem, why this had happend and how to prevent it from happening again. Make some actions.

Suggestions:
1. Monitor your website, with "uptimerobot" or similar cloud services with notification, so that as soon as your website is down, you will get a notification.
2. Monitor your log files with Realtime alerts, ELK log analyzer or other cloud log analyzer.
3. Proper firewall rules.
4. Regular backups, at least you have a choice to go back to the most recent point before it gets hacked.
5. System resource monitoring, cpu, mem, disk, network usages.

I hope you would never need to use this! :)

No comments: