Bunkerweb WAF now protecting OpenCTI Instance
First off, I'd like to apologize for my absence in writing new articles on my Blog. I have been absent due to a grueling personal issue, that is almost resolved! So going forward, I am excited to get more time to dive back into writing more, and helping the open source community! Thank you all for your patience, and it is an absolute joy to be doing all the things I have been to help, instruct and give back to the Cybersecurity, Linux and Open Source Security community.
On to the update regarding the OpenCTI instance having issues lately, which are fixed now! 😄
Bunkerweb WAF for OpenCTI platform
Lately I started to notice as the site got more and more popular, and my OpenCTI Live feed populating my Blog with all the newest Reports and Threat Summaries. The platform as it grew in data and size, along with more users finding these articles via search engines. The OpenCTI instance started to come to a crawl.
Investigating further, I knew it wasn't a resource problem, as I have it running on some very powerful hardware. So looking more closely at the NGINX logs, using a great tool called Goaccess. I started to quickly realize, there were many malicious IP's just blasting and brute forcing and scraping the GraphGL API URL continuously. This is the core central hub of how OpenCTI works, and it was getting so many requests, completely overloading the back-end. Proven by looking at the Docker Logs.
At this point, I knew I had to do something to protect this instance better, now that it is now on the Radar of the "Bad Guys". 😄 So I completely redid the front end Nginx Reverse Proxy and completely rebuilt it from the ground up on a very beefy VM with tons of bandwidth to handle the Bots, DDOS attacks, Scrapers, Malicious IP's etc etc from ruining it for the rest of us!
It is IMO the best open source complete WAF solution, with sooooo many add-on's and features, most of which are totally free. Most of these layers I have leveraged, including all sorts of real time IP blacklists, bad behavior / modsecurity like bad behavior detection, Crowdsec, and much more. All tor exits, open proxies and many VPN's are now banned. Those IP's that show suspicious behavior will be throttled, and then banned automatically. I pull from a very large Fail2ban 24 hr list, where if any malicious IP was detected in the last 24hrs and blocked from any fail2ban instance in the blocklist. They will automatically be proactively blocked.
After tuning and testing it, things are so much better, and the API and site is not getting flooded with brute force, and DDOS packets, bots and scrapers and they are getting dropped on the spot.
In summary for those interested, the new architecture is a beefy VM running in the cloud acting as the front end / front door to the site. Bunkerweb is now setup along with ipset blacklists to drop the biggest offending IP's at the iptables level. Bunkerweb is more higher up in the OSI, and together this works amazingly well. This front end takes all the "arrows" and hits residing in a premium data center with a large "pipe". I then moved the backend OpenCTI platform to On-Prem beefy hardware and the data is proxy passed between the two servers over a secure Wireguard tunnel using our overlay network for optimal efficiently.
This VM has a 10gbit connection, and even with much processing power (cpu / memory). I was still noticing TCP drops and syn / ack issues with the network stack being still overwhelmed. So I added the following to the /etc/sysctl.conf file to turbocharge the entire network stack of the Nginx BunkerWeb Proxy, solving the issue by taking off the "restriction plate" if you will, (the default Linux TCP / Net Settings.
For those of you wanting to greatly boost your Linux machines performance across the board with an easy tune, add this to the end of your sysctl.conf file and then "sysctl -p" to reload the changes.
net.ipv4.ip_local_port_range = 15000 65535
vm.max_map_count=1048575
net.core.wmem_max=16777216
net.core.rmem_max=16777216
net.ipv4.tcp_rmem= 10240 87380 16777216
net.ipv4.tcp_wmem= 10240 87380 16777216
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_max_syn_backlog = 8096
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 50000
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_syn_backlog = 65536
net.ipv4.tcp_syncookies=1
Thanks for the patience, unfortunately, all the malicious dirt-bags always have to try and ruin it for the rest of us! 😀
Make note of the new public user password I changed recently on the login page!
Enjoy!
Dan