Well, that turned into a dumpster fire, quickly.
If you haven’t heard about Log4j or Spring4Shell, consider yourself lucky. Instead of going in-depth on what they are, let me direct you to some relevant pages that will bring you up to speed on these two. Log4j was revealed in December of 2021 and managed to scramble security practitioners, developers, and many others around the world. Not long after that, in March 2022, Spring4Shell was also revealed. Spring, while not (arguably) not as severe as Log4J, only compounded on already wore-out staff.
The challenges (from my perspective).
It was not easy to detect either Log4J or Spring4Shell. These were code-based vulnerabilities that could lie in dormant Git repositories or .jar files anywhere, just waiting to be deployed. Finding them deployed wasn’t exactly easy either. Scanning vendors had a hard task ahead of them creating the detections to find these. In fact, some are still having issues detecting them.
Let’s not forget about detecting vendor manual fixes. If your scanner is only looking at version detection then it may very well be missing that vendor deployments may have made a manual fix to remediate the JNDI in Log4J, as an example.
Clean-up of artifacts, repositories, jar files, and everything else that could contain a vulnerable version of Log4j or Spring4Shell is like finding a needle in a needle-stack. Especially if you work for a very large company with tens of thousands of devices and thousands of developers.
These issues, and others, meant that security teams, management, developers, and many others had to scramble to get this under control before threat actors could find and exploit these vulnerabilities.
Leaving a dumpster fire…
Is this something that we will see more of?
I fear it is. Log4J, in my opinion, opened the door to the development stack being heavily targeted. Sure, there are other vulnerabilities that are found there. However, how long did Log4J sit out there vulnerable until it was found? Is it also a coincidence that Spring4Shell came out not long after Log4J? I don’t think so, I don’t know how many Log4J instances I saw that were deployed with Spring as well.
Failure is a great teacher, and I think when you make mistakes and you recover from them and you treat them as valuable learning experiences, then you’ve got something to share.
Steve Harvey
So what do we do going forward?
That’s the million-dollar question, right? I’ll be eagerly awaiting that check.
All jokes aside, we (security practitioners) must do a better job in preparing for disasters. “Disasters?”, you say. Yes, this would qualify as one. It came out of nowhere and turned things upside down and required all hands-on deck to address. Not all disasters are fires, earthquakes, or floods.
If you haven’t, go back and create a playbook. What worked well? What didn’t work well? Involve stakeholders from multiple areas, including InfoSec, development, infrastructure, management, and anyone else that may have played any part in your efforts. What could you have done to better react, address, and remediate these in a timely manner? How do you quickly activate that plan the next time something like this comes up? This is all very important information to have in your playbook. CISA has some great documentation on how to do this on their site.
Closing thoughts
While I’m sure many of us will always remember the never-ending calls, war rooms, panic, scramble, and long days of Log4J & Spring4Shell, we overcame it.
One thing I would highly recommend to anyone, regardless of role, is to take care of yourself first. You can’t really help your company, your team, or your customers if you’re completely wore out, burned out, and barely hanging on yourself. These things happen. How you react to them is up to you. Don’t panic. Take your time to figure out your next steps and then move swiftly. There is nothing worse than doing a LOT of work that ultimately becomes useless as information changes and priorities shift. Take the time to understand the issue at hand and formulate a game plan to move forward. Again, this is where playbooks mentioned above come in to play.
Stay vigilant and get some well-deserved rest.
Best of luck on your journey!