Pages

Thursday, September 10, 2020

My WHCD exam experience

Introduction

My story starts in July in Budapest, summer is hot. Way too hot. I am working in the cozy air-conditioned office room and look at my daily schedule. I have a business lunch with Sándor Fehér, co-founder & CEO at White Hat IT Security at an Italian restaurant. This will be a casual meeting, catching up with each other, no preparation needed. I show up at the place, meet Sándor, order a pizza, and we discuss average lunch topics like how APT attackers operate nowadays. He mentions that in September they will organize the exam for their WHCD - White Hat Certified Defender course. And he is happy to provide me a slot for the exam for free. But as nothing is free in life. If I pass - and why would not I? :) - I create a blog post about my view on the exam. I happily accept the invitation and finish the pizza. 

Preparation

In 2020, the course consisted of 36 hours of training, touching topics like threat intelligence, IoCs, computer forensics, network forensics, all the blue team-related topics. I felt pretty confident that I don't have to prepare for this exam, even if I did not attend the training and I did zero real forensics investigation in the past eight years.  
Before the exam, I received multiple documents on the rules and practical information for the exam. The exam started at 20:00 Friday and ended on 20:00 Sunday. 48 hours for finding 10 flags and provide documentation. Actually, 7 flags are enough to pass the test. The only preparations I made was to buy some energy drinks and sweets. Time is 19:30, so I still have some time to watch the latest Last week tonight before the exam starts.  

My past experience

I have the following certifications which helped me during the exam: OSCP, OSCE, PSP (Powershell for Penetration Testers). I also have some experience with CTFs. 8 years ago I did some forensics investigations, but that was a loooong time ago, things changed. A lot. 

The exam - Friday

I receive all the IP addresses and credentials to log in to my dedicated Windows AD network, consisting of a workstation, a File Server, and a Domain Controller. I am happy that I have to work with a live system - it is really painful when you work with offline images. I set the goal that today I find at least one flag, have a good night's sleep and find the remaining 9 on Saturday, and write the documentation on Sunday. Let's find the easiest task and get the flag quickly ... The challenge descriptions are clear, and even the challenge IDs nicely map to MITRE ATT&CK®.  
Two hours later I realize I underestimated the difficulty of this exam, and I overestimated my skills. Zero flags found and ...




Spoiler alert - I was logging into the workstation with the admin user, and not with the victim. I realized this on Sunday afternoon ... This first initial mistake took me at least two hours lost in this initial stage. Anyway, always read through the exam instructions and don't rush into connecting to the first machine on the provided list. I finally find a sneaky backdoor, analyze it, and submit my first flag. Phew, this is not going to be easy. I go to sleep, but at 5:30 a.m. my brain starts to figure out the challenges even though I am still in my bed, eyes closed. Based on past experience, it does not make sense to try to force myself to sleep, it is better to wake up and keep rollin'

Saturday

Around 10:00 a.m., I submit another flag. I already spent like seven hours, and I only have two flags. Boy oh boy, this is going to be tough. Time to crack open another energy drink. Hours later I find myself ALT + TAB-bing between the different challenges and making no progress. Time to have a break. When I am back, I make some good progress. Basic programming skills are really needed to solve multiple challenges. Sometimes I mix up AES keys with IVs but luckily it is OK to fail. Flag after flag. It is evening, and I have 6 flags already \o/. I also found our main suspect, a bird! I was like "let's find the seventh flag now and I can rest in peace". Let's connect the "Rubber duck debugging" and writing the documentation together - maybe I find out what I missed. It is already dark outside, I am tired as hell, but I made good progress with the documentation at least. This tactic turned out to be useful, as I submitted my seventh flag! Finally. I feel like rock and I can find all the remaining flags on Sunday. GOTO sleep(28800). 


Sunday

My brain wakes up again earlier than I want. Let's finish the documentation and after that focus on the remaining flags. I spend a lot of time on Stackoverflow on how to insert nice syntax-highlighted code into the report (Hello Microsoft, it is 2020, please make it easier for us with builtin tools ...) and I also spend more time than I want to admit on how to format the flags in Wordart to make it look nice. Documentation finished, let's go back to work. While I close all the opened tabs, I finally find the exam instructions, with the credentials to the user's workspace. "What the fluff, how could I overlook this?". MMMKay, let's log in. Run a bunch of tools. Run even more tools. Let's dig up emails from 2009 for a German forensics tool I don't remember the name anymore. But this whole exam is less about tools and more about perseverance and thoroughness. Time flies fast, no progress. Let's have a break, and kill Sigrun, the Valkyrie Queen in God of War. When I got back, no new flags submitted themselves. Let's listen to a CTF playlist, maybe that helps. Well, no, it does not. Let's download malware information collector software to the workstation and run that for clues. Pro tip: never do this on a live forensics investigation ;) 


It's at 6 o'clock in the evening and I am getting really tired. And the fact that I already have enough flags to pass the exam makes me lazy. I close my eyes and all I can see is obfuscated code flickering. I literally made zero progress on Sunday on the remaining flags. Maybe if the passing limit was 8 flags, I would have tried harder®? We will never know. 

Monday



During the day I ping the organizers for the solutions regarding the remaining flags. They are kind enough to provide these to me. And I am disappointed. Not because of the challenges, but because of me. One flag I missed as I was not thorough enough. I missed another flag because I was focusing on the wrong computer - even though this challenge was pretty hard, there is a chance I could not even have solved it. The third missing flag - I was not thinking outside the box enough. And my boxes were these three machines I had.
 

Conclusion

The whole exam experience was perfect. Access to the machines was fast and responsive. I did not experience any technical glitch. All the challenges and the whole story is very challenging but possible to solve. I would say that high-end financially motivated attackers could attack any corporation the same way, and basic detection or forensics would miss most clues. The whole scenario is 100% realistic. Most attacks/techniques are sneaky. The only thing I missed is that on the official exam Discord, they should create a #rage or #random or #memes channel where you can send all your frustration during the exam. 
 
Lesson l: Never underestimate an exam just because it is not known. 
Lesson 2: Never overestimate the knowledge and skills you did not practice for a long time. 
Lesson 3: This WHCD exam provided me the same - or even better experience than I had at Offensive exams like OSCP or OSCE. 
If someone passes this challenge, it is really true that the person really knows how to technically deal with computer incidents. I vision great success for this certification because as of today, there are not many hands-on practical defender exams available on the market. 
I would like to thank Sándor Fehér for this opportunity and to János Pallagi for his support during the weekend and to share some thoughts with me after the exam. 







Thursday, January 16, 2020

The RastaLabs experience

Introduction


It was 20 November, and I was just starting to wonder what I would do during the next month. I had already left my previous job, and the new one would only start in January. Playing with PS4 all month might sound fun for some people, but I knew I would get bored quickly.

Even though I have some limited red teaming experience, I always felt that I wanted to explore the excitement of getting Domain Admin – again. I got my first DA in ˜2010 using pass-the-hash, but that was a loooong time ago, and things change quickly.
While reading the backlogs of one of the many Slack rooms, I noticed that certain chat rooms were praising RastaLabs. Looking at the lab description, I felt "this is it, this is exactly what I need." How hard could it be, I have a whole month ahead of me, surely I will finish it before Christmas. Boy, was I wrong.



The one-time fee of starting the lab is 90 GBP which includes the first month, then every additional month costs 20 GBP. I felt like I was stealing money from Rastamouse and Hackthebox... How can it be so cheap? Sometimes cheap indicates low quality, but not in this case.



My experience


Regarding my previous experience, I already took OSCP, OSCE, SLAE (Securitytube Linux Assembly Expert), and PSP (Powershell for Pentesters), all of which helped me a lot during the lab. I also had some limited red teaming experience. I had more-than-average experience with AV evasion, and I already had experience with the new post-exploit frameworks like Covenant and Powershell Empire. As for writing exploits, I knew how a buffer overflow or a format string attack worked, but I lacked practice in bypassing ASLR and NX. I basically had zero experience with Mimikatz on Windows 10. I used Mimikatz back in 2012, but probably not since. I also had a lot of knowledge on how to do X and Y, on useful tools and hot techniques, but I lacked recent experience with them. Finally, I am usually the last when it comes to speed in hacking, but I have always balanced my lack of speed with perseverance.

RastaLabs starts in 3,2,1 ...


So I paid the initial entry fee, got the VPN connection pack, connected to the lab, and got my first flag after ... 4 days. And there were 17 of them in total. This was the first time I started to worry. I did everything to keep myself on the wrong track, stupid things like assuming incorrect lab network addresses, scanning too few machines, finding the incorrect breadcrumbs via OSINT, trying to exploit a patched web service (as most OSCPers would do), etc. I was also continually struggling with the tools I was using, as I never knew whether they were buggy, or I was misusing them, or this is just not the way to get the flag. I am sure someone with luck and experience could have done this stage in 2-3 hours, but hey, I was there to gain experience.

During the lab, whenever I got stuck with the same problem for more than 30-40 hours and my frustration was running high, I pinged Rastamouse on the official RastaLabs support channel on https://mm.netsecfocus.com/. I usually approached him like "Hi, I tried X, Y, and Z but no luck", then he replied "yeah, try Y harder". This kind of information was usually all I needed, and 2-3 hours later I was back on track again. His help was always enough, but never too much to spoil the fun. The availability and professionalism of Rastamouse was 10/10. Huge multi-billion dollar companies fail to provide good enough support, this one guy here was always there to help. Amazing. I highly recommend joining the Mattermost channel – it will help you a lot to see that you are not the only one stuck with problems. But please do not DM him or the channel if you have not already tried harder.

What's really lovely in the lab is that you can expect real-world scenarios with "RastaLabs employees" working on their computer, reading emails, browsing the web, etc. I believe it is not a spoiler here that at some point in time you have to deliver malware that evades the MS Defender AV on the machine. Yes, there is a real working Defender on the machines, and although it is a bit out of date, it might catch your default payload very quickly. As I previously mentioned, luckily I had recent experience with AV evasion, so this part was not new to me. I highly recommend setting up your own Win10 with the latest Defender updates and testing your payload on it first. If it works there, it will work in the lab. This part can be especially frustrating, because the only feedback you get from the lab is that nothing is happening, and there is no way to debug it. Test your solution locally first.

Powershell Empire turned out to be an excellent solution for me, the only functionality it lacked was Port Forwarding. But you can drop other tools to do this job efficiently.

A little help: even if you manage to deliver your payload and you have a working C&C, it does not mean your task with AV evasion is over. It is highly probable that Defender will block your post-exploit codes. To bypass this, read all the blog posts from Rastamouse about AMSI bypass. This is important.

Lateral movement


When you finally get your first shell back ...



A whole new world starts. From now on, you will spend significant time on password cracking, lateral movement, persistence, and figuring out how Windows AD works.
In the past, I played a lot of CTF, and from time to time I got the feeling "yeah, even though this challenge was fun, it was not realistic". This never happened during RastaLabs. All the challenges and solutions were 100% realistic, and as the "Ars poetica" of RastaLabs states:



...which is sooooo true. None of the tasks involve any exploit of any CVE. You need a different mindset for this lab. You need to think about misconfigurations, crackable passwords, privilege abuse, and similar issues. But I believe this lab is still harder to own than 90% of the organizations out there. The only help is that there are no blue-teamers killing our shells.

About the architecture of the lab: When connecting to the lab with VPN, you basically found yourself in a network you might label as "Internet", with your target network being behind a firewall, just as a proper corporate network should be.
There are a bunch of workstations – Win10 only, and some servers like fileserver, exchange, DC, SQL server, etc. The majority of servers are Windows Server 2016, and there is one Linux server. The two sites are adequately separated and firewalled.

As time passed, I was getting more and more flags, and I started to feel the power. Then the rollercoaster experience started. I was useless, I knew nothing. Getting the flag, I was god. One hour later, I was useless.



For example, I spent a significant amount of time trying to get GUI access to the workstations. In the end, I managed to get that, just to find out I did not achieve anything with it. For unknown reasons, none of the frameworks I tried had a working VNC, so I set up my own, and it was pain.

On December 18, I finally got Domain Admin privileges. So my estimation to "finish the lab" in one month was not that far off. Except that I was far from finishing it, as I still had to find five other flags I was missing. You might ask "you already have DA, how hard could it be to find the remaining five?". Spoiler alert, it was hard. Or to be more precise, not hard, just challenging, and time-consuming. This was also a time when connections on Mattermost RastaLabs channel helped me a lot. Hints like "flag X is on machine Y" helped me keep motivated, yet it did not spoil the fun. Without hints like this, I would not have written this post but would have been stuck with multiple flags.

About exploitation


And there was the infamous challenge, "ROP the night away." This was totally different from the other 16. I believe this image explains it all:


If you are not friends with GDB, well, you will have a hard time. If you don't have lots of hands-on experience with NX bypass - a.k.a ROP - like me, you will have a hard time with this challenge. The binary exploit challenges during OSCP and OSCE exams are nowhere near as complex as this one. If you have OSEE, you will be fine. For this challenge, I used GDB-Peda and Python pwntools – check them out in case you are not familiar with them. For me, solving this challenge took about 40 hours. Experienced CTF people could probably solve it in 4 hours or less.

Conclusion


I would not recommend taking this lab for total beginners *. I also do not recommend doing the lab if you only have limited time per day, which is especially true if you are working on your home computer. I probably would have saved hours or even days if I had set up a dedicated server in the cloud for this lab. The issue was that the lab workstations were rebooted every day, which meant that I always lost my shells. "Persistence FTW", you might say, but if your C&C is down when the workstation reboots, you are screwed. "Scheduled tasks FTW", you might say, but unless you have a strict schedule on when you start your computer, you will end up with a bunch of scheduled tasks just to get back the shell whenever you start your computer. Day after day I spent the first hour getting back to where I had been the day before. And I just figured out at the end of the lab why some of my scheduled tasks were not working ...

I would be really interested to see how much time I spent connected to the lab. Probably it was around 200–250 hours in total, which I believe is more than I spent on OSCP and OSCE combined. But it was totally worth it. I really feel the power now that I learned so many useful things.

But if you consider that the price of the one-month lab is 20 GBP, it is still a very cheap option to practice your skills. 
* It is totally OK to do the lab in 6 months, in case you start as a beginner. That is still just 190 GBP for the months of lab access, and you will gain a lot of experience during this time. You will probably have a hard time reaching the point when you have a working shell, but it is OK. You can find every information on Google, you just need time, patience and willingness to get there.

Anyway, it is still an option not to aim to "get all the flags". Even just by getting the first two flags, you will gain significant experience in "getting a foothold". But for me, not getting all the flags was never an option.



If you are still unconvinced, check these other blog posts:

Or see what others wrote about RastaLabs.


Footnote


In case you start the lab, please, pretty please, follow the rules, and do not spoil the fun for others. Do not leave your tools around, do not keep shared drives open, do not leave FLAGs around. Leave the machine as it was. If you have to upload a file, put it in a folder others won't easily find. This is a necessary mindset when it comes to real-world red teaming. Don't forget to drop a party parrot into the chat whenever you or someone else gets a new flag. And don't forget:
OSCP has no power here. Cry harder!

I will probably keep my subscription to the lab and try new things, new post-exploit frameworks. I would like to thank @_rastamouse for this great experience, @superkojiman for the ROP challenge. Hackthebox for hosting the lab with excellent uptime.
As for @gentilkiwi and @harmj0y, these two guys probably advanced red-teaming more than everyone else combined together. pwntools from @gallopsled was also really helpful. And I will be forever grateful to Bradley from finance for his continuous support whenever I lost my shells.