Embrace The Red

Survivorship bias is an interesting thing that we can observe nearly daily. It is the success stories of exponentially growing startups, the motivational speaker who shares their insights on how to be successful and so forth. What is it exactly? What is survivorship bias? Wikipedia defines it as “…the logical error of concentrating on the people or things that made it past some selection process and overlooking those that did not, typically because of their lack of visibility.

Security metrics are an interesting topic. Over the years I used “scores” as a tool to identify and shine light on problematic areas or highlight lack of engineering and security quality of certain teams. A security score should not seen as an objective or absoulte measure, but it allows to compare systems with each other at a relative scale, and by sharing the score it makes people ask questions.

Today FireEye shared that they were victim of a cyberattack and internal red teaming tooling was accessed by adversaries. More details in this NYT article. This reminded me that I wanted to do a post on actively protecting pen testers and pen testing assets for a while. Against persistent adversaries it is only a matter of time when they succeed, not if they will succeed. The big question is do you know when an adversary starts poking around, and when they succeed?

What a journey it has been. I wrote quite a bit about machine learning from a red teaming/security testing perspective this year. It was brought to my attention to provide a conveninent “index page” with all Husky AI and related blog posts. Here it is. Machine Learning Basics and Building Husky AI Getting the hang of machine learning The machine learning pipeline and attacks Husky AI: Building a machine learning system MLOps - Operationalizing the machine learning model Threat Modeling and Strategies Threat modeling a machine learning system Grayhat Red Team Village Video: Building and breaking a machine learning system Assume Bias and Responsible AI Practical Attacks and Defenses Brute forcing images to find incorrect predictions Smart brute forcing Perturbations to misclassify existing images Adversarial Robustness Toolbox Basics Image Scaling Attacks Stealing a model file: Attacker gains read access to the model Backdooring models: Attacker modifies persisted model file Repudiation Threat and Auditing: Catching modifications and unauthorized access Attacker modifies Jupyter Notebook file to insert a backdoor CVE 2020-16977: VS Code Python Extension Remote Code Execution Using Generative Adversarial Networks (GANs) to create fake husky images Using Microsoft Counterfit to create adversarial examples Backdooring Pickle Files Backdooring Keras Model Files and How to Detect It Miscellaneous Participating in the Microsoft Machine Learning Security Evasion Competition - Bypassing malware models by signing binaries Husky AI Github Repo Conclusion As you can see there are many machine learning specific attacks, but also a lot of “typical” red teaming techniques that put AI/ML systems at risk.

In this post we will explore Generative Adversarial Networks (GANs) to create fake husky images. The goal is, of course, to have “Husky AI” misclassify them as real huskies. If you want to learn more about Husky AI visit the Overview post. Generative Adversarial Networks One of the attacks I wanted to investigate for a while was the creation of fake images to trick Husky AI. The best approach seemed by using Generative Adversarial Networks (GANs).

There are plenty of examples of artificial intelligence and machine learning systems that made it into the news because of biased predictions and failures. Here are a few examples on AI/ML gone wrong: Amazon had an AI recruiting tool which favored men over women for technical jobs The Microsoft chat bot named “Tay” which turned racist and sexist rather quickly A doctor at the Jupiter Hospital in Florida referred to IBM’s AI system for helping recommend cancer treatments as “a piece of sh*t” Facebook’s AI got someone arrested for incorrectly translating text The list of AI failures goes on…

You might have heard about “NAT Slipstreaming” by Samy Kamkar. It’s an amazing technique that allows punching a hole in your routers firewall by just visiting a website. The attack depends on the router having the Application Layer Gateway enabled. This gateway can be used by anyone inside your network to open a firewall port (totally by design). Protocols such as SIP (Session Initiation Protocol) use it. What I will focus on in this post is the Application Layer Gateway (ALG) and SIP.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. Overview: How Husky AI was built, threat modeled and operationalized Attacks: The attacks I want to investigate, learn about, and try out In this post we are going to look at the “Repudiation Threat”, which is one of the threats often overlooked when performing threat modeling, and maybe something you would not even expect in a series about machine learning.

My GrayHat Red Team Village talk “Learning by doing: Building and breaking a machine learning system” is now live on YouTube. Check it out: https://www.youtube.com/watch?v=-SV80sIBhqY and smash the Like button! :D Question? I thought of turning the content into a hands-on workshop. Let me know if that would be something that would you would attend? Trying to see if there is interest. Cheers, Johann Twitter: @wunderwuzzi23

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. Overview: How Husky AI was built, threat modeled and operationalized Attacks: Some of the attacks I want to investigate, learn about, and try out A few weeks ago while preparing demos for my GrayHat 2020 - Red Team Village presentation I ran across “Image Scaling Attacks” in Adversarial Preprocessing: Understanding and Preventing Image-Scaling Attacks in Machine Learning by Erwin Quiring, et al.

A few years back the Blue Team of a company asked to be targeted in a Red Team Operation. That was a really fun, because Rules of Engagement commonly prevent targeting Blue Teams. Blue’s infrastructure, systems and team members are often out of scope, unfortunately. Blue team infrastructure is a gold mine for credentials, recon but also for remote code execution! Often companies do not have adequate protection, procedures (MFA, multi-person attestation), monitoring and auditing in place when it comes to accessing data from endpoint agents.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. Overview: How Husky AI was built, threat modeled and operationalized Attacks: Some of the attacks I want to investigate, learn about, and try out I wanted to explore the “Adversarial Robustness Toolbox” (ART) for a while to understand how it can be used to create adversarial examples for Husky AI.

For GrayHat 2020 I was asked to create a short intro video for my Red Team Village talk “Learning by doing: Building and breaking a machine learning system”. So I put my green screen to good use and recorded this short clip for Red Team Village. Here is the link to the clip on Twitter: Hope you like it. :) The talk will be October, 31st 2020. Schedule: http://redteamvillage.io/schedule

There is a lot of discussion around terms such as red team, attack team, pentest, adversarial engineering or offensive security team and similar ones. I typically stay away from the (sometimes passionate) discussions that ensue whenever this topic comes up. Personally, I think a good strategy is to define programs and teams who operate in this space by what services the team (or teams) provide(s) to the organization. The business groups, blue team, developers, engineers, employees and clients are the customers.

While building “Husky AI” I started working a lot with Microsoft’s VS Code Python extension. It is a super convinient way to edit Jupyter Notebooks. I just use VS Code’s Remote SSH feature to get to my Linux host and work on modeling and testing there. When threat modeling “Husky AI” I identified backdooring of third party libraries and development tools as a potential issue to be aware of. So finding security issues in the tools is naturally something I am keeping an eye on.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. Overview: How Husky AI was built, threat modeled and operationalized Attacks: The attacks I want to investigate, learn about, and try out We talked about creating adversarial examples and “backdoor images” for Husky AI before. One thing that we noticed was that an adversary with model access can very efficiently come up with adversarial examples.

Excited to announce that I will be presenting at Grayhat - Red Team Village on October 31st 2020. The presentation is about my machine learning journey and how to build and break a machine learning system. If you follow my blog, you can guess that there will be lots of discussion around “Husky AI”. The bits and pieces that make up a machine learning pipeline, and how to threat model such a system.

This was also presented at BSides Singapore 2020. The slides are here and YouTube link is here. The origins of the Shadowbunny A few years ago, around 2016, I went on a relaxing two weeklong vacation. It was great to disconnect from work. I traveled to Austria, enjoying hiking in the mountains, and exploring Vienna. When I came back to the office, the team had placed a giant bunny teddy into my chair.

This year one of my goals was to learn about machine learning and artificial intelligence. I wrote about my journey before - including what classes I took and books I read, the models and systems I built and operationalized, threat modeling it to learn about practical attacks and defenses. My goal is to be knowledge enough in the AI/ML space enough to be able to help bridge the gap between research and operational red teaming - by doing practical things with life systems.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. Overview: How Husky AI was built, threat modeled and operationalized Attacks: The attacks I want to investigate, learn about, and try out Mitigations: Ways to prevent and detect the backdooring threat During threat modeling we identified that an adversary might tamper with model files. From a technical point of view this means an adversary gained access to the model file used in production and is able overwrite it.