Table of Contents

• Executive Summary
• Introduction
• Origin Story
• Standardized Capability Maturity Model
• Addressing Challenges of the Existing Model
• Challenges Encountered While Drafting the New Model
• Maturity Model Overview
  • Assumptions
  • Structure
  • Usage Tips
• Contributors and Contributing

Executive Summary

Cutting to the chase, this blog discusses a new Capability Maturity Model to help guide and report on your Red Team’s maturity. As leadership, you can enable the Red Team’s maturation, or reap larger benefits, in a few key areas. We believe those to be the following: Processes

• Operational Planning – The Red Team can target the organizations objectives and major muscle movements to provide key data, but it can’t do that if it’s not aware of these things.
• Operational Approval – Want more data? Learn to trust your Red Team (earned with evidence, not merely given) and reduce the approval overhead needed to execute operations.
• Resource Management – Your Red Team will need weird stuff. Help provide the top-cover needed to expedite potentially lengthy processes or options to work outside of the organization’s normal channels.

Technology

• Test Environment – A representative test environment that can easily be modified without affecting Production can greatly augment the data your Red Team creates. This takes pulling together the right people and resources, which benefits from top-level support.

People

• Relationships – Encourage your other leaders, and yourself, to understand and use the Red Team. Everyone needs data to make the right decisions, and Red Team help provide that if they understand those needs.
• Training and Skill Development – The Red Team needs to have strong skillsets in many different technologies that are fields in their own right. It’s unrealistic to believe a mature Red Team can build and maintain these skills all in their free time. Providing time and resources to research or stay on top of current practices will yield more realistic threat simulations.

Program

• Strategy – As your Red Team matures and you build trust, think about the Red Team and its strategy when formulating your own strategy. There may be areas where the team can help make a difficult decision a little clearer.
• Giving Back – The security community has a wealth of knowledge to draw from, but that only happens when the members of said community are freed/enabled to contribute to it. Consider ways to allow your team to speak at conferences or make development efforts open-source so other teams can mature.

Introduction

Red teams provide pivotal data for organizations to understand how their security posture would withstand an attack from a motivated adversary. But how does the organization know the data generated by their Red Team is as effective as it could be? How does the Red Team itself understand where it is relative to other threat actors and where it needs to go, and how does it communicate that status to leadership to build further trust and confidence? For many fields, a Capability Maturity Model (CMM) has been one answer to address these questions. Starting in the software engineering field, a CMM describes behaviors and processes across five levels of maturity, moving from unplanned/ad hoc activities through defined and documented practices and ultimately practices that are consistently being tracked and updated with lessons learned. That sounds good, so how can a Red Team get involved, you ask? Great question! The problem, as we’ll discuss in a moment, is that the Red Team field lacked a standardized CMM that teams could use for these purposes. The next few sections provide context on the why and how this CMM came to be. If you want to skip straight to information on the model and considerations for using it, click here. If you want to go to the model itself, click here.

An Origin Story

As we set out on our own internal measurement and planning, we came across a published CMM from some folks from Norton Life Lock and Bishop Fox (you can find it here). The CMM consists of four categories (Processes, Technology, People, and Program) in keeping with other CMMs, but diverges by only addressing three levels of maturity. Through later discussions with one of the authors, they shared that they decided to focus on simplicity with their CMM to enable more use and avoid some of the inherent ambiguity of our field. Their offering filled a critical need where no other option existed. The authors captured many key areas that demonstrate Red Team maturity and presented it in a compact and easy-to-digest format that was able to get us started. We definitely want to thank Jordan and Noah Potti and Trevin Edgeworth for putting a stake in the ground when none existed, despite some of the challenges we’re about to discuss! As we began to the use the simplified CMM, we ran into two primary issues due to its structure:

1. It didn’t provide us with a common language because of its truncated levels. Saying we wanted to be level 3 in the simplified CMM would be the highest level of that model, but other teams using a five-point scale would assume a lower level of maturity.
2. The simplification (consolidation) of subjects led to lack of consistency across levels. Some elements of maturity may appear in only one or two levels, rather than all three, and some of the subjects combined a wide array of fairly disparate topics. That led to us targeting individual bullets rather than consistently targeting a specific level in a given subject.

As we headed into our internal summit, we decided to take on creating a standard-format CMM as one of our summit tasks with two purposes in mind: give ourselves more of what we needed to plan and report on our team’s health and create a resource that other members of the community who may run into the same issues could use.

And in This Corner, A Standardized CMM

After drafting the CMM during our summit, we sought feedback from the community to ensure we didn’t fall prey to biases of prior experience. After four, 2-hour sessions of excellent conversation with three industry partners (see the contributors section below) who brought a wealth of different experiences to the table, we ended with a CMM that contains four categories (Processes, Technology, People, and Program) and 27 subjects spread across the categories. If you want to skip straight to information on the model and considerations for using it, click here. And as a reminder, if you want to go to the model itself, click here. Otherwise, let’s talk nerdy for a moment on what we did, how we did it, and what challenges we faced.

It's Not You, It's Me

Our first considerations were to address the challenges we faced with the simplified CMM. To provide ourselves with a “common language” to use with leadership and other teams, we expanded the CMM to the traditional five levels: 1 – Initial 2 – Repeatable 3 – Defined 4 – Managed (Capable) 5 – Optimizing This change allows users of the model to accurately report on current status or future plans without needing a translation layer. To address the issue of simplified categories, our draft started by inventorying the key pillars of Red Team work that we believed needed to have the full spectrum of maturity levels. To do this, we promoted some of the individual items from the simplified CMM to full subjects and filled remaining gaps with new offerings. As an example, the simplified CMM has a subject for “Team” that includes things like operational capability and work management; both of those became subjects of their own in the standardized CMM. Last, to keep consistency through drafting, we aligned descriptors for the different levels to other known CMM models. These descriptors are high-level phrases that generally describe behavior at a given level; things like “documented” or “automated”.

Nothing Worth Doing is Easy

Obviously that figure of speech isn’t true, but we did face our own, new challenges during drafting. As it pertains to the level descriptors, while we wanted to keep standardization with other known and accepted models, we quickly found that the list did not account for everything we needed to describe Red Team maturity, such as operational security considerations. So, we added additional descriptors that kept with the spirit of the levels but provided us more wiggle room. These descriptors are critical to ensuring consistency across the 27 subjects; we’ll cover them in the section on the CMM itself a little later. Another typical feature of CMMs is that they are “additive” in nature, meaning that as you progress from one level to the next, you’re adding capability to your existing toolset and generally keep doing the lower level in addition to the new stuff. That doesn’t apply to everything we wanted to cover, particularly in the Technology category where some “level ups” replace the lower level. In these cases, what we really needed was a sliding scale of maturity rather than an additive model, so some of the subjects follow this flow rather than the traditional additive approach. Our biggest challenge, though, and one that still warrants review by more eyes over time, was keeping the number of subjects reasonable while not falling into the same pit of consolidation we were trying to avoid. There is undoubtedly overlap that could potentially lead to consolidation in the subjects that made the cut, but our core number of subjects did not fluctuate much beyond the 25-28 range. One of our guiding principles for this endeavor was if a subject did not have more than three relatively obvious levels of maturity, it might be a candidate for consolidation if there was something close to it.

The Capability Maturity Model

Assumptions

Before digging in, there are a few assumptions to be aware of to best understand and utilize the material. Assumption: This CMM is currently written for internal Red Teams. Consultancies would need to modify some of the subjects and maturity levels to match their situation. For example, a consultancy doesn’t need to foster long-term relationships with various defensive or legal teams, but they would need to understand and plan for their maturity of maintaining client relations overall. Assumption: This CMM presumes you have staffed your Red Team with dedicated operators. If an organization only has a general Offensive Security team or perhaps a Red Team manager who coordinates third-party Red Team assessments, this CMM doesn’t apply. Assumption: Generally speaking, except for levels that describe a negative or some of the technology subjects noted below, the Red Team must exhibit the prior level’s behavior before it can progress to the next level. This is to keep with the additive nature of other CMMs as much as possible. In other words, you can’t skip over defining a process simply because you have metrics for it. The model also includes some definitions, such as potentially foreign terms and the descriptions of each level. Those can be found on the model page.

The Structure

The model is broken into 4 categories and 27 subjects spanning elements that affect Red Team maturity. The model itself can be found here, this blog will only discuss it from a mid-level view and provide context that may help shape its usage, such as the intent of categories or particular things to be aware of when using a particular category or level within it. Processes Processes is one of the largest categories in the CMM, containing 9 of the subjects. The subjects within this category focus on a broad range of procedural elements that improve the team’s maturity, from maintaining a knowledge base to selecting and planning for operational targets. The subjects:

• Continuous Improvement: How the Red Team iteratively improves itself through better planning and self-reflection.
• Knowledgebase: How the Red Team captures and maintains its knowledge so that it avoids recreating the wheel and can onboard new members quickly.
• Work Management: We don’t want to say Agile because you should do what makes sense to you, but think things like Agile. Whatever processes that best fit how your team tracks and executes work in a transparent and organized way.
• Operational Planning and Selection: How much forethought the Red Team puts into planning operations and from what sources it draws operation objectives.
• Operation Approvals: This was a tricky one, ultimately leading to a forked path where the paths join at level 3 and continue together from there. On the one side, there are organizations with very little leadership engagement (by choice or ignorance) where teams are operating without much approval. This fork follows a bell-curve, first gaining leadership engagement before later reducing the amount of approval needed because trust is built. On the other side, there are organizations where the Red Team has not built trust with leadership and every little thing the team does needs to be approved. This follows a declined slope, ultimately landing in the same place of relaxed approvals due to trust.
• Operational Documentation: How the team tracks its artifacts and actions to share with defensive teams after an operation.
• Operation Reporting: How the Red Team engages with the appropriate partners to disseminate the results of its operations.
• Configuration Management: How the Red Team manages artifacts it generates, from reports to tooling and infrastructure.
• Resource Management: How the Red Team or organization manages resources like licenses or accounts. Note: For some organizations, tracking of things like licenses may fall outside the Red Team. For others, it’s all in-house. We tried to keep this vague enough that whatever your internal situation is, you can still progress along the levels.

Technology While we believe red teaming is a mindset, in practice it does require tools. This category discusses how those tools are built and maintained, and how effective they are at meeting the team’s needs. The subjects:

• Tooling: Broadly encompasses any toolsets the team uses to accomplish its objectives. Tools can include commercial, open source, or in-house. Note: This is an area where a sliding scale is used rather than explicit “additive” maturity, as discussed in a prior section.
• Infrastructure: Deployment and configuration of the team’s infrastructure, growing in automation and operational security considerations as the team progresses. Note: This is an area where a sliding scale is used rather than explicit “additive” maturity, as discussed in a prior section.
• Test Environment: The Red Team’s capability to develop and test its tools or other test cases in a representative environment.

People The People category includes relationships with key stakeholders outside the Red Team as well as the team’s internal capacity to accomplish its tasks. For consultancies, the People category is likely the area to focus most of your modification efforts. P.S. – if someone does take that on, we’d love to hear about it and perhaps offer an A/B version. The subjects:

• Relationships with Responders: Traditional responders like SOC/IR, but also Physical security if your team runs physical operations
• Relationships with Engineering Teams: Enterprise, endpoint, detection engineers, etc.
• Relationship with Cyber Threat Intelligence
• Relationship with Legal
• Relationship with Human Resources: Note: From discussions, this category differs a little from other key stakeholders. To summarize the conversations, HR can be a strong enabling partner for certain circumstances and building this relationship can aid the Red Team in a similar fashion to Legal, but generally HR isn’t considered an operational stakeholder unless they were consulted during the operation.
• Relationship with Leadership: Note: Who constitutes “Leadership” will differ per organization. We offer security leadership, IT leadership, and various segment leaders as key contenders.
• Knowledge of Business and Technical Environment: How much knowledge the Red Team has about the organization’s objectives and technology stacks, leading to better operational selection and reporting.
• Operational Capability: The Red Team’s personnel skillsets and knowledge to enable robust operations in the face of a more mature defensive posture.
• Development Capability: The team’s employed capacity to modify and develop tools to meet operational needs. Note: Through discussions, we decided to focus on “employed” to provide tangible evidence of a given maturity level. Simply saying that the team can do something isn’t measurable. To offset this focus, we do provide a caveat of “operationally relevant” for more advanced development like custom implants or C2 frameworks.
• Training and Skill Development: Focused on the availability of time and resources to pursue appropriate training that meets personal and team gaps.

Program Program focuses on the elements that make the Red Team part of the business rather than a mere collection of folks doing hackery things. The subjects:

• Red Team Product Lines: Focused on how the Red Team is a distinct member of the Offensive Security function overall and how it provides meaningful data to the organization. Note: This will look different for every team, both in nomenclature for the activities and what activities are needed at any given point in the organization’s defensive maturity.
• Strategy
• Metrics: Note: This section is intentionally descriptive rather than prescriptive. Red Team metrics can be difficult and will be highly dependent on the organization’s priorities.
• Knowledge Sharing: How the Red Team contributes knowledge within the organization and back to the greater security community. We consume so much of others’ work; a mature team will help newer teams in the same way.

Usage Tips

Planning If this is your first time using a CMM, there are two things you should know about planning:

• You don’t have to target the next level up as your goal. Except for those situations noted in the pre-amble sections, we tried diligently to keep this CMM’s usage in step with other CMMs. What that means is generally you still need to meet the skipped level’s characteristics, but otherwise aim high.
• You don’t have to get to level five in everything. Don’t have enough development to merit needing CI/CD in your development process? All good. This is for you and your priorities. Just make sure you communicate that to the right stakeholders.

Reporting Teams can report on status and progress in three ways: a specific subject’s maturity level (goal or current); a category’s aggregate maturity level (goal or current); overall aggregate maturity level (goal or current) For category and overall scores, the score is an average of the contained subjects. If a team does not meet level one in any subject, it is still counted towards the average (i.e., in the total count of applicable subjects with a score of zero for that subject). Some teams may wish to weight certain categories or subjects heavier in their scoring. We provide no such recommendations or capability in the CMM spreadsheet as that will be highly specific to each team, but we’d love to hear how different teams have elected to do that. Modifications We tried to make this generalized, but we know there’s a broad range of needs that the Red Teams will need to fill for their respective organizations. First off, we welcome feedback. Please use our GitHub page to submit modifications that you think would be broadly useful to other teams, because this is only V1 we know things aren’t perfect. Secondly, if you plan to modify this CMM for your internal use, please read the challenges section above to avoid pitfalls that are waiting for you. The biggest item here is to ensure you stick to a common set of level descriptors so you keep consistency across levels, otherwise it is very easy (trust us) to accidentally create hard-mode and easy-mode subjects.

Contributors and Contributing

We want to offer a big thanks to the internal and external reviewers and contributors to version one: Andy Grant (Head of Offensive Security, Zoom), Johann Rehberger (Red Team Director, Electronic Arts), Matthew Bjornstad (Principal Security Engineer, Red Team, Charter Communications), Josh Huff (Red Team, Humana) Willa Riggins (Associate Director, Pentesting, Humana), Tim Straub (Threat and Vulnerability Management, Humana), Dion Baccus (Red Team Product Owner, Humana) Note: Individuals participated in their own right and none of the material should be considered indicative of their companies’ stance. If you would like to contribute: Please visit our GitHub page to review our contribution guidance submit suggestions and comments. We plan to update this model periodically and want it to be broadly applicable to the community.