Here are all the metrics, metrics, metrics in this strategy document:
Reduction in our time-to-detect adversary activity affecting federal agencies and critical infrastructure partners.
Reduction in the time-to-remediation across each identified intrusion.*
Reduction in impact of incidents affecting CISA stakeholders
Number of malicious domain requests blocked.
Percentage increase in agencies that have fully automated key vulnerability and asset management processes and can report advanced measurements such as time-to-remediate, scan frequency, and scan quality.
Percentage decrease in prevalence of, and time-to-remediate, vulnerabilities in all participating organizations and percentage increase in visibility across all sectors.
Increase in vulnerabilities identified via agency Vulnerability Disclosure Platforms prior to adversary exploitation.
Increase in eligible organizations enrolled in DotGov.
Number of potential threats detected by the CyberSentry capability prior to identification by participating entity
Reduction in the time-to-remediate Known Exploited Vulnerabilities across critical infrastructure and government networks.
Increase in percentage of recommendations from CISA’s vulnerability and risk assessments adopted by assessed organizations.
Reduction in the number of vulnerabilities disclosed without appropriate coordination or provision of necessary mitigations.
Increase in the volume of unique, timely, and relevant information shared by industry or government partners through our persistent collaboration channels.
Increase in specific actions codified in cyber defense plans adopted by industry and government
Increase in post-incident after-action reports demonstrating that actions developed in cyber defense plans reduced negative outcomes.
Increase in the percentage of recommendations in CISA’s guidance and directives that are directly based upon specific data showing how adversaries successfully execute intrusions and the most effective mitigations to stop them.
Increase in the average number of Cybersecurity Performance Goals effectively adopted by organizations across each critical infrastructure sector.
Where possible, reduction in confirmed impactful incidents in organizations that have adopted a higher number of Cybersecurity Performance Goals.
Increase in the number of organizations outside of the FCEB that have adopted applicable requirements in CISA directives.
Increase in the percentage of FCEB agency adoption of CISA directive requirements.
Increase in the number of technology providers that have published detailed threat models, describing what the creators are trying to protect and from whom.
Increase in the number of technology providers that have regularly and publicly attested to implementation of specific controls in the Secure Software Development Framework (SSDF).
Increase in the number of technology providers that have published a commitment to ensure that product CVE entries are correct and complete.
Increase in the number of technology providers that have published a secure-bydesign roadmap, including how the provider is making changes to their software development processes, measuring defect rates, and setting goals for improvement, and transitioning to memory-safe programming languages.
Increase in the number of technology providers that regularly publish securityrelevant statistics and trends, such as MFA adoption, use of unsafe legacy protocols, and the percentage of customers using unsupported product versions.
Help organizations safely use AI to advance cybersecurity.
Protect AI systems from adversarial manipulation or abuse, building upon NIST’s AI Risk Management Framework.
Protect critical infrastructure organizations from adversarial AI systems.
Publish evaluation of potential cryptographic vulnerabilities in critical infrastructure, particularly focused on ICS/OT systems.
As verifiably quantum-safe products enter the market, increase in migration to quantum-safe cryptography by Systemically Important Entities and FCEB agencies.
Increase in the number of cybersecurity students trained in courses offered or funded by CISA.
Increase in the percentage of cybersecurity courses offered or funded by CISA that target underrepresented populations.
Increase in the number of organizations provided with training and resources to deliver cybersecurity training
As these things go, these are pretty straightforward, relatively forward thinking, and mostly pragmatic. For context: if the USG was (say) Walmart, CISA would be one small arm of its corporate security team.
I am not suggesting the metrics are meaningless, I am suggesting everything else you need to do not covered under these or nist csf becomes lower priority or even seen as wasteful. In case you are not aware if it look up "mcnamara fallacy", a phenomena I have seen many times in corporate infosec, measurement itself is (wrongly) the goal instead of understanding and interpreting the measurement.
Check this:
> Number of malicious domain requests blocked.
Does this mean your domain reputation system sucks if that number goes down or does it mean you are cleaning up your assetts well? If you knew for certain they were malicious, and didn't block them, that is what it is meant to capture but in reality a downward linechart is all that is needed to fulfill the metric.
> Increase in the number of cybersecurity students trained in courses offered or funded by CISA.
> Increase in the percentage of cybersecurity courses offered or funded by CISA that target underrepresented populations.
The few "cyber security" grads I have seen start their career knew less than a helpdesk analyst. From well reputed colleges! Where is the qualitative metric?
> Reduction in our time-to-detect adversary activity affecting federal agencies and critical infrastructure partners.
Which adversaries? So, if defender remediates 10000 malware infection attempts within a day and you have 2 APTs with 90day+ dwell time, how does this work out? How does one's efforts to reduce APT dwell time fit in?
> Reduction in the time-to-remediation across each identified intrusion
This seems like a good idea on the surface but really, it should be time-to-containment. The IRL impact is, analysts will rush to remediate without properly analyzing and scoping the compromise. So long as the containment was effective and the eradication time is not unreasonable, who cares? The worst APTs are very hard to contain if you don't take your time to analyze their behavior, even containment is discourages in certain contexts to avoid tipping them off.
But stepping back a bit, I am with you that some of the metrics (especially around vulnmgmt) are solid. Lack of details, what isn't said and lack of emphasis on understanding are what make this harmful if taken as-is. For an org like the US government, I can see how this can be a good set of metrics for governmental departments and agencies, to enforce some reasonable level of security posture, expecting security teams to go beyond this and implement a much better set of goals and metrics according to their resources. But in the corporate world, this becomes the helm that drives the ship. You have one security org in the company and managers look bad when resources are spent doing things that don't help this metric and analysts "make" their numbers, unable to change deficiencies they see day to day that could help management understand the metrics (because that isn't the goal, the metric alone is the goal!).
Does this mean your domain reputation system sucks if that number goes down or does it mean you are cleaning up your assetts well? If you knew for certain they were malicious, and didn't block them, that is what it is meant to capture but in reality a downward linechart is all that is needed to fulfill the metric.
Per the document, I think they're referring to a particular DNS service they themselves operate.
I am guessing they have undisclosed private intel vendor telling them which domains are "malicious", or they are looking at compromises and auditing how malware domain is/isn't being blocked as part of their response.
Reduction in our time-to-detect adversary activity affecting federal agencies and critical infrastructure partners.
Reduction in the time-to-remediation across each identified intrusion.*
Reduction in impact of incidents affecting CISA stakeholders
Number of malicious domain requests blocked.
Percentage increase in agencies that have fully automated key vulnerability and asset management processes and can report advanced measurements such as time-to-remediate, scan frequency, and scan quality.
Percentage decrease in prevalence of, and time-to-remediate, vulnerabilities in all participating organizations and percentage increase in visibility across all sectors.
Increase in vulnerabilities identified via agency Vulnerability Disclosure Platforms prior to adversary exploitation.
Increase in eligible organizations enrolled in DotGov.
Number of potential threats detected by the CyberSentry capability prior to identification by participating entity
Reduction in the time-to-remediate Known Exploited Vulnerabilities across critical infrastructure and government networks.
Increase in percentage of recommendations from CISA’s vulnerability and risk assessments adopted by assessed organizations.
Reduction in the number of vulnerabilities disclosed without appropriate coordination or provision of necessary mitigations.
Increase in the volume of unique, timely, and relevant information shared by industry or government partners through our persistent collaboration channels.
Increase in specific actions codified in cyber defense plans adopted by industry and government
Increase in post-incident after-action reports demonstrating that actions developed in cyber defense plans reduced negative outcomes.
Increase in the percentage of recommendations in CISA’s guidance and directives that are directly based upon specific data showing how adversaries successfully execute intrusions and the most effective mitigations to stop them.
Increase in the average number of Cybersecurity Performance Goals effectively adopted by organizations across each critical infrastructure sector.
Where possible, reduction in confirmed impactful incidents in organizations that have adopted a higher number of Cybersecurity Performance Goals.
Increase in the number of organizations outside of the FCEB that have adopted applicable requirements in CISA directives.
Increase in the percentage of FCEB agency adoption of CISA directive requirements.
Increase in the number of technology providers that have published detailed threat models, describing what the creators are trying to protect and from whom.
Increase in the number of technology providers that have regularly and publicly attested to implementation of specific controls in the Secure Software Development Framework (SSDF).
Increase in the number of technology providers that have published a commitment to ensure that product CVE entries are correct and complete.
Increase in the number of technology providers that have published a secure-bydesign roadmap, including how the provider is making changes to their software development processes, measuring defect rates, and setting goals for improvement, and transitioning to memory-safe programming languages.
Increase in the number of technology providers that regularly publish securityrelevant statistics and trends, such as MFA adoption, use of unsafe legacy protocols, and the percentage of customers using unsupported product versions.
Help organizations safely use AI to advance cybersecurity.
Protect AI systems from adversarial manipulation or abuse, building upon NIST’s AI Risk Management Framework.
Protect critical infrastructure organizations from adversarial AI systems.
Publish evaluation of potential cryptographic vulnerabilities in critical infrastructure, particularly focused on ICS/OT systems.
As verifiably quantum-safe products enter the market, increase in migration to quantum-safe cryptography by Systemically Important Entities and FCEB agencies.
Increase in the number of cybersecurity students trained in courses offered or funded by CISA.
Increase in the percentage of cybersecurity courses offered or funded by CISA that target underrepresented populations.
Increase in the number of organizations provided with training and resources to deliver cybersecurity training
As these things go, these are pretty straightforward, relatively forward thinking, and mostly pragmatic. For context: if the USG was (say) Walmart, CISA would be one small arm of its corporate security team.