Blog Article

  • Home /
  • News /
  • Beyond Worthless Metrics

Beyond Worthless Metrics...
Adding Value to Your Information Security Program Part II

Written by Shohn Trojacek

Tuesday 19 September 2017

In my "Beyond Worthless Metrics...Part I" blog , I highlighted a key problem that many IT teams and leaders face-- devoting the proper time and resources to their security programs. Often a Security Program will become focused on paperwork instead of providing useful information and communicating essential security needs to the Executive Team who hold the purse strings.

The goal is to move beyond bureaucracy and paperwork and transition into real metrics that effectively measure performance, anticipate trends, and use such metrics to optimize the security spend on where it matters the most.

How do we get there?

3 Things to Consider:

1. Know Yourself.
It is cliché for information security professionals to cite Sun Tzu’s, “Art of War” when emphasizing the importance of knowing one’s enemy, but what is often left out is the ancient adage to “Knowing Yourself”.

The first steps in Knowing Yourself involve understanding the market and regulatory environment in which your organization operates, as well as business goals:

On one hand, your organization may be very regulatory-driven and could be faced with grave legal or regulatory consequences. On the other hand, you may have security needs based on your type of business (i.e. medical, financial, etc.). Disclosure of client data could prove completely undermine public or customer trust in your organization.

Defining your business needs, regulations and drivers can help when considering how to strategically protect your company. Questions to ask include:
  • Is confidentiality more important, or availability?
  • If one is confronted with budget decisions, which is more important?
  • For a specific example, should you transition your Exchange environment to the cloud, to improve availability, or would that risk exposing company information?
  • How might changes to the Security Program impact organizational culture, or are their ways to integrate with existing culture?
The answers to these questions will be driven by a solid understanding of your information security needs AND what you have, again – Know Yourself! Take the extra step to understand what you have:
  • Maintain inventories of business processes, applications, data, supporting systems, and their connectivity;
  • Maintain baselines of normal system performance characteristics including basics such as memory utilization and normal process execution; and
  • Maintain an understanding of your organization’s information security mentality, and more importantly – awareness.
2. Know Your Enemy.
Knowing yourself through your enemy really means knowing of yourself, and how your enemy may impact you. Understanding the enemy is relatively straight-forward and could include:
  • Maintain an ongoing understanding of threat trends and likely future scenarios;
  • Maintain an understanding of how threat trends might translate into real risks to your organization; and
  • Maintain drills and exercises similar to what the enemy may do and include in your metrics and disaster recovery program.
3. Know Your Friends.
We have already quoted ancient wisdom, so why not add some more? There is an ancient Mexican proverb which states: “Show me who your friends are, and I will tell you who you are.”

Once you have begun to Know Yourself and your Enemy, you would want to communicate with everyone in your organization, but there are often language barriers to overcome – each department has its own drivers and needs in the same way that the organization as a whole does.

Useful metrics for the Executive team may be completely useless for others. Questions to ask include how the Security Program can be of value to the various departments within an organization and since security often touches all elements of an organization, it will be critical to get more than one view to understand what is truly important to the organization.

Candidly, Information Security professionals may sound to outsiders more like astrologers or soothsayers – engrossed in a language of our own, and despite possessing insights, when such insights are shrouded in mystery they would naturally cause distrust and resentment. To outsiders, information security presentations may sound something like this: “We need the new Flux Capacitor 1000 because the old one no longer connects with the cross-fibulator”. In response, purse-string holders may reluctantly approve of additional expenditure, all the while becoming resentful of what seems like an ever increasing demand for “new toys”.

Commun-icating means building community within your organization and, by proxy, support for information security in a way that most everyone will want to support. This means sharing things that are of value to the community. Consolidated metrics and reports MUST be combined with story-telling that your audience can connect with. Even with the most robust data capture, the information must be distilled to find patterns and ask questions that weren’t previously thought of and this process should be of value to the organization. Some metrics initially adopted may prove worthless, and yet new ones not thought of, unique to your organization may emerge – providing true value for guiding decision making.

For an example of story-telling, consider this narrative: we noticed that Internet ads were increasingly the source of the uptick in virus infection over the last year, resulting in 25% increase in help desk calls. We implemented a software package to block Internet ads, which has contributed to a significant decline in help desk calls.

Similarly, reach out and build community within the organization through story telling bulletins that aren’t perceived as a waste of time. This would require understanding your audience – what are their pain points?

Outside of IT, information security can often be viewed as a bottomless money pit because the nature of the evolution is not well understood. In some cases, a simple quarterly threat trends presentation can be helpful for briefing Executives on changes in the threat paradigm and the IT environment.
  • Information needs to be fresh and interesting;
  • Highlight progress or stalled efforts;
  • Include estimates on when evolving threats will become more pertinent or mitigated within an organization along with cost savings or recouped hours or avoided rework.
The true value of the information security program is in anticipating the movements of the bad actors with enough lead time to map budget to the risks accordingly. Within the user community, create threat bulletins that aren’t annoying and repetitive. These presentations can include recommendations to either: accept that risk, in other cases it may be recommendations to increase headcount or budget. These presentations need to balance the reality of business needs against security risk. If assumptions about business needs are in error, that is up to the business to correct.

4. Know What Matters.
Consider this quote: “Objectivity in history writing is impossible, because history can only exist as a narrative and every narrative has to have a narrator” taken from “Like a Baird in a Cage: The Invasion of Sennacherib in 701BCE”. At the end of the day, the “big breach” is coming for all of us connected to information security. We must look from that day backwards, to what we are doing now, and ask: on that day, “What will truly matter?”

Each of us makes the best decisions with what is known at the time and people understand that mistakes will be made. However, once that breach happens, if it appears that there was a lack of effort, failure on our part to stand up and do our jobs – to say the difficult things, then paradoxically, we will not be so easily forgiven.

Equifax recently suffered a major breach. The CEO of Equifax was grilled before Congress and has effectively assigned blame to one individual for not applying a security patch. That was his narrative. I suppose that he felt he had to say that or face some legal repercussions, but I fear his assertion was far from reality, and the real problem was more fundamental.

The narrative sounds reminiscent of the focus on the lone gunman part of the assassination of JFK. The headlines didn’t read: “President gets himself assassinated by driving around in a convertible”. In the case of Equifax – perhaps connecting millions of records to the Internet was the real problem, or in other hypothetical cases – connecting that wire transfer terminal to the Internet, or that process control network to the Internet. But no one will say that, will they? In the world of information security, we are tasked with keeping people safe, so we must speak up when necessary.

Often the difficult questions boil down to convenience and profits vs. the amount of risk an organization is willing to take on. The organization’s mission pays for the whole security party, lest we forget, yet we are tasked with protecting the organization’s mission. As security practitioners, we say fancy terms such as defense-in-depth, make suggestions on new toys to purchase, and beg for an ever-increasing budget – but are we missing some fundamentals? To borrow a football metaphor, are we focusing on some fancy new offense, all the while neglecting blocking and tackling? I think most of us are. The amount of risk that organization is taking on needs to be made plain.

Consider: cybersecurity used to not matter – it was an after-thought in most organizations, but times are changing. Since heads are rolling, such as the CEO of Equifax, now it matters. But, perhaps it taking this long is more a measure of our ability to communicate and prioritize. We must learn to discern between toys and promises of “blue sky” vs. facing or communicating the candid reality that there will be a risk of breach for almost anything connected to the Internet, and to present the risks in a manner that the organization can deal with and process.

IT Directors and CSOs need proper instrumentation so they can make help the organization they are a part of make sound decisions. In the Equifax case, they are likely driven by competition against the other credit bureaus. Once one firm made the decision to connect these records to the Internet, the others had to as well, if they were to survive, but perhaps the real problem was that the costs of doing so weren’t properly weighed. This could be analogous to companies in the 1970s failing to account for potential lawsuits from polluting the environment.

Oh where shall we begin on a practical level? For starts, the SANS Top 20 list provides a solid starting point, and more importantly, metrics, but who actually uses those metrics? Yet, let’s not make the mistake of failing to think - simply checking the box. The metrics need to be tailored to the organization each of us are a part of.

Some example technical metrics include: how many days from security patch release to deployment on average? Can we slice that into different dimensions? What about for Java vs. Microsoft patches? How about by Business Unit or application to find deeper root causes? How many days from security patch release to deployment for Critical, High, or Medium Risk patches? Are we wasting time on Low risk items? How many risk acceptances are registered with no plans to address? How much time passes before an “unauthorized device” connected to the company network would be detected?

Getting to these metrics requires collecting data. We suggest leveraging modern log consolidation software and reporting engines to create consolidated Executive dashboard views of key metrics from a variety of tools; however, there are ways to get things done without fancy new toys. Consider that a few police officers equipped with radar guns can keep the speeding under control in most cities.

For those without sufficient budget for tooling, use what you already have, starting with Excel. You don’t have time to check everything? Then do random spot checks and note in Excel and indicate the amount of budget or time necessary to improve.

5. Know What You Don't Know.
While he wasn’t the first person to say it, Donald Rumsfeld reportedly said:

“There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don't know. But there are also unknown unknowns. There are things we don't know we don't know.”

As security practitioners, we must be forthright in what data we don’t have. This is doing our job! If your assessments are based on assumptions derived from poor data, then say that! Businesses deal with incomplete data all the time.

Look at the SANS Top 20 list, and map out what is missing and what is most cost-effective to collect.. It is important to be candid about priority – “we don’t have that data today, but we could, if ______”.

What are my options?

You can start by pulling together all of the requirements and business needs associated with your organization. Next, consider the metrics that you have been providing to your Executives. Are they helping further your case about taking security more seriously? Or have your Executives still been left scratching their heads after they see your reports and not quite “getting it?”

This may all seem overwhelming, so consider whether you’d like us to lend a hand. DYONYX has been working with federal and state agencies, and energy, financial, healthcare, and other organizations to help organizations make the most of their security spend. We can help in at least two areas:

1. Gap Assessment & Instrumentation Refresh: We can work with your team by taking a look at your existing toolset and create new dashboards that help communicate and measure the right information for “worthwhile metrics.” We can also make recommendations on other tools that help fill in the gap.

From analyzing the types of controls you are using to who has access to everything, a.k.a. your “god accounts”, we can help put these numbers into a format where you and your Executives can see changes and improvements month-to-month. This will enable you and your team to be able to see if your organization is where it needs to be or at least moving in the right (and hopefully not wrong!) direction.

2. Beyond the Metrics- Translating into a Common Tongue: In addition to helping put together real, measurable statistics that communicate meaningful information and progress, we can help align the IT and business strategic visions on more of an organizational/tactical level. In other words, we help put the “big picture” into perspective and bridge the gap by translating the different voices on the business and technology sides into a common tongue.

A Quick Takeaway:

Using the proper security instrumentation and measurements are truly vital to the success of your company. The brunt of the attacks are coming through spear phishing. If it were my budget, I would focus on not only instrumentation, but measuring how well my employees are doing with respect to not clicking on things they ought to not click on.

Once you have this data in front of you, it needs to be translated into an actionable plan that includes experiential employee training and follow-up so that each individual at your company knows and understands the important role they play in keeping your company safe.

DYONYX LP, is an award winning IT total solutions provider dedicated to helping clients across the globe solve multiple problems through one delivery system. Consulting services include, security assessments, IT assessments, and O365 Migration services. Managed Services include hosting, hybrid Cloud Services, ITILv3-aligned Service Desk (Help Desk), managed network and security, disaster recovery, and Security as a Service. Learn more at

All posts

About Us

DYONYX is an innovative, award-winning IT solutions provider dedicated to helping you solve multiple problems through one delivery system. Our aim is to improve your productivity and security while reducing your cost and risk.

Contact Us

13430 Northwest Freeway, Suite 1000
Houston, TX 77040