I’ve been talking about vulnerability management with a colleague who works in that field. Specifically, we’ve been talking about evaluating the effectiveness of vulnerability remediation and how to measure it. This ended up focusing on Mean Time to Remediate (MTTR), which is a popular metric you can find recommended many places on the internet. However, as we talked about it and I began thinking about it in more depth, I had some questions and concerns that didn’t really seem to be brought up or addressed elsewhere. Admittedly, I might not have looked very hard, but I still wanted to document my thinking.

What is Mean Time to Remediate?

The Mean Time to Remediate (MTTR) is the average Time to Remediate (TTR) for some collection of vulnerabilities. The TTR for a vulnerability is just the number of days between when the vulnerability was opened (i.e., discovered) and when it is closed (i.e., fixed). In fancy mathematical notation, it would look something like this:

$$\frac{1}{n}\sum_{1}^{n} (V_i^c - V_i^o )$$

where $n$ is the number of vulnerabilities, $V_i^c$ is the close date of the $i$ -th vulnerability, and $V_i^o$ is the open date of the $i$ -th vulnerability.

As an example, if we have three vulnerabilities that were closed after 4, 9, and 3 days, the MTTR will be $\frac{4 + 9 + 3}{3} \approx 5.3$ days. So, the MTTR for these vulnerability is just a bit over 5 days.

However, there are certain choices that go into calculating the MTTR that can affect the value and what it actually represents. Without understanding these choices and limitations, it seems that this metric could be inaccurate or deceptive, if done purposefully.

Open Vulnerabilities

As mentioned above, to calculate the MTTR, you need the TTRs for a collection of vulnerabilities. For open vulnerabilities, the closed date is undefined, so the TTR is undefined and they can’t be included within the MTTR calculation. Without acknowledging that open vulnerabilities aren’t included in the calculation, the value can make the state of your remediation efforts look better than it really is.

As an example, if there are twenty vulnerabilities and one has a TTR of one day while the rest remain open, the MTTR will be $\frac{1}{1} = 1$ day. An MTTR of 1 is pretty impressive, but it is less meaningful when you know that 95% of the vulnerabilities remain open.

Open Vulnerabilities with Closed Dates?

But, what if we do assign a closed date to open vulnerabilities?

Before talking about the options that I’ve considered, I want to say explicitly that when this is done, the MTTR becomes an approximation or estimate. This needs to be explicitly stated if this is done to avoid being deceptive.

One option would be to assume that vulnerabilities will be resolved as soon as possible, which is today. When we assume this, the TTR for all open vulnerabilities will be the lowest it can be. In the future, the TTR for open vulnerabilities will continue to go up, which increases the MTTR. As a result, this method will produce a lower bound of the MTTR that can only increase in the future.

As an example, if there are open vulnerabilities that are three and five days old and a closed vulnerability resolved after two days, the calculation is $\frac{3 + 5 + 2}{3} \approx 3.3$ days. Therefore, we can state that the lower bound of the MTTR for these vulnerabilities is at least 3.3 days. If the two open vulnerabilities are still open tomorrow, the calculation would become $\frac{4 + 6 + 2}{3} = 4$ days.

As a second example, if there is an open vulnerability that is one day old and two closed vulnerabilities resolved after three and four days, the calculation is $\frac{1 + 3 + 4}{3} \approx 2.6$ days. This illustrates that this option can result in a lower bound that is either lower or higher than the actual MTTR of closed vulnerabilities.

Another option would be to use historical data to create an estimate for when open vulnerabilities are closed. There is a lot of nuance to this option, depending on the available data and choices you make. Some example are the scope you use to calculate the historical estimate and what you do with vulnerabilities open longer than that estimate. While I think this could be a useful option, analyzing it in greater depth would require making a lot of assumptions. I’ll leave the idea here and maybe revisit this option in the future.

Time Period

Another question is how you select the vulnerabilities you want to report on. The meaning of the number will change depending on which vulnerabilities you use to calculate your MTTR. I’ve contemplated a few ways of doing this, and they all seem to have their own considerations and limitations.

Everything

You could calculate the MTTR over every vulnerability that you have ever tracked. However, improvements–even significant improvements–in your MTTR can be slow to improve your metric if there is a long history overpowering the improvements. This can make it difficult to show that changes in process are actually improving the situation, which can lead to poor decision making if this number is used.

As an example, if you have 100 vulnerabilities all closed at 60 days and improve the process where vulnerabilities can be closed in 5 days. The first vulnerability resolved at 5 days will make your MTTR $\frac{100 * 60 + 5}{101} \approx 59.5$ days. The first 100 vulnerabilities resolved at 5 days will make your MTTR $\frac{100 * 60 + 100 * 5}{200} = 32.5$ days. It may take a long time to get close to the current MTTR and, even then, won’t reach the actual number.

Vulnerabilities Closed over Period

An option for not using the entire history of vulnerabilities is to select a time period, such as the first quarter of the year, and measure the MTTR over that period. However, there are two dates that go into calculating MTTR, the open date and the close date, and which one you choose will affect the meaning of your MTTR. In this section, I’ll talk about calculating MTTR using the closed date.

A benefit is that the vulnerabilities closed over a period is a fixed collection of vulnerabilities unless there is a problem in collecting the data. Once a period ends, no new closed vulnerabilities will be added to it and the MTTR can be calculated and used into the future without it changing. However, it can lead to a slightly unintuitive situation.

As an example, over a three month period of time, three vulnerabilities are resolved after 5 days, 5 days, and 365 days. The MTTR will be $\frac{5 + 5 + 365}{3} \approx 123.6$ days. In this case, two vulnerabilities were resolved quickly, but a final vulnerability was resolved a year after it was opened. As a result, the MTTR was inflated to almost 124 days.

There are a few things to notice in this example. First, the MTTR can be longer than the period of which it is calculated. While this is to be expected, it can be unintuitive that a 90 day period has a 124 day MTTR. Second, remediation efforts on old systems make your MTTR worse. Closing old vulnerabilities should be encouraged, but will be disincentivized if it negatively affects your metrics.

Vulnerabilities Opened over Period

The other option is to use the open date and finding all vulnerabilities that are opened during this period.

Unlike the vulnerabilities closed during a period, this value is subject to change after the period as those vulnerabilities continue to be closed. The MTTR for the first quarter of the year is going to be different depending on when you calculate the MTTR. As time goes on and more vulnerabilities are closed, the MTTR will increase.

A related concern is the open vulnerabilities issue mentioned above. The vulnerabilities opened later in the period are less likely to be closed than vulnerabilities earlier in the period. If calculated at the end of the period, there is likely to be a higher than normal number of open vulnerabilities excluded from the final number. This could be mitigated by waiting a short period of time after the period ends to calculate the MTTR.

Continuous Evaluation

While I mention a fixed period (e.g., the first quarter of the year) above, it could also be a relative period (e.g., the last 90 days). This enables continuous evaluation that can be done on a regular schedule to monitor the trend of the MTTR. You still have the option on using either the open or close date to calculate the MTTR with the same considerations as talked about in the previous sections. Open vulnerabilities that fall out of the relative period need another reporting method to avoid being lost.

Final Thoughts

Overall, my opinion is that MTTR can have a nuanced interpretation and has several options when calculating it. Unless other considerations are already resolved, I don’t believe that the MTTR is sufficient to evaluate the effectiveness of a vulnerability management program.

The most important point for me is that the rate that vulnerabilities are closed needs to be measured and understood. If only a small number of vulnerabilities are being closed, the MTTR may not have much meaning and focus should be on increasing the close rate. If the close rate is already at an acceptable level, it may be sufficient, but it should still be measured to ensure that it doesn’t drop below an acceptable level.

In terms of choices for measuring the MTTR, I think measuring by open date makes the most sense. The goal is fix vulnerabilities before they can be exploited and it seems that at some point the period for achieving that is less meaningful and including these metrics in the report is less valuable. Does it mean much if a vulnerability is closed after 720 days or 850 days? I’m not sure it does. I think that vulnerabilities open for a long period of time need a different reporting method. For that reason, I also think that calculating the MTTR on the close date is not very useful (although I think this is the most common way it is calculated, though I don’t have a source for that claim).

I am rather interested in the metric calculated from using today’s date for the close date of open vulnerabilities to calculate the lower bound of the MTTR. I think it can provide an interesting perspective of all the vulnerabilities that are being tracked. It’s something that might be worth thinking about further.

I also think that continuously evaluating the MTTR is valuable and important. I think this is the best way to track trends in the metric, especially to quickly evaluate new initiatives that attempt to improve it.

Future

I have some other thoughts on various metrics that I think may be useful and possible visualizations for those metrics. I think my next post will be about those, but we’ll see.