Wednesday, December 23, 2009

On The Use Of Models By Standard & Poor's

Original posted on the Standard & Poor's website:

Standard & Poor's Ratings Services' ratings are opinions of creditworthiness based on our analysis. Ratings are not precise probabilities of default but rather a relative ranking of creditworthiness. One important tool we use in assigning ratings, but by no means the only one, is quantitative modeling. This report summarizes our definition of models, briefly describes what Standard & Poor's models are and are not used for, and discusses in general terms our view on methods of combining qualitative and quantitative considerations in the ratings process. The paper concludes with a discussion of our views on some distinctions we see between models suitable for ratings analysis and models more suitable for use in valuation, portfolio optimization, and risk measurement.


What Is A Quantitative Financial Model?

A quantitative model is a controlled view of certain real world dynamics that is used to infer the likely consequences of some pre-specified assumptions under various circumstances. However, in the process of moving from inputs to outputs, a model may not capture all the nuances of the real world. The distinctive feature of a quantitative financial model is that it is a quantitative calculation based on one or more assumptions. Models are not black boxes of revealed truth but merely numerical expressions of some view of how the world would be likely to behave. The models used in finance rely on assumptions about the behavior of people, organizations, acts of the natural world, and the use of other models by market participants. Quantitative financial models embody a mixture of behavioral psychology, statistics, numerical methods, and subjective opinions.

The physical sciences have laws of nature called "theories," that observation or experiments can verify or disprove. In finance, however, there are merely statistically significant tendencies and patterns, and there are always exceptions that do not fit these patterns. These exceptions do not mean that a given model is not useful. Rather, all quantitative financial models are necessarily generalizations that events in the real world will sometimes contradict. Because mathematicians, scientists, and engineers have developed powerful techniques for solving certain types of equations, quantitative financial analysts tend to express the underlying financial dynamics and assumptions as the same type of mathematical equations, and employ analogies to physical laws. This enables the financial models to use the same techniques from the physical sciences to compute the numerical solutions to these financial equations.

Different assumptions and different intended uses will in general lead to different models, and those intended for one use may not be suitable for other, unintended uses. Weaker performance under such circumstances does not necessarily indicate defects in a model but rather that the model is being used outside the realm for which it was optimized. In our view, the test of a financial model is how suitable it is for its intended use, which involves a simultaneous test of assumptions, inputs, implementation, and usage.

We believe that quantitative financial models can be very useful tools for analyzing credit risk.


The Use Of Quantitative Models By Standard & Poor's Ratings Services

The ratings process at Standard & Poor's can involve use of quantitative (models) in addition to qualitative (analytical judgment) considerations. The applicable published Standard & Poor's criteria will, in general, outline the nature of those considerations. The importance of models in the rating process varies. Analysis of more complicated debt often calls for more elaborate tools. Conversely, analysis of an unsecured senior bond might be, in effect, as simple as that the firm will pay interest unless it defaults. In this case, an analyst familiar with the issuer might not need models to form a credit opinion. On the other hand, when analyzing certain structured finance products, for example, model results could be of greater importance in a rating committee's decision. In between these two cases, such as when the issuer has a complicated portfolio of risks or has significant asset holdings in structured products, the debt we rate might have some risks we evaluate quantitatively and others qualitatively.

An essential part of any financial model is the set of assumptions on which the model is based. In some cases, the analysis of assumptions may call for a quantitative model. In other instances, models may not be necessary. Indeed, ratings determinations where no formal model is involved are common.

When we use a model, our ratings determination can follow one of several different paths, depending on the nature of the information and the circumstances. A path could involve qualitative assessments, quantitative models, or both, and it could have any number of steps. Models can fit into this analytical process either before or after qualitative judgments. For a front-end process, the analysts use the model output as one of the pieces of information on which to base their qualitative opinion. In a back-end process, analysts render a set of subjective opinions, which are the inputs to a model, and the model combines these and generates output. The model output could be a preliminary ratings indication, a numerical estimate such as a projected probability of default (PD), or the input to some other process.


One-step processes

Ratings determinations that employ only fundamental analysis without relying on formalized models emphasize qualitative factors, even when they are difficult to quantify. Such a process may include both qualitative and quantitative considerations without using a model. It may also include stress testing and scenario analysis.

Conversely, the financial analysis some market participants perform for certain types of instruments, such as collateralized debt obligations (CDOs), could largely or solely result from automated systems. Such systems might use a purely quantitative process in which the design of the model can potentially reflect certain qualitative considerations participants identify that the analysis of individual deals doesn't reconsider. And new information can immediately produce new results. On the other hand, this approach might fail to capture the impact of rapidly changing market conditions. Key relationships among variables in the model could suddenly change, diminishing the model's ability to produce useful and meaningful results. Accordingly, even a one-step-model-driven process requires close oversight from analysts to assess when the model's underlying assumptions begin to break down. Standard & Poor's generally uses an approach that incorporates additional assessments made outside of a formal quantitative model framework.


Multi-step processes

In contrast, many types of rating analysis involve a multi-step process in which models may play a role in one or more steps. For example, the analysis of residential mortgage-backed securities (MBS) involves a quantitative model (LEVELS) in the first step of the process to estimate tentative credit enhancement levels consistent with our criteria at the applicable rating levels for the liability side of a transaction. After the initial step, pursuant to our criteria, analytic adjustments could be made to reflect Standard & Poor's view regarding originator quality, the quality of due diligence, and the quality of a transaction's representations and warranties. After that, we may use a different quantitative model (SPIRE RMBS) to test the cash flow structure of an MBS transaction. We could also make qualitative analytic adjustments to the SPIRE output.

In other settings, qualitative analytic judgments could become inputs to a quantitative model. An example is the set of analytic adjustments made to financial ratios used in ratings analysis, based on a rating committee's views of potential sources of distortion and future volatility in the unadjusted ratios.


Multiple-model processes

In many cases, Standard & Poor's uses more than one model to help analyze whether the assets supporting a transaction are sufficient to fund the liabilities involved at the applicable rating stress level. It is quite common for the model used for the analysis of the assets to be distinct from the "waterfall" model of payment priorities used for the liability side. In most of these cases, we run both models under a prespecified set of stress scenarios. Either or both models could have separate qualitative adjustments applied.

Some ratings rely on models for certain components of analysis but not others. As an example, in insurance analysis, we use quantitative models to effectively analyze certain risk factors, such as property catastrophe risk or financial market risks. Other areas may not be amenable to quantitative modeling, such as management actions or regulatory risk. In these situations, the rating committee will determine the rating based on both quantitative and qualitative approaches, using its judgment about the significance of each risk and the appropriate weight to apply to each of those approaches.


Calibration

A more subtle subjective judgment lies in calibrating the model. In calibrating models, analysts should be careful to choose relevant data on which to base assumptions, including data from analogous areas. Standard & Poor's has taken steps toward the implementation of processes whereby models are independently reviewed, tested for parameter sensitivity, and recalibrated at regular intervals -- as and when more data become available or as market conditions change. We have recently published a list of stressful historical examples (see Appendix V of "Understanding Standard & Poor's Rating Definitions," June 3, 2009) as an aid in calibration. This list is merely suggestive and is not a definitive guide to calibrating any particular model or any particular asset class.


Quantitative Models For Credit Risk In Contrast To Other Financial Models

The intended uses of quantitative financial models include valuation, portfolio selection and optimization, risk measurement, and stress testing. Some features of any one model type might be incompatible with other model types.

For instance, some risk-measurement models, such as the Basel-mandated 10-day 99% Value-at-Risk (VaR) model, assume the probabilities of certain hypothetical future events and infer the likely consequences if these events occur. The VaR model is calibrated based on the assumption that market behavior in the near future will usually be similar to the recent past. Other risk-measurement models, collectively called stress tests, infer the likely consequences in assumed severe but plausible scenarios, without necessarily assigning any numerical probability to the scenarios. These stresses are calibrated based on the assumption that the market may occasionally behave very differently than it has in the recent past. An important distinction, in our view, between these two models is that, broadly speaking, the former addresses a bad day in a normal market, whereas the latter addresses an abnormal market.

Models used for different purposes could call for different assumptions. For example, because our assessment of creditworthiness is not a unique number but rather a band of expected performance, a credit rating model might calculate a modest range of values consistent with a specified stress assumption, whereas a valuation model would in general produce a much tighter range of values consistent with the current market with no extra assumed stresses. Also, the number of assumptions needed can depend on the characteristics of the intended use. As an example, consider a complex interest rate option. A stand-alone valuation model for the option is based on the assumed dynamics of the yield curve. If that same option is embedded in a high-yield corporate structured note, a model for valuing the incremental change in the note's price due to the embedded option must make additional assumptions about the relationship between interest rates and the issuer's credit spread, and about the likelihood of sub-optimal option exercise in different hypothetical yield curve scenarios. A model for estimating the incremental change in creditworthiness of that same structured note due to the same embedded option must make additional assumptions about both the likely option exercise behavior of the issuer in different hypothetical credit stress scenarios and the likely relationships between credit scenarios and interest-rate scenarios.

Financial risk consists of various components such as credit risk, franchise risk, liquidity risk, market risk, operational risk, and political risk. Most of the models Standard & Poor's uses are intended for our analysis of credit risk -- the likelihood and probable magnitude of credit events under different economic conditions. These models can concern many aspects of credit risk, including:

  • Expected cash flows under assumed stress scenarios,
  • Expected and unexpected losses from a financial instrument,
  • The correlation between debtors,
  • Contagion from the distress of one debtor to others,
  • Use of various indicators to infer the magnitude of credit risk, and
  • Use of various indicators and market observations to derive signals.

It's important to note that rating committees assign ratings based on qualitative and quantitative assessments. Where applicable, model outputs can provide relevant, useful information for purposes of a committee's analysis.

Many quantitative tools can be classified as either true models or rule-based calculations. True models (such as Standard & Poor's CDO Evaluator and LEVELS) make assumptions about possible future scenarios, while rule-based calculations (such as cashflow "waterfall" computations) compute the consequences of a given set of inputs without making any assumptions. For example, suppose that, in one particular scenario of a model, a CDO would collect $100. A rule-based cashflow tool could calculate how that hypothetical $100 would be paid out to the various tranches and accounts through the waterfall spelled out in the contract.

For a model intended for stress tests, some important features, in our view, may include:

  • Providing a meaningful answer over the widest possible range of circumstances;
  • No underlying implicit assumptions that certain inputs will remain small even in a stressed environment;
  • The ability to handle extraordinary relationships, such as when a bad asset affects previously unrelated assets. This is called contagion;
  • Features that are important only in extreme conditions;
  • Precision of results that was necessary in normal circumstances is not as important in a stress situation; and
  • Model calibration that combines extrapolation from past experience with expert judgment because of unprecedented stresses.

For models intended for risk measurement in less-extreme conditions, such as VaR, some important features, in our view, may include:

  • Giving meaningful answers in unremarkable circumstances;
  • Eliminating some less significant, or unobservable, inputs;
  • Hypothetical relationships not seen in the model calibration period might not be as important to model as carefully as those actually observed;
  • A business decision regarding which features can be eliminated;
  • Results in normal circumstances that are expressed in the same units as those in more extreme conditions to facilitate numerical comparisons; and
  • Model calibration that combines historical experience, current market expectations, and smaller amounts of judgment than for stress models.

Valuation models assume some relationships between some observed "calibrating" prices for liquidly traded instruments and the theoretical price of a less-liquid instrument, and they infer this unobserved price. Here, the analogous important features, in our view, may include:

  • Valuations for calibrating instruments that match the observed prices for all model inputs while using the same model for the calibrating instruments and the illiquid one;
  • A minimal number of unobservable inputs;
  • The need to incorporate relationships seen during the model calibration period into the model;
  • The importance of comparing results to the market consensus;
  • Quality of results in more extreme conditions is not as important; and
  • Calibration is based only on current market expectations and on historical relationships, with little or no judgment.

Predictive models, such as those used for investment and portfolio optimization, attempt to strike a balance between some measure of risk and a prediction of future reward. In general, these do not target market mispricing. Rather, they attempt to evaluate the market price of risk. They can address the entire market or a particular segment, or they can aim at a particular risk profile. Some key features, in our view, may include:

  • Giving a meaningful estimate, averaged over a range of expected future circumstances, including the likely error of the estimate;
  • Inputs include investor preferences, and different investors using the same model could get different answers;
  • Relationships expected for the future, up to the investment horizon, need not depend on what was seen during the model calibration period;
  • Features included in the model may depend on investor preferences;
  • The degree of applicability may also depend on investor preferences; and
  • Judgmental decisions and model calibration may, in many cases, disagree with the market consensus.

Statistical arbitrage models attempt to find and exploit perceived errors in the market consensus. In some sense, this is the opposite of a valuation model. The goal is, in CAPM (capital assets pricing model) jargon, to enhance alpha, either with a single security or on the portfolio level. In our view, key features may include:

  • Meaningful prediction of returns over a range of future circumstances;
  • Input could include proprietary views, and different arbitrageurs using the same model may get different answers;
  • Relationships expected for the future, up to the investment horizon, could be inputs and need not depend on what was seen during the model-calibration period;
  • The importance of modeling features not priced by the broader market, because an arbitrage model attempts to predict market mispricing;
  • A degree of model applicability that depends on investor expectations; and
  • Calibration that combines historical experience, current market expectations, and proprietary views.

Conclusion

For certain securities, the use of quantitative financial models can be an important tool in the rating process. The models Standard & Poor's uses are built to embody our assumptions and are specifically designed for use in our ratings process. These models can differ from those intended for other purposes and from those embodying different assumptions. We may use these models earlier in the ratings process than qualitative analysis, or later, or simultaneously, depending on our view on how best to analyze a particular aspect of credit risk.

No comments: