Statistics Seminar:: Rapid Detection of Bias in Forecasts of Financial Risk

GUEST LECTURER
Dr. Joe H. Sullivan

AFFILIATION
College of Business, Mississippi State University

ABSTRACT
Banking and insurance regulators, portfolio managers, corporate finance officers and others charged with the oversight or management of financial risk rely on daily forecasts of financial risk, such as Value-at-Risk (VaR). Financial risk may be characterized in many ways, all fundamentally related to the distribution of the gain over some time horizon, where a loss is represented as a negative gain. The value-at-risk measure simplifies the complex matter of financial risk characterization into a statement of the following form: “We are X percent certain that we will not lose more than V dollars in the next N days.” (Hull, 2003, p. 346). The value V is the VaR, which depends on the time horizon N and the confidence level X. Common values are 95% or 99% for X and 1 trading day for N. The VaR is the endpoint of a 1-sided prediction interval, which is the Xth percentile of the estimated distribution for the portfolio loss.
Estimation of the VaR is complex and almost always proprietary, based on estimation of the distributions of assets in the portfolio, their correlations, and their correlations with financial market factors. These methods of estimation are subject to various forms of inaccuracy, including systematic bias, such as systematically understating (or overstating) the risk. For risk managers it is imperative that the presence of systematic bias in these estimates not go undetected for very long. However, past research has focused on backtesting large samples of fixed size, often thousands of trading days, so that many years would be required to determine the presence of bias using such methods. The use of a historical sample of fixed size may justified in some regulatory situations, but is of little use to an entity that risks exhausting its capital in a single unfavorable market movement.
We develop a class of sequential testing tools for rapid assessment of the current accuracy, based on a time window that varies adaptively with the data. We are able to detect systematic bias in about 25 trading days or so, depending on the severity of the bias and other factors.
In addition to developing a real-time assessment procedure, we also introduce the concept of detection purity. Since the VaR is determined by a single point of the estimated distribution of the portfolio gain, with a loss represented as a negative gain, a pure test of VaR accuracy is obtained by converting each observed gain into a binary value indicating whether it was covered by the corresponding VaR forecast or not. Previous literature has pointed out that a more powerful test results from using the entire distribution, by transforming the observed gain to a random variable that has a known distribution when the forecast is accurate. Although more powerful, testing the entire distribution makes the test sensitive to errors in the upper tail that are not related to VaR or risk assessment. The previously published methods have been one or the other, either pure or testing the entire distribution.
We propose an adjustable, continuous compromise between detection power and detection purity, where power refers to quick detection of systematic bias and purity refers to insensitivity to errors not relevant to VaR estimation accuracy. We suggest that there are few practical situations for which the choice of either extreme would be optimal. Instead, we give a compromise that would be more effective in most practical applications.
--------------------------
Reference:
Hull, John C. (2003), Options, Futures and Other Derivatives, Prentice Hall, Upper Saddle River, NJ.

DATE & TIME
Wednesday, April 13, 2005 -- 12:00 PM

DURATION
1 hour

LOCATION
228 ISyE Main Building

CONTACT PERSON
roshan@isye.gatech.edu