CONGRESS OF THE UNITED STATES CONGRESSIONAL BUDGET OFFICE An Evaluation of CBO’s Past Outlay Projections Percent 40 Since 1984, CBO’s projections of total outlays for the upcoming fiscal year have exceeded actual amounts by an average of 1.7 percent, excluding the effects of legislative changes. 20 0 -20 1985 1989 1993 1997 2001 2005 2009 2013 November 2017 Notes Unless otherwise indicated, all years referred to in this report are federal fiscal years, which run from October 1 to September 30 and are designated by the calendar year in which they end. Numbers in the text and tables may not add up to totals because of rounding. Supplemental data are posted along with this report on CBO’s website. www.cbo.gov/publication/53328 Contents Summary 1 How Close Were CBO’s Budget-­ ear Projections to Actual Outlays? Y 1 How Close Were CBO’s Sixth-­ ear Projections to Actual Outlays? Y 2 What Factors Have Contributed to Differences Between CBO’s Projections and Actual Outlays? 2 CBO’s Method for Projecting Outlays 3 CBO’s Method for Updating Its Projections 3 CBO’s Method for Assessing Its Projections 4 Sources of Data 4 Calculation of Differences Between Projected and Actual Outlays 5 BOX 1. HOW CBO CLASSIFIES CHANGES TO ITS ESTIMATES OF THE BUDGETARY EFFECTS OF LEGISLATION 5 BOX 2. PREVIOUSLY PUBLISHED EVALUATIONS OF CBO’S OUTLAY PROJECTIONS 6 Measuring the Quality of Projections 7 Assessment of the Projections 8 Total Outlays 9 Mandatory Outlays 12 Discretionary Outlays 23 Net Interest Outlays 27 Comparison of CBO’s and the Administration’s Outlay Projections 32 Adjusting the Administration’s Data for Comparison 32 Comparing the Quality of CBO’s and the Administration’s Projections 32 Appendix: Calculation of Economic and Technical Errors in CBO’s Projections of Outlays 35 List of Tables and Figures 39 About This Document 40 An Evaluation of CBO’s Past Outlay Projections Summary projected and actual outlays, which are related to CBO’s Since its inception, the Congressional Budget Office economic forecast or other factors (referred to as techni- has regularly published baseline projections of federal cal factors). For the purposes of this analysis, the agency revenues and outlays. Those projections, which reflect also removed outlays for the housing entities Fannie Mae the assumption that current laws will generally remain and Freddie Mac from its projections and the actual unchanged, typically underlie the budget resolutions amounts reported by the Treasury Department because prepared by the House and Senate Budget Committees CBO and the Administration account for those entities’ as well as CBO’s cost estimates for proposed legislation. transactions differently. The baseline projections can also be useful to policymak- ers seeking to identify and address budgetary trends that All told, CBO’s projections of outlays for both the are likely to play out over the coming years if current budget year and the sixth year have generally been close laws remain in place. to actual amounts, although they have been too high, on average. The Administration has also tended to As part of the process of preparing its projections, the overestimate baseline outlays for the budget year. (The agency regularly assesses the quality of its past estimates Administration has not published detailed information of federal spending and revenues to refine its methods on differences between its projections and actual outlays and improve the accuracy of its projections in the future. over the six-­ ear horizon, so CBO could not compare y For this analysis, CBO reviewed the baseline projections those longer-­ erm projections.) In general, both CBO’s t of total outlays as well as those of broad categories of and the Administration’s projection errors followed sim- spending that it has issued each spring, focusing on two ilar patterns and were larger for years in which unantic- fiscal years in the period spanned by each projection: ipated events that had large budgetary effects occurred. the second year (often called the budget year), which The Administration’s projections for budget years 1993 usually begins about six months after a spring baseline is through 2005 were about as accurate as CBO’s pro- released, and the sixth year. To assess the relative quality jections. Since 2005, however, the Administration has of its estimates, CBO compared its budget-­ ear projec- y overestimated spending in the budget year by more than tions with those of the Administration. CBO has in all but one year. The quality of CBO’s projections can be measured in var- How Close Were CBO’s Budget-­Year Projections to ious ways, but in this assessment, CBO focuses primarily Actual Outlays? on two characteristics: statistical bias (the tendency of a Since 1984, CBO has tended to overestimate total set of projections to err in the same direction) and accu- outlays (after adjustments for legislative changes) for the racy (how close projected values are to actual amounts). budget year; the average error for total outlays is 1.7 per- cent. Of the 32 budget-­ ear projections produced from y Any comparison of actual outcomes with projections is 1984 to 2015, 25 exceeded actual outlays. complicated by changes in law made after the projections are prepared. Although CBO does not attempt to predict Although often too high, CBO’s projections of total future legislative changes or their effects on outlays when outlays for the budget year have generally been close to preparing its baseline budget projections, actual outlays actual amounts. Half of the agency’s projections made are affected by those changes. CBO therefore adjusted since 1984 have differed from actual outlays by less than its projections to incorporate the estimated effects of 2 percent of the actual amounts. A few projections, legislation that was enacted after the projections were however, had significantly larger errors. For example, in produced. Thus, the analysis presented in this report 1992, CBO’s estimate of total outlays for 1993 was too concentrates on the remaining differences between high by about 8 percent. 2 An Evaluation of CBO’s Past Outlay Projections November 2017 The mean absolute error of the budget-­ ear projec- y proved difficult. Anticipating large and sudden increases tions—that is, the average of all errors without regard or decreases in spending for government programs for whether they were positive or negative—equals related to the financial sector has also been challenging. 2.3 percent. If CBO’s current budget-­ ear projection of y Although such developments related to the financial sec- $4.1 trillion in total outlays had an error the size of that tor have been rare, when they have occurred, they have mean absolute error, actual outlays for 2018 would be resulted in CBO’s misestimating outlays in its budget-­ higher or lower than the agency projected by $0.1 tril- year projections. lion—or about 0.5 percent of gross domestic product (GDP).1 After observing how those and other factors affected the accuracy of its past outlay projections, CBO noted the How Close Were CBO’s Sixth-­Year Projections to following: Actual Outlays? CBO’s sixth-­ ear projections also tended to be too high, y ■■ Projections of net interest outlays were not as accurate exceeding actual outlays in 20 of 28 years. Both the as those of most other categories of spending, average error (3.0 percent) and the mean absolute error primarily because CBO significantly overestimated (5.9 percent) of the sixth-­ ear projections made between y interest rates during the most recent recession 1984 and 2011 were larger than those of the budget-­ ear y and subsequent recovery (as did other economic projections covered by this analysis. Of the 28 sixth-­ forecasters). The sixth-­ ear projections of such outlays y year projections, 9 were within 4 percent of actual were substantially less accurate than the budget-­ ear y outlays, and 4 were off by 10 percent or more. If CBO’s projections. When net interest outlays are excluded June 2017 projection of $5.2 trillion in total outlays for from the evaluation of the projections for 1997 to 2022 had an error equal to that mean absolute error, 2016, the mean error of CBO’s sixth-­ ear projections y actual outlays would be higher or lower than the agency of overall outlays over that period is reduced by more projected by $0.3 trillion (or 1.3 percent of GDP). than half. What Factors Have Contributed to Differences ■■ Sixth-­ ear projections of Medicare and Medicaid y Between CBO’s Projections and Actual Outlays? spending were significantly less accurate than those of Many factors account for the differences between CBO’s all other categories of spending except for net interest projections and actual outlays. Some of those factors outlays, in part because it took CBO several years relate to the agency’s economic forecast. Forecasting to fully incorporate into its projections the slowing interest rates has been particularly challenging, even growth in spending for those programs that occurred during relatively stable periods, and errors in interest rate between 1996 and 2002 and between 2008 and forecasts have significantly affected CBO’s projections 2014. (Budget-­ ear projections for those programs y of interest payments. In addition, turning points in the were considerably more accurate.) economy can have significant effects on the budget. During recessions, outlays for countercyclical income ■■ Occasionally, large errors in projections of spending security programs—to pay unemployment benefits, for for other programs—mostly related to financial example—can increase rapidly. Such turning points, and activities—significantly influenced overall measures the cumulative budgetary effects that result from them, of bias and accuracy. For example, unanticipated are very difficult to anticipate. changes in spending for deposit insurance that stemmed from the savings and loan crisis accounted Other factors unrelated to the economic forecast have for roughly 90 percent of the large difference between also contributed to misestimates in CBO’s projections. CBO’s budget-­ ear projection of total outlays for y For example, quickly identifying new trends—particu- 1993 and the actual total. In 2010, the discrepancies larly in federal spending on health care—and incorpo- between the projections of outlays for deposit rating them into the agency’s baseline projections has insurance and for the Troubled Asset Relief Program (TARP) prepared the previous year and the actual amounts together exceeded the difference between 1. Congressional Budget Office, An Update to the Budget and Economic Outlook: 2017 to 2027 (June 2017), www.cbo.gov/ the projected and actual amounts of total outlays. publication/52801. Excluding those two programs would lower the November 2017 An Evaluation of CBO’s Past Outlay Projections 3 average error of CBO’s budget-­ ear projections of y caps—appropriations designated for overseas military total outlays for 1993 to 2016 by about two-­ hirds of t operations, for example—are assumed to keep pace a percentage point. with inflation throughout the baseline projection period.3 ■■ Differences between CBO’s estimates of total outlays in the budget year and actual outlays increased ■■ Projections of net interest outlays depend on the notably for the years immediately following the current amount of debt held by the public, CBO’s 2007–2009 recession and have narrowed somewhat forecast of interest rates, and the agency’s baseline since then. The differences between CBO’s sixth-­ ear y estimates of future borrowing needs. projections and actual outlays also increased during those years, but those larger differences—stemming CBO’s baseline projections of outlays are not intended from projections made before 2012—have persisted to be a prediction of budgetary outcomes. Rather, they in recent years. reflect CBO’s best judgment about what the economy and the budget would look like in future years under CBO’s Method for Projecting Outlays existing laws. That approach allows the baseline to serve In accordance with the rules governing baseline pro- as a neutral benchmark against which the effects of pro- jections, CBO constructs its 10-­ ear projections of y posed legislation or alternative policies can be measured. federal spending and revenues under the assumption that current laws will generally remain unchanged.2 The CBO’s Method for Updating Its Projections agency projects outlays for the three main categories of CBO generally releases baseline budget projections three spending—mandatory, discretionary, and net interest— times a year: near the beginning of the calendar year, although the procedures used to make projections for in early spring (to coincide with CBO’s estimate of the those three categories typically differ: President’s annual budgetary proposals), and in late summer. Whenever CBO updates its baseline, the agency ■■ Projections of mandatory spending reflect changes divides the changes in estimated outlays for each of the that CBO anticipates will occur in the economy, three types of spending into three categories: legislative, demographics, and other factors that affect spending economic, and technical. Legislative changes are the for various programs. estimated effects of new laws enacted since the last set of baseline projections was completed. Economic changes ■■ Projections for discretionary spending incorporate the are those that result from an update to CBO’s economic caps on such funding when they are in place; for years forecast. The agency typically updates its economic fore- for which no caps have been legislated, the baseline cast—which includes projections of GDP, interest rates, reflects the assumption (as required by section 257 of the unemployment rate, and other factors that affect the the Deficit Control Act) that discretionary funding budget—twice a year. Any updates to projections that are will keep pace with inflation. Those categories of neither legislative nor economic are considered technical discretionary funding that are not constrained by the changes. Examples of such changes include those stem- ming from modeling improvements, the incorporation of new demographic information, recent agency actions 2. CBO constructs its baseline in accordance with provisions set or judicial decisions, and updated data from federal forth in the Congressional Budget and Impoundment Control agencies or other sources. Act of 1974 and the Balanced Budget and Emergency Deficit Control Act of 1985, as amended by the Balanced Budget Act of 1997. Section 257 of the Deficit Control Act specifies some The legislative effects that are incorporated into CBO’s exceptions to the requirement that baseline projections reflect the baseline updates normally reflect the cost estimate that assumption that current laws remain unchanged. In particular, was prepared when the legislation was enacted. CBO the law requires CBO to assume full funding of entitlement authority. Another exception relates to mandatory programs whose authorization expires during the baseline period. All 3. In 2000, CBO estimated discretionary spending for its baseline such programs that predate the Balanced Budget Act and have projections using three different methods. This analysis current-­ ear outlays greater than $50 million are assumed to y incorporates the baseline from that year in which discretionary continue in CBO’s baseline projections. For mandatory programs funding adhered to the caps through 2002. In 2001, CBO established after 1997, continuation is assessed on a program-­ y-­ b departed from convention by publishing projections of program basis, in consultation with the House and Senate Budget discretionary spending for 2002 that did not incorporate the caps Committees. that had been legislated for that year. 4 An Evaluation of CBO’s Past Outlay Projections November 2017 does not typically revisit its initial estimates of the bud- Sources of Data getary effects of legislation. Rather, if it makes any sub- Although CBO has regularly published baseline bud- sequent revisions to its baseline to account for changes get projections since it was established, the number of in programs affected by that legislation, CBO considers years covered by those projections has changed over all aspects of each program and classifies the revisions time. Between 1984 (the earliest year included in this as economic or technical. (For more information about evaluation) and April 1995, CBO’s projections typically how CBO categorizes changes to its original estimates of covered the current year and the next 5 fiscal years. Since legislation, see Box 1.) that time, the agency’s baseline projections have covered the fiscal year in progress and the next 10 fiscal years. For After each fiscal year ends, CBO undertakes a detailed example, in March 2016, CBO released projections of review of the actual outlays reported by the Treasury outlays for fiscal year 2016 (the current year), for fiscal for the year to assess the quality of its projections for year 2017 (the budget year), and for each of the next that year. Those evaluations are important for two main nine years. The Treasury Department reports the actual reasons. First, they can help CBO improve its projec- amounts of outlays for each year in October, shortly after tions by identifying the factors that might have caused the end of the fiscal year (see Figure 1). (Those actual CBO to misestimate the trajectory of particular cate- amounts are subject to later revisions, which, if made, are gories of spending. Second, such evaluations provide typically small.) historical data that CBO can use to quantify the likely errors or uncertainty in future projections. (In addition This analysis uses estimates from the outlay projections to conducting those annual internal reviews, CBO has that CBO released each spring (generally around March), previously published evaluations of the quality of its past primarily because the budgetary effects of legislation outlay projections. For more information about those being considered by the Congress are typically measured previous evaluations of CBO’s projections, see Box 2.) in relation to those estimates.4 The span of years exam- ined for this evaluation differs by category of spending CBO’s Method for Assessing Its Projections on the basis of data availability. The oldest projections To assess its past projections of outlays, CBO compared of total outlays included here were made in 1984; the them with actual amounts recorded in the budget and earliest projections of mandatory spending (including attempted to determine the sources of any differences details for Social Security, Medicare, Medicaid, and other between the two. Because the agency intentionally does mandatory programs), total discretionary spending, and not incorporate the effects of possible legislative changes net interest outlays were made in 1992; and the oldest into its baseline projections, this report focuses on those projections of defense and nondefense discretionary differences between projected and actual outlays that outlays were made in 1998. result from economic and technical factors. However, categorizing the changes is an imperfect process, in part The data necessary to categorize differences as legislative, because all three categories interact with one another. economic, or technical for most types of spending are not available for projections made before 1992, so this This evaluation focuses on two characteristics of CBO’s analysis focuses on projections made since then. As a outlay projections: statistical bias (the tendency of a set result, the sample size of the projection errors is relatively of projections to be too low or too high over a period small by statistical standards: 24 budget-­ ear projections y of time) and accuracy (how close projected values are to and only 20 sixth-­ ear projections. Because CBO did not y actual values). To measure statistical bias, CBO uses the extend its baseline beyond six years until late 1995 and average, or mean, error. To assess accuracy, the agency because this analysis includes actual data only through uses two other measures: the mean absolute error (which 2016, the sample size of projection errors drops off is the average of the magnitude of all errors, without precipitously for longer projection periods. (For example, regard for whether they are positive or negative) and the root mean square error (or RMSE, which is calculated by 4. CBO’s evaluation of its revenue projections used projections squaring the errors, averaging those squares, and taking released near the beginning of the calendar year instead of the square root of that average). Both of those measures those published in the spring. Using the spring updates for that analysis would have had very minimal effects on the results. See indicate the degree to which projections are dispersed Congressional Budget Office, CBO’s Revenue Forecasting Record around the actual values. (November 2015), www.cbo.gov/publication/50831. November 2017 An Evaluation of CBO’s Past Outlay Projections 5 Box 1. How CBO Classifies Changes to Its Estimates of the Budgetary Effects of Legislation Each time the Congressional Budget Office updates its baseline additional spending for Medicaid created by the expansion of projections, the agency divides the changes in estimated the program under the Affordable Care Act (ACA). outlays into three categories: legislative, economic, and Emergency Unemployment Compensation. During and after technical. Because CBO constructs its baseline projections the most recent recession, lawmakers enacted several laws under the assumption that current laws will generally remain that temporarily extended unemployment benefits under the unchanged, the agency adjusted its projections for this report Emergency Unemployment Compensation (EUC) program. to account for the effects on outlays of legislation enacted after CBO’s original cost estimates for EUC extensions were based the projections were produced, thereby allowing it to focus on on its current economic forecasts and technical factors (such as economic and technical errors. the number of people currently receiving regular unemployment The legislative effects that are incorporated into CBO’s updated benefits and historical information on average benefit amounts baseline projections normally reflect the cost estimate that the and duration of unemployment spells) that were known when agency prepared when the legislation was enacted. However, each estimate was prepared. In many cases, however, the the costs and savings that result from legislation depend actual unemployment rate ended up differing from the rate on how the changes are implemented, prevailing economic that CBO projected when it prepared its cost estimate for a conditions, and other factors. If any of those factors differ from particular piece of legislation. As a result, updates to CBO’s the conditions that CBO initially anticipated, the agency may baseline projections that included EUC often included changes subsequently adjust its estimate of outlays for the affected to estimates of spending for that program that were classified programs to account for those variations. as economic or technical rather than legislative. CBO does not normally revisit its initial estimates of legisla- Medicaid Expansion Under the ACA. The ACA required tion, however. In fact, in most cases it would be impossible all states to expand Medicaid coverage, so when the law to disentangle the effect of the legislation from other factors was enacted on March 23, 2010, CBO updated its baseline that affect spending for a program. Rather, any subsequent projections to account for the additional Medicaid outlays, a changes to the baseline for programs affected by that legis- change it classified as legislative. But in 2012 the Supreme lation, including the estimated effects of implementing the Court ruled that the Medicaid expansion was optional for new law through the administrative or regulatory process, states, and it became clear that some states would not expand are classified as economic or technical rather than legislative. coverage. Because of that decision, CBO reduced its projec- Two examples of such treatment are CBO’s projection of the tions of spending for Medicaid and recorded that revision as a cost of extending unemployment benefits and its projection of technical change. only 15 seventh-­ ear projections could be evaluated.) y effects of subsequently enacted legislation.5 Such adjust- CBO therefore chose to focus on evaluating its budget-­ ments isolate discrepancies between projected and actual year and sixth-­ ear projections in this report. y amounts that stem from economic and technical factors that CBO attempts to account for in its projections. Calculation of Differences Between Projected and Actual Outlays Before comparing its baseline projections with the actual 5. On average, CBO estimates that legislative changes have outlays reported by the Treasury Department, CBO increased total outlays in the budget year by 2 percent and in adjusted the projections to account for the effects of the sixth year by 4 percent. However, those averages result from legislation enacted after they were made. Because CBO’s wide ranges of effects. For budget-­ ear projections made between y baseline is constructed under the assumption that current 1984 and 2015, the effect of legislation enacted after they were laws will generally remain in place, it was necessary to prepared ranged from reducing budget-­ ear outlays by 2 percent y to increasing them by 14 percent. For the sixth-­ ear estimates y adjust the baseline projections to include the estimated published between 1984 and 2011, the variation in legislative effects was even greater, ranging from a reduction in spending of 16 percent to an increase of 21 percent. 6 An Evaluation of CBO’s Past Outlay Projections November 2017 Box 2. Previously Published Evaluations of CBO’s Outlay Projections Although this report is the Congressional Budget Office’s In past editions of The Budget and Economic Outlook, which most comprehensive evaluation of the quality of its past CBO publishes each winter (typically in January), the agency outlay projections to date, the agency has previously included evaluated its track record of projecting baseline revenues, evaluations of the accuracy of its outlay projections in various outlays, and the surplus or deficit (after adjusting the projec- publications. Those evaluations, which have taken different tions to account for legislative changes) in the current year, approaches over the years, have detailed the reasons why the budget year, and each of the following four years.2 Those CBO’s outlay projections differed from actual amounts. evaluations, which covered projections made between 1981 and 2003, included discussions of reasons for the projection One approach that CBO has used in the past to evaluate the errors.3 accuracy of its projections has been to analyze the sources of differences between targets set in the budget resolution More recently, in 2012, CBO published a short report that com- in a given year and actual budgetary outcomes for that year.1 pared its January 2001 baseline projections for 2001 to 2011 The annual budget resolutions, adopted by both Houses of with the actual outlays recorded in those years and examined Congress, specify target levels for revenues, spending, and the the reasons for differences between the projected and actual deficit or surplus for the upcoming fiscal year (the budget year). amounts.4 The agency has also looked at the accuracy of its Typically those targets are based on CBO’s economic assump- projections of outlays for specific programs (Medicare’s Part D tions and baseline projections for the budget year and incor- prescription drug program, for example) and examined why its porate estimates of the budgetary effects of proposed policy projections for broad categories of spending (such as spending changes to be enacted in the coming year. Thus, by evaluating for the major health care programs) have changed over time.5 the accuracy of the budget resolutions, CBO was essentially In addition to those evaluations of its outlay projections, CBO evaluating its own budget-­ ear projections. Using that method, y has published assessments of the quality of its economic fore- CBO has analyzed the accuracy of its budget projections for casts and revenue projections.6 the years between 1977 and 1999. In its evaluations, CBO classified the differences between the targets specified in budget resolutions and actual amounts as stemming from policy, economic, or technical factors. (In 2. See, for example, Congressional Budget Office, The Budget and Economic some evaluations, a fourth category was included to account Outlook: Fiscal Years 2005 to 2014 (January 2004), Appendix A, www.cbo. for differences that resulted from administrative actions; in gov/publication/15179. other evaluations, such differences were attributed to technical 3. Unlike the evaluations for projections made between 1981 and 2003, this factors.) Differences attributed to policy changes included the analysis begins with the projections made in 1984 for 1985. The primary budgetary effects of proposals that were anticipated at the reason for the different starting point is that the current study evaluates time a resolution was prepared but not enacted during the outlay projections in CBO’s March baselines, whereas earlier evaluations year, proposals that were enacted in a different form than orig- analyzed the agency’s January baseline projections. The detailed data inally envisioned, and proposals that were not envisioned by necessary to evaluate outlay projections in the agency’s March baseline are lawmakers when they adopted a resolution but were enacted. available only for 1984 and later years. Differences between the targets and actual amounts that 4. See Congressional Budget Office, Changes in CBO’s Baseline Projections resulted from economic and technical factors were accounted Since January 2001 (June 2012), www.cbo.gov/publication/41463. for in ways that were largely consistent with the approach used 5. See, for example, Congressional Budget Office, “The Accuracy of in this analysis except for years when the budget resolution CBO’s Budget Projections,” CBO Blog (March 25, 2013), www.cbo.gov/ incorporated the Administration’s economic assumptions rather publication/44017, and “How Have CBO’s Projections of Spending for than CBO’s. Medicare and Medicaid Changed Since the August 2012 Baseline?” CBO Blog (February 21, 2013), www.cbo.gov/publication/43947. 6. See, for example, Congressional Budget Office, CBO’s Economic 1. See, for example, Congressional Budget Office, The Economic and Forecasting Record: 2017 Update (October 2017), www.cbo. Budget Outlook: An Update (August 1987), Chapter III, www.cbo.gov/ gov/publication/53090, and CBO’s Revenue Forecasting Record publication/16362. (November 2015), www.cbo.gov/publication/50831. November 2017 An Evaluation of CBO’s Past Outlay Projections 7 Figure 1 . Sample Timeline for Measuring Errors in CBO’s Projections of Outlays for the Budget Year CBO releases initial baseline budget projections for The Treasury Department releases 2015 to 2025, which include current-year projections actual outlay amounts for fiscal year for 2015, budget-year projections for 2016, and 2016 (the budget year in CBO's sixth-year projections for 2020. March 2015 baseline projections). Fiscal Year 2015 Fiscal Year 2016 Fiscal Year 2017 (Current Year) (Budget Year) (Third Year) Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep The Administration CBO releases an update to its releases the President's outlay projections for 2015 to 2016 budget and 2025. The baseline updates updated baseline released each spring, such as projections. this one, are evaluated in this report. Source: Congressional Budget Office. Dates for the budget-­ ear projection of outlays for 2016 (the most recent examined in this study) are used in this timeline for illustrative purposes. The y timelines for measuring errors in all budget-­ ear projections of outlays are similar, although the President’s budget is not always released in February. y The agency also adjusted its baseline projections and from the adjusted projections and dividing that differ- the actual amounts reported to remove outlays related ence by actual outlays. Thus, the errors are expressed to Fannie Mae and Freddie Mac, two entities that help as percentages of actual outlays, with negative values finance the majority of mortgages in the United States. representing underestimates and positive values, overesti- CBO did so because it accounts for the activities of those mates. For example, in March 2015, CBO released base- entities differently than the Administration does in the line projections for 2015 through 2025 and projected budget or the Treasury does in its reports. Since 2008, that outlays in the budget year, 2016, would total a little when the federal government placed Fannie Mae and more than $3.9 trillion under current law. That projec- Freddie Mac into conservatorship, CBO and the budget tion was $55 billion more than actual outlays in 2016 committees have considered the activities of those two (excluding spending for Fannie Mae and Freddie Mac). institutions to be governmental. In CBO’s view, trans- Adjusting CBO’s projection for the estimated $49 billion actions between Fannie Mae and Freddie Mac and the in outlays that resulted from legislation enacted into law Treasury should be considered intragovernmental. In after the baseline was completed increases the total differ- contrast, the Administration considers Fannie Mae and ence between projected and actual outlays to $104 bil- Freddie Mac to be outside the federal government for lion. Dividing that amount by actual outlays recorded in budgetary purposes and thus records cash transactions 2016 ($3.9 trillion) yields a projection error of 2.7 per- between them and the Treasury as increases or decreases cent. (See the appendix for a detailed explanation of how in federal outlays.6 Because the accounting for those two CBO calculates and characterizes the differences from concepts is entirely different, comparing CBO’s estimates one baseline update to the next.) with the amounts recorded by the Treasury would not contribute to a meaningful assessment of CBO’s statisti- Measuring the Quality of Projections cal bias or accuracy in estimating outlays. Like CBO’s previous studies of its economic forecasts and revenue projections, this evaluation of the agency’s After making those two adjustments, CBO calculated outlay projections focuses on two indicators of quality: the errors in its projections by subtracting actual outlays statistical bias and accuracy. 6. To make its estimate of current-­ ear outlays reflect what the y Statistical Bias. The statistical bias of a set of projec- Treasury will ultimately report, CBO adopts the Administration’s tions indicates the tendency of the projections to err in treatment for outlays in the current year. 8 An Evaluation of CBO’s Past Outlay Projections November 2017 a particular direction. CBO aims to provide a baseline The sum of the budget-­ ear projections of spending that y projection of outlays that is free of such bias—that is, CBO prepared between 1992 and 2015 for the years one that is equally likely to be higher than actual outlays 1993 to 2016 and the estimated outlays in those years as it is to be lower. To measure statistical bias, CBO uses resulting from legislation enacted after those projections the mean error—the arithmetic average of the projection were prepared is $61.9 trillion (excluding outlays related errors—which is the simplest measure of bias. However, to Fannie Mae and Freddie Mac). However, actual because the positive values of overestimates offset the outlays for those years were $1.3 trillion, or 2.2 percent, negative values of underestimates in the calculation of less than that amount. Errors in projections of outlays the mean error, the measure provides an imperfect view for mandatory programs other than Social Security, of the accuracy of a projection. A number of projections Medicare, and Medicaid accounted for about 40 per- with small errors in both directions that largely offset one cent of the total difference between projected and actual another would produce a small mean error. But so, too, amounts, even though such spending represented only would relatively large overestimates and underestimates, about 14 percent of all outlays over that time. Similarly, as long as they were approximately the same magnitude the projections of outlays for net interest, which made and counterbalanced one another. up about 9 percent of total outlays, were responsible for 20 percent of the overall difference (see Figure 2). By Accuracy. The accuracy of a set of projections is the contrast, projections of discretionary spending accounted degree to which projected values are dispersed around for about a quarter of the total difference even though actual outcomes. In this evaluation, CBO used two such spending has constituted about 36 percent of total standard measures of accuracy: the mean absolute error outlays since 1993. Social Security made up 22 percent and the RMSE. The mean absolute error is the arith- of total outlays but had almost no effect on the mean metic average of the errors without regard to the sign of error of projections of total outlays. the error (that is, the negative signs are removed from underestimates before averaging), so errors in different For the sixth-­ ear projections of total outlays that CBO y directions do not offset one another. The RMSE also made for the years 1997 to 2016, overestimates of net measures the size of errors without regard to direction, interest outlays were responsible for most of the gap but by squaring the errors, it places a greater weight on between projected outlays and actual amounts. Although larger deviations.7 such spending made up only about 8 percent of all out- lays over that period, it accounted for roughly 70 percent Assessment of the Projections of the total difference between estimated outlays and CBO’s projections of total outlays and of most broad the actual amounts reported.8 Projections of Medicare categories of spending for both the budget year and the spending—the next largest contributor to the overall sixth year have generally been too high. Projections for error—accounted for about 14 percent of the difference the budget year have been more accurate than those for between projected and actual outlays, an amount roughly the sixth year for most categories of spending, primarily equal to such spending’s share of total outlays since because changes in the economy, demographics, and a 1997. As was the case for the budget-­ ear projections, y variety of other factors are more difficult to anticipate discrepancies between projections of discretionary spend- over longer time horizons. ing and actual amounts accounted for a portion of the overall mean error (13 percent) that was much smaller than such spending’s share of total outlays between 1997 and 2016 (36 percent). In its sixth-­ ear projections, y CBO underestimated spending for mandatory programs 7. The RMSE is calculated by squaring the projection errors, other than Social Security, Medicare, and Medicaid, off- averaging those squares, and taking the square root of that average. The mean square projection error is equal to the square setting a small part of the agency’s overestimates in other of the bias in the errors plus the variance (that is, the square of categories of spending and reducing the mean error of the standard deviation) of the errors. Because the projection its projections of total outlays by 7 percent. Those other errors in this assessment are measured as a percentage of the actual amount, the RMSE measurement reported here is often referred to as the root mean square percentage error. Likewise, the 8. If net interest is excluded, the average error of CBO’s sixth-­ ear y mean absolute error reported in this assessment is often referred projections made for 1997 to 2016 drops from 5.9 percent to to as the mean absolute percentage error. 2.3 percent. November 2017 An Evaluation of CBO’s Past Outlay Projections 9 Figure 2 . Share of Total Spending and of Projection Errors for Total Outlays, by Category of Spending Share of Total Spending Recorded Between Share of Mean Error of 1993 and 2016 Projections of Total Outlays Mandatory Social Security 21.9% Share of Mean Error of Budget-Year Projections for 1993 to 2016 Medicareª 12.6% Share of Mean Error of Sixth-Year Projections for 1997 to 2016 Medicaid 7.4% Other mandatory 13.9% Discretionary 35.8% Net Interest 8.5% -10 0 10 20 30 40 50 60 70 80 Percent Source: Congressional Budget Office. Errors are projected amounts minus actual amounts, expressed as percentages of actual amounts; thus, a negative error indicates an underestimate, and a positive error, an overestimate. The mean error is the arithmetic average of the projection errors. Outlays related to the activities of Fannie Mae and Freddie Mac are excluded from the data above. The projection errors also exclude the estimated budgetary effects of legislation enacted after the projections were produced. a.Includes offsetting receipts. programs accounted for about 14 percent of total outlays categories of spending other than net interest and Social over the period. Security were largely technical. About three-­ uarters of the mean error in the budget-­ q The 2007–2009 recession provides one example of the year projections of total outlays (after the adjustments significant effect that economic developments can have to remove the discrepancies attributable to legislative on the budget. Both budget-­ ear projections and sixth-­ y changes) stems from technical factors and the remainder year projections for fiscal year 2010 were less accurate of the error, from economic factors. Errors in projections than those for most previous years because of the 2007– of specific categories of spending were overwhelmingly 2009 recession and the difficulty of projecting interest technical in nature, with the exception of errors in rates during the subsequent economic recovery and projections of net interest, which stemmed primarily expansion. Although the mean error and the mean abso- from errors in CBO’s forecast of interest rates, and in lute error of the budget-­ ear projections have returned to y projections of outlays for Social Security. About 60 per- levels similar to those recorded before the recession, the cent of the mean error of the sixth-­ ear projections of y errors for the sixth-­ ear projections have not. y total outlays is attributable to economic factors, namely the difficulty CBO and others experienced in forecast- Total Outlays ing interest rates. As with the budget-­ ear projections, y In the budget-­ ear projections that CBO made y errors in the agency’s sixth-­ ear projections of specific y between 1984 and 2015, the agency was more likely to 10 An Evaluation of CBO’s Past Outlay Projections November 2017 Table 1 . Summary Measures of the Quality of CBO’s Projections of Outlays Percent Budget-Year Projections Sixth-Year Projections Mean Absolute Root Mean Mean Absolute Root Mean Mean Error Error Square Error Mean Error Error Square Error By Category of Error (For baseline projections published since 1984)a Total Outlays 1.7 2.3 3.1 3.0 5.9 6.7 Economic errors 0.4 0.7 0.9 3.1 3.5 3.9 Technical errors 1.3 2.0 2.6 -0.1 4.0 4.9 By Category of Spending (For baseline projections published since 1992)b Total Outlays 2.2 2.3 2.9 5.9 6.0 6.9 Mandatory 2.4 2.9 4.1 2.5 3.4 5.0 Social Security * 0.8 0.9 0.3 2.9 3.6 Medicarec 1.9 3.3 4.1 7.3 8.0 10.5 Medicaid 2.5 4.0 5.1 10.3 13.5 17.5 Other mandatory 7.0 8.5 13.8 -1.9 5.2 6.3 Discretionary 1.3 1.4 1.7 1.9 2.0 2.5 Net Interest 5.2 7.7 9.4 54.0 56.0 67.0 Source: Congressional Budget Office. Errors are projected amounts minus actual amounts, expressed as percentages of actual amounts; thus, a negative error indicates an underestimate, and a positive error, an overestimate. The mean error is the arithmetic average of the projection errors. The mean absolute error is the average of the errors with the negative signs removed from the underestimates. The root mean square error is calculated by squaring the projection errors, averaging those squares, and taking the square root of that average. The estimated budgetary effects of legislation enacted after the projections were produced, as well as outlays related to the activities of Fannie Mae and Freddie Mac, are excluded from the errors. * = between zero and 0.05 percent. a.Measures are based on budget-­ ear projections for 1985 to 2016 and sixth-­ ear projections for 1989 to 2016. The data necessary to calculate the y y projection errors for total outlays are not available for projections made before 1984. b.Measures are based on budget-­ ear projections for 1993 to 2016 and sixth-­ ear projections for 1997 to 2016. The data necessary to calculate the y y projection errors for most types of spending are not available for projections made before 1992. c.Includes offsetting receipts. overestimate total outlays (excluding the effects of legis- 2 percent of the actual outlays. A few errors, however, lative changes) than to underestimate them. The mean were much larger than that. In 1992, for example, CBO error of projections made in those years is 1.7 percent, overestimated total outlays for 1993 by about 8 percent the mean absolute error is 2.3 percent, and the RMSE is for economic and technical reasons (see Figure 3). The 3.1 percent (see Table 1). If the projection of $4.1 tril- unusually large error in the projection for that year was lion in total outlays for 2018 that CBO published in primarily the result of an unanticipated delay in provid- June 2017 had an error equal to that mean absolute ing the funding necessary to allow the government to error, actual outlays would be $0.1 trillion higher or resolve failed savings and loan associations. (The projec- lower than the agency projected. tion for 1992 was off by a similar amount.) When such spending is excluded, CBO’s projection for 1993 is only CBO’s projections of total outlays for the budget year about 1 percent higher than actual outlays recorded for have generally been close to actual amounts. Half of the that year. projections made between 1984 and 2015 were within November 2017 An Evaluation of CBO’s Past Outlay Projections 11 Figure 3 . Errors in CBO’s Projections of Total Outlays Budget-Year Projections Percent 50 40 30 Mean Projection Error, 1985 to 2016 20 (1.7%) 10 0 -10 -20 Sixth-Year Projections Percent 50 40 30 Mean Projection Error, 1989 to 2016 20 (3.0%) 10 0 -10 -20 1985 1987 1989 1991 1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 2013 2015 Source: Congressional Budget Office. The years shown are those for which each projection was made. For example, the values shown for 2016 are for the budget-­ ear projection published y in March 2015 and the sixth-­ ear projection published in March 2011. y Errors are projected amounts minus actual amounts, expressed as percentages of actual amounts; thus, a negative error indicates an underestimate, and a positive error, an overestimate. The mean error is the arithmetic average of the projection errors. The estimated budgetary effects of legislation enacted after the projections were produced, as well as outlays related to the activities of Fannie Mae and Freddie Mac, are excluded from the errors. CBO’s sixth-­ ear projections have also tended to be too y projections indicate that they had larger errors than the high. Of the 28 sixth-­ ear projections of total outlays y budget-­ ear projections had and that those errors were y evaluated, 20 exceeded the actual amounts recorded. more widely dispersed. Of the 28 sixth-­ ear projections, y The mean error (3.0 percent), the mean absolute error 9 were within 4 percent of actual outlays, and 4 were off (5.9 percent), and the RMSE (6.7 percent) of the by 10 percent or more. If CBO’s June 2017 projection of projections made between 1984 and 2011 for 1989 to $5.2 trillion in total outlays for 2022 (the current sixth-­ 2016 were larger than those of the budget-­ ear projec- y year projection) had an error equal to that mean absolute tions analyzed in this report. The larger values of the error of 5.9 percent, actual outlays would differ from the three summary measures of the quality of the sixth-­ ear y projected amount by $0.3 trillion. 12 An Evaluation of CBO’s Past Outlay Projections November 2017 Mandatory Outlays to that of the budget-­ ear projections. Of the 20 sixth-­ y Mandatory, or direct, spending consists of outlays for year projections made between 1992 and 2011 (for some federal benefit programs and for certain other 1997 to 2016), 14 were overestimates. The mean error payments to people, businesses, nonprofit institutions, of CBO’s sixth-­ ear projections was 2.5 percent—only y and state and local governments. It includes outlays for 0.1 percentage point higher than the mean error of the Social Security, Medicare, and Medicaid as well as other budget-­ ear estimates. The size of overestimates in the y spending that is generally governed by statute rather than two sets of projections is also similar. For the budget-­ by specific annual appropriations. (Some mandatory year projections, the largest overestimate was 12.7 per- programs receive annual funding in appropriation acts.) cent, and the average overestimate was 3.4 percent; for In 2016, mandatory spending amounted to $2.4 trillion, the sixth-­ ear projections, the largest overestimate was y or 63 percent of total federal outlays for the year (net of 11.8 percent, and the average overestimate, 4.2 percent. offsetting receipts, which reduce outlays). The share of In addition, when CBO underestimated mandatory all federal outlays that goes to mandatory spending has outlays in its sixth-­ ear projections, it did so by slightly y grown significantly over time; such spending accounted larger amounts than it did in its budget-­ ear projections. y for about 47 percent of federal spending in 1992. The average underestimate in the sixth-­ ear projections y was 1.4 percent, whereas the average underestimate in The Deficit Control Act requires CBO to construct the budget-­ ear projections was 1.1 percent. The mean y baseline projections of spending for most mandatory absolute error of the sixth-­ ear projections (3.4 percent) y programs under the assumption that current laws will was about half a percentage point greater than the mean continue unchanged.9 CBO’s projections of mandatory absolute error of the budget-­ ear projections. y spending also reflect changes in the economy, demo- graphics, and other factors that it anticipates will occur In general, both CBO’s budget-­ ear and sixth-­ ear y y under current law. projections of Social Security outlays have been close to actual outlays and have exhibited little statistical bias. Excluding legislative changes, CBO has tended to over- Projections of Medicare and Medicaid have been further estimate mandatory outlays since 1992 (the first year off, especially those for the sixth year. CBO’s budget-­ ear y for which the data necessary for evaluation are avail- projections of other mandatory spending have differed able). Indeed, 19 of 24 budget-­ ear projections made y significantly from actual outlays, mostly because of large for the years 1993 to 2016 exceeded actual amounts errors in projections of outlays for a few programs. The (see Figure 4). The mean error of those projections was sixth-­ ear projections of other mandatory spending, y 2.4 percent, and the mean absolute error, 2.9 percent however, have been much closer to actual spending. (see Table 2). Most of the projections—17 of 24 (includ- That is the only major category for which the sixth-­ ear y ing 16 of the 18 most recent projections)—had errors of projections were more accurate than the budget-­ eary 2.9 percent or less. Only 5 projections differed from the projections. actual amounts by a percentage greater than the RMSE of the projections (4.1 percent). If CBO’s June 2017 pro- Social Security. Social Security comprises two parts: jection of $2.6 trillion in mandatory outlays for 2018 Old-­ ge and Survivors Insurance (OASI) and Disability A (the current budget-­ ear projection) had an error equal y Insurance (DI). To receive either type of benefit, an to that mean absolute error of 2.9 percent, actual outlays individual must have a substantial work history and meet for those programs would be $74 billion higher or lower other criteria. (For old-­ ge benefits, the claimant must a than projected. have reached a minimum retirement age; for disability benefits, a claimant must be unable to work because of The statistical bias and accuracy of the sixth-­ ear projec- y a disability.) Certain dependents of deceased, retired, or tions of mandatory spending followed a pattern similar disabled workers also qualify for benefits. CBO’s projec- tions of outlays for those benefits are based on the num- 9. In keeping with rules established by the Deficit Control Act, ber of beneficiaries and their average benefits. In 2016, CBO’s baseline projections reflect the assumption that certain mandatory outlays for Social Security totaled $910 bil- mandatory programs whose authorization expires within the lion, almost 85 percent of which was for OASI.10 current projection period will continue. In addition, the Deficit Control Act directs CBO to assume that entitlement authority will be fully funded and that a cost-­ f-­iving adjustment for o l 10. Administrative expenses for Social Security are classified as veterans’ compensation will be granted each year. discretionary spending. November 2017 An Evaluation of CBO’s Past Outlay Projections 13 Figure 4 . Errors in CBO’s Projections of Mandatory Outlays Budget-Year Projections Percent 50 40 30 Mean Projection Error, 1993 to 2016 20 (2.4%) 10 0 -10 -20 Sixth-Year Projections Percent 50 40 30 Mean Projection Error, 1997 to 2016 20 (2.5%) 10 0 -10 -20 1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 2013 2015 Source: Congressional Budget Office. The years shown are those for which each projection was made. For example, the values shown for 2016 are for the budget-­ ear projection published y in March 2015 and the sixth-­ ear projection published in March 2011. y Errors are projected amounts minus actual amounts, expressed as percentages of actual amounts; thus, a negative error indicates an underestimate, and a positive error, an overestimate. The mean error is the arithmetic average of the projection errors. The estimated budgetary effects of legislation enacted after the projections were produced, as well as outlays related to the activities of Fannie Mae and Freddie Mac, are excluded from the errors. Statistical Bias and Accuracy. CBO’s budget-­ ear projec- y between projected and actual outlays was 2.6 percent. In tions of Social Security spending were lower than the general, relatively small errors in CBO’s forecasts of infla- actual outlays almost as often as they were too high—and tion, which determine annual cost-­ f-­iving adjustments o l by roughly similar amounts—resulting in a mean error (COLAs), have been offset by relatively small errors close to zero (see Figure 5). Those projections also have related to other determinants of benefit amounts and consistently been accurate, with a mean absolute error of projections of caseloads. (Caseloads for Social Security 0.8 percent. Of the 24 budget-­ ear projections evaluated, y are fairly stable from year to year, which contributes 17 had an error of 1 percent or less; the largest difference significantly to the accuracy of CBO’s estimates.) 14 An Evaluation of CBO’s Past Outlay Projections November 2017 Table 2 . Summary Measures of the Quality of CBO’s Projections of Mandatory Outlays Percent Budget-Year Projections Sixth-Year Projections Mean Absolute Root Mean Mean Absolute Root Mean Mean Error Error Square Error Mean Error Error Square Error Total Mandatory 2.4 2.9 4.1 2.5 3.4 5.0 Economic errors 0.1 0.6 0.8 0.2 2.0 2.6 Technical errors 2.3 2.7 3.9 2.3 3.0 4.0 Social Security * 0.8 0.9 0.3 2.9 3.6 Economic errors -0.2 0.5 0.7 -0.4 1.7 2.4 Technical errors 0.3 0.5 0.6 0.7 1.8 2.4 Medicarea 1.9 3.3 4.1 7.3 8.0 10.5 Economic errors 0.1 0.3 0.4 1.0 1.6 2.0 Technical errors 1.8 3.1 3.9 6.2 7.1 9.2 Medicaid 2.5 4.0 5.1 10.3 13.5 17.5 Economic errors * 0.4 0.5 * 1.4 1.7 Technical errors 2.5 4.0 5.0 10.3 13.1 17.0 Other Mandatory 7.0 8.5 13.8 -1.9 5.2 6.3 Economic errors 0.7 1.9 2.3 0.8 4.4 5.7 Technical errors 6.2 8.1 13.5 -2.6 6.2 8.2 Source: Congressional Budget Office. Measures are based on budget-­ ear projections for 1993 to 2016 and sixth-­ ear projections for 1997 to 2016 from the baseline projections that CBO y y published between 1992 and 2015. Errors are projected amounts minus actual amounts, expressed as percentages of actual amounts; thus, a negative error indicates an underestimate, and a positive error, an overestimate. The mean error is the arithmetic average of the projection errors. The mean absolute error is the average of the errors with the negative signs removed from the underestimates. The root mean square error is calculated by squaring the projection errors, averaging those squares, and taking the square root of that average. The estimated budgetary effects of legislation enacted after the projections were produced, as well as outlays related to the activities of Fannie Mae and Freddie Mac, are excluded from the errors. * = between −0.05 and 0.05 percent. a.Includes offsetting receipts. As with the budget-­ ear projections, the sixth-­ ear pro- y y of beneficiaries collecting benefits at the beginning of jections of Social Security outlays were too low almost a projection period is large, and the entries to and exits as often as they were too high, resulting in small mean from the program are relatively small. Thus, projections error (0.3 percent). The mean absolute error (2.9 percent) of future caseloads are not particularly volatile. and RMSE (3.6 percent) were larger than those for the budget-­ ear projections but still relatively small, indicating y Projecting the average benefits per beneficiary depends that the sixth-­ ear projections were fairly accurate. Of the y on accurate forecasts of the growth in wages and of 20 projections evaluated, 15 differed from actual outlays by the rate of inflation. Initial Social Security benefit less than the RMSE of 3.6 percent, and only 2 were off by amounts are based on workers’ individual earnings more than 6 percent. histories (indexed to changes in average annual earnings for the U.S. workforce). Because the number of new Factors Underlying the Errors. The large size and stability entrants to the program is relatively small, the main of OASI caseloads enable CBO to project outlays for determinant of the change in average benefits each year is Social Security fairly accurately. For OASI, the number the COLA, which is based on the annual increase, if any, November 2017 An Evaluation of CBO’s Past Outlay Projections 15 Figure 5 . Errors in CBO’s Projections of Social Security Outlays Budget-Year Projections Percent 50 40 30 Mean Projection Error, 1993 to 2016 20 (0%) 10 0 -10 -20 Sixth-Year Projections Percent 50 40 30 Mean Projection Error, 1997 to 2016 20 (0.3%) 10 0 -10 -20 1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 2013 2015 Source: Congressional Budget Office. The years shown are those for which each projection was made. For example, the values shown for 2016 are for the budget-­ ear projection published y in March 2015 and the sixth-­ ear projection published in March 2011. y Errors are projected amounts minus actual amounts, expressed as percentages of actual amounts; thus, a negative error indicates an underestimate, and a positive error, an overestimate. The mean error is the arithmetic average of the projection errors. The estimated budgetary effects of legislation enacted after the projections were produced are excluded from the errors. in the consumer price index for urban wage earners and prepared for those years, CBO overestimated inflation clerical workers. in the consumer price index, on average (as did the Administration and the Blue Chip consensus, an average Most of the overestimates in CBO’s sixth-­ ear projec- y of about 50 private-­ ector forecasts).11 s tions for Social Security occurred in the projections made between 1992 and 1997 (for 1997 to 2002) and were the result of both technical and economic factors. In the underlying economic forecasts that the agency 11. Congressional Budget Office, CBO’s Economic Forecasting Record: 2017 Update (October 2017), www.cbo.gov/publication/53090. 16 An Evaluation of CBO’s Past Outlay Projections November 2017 Most of the underestimates in CBO’s sixth-­ ear projec- y projections of Medicare outlays was 3.3 percent, and the tions of Social Security outlays occurred in the projec- RMSE was 4.1 percent. tions made between 1999 and 2010 (for 2004 to 2015), and like the underestimates in the budget-­ ear projec- y The differences between CBO’s sixth-­ ear projections of y tions, they are largely related to CBO’s forecasts of infla- Medicare outlays and actual outlays were much larger; tion. One notable example is CBO’s March 2004 base- their mean error was 7.3 percent. CBO overestimated line projection, in which the agency projected that Medicare outlays in about three-­ uarters of its sixth-­ ear q y outlays for Social Security would total $618 billion in projections, and on average, those overestimates were 2009. Actual outlays amounted to $678 billion that significantly larger (10.2 percent) than the underesti- year—$60 billion more than CBO projected. According mates (1.5 percent), resulting in a mean absolute error to CBO’s calculations, about three-­ uarters of that q (8.0 percent) that is only modestly higher than the difference was the result of economic factors. Much mean error. The RMSE of the sixth-­ ear projections was y of that error arose because COLAs for 2005 through 10.5 percent, about twice as large as the RMSE of the 2009 ended up being larger than CBO anticipated. For projections of all mandatory outlays, indicating that dif- example, in 2004, CBO projected that the COLA for ferences between projected and actual Medicare outlays 2009 would be 2.2 percent, but the COLA that year was were particularly large in some years. In fact, the errors of unusually high—5.8 percent.12 5 of the 20 sixth-­ ear projections exceeded the RMSE— y those produced for 1999 to 2003. The errors in CBO’s Medicare. Medicare provides subsidized medical insur- projections of Medicare outlays for the sixth year of its ance to the elderly and to some people with disabilities. baseline projection period have ranged from an under- The program has three principal components: Part A estimate of 3.3 percent in its 2004 forecast (for 2009) to (Hospital Insurance), Part B (Medical Insurance, which an overestimate of 23.4 percent in its 1997 forecast (for covers doctors’ services, outpatient care, home health 2001).13 The large errors highlight the growing difficulty services, and other medical services), and Part D (which of projecting spending for Medicare as the projection covers outpatient prescription drugs). Part D coverage period extends further into the future. began in 2006, more than halfway through the period covered in this report. People generally become eligible Factors Underlying the Errors. Most of the differences for Medicare when they reach age 65 or two years after between projected and actual outlays stem from techni- they qualify to receive disability insurance. Nearly all cal factors. For example, growth in Medicare spending Medicare beneficiaries begin participating in the program slowed significantly from 1996 to 2002, and it took soon after they become eligible. In 2016, net outlays for CBO several years to recognize that the change was more Medicare (including the offsetting effects of premium than a short-­ erm deviation from past trends and to fully t payments and other receipts to the program) totaled incorporate the slowdown into its projections of spend- $588 billion. ing. That slowdown appears to have been caused by fac- tors affecting beneficiaries’ demand for care and changes Statistical Bias and Accuracy. CBO overestimated in providers’ behavior. A similar lag between a sustained Medicare outlays for the budget year about 60 percent slowdown and CBO’s recognizing it and incorporating of the time, less often than it overestimated either total it into the agency’s projections of Medicare spending outlays or mandatory outlays as a whole. The mean occurred after the 2007–2009 recession.14 error of the budget-­ ear projections of Medicare outlays y (1.9 percent) is slightly smaller than the corresponding error for total outlays. Of the 24 budget-­ ear projec- y tions, 10 were within 2 percent of the actual outlays for 13. For most years of the analysis, sixth-­ ear projections of Medicare y Medicare, but 1 projection (that was made in 1998 for outlays made by the Centers for Medicare & Medicaid Services 1999) was 11 percent higher than the actual amount (see were similar to those made by CBO. See Centers for Medicare & Medicaid Services, “Trustees Report & Trust Funds” (last Figure 6). The mean absolute error of CBO’s budget-­ eary updated July 13, 2017), https://go.usa.gov/xnCkg. 14. See Michael Levine and Melinda Buntin, Why Has Growth in 12. Congressional Budget Office, The Budget and Economic Outlook: Spending for Fee-­or-­ ervice Medicare Slowed? Working Paper f S Fiscal Years 2005 to 2014 (January 2004), p. 61, www.cbo.gov/ 2013-­ 6 (Congressional Budget Office, August 2013), www.cbo. 0 publication/15179. gov/publication/44513. November 2017 An Evaluation of CBO’s Past Outlay Projections 17 Figure 6 . Errors in CBO’s Projections of Medicare Outlays Budget-Year Projections Percent 50 40 30 Mean Projection Error, 1993 to 2016 20 (1.9%) 10 0 -10 -20 Sixth-Year Projections Percent 50 40 Mean Projection Error, 30 1997 to 2016 (7.3%) 20 10 0 -10 -20 1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 2013 2015 Source: Congressional Budget Office. Medicare outlays include offsetting receipts. The years shown are those for which each projection was made. For example, the values shown for 2016 are for the budget-­ ear projection published y in March 2015 and the sixth-­ ear projection published in March 2011. y Errors are projected amounts minus actual amounts, expressed as percentages of actual amounts; thus, a negative error indicates an underestimate, and a positive error, an overestimate. The mean error is the arithmetic average of the projection errors. The estimated budgetary effects of legislation enacted after the projections were produced are excluded from the errors. The largest overestimate in CBO’s sixth-­ ear projections y new prescription drug benefit, proved to be too high, of Medicare outlays (23.4 percent in its 1997 projec- and adjustments to those original estimates were clas- tion for 2001) was so large because CBO made the sified as technical changes. When technical errors are projection immediately after a period of rapid growth excluded and only differences attributable to economic in the program and before the agency recognized that factors are considered, the mean error of the budget-­ ear y a marked and lasting slowdown had begun. In other projections falls to 0.1 percent, and that of the sixth-­ ear y years, estimates of the effects of legislation, such as the projections drops to 1.0 percent. Those economic errors 18 An Evaluation of CBO’s Past Outlay Projections November 2017 stem from misestimates of various measures of inflation, improved over time. The mean absolute error of projec- which are the basis for adjusting payment rates for most tions made between 1992 and 2000 (for 1997 to 2005) services in the fee-­ or-­ ervice component of Medicare. f s averaged 20.3 percent, but that of projections made for 2006 to 2016 was much smaller: 8.0 percent. Most of Medicaid. Medicaid is a joint federal-­ tate program s the differences between projected and actual spending that pays for health care services for people with low for Medicaid are classified as technical errors. income. State governments operate the program under federal oversight, and the federal government Factors Underlying the Errors. CBO’s projections of reimburses each state a portion of its costs for medical spending for Medicaid have been heavily influenced by care and program administration. The states admin- volatile patterns of past growth. Growth in Medicaid ister the program’s daily operations, reimburse health outlays has fluctuated significantly since 1980: Annual care providers and health plans, and determine which changes in such spending have ranged from an increase eligibility and service options to adopt. Federal outlays of 29 percent to a decrease of 9 percent. From 1980 for Medicaid are affected by whether states choose to to 1993, Medicaid spending increased at an average cover optional groups of enrollees and optional services, annual rate of 14 percent (including three years, 1990 to states’ reimbursement rates for providers, and economic 1992, in which there was extremely high growth). CBO growth. (Medicaid is generally countercyclical—that is, expected that strong growth to persist, and that expecta- spending for Medicaid tends to rise when the economy tion is reflected in the March 1992 baseline projections falters.) In 2016, federal spending for Medicaid totaled it made for 1993 to 1997, which showed average annual $368 billion. growth of about 13 percent in spending for Medicaid. But the growth in spending slowed abruptly beginning Statistical Bias and Accuracy. Even though CBO has in 1994, in part because of legislation designed to limit tended to overestimate federal spending for Medicaid the ability of states to make payments to institutions that about as often as it has spending for Medicare, the differ- care for large numbers of indigent patients. Medicaid ences between projected and actual outlays are larger for spending continued to grow at a slower pace, largely Medicaid. CBO overestimated outlays for Medicaid in because rapid economic growth held down enrollment about 60 percent of the budget-­ ear projections evalu- y in the program. Another factor during that period ated. The errors in the projections ranged from an under- was legislation that changed the relationship between estimate of 5.6 percent (in the projection for 2015) to an Medicaid and people who qualified for income support overestimate of 12.0 percent (in the projection for 1994) programs. For a time, the new laws created confusion and averaged 2.5 percent overall (see Figure 7). The about Medicaid eligibility rules among the state agencies mean absolute error of those projections was 4.0 percent, that administer the program, which led to some enroll- and the RMSE, 5.1 percent. About three-­ uarters of q ees losing their eligibility. Because a significant portion CBO’s budget-­ ear projections were within 5 percent of y of the budgetary effects of those developments were not the actual outlays for Medicaid. recognized at the time legislation was enacted or when economic updates were made to CBO’s baselines, the CBO overestimated outlays for Medicaid more fre- differences between CBO’s projections and actual outlays quently—80 percent of the time—in its sixth-­ ear pro- y were recorded as technical. jections. Those projections were, on average, too high by 10.3 percent. The mean absolute error of the projections By 1997, CBO had lowered its projection of annual was 13.5 percent, significantly higher than the 4.0 per- growth in Medicaid outlays significantly; the agency esti- cent mean absolute error of the budget-­ ear projections, y mated that federal outlays for Medicaid would increase indicating that the challenge of projecting outlays for at an average annual rate of nearly 8 percent over the Medicaid increases as the projection period extends 1998–2007 period.15 On average, actual growth over further into the future. Of the 20 sixth-­ ear projec- y that period was even lower, at about 7 percent annually. tions evaluated, 6 were off by a percentage greater than (However, annual growth in Medicaid spending spiked the mean absolute error, some by substantially larger amounts. The differences exceeded 30 percent on three 15. Congressional Budget Office, The Economic and Budget occasions, contributing to an RMSE of 17.5 percent. Outlook: Fiscal Years 1998–2007 (January 1997), www.cbo.gov/ CBO’s sixth-­ ear projections of Medicaid outlays have y publication/10330. November 2017 An Evaluation of CBO’s Past Outlay Projections 19 Figure 7 . Errors in CBO’s Projections of Medicaid Outlays Budget-Year Projections Percent 50 40 30 Mean Projection Error, 1993 to 2016 20 (2.5%) 10 0 -10 -20 Sixth-Year Projections Percent 50 40 Mean Projection Error, 1997 to 2016 30 (10.3%) 20 10 0 -10 -20 1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 2013 2015 Source: Congressional Budget Office. The years shown are those for which each projection was made. For example, the values shown for 2016 are for the budget-­ ear projection published y in March 2015 and the sixth-­ ear projection published in March 2011. y Errors are projected amounts minus actual amounts, expressed as percentages of actual amounts; thus, a negative error indicates an underestimate, and a positive error, an overestimate. The mean error is the arithmetic average of the projection errors. The estimated budgetary effects of legislation enacted after the projections were produced are excluded from the errors. to 14 percent in 2002—the result of a combination of program, by an average of about 8 percent. Growth in higher prices and rising enrollment and utilization—and spending for Medicaid slowed significantly from 2011 was 9 percent in 2003 and 10 percent in 2004, causing to 2013. Indeed, in 2012, when the additional federal CBO’s budget-­ ear and sixth-­ ear projections for those y y matching funds that were provided to states by the years to be too low.) American Recovery and Reinvestment Act (ARRA) from 2009 to 2011 were no longer available, federal spend- In its sixth-­ ear projections for later years, 2007 through y ing fell by nearly 9 percent. As a result, the sixth-­ ear y 2016, CBO continued to overestimate outlays for the 20 An Evaluation of CBO’s Past Outlay Projections November 2017 projections that CBO made between 2006 and 2008 for Most of the errors in the budget-­ ear projections y those years overestimated outlays by an average of stemmed from overestimates of outlays for programs 13 percent. aimed at mitigating financial crises—the 2007– 2009 recession and the savings and loan crisis of the In its most recent baseline projections, published late 1980s (which continued to affect outlays into the in June 2017, the agency projected that outlays for 1990s). When spending for those programs is removed Medicaid would grow at an average annual rate of from the analysis, the measures of bias and accuracy for 5 percent over the 2018–2027 period.16 The primary the budget-­ ear projections of other mandatory spending y reason that CBO expects Medicaid outlays to grow at a improve dramatically: The mean error falls to 1.7 per- slower rate in the future is that it projects lower growth cent; the mean absolute error, to 4.4 percent; and the in spending for long-­ erm care. t RMSE, to 5.3 percent. Other Mandatory Spending. Other mandatory spend- Together, the economic and technical errors in CBO’s ing includes some federal expenditures for health care sixth-­ ear projections of other mandatory spending are y (other than those for Medicare and Medicaid); outlays smaller than those for the budget-­ ear projections. The y for income security programs, such as unemployment mean error of the sixth-­ ear projections is −1.9 percent, y compensation and nutrition assistance; and spending for meaning that, on average, the projections were less civilian and military retirement, certain veterans’ bene- than actual outlays. The overestimates in the sixth-­ ear y fits, agricultural support, and higher education (includ- projections of other mandatory spending were fewer and ing the net cost of student loans). It also includes outlays smaller than those in the budget-­ ear projections. Of the y related to programs designed to stabilize the financial 20 sixth-­ ear projections, 14 (or 70 percent) were lower y sector, such as deposit insurance and, during the most than actual spending. The mean absolute error of the recent recession, TARP. Outlays for such programs can sixth-­ ear projections of other mandatory spending was y swing positive or negative and are significantly affected 5.2 percent, and the RMSE was 6.3 percent. by financial crises. In 2016, other mandatory spending amounted to $575 billion. Factors Underlying the Errors. The errors in the budget-­ year projections were heavily influenced by spending Statistical Bias and Accuracy. Budget-­ ear projections y for programs that help resolve financial crises. About of other mandatory spending have tended to be sig- 70 percent of the cumulative difference between the nificantly too high. The mean error of the budget-­ ear y budget-­ ear projections of other mandatory spending y projections of such spending (after the adjustments to made for 1993 through 2016 and the actual outlays account for the effects of legislative changes) is 7.0 per- recorded in those years stems from misestimates of cent—significantly greater than the mean errors of the outlays for deposit insurance and for the Troubled Asset budget-­ ear projections of outlays for Social Security, y Relief Program, which was established in 2008. Nearly Medicare, and Medicaid. CBO overestimated outlays for 90 percent of the errors in the budget-­ ear projections y other mandatory spending in about three-­ uarters of the q of outlays for those two programs is concentrated in projections, and the average overestimate was 10.3 per- the projections made for 1993, 2010, and 2011 (see cent. Moreover, CBO overestimated other mandatory Figure 8). For the sixth-­ ear projections of other man- y spending in every budget-­ ear projection made for the y datory spending, the errors are mostly from projections years 2009 to 2016. Underestimates were fewer and of spending on programs other than TARP and deposit smaller. In the six projections in which CBO underes- insurance. timated other mandatory spending for the budget year, the average underestimate was 3.0 percent. The mean The largest percentage error in a single budget-­ ear y absolute error of the budget-­ ear projections of other y projection of other mandatory spending was 48 percent mandatory spending was 8.5 percent, and the RMSE, in CBO’s 1992 projection for 1993. (Actual outlays were 13.8 percent. $80 billion less than CBO projected, before any adjust- ments to account for legislative differences.) Most of that error arose from a delay in the funding of the Resolution 16. Congressional Budget Office. An Update to the Budget and Economic Outlook: 2017 to 2027 (June 2017), www.cbo.gov/ Trust Corporation (RTC). publication/52801. November 2017 An Evaluation of CBO’s Past Outlay Projections 21 Figure 8 . Errors in CBO’s Projections of Other Mandatory Spending Budget-Year Projections Percent 60 Deposit Insurance and TARP 30 Other Mandatory, Excluding All Other Mandatory Deposit Insurance and TARP 0 -30 Sixth-Year Projections Percent 60 30 0 -30 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012 2014 2016 Source: Congressional Budget Office. The years shown are those for which each projection was made. For example, the values shown for 2016 are for the budget-­ ear projection published y in March 2015 and the sixth-­ ear projection published in March 2011. y Errors are projected amounts minus actual amounts, expressed as percentages of actual amounts; thus, a negative error indicates an underestimate, and a positive error, an overestimate. The mean error is the arithmetic average of the projection errors. The estimated budgetary effects of legislation enacted after the projections were produced, as well as outlays related to the activities of Fannie Mae and Freddie Mac, are excluded from the errors. TARP = Troubled Asset Relief Program. The RTC, which was established in 1989, was charged activities of the RTC to be a mandatory obligation and with purchasing and liquidating assets (such as mortgage estimated its operations on an unconstrained basis, antic- loans) of savings and loan associations that were declared ipating that the program would need additional resources insolvent by the Office of Thrift Supervision in the late beyond those that had been provided under current law 1980s. (In 1995, the duties of the RTC were transferred at the time. Organizations such as the RTC that aim to to the Federal Deposit Insurance Corporation.) Both resolve failed financial institutions have both positive and CBO and the Administration considered outlays for the negative outlays: Amounts that the government spends 22 An Evaluation of CBO’s Past Outlay Projections November 2017 to purchase assets are recorded as positive outlays, and budget estimates for such programs, the riskiness of amounts it receives for selling those assets are recorded credit-­ elated activity is reevaluated each year, and any r as negative outlays—or offsetting receipts. (Premiums changes to the assessment of whether a program would for certain programs, such as deposit insurance, are also result in costs or savings to the government are recorded reflected on the federal budget as offsetting receipts.) that year as “credit reestimates.” Those changes are con- Over the 1993–2016 period, net outlays for the RTC sidered technical, and they may be either positive or neg- and deposit insurance totaled −$127 billion, about ative. Net outlays for TARP have amounted to roughly 1½ percent of all other mandatory spending; however, $24 billion since the program was created, but both large that total reflects several years of substantial outlays offset positive and large negative outlays were recorded from by other years of sizable receipts. 2009 to 2012. In 1992, CBO projected that insurance losses and Initially, CBO estimated that the entire $700 billion disbursements of the RTC would exceed the proceeds made available to TARP would be disbursed and that, from asset sales in 1993 and result in net outlays of as a result of significantly weakened financial markets, $50 billion. However, funding for the program was much of the value of the assets purchased under TARP interrupted from April 1992 to December 1993, and as would not be recovered. As the Treasury’s approach to a result, the RTC had limited authority to incur losses. It implementing the program became clear and financial could, however, continue to sell off its portfolio of assets, markets stabilized to the point that several large banks and consequently, the RTC recorded negative outlays of were able to repurchase preferred stock that they had $28 billion in 1993. sold to the government, the program’s net cost to the government and its effect on the deficit declined. Most For the next 15 years or so, the discrepancies between of the discrepancies between the 2009 and 2010 budget-­ CBO’s projections of other mandatory spending and year projections for 2010 and 2011 and the actual the actual amounts were relatively small. But the errors outlays for those years arose because TARP actually spiked again in the wake of the 2007–2009 recession, disbursed far less than originally anticipated and because primarily because of overestimates in projections of the disbursements that were made were not as costly as outlays for deposit insurance and TARP. Over succes- initially envisioned by CBO and the Administration’s sive baselines, CBO revised downward its budget-­ ear y Office of Management and Budget (OMB). Because of projections of outlays for deposit insurance for 2010 and those changes, OMB revised its estimate of the cost of 2011—by $68 billion and by $22 billion, respectively— the credit-­ elated activities under TARP, decreasing the r to account for changes in the timing of expenditures and estimated cost of the program by $117 billion in 2010 receipts stemming from the failure of federally insured and by $58 billion in 2011 by recording negative outlays banks, thrift institutions, and credit unions during the of those amounts in those years.18 financial crisis. During those years, the Federal Deposit Insurance Corporation also accelerated premium pay- Aside from errors related to outlays for the RTC, deposit ments (which are recorded as negative outlays) from cov- insurance, and TARP, some of the largest errors in CBO’s ered institutions to replenish the balances of the Deposit budget-­ ear and sixth-­ ear projections of other manda- y y Insurance Fund, which had been depleted as a result of tory spending include those associated with the following the financial crisis. categories of outlays: CBO’s estimates of outlays for TARP in those projections ■■ Outlays for countercyclical income security programs were also too high. TARP was created by the Emergency (such as unemployment compensation), projections Economic Stabilization Act of 2008, which granted the Treasury broad authority to invest up to $700 billion in 18. Comparing estimates that account for market risk with actual troubled financial assets. The law specified that costs of budgetary outlays can be considered conceptually inconsistent, TARP should be recorded in the budget by calculating because the estimates include premiums to account for market risk, whereas the actual amounts do not incorporate that the present value of the program’s anticipated costs, additional cost. Even if losses were exactly as forecast, there adjusted for market risk.17 Under the rules that govern would still be downward reestimates after the transactions were completed and the actual cash flows were discounted 17. A present value is a single number that expresses a flow of current using risk-­ ree rates. For more information, see Congressional f or future income or payments in terms of an equivalent lump Budget Office, Fair-­ alue Accounting for Federal Credit Programs V sum received or paid at a specific point in time. (March 2012), www.cbo.gov/publication/43027. November 2017 An Evaluation of CBO’s Past Outlay Projections 23 of which are sensitive to CBO’s forecasts of economic past 25 years, discretionary spending has decreased as a factors, such as the unemployment rate, and tend to share of total federal outlays, falling from 39 percent in be too low at the start of recessions and too high in 1992 to 31 percent in 2016. periods of recovery; Because funding for most discretionary programs is pro- ■■ The subsidy costs of student loans, revisions to which vided in annual appropriation acts, the largest differences can be positive or negative and are recorded in the between CBO’s projections of discretionary spending federal budget each year using the Administration’s and the actual amounts recorded tend to be legislative in estimates; and nature. They result from differences between the appro- priated amounts and the baseline projections and reflect ■■ Receipts from the Federal Communications any adjustments made to caps on discretionary funding, Commission’s periodic auctions of licenses for use of such as those currently in place through 2021. CBO’s the electromagnetic spectrum. baseline projections generally reflect the assumption, specified in section 257 of the Deficit Control Act, that Some other components of other mandatory spending funding keeps pace with inflation. For years in which are more easily estimated, and errors in CBO’s pro- caps on discretionary funding were in place, the pro- jections of outlays for those programs are smaller. In jected funding levels reflect those caps. particular, federal civil service and military retirement programs exhibit characteristics similar to Social Security, Almost all of the small economic errors in projections including fairly stable caseloads, detailed data on benefit of discretionary outlays arise because of errors in the payments, and annual cost-­ f-­iving adjustments. Errors o l agency’s forecast of inflation. The remaining differences in projections of outlays for those retirement programs between projections of discretionary spending and actual thus follow a pattern similar to those in projections for amounts are classified as technical and largely stem from Social Security—economic errors stemming from errors the agency’s misestimating the rate at which budget in forecasts of inflation are generally offset by technical authority would be spent. errors, so the mean error is near zero. CBO’s budget-­ ear projections of discretionary outlays y Discretionary Outlays have generally been too high, but by only a relatively Funding for most discretionary programs is provided small amount; the mean error of those projections is through annual appropriation acts in the form of budget 1.3 percent (see Table 3). Nevertheless, of the 24 such authority—that is, the authority to incur financial estimates produced for 1993 to 2016, 20 exceeded the obligations that result in federal outlays. Discretionary actual outlays in those years. As for accuracy, the mean spending is divided into two main categories: defense absolute error of those projections was 1.4 percent, less and nondefense. Lawmakers appropriate funding each than that of budget-­ ear projections of any other cate- y year for the majority of defense and nondefense activ- gory of spending except for Social Security. An error of ities; the latter include activities and programs such as that size in CBO’s June 2017 budget-­ ear projection of y law enforcement, transportation, national parks, disaster $1.2 trillion in discretionary outlays would mean that relief, and foreign aid. (A few programs receive multiyear the agency misestimated actual discretionary outlays for appropriations.) Depending on the activity or program, 2018 by $17 billion. federal spending that arises from that budget authority can occur quickly (to pay salaries, for example) or slowly An RMSE of 1.7 percent—only slightly higher than the (to pay for long-­ erm research and construction). Thus, t mean absolute error—indicates that the errors of CBO’s discretionary outlays recorded each year come not only budget-­ ear projections of discretionary spending were y from new budget authority but also from prior appro- narrowly dispersed. Only 8 of the 24 projections evalu- priations. To estimate the amount of outlays that would ated differed from actual outlays by more than 1.7 per- result from projected discretionary funding levels, CBO cent. (But 7 of those 8 projections were made in the past typically evaluates factors such as the historical rate at 10 years.) The largest difference was between the projec- which budget authority has been obligated and spent tion published in 2009 and the actual outlays recorded as well as the amount of outstanding budget authority in 2010; CBO overestimated discretionary outlays by available from previous years’ appropriations. Over the 3.5 percent, or $47 billion, mostly because the funds 24 An Evaluation of CBO’s Past Outlay Projections November 2017 Table 3 . Summary Measures of the Quality of CBO’s Projections of Discretionary Outlays Percent Budget-Year Projections Sixth-Year Projections Mean Absolute Root Mean Mean Absolute Root Mean Mean Error Error Square Error Mean Error Error Square Error By Category of Error (For baseline projections published since 1992)a Total Discretionary 1.3 1.4 1.7 1.9 2.0 2.5 Economic errors -0.1 0.1 0.2 -0.1 0.8 1.1 Technical errors 1.4 1.4 1.7 2.0 2.2 2.6 By Category of Spending (For baseline projections published since 1998)b Defense 0.9 1.4 1.7 1.4 1.9 2.4 Nondefense 2.1 2.4 2.7 2.9 3.0 3.4 Source: Congressional Budget Office. Errors are projected amounts minus actual amounts, expressed as percentages of actual amounts; thus, a negative error indicates an underestimate, and a positive error, an overestimate. The mean error is the arithmetic average of the projection errors. The mean absolute error is the average of the errors with the negative signs removed from the underestimates. The root mean square error is calculated by squaring the projection errors, averaging those squares, and taking the square root of that average. The estimated budgetary effects of legislation enacted after the projections were produced are excluded from the errors. a.Measures are based on budget-­ ear projections for 1993 to 2016 and sixth-­ ear projections for 1997 to 2016. The data necessary to calculate the y y projection errors for total discretionary outlays are not available for projections made before 1992. b.Measures are based on budget-­ ear projections for 1999 to 2016 and sixth-­ ear projections for 2003 to 2016. The data necessary to calculate the y y projection errors for defense and nondefense discretionary spending are not available for projections made before 1998. provided by the American Recovery and Reinvestment both defense spending and nondefense spending were Act of 2009 were spent more slowly than CBO antici- overestimated. pated (see Figure 9). Currently, funding for the two categories of discretionary The sixth-­ ear projections of discretionary spending y spending are subject to separate caps (with adjustments follow a pattern similar to that of the budget-­ ear pro- y for spending related to overseas contingency operations, jections in terms of both statistical bias and accuracy. Of disaster relief, designated emergencies, and initiatives the 20 sixth-­ ear projections of discretionary spending y designed to enhance program integrity). From 1992 to evaluated, 18 exceeded actual amounts. Nevertheless, 1998, however, total funding for discretionary programs, the measures of bias and accuracy have generally been regardless of whether it was for defense or nondefense much smaller than those for projections of other major purposes, was subject to a single cap for some years. categories of spending. On average, CBO overestimated CBO did not consistently make separate budget-­ ear and y discretionary spending by 1.9 percent in its sixth-­ ear y sixth-­ ear projections for the two categories; it projected y projections; the mean absolute error of the projections only total discretionary spending. As a result, CBO was 2.0 percent. The variation in the errors was relatively does not have the historical data needed to distinguish narrow. In its sixth-­ ear projections, CBO underesti- y between defense and nondefense spending in projections mated discretionary outlays by as much as 0.8 percent made before 1998.19 In general, CBO has tended to (in its 2004 projection for 2009) and overestimated them by as much as 5.2 percent (in its 2008 projection 19. For information on the accuracy of the estimates of discretionary for 2013), resulting in an RMSE of 2.5 percent. The spending that CBO prepared for appropriation bills before 1998, see Congressional Budget Office, An Analysis of CBO’s largest misestimates tended to be in projections in which Outlay Estimates for Appropriation Bills, Fiscal Years 1993–1997 (October 1998), www.cbo.gov/publication/11255. November 2017 An Evaluation of CBO’s Past Outlay Projections 25 Figure 9 . Errors in CBO’s Projections of Total Discretionary Outlays Budget-Year Projections Percent 50 40 30 Mean Projection Error, 1993 to 2016 20 (1.3%) 10 0 -10 -20 Sixth-Year Projections Percent 50 40 30 Mean Projection Error, 1997 to 2016 20 (1.9%) 10 0 -10 -20 1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 2013 2015 Source: Congressional Budget Office. The years shown are those for which each projection was made. For example, the values shown for 2016 are for the budget-­ ear projection published y in March 2015 and the sixth-­ ear projection published in March 2011. y Errors are projected amounts minus actual amounts, expressed as percentages of actual amounts; thus, a negative error indicates an underestimate, and a positive error, an overestimate. The mean error is the arithmetic average of the projection errors. The estimated budgetary effects of legislation enacted after the projections were produced are excluded from the errors. overestimate both defense and nondefense outlays since systems and other equipment, and pay and benefits for 1998, but it has overestimated nondefense spending by military personnel—as well as with the atomic energy larger amounts. activities of the Department of Energy. A small por- tion of defense spending is for various defense-­ elated r Defense Spending. Defense spending consists of the activities in other agencies. In 2016, outlays for defense outlays associated with the military activities of the totaled $585 billion and accounted for almost half of all Department of Defense (DoD)—including those for discretionary outlays and 15 percent of total outlays. operation and maintenance, procurement of weapon 26 An Evaluation of CBO’s Past Outlay Projections November 2017 Statistical Bias and Accuracy. Overall, CBO has tended spending for defense for the next few years. In part, that to slightly overestimate defense spending for the budget is because uncertainty over future funding created by year. Of the 18 budget-­ ear projections made for 1999 y the Budget Control Act of 2011 led DoD to spend its through 2016, 16 were overestimates, leading to a mean budget authority more slowly than CBO had expected. error of 0.9 percent. All but one of the overestimates In its projections for 2015 and 2016, CBO still overes- were made in consecutive years between 2001 and 2015 timated defense discretionary spending, but it did so by (in projections for 2002 to 2016), and the mean error considerably smaller amounts—less than 1 percent of of those projections was 1.3 percent. Despite CBO’s actual outlays. tendency to overestimate such spending, its budget-­ ear y projections of defense outlays have been fairly accu- Although projections of all categories of defense spend- rate: A third of the projections fell within 1 percent of ing contributed to the overestimates of total defense the actual outlays, and none differed from the actual spending for the years 2002 to 2016, some categories amounts by more than 3.3 percent. The mean absolute affected the accuracy of CBO’s projections more than error of those projections was 1.4 percent. others. For example, projections of spending for mili- tary personnel have been both relatively unbiased and Much like CBO’s sixth-­ ear projections for discretionary y accurate. The mean error and mean absolute error of the spending as a whole, CBO’s sixth-­ ear projections of y projections of such spending made between 2001 and defense spending tended to be slightly high (about two-­ 2015 (for 2002 to 2016) were both less than 1.0 percent. thirds of the projections were overestimates) but were By contrast, projections of spending for operation and among the most accurate of the projections of all the var- maintenance, including DoD’s working capital funds, ious categories of spending evaluated. On average, CBO and for procurement have differed more significantly overestimated defense spending by 1.4 percent in the from actual spending. The majority of those projections sixth-­ ear projections it made between 1998 and 2011 y have been too high for the past 14 years. (for 2003 to 2016). The mean absolute error of those projections was 1.9 percent. The projection errors ranged Nondefense Spending. Nondefense programs cover a from an underestimate of 1.7 percent (in the projection wide array of government activities, including those made in 1999 for 2004) to an overestimate of 4.4 per- related to primary and secondary education, transpor- cent (in the projection published in 2008 for 2013). tation, federal law enforcement, and pay and benefits for federal employees. In 2016, discretionary outlays for Factors Underlying the Errors. Although CBO underesti- nondefense purposes totaled $600 billion and accounted mated the rate at which defense budget authority would for just over half of all discretionary outlays and 16 per- be spent in the budget-­ ear projections it made between y cent of total outlays. 1999 and 2000 (for 2000 and 2001), it overestimated the rate in the projections it made between 2001 and Statistical Bias and Accuracy. CBO overestimated non- 2015 (for 2002 to 2016). One reason for those overes- defense spending for the budget year as frequently as timates was that significant changes in funding levels it did defense spending in the projections made for and uncertainty about future funding amounts resulted 1999 to 2016, but the magnitude of the differences in patterns of spending that differed from those CBO was larger, on average, for nondefense spending. Of the had anticipated. The differences between projected and 18 budget-­ ear projections evaluated, CBO overesti- y actual outlays for 2002 to 2016 largely stemmed from mated nondefense discretionary spending in 15 of them sharp increases in funding in the early and mid-­ 000s for 2 (including the 6 most recent projections, those for 2010 overseas contingency operations (war-­ elated activities, r to 2016). The mean error of the budget-­ ear projections y primarily in Iraq and Afghanistan) and changes in the of nondefense discretionary spending was 2.1 percent, tempo of total spending. In its projections for 2003 to 1.2 percentage points higher than the mean error of the 2010, CBO overestimated defense outlays, partly because budget-­ ear projections of defense discretionary outlays. y the number and timing of appropriation bills varied Furthermore, the standard measures of accuracy indicate during that period, making it difficult to estimate the that CBO’s nondefense discretionary projections have rates at which the regularly increasing budget authority been somewhat less accurate than its defense projec- would be spent. After defense funding began to decline tions. The mean absolute error (2.4 percent) and the in 2010, CBO continued to overestimate budget-­ eary RMSE (2.7 percent) of CBO’s budget-­ ear projections y November 2017 An Evaluation of CBO’s Past Outlay Projections 27 of nondefense outlays were 1 percentage point higher CBO’s projections of discretionary outlays for Pell grants than the corresponding measures for the agency’s defense also vary in terms of accuracy more than other discre- projections. tionary programs. The program was created to improve the access of low-­ncome students to postsecondary i The differences between CBO’s sixth-­ ear projections y education. CBO bases its projections of spending for of nondefense discretionary spending and actual out- the program on estimates of the number of Pell grant lays follow a pattern similar to those of its budget-­ ear y recipients in each year. The agency underestimated costs projections (that is, large errors tend to occur in budget-­ in projections it made just before economic downturns year projections and sixth-­ ear projections for the same y (because as unemployment rises during recessions, years). Although the sixth-­ ear projections of nondefense y enrollment in higher education increases) and overesti- spending were a bit worse than the agency’s sixth-­ ear y mated costs for periods when the economy was stabiliz- defense projections in terms of both statistical bias and ing following a recession.21 accuracy, they were nevertheless slightly more accurate and less statistically biased than the agencies’ projections Some of the errors in CBO’s projections of nondefense of most other categories of spending. The sixth-­ ear pro- y spending for 2010 to 2016 can be attributed to the jections of nondefense discretionary outlays exceeded the agency’s overestimating the rate at which ARRA funding actual amounts recorded in every year except 2009, but would be spent. (Enacted in 2009 in response to signifi- they did so by relatively small amounts: 9 of the 14 pro- cant weakness in the economy, that legislation provided jections were within 3.0 percent of the actual amounts. discretionary budget authority of $268 billion for a The overestimates in the projections made between 2005 number of governmental activities that year and smaller and 2011 (for 2010 to 2016) were a little larger, averag- amounts of funding in the following years.) A smaller ing 4.0 percent. portion of the funding appropriated for ARRA was spent in the first few years after enactment than CBO Factors Underlying the Errors. Although most of the over- had originally estimated (even though CBO’s original estimates for nondefense outlays were relatively small, estimates took account of lags in disbursing funding that projections of spending for certain programs have tended are typical for the programs like those funded by that to have larger errors. For instance, projected budget-­ ear y legislation). collections for the Federal Housing Administration’s (FHA’s) single-­ amily mortgage insurance program have f Additionally, uncertainty around sequestration in differed by more than 10 percent from actual collections 2013 may have led some agencies to temporarily slow on multiple occasions. The budgetary effect recorded for obligations and spending.22 CBO’s projections of nonde- that program each year is the estimated present value fense discretionary outlays were thus higher than actual of the government’s receipts from fees and its expenses spending for 2012 and 2013 because, when preparing its from defaults (net of recoveries) over many years from budget-­ ear and sixth-­ ear projections, CBO had antici- y y the guarantees issued in that year.20 Those amounts pated that budget authority would be spent at historical are recorded as offsetting collections—that is, funds rates. collected from other government accounts or from the public that are treated as negative budget authority and Net Interest Outlays outlays. Most of the difference between CBO’s projec- Net interest outlays include interest paid on Treasury tion and the actual amount is considered technical, and securities and other interest that the government pays that difference tends to be larger when there are signifi- (interest paid for late refunds issued by the Internal cant fluctuations in the housing market. 21. For a detailed explanation of why the Federal Pell Grant Program’s costs increase during recessions, see Congressional 20. The program provides guarantees for first-­ ime homebuyers and t Budget Office, The Federal Pell Grant Program: Recent other borrowers who might otherwise find it difficult to obtain a Growth and Policy Options (September 2013), www.cbo.gov/ mortgage, and the FHA is required to set fees for its insurance so publication/44448. that projected receipts exceed the projected losses from defaults on mortgages insured under the program. See Congressional 22. Sequestration is an enforcement mechanism by which the Budget Office, “FHA’s Single-­ amily Mortgage Guarantee F President orders the cancellation of budgetary resources to Program: Budgetary Cost or Savings?” CBO Blog (October 21, enforce the discretionary spending caps specified in the Budget 2013), www.cbo.gov/publication/44628. Control Act of 2011. 28 An Evaluation of CBO’s Past Outlay Projections November 2017 Revenue Service, for example) minus the interest that CBO’s forecast of interest rates (see Table 4).23 Since the it collects from various sources (such as from states that early 2000s, CBO—like most forecasters—has consis- pay the federal unemployment trust fund interest on tently overestimated interest rates. The most significant advances they received when the balances of their state errors were in forecasts for years following recessions, unemployment accounts were insufficient to pay benefits when the Federal Reserve lowered interest rates more in a timely fashion). Net interest costs are determined than expected and kept them low for longer than antic- primarily by the size and composition of the govern- ipated. All told, economic factors accounted for a mean ment’s debt and by market interest rates. As a share of error of 5.6 percent and a mean absolute error of 7.5 per- total outlays, net interest spending has decreased by more cent in budget-­ ear projections of interest costs. y than half since 1992, falling from 14 percent in that year (when the benchmark 10-­ ear interest rate averaged y When those economic errors are excluded and only 7.2 percent) to 6 percent in 2016 (when the 10-­ ear rate y differences attributable to technical factors are consid- averaged only 1.9 percent). ered, the mean error of the budget-­ ear projections falls y to −0.4 percent, and the mean absolute error, to 3.0 per- Statistical Bias and Accuracy. CBO has tended to overes- cent. Those technical factors include the amount of each timate net interest outlays for the budget year—the pro- type of security issued by the Treasury, intergovernmental jections (after the adjustment to account for the effects interest payments, and amounts received by nonbudget- of new legislation) exceeded actual outlays in 19 of the ary credit financing accounts. Also included in CBO’s past 24 years (see Figure 10). The mean error of the accounting of technical errors are the differences in debt-­ budget-­ ear projections of net interest was 5.2 percent, y service costs that arise from the federal government’s which is higher than the mean error of the projections of borrowing more or less than CBO projected because of all other spending categories except for other mandatory technical errors in estimates of other types of outlays.24 spending. Economic factors also explain most of the net interest CBO’s budget-­ ear projections of net interest outlays y overestimates in sixth-­ ear projections, contributing y have been less accurate than those of most other cate- 50 percentage points to each measure of error. The effect gories of spending. The errors were widely dispersed, of errors in the economic forecast was most pronounced ranging from an underestimate of 13.9 percent (in the after the 2007–2009 recession, resulting in an aver- agency’s 2009 projection for 2010) to an overestimate age error of about 88 percent in the projections made of 19.2 percent (in the 2014 projection for 2015). The between 2004 and 2011 for 2009 to 2016. mean absolute error of the projections was 7.7 percent, and the RMSE, 9.4 percent. If CBO’s June 2017 pro- CBO’s tendency to overestimate interest rates for the jection of $307 billion of net interest had an error of the years after the recessions in 2001 and from 2007 to 2009 same magnitude as that mean absolute error, actual net stems partly from the fact that CBO did not anticipate interest outlays in 2018 would be $24 billion higher or the recessions when it was preparing its projections for lower than CBO projected. those years and partly from the unusually slow recov- eries that followed the recessions. Interest rates were Of all the projections of specific categories of spending historically low after those recessions, and CBO did evaluated in this report, CBO’s projections of net interest not anticipate that the rates would remain below their for the sixth year produced the largest percentage differ- historical averages for such an extended period of time ences between projected and actual outlays. The mean (see Figure 11). error of the sixth-­ ear projections was 54.0 percent, y and the mean absolute error, 56.0 percent. The corre- For example, by 2009, the rate on 3-­ onth Treasury m sponding errors for Medicaid, the category with the next bills, which averaged 4.3 percent over the previous largest errors, were 10.3 percent and 13.5 percent. 23. For more information on how CBO forecasts interest rates, see Factors Underlying the Errors. The discrepancies between Congressional Budget Office, CBO’s Economic Forecasting Record: the projected and actual amounts are primarily attribut- 2017 Update (October 2017), www.cbo.gov/publication/53090. able to economic factors, namely to overestimations in 24. Debt service refers to a change in interest payments resulting from a change in estimates of the deficit. November 2017 An Evaluation of CBO’s Past Outlay Projections 29 Figure 10 . Errors in CBO’s Projections of Net Interest Outlays Budget-Year Projections Percent 125 100 75 Mean Projection Error, 50 1993 to 2016 (5.2%) 25 0 -25 Sixth-Year Projections Percent 125 Mean Projection Error, 100 1997 to 2016 (54.0%) 75 50 25 0 -25 1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 2013 2015 Source: Congressional Budget Office. The years shown are those for which each projection was made. For example, the values shown for 2016 are for the budget-­ ear projection published y in March 2015 and the sixth-­ ear projection published in March 2011. y Errors are projected amounts minus actual amounts, expressed as percentages of actual amounts; thus, a negative error indicates an underestimate, and a positive error, an overestimate. The mean error is the arithmetic average of the projection errors. The estimated budgetary effects of legislation enacted after the projections were produced are excluded from the errors. 20 years, had plummeted to an annual average of 2.1 percent.25 CBO’s overestimates of short-­ erm rates t 0.2 percent. However, the rates that CBO used to have been exacerbated in the years since then by the construct its budget-­ ear projection of outlays for y 2009 were those from the economic forecast it 25. CBO typically updates its economic forecast only twice per year, made in March 2008, before the National Bureau of once in January and once during the summer; however, because Economic Research recognized that a recession had economic conditions were rapidly deteriorating in early 2008, the agency also updated its forecast before preparing its March begun in December 2007, so they remained elevated at baseline budget projections that year. 30 An Evaluation of CBO’s Past Outlay Projections November 2017 Table 4 . Summary Measures of the Quality of CBO’s Projections of Outlays for Debt Service and Other Net Interest Percent Budget-Year Projections Sixth-Year Projections Mean Absolute Root Mean Mean Absolute Root Mean Mean Error Error Square Error Mean Error Error Square Error Total Net Interest 5.2 7.7 9.4 54.0 56.0 67.0 Economic errors 5.6 7.5 9.5 50.0 50.0 61.9 Rate effectsa 5.4 7.5 9.5 47.2 47.2 61.0 Debt serviceb 0.1 0.6 0.9 2.8 8.2 10.7 Technical errors -0.4 3.0 4.7 4.0 13.5 15.6 Mix of securities and otherc -1.0 2.2 4.1 1.1 4.2 6.2 Debt serviceb 0.6 1.6 2.2 2.8 14.1 16.8 Source: Congressional Budget Office. Measures are based on budget-­ ear projections for 1993 to 2016 and sixth-­ ear projections for 1997 to 2016 from the baseline projections that CBO y y published between 1992 and 2015. Errors are projected amounts minus actual amounts, expressed as percentages of actual amounts; thus, a negative error indicates an underestimate, and a positive error, an overestimate. The mean error is the arithmetic average of the projection errors. The mean absolute error is the average of the errors with the negative signs removed from the underestimates. The root mean square error is calculated by squaring the projection errors, averaging those squares, and taking the square root of that average. The estimated budgetary effects of legislation enacted after the projections were produced are excluded from the errors. a.Errors in projections of net interest outlays caused by misestimates of interest rates. b.Errors stemming from changes in interest payments that result from differences between projections of deficits and of outstanding debt and the actual amounts. c.Errors resulting from misestimates of the relative amounts of the types of securities the Treasury will issue to finance deficits. For example, the time to maturity of those securities or the schedule of interest payments on them might have differed from CBO’s estimates. unusually slow recovery and by the Federal Reserve’s used in its sixth-­ ear baseline projections for those years y decision to maintain the federal funds rate at nearly averaged 4.7 percent and 5.7 percent, respectively. zero for an unprecedented length of time.26 Short-­term borrowing rates, which are closely tied to the rates set by In addition to those recessions and unusually slow recov- the Federal Reserve, have averaged just 0.1 percent since eries, several other factors account for CBO’s having 2008. Similarly, rates on 10-­ ear notes have averaged y persistently overestimated interest rates since the early 2.5 percent since 2008, well below their previous histori- 2000s. Over the past 15 years or so, inflation, growth in cal average of 5.9 percent. hours worked, and growth in labor productivity—all of which are important determinants of interest rates—have Like its forecasts of interest rates two years out, CBO’s been below expectations. Low foreign interest rates, forecasts of interest rates for six years in the future have heightened concern about global growth, and increased tended to be too high. Over the past 20 years, actual demand for Treasury securities as a hedge against possible rates for 3-­ onth Treasury bills have averaged 2.2 per- m adverse economic outcomes have further contributed cent, and those for the 10-­ ear Treasury note have y to low interest rates, especially since the most recent averaged 4.0 percent. By comparison, the rates that CBO recession ended in 2009. The Federal Reserve’s policy of quantitative easing—that is, its practice of directly purchasing long-­ erm Treasury securities, mortgage-­ t 26. The federal funds rate is the interest rate that financial backed securities, and agency debt—has also contributed institutions charge each other for overnight loans of their to the unusually low long-­ erm interest rates since that t monetary reserves. The Federal Reserve uses the federal funds rate recession. to conduct monetary policy. November 2017 An Evaluation of CBO’s Past Outlay Projections 31 Figure 11 . Comparison of CBO’s Sixth-­ ear Forecasts of Interest Rates on 3-­ onth Treasury Bills and Y M 10-­ ear Treasury Notes With Actual Rates Y 3-Month Treasury Bill Percent 8 6 Sixth-Year Forecast 4 2 Actual Rate 0 10-Year Treasury Note Percent 8 Sixth-Year Forecast 6 4 Actual Rate 2 0 1997 1999 2001 2003 2005 2007 2009 2011 2013 2015 Source: Congressional Budget Office. The years shown are those for which each projection was made. For example, the rates shown for 2016 are for the sixth-­ ear projection published in y January 2011. Overly optimistic forecasts of real (inflation-­ djusted) a Even when projections for the most recent period of output growth account for the bulk of the overestimates extremely low rates are excluded, the remaining projec- in the interest rate forecasts during the expansionary tions still indicate that CBO had a tendency to overes- period of the mid-­ 000s. Given the positive outlook 2 timate interest rates. The average actual rates for short-­ for growth in the economy, CBO (and other forecast- term and long-­ erm debt from 1996 to 2008 were both t ers) expected the Federal Reserve to try to temper that roughly 1 percentage point lower than the forecast rates. growth and the inflationary pressures that could result As a result of those misestimates as well as differences in from it by raising interest rates. the amount of federal debt, that period accounted for large errors in projections of interest outlays. The mean 32 An Evaluation of CBO’s Past Outlay Projections November 2017 error of the sixth-­ ear projections of net interest outlays y Comparing the Quality of CBO’s and the made between 1992 and 2003 (for 1997 to 2008) was Administration’s Projections 28.8 percent, and the mean absolute error of those pro- The Administration has tended to overestimate total jections was 32.1 percent. outlays for the budget year by a slightly greater amount than CBO has. The Administration overestimated total Comparison of CBO’s and the outlays for 1993 to 2016, on average, by 2.9 percent, Administration’s Outlay Projections about 0.7 percentage points greater than the mean error In addition to evaluating its own record, CBO com- of CBO’s budget-­ ear projections. The mean absolute y pared its budget-­ ear projections with those released by y error of the Administration’s budget-­ ear projections was y the Administration. The Administration’s budget-­ ear y 3.0 percent, and the RMSE was 3.6 percent—both were projections of total outlays, as well as its projections of larger than CBO’s corresponding errors of 2.3 percent mandatory, discretionary, and net interest outlays, have and 2.9 percent. been close to CBO’s projections in most years since 1992. But CBO’s projections for the major spending Through the projection for 2005, the Administration’s categories—particularly those made between 2005 and projections were similar to CBO’s in terms of statisti- 2015 (for 2006 to 2016)—were slightly more accurate cal bias and accuracy. Since 2005, both CBO and the and less statistically biased than the Administration’s Administration have continued to overestimate budget-­ corresponding projections. year outlays, but the Administration’s overestimates have tended to be greater than CBO’s. The budget-­ ear y Adjusting the Administration’s Data for Comparison projections that CBO and the Administration published Like CBO, the Administration publishes baseline pro- between 1992 and 2004 (for 1993 to 2005) had simi- jections when it submits the budget to the Congress. In lar mean errors: 1.9 percent for CBO’s projections and many years, the Administration published an adjusted 2.1 percent for the Administration’s (see Figure 12). But baseline that incorporated the estimated effects on the errors in the projections for 2006 to 2016 diverged outlays and revenues of certain proposals that had not by a much greater amount. Whereas the Administration yet been enacted, along with other adjustments. To overestimated outlays for those years by an average of make those projections comparable to its own baseline 3.8 percent of the actual amounts, CBO overestimated projections, CBO removed those adjustments from the by an average of 2.4 percent. Administration’s estimates. Additionally, CBO removed outlays related to Fannie Mae and Freddie Mac from The mean absolute errors for the two sets of projections both its and the Administration’s baseline projections. followed a similar pattern. For budget-­ ear projections y for 1993 to 2005, the Administration’s mean absolute CBO was able to compare only its budget-­ ear base- y error was 2.3 percent, nearly the same as CBO’s mean line projections with those of the Administration, not absolute error of 2.2 percent. But for projections for its sixth-­ ear projections. In the Analytical Perspectives y 2006 to 2016, the Administration’s mean absolute error volume that it publishes annually along with the budget, was 3.8 percent, more than 1 percentage point greater the Administration provides information on differences than CBO’s mean absolute error. between its budget-­ ear baseline projections and actual y outlays for broad categories in the budget; however, it Mandatory Spending. As with projections of total does not generally provide details for specific programs. outlays, the Administration’s projections of total As a result, CBO was able to compare only the projec- mandatory spending were slightly more statistically tions of total outlays and the projections of the three biased than CBO’s, and they have tended to be less main subcategories of spending—mandatory, discre- accurate. By CBO’s calculations, the mean error of tionary, and net interest. The Administration has not the Administration’s budget-­ ear projections of total y published detailed information on the accuracy of its mandatory outlays for 1993 to 2016 was 3.1 percent, projections beyond the budget year, so CBO could not 0.7 percentage points larger than the mean error of compare the Administration’s sixth-­ ear projections with y CBO’s corresponding projections. The pattern of errors its own. in the Administration’s and CBO’s projections of man- datory spending was similar, but the Administration’s projections tended to have larger errors. As a result, the November 2017 An Evaluation of CBO’s Past Outlay Projections 33 Figure 12 . Comparison of Mean Errors in CBO’s and the Administration’s Budget-­ ear Projections of Outlays Y Percent 8 8 1993 to 2005 2006 to 2016 6 6 Administration 4 4 Administration CBO CBO 2 2 0 0 Total Outlays Mandatory Discretionary Net Interest Total Outlays Mandatory Discretionary Net Interest Sources: Congressional Budget Office; Office of Management and Budget. The years shown are those for which each projection was made. For example, the right panel shows the mean errors for the budget-­ ear projections y for 2006 to 2016 that were made between 2005 and 2015. Errors are projected amounts minus actual amounts, expressed as percentages of actual amounts; thus, a negative error indicates an underestimate, and a positive error, an overestimate. The mean error is the arithmetic average of the projection errors. The estimated budgetary effects of legislation enacted after the projections were produced, as well as outlays related to the activities of Fannie Mae and Freddie Mac, are excluded from the errors. mean absolute error of the Administration’s mandatory Administration overestimated discretionary spending spending projections (3.7 percent) was larger than that by less than 1 percent. In the projections it published of CBO’s projections (2.9 percent). between 2005 and 2015 (for 2006 to 2016), the Administration overestimated total discretionary spend- Discretionary Spending. The Administration’s budget-­ ing by an average of 4.5 percent because both defense year projections that differed most from CBO’s were and nondefense programs spent appropriated funds more those of discretionary outlays. The Administration’s slowly than it had expected. CBO also overestimated budget-­ ear projections of total discretionary spending y discretionary spending in the budget-­ ear projections y were more statistically biased than CBO’s, and they it published during that period, but it did so by only have tended to be less accurate. The Administration 2.0 percent, on average. overestimated discretionary outlays in the projections it made for 1993 to 2016 by 2.3 percent, on average; the Net Interest Outlays. The Administration’s budget-­ eary mean error of CBO’s projections of such outlays was projections of net interest have followed a pattern similar 1.3 percent. By CBO’s calculation, the mean absolute to CBO’s. The Administration and CBO generally made error of the Administration’s budget-­ ear projections of y large overestimates in the same years, although CBO’s discretionary spending was 2.6 percent, and the RMSE projections have been somewhat more accurate. The was 3.5 percent—both were more than 1 percentage mean error of the Administration’s budget-­ ear projec- y point larger than the corresponding errors of CBO’s tions of net interest for 1993 to 2016 was 5.7 percent, projections. and the mean absolute error, 8.7 percent. The mean error of CBO’s projections for the same years was 5.2 percent, The Administration’s errors have been larger in recent and the mean absolute error, 7.7 percent. Most of the years. In the projections they published between 1992 difference between the measures of bias and accuracy and 2004 (for 1993 to 2005), both CBO and the of the two sets of projections is attributable to errors in 34 An Evaluation of CBO’s Past Outlay Projections November 2017 projections of net interest for years after the most recent 2016, the Administration overestimated net interest out- recession. In the projections they made between 1992 lays by 7.6 percent, on average—more than 1 percentage and 2004 (for 1993 to 2005), both the Administration point higher than the mean error of CBO’s correspond- and CBO overestimated net interest outlays by 4.1 per- ing projections. cent, on average. In the projections it made for 2006 to appendix Appendix: Calculation of Economic and Technical Errors in CBO’s Projections of Outlays Each time that it publishes a new baseline, the rather, they reflect CBO’s best judgment about what the Congressional Budget Office includes an explanation of economy and the budget would look like in future years the changes it made to its previous projections.1 Those under existing laws. That approach allows the baseline changes are divided into three categories—legislative, to serve as a neutral benchmark against which the effects economic, and technical. Legislative changes are the of proposed legislation or alternative policies can be estimates of the budgetary effects of laws enacted since measured. However, the practice makes it difficult to the agency published its previous baseline. CBO classifies evaluate the accuracy of the agency’s projections because changes as economic if they result from an update to actual outlays are affected by legislation enacted since the its economic forecast, such as revisions to the agency’s projections were made. projections of gross domestic product, interest rates, or the unemployment rate. Technical changes are those For this analysis, CBO was concerned primarily with that arise from factors other than new laws or updated those differences between its projections of outlays and assessments of the economy. They stem from such devel- the actual amounts recorded that stem from economic opments as the incorporation of new information or data and technical factors. To isolate those differences, the from federal agencies, changes made to the way programs agency adjusted its published baseline projections to are administered, updates to regulations, or improve- include the estimated effects of legislation enacted after ments in modeling techniques. the original baselines were released.2 The data that CBO generates as part of the process of For example, CBO’s March 2015 projection of total preparing its baselines form the basis of the calculations outlays for 2016 was $3,925 billion (see Table A-1). in this report. To illustrate CBO’s method of adjusting However, legislation enacted after March 2015 increased its projections to compare them with actual outlays, this outlays for 2016 by $49 billion, according to CBO’s appendix looks at the legislative, economic, and technical estimates. Adding those outlays to CBO’s projection for changes CBO made to its projections for 2016 between 2016 brought the estimated total to $3,974 billion. March 2015, when the projections were first published, and when the Treasury Department released the actual The largest legislative changes affecting the projection outlay amounts recorded in the fiscal year. for 2016 occurred between the publication of CBO’s August 2015 and January 2016 baselines.3 During Adjustments to CBO’s Published Projections that period, CBO revised its estimates of discretionary As mandated by law, CBO constructs its 10-­ ear pro- y outlays for 2016 upward by $25 billion, mostly because jections of federal spending and revenues under the assumption that current laws will generally remain 2. In addition, for this analysis, CBO reclassified the incorporation unchanged. The agency’s projections are thus not of the automatic enforcement procedures in the Budget intended to be a prediction of budgetary outcomes; Enforcement Act of 2011 as legislative changes. The budgetary effects of triggering those procedures were originally incorporated as technical changes in CBO’s January 2012 baseline projections. 1. Updates to the agency’s baseline projections can be found 3. Congressional Budget Office, The Budget and Economic Outlook: in the “Budget and Economic Outlook and Updates” 2016 to 2026 (January 2016), www.cbo.gov/publication/51129, section of CBO’s website (www.cbo.gov/about/products/ and An Update to the Budget and Economic Outlook: 2015 to 2025 major-recurring-reports#1). (August 2015), www.cbo.gov/publication/50724. 36 An Evaluation of CBO’s Past Outlay Projections November 2017 Table A-1 . How CBO Adjusts Its Budget-­ ear Projections of Total Outlays to Compare Them With Actual Spending: Y An Example Using the March 2015 Projections for 2016 Billions of Dollars 2016 March 2015 Baseline Projection of Total Outlays 3,925 Adjustments to Include Estimated Effects of Legislation March 2015 to August 2015 19 August 2015 to January 2016 30 January 2016 to March 2016 * March 2016 to August 2016 a Difference between August 2016 baseline and actual amount a ____ Subtotal 49 Adjustment to Exclude Outlays Associated With Fannie Mae and Freddie Mac -3 Total Adjustments 46 Adjusted March 2015 Baseline Projection of Total Outlays 3,971 Economic Changes March 2015 to August 2015 -26 August 2015 to January 2016 -16 January 2016 to March 2016b * March 2016 to August 2016 -4 Difference between August 2016 baseline and actual amount -4 ____ Subtotal -51 Technical Changes March 2015 to August 2015 10 August 2015 to January 2016 * January 2016 to March 2016 -23 March 2016 to August 2016 -31 Difference between August 2016 baseline and actual amount -10 ____ Subtotal -54 Actual Total Outlays, Excluding Those Associated With Fannie Mae and Freddie Mac 3,867 Source: Congressional Budget Office; Office of Management and Budget. The periods identified above correspond to the intervals between successive baseline projections, which CBO typically publishes each winter, early spring, and summer. The economic and technical changes shown in the reports on the budget and economic outlook that CBO published between August 2015 and August 2016 do not add up to the total amount for such changes shown here because this analysis adds the difference between CBO’s August 2016 baseline projection (the final projection for fiscal year 2016) and actual outlays to those previously published amounts. In addition, this analysis excludes from the technical changes the budgetary transactions of Fannie Mae and Freddie Mac, whereas the earlier reports included them. * = between −$500 million and $500 million. a.CBO did not estimate any legislative effects for fiscal year 2016 between March 2016 and the end of the fiscal year. b.CBO typically updates its economic forecast only twice a year, once before preparing its January baseline and again before the August baseline. However, CBO modified the economic forecast that it had prepared in early December 2015 later that same month to account for new legislation— too late in the process to fully incorporate the modifications in the budget projections it published in January 2016. APPENDIX An Evaluation of CBO’s Past Outlay Projections 37 the Bipartisan Budget Act of 2015 (Public Law 114-­ Calculation of Economic and 74) had increased the caps on budget authority for Technical Errors defense and nondefense programs, and the resulting After adjusting for new legislation and transactions appropriations were equal to those limits in 2016. Most with Fannie Mae and Freddie Mac, CBO calculated the of the other revisions to CBO’s projection of outlays remaining differences between its projections and actual for 2016 that were attributable to legislative changes outlays (that is, the economic and technical errors) by stemmed from the enactment of the Medicare Access adding up all revisions of each type that were recorded in and CHIP Reauthorization Act of 2015 (P.L. 114-­ 0), 1 the baseline updates as well as the difference between the which increased outlays for Medicare, Medicaid, and the final revised projections and the actual amounts recorded Children’s Health Insurance Program in 2016. Other leg- at the end of the fiscal year. islation enacted between March 2015 and the end of the 2016 fiscal year had relatively small effects on projections In March 2015, CBO overestimated outlays for 2016 by of outlays for 2016. $104 billion, or 2.7 percent (after the baseline projec- tions were adjusted to account for new legislation and CBO also adjusted its projections and the actual totals the outlays associated with Fannie Mae and Freddie to remove outlays for Fannie Mae and Freddie Mac (two Mac). That percentage difference is the error in the institutions that help finance the majority of mortgages budget-­ ear projection for 2016. That error was attrib- y in the United States). Those outlays were removed utable to economic and technical factors in almost equal because CBO and the Administration treat transactions proportions. between those two institutions and the Treasury differ- ently. The Administration—in the President’s budget and Economic Errors in data compiled by the Department of the Treasury— Economic factors were responsible for $51 bil- treats Fannie Mae and Freddie Mac as nongovernmental lion, or 1.3 percent, of the difference between the organizations and records payments between the two March 2015 budget-­ ear projection of total outlays for y entities and the Treasury on a cash basis. By contrast, 2016 and actual spending. Most of that difference was CBO projects the budgetary impact of the two entities’ accounted for by downward revisions that CBO made operations as if they were conducted by a federal agency between March 2015 and January 2016. because of the degree of management and financial control that the government exercises over them. CBO An overestimate of net interest costs contributed $37 bil- therefore estimates the net lifetime costs—that is, the lion to the overestimate attributable to economic factors, subsidy costs—of the new loans and guarantees that and an overestimate of mandatory spending added the entities are expected to issue and counts those costs $14 billion. Much of the overestimate of net interest as federal outlays in the year of issuance for all years occurred because CBO’s forecast of interest rates was too after the current one in its projections. (In the current high. The agency lowered that forecast before preparing year, CBO projects payments between Fannie Mae and both its August 2015 and January 2016 baselines and Freddie Mac and the Treasury on a cash basis.) In its revised its projection of net interest for 2016 accord- March 2015 baseline, CBO estimated that the net life- ingly, lowering it by about $14 billion in each of those time costs of new loans and guarantees issued by the two baselines. CBO’s March 2015 projection of interest costs entities in fiscal year 2016 would total $3 billion; that also reflected an overestimate in the agency’s forecast of amount was removed from CBO’s baseline projection of inflation, which boosted the projected costs of Treasury total outlays for the year. inflation-­protected securities. After the effects of new legislation were accounted for As for the agency’s projections of mandatory spending, and the outlays associated with Fannie Mae and Freddie almost half of the total economic error was in its projec- Mac were removed, CBO’s March 2015 projection tions of outlays for Social Security. The largest discrep- of total outlays for 2016 amounted to $3,971 billion. ancy occurred because CBO overestimated the cost-­ f-­ o Actual total outlays, excluding $14 billion in offset- living adjustment for 2016 in its March 2015 baseline, ting receipts from Fannie Mae and Freddie Mac, were projecting that the adjustment would be 0.9 percent. $3,867 billion. However, there was no cost-­ f-­iving adjustment in o l 2016 because the rate of inflation was lower than 38 An Evaluation of CBO’s Past Outlay Projections November 2017 originally forecast. CBO had anticipated that outcome An additional $6 billion overestimate classified as tech- and reduced its estimate of the adjustment to zero in its nical stemmed from agencies’ updating their estimates of August 2015 baseline. the net costs of several credit programs. Each year various agencies revise their estimates of the subsidy costs for Technical Errors outstanding credit programs. Such revisions—known as Technical factors explain $54 billion of the overes- credit reestimates—reflect changing economic conditions timate of total outlays for 2016 in CBO’s adjusted and actual loan performance and are recorded in the March 2015 baseline projection. CBO classified as tech- budget as an increase or decrease in mandatory outlays. nical a $35 billion overestimate of mandatory spending, In 2016, agencies determined that previous subsidy esti- an $18 billion overestimate of discretionary outlays, and mates were too high and therefore reported downward an overestimate of net interest payments of less than half credit reestimates of $6 billion. CBO does not estimate a billion dollars. future credit reestimates because it does not attempt to predict when the various agencies will revise their The $35 billion overestimate in CBO’s projections of credit estimates. The $6 billion in downward reestimates mandatory spending stemmed from a $41 billion overes- reported in 2016 is therefore considered a technical error timate for programs other than Medicaid, Medicare, and in CBO’s March 2015 projection of other mandatory Social Security that was offset by an underestimate of spending. Medicare spending. For example, for technical reasons, the agency overestimated outlays for health insurance Because of technical factors, CBO overestimated both purchased through the marketplaces established under defense and nondefense discretionary outlays for 2016 in the Affordable Care Act and related spending by $17 bil- its March 2015 baseline—defense spending by about lion and outlays for income security programs by $8 bil- $4 billion and nondefense spending by $14 billion. The lion in its March 2015 baseline. In the baselines CBO overestimates were not concentrated in any particular produced after March 2015, the agency made technical program. adjustments to spending projections for those programs and others, which brought the estimates closer to actual spending and reduced its overall projection of mandatory outlays. List of Tables and Figures Tables 1. Summary Measures of the Quality of CBO’s Projections of Outlays 10 2. Summary Measures of the Quality of CBO’s Projections of Mandatory Outlays 14 3. Summary Measures of the Quality of CBO’s Projections of Discretionary Outlays 24 4. Summary Measures of the Quality of CBO’s Projections of Outlays for Debt Service and Other Net Interest 30 A-1. How CBO Adjusts Its Budget-­ ear Projections of Total Outlays to Y Compare Them With Actual Spending: An Example Using the March 2015 Projections for 2016 36 Figures 1. Sample Timeline for Measuring Errors in CBO’s Projections of Outlays for the Budget Year 7 2. Share of Total Spending and of Projection Errors for Total Outlays, by Category of Spending 9 3. Errors in CBO’s Projections of Total Outlays 11 4. Errors in CBO’s Projections of Mandatory Outlays 13 5. Errors in CBO’s Projections of Social Security Outlays 15 6. Errors in CBO’s Projections of Medicare Outlays 17 7. Errors in CBO’s Projections of Medicaid Outlays 19 8. Errors in CBO’s Projections of Other Mandatory Spending 21 9. Errors in CBO’s Projections of Total Discretionary Outlays 25 10. Errors in CBO’s Projections of Net Interest Outlays 29 11. Comparison of CBO’s Sixth-­ ear Forecasts of Interest Rates on Y 3-­ onth Treasury Bills and 10-­ ear Treasury Notes With Actual Rates M Y 31 12. Comparison of Mean Errors in CBO’s and the Administration’s Budget-­ ear Projections of Outlays Y 33 About This Document Each year, typically in January, the Congressional Budget Office issues a report on the state of the budget and the economy. The agency usually updates that report in March and August. The first set of updated projections often serve as the basis for CBO’s estimates of legislation as well as the Congress’s budget resolution for the year to come. This document provides background information on the statistical bias and accuracy of the outlay projections included in those reports. In keeping with CBO’s mandate to provide objective, impartial analysis, the report makes no recommendations. Christina Hawley Anthony, Barry Blom, Meredith Decker, Avi Lerner, Amber Marcellino, and Dan Ready wrote the report with guidance from Theresa Gullo, Holly Harvey, Jeffrey Holland (formerly of CBO), and Sam Papenfuss. Mark Booth, Tom Bradley, Kent Christensen, Kim Kowalewski, Noah Meyerson, Claire Sleigh, Emily Stern, and Rob Stewart provided helpful comments. Megan Carroll, Jacob Fabian, Ann Futrell, Sarah Puro, Dawn Sauter Regan, Lara Robillard, and Victoria Rosenboom (formerly of CBO) fact-­ hecked the report. c Jim Hearn of the Public Company Accounting Oversight Board, Kevin Kliesen of the Federal Reserve Bank of St. Louis, and Paul Van de Water of the Center on Budget and Policy Priorities also reviewed the report. The assistance of external reviewers implies no responsibility for the final product, which rests solely with CBO. Wendy Edelberg, Mark Hadley, Jeffrey Kling, and Robert Sunshine reviewed the report. Bo Peery edited it, and Jorge Salazar prepared it for publication. The report is available on CBO’s website (www.cbo.gov/publication/53328). Keith Hall Director November 2017