Shibboleths in Modeling Public Policy1

Publication Details

Richard P. O'Neill

Federal Energy Regulatory Commission

Over the last 25 years, the principal direction of the government's modeling of public policy in the energy area has been to analyze the effects of more market-driven and incentive-driven outcomes. Similar efforts have been made in health care. Many of the regulations hastily put in place in the 1970s after the oil embargoes and price run-ups are still being unraveled today. As a result, paradigms that have been accepted for more than a century are changing.

People consume some services and commodities without knowing the price, then pay the bill without fully understanding how the price was determined. One of those commodities has been electricity. Attempts to create market forces in this area have been made since the 1970s, when legislation was passed to begin to open up natural gas and electricity markets. But paradigms shifts are not easy.

One of the most interesting paradigm shifts in history took place during the tenure of Pope Urban VIII. In 1530, Copernicus published a book stating that the earth revolved around the sun. At the time, Church theology held that the earth was the fixed, immovable center of the universe. But Galileo read Copernicus, looked at the skies through his newly invented telescope, and agreed with him. Soon after, Galileo published the Dialogue, his most controversial work, which presented the arguments for and against heliocentrism. The Inquisition banned the book, and Galileo was found guilty of heresy and condemned to spend the rest of his life under house arrest (in a palace). Writings by Copernicus and Galileo were placed on the Church's index of forbidden works, where they remained for more than 200 years. All of this happened despite the fact that the Church had been debating the truth of Copernican discoveries for decades and despite Pope Urban VIII's admiration for Galileo.

Western science doesn't always get things right the first time. Priestley, who is usually credited with discovering oxygen, went to his grave believing in the phlogiston theory of combustion. History shows that paradigm shifts are difficult.

In our day, the move from a centralized, regulated system of energy to a more decentralized system based on competitive incentives has been very difficult. In some ways, the electricity system is like a hospital, a centrally run institution with many agents (e.g., doctors, nurses, and administrators) operating with different incentives—some of them at odds with the overall mission of the organization. In the energy system, a key goal has been getting the incentives right. Very quickly, one realizes that entrenched cultural beliefs present major barriers to change. Some social scientists believe that cultural paradigm shifts can take several generations.

Modelers are often compared to carpenters with hammers looking for nails; if they find a screw instead of a nail, they pound it anyway. Many in the energy field assume that the market was in a Nash equilibrium (i.e., entities may not collude explicitly, but they do collude implicitly) in part because of a popular book, successful movie and Nobel prize on Nash's life and work. Much modeling was done based on Nash's theory, but it turned out that there was explicit collusion in Western markets.

Different jargons and market dialects often present barriers to paradigm shifts. Enormous efforts have been made to introduce competition and competitive market paradigms over the last quarter century, but many people in the field have been trained to think and speak in cost-of-service or cost-base dialect. In fact, a huge segment of the industry still talks and thinks in this dialect. Like Eskimos who have many words for snow but few words for heat stroke, market participants trying to talk about auctions and market processes do not have the appropriate grammar or vocabulary to discuss the topic.

Small, unwritten rules matter. In New Zealand, unwritten rules for government-owned electricity corporations were an important factor in market outcomes. In other countries, when government-owned electric assets were sold to private interests, often one of the first things the CEOs did was increase their own salaries and buy private jets—hardly a confidence builder for competition.

Some popular analogies in policy discussions about electricity market reforms are to the natural gas market and the air traffic control system. Natural gas is a poor analogy for the electricity market, because natural gas can be economically stored. There is no simple equivalent of a valve in the electricity system. Some have argued that the electricity system controller should be like an air traffic controller, meaning there should be no central market-control process. These same people seem to want rules that direct behavior with no regard for cost. Most of these analogies are misleading because, even though they argue for market forces, they also lead either to greater socialization or easier manipulation of the market.

The California electricity market is an interesting case study. In the early 1990s, there was a great deal of discussion about liberalizing the California market. Technical people spent two years designing the new market, but when politicians got involved, they threw out all of the market designs and cut a deal that emerged as legislation (AB1890), which was passed unanimously. Environmentalists agreed to support the plan if they could be given money for their programs. Marketers pushed for a bad market design to ensure more profits for themselves later.

Employees of utilities who had been involved in the discussions before the compromise were told not to discuss the previously proposed market designs. A number of staff at the Federal Energy Regulatory Commission noted that the model did not provide good incentives and that the process could get out of control. For two years, the legislatively mandated market design underwent constant changes. Confusion and gaming masked what was to happen—no new generators, demand growth, and a drought. In the end, the state risked everything on a model designed by a legislative committee.

Prices in the wholesale electricity market are typically in the range of $30 to $50 per megawatt-hour. In California and most of the west, spot prices remained more than $100 per megawatt-hour for months. Then, suddenly, prices dropped, for a number of reasons: the utilities lost their credit ratings; the governor bought power under high-priced, long-term contracts; new generators came on line; and the weather changed.

Since then, hundreds of thousands of dollars have been spent on litigation to determine what went wrong and who should be punished. Interestingly, the smartest people turned out to be the consumer representatives who had been skeptical about the program and had called for a retail rate freeze and guaranteed rate reductions. Initially, they came out looking good, but in the end, the rate freeze contributed to the market disequilibrium. Consumers will be paying for these mistakes for years to come.

In retrospect, it can be seen that another player in this market caused a lot of the mischief. That player constantly proposed and funded campaigns for market designs that would not work, but that it could take advantage of. That player supported an array of approaches that all eventually failed. That player was Enron, the darling of Wall Street at the time. After Enron went bankrupt, it released a memo outlining the strategies it had used to manipulate the market in California. Wall Street lost its exuberance and abandoned its desire for Enron clones.

Some interesting questions have been asked about the California experience. For instance, was this a six-sigma event (i.e., was the outcome a low-probability event)? Or was this a one-sigma or two-sigma event (i.e., the wrong paradigm)? If we look at the debacle as an enormous experiment, some good may come of it. Theory can predict performance, and theory predicted that the market design in California would fail under stress. The California experiment cost billions of dollars, but it did prove the strength of the theory.

Another lesson we learned from California is that incentives matter. Financial incentives are very important, in energy markets and in the health care market. Fee-for-service versus salary systems can change the incentives significantly. You always have to ask what incentive a doctor or dentist has to cure a problem; non-monetary incentives are far more important in health care than in the energy industry. Dedication is a very important factor in health care and hard to model. Another issue is the principal agent problem—who acts on the patient's behalf? Programming this behavior into a model is difficult.

Bad incentives yield bad practice. Enron is a case in point in the energy market. Good accounting theory is mark-to-market accounting. In theory, mark-to-market accounting is theoretically correct, but in practice, in thin markets the market price is prone to manipulation. In fact, it can often simply be made up. Enron took advantage of this gap and produced a grossly distorted set of accounts.

In energy and health care markets, good market designs can yield benefits. Here are some don'ts in market design: don't oversimplify; don't create gaming opportunities; don't favor large players; don't use market jargon or model jargon to explain things; don't ignore the extreme model outcomes, because outliers often provide interesting information and stories. The modeler should be immersed in the problem. Outsiders may be brought in to help, but somebody must intimately understand the model, as well as the process being modeled.

Clients should understand that models provide insights. Forecast are often wrong, but everyone must forecast. Forecasting often involves insight into possible outcomes, rather than numbers. The model must be tuned based on experience. A model should not be a black box. It should be available to people; it should be testable; and it should be auditable.

A lesson for health care modeling is that modeling can be a useful tool, but a healthy skepticism is necessary for success.



This report does not necessarily reflect the view of the Federal Energy Regulatory Commission.