3Assessing the Value of Research at IBM

Publication Details

Joseph M. Jasinski

IBM Research

Thomas J. Watson Research Center


Almost all scientists, I assert, believe that knowledge is a good thing, and I also believe that the public at large agrees with this statement. Most people, especially scientists, would also agree that the goal of scientific research is to produce new knowledge through discovery. The fly in the ointment is, of course, that gaining knowledge through scientific research costs money. It requires an investment in the future, and necessarily in the unknown. Whether the supplier of this investment money is a foundation, a corporation, or the American people through their government, the investor naturally wants and often demands to know that the investment is paying off—thus, the need for assessment of research.

In this paper I describe how IBM Research attempts to perform such an assessment of its own results against its mission and responsibilities to IBM. Since assessment is meaningless taken out of context, I begin with a brief overview of IBM and IBM Research and the relationship between IBM and its research organization. Needless to say, IBM is not a part of the chemical industry, although there are many chemicals and chemical processes involved in the production of computer chips, displays, and disk drives. The assessment process is, however, fairly general, and the key ideas should be transferable regardless of the specific nature of the research work.

International Business Machines Corporation is a large, global, vertically integrated information technology (IT) company, with 1996 revenues of $75.9 billion. The corporation can be viewed as consisting of four functional blocks: corporate headquarters, manufacturing and development businesses, marketing and service operations, and research. The manufacturing and development businesses include servers, personal computers, software, semiconductor chips, and magnetic storage. Marketing and services includes marketing and sales function for all product lines, as well as industry-specific technology solutions and services such as operating computer systems for large corporate customers.

IBM Research is a largely autonomous function that reports fairly directly to the CEO's office and is closely aligned with and coupled to (but not tied organizationally to) all other parts of the corporation, as well as IBM customers and the worldwide scientific and technical community. The technical work within IBM Research is segmented into four major parts: silicon technology; magnetic storage; computer systems and software; and services, applications, and solutions (see Figure 3.1). The first three areas are self-explanatory. The fourth is a segment of IBM Research that works closely with IBM customers to develop integrated solutions (hardware and software) for major industry segments such as health care, finance, manufacturing, utilities, and so on. Its goal is to bring IBM Research into direct contact with IBM's leading-edge corporate customers—in other words, to provide Research with an entry into the demands of the marketplace.

FIGURE 3.1. Segmentation of technical activities within IBM Research.


Segmentation of technical activities within IBM Research.

The fifth segment of technical work in IBM Research is science. As illustrated in Figure 3.1, our science programs touch all four "aligned" segments in a major way. We also conduct long-term exploratory work that is not directly aligned with a current product or development plan. This diverse mixture of work including basic science must and can be assessed. The assessment process, particularly assessment of the longer-term work, is the main topic of this paper.

IBM Research

IBM Research is just over 50 years old, tracing its roots to the Watson Computing Laboratory at Columbia University, which started in 1945. Box 3.1 lists some of the major contributions of IBM Research to science and technology. IBM scientists have garnered three U.S. National Medals of Science and six National Medals of Technology, shared three physics Nobel Prizes, and received numerous other forms of recognition in the external scientific and technical community, including memberships in the U.S. National Academies of Sciences and Engineering.

Box Icon

BOX 3.1

Major Contributions by IBM Research to Science and Technology. One-Device Memory Cell Reduced Instruction Set Computing

IBM Research currently consists of approximately 2,800 employees worldwide, or about 1 percent of IBM's total global work force. Research currently operates facilities at seven locations around the globe: the Thomas J. Watson Research Center in Yorktown Heights, N.Y.; the Almaden Research Center in San Jose, Calif; the Zurich Research Laboratory in Ruschlikon, Switzerland; the Tokyo Research Laboratory in Yamato, Japan; the Haifa Research Laboratory in Haifa, Israel; the China Research Laboratory in Beijing, China; and the Austin Research Laboratory in Austin, Texas. The China and Austin laboratories have opened within the past few years, and IBM has announced plans to open an eighth location, the India Solutions Research Center in Delhi, India.

In the early 1990s, IBM Corporation experienced a sudden, dramatic change in its business environment. In one year it went from a highly profitable enterprise to one in financial and strategic disarray, largely owing to major changes in the information technology industry. In 1992, IBM reevaluated everything about itself, including what its key business should be, what its long-term strategy should be, and whether or not a central research organization was still important to its future. This necessarily caused IBM Research to evaluate and change itself. This evaluation led to a definition of the Research role as "vital to IBM's future success," and resulted in the organization as it exists, healthily, today.

As its high-level mission, IBM Research seeks to transform basic scientific and engineering knowledge in fields such as chemistry, physics, mathematics, electrical engineering, and computer science into new products and technologies that affect IBM's existing businesses or lead to new business directions for IBM (Figure 3.2). The values and assets we create in the process include technical leadership in selected fields and in IBM's products and patents; a vision of the future in information technology; people who are technical leaders and in many cases future executive leaders for IBM; support for our marketing efforts by direct technical interaction with leading-edge customers; and luster—the substantiated perception by the industry, and the world at large, that IBM really does have the best people, the best ideas, and therefore the best IT products.

FIGURE 3.2. IBM Research seeks to transform IBM by advancing knowledge in critical scientific and engineering disciplines and using that knowledge to have a positive impact on IBM businesses.


IBM Research seeks to transform IBM by advancing knowledge in critical scientific and engineering disciplines and using that knowledge to have a positive impact on IBM businesses.

The Research Division Business Process

A schematic of the annual business process cycle for IBM Research is shown in Figure 3.3. There are four major elements. These are the technology outlook, or "Ten Year Outlook"; the environment, vision, and strategy; the technical plan; and the end-of-year assessment. The process begins in the January-February time frame, when senior management and key technical staff take a look at the Information Technology (IT) industry and try to envision where it is headed and what IT will look like over the next 10 years. This outlook drives the strategy process for each Research segment. The strategic goals and opportunities we identify, combined with our vision of the industry, are the main elements used to develop our financial request for the next year. The budget is then negotiated with corporate headquarters. These three factors—outlook, strategy, and budget—then lead to a technical plan of work for the following year. The technical plan closes in the fall. It defines our intended work for the following year and sets a number of the parameters against which we will assess our performance that year. The cycle closes with the assessment phase of the current year's work in December, against goals set in the current technical plan, which was created in the previous business plan cycle.

FIGURE 3.3. Outline of the annual business cycle for IBM Research.


Outline of the annual business cycle for IBM Research.

The assessment consists of a report on seven distinct items that constitute Research's "contract" with the IBM Corporation. The preface to the report states:

This report on the IBM Research Division results is focused on values created for and having impact on IBM. Success and impact are measured by success in delivering technology to IBM's customers, except for long-term science, where the measurement is major contributions to knowledge in a specific area.

The following assessment items are described more fully below:

  •  Measurement projects,
  •  Accomplishments,
  •  Partnership assessments,
  •  Intellectual property,
  •  External recognition,
  •  Financial goals, and
  •  Division goals.

The weighted outcomes of these items determine the overall assessment of Research's performance for the year. The only immediate impact of this result is that it is a factor in determining the annual bonus for all Research employees. Obviously, there is a long-term feedback between the assessment process and the rest of the cycle, but it does not, for example, determine the next year's budget in a direct way.

Of the seven assessment items, three are easily measured and easily described. These are intellectual property, external recognition, and financial goals. We count the number of patents filed by Research in the year and make a semiquantitative assessment of their potential value. The result is then compared with a target set by the IBM Intellectual Property and Licensing organization. We tabulate external scientific and technical awards and honors received by our staff, and we determine whether or not we met our budget. The division goals item is a set of high-level goals that are essentially a confidential performance plan for the executive management of the Research Division and are assessed by the CEO's office. Partnership assessments are "customer feedback surveys" from the other parts of IBM with which we work. The remaining two items, measurement projects and accomplishments, are designed to assess the results of our technical work in a very broad way.

Measurement projects are defined during the technical plan cycle and are typically projects that have a well-defined goal, a "deliverable," and a "customer" to receive and evaluate the deliverable. The deliverable could be an advanced chip design, a new process for fabricating devices, a piece of software, or a prototype solution, such as an advanced voice recognition system. The customer might be another part of IBM, an external customer, or (in the case of work done with government contract funding) the funding agency. The intent of the measurement project item is to assess our ability to deliver what we promise to our business partners.

Finally, and perhaps of most relevance to this proceeding, we assess our exploratory work through the item referred to as accomplishments. Accomplishments are used to assess our entire portfolio of technical work, including, and perhaps of most relevance to these proceedings, our exploratory and basic science work. Accomplishments represent a modest subset (on an annual basis) of IBM Research's exploratory technology and basic science work. Candidate accomplishments are proposed by line management and evaluated and "graded" according to a set of well-defined criteria. There are three classes of accomplishments: accomplishment, outstanding accomplishment, and extraordinary accomplishment (Table 3.1 ). The criteria for each grade are given below. There are two sets of criteria for each level: one that applies to work intended to have an impact on the scientific and technical community, and one that applies to work intended to have direct IBM impact. These criteria can be orthogonal, although the very best accomplishments—the ones that are truly extraordinary—will frequently meet the criteria in both categories.

TABLE 3.1. Criteria for IBM Research's Three Classes of Accomplishments.


Criteria for IBM Research's Three Classes of Accomplishments.

An extraordinary accomplishment is a rare event. Very few, if any, accomplishments will achieve the extraordinary (E) rating when first presented as an accomplishment. It is expected that the E rating will only be achieved after sufficient time has elapsed to allow demonstration of the outstanding long-term impact and importance of the accomplishment; this will generally be after a number of years.

An important feature of the accomplishment assessment is the long-term tracking of accomplishments and the option to revisit and upgrade accomplishments in later years as they become more significant. A sampling of IBM Research accomplishments for 1996, the most recent year available at this time, is given in Figure 3.4. A comprehensive description of each is not possible here, but collectively they span a wide range of topics, including basic, numerically intensive computational tests of quantum chromodynamics; prototype products or significant enhancements to IBM product lines; and Deep Blue, the chess-playing supercomputer that in 1996 beat World Chess Champion Gary Kasparov in a single game (but lost the match) and in 1997 won a six-game match against Kasparov.

FIGURE 3.4. Sampling of IBM Research accomplishments in 1996.


Sampling of IBM Research accomplishments in 1996.

Concluding Remarks

I have given a brief description of how IBM Research, a large, complex research organization with multiple goals and responsibilities—ranging from having a direct impact on IBM's bottom line to long-term basic science—assesses its work. My intention was to point out that meaningful assessment of a wide range of technical research is possible and must be flexible enough to allow for unanticipated achievements, and that conducting such an assessment is a serious task in itself. As an organization, thus far we find that such assessment has been a worthwhile endeavor as a method both to explain our value to IBM and IBM's shareholders and to understand more clearly ourselves the value we create.


Charles Zukoski, University of Illinois at Urbana-Champaign: The "Accomplishment" levels that you described were clearly for major breakthroughs. You have a large research organization, and everyone's work does not necessarily fit into one of the three categories that you described. Could you explain how you value the other, smaller accomplishments?

Joseph M. Jasinski: I will answer that in several ways. Some of the work you describe is covered in what I call measurement projects, which I did not discuss in detail. That's an equal portion of the report to the chairman, and it covers activities that are much more short term than those described by accomplishments. Over time, we do expect most of the projects to which we devote significant resources to make it to the "Accomplishment" level. If you are not headed on that trajectory, you are probably not in a very comfortable position, if you are not involved in a measurement project or some other type of activity.

Thom H. Dunning, Jr., Pacific Northwest National Laboratory: Could you explain the process that you use to decide on whether projects have met the criteria for an "Accomplishment" or an "Outstanding Accomplishment"? The requirements for an ''Extraordinary Accomplishment" are fairly clear, but the process that you use to assign projects to the other levels is not.

Joseph M. Jasinski: Once a year we ask our managers to submit their proposals for the projects they think should be "Accomplishments" in their groups. These proposals are collected and matched against the criteria. In some sense, we rank-order them in a not terribly quantitative way, and we select a certain number that we think make the cutoff. At this time we also decide whether we think they are an "Accomplishment," "Outstanding," or "Extraordinary." This information is then passed on to the appropriate vice president, who submits his or her list of proposed accomplishments to the director of research and his staff. If necessary, they investigate or challenge the criteria by which proposed projects were graded, whether they are in fact accomplishments, whether they're outstanding, or whether they're extraordinary. We occasionally solicit outside input from customers. If the proposed project is a purely scientific activity, we may actually ask for outside evaluation to verify that the result is as important as we think it is. Finally, the director of research officially assigns the ratings.

Francis A. Via, Akzo-Nobel Chemicals, Inc.: I really enjoyed your presentation, which identified the outstanding scientific accomplishments of IBM, including voice recognition and superconductivity. Within the chemical industry, many of us are constantly confronted with issues associated with whether or not we can capture the value of our exploratory research.

In your industry, we have recently seen an abundance of competitive products in each new area of technology development, even in voice recognition. Of the wonderful technical accomplishments that you have listed, have these contributed significantly to the bottom line, or have you found that capturing the value of technology is more difficult than it has been in previous years?

Joseph M. Jasinski: That is a very complicated question. Speech recognition is one of the oldest projects in IBM research. It started long before it was practical. Our position is that IBM will be the dominant player in speech recognition. Whether this is true or not is not for me to say. There are clearly other products on the market, and the market has not yet sorted out the winners and losers.

You also mentioned high-temperature superconductivity. We don't make any products that use either low-temperature or high-temperature superconductors. There are obviously small niche markets for such products. In fact, we do little research in high-temperature superconductivity now. When we discovered high-temperature superconductivity, there was a great flurry of interest in it that led to the award of a Nobel Prize in physics to the IBM physicists involved. It also led to a lot of very good work in materials science, very fundamental work aimed at trying to understand the mechanism of high-temperature superconductivity. But that activity peaked and quickly died away. We now have only a small effort trying to resolve the mechanism—not a major effort or investment. I do not expect that we will ever make much of an investment in this area, unless for some reason we decide that high-temperature superconductivity is going to be important to one of our technologies.

As Professor Hounshell reminded us, if I tried to give you a return on investment number for our exploratory research, I would be lying. I will not pretend to give one, because we do not calculate ROIs for exploratory research, as far as I know. There may be someone somewhere in IBM who does try to quantify such investments, but if there is, I am not aware of it. As Professor Hounshell noted, it is very difficult, if not impossible, to do.

The guiding principle behind what we do in exploratory research is common sense and flexibility. We want to have impact, but we define impact broadly. We have a research division—I was somewhat facetious in the introduction, telling you research is what I do—but we believe in having a research organization that is differentiated from product development organizations. If we cannot differentiate ourselves and show that the mission has value in and of itself, then we would not be around for very much longer, at least not in the form we are in today.

As I noted in my talk, the earthquake hit IBM in 1992. Research seems to have come through that period relatively intact. In fact, in the last couple of years, we have been growing in terms of numbers in the Research Division. We have opened two new laboratories in areas that we view as emerging markets for information technology—India and the People's Republic of China. That shows you that we do believe research is part of IBM's business at all levels, not just in technology development but in marketing and everything else.

Andrew J. Lovinger, National Science Foundation: I have two related questions. The three categories of "Accomplishments" that you discussed are very broad. Do you use any finer scale and are people ranked against each other? How and when do you decide to terminate research projects, and what mechanism do you use?

Joseph M. Jasinski: I spoke about assessing the performance of the Research Division as a whole to IBM Corporation. I alluded to a small part of your question in my talk when I stated that a fraction of all nonexecutive employees' variable pay depends on the outcome of that assessment. But I did not address the question of how we manage our scientists.

We use a system very similar to Bell Laboratories for managing our professional staff, usually Ph.D.s, that we call RSMs, or research staff members (at Bell they are called members of the technical staff). At the end of the year, we rank them based on their individual and team accomplishments. We assign a number to every RSM on a scale of 1 to 100. An individual employee's compensation is based largely on that number and the change in that number over time.

Now your other question: How do we terminate projects? This is an extremely difficult thing to do. When you talk to academic researchers and you ask them why they have worked in the same area for 10 years or 15 years (or even their entire career), it is either because it is really good work and really exciting, or they appear to be uncomfortable because they know that you know and they know that everybody knows that they are doing fine work, but it is not leading edge. In that case, they say something like, "Well, that's where my funding is," or "that's where my reputation is established," or "that's the only area I can get funded to work in," or a similar explanation. It is completely different for us; we don't have those constraints.

We necessarily create a potentially difficult situation because we hire highly motivated, intelligent, driven people, usually very contrary to the mainstream. These, of course, are just the people who are going to discover things. The problem comes when they've been doing their work for a while, and somewhere someone decides that this is not the best work for them to be doing. Telling them this is awkward, uncomfortable—you name the adjective. But, in the end, we tell them that we just don't think the project they are working on is the best thing that they could be doing for IBM. They don't necessarily agree with us, and there may be some contention about the decision. Given their input, we think about the issue very long and hard and eventually either the project is terminated, or the individual prevails.

The process described above also applies to very large projects, projects that have gone well beyond what I call the research stage. For example, when I first joined IBM 15 years ago, we were developing Josephson junction technology for superconducting computers. The project involved 50 to 100 people. When it was decided that Josephson junction technology was never going to be a viable technology, the project was simply terminated. It is much easier to terminate a large project of that sort—the costs are high, the hoped-for return dwindling. You simply place the staff into other projects or try to get them interested in other scientific and/or technical problems.

It is much more difficult when only one or two research staff are involved in a project. It is not costing a lot of money, and even if you really don't believe it's the best thing for them to be doing for the corporation, or necessarily even for their own career, it is difficult to pull the plug when they disagree. At that point it becomes a matter of management judgment and employee persistence.