4SBIR Program Outputs

Publication Details

4.1. INTRODUCTION

Congress has tasked the National Academies to assess whether and to what extent the Small Business Innovation Research (SBIR) program has met the congressionally-mandated objectives for the program, and to suggest possible areas for improvement in program operations. Congress has, over the years, identified a number of objectives for the program, and these mandated objectives can be summarized as follows:

  • Supporting the commercialization of federally funded research.
  • Supporting the agency’s mission.1
  • Supporting small business and, in particular, woman- and minority-owned businesses.
  • Expanding the knowledge base.

Congress has not prioritized among the four objectives, although report language and discussions with congressional staff suggest that commercialization has become increasingly important to Congress. Still, it remains important to assess each of the four objectives, and each should therefore be taken as equally important in evaluating the achievements and challenges of the SBIR program. These four objectives help to define the structure and content of this chapter.

Assessing program outcomes against these four objectives entails numerous methodological challenges. These challenges are discussed in detail in the National Research Council’s (NRC) Methodology Report.2

4.1.1. Compared to What?

Assessment usually involves comparison—comparing programs and activities, in this case. Three kinds of comparison seem possible: with other programs at each agency, between SBIR programs at the various agencies, and with early-stage technology development funding in the private sector, such as venture capital activities. Yet, the utility of each of these three types of comparison is limited.

Other award programs at the agencies have fundamentally different objectives, such as promoting basic research through grant programs (at the National Institutes of Health [NIH] and the National Science Foundation [NSF]), developing capacity (awards for university infrastructure), or training. There are often no other dedicated programs for innovative small businesses. And no other programs for small businesses have as a primary goal the commercial exploitation of research. This fundamental difference in objectives makes it difficult to usefully compare an SBIR program with other programs at the relevant agency.

Comparisons between SBIR programs at different agencies appear superficially more useful, but must be regarded with considerable caution. As discussions in Chapter 1 of this volume and in the separate agency volumes indicate, the widely differing agency missions have shaped the agency SBIR programs, focusing them on different objectives and on different mechanisms and approaches. Agencies whose mission is to develop technologies for internal agency use via procurement—notably the Department of Defense (DoD) and the National Aeronautics and Space Administration (NASA)—have a quite different orientation from agencies that do not procure technology and are instead focused on developing technologies for use outside the agency.

Finally, SBIR might be compared with venture capital (VC) activities, but there are important differences. VC funding is typically supplied later in the development cycle when innovations are in, or close to, market—most venture investments are made with the expectation of an exit from the company within three years. VC investments are typically larger than SBIR awards—the average investment made by VC firms in a company was over $7 million in 2005, compared to less than $1 million for SBIR over a two to three year cycle.3 VC investments are also focused on companies, not projects, and often come both with substantial management support and influence (e.g., through seats on the company’s board). None of this is true for SBIR.

BOX 4-1Multiple Sources of Bias in Survey Response

Large innovation surveys involve multiple sources of bias that can skew the results in both directions. Some common survey biases are noted below.a

  • Successful and more recently funded firms are more likely to respond. Research by Link and Scott demonstrates that the probability of obtaining research project information by survey decreases for less recently funded projects and it increased the greater the award amount.b Nearly 40 percent of respondents in the NRC Phase II Survey began Phase I efforts after 1998, partly because the number of Phase I awards increased, starting in the mid-1990s, and partly because winners from more distant years are harder to reach. They are harder to reach as time goes on because small businesses regularly cease operations, are acquired, merge, or lose staff with knowledge of SBIR awards.
  • Success is self reported. Self-reporting can be a source of bias, although the dimensions and direction of that bias are not necessarily clear. In any case, policy analysis has a long history of relying on self-reported performance measures to represent market-based performance measures. Participants in such retrospectively analyses are believed to be able to consider a broader set of allocation options, thus making the evaluation more realistic than data based on third party observation.c In short, company founders and/or principal investigators are in many cases simply the best source of information available.
  • Survey sampled projects at firms with multiple awards. Projects from firms with multiple awards were underrepresented in the sample, because they could not be expected to complete a questionnaire for each of dozens or even hundreds of awards.
  • Failed firms are difficult to contact. Survey experts point to an “asymmetry” in their ability to include failed firms for follow-up surveys in cases where the firms no longer exist.d It is worth noting that one cannot necessarily infer that the SBIR project failed; what is known is only that the firm no longer exists.
  • Not all successful projects are captured. For similar reasons, the NRC Phase II Survey could not include ongoing results from successful projects in firms that merged or were acquired before and/or after commercialization of the project’s technology. The survey also did not capture projects of firms that did not respond to the NRC invitation to participate in the assessment.
  • Some firms may not want to fully acknowledge SBIR contribution to project success. Some firms may be unwilling to acknowledge that they received important benefits from participating in public programs for a variety of reasons. For example, some may understandably attribute success exclusively to their own efforts.
  • Commercialization lag. While the NRC Phase II Survey broke new ground in data collection, the amount of sales made—and indeed the number of projects that generate sales—are inevitably undercounted in a snapshot survey taken at a single point in time. Based on successive data sets collected from NIH SBIR award recipients, it is estimated that total sales from all responding projects will likely be on the order of 50 percent greater than can be captured in a single survey.e This underscores the importance of follow-on research based on the now-established survey methodology.
FIGURE B-4-1. Survey bias due to commercialization lag.

FIGURE B-4-1

Survey bias due to commercialization lag.

These sources of bias provide a context for understanding the response rates to the NRC Phase I and Phase II Surveys conducted for this study. For the NRC Phase II Survey, of the 4,523 firms that could be contacted out of a sample size of 6,408, 1,916 responded, representing a 42 percent response rate. The NRC Phase I Survey captured 10 percent of the 27,978 awards made by all five agencies over the period of 1992 to 2001. See Appendix A and B for additional information on the surveys.

a

For a technical explanation of the sample approaches and issues related to the NRC surveys, see Appendix A.

b

Albert N. Link, and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative, London: Routledge, 2005.

c

While economic theory is formulated on what is called “revealed preferences,” meaning individuals and firms reveal how they value scarce resources by how they allocate those resources within a market framework, quite often expressed preferences are a better source of information especially from an evaluation perspective. Strict adherence to a revealed preference paradigm could lead to misguided policy conclusions because the paradigm assumes that all policy choices are known and understood at the time that an individual or firm reveals its preferences and that all relevant markets for such preferences are operational. See {1} Gregory G. Dess and Donald W. Beard, “Dimensions of Organizational Task Environments,” Administrative Science Quarterly, 29: 52-73, 1984; {2} Albert N. Link and John T. Scott, Public Accountability: Evaluating Technology-Based Institutions, Norwell, MA: Kluwer Academic Publishers, 1998.

d

Albert N. Link, and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative, op. cit.

e

Data from NIH indicates that a subsequent survey taken two years later would reveal very substantial increases in both the percentage of firms reaching the market, and in the amount of sales per project. See National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, Charels W Wessner, ed., Washington, DC: The National Academies Press, 2009.

The lack of available comparators means that we must assess each program in terms of the benchmarks developed to review the program in the Methodology report described below.4

4.2. COMMERCIALIZATION

4.2.1. Challenges of Commercialization

Commercialization of the technologies developed under the research supported by SBIR awards has been a central objective of the SBIR program since its inception. The program’s initiation in the early 1980s, in part, reflected a concern that American investment in research was not adequately deployed to the nation’s competitive advantage. Directing a portion of federal investment in R&D to small businesses was thus seen as a new means of meeting the mission needs of federal agencies, while increasing the participation of small business and thereby the proportion of innovation that would be commercially relevant.5

Congressional and Executive branch interest in the commercialization of SBIR research has increased over the life of the program.

A 1992 GAO study6 focused on commercialization in the wake of congressional expansion of the SBIR program in 1986.7 The 1992 reauthorization specifically “emphasize[d] the program’s goal of increasing private sector com mercialization of technology developed through federal research and development.”8 The 1992 reauthorization also changed the order in which the program’s objectives are described, moving commercialization to the top of the list.9

The term “commercialization” means “reaching the market,” which some agency managers interpret as “first sale”—that is, the first sale of a product in the market place, whether to public or private sector clients. This definition, however, misses significant components of commercialization that do not result in a discrete sale. It also fails to provide any guidance on how to evaluate the scale of commercialization, an important element in assessing the degree to which SBIR programs successfully encourage commercialization. The metrics for assessing commercialization can also be elusive,10 and it is important to understand that it is not possible to completely quantify all commercialization from a research project:

  • The multiple steps needed after the research has been concluded mean that a single, direct line between research inputs and commercial outputs rarely exists in practice; cutting edge research is only one contribution among many leading to a successful commercial product.
  • Markets themselves have major imperfections, or information asymmetries so high quality, even path-breaking research, does not always result in commensurate commercial returns.
  • The lags involved in the timeline between an early stage research project and a commercial outcome mean that for a significant number of the more recent SBIR projects, commercialization is still in process, and sales—often substantial sales—will be made in the future. The current “total” sales are in this case just a “snapshot half way through the race,” and will require updating as the full impact of the award becomes apparent in sales.
  • Yet the impact of SBIR awards also needs to be qualified. Research rarely results in stand-alone products. Often, the output from an SBIR project is combined with other technologies. The SBIR technology may provide a critical element in developing a winning solution, but that commercial impact—the sale of the larger combined product—is not captured in the data.
  • In some cases, the full value of an “enabling technology” that can be used across industries is difficult to capture.

All this is to say that commercialization results must be viewed with caution, first because our ability to track them is limited (indeed it appears highly likely that our efforts at quantification of research awards may understate the true commercial impact of SBIR projects) and because an award, and a successful project, cannot lay claim to all subsequent commercial successes, though it may contribute to that success in a significant fashion.

These caveats notwithstanding, it is possible to deploy a variety of assessment techniques to measure commercialization outcomes.

4.2.2. Commercialization Indicators and Benchmarks

This report uses three sets of indicators to quantitatively assess commercialization success:

  1. Sales and Licensing Revenues (“sales” hereafter, unless otherwise noted). Revenues flowing to a company from the commercial marketplace and/or through government procurement constitute the most obvious measure of commercial success. They are also an important indicator of uptake for the product or service. Sales indicate that the result of a project has been sufficiently positive to convince buyers that the product or service is the best available solution.
    Yet if there is general agreement that sales are a key benchmark, there is no such agreement on what constitutes “success.” Companies, naturally enough, focus on projects that contribute to the bottom line—that are profitable. Agency staff provide a much wider range of views. Some view any sales a substantial success for a program focused on such an early stage of the product and development cycle, while others seem more ambitious.11 Some senior executives in the private sector viewed only projects that generated cumulative revenues at $100 million or more as a complete commercial success.12
    Rather than seeking to identify a single sales benchmark for “success,” it therefore seems more sensible to simply assess outcomes against a range of benchmarks reflecting these diverse views, with each marking the transition to a greater level of commercial success:
    1. Reaching the market—a finished product or service has made it to the marketplace
    2. Reaching $1 million in cumulative sales (beyond SBIR Phases I and II)—the approximate combined amount of standard DoD Phase I and Phase II awards
    3. Reaching $5 million in cumulative sales—a modest commercial success that may imply that a company has broken even on a project
    4. Reaching $50 million in cumulative sales—a full commercial success.
  2. Phase III Activities Within DoD. As noted above, Phase III activities within DoD are a primary form of commercialization for DoD SBIR projects. These activities are considered in Section 4.3 (Agency Mission) of this report and Chapter 5 (Phase III Challenges and Opportunities) of the DoD report.13
  3. R&D Investments and Research Contracts. Further R&D investments and contracts are good evidence that the project has been successful in some significant sense. These investments and contracts may include partnerships, further grants and awards, or government contracts. The benchmarks for success at each of these levels should be the same as those above, namely:
    1. Any R&D additional funding
    2. Additional funding of $1 million or more
    3. Additional funding of $5 million or more
    4. Funding of $50 million or more
  4. Sale of Equity. This is a clear-cut indicator of commercial success or market expectations of value. Key metrics include:
    1. Equity investment in the company by an independent third party
    2. Sale or merger of the entire company

4.2.3. Sales and Licensing Revenues

The most basic of all questions on commercialization is whether results from a project reached the marketplace. The NRC Phase II Survey14 indicates that just under half (47 percent) of respondents had generated some sales, and that a further 18 percent still expected sales, though they had none at the time of the survey. In addition, 5 percent were still in the research stage of the project. There is variation among the agencies, but these data are consistent with program objectives.15

FIGURE 4-1. Sales from Phase II projects.

FIGURE 4-1

Sales from Phase II projects.

SOURCE: NRC Phase II Survey.

4.2.3.1. Distribution of Sales

Research on early-stage financing strongly suggests a pronounced skew to the results, and this turns out to be the case. Most projects that reach the market generate minimal revenues. A few awards generate substantial results, and a small number bring in large revenues.

Of the 790 SBIR Phase II projects reporting sales greater than $0, average sales per project were $2,403,255. Over half of the total sales dollars were due to 26 projects (1.4 percent of the total), each of which had $15,000,000 or more in sales. The highest cumulative sales figure reported was $129,000,000.

This distribution is reflected in Figure 4-2.

FIGURE 4-2. Distribution of projects with sales >$0.

FIGURE 4-2

Distribution of projects with sales >$0. SOURCE: NRC Phase II Survey.

Almost three quarters of the projects reporting sales greater than zero had $1 million or less in sales; two projects reported sales greater than $100 million. The latter by themselves accounted for 16.5 percent of all the revenues reported; together, the 1.7 percent of respondents reporting sales greater than $20 million accounted for 43.7 percent of all revenues reported.

These distributions are similar to those reported from other SBIR data sources.16 For example, at NIH, sales data are available from the NRC survey, a previous NIH survey, and the DoD commercialization database.17 These data in Figure 4-4 show that about 10 percent of NIH projects that report any revenues report more than $5 million.

FIGURE 4-4. NIH sales by sales range (percent of projects in each range).

FIGURE 4-4

NIH sales by sales range (percent of projects in each range). Total for 1992-2002. SOURCE: NIH data, NRC Phase II Survey, and DoD Commercialization database.

FIGURE 4-3. Distribution of sales, by total sales (percent of total sales dollars).

FIGURE 4-3

Distribution of sales, by total sales (percent of total sales dollars).

SOURCE: NRC Phase II Survey.

4.2.3.2. Sales Expectations

Because it may take years after the end of a Phase II for a commercial product to reach the market, many projects that do not yet report sales still expect them eventually. About 36 percent of NRC survey respondents with no sales (19 percent of all projects) still expected sales in the future. Analysis elsewhere suggests that these expectations are often optimistic, and that a considerably smaller number of these projects will, in the end, reach the market.

However, it is equally important to note that a complete accounting of all sales from the projects funded during 1992-2001 (the focus of the NRC survey) will be possible only some years in the future. Many projects have only recently reached the market, so the bulk of their sales will be made in the future and are not captured in these survey data, which effectively capture initial sales (see Box 4-2).

Box Icon

BOX 4-2

Underestimating Commercial Outcomes from the SBIR Program: The Impact of Systematic Characteristics of Survey Analysis. Among the SBIR agencies, only DoD requires that firms enter commercialization data into a database when applying for subsequent awards. (more...)

TABLE 4-1. Most Important Customer (Percent of Responses).

TABLE 4-1

Most Important Customer (Percent of Responses).

4.2.3.3. Sales by Sector

The NRC Phase II Survey asked respondents to identify the customer base for the products. There are substantial differences between agencies. Only 1 percent of NIH respondents, for example, reported sales to DoD, in contrast to 38 percent for DoD respondents.18

4.2.3.4. Licensee Sales and Related Revenues

The indirect impact of licensee sales, where survey respondents report sales not made by their own company, is an important measure of success. These data are important as an indicator of the extended effects of SBIR beyond the immediate awardee company. However, they should be treated with an additional degree of caution, as respondents do not necessarily have as accurate information about another company as they have about their own.

Licensing activity within the program is significant: Just over 35 percent reported a finalized licensing agreement or ongoing negotiations towards one.19

Just over 5 percent of respondents reported licensee sales greater than $0, with three licenses reporting more than $70 million in sales each and accounting for more than half of all reported licensee sales. One project alone reported more than $200 million in licensee sales.

As the case study in Box 4-3 indicates, in some cases licensing has been pivotal in the commercialization of a successful technology: In some industries, there are no alternatives to using established channels.

Box Icon

BOX 4-3

Licensing Case Study. Applied Health Science and the Wound and Skin Intelligence System™ Applied Health Science (AHS) received a Phase I award to validate and automate the Pressure Sore Status Tool, a standardized assessment instrument for use (more...)

4.2.4. Additional Investment Funding

Further investment in a recipient company related to the SBIR award project is another indication that the project work is of value. On average, SBIR projects received almost $800,000 from non-SBIR sources, with over half of respondents (51.6 percent) reporting some additional funds for the project from a non-SBIR source.20

Focusing just on the 839 projects that reported receiving more than $0 in additional funding, these projects reported average additional funding of $1,538,438, with almost $260,000 from federal agencies themselves, mostly from DoD. Three hundred eighteen projects reported some federal funding, averag ing just under $1.6 million per project. At DoD, just over 30 percent of projects (205) reported additional federal funding outside SBIR, averaging just over $1.6 million per project.

TABLE 4-2. Sources of Additional Investments in SBIR Projects.

TABLE 4-2

Sources of Additional Investments in SBIR Projects.

About $165,000 per project came from venture capital sources. However, in those cases where venture funding was present, the amounts of funding were substantial: For the 50 responding projects with VC funding, the average per project was just under $8.3 million.

These figures reflect the well-known concentration of venture funding on a few, highly desirable projects. They also show that SBIR supports a wide range of projects which do have commercial prospects (as well as other possible benefits) that are not likely to be funded by venture capital.

Focusing more closely on venture funding at NIH, initial research indicates that approximately 50 of the 200 companies that won the most Phase II awards at NIH have received some venture funding.21 This is reflected in Figure 4-5.

FIGURE 4-5. Distribution of venture funding for NIH top 200 Phase II winners.

FIGURE 4-5

Distribution of venture funding for NIH top 200 Phase II winners. SOURCE: VentureSource and other venture capital databases; National Institutes of Health awards database. See also National Research Council, An Assessment of the SBIR Program at the National (more...)

Total VC investment is approximately $1.59 billion in these 50 companies, a total that dwarfs the $272 million NIH SBIR investment in these companies.

In addition, the NRC Firm Survey determined that 15 firms had had initial public offerings, and that a further six firms planned such offerings for 2005-2006. SBIR firms also generate a significant number of new companies. Fourteen percent of responding firms indicated they had formed a spin-off company, with a total of 242 spin-off companies reported.

In contrast to the tightly concentrated distribution of venture funding, inter nal funding was by far the most widespread form of support, being reported by almost 50 percent of all respondents. Average internal funding ($113,000), was much lower than average venture funding.

About 7 percent of all respondents reported receiving “other private equity.” This funding averaged $1.9 million per project for those projects receiving this kind of funding. Investments from government and academic sources were relatively scarce (less than 5 percent of respondents), and provided relatively small amounts (just over $225,000 per respondent receiving these funds).

4.2.5. Additional SBIR Funding

Aside from providing non-SBIR funds, the federal government in many cases makes further investments via the SBIR program itself. The NRC surveys asked respondents how many additional Phase I and Phase II awards followed each initial award, related to the original project.

About 40 percent of respondents reported receiving at least one additional related Phase II award, and slightly over half (53.8 percent) reported at least one additional Phase I award.

Relatively few projects received many related awards: Only 8.4 percent of respondents reported at least 5 related Phase I awards, and 9.7 percent received at least three related Phase IIs. This suggests that the “clustering” hypothesis—that SBIR type projects often require multiple awards, sometimes looping back to Phase I, before reaching the market—only applies to a limited number of projects, although case studies (e.g., SAM Technologies22) indicate that there are also important cases where such clustering does occur. In short, the data suggest that some companies and projects do attract a cluster of SBIR awards, but most do not. It may be, however, that this concentration of clustered awards reflects the limited number of commercial successes as well. Just under one half of respondents (47 percent) reported no additional related Phase I awards, and 60 percent reported no related Phase II.

TABLE 4-3. Related SBIR Awards.

TABLE 4-3

Related SBIR Awards.

4.2.6. SBIR Impact on Further Investment

Both the NRC and NIH Surveys sought additional information about the impact of the SBIR program on company efforts to attract third party funding—the “halo effect” mentioned by some interviewees, who suggested that an SBIR award acted as a form of validation for external inventors.23

The fact that two-thirds of SBIR respondents did not attract outside funding, and that only 3.5 percent received venture funding, suggests that receiving a Phase II SBIR award does not in itself guarantee external funding. Survey responses from other surveys did, however, paint a more positive picture of these effects: 78 percent of NIH respondents said that they believed that additional capital had “resulted from” their SBIR participation—a strong statement.24

Case study interviews provided mixed views on this, with some interviewees strongly supporting the view that SBIR helps to attract investment and others suggesting that SBIR awards had had relatively little impact, although these views obviously reflect individual company experiences.

4.2.7. Small Company Participation and Employment Effects

Employment is another indicator of commercial success and also that the program is supporting small business.

The median size of a company receiving SBIR awards is relatively small—far lower than the 500-employee limit imposed by the SBA (see Figure 4-6).

FIGURE 4-6. Distribution of companies, by number of employees at time of award.

FIGURE 4-6

Distribution of companies, by number of employees at time of award. SOURCE: NRC Phase II Survey.

The program focuses the bulk of its awards on very small companies. More than a third of awardees had between one and five employees at the time of award. A very substantial number (seventy percent) of respondent companies had 20 employees or fewer at the time of the Phase II award.

The NRC Survey sought detailed information about the number of employees at the time of the award and at the time of the survey and about the direct impact of the award on employment. Overall, the survey data showed that the average employment gain at each responding firm from the date of the SBIR award to the time of the survey was 29.9 full-time equivalent employees. Of course, very few of the companies that went out of business responded to the survey, so this question is particularly skewed toward firms that have been at least somewhat successful.

Most responding companies have expanded since the date of the Phase II award. The NRC Phase II Survey also shows that respondents enjoyed strongly positive employment growth after receiving a Phase II award. Table 4-4 shows that the percentage of companies with at least 50 employees more than doubled, from 16.5 percent to 35.4 percent of all respondents. Overall, survey respondents reported gains of 57,808 full time equivalent employees, with the top five respondents accounting for 18.4 percent of the overall net gain.

TABLE 4-4. Employment at Phase II Respondent Companies, at the Time of Award and Currently.

TABLE 4-4

Employment at Phase II Respondent Companies, at the Time of Award and Currently.

The NRC survey also sought to directly identify employment gains that were the direct result of the award. Respondents estimated that specifically as a result of the SBIR project, their firm was able to hire an average of 2.4 employees, and to retain 2.1 more.25

FIGURE 4-7. Employment distribution at responding companies, at time of award and currently.

FIGURE 4-7

Employment distribution at responding companies, at time of award and currently.

SOURCE: NRC Phase II Survey.

TABLE 4-5. Company-level Activities.

TABLE 4-5

Company-level Activities.

4.2.8. Sales of Equity and Other Company-level Activities

Company-level activities offer another set of indicators for measuring commercial activity, as these suggest that something of commercial value is being developed, even if no sales have as yet resulted. For example, although Neurocrine, an NIH SBIR awardee has yet to produce a product, it achieved an IPO that raised more than $100 million.26

The NRC Phase II Survey explored whether SBIR awardees had finalized agreements or ongoing negotiations on various company-level activities. The data show that marketing-related activities were most widespread. Activities with foreign partners were lower than similar activities with U.S. partners. Note, however, that the question asked specifically for outcomes that were the “result of the technology developed during this project”—a tight description.27

The impact of these activities is hard to gauge using quantitative assessment tools only. Box 4-4 illustrates how research conducted using SBIR funding seeded an entire generation of spin-off companies and joint ventures in a technology of potential significance for homeland security.

Box Icon

BOX 4-4

Intelligent Optical Systems: Intelligent, Distributed, Sensitive Chemical and Biochemical Sensors and Sensor Networks. Intelligent Optical Systems has developed a system for using the entire length of a specially designed fiber-optic cable as a sensor (more...)

4.2.9. Commercialization: Conclusions

While accepting the view that there is no single, simple metric for determining the commercial success of an early stage R&D program such as SBIR, numerous metrics do provide the basis for making a broad determination of commercial outcomes at SBIR.

These data, taken together, strongly support the view that the program has a strong commercial focus, with considerable efforts to bring projects to market, with some success. Even though the number of major commercial successes has been few, that is normal for early stage high-risk projects, and the overall commercialization effort is substantial. Products are coming to market quickly, significant licensing and marketing efforts are under way for many projects, and approximately 30-40 percent of projects generate products that do reach the marketplace. These data all paint a picture of a program that is successful in commercializing innovative technologies in a variety of ways.

4.3. AGENCY MISSION

Each agency with an SBIR program has a different mission, and the contribution of SBIR to each specific mission must be assessed individually. These assessments, found in the individual agency reports, conclude that SBIR is indeed supporting each agency’s pursuit of its specific mission.

Some more general observations can be made, however. An assessment of the extent to which SBIR supports agency mission can be divided into two areas:

  • procedural alignment—the extent to which the procedures of the agency SBIR program are aligned with the needs of the agency
  • program outcomes—the extent to which outcomes from the program have the effect of supporting the agency mission.

It is important to note that the different missions of the agencies mean that some agencies define agency mission support more narrowly, or at least have much tighter metrics for assessing this element of the program. In particular, the procurement agencies—primarily DoD and NASA—assess contribution to agency mission primarily against the extent to which the agency itself uses outputs from the SBIR program. In contrast, the non-procurement agencies—NIH, NSF, and, to a great extent, the Department of Energy (DoE)—see support for mission much more broadly: For NIH, for example, support for mission can be construed as anything that improves medical knowledge or public health.

4.3.1. Procedural Alignment of SBIR Programs and Agency Mission

A procedural assessment reviews the steps taken by each agency program to ensure that the design and procedures of their SBIR program are aligned with the needs of the agency.

Agencies do this in different ways, but the following areas of analysis are broadly shared by all the agencies.

4.3.1.1. Topics and Solicitations

SBIR proposals are received by the agencies in response to published solicitations for proposals. These solicitations are the primary vehicle through which the agency expresses its areas of research interest to the scientific and technical community of small businesses. Within each solicitation, specific subject areas of interest are defined by individual topics. Topics can be focused tightly on a specific problem or requirement, or they may broadly outline an area of technical interest to the agency.

Aside from NIH, which expressly indicates that its topics are guidelines, not mandatory limits or boundaries on research that could be funded, all the agencies use topics to specify boundaries. In doing so, they are specifically delimiting areas of technical interest to the agency.

This is prima fascia confirmation that the SBIR programs support agency mission: Unless there is evidence that agencies are generating topics that are not aligned with the agency mission—and our analysis and interviews with staff and awardees found no trace of this—the use of topics and solicitations indicates that agencies are working to ensure that awards are aligned with the stated scientific and technical needs of the agency.

4.3.1.2. Topic Development Process—DoD

The agencies—notably DoD but, to a lesser extent, all the SBIR agencies—have an elaborate process for developing proposals for topics, sifting and assessing these proposals, and then finally deciding which topics should be published in the solicitation.

Agency procedures differ, and are described in the agency volumes. At DoD, for example, the topic selection process includes multiple levels of review. While this process has been criticized for adding considerable time between the initial identification of an agency’s needs and the first dollars flowing to those needs, the process also seeks to ensure that all of the stakeholders feel that they have had a say. This is particularly important for DoD because it has a number of different stakeholder communities.

FIGURE 4-8. Topic review process.

FIGURE 4-8

Topic review process.

SOURCE: NRC chart developed from interviews with Department of Defense staff.

Ultimate decision authority on the inclusion of topics in a DoD solicitation lies with the Integrated Review Team, which contains representatives from each of the awarding components. Topics are reviewed initially at DDR&E and are then returned to the agencies for correction of minor flaws, for revision and resubmission, or as discards.

DoD has also made considerable efforts to improve the topic selection process. In the late 1990s, DoD determined that topics were not being linked closely enough to the users of SBIR technologies in the acquisition community, and a conscious effort was made to ensure that more topics were “owned” by that community (with considerable success—see DoD report, Chapter 5: Program Management, for details28).

4.3.1.3. Topic Development Process—NSF

Under the 2005 NSF strategic plan, SBIR solicitation topics fit into three broad areas: (A) investment business focused topics, (B) industrial market driven topics, and (C) technology in response to national needs. (A, B, and C below). The topics list given in the strategic plan includes seven topics:

  1. Investment Business Focused Topics
    • Biotechnology (BT)
    • Electronics Technology (EL)
    • Information Based Technology (IT)
  2. Industrial Market Driven Topics
    • Advanced Materials and Manufacturing (AM)
    • Chemical Based Technology (CT)
  3. Technology in Response to National Needs
    • Security Based Technology (ST)
    • Manufacturing Innovation (MI)

It appears that areas A and B above are expected to be relatively stable, and that area C is expected to have more frequent changes in specific topics as national needs change.

Even at NIH, where topics are viewed as guidelines, not boundaries for permissible research, the topic selection process is designed to ensure that area specialists within the agency’s Institutes and Centers have substantial input. Area specialists are regularly encouraged to ensure that their wider research agendas are reflected in their selection of SBIR topics for publication.

Overall, agencies appear to be well aware that the topic/solicitation process is the primary mechanism through which the SBIR program is aligned with the agency’s S&T objectives and its overall agency missions. Topics appear to be aligned with agency missions.

4.3.1.4. Award Selection Process

The selection of awards can also support an agency’s mission, to the extent that the process reflects the agencies’ priorities. A wide range of awards procedures are used at the various agencies, and these may differ substantially even between components of the same agency. For example, within DoD, Army, and Navy use different approaches, staff, and methodologies for selecting awardees.

All of these agency procedures share some basic characteristics: They assign considerable weight to the technical merit of proposals; They seek to ensure that selectors are, to some extent, independent of sponsoring agency components; and they have a set of written standards against which proposals are supposed to be measured. They differ in the nature of the reviewers (internal or external), the type of scoring (quantitative or qualitative), the extent of appeal and resubmission for nonawarded proposals, the degree to which scores are binding, and the discretionary powers of agency staff to make full or supplementary awards.

It is also in general fair to say that the point of reference with regard to agency mission reference at the point of award is not the agency’s overall mission statement, but the specific expression of that statement in the form of the topic. Topics are used as expressions of the agency’s mission. For example, DoE’s overall mission is to “advance the national, economic, and energy security of the United States; to promote scientific and technological innovation in support of that mission; and to ensure the environmental cleanup of the national nuclear weapons complex.”29 Specific SBIR awards are made in relation to topics that support that mission—for example, to generate better battery cell technology. A high-quality, effective selection process, one that effectively identifies and funds projects that will help meet agency needs expressed via the topic, is therefore a process that supports the agency mission overall.

While selection procedures could potentially be improved, our analysis concludes that at every agency, the current selection processes largely succeed in aligning the technologies chosen with the expressed needs of the agency.30

4.3.2. Program Outcomes and Agency Mission

In contrast to the discussion above, the program outcomes for agency mission are both more difficult to assess and also more specific to individual agencies. All of the methodological difficulties in assessing outcomes discussed at the beginning of this chapter apply here; moreover, (unlike commercialization) there are no obvious and widely understood benchmarks that apply across agencies.

All agencies maintain a list of “success stories,” describing SBIR awards that meet congressional goals. Some of these are focused on agency mission. However, the stories themselves, while illustrative of the power of the program to help develop new technologies, are of variable quality. Some agencies, such as NIH, use success stories written by the company in question without validating them. Even agencies that take a more systematic approach to case studies do not appear to have clear and transparent criteria for determining what should count as a “success story.”

Each agency has found different ways to explore and describe outcomes related to agency mission. As a result, our overview must reflect these agency differences. Below, we provide a brief summary of agency approaches to identifying and measuring ways in which the SBIR program supports the agency mission.

4.3.2.1. NIH

For NIH, the primary mission is the development of fundamental knowledge and its application for improving health.

The NIH mission is science in pursuit of fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to extend healthy life and reduce the burdens of illness and disability.

SOURCE: National Institutes of Health, <http://www.nih.gov/about/ndex.html#mission>.

A more detailed assessment of outcomes related to these objectives can be found in Chapter 3 of the NIH Report.31 Data on patents and peer-reviewed publications resulting from awards can be used as an indicator of the development of fundamental knowledge. Commercialization measures are another possible indicator of the transfer of knowledge, as commerce marks a transaction that reflects that transfer.

Still, more direct measures would be useful. NIH has tried to develop these in several ways. Like other agencies, NIH maintains a Web page filled with “success stories.” Unfortunately, these are entirely self-posted by companies, and reflect no selection or even verification by staff.

The NIH SBIR program has also led the way at NIH in developing metrics through the implementation of a recipient survey, with the first survey being deployed in late 2003. In this survey, the agency has sought to identify the populations targeted by SBIR projects. Figure 4-9 shows the distribution of projects by sector, for both projects that have already reached the market and those still being commercialized.32

FIGURE 4-9. Distribution of NIH projects, by type of affected population.

FIGURE 4-9

Distribution of NIH projects, by type of affected population. SOURCE: National Institutes of Health, National Survey to Evaluate the NIH SBIR Program: Final Report, July 2003.

The NIH survey also sought to quantify the number of people affected by a technology, asking respondents to place their project within ranges of affected populations by size. Such information, if accurate, would be useful. Unfortunately, respondents have wide latitude in answering, and are given no guidelines that might help to explain how to structure their answers. The responses are essentially guesses—perhaps biased guesses, as respondents may have a tendency to overestimate their projects’ importance.

Still, the data categories show the wide variety of mission-related areas into which SBIR projects can be categorized—and all clearly fall within the broad mission definition of NIH. Responses can also indicate the distribution of projects between mission areas and can be used as a proxy for the degree of NIH interest in these different areas.

4.3.2.2. DoD

The DoD SBIR program’s primary goal is the provision of technologies that are employed as part of defense systems being developed, acquired, or maintained to meet DoD mission needs. In effect, that acquisition stream defines, for operational purposes, agency needs, and the extent to which SBIR is, in fact, part of the acquisition process has become an important indicator of support for agency mission.

The mission of the Department of Defense is to provide the military forces needed to deter war and to protect the security of our country.

SOURCE: Department of Defense, <http://www.defenselink.mil/admin/about.html>.

This was not always the case. As discussed in the DoD volume, the program’s objectives have evolved since the late 1990s. Some elements of the management have recently focused on finding ways to more closely align SBIR projects and programs with the acquisition process of the Services.

Much of the discussion that follows will focus on acquisitions and on Phase III at DoD. However, it is still important to remember that, even at DoD, not all outcomes that support the mission result in acquisitions and Phase III. For example, agency staff have indicated that SBIR can be an important way of assessing technologies that in the end do not pan out. The awards thus act as a low-cost probe of the technological frontier. So, even when no acquisition occurs, SBIR can still provide valuable support for the agency mission in terms of information on technological dead ends, promising technological options, or use resulting from the award itself.

Still, even with these multiple functions, as participants at the NRC’s Phase III Symposium affirmed, Phase III and acquisitions are regarded by DoD as the core focus of the SBIR program, and the key indicator for measuring success in supporting agency mission. Many speakers at the NRC’s Phase III Symposium made clear that the take-up of SBIR-funded technologies into the DoD acquisitions program was the benchmark against which DoD’s SBIR program should be judged.

DoD has made a more conscious effort to measure SBIR impacts on agency performance than any of the other agencies under study. Because DoD is a procurement agency, there is also one clear set of indicators which, if properly captured, could provide critical benchmarks and feedback to the agency.

DoD has two tools for measuring follow-on Phase III contracts. The first is the DD350 contracting forms, which is supposed to capture whether a contract is a follow-on to an SBIR award. However, only well-trained contracting officers who understand SBIR are likely to correctly fill out the form. Internal assessment of the DD350 by DoD suggests that not all contracting officers use the form correctly. Consequently, at many DoD components, SBIR follow-on contracts are only erratically reported, if at all.33 The other tool for measuring follow-on Phase III contracts is the DoD SBIR/STTR Commercialization Database. This database is used to calculate firms’ Commercialization Achievement Index and to generate the Company Commercialization Reports that are packaged with proposals to evaluators.

The Navy has recently made efforts—including the investment of external program dollars—to improve the quality of DD350 data. This effort is reflected in recent DD350 results, which show Navy as accounting for more than 70 percent of all DoD Phase III awards.34

The DD350 results do show that the amount of Phase III contracts generated have been climbing steadily in recent years, particularly at the Navy (see Figure 4-10).

FIGURE 4-10. Phase III awards, annual totals, 1999-2005.

FIGURE 4-10

Phase III awards, annual totals, 1999-2005. SOURCE: DoD SBIR/STTR Commercialization Database, provided by John Williams, U.S. Navy, April 7, 2005.

These data also show that the amount of Phase III contracts being identified is greater than the amount of funding expended on SBIR, once commercialization has been appropriately lagged.

A fuller discussion of DoD’s efforts regarding Phase III can be found in Chapter 5 of the study of SBIR at DoD.35 Leaving complexities aside, however, it is possible to draw the following general conclusions:

FIGURE 4-11. Reported commercializations versus SBIR budget.

FIGURE 4-11

Reported commercializations versus SBIR budget.

SOURCE: Michael Caccuitto, DoD SBIR/STTR Program Administrator, Presentation to SBTC “SBIR in Rapid Transition” Conference, Washington, DC, September 27, 2006.

  • DoD is increasingly making an effort to measure support for agency mission.
  • Part of this assessment is increasingly quantitative, with the Navy leading the way in utilizing data from the DD350 forms. Nonetheless, there are barriers and difficulties that tend to reduce the number and amount of Phase III contracts counted in the DD350 tracking forms.
  • The demonstration effect of the Navy program and growing awareness of SBIR’s potential has increased senior management’s awareness of SBIR’s contributions to the Defense mission.36
  • The growing interest and dedicated management attention by Prime contractors is enhancing the potential for insertion of SBIR outputs into the acquisition process.
  • The reluctance of other agencies to provide management funding at the level of the Navy’s effort remains a constraint on maximizing the return on the nation’s SBIR investment.

In short, recent improvements in DoD activities related to Phase III do suggest a slow but significant change in attitude toward SBIR within DoD. SBIR is seen less as a tax imposed by Congress on otherwise useful R&D activities, and more as an opportunity to bring innovative ideas and products to meet mission objectives.

4.3.2.3. DoE

DoE has, to a considerable extent, relied on process and procedures to ensure that its mission is being supported by the SBIR program. The agency has developed topic and award selection procedures that ensure that the primary driver of the program will be R&D managers within the agency, rather than either external peer reviewers or SBIR program managers. The latter play a minimal role in topic and award selection.

Mission

Discovering the solutions to power and secure America’s future

Vision

The Department’s vision is to achieve results in our lifetime ensuring: Energy Security; Nuclear Security; Science-Driven Technology Revolutions; and One Department of Energy—Keeping our Commitments.

SOURCE: Department of Energy, <http://www.cfo.doe.gov/strategicplan/mission.htm>.

In FY2005, this process resulted in 49 technical topics, from within the 12 program areas (see Table 4-6).

TABLE 4-6. DoE Phase I Awards by Program Area.

TABLE 4-6

DoE Phase I Awards by Program Area.

There does not appear to be a systematic effort under way at DoE to determine whether the SBIR program supports agency mission. As with all other agencies, to the extent that agency mission involves the development of technical knowledge, there are indicators for whether SBIR companies are producing this knowledge.

4.3.2.4. NSF

NSF is similar to DoE in its reliance on procedures to ensure that agency mission is supported. As with DoE, technical managers are involved in the development of SBIR topics, and there is a strong focus on ensuring that awards are made only within defined topic areas. Topic development also aims to involve technical area program managers.

The National Science Foundation (NSF) is an independent federal agency created by Congress in 1950 “to promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense …”

SOURCE: National Science Foundation, <http://www.nsf.gov/about>.

The SBIR program has taken on the specific role and mission of being the commercialization arm of NSF; the NSF program manager has been explicit in focusing the program on commercialization, arguing that the remaining 97.5 percent of NSF is focused on basic science and other primarily noncommercial aspects of scientific inquiry. While at one level this is correct, most of the rest of the NSF budget is largely focused on university research and is not open to small company-based research.

Thus in one sense, support for agency mission is best measured in terms of commercialization (discussed in Section 4.2). In another sense, NSF is indeed focused on support for the generation of new scientific knowledge. Here traditional metrics include patents and peer-reviewed publications. These aspects of mission support for NSF are discussed below. Both topics, and the program as a whole, are covered in more detail in the NSF volume.

4.3.2.5. NASA

NRC used a unique approach to address the question of support for NASA’s agency mission. NRC surveyed agency technical managers (COTRs), who are in charge of the research areas within which SBIR awards are made. The survey sought to measure the quality of the research from the perspective of technical staff who managed both SBIR and non-SBIR programs.

NASA’s mission is to pioneer the future in space exploration, scientific discovery, and aeronautics research.

SOURCE: National Aeronautics and Space Administration, <http://www.nasa.gov/about/highlights/what_does_nasa_do.html>.

The research found that the COTRs believed that 68 percent of surveyed projects generated useful information and found that results from 58 percent of the projects were sufficiently positive to encourage the COTR to seek out additional funding, within or outside the SBIR program.

It is especially encouraging that over 30 percent of projects were sufficiently positive that technical managers sought non-SBIR funding within NASA for further development.

Agency mission at NASA can also be measured in terms of agency take-up of SBIR-funded technologies. This may be especially important at NASA, as the commercial market for space-related technologies is likely to be small. NASA does maintain a set of success stories, but Phase III activities do not appear to be tracked in any coordinated fashion.

FIGURE 4-12. NASA staff perspectives on SBIR awards.

FIGURE 4-12

NASA staff perspectives on SBIR awards.

SOURCE: NRC Project Manager Survey.

4.4. SUPPORT FOR SMALL, WOMAN-OWNED, AND DISADVANTAGED BUSINESSES

4.4.1. Support for Woman- and Minority-owned Firms

Support for women and disadvantaged persons is one of the four primary congressional objectives for the SBIR program. Unfortunately, there are significant concerns regarding the collection of data related to the participation of woman- and minority-owned firms in the program. More importantly, the data that does exist raises questions about whether agencies adequately meet the needs of such firms.

4.4.1.1. Data Concerns

There are problems with the collection of data related to woman- and minority-owned firms at the two largest SBIR agencies. In both cases, these difficulties make it hard to determine how well each agency is meeting this element of the congressional mandate.

At DoD, the problem lies in the collection of data about applications. Even though DoD publishes an annual report which includes data on the number of applications and the success rates of women- and minority-owned firms in winning Phase I and Phase II awards, these data suffer significant deficiencies. For example, the demographic status of applications is entirely self-determined by applicants, and it is likely that in some cases this data is not reported accurately. BRTRC (a data contractor) has identified 53 firms that listed minority or women ownership on some, but not all of the proposals they submitted during FY2005. Looking across years, firms were identified that showed women ownership some years, then no status, then women ownership again. One firm that submitted about 10 proposals annually first listed itself as minority-owned, then for several years claimed no special ownership, then claimed to be woman-owned.

After awards are made and moved to a separate database table, DoD works to correct errors in the demographic status, so the awards data are much more accurate in this regard. Since firms get no preference in selection for being minority- or woman-owned, they may not be motivated to ensure that this part of the application is correct.

Because the applications data are not accurate, conclusions about success rates are also uncertain. And, as discussed below, it is hard to determine what the data mean—and why certain outcomes occur.

At NIH, the NRC study identified anomalies which, on closer inspection, indicated that for some years, NIH was not capturing women- and minority-ownership data accurately. Following discussion with the NRC, NIH made a significant effort to recalculate the data for women and minority participation in the NIH SBIR program. However, apparent anomalies in the NIH data for 2001 and 2002 could not be resolved by the time of publication of this report.37

In addition, it would be helpful if all agencies captured and regularly reported data about the demographics of principal investigators (PIs), as well as company ownership, as many company founders have prior experience as PIs, so PI demographics may be a useful leading indicator of minority- and women-owned businesses in the program.

4.4.1.2. Award Patterns

Of the five agencies studied by the NRC, one—NASA—has no apparent issues in relation to awards to woman- and minority-owned firms. That is to say, their pattern of awards matches those for other firms, and their share of awards is in line with the average for all agencies, and shows no obvious negative trends in any area.

This is not the case for the remaining four agencies. At DoD, there has been a substantial decline in the award shares of minority-owned firms. The share of awards to woman-owned firms has been relatively low at NSF, and the application success rates for woman-owned firms have been lower than those for other businesses. At DoE, success rates for both woman- and minority-owned firms have been lower than for other small businesses. At NIH, issues focus on award shares for minority-owned firms, and on the discordance between the award share to woman-owned firms and the number of female scientists and engineers working in the life sciences. And, as noted the lack of quality data suggest that this issue is not adequately monitored or analyzed, reflecting a need for greater management attention.

DoD

While Phase I awards to woman-owned firms have continued to increase as a percentage of all Phase I awards, the percentage of Phase I awards being made to minority-owned firms has declined since the mid-1990s. The percentage fell below 10 percent for the first time in 2004, and is down by a third since the early 1990s.

The absence of reliable data on the demographics of applications makes it impossible at this point to determine why minority-owned businesses are getting a declining share of Phase I awards. We do not know whether minority-owned business applications are down, whether the success rate of those applications has fallen, or both.

NSF

At NSF, the existence of good applications data makes it relatively easy to determine why woman-owned firms have been doing less well in recent years. Woman-owned businesses have been less successful in getting applications approved than have all applicants in every year since 1994. In half the years, the success rate of woman-owned businesses was less than 70 percent the rate of all applicants. Over the period 1994-2005, woman-owned businesses accounted for 12 percent of all Phase I applications but received less than 10 percent of all Phase I awards.

FIGURE 4-13. Minority-owned business shares of Phase I awards at DoD, 1992-2005.

FIGURE 4-13

Minority-owned business shares of Phase I awards at DoD, 1992-2005.

SOURCE: Department of Defense Awards Database.

FIGURE 4-14. NSF: Comparative success rates for woman-owned and for all applicants in having their Phase I applications approved, 1994-2005.

FIGURE 4-14

NSF: Comparative success rates for woman-owned and for all applicants in having their Phase I applications approved, 1994-2005.

NOTE: This is a correction of the prepublication version released on July 27, 2007.

SOURCE: Developed from data provided by the NSF SBIR program.

FIGURE 4-15. NSF: Comparative Phase II success rates for woman-owned and for all applicants, 1995-2005.

FIGURE 4-15

NSF: Comparative Phase II success rates for woman-owned and for all applicants, 1995-2005.

NOTE: This is a correction of the prepublication version released on July 27, 2007.

SOURCE: Developed from data provided by the NSF SBIR program.

Low success rates occur during Phase II as well. In all except three of the ten years from 1995 through 2005, the success rate of woman-owned businesses fell below that of all businesses in having their Phase II applications funded. Woman-owned businesses contributed 9 percent of all Phase II applications submitted from 1995 to 2003 (203/2,299) and received 7.5 percent of all Phase II grants (79/1,059). From 2002 to 2003, the success rate of woman-owned businesses in getting Phase II grants was particularly low but recently has recovered significantly.

DoE

At DoE, success rates are also an area of interest. Woman-owned businesses accounted for almost 8 percent of all DoE Phase I awards during 1992-2003, and for 10.5 percent of Phase I applications. During the same period, woman-owned businesses accounted for 7.8 percent of Phase II applications in 1995-2003 and for 6.7 percent of actual Phase II awards.

Data on minority-owned firms was similar. They submitted 16.5 percent of all Phase I applications in 1992-2003 and received 13.2 percent of all Phase I awards. Between 1995 and 2003, they accounted for 13.7 percent of Phase II applications and 11.8 percent of Phase II awards.38

DoE is the only agency where minority-owned firms apply more and receive more awards than woman-owned firms.

NASA

The relatively steady trends for both Phase I and Phase II awards at NASA for both woman- and minority-owned businesses are shown below (see Figures 4-16 and 4-17).

FIGURE 4-16. Phase I awards at NASA: woman- and minority-owned businesses’ share of all awards.

FIGURE 4-16

Phase I awards at NASA: woman- and minority-owned businesses’ share of all awards. NOTE: This is a correction of the prepublication version released on July 27, 2007.

FIGURE 4-17. Phase II awards at NASA: woman- and minority-owned businesses’ share of all awards.

FIGURE 4-17

Phase II awards at NASA: woman- and minority-owned businesses’ share of all awards. NOTE: This is a correction of the prepublication version released on July 27, 2007.

The absence of detailed applications data for woman- and minority-owned businesses means that we currently are not able to determine whether these trends are the result of a faster increase in these firms’ number of proposals than other small businesses, improved success rates, or a combination of both.

4.4.2. Small Business Support

At one level, the SBIR program obviously provides support for small business, in that it gives funding only to businesses with no more than 500 employees—the SBA definition of a small business. However, it has been less clear whether SBIR has provided additional support for small business, or simply aggregates existing small business research dollars under the program’s umbrella.

The most direct way to address the question is to review the amount of R&D funding going to small business at the agency as a whole, and then compare that with trends in SBIR funding. Unfortunately, this comparison is publicly available only for one agency, NIH, which publishes separate data on small business shares of research funding.

NIH data (Figure 4-18) show that the share of all NIH small business funding being disbursed through the SBIR program has fallen steadily since soon after the 1983 inception of the NIH program. After peaking at 90 percent of all small business research funding in the mid-1980s, SBIR’s share fell steadily to about 72 percent, before falling further in 2004 (the most recently available data).

FIGURE 4-18

FIGURE 4-18

SBIR share of small business research funding at NIH SOURCE: National Institutes of Health Awards database.

At NIH, the rapid growth of SBIR in recent years has supplemented, not supplanted, small business funding through other mechanisms at NIH. Increasing amounts—and shares—of small business research funding are available outside the SBIR program.

4.4.3. Project-level Impacts

One way of measuring SBIR’s impact is to ask awardees whether their projects would have been implemented without SBIR program funding. Data from the NRC Phase II Survey shown in Figure 4-19 strongly suggest that SBIR provides funding that plays a determinant role to most of the projects that receive it.

FIGURE 4-19. Would the project proceed without SBIR funding? (Percent of respondents.

FIGURE 4-19

Would the project proceed without SBIR funding? (Percent of respondents.) SOURCE: NRC Phase II Survey, Question 13.

According to the respondents, more than 70 percent of projects would likely not have proceeded at all without SBIR. This finding reflects the known difficul ties in funding high-risk early-stage research in all scientific fields. SBIR seems to provide critical funding necessary to fund many early stage projects.

Respondents also indicated that many of the 13 percent of projects that were “definitely” or “likely” to have continued in the absence of SBIR funding would have had significant delays and other changes. More than half of these respondents (54 percent) noted that the project’s scope would have been narrower. Sixty-two percent of projects that would have continued would have been delayed, and 47 percent expected the delay would have been at least 12 months.39

4.4.4. SBIR Impacts Different Types of Companies in Different Ways

Professor Irwin Feller has proposed a typology of companies supported by SBIR funding, capturing the critical differences in company capabilities and aspirations. This section describes, using both data and case studies, how these five different kinds of companies (or different stages of company development) have been supported by the NIH SBIR program.

Feller’s typology differentiates between five types of SBIR-supported company:

  1. Start-up Firm. This is a new firm, typically without marketable products, and usually with minimal funding and limited personnel resources.
  2. R&D Contractor. As described by Cramer, these firms make a strategic choice to specialize in the performance of R&D rather than in marketing products or services.40
  3. Technology Firm. These firms have developed a core technology, which is then deployed into products and services.
  4. Scientific Firm. These businesses are described by Cramer as “firms [that] are generally small and were founded by scientists to explore whether a particular research area can generate ideas or products that might attract investors.”41
  5. Transformational Firm. These companies start out as highly (or partially) dependent on SBIR, or on other government R&D contracts, which they use to develop a product that turns out to have considerable commercial value. This leads the company to become a production-oriented commercial vendor, with a concomitant decrease in the role of SBIR on the firm’s progression.
TABLE 4-7. SBIR Awards and Company Foundation.

TABLE 4-7

SBIR Awards and Company Foundation.

All of these firm types have been identified through case studies as benefiting from SBIR awards. However, SBIR impacts on company formation are an especially important component of the program’s overall support for companies; in addition, the role of R&D contractor companies has been a matter of controversy within the program for some time.

4.4.4.1. Start-ups

Responses to the NRC Firm Survey indicate that just over 20 percent of firms were founded entirely or in part as a result of SBIR awards. These data are supported by a number of the cases examined for this study. One of them is summarized in Box 4-5.42

Box Icon

BOX 4-5

Company Formation: The Case of Sociometrics, Inc. Sociometrics is a successful winner of SBIR awards that has also successfully commercialized some of its SBIR-generated technologies. Over the past twenty years, it has received more than 15 Phase II awards, (more...)

While supporting company formation is not a direct objective of the program, it is easily construed as part of the mandate to support small businesses, and the program does, in fact, meet this important objective.

4.4.4.2. R&D Contractors

R&D contractor-type companies have been the subject of pointed criticism from some observers of the SBIR program. These critics have noted that some of these companies receive substantial SBIR program funds, but produce little in the nature of commercial goods and services. There is also minimal “take-up” of their R&D into the acquisition streams of the procurement agencies. These companies have been derogatively referred to as SBIR “mills,” living off SBIR awards.

This issue is discussed in more detail in the section in Chapter 2 on multiple-award winners. Here, however, we should simply note that this criticism often misses important changes at these companies. The last point has more general application: reliance on SBIR awards tends to decline as the size of the company grows, and most companies that survive do grow. This observation is supported by data from the NRC Phase II Survey.

Table 4-8 shows, for NIH awardees, how responding firms’ SBIR focus compares with their revenue at the time of the survey (2005). The data show that as companies get larger, their reliance on SBIR revenues tends to decline. Of the 36 NIH respondents with at least $5 million in revenues, 30 (78.9 percent) reported no more than a 10 percent focus on SBIR. Conversely, of the 102 firms reporting at least 80 percent focus on SBIR, 100 reported annual firm revenues of no more than $1 million. The data also show that there are a number of companies with revenues of $1-5 million where SBIR accounts for more than half of the revenues.

TABLE 4-8. NIH Phase II SBIR Awards by Overall Company Revenue and Percentage Dependence on SBIR as a Source of Revenue.

TABLE 4-8

NIH Phase II SBIR Awards by Overall Company Revenue and Percentage Dependence on SBIR as a Source of Revenue.

4.4.4.3. Transitioning Firms

A recurring theme in the case studies is that contract research is often used as a bridge to commercialization. For example, Polymer Technologies—an NIH awardee—was originally a contract research house, but it now manufactures a wide range of high technology medical devices at its Berkeley, California plant. It provides OEM services to major device manufacturers around the world.

Similarly, T/J Technologies—an NSF awardee—currently obtains most of its revenue from contract research, but its longer term strategy is to develop partnerships for commercializing its material technologies.

Firms’ strategies evolve, and in particular, companies that now only do con tract research may have ambitions or find it necessary to move to other models. Agencies’ push to commercialization can encourage this process.

4.4.5. Growth Effects

No existing data sets measure the effect of SBIR on company growth. However, NRC survey respondents did provide their own estimates of SBIR impacts on their companies’ growth.

Almost 55 percent of respondents indicated that more than half of the growth experienced by their firm was directly attributable to SBIR. This is evidence of the powerful impact of SBIR on the future development of firms winning SBIR awards.

FIGURE 4-20. SBIR impacts on company growth: Percent of company growth attributable to SBIR awards.

FIGURE 4-20

SBIR impacts on company growth: Percent of company growth attributable to SBIR awards.

SOURCE: NRC Firm Survey.

4.4.6. Conclusions

SBIR supports small high technology businesses at a time when other sources of financial support are especially difficult to find. Businesses use these funds for a variety of purposes, in pursuit of several distinct strategies, as captured in Box 4-6.

Box Icon

BOX 4-6

SBIR as an Enabler and Lifeline for Some High-Tech Companies. Faraday Technologies, Inc. The SBIR program enabled the company to undertake research that otherwise it would not have done. It sped the development of proofs of concept and pilot-scale prototypes, (more...)

4.5. MULTIPLE-AWARD WINNERS AND NEW PROGRAM ENTRANTS

In this section, we will review the data about both the incidence of firms winning multiple SBIR awards and the extent to which the program is open to new companies that have not previously won SBIR awards.

4.5.1. The Incidence of Multiple-Award Winners

The absence of a high quality unified database of award winners makes it difficult to track MAWs. Because there is no single ID for SBIR firms, MAWs must be tracked using firm names, and this method is not reliable: not only do firms change names, even the smallest change in the spelling of a firm name on the application cover sheet will create a second record for that firm.

4.5.2. Commercialization and Multiple-Award Winners

While the DoD commercialization database does not contain information on all companies or all awards,43 it does provide the best data on overall outcomes from awards to multiple SBIR-award winners, partly because it is specifically designed to do so. DoD has been concerned to ensure that companies winning more than four or five awards (the number changed in 2005) are generating results that support DoD’s mission.44

The DoD data covers firms and awards from all agencies, reported within the DoD system (by firms that apply for DoD awards). It is therefore incomplete. However, it does show both the fact that a relatively small number of companies account for a larger share of awards, and also that there are now very many firms that win a great many awards.

The top 1 percent of winners—27 firms—accounted for 16.8 percent of all recorded Phase II projects. The top 7 percent of award winners—200 firms—accounted for 46.5 percent of all awards.

Of course, these data alone do not determine whether any of these firms are “mills”—whether they do not commercialize, and whether they are SBIR-dependent for revenues.

TABLE 4-9. DoD Data on Multiple Award Winners, All Agencies, 1992-2005.

TABLE 4-9

DoD Data on Multiple Award Winners, All Agencies, 1992-2005.

The first question can be answered, at least in aggregate. As noted, data from the DoD commercialization database indicates that the largest award winners are in fact—on average—stronger commercializers than firms with few awards.

The 2,920 firms reporting through the commercialization database reported a total of $20.3 billion in commercial outcomes, including $13.2 billion in sales. The five companies with more than 124 projects generated approximately 60 percent greater commercial returns on average than did the projects with fewer than 15 awards, and accounted for $1.4 billion in sales.

The 7 percent of firms with the highest number of awards—which accounted for 46.5 percent of awards, accounted for 60 percent of the commercial outcomes. This does provide a rebuttal to the claim that multiple winners are simply mills: At the most basic level, the data show that the multiple winners commercialize more than firms receiving fewer awards.

Thus the criticism of MAWs seems in general to be misplaced. It results from an overly negative reading of limited data focused too tightly on raw commercialization data, and perhaps on assessments of the program in its first years of operation.45

A fuller assessment of the role of multiple winners reveals multiple dimensions, and multiple contributions:

  • Active Commercialization. Aggregate data from the DoD commercialization database indicates that the companies winning the most awards are, on average, more successful commercializers than those winning fewer awards.
    • While data from this source are not comprehensive, they do cover the vast majority of MAWs—at least at DoD—and the data indicate that, on average, firms with the largest number of awards commercialize as much or more than do all other groups of awardees; in the aggregate, there is failure of multiple-award winners to commercialize.
    • Case studies also show that some of the biggest award winners have successfully commercialized, and have also met the needs of sponsoring agencies in other ways.
TABLE 4-10. Commercialization Results from Multiple Award Winners.

TABLE 4-10

Commercialization Results from Multiple Award Winners.

  • Declining SBIR Revenues. For some MAWs at least, even though they continue to win a considerable number of awards, the contribution of SBIR to overall revenues has declined: At the commonly cited “mill” firm Radiation Monitoring Devices, for example, SBIR now only accounts for 16 percent of total firm revenues.
  • Graduation. Some of the biggest Phase II winners have graduated from the program either by growing beyond the 500 employee limit or by being acquired. Foster-Miller, for example, was bought by a foreign-owned firm. Legislating to solve a problem with companies that are, in any event, no longer eligible seems inappropriate.
  • Agency Research. Contract research can be valuable in and of itself. Agency staff indicate that SBIR fills multiple needs, many of which do not show up in sales data. For example, some agency staff have noted that they use SBIR awards to conduct low-cost probes of the technological frontier. Awards are conducted on time and on budget, and can effectively test technical hypotheses, potentially saving extensive time and resources later.
  • Spin-off Companies. Some MAWs spin off companies to commercialize innovations. Creating new firms is a valuable impact and a valuable contribution of the program.
  • Rapid Response. MAWs have provided the highly efficient and flexible capabilities needed to solve pressing problems rapidly. For example, Foster Miller Inc. responded to needs in Iraq by developing and the manufacturing add-on armor for Humvees.46

All these points suggest that while there have been companies that depend on SBIR as their primary source of revenue for a considerable period of time, and there are some who fail to develop commercial results, such behavior is limited and does not constitute a significant impediment to SBIR program goals.

There is one final point to consider. Given the fact that SBIR awards meet multiple agency needs and multiple Congressional objectives, it is difficult to see how the program might be enhanced by the imposition of an arbitrary limit on the number of applications per year, as is currently the case at NSF. The evidence—outlined above—supports the conclusion that there is no multiple winner problem per se, although there may be “problem,” i.e., underperforming, firms—a judgment the agency management must make. To the extent agencies continue to see issues in this area, we would strongly suggest that they consider adopting some version of the DoD “enhanced surveillance” model, in which multiple winners are subject to enhanced scrutiny in the context of the award process. The exclusion model used by NSF seems simply inappropriate.

4.5.3. The Incidence of New Entrants

Just as analysis of multiple winners provides a view of award concentration, analysis of new entrants provides a view of award dispersion. The data unequivocally show that the agency SBIR programs are open to significant numbers of new entrants.

4.5.3.1. National Science Foundation

In every year except 1999, the percentage of NSF award-winning companies that have not previously won Phase I awards at NSF is over 50 percent, and the percentage increased by more than half since 1999, to 63.1 percent in 2003.

4.5.3.2. National Institutes of Health

NIH is unique among the agencies in providing data on new applicants—companies not previously funded at NIH—and on new awards to those companies.

The trendline shows that the share of applications from previously unfunded companies has actually increased somewhat, and, in any event, remains within fairly narrow boundaries (58-63.5 percent of applications come from previously unfunded companies). Given that previously funded companies have already passed a quality screen, it is not surprising that the share of awards going to previously unfunded companies is lower.

Nonetheless, at least 35 percent of awards have gone to previous nonwinners in each year since 2000, although that share has declined from 47 percent in 2000 to just over 35 percent in 2005. This decline might partly reflect the fact that the number of previous winners has increased sharply during this period, and that many of these new “previous winner” firms continue to apply for more awards.

4.5.3.3. Department of Defense

At Defense, the data appear to show similar award patterns for Phase II awards. According to the SBIR program office at DoD, 37 percent of FY2005 Phase II awards went to companies that had not previously won a Phase II SBIR award from DoD.

This steady infusion of new firms constitutes a major strength of the program. It underscores the positive impact of SBIR awards, as they encourage innovation activity across a broad spectrum of firms, create additional competition among suppliers for the procurement agencies, and provide agencies with new mission-oriented research and solutions.

4.6. SBIR AND THE EXPANSION OF KNOWLEDGE

Quantitative metrics for assessing knowledge outputs from research programs are well-known, but far from comprehensive. Patents, peer-reviewed publications, and, to a lesser extent, copyrights and trademarks, are all widely-used metrics, and are discussed in detail below.

However, these metrics do not capture the entire transfer of knowledge involved in programs such as SBIR. Michael Squillante, Vice-President for Research at Radiation Monitoring Devices, Inc., points out that there may be benefits from the development and diffusion of knowledge that are not reflected in any quantitative metric:

For example, our research led to a reduction in the incidence of stroke following open-heart surgery. Under an NIH SBIR grant we developed a tool for medical researchers who were examining the causes of minor and major post-operative stroke occurring after open-heart surgery.47

FIGURE 4-21. Percentage of Phase I awards to companies new to the NSF SBIR program, 1996-2003.

FIGURE 4-21

Percentage of Phase I awards to companies new to the NSF SBIR program, 1996-2003.

SOURCE: National Science Foundation.

FIGURE 4-22. “New” Phase I applicants (percent of all applicants) at NIH, FY2000-2005.

FIGURE 4-22

“New” Phase I applicants (percent of all applicants) at NIH, FY2000-2005.

SOURCE: National Institutes of Health data.

FIGURE 4-23. Percentage of Phase I awards to companies new to the NIH SBIR program, FY2000-2005.

FIGURE 4-23

Percentage of Phase I awards to companies new to the NIH SBIR program, FY2000-2005.

SOURCE: National Institutes of Health data.

FIGURE 4-24. New Phase II winners in the DoD SBIR program, FY2005.

FIGURE 4-24

New Phase II winners in the DoD SBIR program, FY2005.

NOTE: Firm data as of March 2006. The “New Awardees” category includes prior Phase I award winners that have not won a Phase II award.

SOURCE: Michael Caccuitto, DoD SBIR/STTR Program Administrator, and Carol Van Wyk, DoD CPP Coordinator, Presentation to SBTC “SBIR in Rapid Transition Conference,” Washington, DC, September, 27 2006.

This was not a project whose success could be measured in commercial terms; the market (stroke researchers) is far too small to generate large revenues. However, the impact of the research was profound.

It is, therefore, quite important to understand that the quantitative metrics discussed below are an indicator of the expansion of knowledge; they reflect that expansion but do not fully capture it. In particular, they say little about the impact of that knowledge. In the case illustrated above, for example, there was no patent, no commercial sales, no impact on the company’s bottom line—but the research, nevertheless, made a remarkable improvement in outcomes for stroke patients undergoing surgery.

4.6.1. Patents

According to the Small Business Administration, small businesses produce 13 to 14 times more patents per employee than do large patenting firms. These patents are twice as likely as are large firm patents to be among the one percent most cited.48

With respect to SBIR, the NRC survey data indicate that about 30 percent of respondents received patents related to their SBIR research. About two-thirds of projects generated at least one patent application, and as with most other SBIR metrics, the distribution of outcomes is highly skewed.

The data show that three companies account for almost 20 percent of all patent applications related to the surveyed SBIR award. However, they were not especially successful, as no company reported more than 20 successful patents granted. Overall, 2.2 percent of the responding companies applied for more than five patents, and 1.3 percent received more than five.

More detailed case study analysis also indicates the importance of intellectual property to firm strategies. At NSF, a number of the case studies showed this clearly.

4.6.2. Scientific Publications

The NRC Phase II Survey determined that slightly more than half of the respondents had published at least one related scientific paper (45.4 percent). About one-third of those with publications had published only a single paper, but one company had published 165 papers on the basis of its SBIR research, and several others had published at least 50 papers.

These data fit well with case studies and interviews, which suggested that SBIR companies are proud of the quality of their research. Publications are featured prominently on many Web sites, and companies like Advanced Brain Monitoring, SAM Technologies, and Polymer Research all made the point during interviews that their work was of the highest technical quality as measured in the single measure that counts most in the scientific community—peer-reviewed publication.

Publications therefore fill two important roles in the study of SBIR programs:

  • First, they provide an indication of the quality of the research being conducted with program funds. In this case, more than half of the funded projects were of sufficient value to generate at least one peer-reviewed publication.
  • Second, publications are themselves the primary mechanism thorough which knowledge is transmitted within the scientific community. The existence of the articles based on SBIR projects is therefore direct evidence that the results of these projects are being disseminated widely, which in turn means that the congressional mandate to support the creation and dissemination of scientific knowledge is being met.

We note that no agency seems to have in place evaluation programs for determining whether similar knowledge effects are being generated at the same or at a different rate outside the SBIR program. Likewise, there is insufficient tracking to determine citation rates for these articles.

4.6.3. SBIR and Universities

According to agency and congressional staff, one of the implicit, though not formal objectives, of the SBIR program is that it supports the transfer of knowledge from universities to the commercial marketplace.

This transfer can happen in many different ways. SBIR funding:

  • Helps university scientists form companies;
  • Enables small firms to use university faculty and to employ graduate students as specialized consultants;
  • Permits the use of university laboratory facilities; and
  • Encourages other less formal types of collaboration.
TABLE 4-11. Distribution of Patents.

TABLE 4-11

Distribution of Patents.

BOX 4-7Intellectual Property and Company Strategy Among NSF Awardee Case Studies

A number of NSF case studies showed extensive use of patenting:

  • Faraday Technology has had 23 patents issued in the U.S. and three foreign issued, since its founding in 1991. Historically, this amounts to 1.4 issued patents per employee. Patents and the fees they generate are the central focus of Faraday’s business strategy, and the company investigates citations by other companies of its patents to obtain knowledge about potential customers.
  • Immersion Corporation has more than 270 patents issued in the U.S. and another 280 pending in the U.S. and abroad. According to Immersion, its patent portfolio is at the heart of its wealth-generation capacity.
  • ISCA has received a trademark on its most recent insect lure technology, which is expected to generate substantial growth for the company.
  • Language Weaver has more than 50 patents pending worldwide, which underpin its commercialization approach. MER has had more than a dozen patents granted.
  • MicroStrain reported patenting to be “very important” to its commercialization strategy.
  • NVE reported 34 issued U.S. patents and more than 100 patents issued worldwide.
  • T/J Technologies has had seven patents granted and has a number of others pending.

Over a third (36.5 percent) of all NRC Phase II Survey respondents indicated that there had been involvement by university faculty, graduate students, and/or a university itself in developed technologies. Based on the responses of all survey respondents, this involvement took a number of forms (see Table 4-13).

TABLE 4-13. University Involvement in SBIR Projects.

TABLE 4-13

University Involvement in SBIR Projects.

The wide range of roles played by university staff and students underscores the multiple ways in which SBIR projects enhance the knowledge base of the nation. Involvement in these projects provides different opportunities for university staff than those available within academia.

Many of the companies interviewed for this study noted that their university connections were extremely important. Radiation Monitoring Devices said that they spent more than $1 million annually on contracted research at universities under their SBIR awards.49 Advanced Targeting Systems, in San Diego, has forged an extended and successful research partnership with a senior scientist at the University of Utah, which they described as “critical to the development and testing of their products.”50

TABLE 4-12. Publications.

TABLE 4-12

Publications.

These stories and the many others like them that we encountered suggest that the flow of information and funding between small businesses and universities working within the SBIR framework is not simple or unidirectional. For example, the constant flow of feedback, testing, and insights between university researchers and Advanced Targeting Systems staff helped to move the company forward toward product deployment into new research areas while providing real-life problems for research to address.

What is now clear from this research—and could certainly be the subject of further analysis—is the extent to which universities themselves see SBIR as a mechanism for technology transfer, commercialization, and additional funding for university researchers.

4.6.3.1. University Faculty and Company Formation

Data from the NRC Firm Survey strongly support the hypothesis that SBIR has encouraged university faculty to form companies to commercialize their inventions and technologies. Two-thirds of all responding companies had at least one academic founder, and more than a quarter had more than one.

The same survey found that about 36 percent of founders were most recently employed in an academic environment before founding the new company.

These data and evidence from case studies strongly suggests that SBIR has indeed encouraged academic scientists to bring their work to a more commercial environment.

4.7. CONCLUSIONS

The extended discussion of metrics and indicators in the Methodology Report show quite conclusively that there is no single simple metric that captures “results” from the program. Each of the four congressional mandates is best assessed separately, and within each, there are a multiple issues to be addressed.

Bearing all these points in mind, it is still possible to summarize the results of our research in straightforward terms.

Commercialization. Approximately 30-40 percent of Phase II projects produce innovations that reach the market, with a few generating substantial returns. Other indicators of commercialization, such as licensing activities, marketing partnerships, and access to and utilization of further investments from both private and public sources, all confirm that while returns are highly skewed, the results in general are positive, and are consistent with other public and private sources of early-stage financing.

Agency Mission. We found many examples of funded projects that clearly made a major contribution to the mission of the awarding agency. To an increasing extent, SBIR is becoming more firmly integrated into the overall R&D strategy of each agency. At the procurement agencies—DoD and NASA—we found that data from the agencies and from NRC surveys indicated that many SBIR projects did result in utilization by the agencies. We also determined that the agencies had made important efforts to improve the alignment between agency needs and SBIR implementation, and that procedures for encouraging or ensuring alignment were in place at all agencies. Traditionally hostile views of the program among many acquisition and R&D managers have begun to shift in a much more positive direction. Many projects made a very valuable contribution to agency mission, and these contributions were often far out of proportion to the commercial returns involved.51

Support for Small Business, and Woman- and Minority-owned Businesses. SBIR significantly supports small high technology businesses in general, and the NRC research team discovered that SBIR had an important catalytic effect in terms of company foundation—providing the critical seed money to fund a company’s first steps. SBIR also had strongly influenced companies’ decisions to initiate individual projects: more than 65 percent of NRC Phase II Survey respondents at every agency believed that their projects would not have gone forward without SBIR, and, of the remainder, most believed that the projects would have been delayed and/or would have had a reduced scope. While some efforts have been made to increase participation by woman- and minority-owned businesses, more needs to be done. At DoD and NIH data on these businesses were incomplete or missing. At all of the agencies except NASA, what data there was suggested that there were areas of concern in the program (different areas at different agencies), and it did not appear that the agencies had focused sufficient resources and management attention on these areas.

Support for the Advancement of Scientific and Technical Knowledge. The program funds cutting edge research, as it was designed to do. Most of the agencies use “technical innovation” in some form as a critical selection parameter. Outcomes in the form of patents and peer-reviewed scientific publications are encouraging, with about two-thirds of recipients publishing at least one related scientific paper. Surveys of program managers and technical points of contact at DoD, NASA, and DoE indicated that they saw SBIR projects as approximately equivalent in research quality to non-SBIR projects.52 Moreover, case studies indicate that knowledge generated in SBIR-funded research can be picked up indirectly in many ways, and often continues to be productive long after the original project has been concluded.

It is therefore appropriate to conclude that the program is meeting all four of the congressional objectives.

Footnotes

1

Each of the SBIR-funding agencies has a different mission, described in the agency volumes of this study. For a review of the different ways the program is operated at the five agencies under review, see National Research Council, SBIR: Program Diversity and Assessment Challenges, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2004.

2

National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, Charles W. Wessner, ed. Washington, DC: The National Academies Press, 2004, pp. 20-21. For a broader discussion of the scope and limitations of surveys by the University of Michigan Survey Research Center, see Robert M. Groves et al., Survey Methodology, Wiley-IEEE, 2004.

3

2005 saw some 3,000 deals worth an average of $7.35 million according to data from National Venture Capital Association. See National Venture Capital Association Web site, <http://www.nvca.org/ffax.html>.

4

For a discussion of this and related methodological challenges, see, National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, op. cit.

5

A growing body of evidence, starting in the late 1970s and accelerating in the 1980s indicates that small businesses were assuming an increasingly important role in both innovation and job creation. See, for example, J. O. Flender and R. S. Morse, The Role of New Technical Enterprise in the U.S. Economy, Cambridge, MA: MIT Development Foundation, 1975, and David L. Birch, “Who Creates Jobs?” The Public Interest, 65:3-14, 1981. Evidence about the role of small businesses in the U.S. economy gained new credibility with the empirical analysis by Zoltan Acs and David Audretsch of the U.S. Small Business Innovation Data Base, which confirmed the increased importance of small firms in generating technological innovations and their growing contribution to the U.S. economy. See Zoltan Acs and David Audretsch, “Innovation in Large and Small Firms: An Empirical Analysis,” The American Economic Review, 78(4):678-690, September 1988. See also Zoltan Acs and David Audretsch, Innovation and Small Firms, Cambridge, MA: The MIT Press, 1990.

6

U.S. General Accounting Office, Federal Research: Small Business Innovation Research Shows Success But Can be Strengthened, GAO/RCED-92-37, Washington, DC: U.S. General Accounting Office, 1992.

7

PL 99-443, October 6, 1986.

8

PL 102-564 October, 28, 1992.

9

These changes are described by R. Archibald and D. Finifter in “Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, Charles W. Wessner, ed., Washington, DC: National Academy Press, 2000.

10

See National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, op. cit.

11

Interviews with SBIR program coordinators at DoD, NIH, NSF, and DoE.

12

Pete Linsert, CEO, Martek, Inc., meeting of the NRC Committee on Capitalizing on Science, Technology, and Innovation: An Assessment of the Small Business Innovation Research Program, June 5, 2005.

13

National Research Council, An Assessment of the SBIR Program at the Department of Defense, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009.

14

Much of the primary data in this section of the report was derived from the NRC’s Phase II Survey. See Appendix A for additional information about the NRC’s Phase II Survey, including response rates.

15

See the section on venture capital in National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009.

16

See National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit., Chapter 4, where data from the DoD Commercialization database and the NIH Phase II Survey are compared with the NRC data.

17

The DoD database captures commercialization data from NIH projects where the firm subsequently applied for a DoD SBIR award.

18

See Chapter 4 in National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit.

19

Eighteen percent of responding projects (242 firms) reported finalized licensing agreements, and 19 percent (249 firms) reported ongoing negotiations. NRC Phase II Survey, Question 12.

20

This included some 989 respondents.

21

Venture funding of awardees at NIH is discussed in more detail in Chapter 4 of the NIH report, National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit.

22

See National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit.

23

For a discussion of the “halo effect” from awards by the Advanced Technology Program, see Maryann Feldman and Maryellen Kelley “Leveraging Research and Development: The Impact of the Advanced Technology Program,” in National Research Council, The Advanced Technology Program: Assessing Outcomes, Charles W. Wessner, ed., Washington, DC: National Academy Press, 2001.

24

See the Chapter 4 in National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit.

25

NRC Phase II Survey, Questions 16.

26

See Neurocrine case study in National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit.

27

NRC Phase II Survey, Question 12.

28

National Research Council, An Assessment of the SBIR Program at the Department of Defense, op. cit.

29

Department of Energy Web site, accessed at <http://www.energy.gov/about/index.htm>.

30

More details about each agency’s selection procedures can be found in the individual agency volumes.

31

National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit.

32

Note that percentages do not add up to 100 percent, as respondents are permitted to select more than one affected population.

33

See more extended discussion of this issue in Chapter 6 of National Research Council, An Assessment of the SBIR Program at the Department of Defense, op. cit.

34

While these results are impressive, it is likely that DD350 data at other components will undercount Phase III successes unless those components undertake a similar effort to improve the quality of their data.

35

See National Research Council, An Assessment of the SBIR Program at the Department of Defense, op. cit.

36

See statements by Charles Holland, Deputy Under Secretary of Defense for Science and Technology in National Research Council, SBIR and the Phase III Challenge of Commercialization, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2007.

37

This is a correction of the text in the prepublication version released on July 27, 2007.

38

Data are unreported by woman- and minority-ownership status for 1992-1994.

39

NRC Phase II Survey, Questions 14 and 15.

40

See Reid Cramer, “Patterns of Firm Participation in the Small Business Innovation Research Program in Southwestern and Mountain States,” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, op. cit.

41

Ibid.

42

See Sociometrics case study in Appendix D for a more detailed review of SBIR at this company.

43

According to BRTRC, which manages the DoD database, the data collected from the agencies on Phase II awards made from 1992 to 2001 identified 2,257 firms that had received at least one Phase II, but were not in the DoD database, and were therefore not included in the table. Of these 2,257, only six had received 15 or more Phase II during the ten years for which BRTRC received award data. Although inclusion of pre-1992 and post-2001 awards would have increased that number, it seems safe to conclude that the firms in the DoD data represent a large majority of multiple winners.

44

Although the database covers all agencies, some agencies are underrepresented owing to the focus on DoD-oriented firms. NIH awardees, for example, account for only 7 percent of entries in the database.

45

See, for example, U.S. General Accounting Office, Federal Research: Small Business Innovative Research Shows Success But Can be Strengthened, GAO/RCED-92-37, op. cit.

46

Foster-Miller’s Last Armor TM, which uses Velcro-backed tiles to protect transport vehicles, helicopters and fixed wing aircraft from enemy fire, was developed on two Phase I SBIRs and a DARPA Broad Agency Announcement. The technology helped improve the safety of combat soldiers and fliers in Bosnia and Operation Desert Storm. Access at <http://www.dodsbir.net/SuccessStories/fostermiller.htm>. (This is a correction of the text in the prepublication version released on July 27, 2007.)

47

Presentation to NRC Research Team by Michael Squillante, vice president, Radiation Monitoring Devices, Inc. June 11, 2004.

48

Accessed on May 16, 2007 at <http://app1.sba.gov/faqs/faqindex.cfm?areaID=24>. Drawing on seminal empirical research, Acs and Audretsch found that small business have comparatively higher rates of innovation—specifically, that “the number of innovation increases with increased industry R & D expenditures but at a decreasing rate. Similarly, while the literature has found a somewhat ambiguous relationship between concentration and various measures of technical change, our results are unequivocal—industry innovation tends to decrease as the level of concentration rises.” See Zoltan J. Acs and David B. Audretsch, “Innovation in Large and Small Firms: An Empirical Analysis,” op. cit.

49

Interview with Michael Squillante, vice president, Radiation Monitoring Devices, June 10, 2005.

50

See Advanced Targeting System case study in National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit.

51

The use of Savi RFID tracking systems reportedly enabled large reductions in shipping requirements for U.S. force deployments. The potential savings far outweighs the commercial value of the tracking devices. See the Savi Technology case study in Appendix D of National Research Council, An Assessment of the SBIR Program at the Department of Defense, op. cit.

52

After results were normalized to eliminate the most favorable and most negative responses.