NRC STUDY FINDINGS
The SBIR program at the Department of Defense is meeting the legislative and mission-related objectives of the program. The program is contributing directly to enhanced capabilities for the Department of Defense and the needs of those charged with defending the country.
- A substantial percentage of SBIR projects at DoD commercialize. 1
- The NRC Phase II Survey, which was sent to all firms with Phase II awards from 1992 to 2002, provides evidence of substantial, if highly skewed, commercialization. 2
- Nearly half (46 percent) of respondents indicated that the surveyed SBIR project had reached the marketplace (i.e., they reported more than $0 in sales and licensing revenues from the project by May 2005, which is the closing date of the survey).3
- Of the 420 projects reporting some sales, just under 1 percent reported sales greater than $50 million, another 9.2 percent of projects reporting some commercialization, indicated sales between $5 million and $50 million.4
- For projects that have received sales, survey responses indicate that 87.6 percent of first sales occurred within 4 years of the Phase II award date.7 Interviews and cases, however, support the view that the bulk of sales will be realized in the longer run—that is, beyond the date of first sale.
- These figures, while positive, necessarily reflect the concentration and skewed outcomes often associated with early-stage funding and the special challenge of the procurement process. The figures also understate, perhaps substantially, the amount of commercialization ultimately to be generated from the funded projects. It is important to recognize that these data constitute only a snapshot of sales and licensing revenues, as of May 2005. Projects completed in more recent years will continue to generate revenues well into the future. Consequently, the data aggregated for the May 2005 snapshot necessarily under-reports the eventual return from the SBIR Phase II awards that were made during the latter part of the study period (1992–2002).8
- Success in attracting further research funding for ongoing development offers evidence indicating that a project is on the path to commercialization, even if no sales have yet been made.
- Over one-quarter of projects that received additional funding reported the acquisition of additional funds from other federal sources and 13.2 percent reported funding from other companies.9 This suggests significant interest in these projects—not least from DoD, and possibly also among the prime contractors (a likely source of funding from “other companies”).
- Venture capital is not widely available to companies primarily focused on the Defense market. Hurdles associated with regulations in federal acquisition, and the limited size of many defense markets tends to limit venture funds’ interest in the DoD market. Only 30 projects—3.8 percent of respondents with some additional funding—reported receiving venture capital,10 although the average VC investment that was received is much higher than the average investment received in each other category, at more than $5 million per project.11
- Additional SBIR awards are a further signal of commercial potential.
- Given that SBIR is a highly competitive program, the acquisition of related SBIR awards also suggests that a project is moving along the development path toward commercialization, not least because commercialization potential has become a significant component in the decision to make an SBIR award. 43.5 percent of respondents indicated that they had received at least one additional related SBIR award.12
- SBIR is in broad alignment with the needs of the DoD agencies and components.
- DoD’s SBIR program has contributed to significant enhancements of its mission capabilities.
- A central mission of the DoD SBIR program is to use the inventiveness of small companies to solve DoD’s technical problems, and to develop new technologies that can be applied to the weapons and logistics systems that are eventually used by the Armed Forces.13
- Improved mission capabilities in the context of DoD relate to maintaining technological dominance in battle space conditions, increased responsiveness to new, unexpected situations, such as responding to improvised explosive devices (IEDs), and reductions in the cost of operations and support systems.
- DoD SBIR program managers also speak favorably about the creativity of the small- and medium-sized firms that comprise the SBIR community.14
- DoD appears to be making a concerted effort to ensure that SBIR projects are aligned with the needs of the weapons system managers. 15
- SBIR offers an unusual degree of execution year flexibility and a short planning horizon, permitting the program to rapidly address urgent mission needs.
- SBIR-supported innovations have contributed to enhanced U.S. combat capabilities, and provided technological solutions to meet sudden, unexpected challenges to our military. Indeed, the high degree of flexibility characteristic of small firms means that SBIR has provided DoD with an increased number of suppliers capable of quickly responding on short notice to unexpected battlefield situations.20
- For example, the Navy has taken advantage of this flexibility to issue a “quick response IED topic” in 2004, and made 38 Phase I awards within 5 months of topic development. These developed into 18 Phase II awards, and results from these will be available in 2006–2007. The first prototypes were expected in Iraq in the Fall of 2006.
- By contrast, most RTD&E accounts require considerable forward planning.
- SBIR increases the number of potential suppliers for new technologies, and also creates new opportunities for these firms to partner together in new undertakings.
- Used effectively, SBIR can act as a low-risk, low-cost technology probe and a search tool for finding new technology suppliers. It has helped DoD personnel learn about new technologies, new applications, and a new set of high-tech firms with whom they would not otherwise have contact.21
- The laws governing the SBIR program permit the use of sole source procedures when federal agencies acquire technologies developed with SBIR funding. This allows substantially faster acquisition than through standard channels and acts as a powerful incentive for SBIR firms and their partners.
- The quality of SBIR-funded research is broadly comparable to that of other non-SBIR research according to the NRC Project Manager Survey.
- In the NRC Project Manager Survey, 53 percent of Technical Points of Contact (TPOC) respondents indicated that the specific SBIR project identified in the survey produced results that were useful to them and that they had followed up on this work with other research. SBIR projects are normally part of a wider portfolio of research responsibilities handled by TPOCs.22
- The NRC Project Manager Survey also indicated that the quality of SBIR funded research is comparable to non-SBIR research they manage. Normalized survey scores indicate that the quality of SBIR research is equivalent to that of other research at DoD.23
- The Department has devoted considerable recent effort to strengthen the critical connection between SBIR and the acquisition programs through Phase III. While this focus is to be commended, the Phase III process can be considerably widened and improved.
- A striking aspect of SBIR Phase III at DoD is the extraordinarily uneven character of outcomes and activities between services, and between components within services.
- Notably, Phase III transitions at PEO SUBS account for approximately 86 percent of all Navy Phase III contracts, and Navy in turn accounts for about 70 percent of all DoD Phase III contracts, as captured by the DD350 forms completed by contracting officers.24
- This skew partly results from the additional effort made by Navy to ensure that DD350 forms are completed and accurately reflect SBIR contributions, which in turn reflects different views of the importance of supporting SBIR, as evidenced by the amount of resources, staff, and funding that services and components allocate for program support.
- At some components, such as Navy, senior management recognizes the potential value of SBIR and has supported extensive efforts to build effective bridges between SBIR and the acquisition programs. At other components, efforts have been less well supported, and on the basis of the data presented in Chapter 5, they appear to be considerably less effective.
- There is no effective and comprehensive tracking system within DoD to follow SBIR-funded technologies to their final outcome.
- The Company Commercialization Report is self-reported data, and must be updated only when a company applies for a new DoD SBIR.
- The agency’s DD350 reporting system may substantially undercount Phase III awards, as contracting officers must be specifically trained to capture this data correctly.25
- The lack of a reliable and effective tracking system for SBIR awards that would identify follow-on funding sets back efforts to assess the impact of the program and to document its successes.
- SBIR awards made by DoD support small businesses in a number of important ways.
- SBIR awards have had a substantial impact on participating companies.
- Company Creation. Just over 25 percent of companies responding to the NRC Firm Survey indicated that they were founded entirely or partly because of a prospective SBIR award.26
- The Decision to Initiate Research. Only 13 percent of DoD project respondents thought that their project would “definitely” or “probably” have gone ahead without SBIR funding. Over two-thirds (about 70 percent) thought they definitely or probably would not have initiated the research; most of those who anticipated that their project would have gone ahead without the award acknowledged the likelihood of substantial delays without the award.27
- Company Growth. Almost half (48 percent) of the respondents indicated that more than half the growth experienced by their firm was directly attributable to SBIR.28
- Partnering. SBIR funding is often used by small firms to gain access to outside resources, especially 2-02 academic consultants and, often, to seek company partners.
- SBIR Awards Attract Participation by New Firms.
- FY2005 data from DoD show that 29 percent of Phase I awards went to firms that had not previously won a DoD Phase II SBIR award.
- DoD data indicate that a further 50 percent of awards went to companies with five or fewer previous Phase II awards. Only 13 percent of awards went to companies with at least six Phase II awards.29
- Data from the Naval Air Systems Command (NAVAIR) show that about half of all its Phase I contracts go to companies that have never won an SBIR award from NAVAIR before. In addition, about 40–45 percent of Phase II contracts go to newly participating firms.30
- Multiple SBIR awards serve multiple objectives for firms and agencies. A small number of companies receive multiple awards. A few companies have received more than 100 awards at DoD (over 20 years), and are higher in commercialization (on average).31
- The myth of the linear innovation model:
- The program has often encountered criticism with regard to firms—sometimes labeled SBIR “mills”—that are described as winning large numbers of awards but that achieve lower levels of commercialization than firms with significantly fewer awards.32 Implicit in this view of multiple-award winners is the simple linear model of the program. In reality, multiple awards serve a variety of functions, as noted below.
- Role of multiple awards. More than one award is often required to develop a technology or firm capability, quite apart from the commercialization of a complete product. These needs reflect the complex and differing characteristics of firms.
- Diversity of firm objectives. Reflecting the diversity of the program’s objectives and of the participants, some firms approach the SBIR award process at different stages of development and with different objectives. Some firms are developing technology concepts; some firms see their vocation as contract research organizations; others actively seek to develop commercial products, either for public agencies or for the marketplace.33
- Diversity in firm strategy. For example, investigator-led firms, limited in size and focused on a single concept may seek multiple awards as they advance research on a promising technology.34 For firms that carry out research as a core activity, success is often measured in multiple contract awards
- Addressing agency missions. Some firms, mainly at DoD, have won large numbers of awards over the life of the program. Yet, even with many awards, there is nothing intrinsically wrong with a process that makes high-quality research available to the department at relatively low cost. Some of this research is intrinsically noncommercial, but may have considerable value.35
- Identifying dead ends. Inexpensive exploration of new technological approaches can be valuable, particularly if they limit expenditure on technological dead ends. For research oriented firms, the key issue is the quality of the research and its alignment with service and agency needs.36 Each of the seven most frequent winners, who have received over 100 Phase II awards since the program inception, has a large number of researchers who submit proposals. The high number of quality proposals can produce a high number of awards. Some successful applicants use spin-off firms to commercialize the results of their SBIR awards.
- Providing solutions. In some cases, firms respond to an agency solicitation and “solve” the problem, provide the needed data, or propose a solution that can then be adopted by the agency with no further “commercialization” revenues for the firm.37
- Developing technologies. Those firms that seek to develop commercial products may, in an initial phase, seek multiple awards to rapidly develop a technology. For the high-growth firms, this period is limited in time, before private investment becomes the principal source of funding.38
- Flexibility and speed. Some multiple-award winners have provided the highly efficient and flexible capabilities needed to solve pressing problems rapidly. For example, Foster-Miller, Inc., responded to needs of U.S. forces in Iraq by developing and the manufacturing add-on armor for Humvees that provide added protection from insurgent attacks.39
- There is evidence that companies winning multiple awards commercialize their projects at least as effectively as firms with fewer awards. The capacities built up through multiple awards can also enable them to meet agency needs in a timely fashion.
- Commercialization success. Aggregate data from the DoD commercialization database indicates that the companies winning the most awards generate more commercialization per award than those winning few awards.40 The 27 firms with more than 50 total Phase II awards account for 16.4 percent of all awards as reported through the CCR database, and for 30 percent of all reported commercialization.41 Among these, firms with 50–75 Phase II awards were the most successful.
- Meeting agency needs. Case studies show that some companies that have substantial numbers of awards have successfully commercialized products and have also met the needs of sponsoring agencies in other ways.42
- Graduation. Some multiple-award winners eventually “graduate” from the program, either by exceeding the 500-employee limit to qualify as a small firm or by being acquired by another firm. Successful firms such as Digital Systems Resources and Martek have provided valuable products, shown commercial success, and also received numerous awards.43
- Shifting revenues. Some firms with multiple awards show a declining percentage of revenues over time. Radiation Monitoring Devices, for example, testified that it currently generates only about 16 percent of company revenues from SBIR.44 In general, the larger the firm, the lower the percentage of revenues reported from SBIR awards.
- Company creation. Some multiple winners—like Optical Sciences, Creare, and Luna Innovations—frequently spin off companies. Creating new firms is a valuable contribution of the SBIR program especially with regard to the defense industrial base. Newly created firms create new opportunities for defense contractors, greater competition, and permit more rapid development of new defense solutions.
- While the DoD SBIR program supports woman- and minority-owned businesses, the steady decline in the share of Phase I awards to minority-owned businesses (falling to below 10 percent in 2004 and 2005) is a matter for concern and further review. 45
- A caveat on measurement. A stated objective of the SBIR program is to expand opportunities for women and minorities in federal S&T. One way to measure program performance in this area is to review the share of awards being made to woman- and minority-owned firms. In doing so, we must keep in mind the overall percentage of the population is a less relevant benchmark than are the number of science, technology, and engineering graduates, the demographics of high-tech firm ownership, as well as other variables such as the greater difficulty faced by these groups in accessing capital from other sources.
- Awards to woman-owned firms continue to increase. Figure 2-3 shows two divergent trends. Phase I awards to woman-owned firms continue to increase, increasing even as a percentage of the rising number of overall awards. However, the share awarded to minority-owned firms declined quite substantially since the mid-1990s, and fell below 10 percent for the first time in 2004 and 2005.
- Decline in award shares for minority-owned firms. Data on Phase II awards suggest that the decline in Phase I award shares for minority-owned firms has been reflected in Phase II. It is an open question whether the increase in awards to minority-owned firms in 2005 is the start of a reversal of this trend (see Figure 2-4).
- No apparent bias for or against woman- and minority-owned firms in Phase II awards. These data also indicate that both woman- and minority-owned firms are converting Phase I awards into Phase II at a rate very close to that of all award winners. This suggests that these firms are being invited to compete for Phase II awards at about the same rate as other firms, and that as a result there is no discernable bias for or against woman- and minority-owned firms in the selection of Phase II awards.
- The SBIR program at DoD generates considerable new technical knowledge, and is helping to expand the nation’s science and technology base.
- A key element in award selection is the technical merit and innovative character of the proposal. This criterion is applied to ensure that projects that do receive awards have the potential to generate new knowledge.
- In general, the SBIR program at DoD is highly competitive; since 1992 about 15 percent of Phase I proposals have been funded. Approximately half of Phase I winners receive Phase II contracts.
- The DoD SBIR program contributes new scientific and technological knowledge in several forms.
- These include intellectual property rights secured by the invent ing firm. Intellectual property rights create publication and licensing opportunities. Publications in the scientific and technical literatures and presentations at professional and technical meetings help the knowledge created by intellectual property rights to spill over into the public domain.
- SBIR companies have generated numerous patents and publications, the traditional measures of knowledge activity. 34.4 percent of DoD projects surveyed by NRC generated at least one patent application, and just over 25 percent had received a patent.47 (There is typically a 2–4 year lag between patent application and patent award or declination for recent patents.)
- Patents. For many firms, patents are an essential means of establishing intellectual property rights. This is especially important for firms planning to sell or license their technology for use in larger weapons and logistics systems where the immediate purchaser of their innovation will be a far larger defense contractor.
- Trade secrets. Other successful DoD awardees, however, see patents as less important. Some firms prefer to use trade secrets, saving the costs of filing and defending the patent. Some firms also see little economic profit from a patent if target market is primarily the federal government, which under SBIR has the right to its use royalty free. Thus a focus on patent data alone will understate the intellectual property generated through the DoD SBIR program.
- SBIR supports the transfer of technology from the university into the marketplace.
- Responses to the NRC Phase II and Firm Surveys suggest that SBIR awards are supporting the transfer of knowledge, firm creation, and partnerships between universities and the private sector:
- In more than 66 percent of responding companies (at all agencies), at least one founder was previously an academic50;
- About 36 percent of founders (at all agencies) were most recently employed as academics before the creation of their company.51
- At DoD, 13.6 percent of reporting projects had university faculty as subcontractors or consultants on the project, at 12.5 percent a university was itself a subcontractor, at 9.2 percent university facilities were used, and at 11.4 percent graduate students worked on the project.52
- There is anecdotal evidence concerning beneficial “indirect path” effects of SBIR.
- These indirect effects refer to the existence of projects that provide investigators and research staff with knowledge that may later become relevant in a different context often in another project or even another company. While these effects are not easily measurable, comments made during interviews and case studies suggest they exist.53
- DoD has taken steps to improve program performance, with some significant successes. 54 However, implementation of these best practices across the services and agencies is uneven.
- Improved information flows.
- DoD has a very extensive SBIR/STTR Web site, with exceptional support for potential SBIR applicants.
- The Pre-Release Program provides detailed guidance on specific topics.
- The Help Desk Program removes a significant burden from military staff and provides better service using professional civilian contractors.
- Efforts to address funding gaps and timeline issues.
- DoD and its components have made a number of efforts to address the gaps between topic conceptualization and Phase I funding, Phase I and Phase II, and the “TRL Gap” that often emerges after Phase II, before acquisitions programs can accept a technology as ready for acquisition.
- Acquisition alignment and Phase III. DoD is increasingly aware that success in Phase III requires strong support from program offices, and long-term alignment between SBIR activities and program needs. Efforts to improve topic alignment have been underway for several years
- Administrative funding. The Navy has taken the lead in providing extended administrative funding and support to its SBIR program. It may not be a coincidence that Navy’s Phase III results—as reported in the DD350 forms—are better than those of all other services and components combined.55
- Data collection and analytic capabilities. Data collection, reporting requirements, and analytic capability have all been improved. The CCR database represents the most comprehensive source of data on outcomes from SBIR projects at any of the agencies. The NRC study and other recent reviews represent a positive effort to connect data and analysis to practice.
- Phase III results remain uneven across the services and among components within the services.
- This suggests that other components and other services could improve the performance of their programs.
- Interviews suggest that some elements of the department have not fully integrated SBIR within their own program missions and have not provided SBIR with the resources and management attention needed to maximize its effectiveness.57
- The 2005 NRC Symposium on SBIR Phase III contributed to the awareness of the SBIR program’s potential, the challenge that promising products face in the Phase III transition, and the need for additional efforts to “team” across agencies, with SBIR program managers, Program Executive Officers, and prime contractors.58
- Prime contractors are taking a positive approach towards the SBIR program.
- Increased interest in SBIR. As the program has grown in size and performance, it has garnered greater attention from the DoD upper management and, importantly, the prime contractors.
- Interest has been followed by action. This increased focus on the prospective contributions of the SBIR program by the prime contractors appears to represent a significant positive endorsement of the contributions of the program.
- Steps taken by the prime contractors to integrate SBIR within their strategic roadmaps reveal that they see the program contributing to technological innovations that further the Defense mission.60
- At the Academies’ Phase III conference, representatives of prime contractors stated that there was already a substantial amount of prime involvement with the SBIR program. Moreover, several of the primes affirmed that they had made significant efforts to increase their levels of involvement.
- For example, Boeing had recently decided to increase its emphasis on SBIR.
- Similarly, at Raytheon, some divisions (e.g., Integrated Defense Systems) had formal working arrangements with SBIR for several years.
NRC STUDY RECOMMENDATIONS
As noted in the Findings section above, the Department of Defense has an effective SBIR program. The recommended improvements listed below should enable the DoD SBIR managers to address the four mandated congressional goals in a more efficient and effective manner.
- Improve the Phase III transition. DoD should continue to expand its work on improving the Phase III transition (the transition from SBIR-funded Phase I and Phase II research to commercialization—especially testing and evaluation funded by other DoD sources). It is important to recognize that the transition of new technologies is a complex process requiring teaming across areas of responsibility, additional resources, and often coping with some element of additional risk.61 Areas for possible action include:
- Aligning incentives.
- For the SBIR program to achieve its full potential, better incentives are required. Expansion of positive incentives for program officers to utilize the SBIR program for their own research needs, beyond the current requirements for their involvement in topic development.
- Management needs to improve incentives so that acquisitions officers perceive reduced risks and enhanced benefits from participating in the program.
- Increasing resources. SBIR managers need greater resources to “match” program funds to encourage uptake. In addition to increased Phase III SBIR funding, linkages with other programs (e.g., ManTech) might be enhanced to facilitate the Phase III transition.
- Developing an evaluation culture. Agency and service managers should have effective data collection and analysis as performance metrics.
- Involving acquisition officers. Active participation by acquisition officers is key to successful Phase III transitions.
- Acquisition officers control the funding, and their involvement is important for successful commercialization of SBIR technologies. A cultural shift in program participation and use seems to have occurred at Navy once Program Executive Offices (PEO) became active champions of SBIR involvement in acquisitions.
- Senior management support and encouragement, better information flows, improved PEO education about SBIR, and additional incentives for PEOs to use SBIR are all elements of an effective overall program.
- Integrating with roadmaps. The long technology development and acquisition cycle for major weapons and logistics systems means that effective Phase III transition requires early integration of SBIR topics and firms into the planning process.62
- Linking SBIR with other programs. Linkages with other programs (e.g., ManTech) might be enhanced to facilitate the Phase III transition.
- Improving outreach and matchmaking. There are significant barriers to the flow of information among SBIR firms, prime contractors, and acquisitions offices. Effective transition requires that these barriers be overcome, most likely through implementation of a range of activities, including improved electronic communications methods and matchmaking services like the Navy Opportunity Forum. In particular, efforts should be made, as appropriate, to align the SBIR program with the needs of the prime contractors responsible for the development of major systems.
- Connecting with the primes. The growing interest among prime contractors of the SBIR program’s outputs and opportunities for partnering with SBIR companies should be encouraged. Consideration should be given to performance incentives to further encourage development of SBIR supported technologies.
- Assessing and expanding commercialization programs. Commercialization programs that provide training, counseling, and networking opportunities should be assessed and, as appropriate, expanded.
- DoD should take immediate steps to enhance the perception of the SBIR program’s potential and accomplishments, promoting SBIR as an opportunity.
- A key element in the program’s operation is the attitude taken towards the program by the different levels of management in the Defense research and development community.
- Where SBIR is seen as an unwarranted intrusion on program management, a “tax” on R&D resources, it is less likely to be effectively aligned with service needs and less likely to have the resources to develop and ultimately insert the results of successful Phase II technologies in weapons and logistics systems and other programs.
- When the program is seen as an effective tool to engage the ingenuity of small companies in support of the Defense mission, with shorter lead times and more flexibility, it is much more likely to have its results adopted and incorporated.
- In short, there is an element of circularity in developing measures to enhance program effectiveness and management’s guidance and rewards for those managers who use the program effectively. Providing the resources and incentives for managers to see opportunity rather than obligation may well enhance program effectiveness.
- DoD should substantially strengthen and expand its evaluation efforts in order to further develop a program culture that is driven by outcomes, data, and internal and external evaluations.
- Efforts to identify outcomes should be improved, and evaluations63 should be connected much more directly to program management. More attention should be devoted to the role and contributions of the Contracting Officer’s Technical Representative (COTR). It is important that DoD create the capability to use outcomes data to help assess best practice.
- New mechanisms need to be developed that allow for the efficient design, implementation, and subsequent assessment of pilot programs.
- Efforts should be increased to make sure that appropriate metrics and benchmarks are adopted and implemented by all units, components, and Services.
- DoD should encourage and support the development of a results-oriented SBIR program with a focused evaluation culture.
- Effective oversight requires additional staff and funding. Effective management of a data-driven SBIR program requires the regular collection of higher-quality data and systematic assessment. Currently, sufficient resources are not available for these functions. Additional funding should be provided for program management and assessment.
- This funding should also be used to provide management oversight, including site visits, improved data collection and analysis, regular reporting, program review, and systematic third-party assessments.
- To help foster an active evaluation culture, DoD should consider:
- Preparing an expanded annual SBIR program report. DoD should prepare an annual SBIR program report, which gathers all relevant data about awards, outcomes, program activities, and management initiatives. In particular, the Department should publish detailed data annually about Phase III take-up at each service, and at each component within each service, as well as providing information about program initiatives.
- Commissioning regular assessments. SBIR programs at both DoD and the individual DoD components should seek to enhance a data-driven management approach, with regular assessment supporting policy development and program management.
- Instituting systematic and objective, outside review. The internal assessment program should be supported and supplemented by systematic, objective outside review and evaluation, as envisaged in the reauthorization legislation.
- Convening an advisory board. DoD should consider development of a formal advisory board, which would receive the annual program report and provide its own supplementary review of the report and management practices on an annual basis to senior DoD officials in charge of the SBIR program, or possibly to a subpanel of the Defense Sciences Board.
- DoD should consider greater internal review and adoption of best practices.Such an assessment would identify best practices within DoD and develop mechanisms for encouraging other components to implement these practices within their SBIR programs.
- One important example of best practice might be one focused on the Phase III transition at the Navy (see Box 2-3).
- Additional research should be undertaken to address, inter alia, three questions related to Navy practice.
- What unique factors make the Navy SUBS program successful?
- What role is played by initiatives at the service level in supporting the SUBS program?
- What elements of that success can best be transferred elsewhere in DoD as best practices to be followed? What changes will be needed to make those transfers successful?
- DoD should encourage and support pilot programs that evaluate new tools for improving the program’s overall performance.
- Innovation through pilot programs. Making changes initially through pilot programs allows DoD to alter selected areas on a provisional basis; a uniform approach is unlikely to work well for all components of a program that funds highly diverse projects with very different capital requirements and very different product development cycles.
- Some possible pilot projects include:
- Small Phase III awards. These could be a key to bridging the financing ‘Valley of Death’ that many firms face in converting research to innovation to products.64 NASA for example sometimes provides a small Phase III award—perhaps enough money to fly a demonstration payload—for a technology not ready for a full Phase III. These might be combined with milestones or gateways to additional rounds of Phase II funding.
- Unbundling larger contracts. Organizing larger contracts into smaller components would tend to open more Phase III opportunities for SBIR firms.65
- Redefining testing and evaluation within SBIR. DoD could pilot adoption of a wider view of RDT&E, so that SBIR projects could qualify for limited testing and evaluation funding. That in turn would help fund improvements in readiness levels.
- “Spring loading” Phase III, by putting in place Phase II milestones that could help to trigger initial Phase III funding. This could possibly occur in the context of larger, staged, Phase II awards in which additional stages fund more Demonstration and T&E, where non-SBIR funds or resources are leveraged.
- Evaluating pilots. DoD should develop a formal mechanism for designing, implementing, and evaluating pilot programs. Pilot programs allow agencies to investigate program improvements at lower risk and potentially lower cost. Effective pilot programs require rigorous design and evaluation, clear metrics for success, and the necessary resources and internal support.
- A flexible approach is required. In some cases, pilot programs may require waivers from SBA for activities not otherwise permitted under the SBA guidelines.66 SBA should be encouraged to take a highly flexible view of all agency proposals for pilot programs.
- Provide additional management resources. To carry out the measures recommended above to improve program utilization, management, and evaluation, the program should be provided with additional funds for management and evaluation.
- Effective oversight relies on appropriate funding.67 A data-driven program requires high quality data and systematic assessment. As noted above, sufficient resources are not currently available for these functions.
- Increased funding is needed to provide effective oversight, including site visits, program review, systematic third-party assessments, and other necessary management activities.
- In considering how to provide additional funds for management and evaluation, there are three ways that this might be done:
- Additional funds might be allocated internally, within the existing budgets of the services and agencies, as the Navy has done.
- Funds might be drawn from the existing set-aside for the program to carry out these activities.
- The set-aside for the program, currently at 2.5 percent of external research budgets, might be marginally increased, with the goal of providing management resources necessary to maximize the program’s return to the nation.68
- DoD should take steps to increase the participation and success rates of woman- and minority-owned firms in the SBIR program.
- Encourage participation. Develop targeted outreach to improve the participation rates of woman- and minority-owned firms, and strategies to improve their success rates. These outreach efforts and other strategies should be based on causal factors determined by analysis of past proposals and feedback from the affected groups.69
- Encourage emerging talent. The number of women and, to a lesser extent, minorities graduating with advanced scientific and engineering degrees has been increasing significantly over the past decade, especially in the biomedical sciences. This means that many of the woman and minority scientists and engineers with the advanced degrees usually necessary to compete effectively in the SBIR program are relatively young and may not yet have arrived at the point in their careers where they own their own companies. However, they may well be ready to serve as principal investigators (PIs) and/or senior co-investigators (Co-Is) on SBIR projects. Over time, this talent pool could become a promising source of SBIR participants.
- Improve data collection and analysis. The Committee also strongly encourages the agencies to gather and report the data that would track woman and minority firms as well as principal investigators (PIs), and to ensure that SBIR is an effective road to opportunity.
All data in this section are drawn from the NRC Phase II Survey, unless otherwise stated. Commercialization refers here to the extent to which projects generate outcomes that have market value. Commercialization in the context of DoD also refers to the take-up of projects within DoD, often (but not always) in the context of Phase III funding from non-SBIR resources. This aspect of commercialization is taken up in Section 4.3.
See Appendix B for a detailed description of the survey, response rate, and related issues. For DoD, the response rate was 42 percent of the awards contacted. See also National Research Council, Assessment of the Small Business Innovation Research Program—Project Methodology, Washington, DC: The National Academies Press, 2004, accessed at <http://books
See Figure 4-1 (NRC Phase II Survey, Question 1 and Question 4).
See Figure 4-2 (NRC Phase II Survey, Question 4).
This type of “skew”—in which a majority of projects fail or are minimally successfully while a small proportion generates large revenues—is typical of early-stage finance and has been noted in previous Academy research. See National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, Charles W. Wessner, ed., Washington, DC: National Academy Press, 2000. See also Joshua Lerner, “Public Venture Capital: Rationales and Evaluation,” in National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities, Charles W. Wessner, ed., Washington, DC: National Academy Press, 1999.
NRC Phase II Survey, Question 4.
The total eventual return from these awards is estimated to be approximately 50 percent higher than the data captured at the time of the survey. (For an explanation of the methodology underlying this analysis, see Chapter 4.) This suggests that the actual sales and licensing revenues that will in the end be generated by projects funded during the study period, on average, are approximately $2.2 million, and about $5.6 million for each project that did report some sales or licensing revenues. For DoD as a whole, the SBIR Program Manger Michael Caccuitto, reports that the amount of commercialization generated from SBIR projects now leads the total amount spent on SBIR, with about a 4-year lag from the year of Phase II award. See National Research Council, SBIR and the Phase III Challenge of Commercialization, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2007. This corresponds with the findings of the 1992 GAO report’s assessment of commercialization, which found that not enough time had elapsed since the program’s inception for projects to mature. See U.S. General Accounting Office, Small Business Innovation Research Program Shows Success But Can Be Strengthened, RCED–92–32, Washington, DC: U.S. General Accounting Office, 1992.
Derived from NRC Phase II Survey, Question 23. See also Table App-A-37.
NRC Phase II Survey, Question 23. See also Table App-A-37. Similar to sales, the amount of venture funding is skewed with only eight projects reporting $5 million or more in venture funding.
See Table 4-12.
Interviews with SBIR program managers.
The agency’s Technical Point of Contacts (TPOCs) were surveyed by the NRC. See NRC Project Manager Survey in National Research Council, An Assessment of the SBIR Program at the National Aeronautics and Space Administration, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009.
A number of initiatives in this direction are described in Chapter 5: The Phase III Challenge.
See Section 6.2.
The Navy finds that 80 percent of its topics are acquisition linked. Comments by Dr. Holland and Mr. Caccuitto at the NRC Phase III Symposium, June 2005. See National Research Council, SBIR and the Phase III Challenge of Commercialization, op. cit.
Many Phase III contracts are not captured effectively by the DD350 reporting system, which undercounts the number and size of Phase III contracts awarded, to a varied degree at different DoD components. See Chapter 5. In addition, commercialization occurring intraindustry, at a subcontract level, or in the commercial marketplace (and then perhaps finding its way to DoD) is not measured by this data source. The DD350 has been replaced by the Federal Procurement Data System—Next Generation (FPDS-NG) as the system tracking, and thus characterizing, all prime contract actions. This system is now standard for the federal government, with a small number of exceptions.
$450 million is based on the DD 350: however, the 2005 DoD Annual SBIR Report to SBA lists by contract $565 million in known Phase III contracts and that listing is considered to be incomplete. It includes only contracts known to the SBIR program managers. It should be kept in mind that these contracts are for direct sales to DoD or direct further DoD R&D funding. Sales and further R&D from DoD primes are not included. The NRC Phase II Survey and the DoD Commercialization data indicate the DoD sales and funding are actually less than half of the total commercialization from DoD SBIR.
See, for example, the case of ArmorWorks, Inc., in Box 2-1.
Firms express this outcome as follows: Had it not been for SBIR, their business with DoD services or agencies would not have developed. Services likely would have stayed with their pre-existing sources of supply. Program managers are normally too busy administering multiple contracts to search out or respond attentively to new sources of technology. Their propensity is to hire a contractor to solve problems rather than seek out the most technologically innovative performer. The SBIR program requires that program managers become involved with small firms, to look at technical options, and to allow for increased competition in the selection of R&D performers.
See the related discussion in Section 220.127.116.11.
The scores were normalized scores by removing the outliers in the top and bottom 5 percent of scores. Statistical procedures often assume that the variables are normally distributed. A significant violation of the assumption of normality can seriously increase the chances of a Type I (overestimation) or Type II (underestimation) error. Nonnormality can occur in the presence of outliers (scores that are extreme relative to the rest of the sample). Removing the outliers can improve the normality of the distribution. See C. M. Judd and G. H. McClelland, Data Analysis: A Model-Comparison Approach. San Diego, CA: Harcourt Brace Jovanovich, 1989.
There are multiple limitations to current systems for tracking SBIR awards. The DD350 reporting system can be used to extract some SBIR data; however, the system was not designed to gather SBIR award data, and is used differently by the services and agencies. For example the Navy reports Phase II Enhancements as Phase III, while the other services report such awards as Phase II. The Phase I and Phase II data in the DD350 does not match the SBIR budget or DoD Annual Report to SBA. The Annual Report to SBA, prepared by the DoD SBIR program managers, accounts for the budgeted SBIR funding. The SBIR program managers also have less knowledge of the Phase III awards, since these are made with funds that are not under control of the SBIR program.
See Figure 4-10 (NRC Phase II Survey, Question 13).
See Table 4-16.
Michael Caccuitto and Carol van Wyk, “Enhancing the Impact of Small Business Innovation Research (SBIR) Program: The Commercialization Pilot Program,” Presentation, September 27, 2006.
See Section 3.2.2.
It is important to keep in mind the difficulties in tracking companies over time. Companies regularly change names, locations, even employer identification numbers, which makes tracking them across time within the DoD awards database difficult.
See the 1992 GAO report, U.S. General Accounting Office, Small Business Innovation Research Program Shows Success But Can Be Strengthened, RCED–92–32, op. cit. That report focused on firms that had received twenty or more awards and on a single program metric, that is, commercialization. Congressional legislation later fixed fifteen awards in a five-year period as the level where efforts to commercialize should be taken into account.
See Reid Cramer, “Patterns of Firm Participation in the Small Business Innovation Research Program in the Southwestern and Mountain States,” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, op. cit., p. 151. The author describes the incremental nature of technical advance, which sometimes necessitates several awards. See also John T. Scott, “An Assessment of the Small Business Innovation Research Program in New England: Fast Track Compared with Non-Fast Track,” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, op. cit., p. 109 for a discussion of Foster-Miller, Inc.
Ibid. The mirror image of this approach is the program manager who makes several awards for similar technologies among different companies. In fact, it is not uncommon to have multiple awardees on the same Phase I topic. For an example, see the Navy’s SBIR Web site selections page for their FY-06.1 awardees at <http://www
.navysbir.com/06_1selections .html>. This page not only shows several awardees for each topic, but if you click on the “Details” link, you can see the differences in companies’ approaches to the topics.
To secure additional awards, a small company has to submit its proposal for follow-on research through the regular review process. These awards are relatively small in amount—the normal Phase I and Phase II awards would total $850,000. As a point of comparison, the top three U.S. prime contractors in 2007 garnered over $20.5 billion in defense revenues. See Washington Technology, “Top 100 Federal Prime Contractors: 2004,” May 14, 2007.
For example, Foster-Miller, the most frequent SBIR award winner, provided armor for Humvees and aircraft and developed robots for use in Iraq to identify roadside improvised explosive devices. (This company has since been acquired, making it ineligible to participate in SBIR. See Financial Times, “Qinetiq set to make its first US acquisition,” September 8, 2004.) Their LAST® Armor has had sales in excess of $170 million.
There are cases where a small business successfully completes the requirements and objectives of a Phase II contract, meeting the needs of the customer (e.g., by delivering specialized software or hardware), without gaining additional commercialization revenues. For example, Aptima, Inc., designed an instructional system to improve boat-handling safety by teaching the use of strategies that mitigate shock during challenging wave conditions. A secondary goal was to demonstrate how an innovative learning environment could establish robust skill levels while compressing learning time. Phase I of the project developed a training module, and in Phase II, instructional material, including computer animation, videos, images, and interviews were developed. The concept and the supporting materials were adopted as part of the introductory courses for Special Operations helmsmen with the goal of reducing injuries and increasing mission effectiveness.
For a discussion of Martek as an example, see Maryann P. Feldman, “Role of the Department of Defense in Building Biotech Expertise,” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, op. cit., pp. 266–268. See also Reid Cramer, “Patterns of Firm Participation in the Small Business Innovation Research Program in the Southwestern and Mountain States,” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, op. cit., pp. 146–147, who discusses several firms that realized commercial success after several awards.
Foster-Miller’s LAST® Armor, which uses Velcro-backed tiles to protect transport vehicles, helicopters and fixed wing aircraft from enemy fire, was developed on two Phase I SBIRs and a DARPA Broad Agency Announcement. The technology has helped improve the safety of combat soldiers and fliers in Bosnia and Operation Desert Storm. Access at <http://www
See Table 3-5.
CCR table provided by the database contractor, BRTRC, December 18, 2006.
See, for example, the case studies of Creare and Foster-Miller. The latter responded to needs in Iraq by providing add-on ceramic armor for HMMWVs. As noted above, contract research can be valuable even in the absence of commercialization. Agency staff report that SBIR fills multiple needs, many of which do not show up in sales data. For example, efficient probes of a technology’s potential, conducted in a relatively short time frame within a limited budget may save substantial time and resources. A key question to ask with regard to this and other aspects of the program is “compared to what?”—that is, what realistically are the probably alternative modes of exploration open to program managers?
VIASAT, Inc., garnered 24 Phase II awards in growing from a three-person start-up in 1986 to over 500 employees in 2001 (thus ineligible for SBIR). It has grown to over $350 million in annual revenue, mostly in sales to DoD and DoD prime contractors. Products include some of the most critical communications systems in DoD.
Michael Squillante, Vice President, Radiation Monitoring Devices, private communication, June 2004.
The absolute number of DoD Phase I awards to minorities has grown by 12 percent since 1995, while the overall number of DoD Phase I awards grew by 57 percent.
Award figures were provided by DoD.
See Table 4-18.
Without detailed identifying data on these patents and publications, it was not possible to apply bibliometric and patent analysis techniques to assess their relative importance.
NRC Phase II Survey, Question 18.
NRC Firm Survey, Question 2.
See NRC Firm Survey, Question 3.
See Table 4-19. (NRC Phase II Survey, Question 32).
For a discussion of the indirect path for awards made under the Advanced Technology Program, see Rosalie Ruegg, “Taking a Step Back: An Early Results Overview of Fifty ATP Awards,” in National Research Council, The Advanced Technology Program: Assessing Outcomes, Charles W. Wessner, ed., Washington, DC: National Academy Press, 2001.
See Box 2-3 discussing the Navy’s approach.
See also a recent RAND Report requested by DoD: Bruce Held, Thomas Edison, Shari Lawrence Pfleeger, Philip S. Anton, and John Clancy, Evaluation Recommendations for Improvement of the Department of Defense Small Business Innovation Research (SBIR) Program, Santa Monica, CA: RAND Corporation, 2006.
Some DoD program managers see the SBIR program as a “tax,” that is an unwarranted allocation of funds to small business, one that hinders effective R&D program management and one that is time consuming to manage and without funds to cover the cost of management. For a more positive view of the program, see the discussion of the Navy program in the Findings section. The recommendations address program perception and incentives.
Following the National Academies meeting on the SBIR commercialization challenge, the Senate Small Business & Entrepreneurship Committee proposed legislation that established a Commercialization Pilot Program. See Section 252 of the 2006 National Defense Authorization Act. The bill was passed in bipartisan spirit by the Senate Committee on Small Business & Entrepreneurship (SBE) under the leadership of the Committee Chair, Olympia Snowe (R-ME) and Ranking Member, John Kerry (D-MA). Further reflecting the growing appreciation of the program’s role and the increased focus on Phase III, Dr. Finley has described the SBIR program as a means of accelerating innovation and putting better equipment into the hands of the war-fighter. See remarks by Deputy Under Secretary of Defense for Acquisition and Technology Finley at the Small Business Technology Coalition Conference on September 27, 2006.
Raytheon, for example, estimates that the value of technology leveraged through SBIR jumped from $3.8 million in 2004 to $11.6 million in 2005, and looks set to grow as rapidly in 2006. Raytheon is involved with 36 Phase I projects, 17 Phase II projects, 4 Phase III projects (with three more in the works), and has been a subcontractor on other projects. See Lani Loell, SBIR Program Manager Raytheon Integrated Defense Systems, Presentation to SBTC SBIR in Transition Conference, September 27, 2006, Washington, DC.
See the discussion of these problems and potential remedies in the section on Phase III Transition in this volume. See also National Research Council, SBIR and the Phase III Challenge of Commercialization, op. cit.
In the case of the semiconductor industry, the industry perceived early on that problems of coordination could arise with a complex technology, multiple participants, and many ways of proceeding. This realization led to cooperative efforts led by SEMATECH to develop a technology roadmap setting out the relationships among science, technology, and applications as a point of reference for the researchers, technologists, project managers, suppliers, and users involved in and affected by the consortium’s work. As a general approach, roadmaps can advance similar coordination functions in other industry partnerships and in this way contribute to more efficient and more cooperative research. See National Research Council, Government-Industry Partnerships for the Development of New Technologies: Summary Report, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2003.
For example, each SBIR award has a DoD technical monitor who serves as the contracting officer’s technical representative (COTR). This individual monitors the contractor’s performance during Phase I and recommends or issues the invitation for Phase II, monitors performance of Phase II, and receives the contractor’s Phase I and II reports. Despite the importance of the TPOC’s role, there is no systematic attempt to use the technical monitor to evaluate the quality of the SBIR efforts and to facilitate Phase III. Since these individuals are not funded separately for SBIR, dissemination of SBIR outcomes and assistance in transition is a function of individual initiative and competing responsibilities. DoD should consider training, funding, and making better use of this valuable asset.
See Lewis Branscomb and Philip Auerswald, Between Invention and Innovation: An Analysis of Funding for Early-Stage Technology Development, NIST GCR 02-841, Gaithersburg, MD: National Institute of Standards and Technology, Prepared for the Economic Assessment Office, Advanced Technology Program, November 2002. See also National Research Council, SBIR and the Phase III Challenge of Commercialization, op. cit.
The recent focus for DoD research, development, and engineering centers has been to take many small support contracts, which were then done by small business and bundle them into a large solicitation, which can only be won by a firm which has significant resources. Bundling limits options for follow-on engineering support by SBIR firms. An alternative to unbundling is to provide incentives within large omnibus support contracts that make subcontract awards to SBIR firms.
In the past, SBA has shown commendable flexibility in allowing agencies to deviate from standard award sizes to accommodate the needs of the technology in question and to devise program innovations, such as the NSF Phase IIB, to provide incentives for commercialization.
According to recent OECD analysis, the International Benchmark for program evaluation of large SME and Entrepreneurship Programs is between 3 percent for small programs and 1 percent for large-scale programs. See “Evaluation of SME Policies and Programs: Draft OECD Handbook,” OECD Handbook, CFE/SME(2006)17.
Each of these options has its advantages and disadvantages. For the most part, the departments, institutes, and agencies responsible for the SBIR program have not proved willing or able to make additional management funds available. Without direction from Congress, they are unlikely to do so. With regard to drawing funds from the program for evaluation and management, current legislation does not permit this and would have to be modified; the Congress has clearly intended program funds to be for awards only. The third option, involving a modest increase to the program, would also require legislative action and would perhaps be more easily achievable in the event of an overall increase in the program. In any case, the Committee envisages an increase of the “set-aside” of perhaps 0.03 percent to 0.05 percent on the order of $35 million to $40 million per year, or roughly double what the Navy currently makes available to manage and augment its program. In the latter case (0.05 percent), this would bring the program “set-aside” to 2.55 percent, providing modest resources to assess and manage a program that is approaching an annual spend of some $2 billion. Whatever modality adopted by the Congress, without additional resources the Committee’s call for improved management, data collection, experimentation, and evaluation may prove moot. See also the National Research Council, SBIR and the Phase III Challenge of Commercialization, op. cit.
This recommendation should not be interpreted as lowering the bar for the acceptance of proposals from woman- and minority-owned companies. Rather it should be seen as assisting them to become able to meet published criteria for grants at rates similar to other companies on the basis of merit, and to ensure that there are no negative evaluation factors in the review process that are biased against these groups.
National Academies Press (US), Washington (DC)
National Research Council (US) Committee for Capitalizing on Science, Technology, and Innovation: An Assessment of the Small Business Innovation Research Program, Policy and Global Affairs; Wessner CW, editor. An Assessment of the SBIR Program at the Department of Defense. Washington (DC): National Academies Press (US); 2009. 2, Findings and Recommendations.