The first section of this appendix describes the methodology used to survey Phase II SBIR awards (or contracts.) The second part presents the results—first of the awards and then of the firm survey. (Appendix C presents the Phase I survey.)
ABOUT THE SURVEYS
Starting Date and Coverage
The survey of SBIR Phase II awards was administered in 2005, and included awards made through 2001. This allowed most of the Phase II awarded projects (nominally two years) to be completed, and provided some time for commercialization. The selection of the end date of 2001 was consistent with a GAO study, which in 1991, surveyed awards made through 1987.
A start date of 1992 was selected. The year 1992 for the earliest Phase II project was considered a realistic starting date for the coverage, allowing inclusion of the same (1992) projects as the DoD 1996 survey, and of the 1992, and 1993 projects surveyed in 1998 for SBA. This adds to the longitudinal capacities of the study. The 10 years of Phase II coverage spanned the period of increased funding set-asides and the impact of the 1992 reauthorization. This time frame allowed for extended periods of commercialization and for a robust spectrum of economic conditions. Establishing 1992 as the cut-off date for starting the survey helped to avoid the problem that older awards suffer from several problems, including meager early data collection as well as potentially irredeemable data loss; the fact that some firms and PIs are no longer in place; and fading memories.
While adding the annual awards numbers of the five agencies would seem to define the larger sample, the process was more complicated. Agency reports usually involve some estimating and anticipation of successful negotiation of selected proposals. Agencies rarely correct reports after the fact. Setting limitations on the number of projects to be surveyed from each firm required knowing how many awards each firm had received from all five agencies. Thus, the first step was to obtain all of the award databases from each agency and combine them into a single database. Defining the database was further complicated by variations in firm identification, location, phone numbers, and points of contact within individual agency databases. Ultimately, we determined that 4,085 firms had been awarded 11,214 Phase II awards (an average of 2.7 Phase II awards per firm) by the five agencies during the 1992–2001 time frame. Using the most recent awards, the firm information was updated to the most current contact information for each firm.
Sampling Approaches and Issues
The Phase II survey used an array of sampling techniques, to ensure adequate coverage of projects to address a wide range both of outcomes and potential explanatory variables, and also to address the problem of skew. That is, a relatively small percentage of funded projects typically account for a large percentage of commercial impact in the field of advanced, high-risk technologies.
- Random samples. After integrating the 11,214 awards into a single database, a random sample of approximately 20 percent was sampled. Then a random sample of 20 percent was ensured for each year; e.g., 20 percent of the 1992 awards, of the 1993 awards, etc. Verifying the total sample one year at a time allowed improved ability to adapt to changes in the program over time, as otherwise the increased number of awards made in recent years might dominate the sample.
- Random sample by agency. Surveyed awards were grouped by agency; additional respondents were randomly selected as required to ensure that at least 20 percent of each agency’s awards were included in the sample.
- Firm surveys. After the random selection, 100 percent of the Phase IIs that went to firms with only one or two awards were polled. These are the hardest firms to find for older awards. Address information is highly perishable, particularly for earlier award years. For firms that had more than two awards, 20 percent were selected, but no less than two.
- Top performers. The problem of skew was dealt with by ensuring that all Phase IIs known to meet a specific commercialization threshold (total of $10 million in the sum of sales plus additional investment) were surveyed (derived from the DoD commercialization database). Since 56 percent of all awards were in the random and firm samples described above, only 95 Phase IIs were added in this fashion.
- Coding. The project database tracks the survey sample, which corresponds with each response. For example, it is possible for a randomly sampled project from a firm that had only two awards to be a top performer. Thus, the response could be analyzed as a random sample for the program, a random sample for the awarding agency, a top performer, and as part of the sample of single or double winners. In addition, the database allows examination of the responses for the array of potential explanatory or demographic variables.
- Total number of surveys. The approach described above generated a sample of 6,410 projects, and 4,085 firm surveys—an average of 1.6 award surveys per firm. Each firm receiving at least one project survey also received a firm survey. Although this approach sampled more than 57 percent of the awards, multiple-award winners, on average, were asked to respond to surveys covering about 20 percent of their projects.
Administration of the Survey
The questionnaire drew extensively from the one used in the 1999 National Research Council assessment of SBIR at the Department of Defense, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative.1 That questionnaire in turn built upon the questionnaire for the 1991 GAO SBIR study. Twenty-four of the 29 questions on the earlier NRC study were incorporated. The researchers added 24 new questions to attempt to understand both commercial and noncommercial aspects, including knowledge base impacts, of SBIR, and to gain insight into impacts of program management. Potential questions were discussed with each agency, and their input was considered. In determining questions that should be in the survey, the research team also considered which issues and questions were best examined in the case studies and other research methodologies. Many of the resultant 33 Phase II Award survey questions and 15 Firm Survey questions had multiple parts.
The surveys were administered online, using a Web server. The formatting, encoding, and administration of the survey was subcontracted to BRTRC, Inc., of Fairfax, VA.
There are many advantages to online surveys (including cost, speed, and possibly response rates). Response rates become clear fairly quickly, and can rapidly indicate needed follow up for nonrespondents. Hyperlinks provide amplifying information, and built-in quality checks control the internal consistency of the responses. Finally, online surveys allow dynamic branching of question sets, with some respondents answering selected subsets of questions but not others, depending on prior responses.
Prior to the survey, we recognized two significant advantages of a paper survey over an online one. For every firm (and thus every award), the agencies had provided a mailing address. Thus, surveys could be addressed to the firm president or CEO at that address. That senior official could then forward the survey to the correct official within the firm for completion. For an online survey we needed to know the email address of the correct official. Also, each firm needed a password to protect its answers. We had an SBIR Point of Contact (POC) and email address and password for every firm, which had submitted for a DoD SBIR 1999 survey. However, we had only limited email addresses and no passwords for the remainder of the firms. For many, the email addresses that we did have were those of Principal Investigators rather than an official of the firm. The decision to use an online survey meant that the first step of survey distribution was an outreach effort to establish contact with the firms.
Outreach by Mail
This outreach phase began with the establishing a NAS registration Web site which allowed each firm to establish a POC, email address and password. Next, the Study Director, Dr. Charles Wessner, sent a letter to those firms for which email contacts were not available. Ultimately only 150 of the 2,0802 firms provided POC/email after receipt of this letter. Six hundred fifty of those letters were returned by the post office as invalid addresses. Each returned letter required thorough research by calling the agency provided phone number for the firm, then using the Central Contractor Registration database, Business.com (powered by Google) and Switchboard.com to try to find correct address information. When an apparent match was found, the firm was called to verify that it was in fact the firm, which had completed the SBIR. Two hundred thirty-seven of the 650 missing firms were so located. Another ten firms were located which had gone out of business and had no POC.
Two months after the first mailing, a second letter from the Study Director went to firms whose first letter had not been returned, but which had not yet registered a POC. This letter also went to 176 firms, which had a POC email, but no password, and to the 237 newly corrected addresses. The large number of letters (277) from this second mailing that were returned by the postal service, indicated that there were more bad addresses in the first mailing than indicated by its returned mail. (If the initial letter was inadvertently delivered, it may have been thrown away.) Of the 277 returned second letters, 58 firms were located using the search methodology described above. These firms were asked on the phone to go to the registration Web site to enter POC/email/password. A total of 93 firms provided POC/email/password on the registration site subsequent to the second mailing. Three additional firms were identified as out of business.
The final mailing, a week before survey, was sent to those firms that had not received either of the first two letters. It announced the study/survey and requested support of the 1,888 CEOs for which we had assumed good POC/email information from the DoD SBIR submission site. That letter asked the recipients to provide new contact information at the DoD submission site if the firm information had changed since their last submission. One hundred seventy-three of these letters were returned. We were able to find new addresses for 53 of these, and ask those firms to update their information. One hundred fifteen firms could not be found and five more were identified as out of business.
The three mailings had demonstrated that at least 1,100 (27 percent) of the mailing addresses were in error, 734 of which firms could not be found, and 18 were reported to be out of business.
Outreach by Email
We began Internet contact by emailing the 1,888 DoD Points of Contact (POCs) to verify their email and give them opportunity to identify a new POC. Four hundred ninety-four of those emails bounced. The next email went to 788 email addresses that we had received from agencies as PI emails. We asked that the PI have the correct company POC identify themselves at the NAS Update registration site. One hundred eighty-eight of these emails bounced. After more detailed search of the list used by NIH to send out their survey, we identified 83 additional PIs and sent them the PI email discussed above. Email to the POCs not on the DoD Submission site resulted in 110 more POC/email/Password being registered on the NAS registration site.
We began the survey at the end of February with an email to 100 POCs as a beta test and followed that with another email to 2,041 POCs (total of 2,141) a week later.
By August 5, 2005, five months after release of the survey, 1,239 firms had begun and 1,149 firms had completed at least 14 of 15 questions on the firm survey. Project surveys were begun on 1,916 Phase II awards. Of the 4,085 firms that received Phase II SBIR awards from DoD, NIH, NASA, NSF, or DoE from 1992 to 2001, an additional seven firms were identified as out of business (total of 25) and no email addresses could be found for 893. For an additional 500 firms, the best email addresses that were found were also undeliverable. These 1,418 firms could not be contacted, thus had no opportunity to complete the surveys. Of these firms, 585 had mailing addresses known to be bad. The 1,418 firms that could not be contacted were responsible for 1,885 of the individual awards in the sample.
Using the same methodology as the GAO had used in the 1992 report of their 1991 survey of SBIR, undeliverables and out-of-business firms were eliminated prior to determining the response rate. Although 4,085 firms were surveyed, 1,418 firms were eliminated as described. This left 2,667 firms, of which 1,239 responded, representing a 46 percent response rate by firms,3 which could respond. Similarly when the awards, which were won by firms in the undeliverable category, were eliminated (6,408 minus 1,885), this left 4,523 projects, of which 1,916 responded, representing a 42 percent response rate. Table App-B-1 displays by agency the number of Phase II awards in the sample, the number of those awards, which by having good email addresses had the opportunity to respond, and the number that responded.4 Percentages displayed are the percentage of awards with good addresses, the percentage of the sample that responded, and the responses as a percentage of awards with the opportunity to respond.
The NRC Methodology report had assumed a response rate of about 20 percent. Considering the length of the survey and its voluntary nature, the rate achieved was relatively high and reflects both the interest of the participants in the SBIR program and the extensive follow-up efforts. At the same time, the possibility of response biases that could significantly affect the survey results must be recognized. For example, it may be possible that some of the firms that could not be found have been unsuccessful and folded. It may also be possible that unsuccessful firms were less likely to respond to the survey.
NRC Phase II Survey Results for DoD
NOTE: SURVEY RESPONSES APPEAR IN BOLD, AND EXPLANATORY NOTES APPEAR IN TYPEWRITER FONT. FOR FURTHER DETAIL AND ANALYSIS OF SURVEY RESULTS, SEE APPENDIX A.
Project Information 920 respondents answered the first question. Since respondents are directed to skip certain questions based on prior answers, the number that responded varies by question. Also some respondents did not complete their surveys. 837 completed all applicable questions. For computation of averages, such as average sales, the denominator used was 920, the number of respondents who answered the first question. Where appropriate, the basis for calculations is provided in red after the question.
PHASE II CONTRACT/GRANT NUMBER:
Part I. Current status of the Project
1. What is the current status of the project funded by the referenced SBIR award? Select the one best answer. Percentages are based on the 920 respondents who answered this question.
2. Did the reasons for discontinuing this project include any of the following?
PLEASE SELECT YES OR NO FOR EACH REASON AND NOTE THE ONE PRIMARY REASON
300 projects were discontinued. The % below are the percent of the discontinued projects that responded with the indicated response.
The next question to be answered depends on the answer to question 1. If c, go to question 3. If b, skip to question 16.
Part II. Commercialization activities and planning
Questions 3–7 concern actual sales to date resulting from the technology developed during this project. Sales includes all sales of a product, process, or service, to federal or private sector customers resulting from the technology developed during this Phase II project. A sale also includes licensing, the sale of technology or rights, etc.
3. Has your company and/or licensee had any actual sales of products, processes, services or other sales incorporating the technology developed during this project? Select all that apply. This question was not answered for those projects still in Phase II (5%) or for projects, which were discontinued without sales or additional funding (23%). The denominator for the percentages below is all projects that answered the survey. Only 72% of all projects, which answered the survey, could respond to this question.
From the combination of responses 1b, 3a, and 3b, we can conclude that 31% had no sales and expect none, and that 18% had no sales but expect sales.
4. For your company and/or your licensee(s), when did the first sale occur, and what is the approximate amount of total sales resulting from the technology developed during this project? If multiple SBIR awards contributed to the ultimate commercial outcome, report only the share of total sales appropriate to this SBIR project. Enter the requested information for your company in the first column and, if applicable and if known, for your licensee(s) in the second column. Enter approximate dollars. If none, enter 0 (zero).
Display this box for Q 4 & 5 if project commercialization is known.
Your company reported sales information to DoD as a part of an SBIR proposal or to NAS as a result of an earlier NAS request. This information may be useful in answering the prior question or the next question. You reported as of (date): DoD sales ($ amount), Other federal sales ($ amount), Export sales ($ amount), Private-sector sales ($ amount), and other sales ($ amount).
5. To date, approximately what percent of total sales from the technology developed during this project have gone to the following customers? If none enter 0 (zero). Round percentages. Answers should add to about 100%.5 920 firms responded to this question as to what percent of their sales went to each agency or sector.
The following questions identify the product, process, or service resulting from the project supported by the referenced SBIR award, including its use in a fielded federal system or a federal acquisition program.
6. Is a federal system or acquisition program using the technology from this Phase II?
If yes, please provide the name of the federal system or acquisition program that is using the technology. 12% reported use in a federal system or acquisition program.
7. Did a commercial product result from this Phase II project? 27% reported a commercial product.
8. If you have had no sales to date resulting from the technology developed during this project, what year do you expect the first sales for your company or its licensee? Only firms that had no sales but answered that they expected sales got this question.
9. For your company and/or your licensee, what is the approximate amount of total sales expected between now and the end of 2006 resulting from the technology developed during this project? (If none, enter 0 [zero].) This question was seen by those who already had sales and those w/o sales who reported expecting sales; however, averages are computed for all who took the survey since all could have expected sales.
10. How did you (or do you expect to) commercialize your SBIR award?
11. Which of the following, if any, describes the type and status of marketing activities by your company and/or your licensee for this project? Select one for each marketing activity. This question answered by 620 firms, which completed Phase II and have not discontinued the project, w/o sales or additional funding.
Part III. Other outcomes
12. As a result of the technology developed during this project, which of the following describes your company’s activities with other companies and investors? (Select all that apply.) Percentage of the 620 who answered this question.
13. In your opinion, in the absence of this SBIR award, would your company have undertaken this project? Select one. Percentage of the 618 who answered this question.
14. If you had undertaken this project in the absence of SBIR, this project would have been Questions 14 and 15 were answered only by the 13% who responded that they definitely or probably would have undertaken this project in the absence of SBIR.
15. In the absence of SBIR funding, (Please provide your best estimate of the impact.)
a. The start of this project would have been delayed about an average of 11 months.
62% of the 81 firms expected the project would have been delayed. 49% (40 firms) expected the delay would be at least 12 months. 22% anticipated a delay of at least 24 months.
b. The expected duration/time to completion would have been
c. In achieving similar goals and milestones, the project would be
16. Employee information. Enter number of employees. You may enter fractions of full-time effort ( e.g., 1.2 employees). Please include both part-time and full-time employees, and consultants, in your calculation.
17. The Principal Investigator for this Phase II Award was a (check all that apply)
18. Please give the number of patents, copyrights, trademarks, and/or scientific publications for the technology developed as a result of this project. Enter numbers. If none, enter 0 (zero). Results are for 816 respondents to this question.
Part IV. Other SBIR funding
19. How many SBIR awards did your company receive prior to the Phase I that led to this Phase II?
a. Number of previous Phase I awards. Average of 22. 29% had no prior Phase I and another 42% had 5 or less prior Phase I.
b. Number of previous Phase II awards. Average of 8. 46% had no prior Phase II and another 36% had 5 or less prior Phase II.
20. How many SBIR awards has your company received that are related to the project/technology supported by this Phase II award?
a. Number of related Phase I awards. Average of two awards. 46% had no prior related Phase I and another 48% had 5 or less prior related Phase I.
b. Number of related Phase II awards. Average of one. 56.5% had no prior related Phase II and another 42.1% had 5 or less prior related Phase II.
Part V. Funding and other assistance
21. Prior to this SBIR Phase II award, did your company receive funds for research or development of the technology in this project from any of the following sources? Of 854 respondents.
Commercialization of the results of an SBIR project normally requires additional developmental funding. Questions 22 and 23 address additional funding. Additional Developmental Funds include non-SBIR funds from federal or private sector sources, or from your own company, used for further development and/or commercialization of the technology developed during this Phase II project.
22. Have you received or invested any additional developmental funding in this project?
23. To date, what has been the total additional developmental funding for the technology developed during this project? Any entries in the Reported column are based on information previously reported by your firm to DoD or NAS. They are provided to assist you in completing the Developmental Funding column. Previously reported information did not include investment by your company or personal investment. Please update this information to include breaking out Private investment and Other investment by subcategory. Enter dollars provided by each of the listed sources. If none, enter 0 (zero).</eei> The dollars shown are determined by dividing the total funding in that category by the 920 respondents who started the survey to determine an average funding. Only 462 of these respondents reported any additional funding.
24. Did this award identify matching funds or other types of cost sharing in the Phase II Proposal?6
25. Regarding sources of matching or co-investment funding that were proposed for Phase II, check all that apply. The percentages below are computed for those 161 projects, which reported matching funds.
26. Did you experience a gap between the end of Phase I and the start of Phase II?
The average gap reported by 584 respondents was 5 months. 3% of the respondents reported a gap of two or more years.
27. Project history. Please fill in for all dates that have occurred. This information is meaningless in aggregate. It has to be examined project by project in conjunction with the date of the Phase I end and the date of the Phase II award to calculate the gaps.
28. If you experienced funding gap between Phase I and Phase II for this award, (select all answers that apply)
29. Did you receive assistance in Phase I or Phase II proposal preparation for this award? Of 791 repondents.
Was this assistance useful?
30. In executing this award, was there any involvement by university faculty, graduate students, and/or university developed technologies? Of 837 respondents.
31. This question addresses any relationships between your firm’s efforts on this Phase II project and any University (ies) or College (s). The percentages are computed against the 837 who answered question 30, not just those who answered yes to question 30. Select all that apply.
In remarks enter the name of the University or College that is referred to in any blocks that are checked above. If more than one institution is referred to, briefly indicate the name and role of each.
32. Did commercialization of the results of your SBIR award require FDA approval?
In what stage of the approval process are you for commercializing this SBIR award?
NRC Firm Survey Results
NOTE: ALL RESULTS APPEAR IN BOLD. RESULTS ARE REPORTED FOR ALL 5 AGENCIES (DoD, NIH, NSF, DoE, AND NASA).
1,239 firms began the survey. 1,149 completed through question 14. 1,108 completed all questions.
If your firm is registered in the DoD SBIR/STTR Submission Web site, the information filled in below is based on your latest update as of September 2004 on that site. Since you may have entered this information many months ago, you may edit this information to make it correct. In conjunction with that information, the following additional information will help us understand how the SBIR program is contributing to the formation of new small businesses active in federal R&D and how they impact the economy. Questions A–G are autofilled from Firm database, when available.
A. Company Name: _______________________________________________
B. Street Address: ________________________________________________
C. City: _________________________________ State: ____ Zip: _________
D. Company Point of Contact: ______________________________________
E. Company Point of Contact Email: _________________________________
F. Company Point of Contact Phone: (___) ___ - ____ Ext: ______________
G. The year your company was founded: ___________
1. Was your company founded because of the SBIR Program?
2. Information on company founders. Please enter zeros or the correct number in each pair of blocks.
3. What was the most recent employment of the company founders prior to founding this company? Select all that apply. Total >100% since many companies had more than one founder.
4. How many SBIR and/or STTR awards has your firm received from the federal government?
a. Phase I: _________ Average number of Phase I reported was 14.
What year did you receive your first Phase I Award? _______
b. Phase II: ________ Average number of Phase II reported was 7
What year did you receive your first Phase II Award? _______
5. What percentage of your company’s growth would you attribute to the SBIR program after receiving its first SBIR award?
6. Number of company employees (including all affiliates):
7. What percentage of your total R&D effort (Man-hours of Scientists and Engineers) was devoted to SBIR activities during the most recent fiscal year?___%
8. What was your company’s total revenue for the last fiscal year?
9. What percentage of your company’s revenues during its last fiscal year is federal SBIR and/or STTR funding (Phase I and/or Phase II)? ____________
10. This question eliminated from the survey as redundant.
11. Which, if any, of the following has your company experienced as a result of the SBIR Program? Select all that apply.
12. How many patents have resulted, at least in part, from your company’s SBIR and/or STTR awards?
A total of over 3,350 patents were reported; an average of almost 3 per firm
The remaining questions address how market analysis and sales of the commercial results of SBIR are accomplished at your company.
13. This company normally first determines the potential commercial market for an SBIR product, process or service
14. Market research/analysis at this company is accomplished by: (Select all that apply.)
15. Sales of the product(s), process(es) or service(s) that result from commercialising an SBIR award at this company are accomplished by: Select all that apply.
National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, Charles W. Wessner, ed., Washington, DC: National Academy Press, 2000.
The letter was also erroneously sent to an additional 43 firms that had received only STTR awards.
Firm information and response percentages are not displayed in Table App-B-1, which displays by agency, since many firms received awards from multiple agencies.
The average firm size for awards, which responded, was 37 employees. Nonresponding awards came from firms that averaged 38 employees. Since responding Phase IIs were more generally more recent than nonresponding, and awards have gradually grown in size, the difference in average award size ($655,525 for responding and $649,715 for nonresponding) seems minor.
Please note: If a NASA SBIR award, the prime contractors line will state “Prime contractors for NASA.” The “Agency that awarded the Phase II” will only appear if it is not DoD or NASA. The name of the actual awarding agency will appear.
The words underlined appear only for DoD awards.
National Academies Press (US), Washington (DC)
National Research Council (US) Committee for Capitalizing on Science, Technology, and Innovation: An Assessment of the Small Business Innovation Research Program, Policy and Global Affairs; Wessner CW, editor. An Assessment of the SBIR Program at the Department of Defense. Washington (DC): National Academies Press (US); 2009. B, NRC Phase II and Firm Surveys.