Scaling of an antibody validation procedure enables quantification of antibody performance in major research applications

Antibodies are critical reagents to detect and characterize proteins. It is commonly understood that many commercial antibodies do not recognize their intended targets, but information on the scope of the problem remains largely anecdotal, and as such, feasibility of the goal of at least one potent and specific antibody targeting each protein in a proteome cannot be assessed. Focusing on antibodies for human proteins, we have scaled a standardized characterization approach using parental and knockout cell lines (Laflamme et al., 2019) to assess the performance of 614 commercial antibodies for 65 neuroscience-related proteins. Side-by-side comparisons of all antibodies against each target, obtained from multiple commercial partners, demonstrates that: i) more than 50% of all antibodies failed in one or more tests, ii) yet, ~50–75% of the protein set was covered by at least one high-performing antibody, depending on application, suggesting that coverage of human proteins by commercial antibodies is significant; and iii) recombinant antibodies performed better than monoclonal or polyclonal antibodies. The hundreds of underperforming antibodies identified in this study were found to have been used in a large number of published articles, which should raise alarm. Encouragingly, more than half of the underperforming commercial antibodies were reassessed by the manufacturers, and many had alterations to their recommended usage or were removed from the market. This first such study helps demonstrate the scale of the antibody specificity problem but also suggests an efficient strategy toward achieving coverage of the human proteome; mine the existing commercial antibody repertoire, and use the data to focus new renewable antibody generation efforts.


Introduction
Antibodies are critical reagents used in a range of applications, enabling the identification, quantification, and localization of proteins studied in biomedical and clinical research. The research enterprise spends significantly on the ~1.6M commercially available antibodies targeting ~96% of human proteins [1]. Unfortunately, a significant percentage of these antibodies do not recognize the intended protein or recognize the protein but also recognize non-intended targets, with estimates that $0.375 to $1.75 billion is wasted yearly on non-specific antibodies [2][3][4].
Perhaps worse, the use of poor-quality antibodies is a major factor in the scientific reproducibility crisis [3][4][5]. With tens to hundreds of antibodies available for any given protein target, it is difficult for antibody users to select the best performing antibody [6], and a growing number of cases reveal that use of previously published antibodies is not a reliable method to assess performance [7][8][9][10][11][12]. Academic and industry scientists aspire to have at least one, and ideally more, potent and selective and renewable antibody for each human protein for each of the most common applications [13]. Unfortunately, there is no agreed-upon mechanism to determine, validate or compare antibody performance and there are multiple strategies for antibody validation [14], with unequal scientific value. Most information on how commercial antibodies perform is anecdotal. It is thus difficult to assess progress toward the objective of well-validated antibodies for each human protein, or to design a strategy to accomplish this aim.
We sought to address this issue by developing optimized protocols to assess antibody specificity in the three most common uses of antibodies in biomedical research laboratories; Western blot (WB), immunoprecipitation (IP), and immunofluorescence (IF). We used these protocols to test antibodies against a variety of neuroscience targets, chosen by funders, to predict requirements for the larger goal of coverage of an entire mammalian proteome. The optimal antibody testing methodology is largely settled; using an appropriately selected wild type cell and an isogenic CRISPR knockout (KO) version of the same cell as the basis for testing, yields rigorous and broadly applicable results (this study, as well as [7,15,16]). However, the cost of antibody characterization using engineered KO cells is higher than that of other methods, mainly because of the cost of custom edited cells. Commercial antibody suppliers support a large and diverse catalogue of products, with most antibody products generating <$5,000 in total sales, far less than the costs of KO-based validation, estimated at $25,000. While leading companies are increasingly assessing antibody performance, it is exceedingly difficult, and cost restrained, to properly characterize all their products. Even when available, high-performing antibodies may remain hidden within the millions of reagents of unknown quality. 4 To begin the process of large-scale antibody validation and to provide a large enough dataset to allow for more accurate estimates of the work and financing required to complete such a project, we began with the human proteome. We created a partnership of academics, funders, and commercial antibody manufacturers, including 10 companies representing approximately 27% of antibody manufacturing worldwide. For each protein target, we tested commercial antibodies from various manufacturers in parallel using standardized protocols, agreed upon by all parties, in WB, IP, and IF applications. All data are shared rapidly and openly on ZENODO, a preprint server.
We have tested 614 commercially available antibodies targeting 65 proteins, and found that approximately two thirds of this protein set was covered by at least one high-performing antibody, and half was covered by at least one high-performing renewable antibody, suggesting that coverage of human proteins by high-performing antibodies is significant. This sample is large enough to observe several trends in antibody performance across various parameters and estimate the scale of the antibody liability crisis. The primary antibody specifically recognizes the target protein, but also unrelated protein(s) (Supplemental Figure   1A).

Successful antibody for
immunoprecipitation Under the conditions used, a successful primary antibody immunocaptures the target protein to at least 10% of the starting material (Supplemental Figure 1B).

Successful antibody for
immunofluorescence A successful primary antibody immunolocalizes the target protein by generating a fluorescence signal in WT cells that is at least 1.5-fold higher than the signal in KO cells (Supplemental Figure 1C). Signal provided by such antibody staining can be easily distinguished from unspecific background and noise.
. CC-BY 4.0 International license available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made

Assembling KO cell lines and antibodies
Our initiative has thus far validated antibodies for 65 protein targets, which were chosen by disease charities, academia, and industry without consideration of antibody coverage. The list comprises 32 Alzheimer's disease (AD)-related proteins that were community-nominated through an NIH-funded project on dark AD genes (https://agora.adknowledgeportal.org/), 22 proteins nominated within the amyotrophic lateral sclerosis (ALS) Reproducible Antibody Platform project, 5 Parkinson's disease (PD)-linked proteins nominated by the Michael J. Fox Foundation, and 6 proteins nominated by industry ( Figure 1A). Within the 65 target proteins, 56 are predicted intracellular and 9 are predicted secreted. The description of each protein target is indicated in Table 1.
The proteins were searched to determine the Uniprot identifier, the predicted molecular mass, and whether the protein is secreted or intracellular ( Figure 1B). Our strategy was predicated on identifying a parental cell line that expressed sufficient levels of the target protein to be detected by an antibody with a binding affinity of 1-50 nM. To identify candidate lines, we searched the Cancer Dependency Map Portal (DepMap) using the "Expression 22Q1" database, which houses the RNA-level analysis of >1800 cancer cell lines [17] ( Figure 1B). After our initial experience with a few dozen targets comparing RNA expression and the ability to detect a clear signal, we selected 2.5 log2(TPM+1) as an RNA-level threshold to select a candidate cell line to create a KO.
Among the cell lines showing expression above this level, we prioritized a group of 8 common cell line backgrounds representing different cell/tissue types because their doubling time is short, and they are amenable to CRISPR-Cas9 technology ( Figure 1B). These 8 cell lines were used in 62 out of the 65 antibody characterization studies (Table 1).
After identifying candidate cell lines for each target, we either obtained KO lines from our industry consortium partners or generated them in-house ( Figure 1C). Antibodies were provided from antibody manufacturers, who were responsible for selecting antibodies to be tested from their collections ( Figure 1D). Most antibody manufacturers prioritized renewable antibodies. The highest priority was given to recombinant antibodies as they represent the ultimate renewable reagent [13] and have advantages in terms of adaptability, such as switching IgG subclass [18] or using molecular engineering to achieve higher affinity binding than B-cell generated antibodies [19].

7
All available antibodies from all companies were tested side-by-side in parental and KO lines. The protocols used were established by our previous work [7] and refined in collaboration with antibody manufacturers. On occasion, our protocols differed from those the companies used in their internal characterization. All antibodies were tested for all 3 applications (except that secreted proteins were not tested in IF), independent of the antibody manufacturers' recommendations. We received on average 9.5 antibodies per protein target contributed from an average of 5 different antibody manufacturers (Figure 1E, F, G). Companies often contributed more than one antibody per target (Table 1).

Antibody and cell line characterization
For WB, antibodies were tested on cell lysates for intracellular proteins or cell media for secreted proteins ( Figure 1E). For 55/65 of the target proteins, we identified one or more antibodies that successfully immunodetected their cognate protein, identifying well-performing antibodies and validating the efficacy of the KO lines. For the remaining 9 targets, we identified at least one specific, non-selective antibody that detects the cognate protein by WB, but also recognizes unrelated proteins, that is, non-specific bands not lost in the KO controls. All 614 antibodies were tested by immunoprecipitation on non-denaturing cell lysates for intracellular proteins or cell media for secreted proteins, using WB with a successful antibody from the previous step to evaluate the immunocapture ( Figure 1F). All antibodies against intracellular proteins were tested for IF using a strategy that imaged a mosaic of parental and KO cells in the same visual field to reduce imaging and analysis biases ( Figure 1G).
For each protein target, we consolidated all screening data into a report, which is made available without restriction on ZENODO, a data-sharing website operated by CERN. On ZENODO, all 65 reports are gathered under the Antibody Characterization through Open Science (YCharOS) community: https://ZENODO.org/communities/ycharos/ ( Figure 1). Prior to release, each antibody characterization report underwent technical peer review by a group of scientific advisors from academia and industry ( Figure 1).

Coverage of human proteins by renewable antibodies
The Antibody Registry (www.antibodyregistry.org, SCR_006397) indicates that there are ~1.6 million antibodies covering ~96% of human proteins [1], with 53% covered by at least five renewable antibodies (Figure 2A, Supplementary Data 1). Approximately 21% of human proteins are covered by only one or two renewable antibodies, and ~15% have no renewable antibodies 8 available ( Figure 2A). In our set of 65 proteins, and from the manufacturers represented, 49 were covered by at least 3 renewable antibodies, 15 by one or two renewable antibodies, and one was not covered by any renewables (Table 1).
We found a well-performing renewable antibody for 50 targets in WB ( Figure 2B, left bar graph), for 49 targets in IP ( Figure 2B, middle bar graph), and for 30 targets in IF ( Figure 2B, right bar graph). For some proteins lacking coverage by renewable antibodies or lacking successful renewable antibodies, well-performing polyclonal antibodies were identified ( Figure 2B). Some proteins were not covered by any successful antibodies depending on application; notably ~40% of our protein set lacked a successful antibody for IF ( Figure 2B, right bar graph).

Optimizing an antibody characterization strategy
While the parental versus KO method is the consensus superior method for antibody validation [7,15,16,20], not all antibodies on the market are characterized this way, largely due to cost and the range of alternative methods [14]. To assess if the cost of KO characterization is justified, we compared the performance of antibodies in our dataset to the performance predicted by the characterization methods used by the companies. In all, 578 of the 614 antibodies tested were recommended for WB by the manufacturers. Of these, 44% were successful, 35% were specific but non-selective, and 21% failed (Supplemental Figure 2, left bar graph). Most antibodies are not recommended for IP by the suppliers, perhaps because they are not tested. 143 of 614 . CC-BY 4.0 International license available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint this version posted August 17, 2023. ; https://doi.org/10.1101/2023.06.01.543292 doi: bioRxiv preprint 9 antibodies were recommended for IP, and 58% enriched their cognate target from cell extracts.
Interestingly, of the 471 remaining antibodies that had no recommendation for IP, 37% were able to enrich their cognate antigen (Supplemental Figure 2, middle bar graph). In this regard, the manufacturers are under recommending successful products. Of the 529 antibodies tested in IF, 293 were recommended for this application by the suppliers and 236 were not. Only 39% of the antibodies recommended for IF were successful (Supplemental Figure 2, right bar graph).
We next investigated if antibody validation strategies have equal scientific value. Broadly, antibodies are characterized using genetic approaches, which exploit KO or knockdown (KD) samples as controls, or using orthogonal approaches, which use known information about the target protein of interest as a correlate to validate performance. For WB, 61% of antibodies were recommended by manufacturers based on orthogonal approaches, 30% based on genetic approaches and 9% using other strategies. For IF, 83% of the antibodies were recommended based on orthogonal approaches, 7% using genetic approaches and 10% using other strategies antibodies that failed in our hand, one has already been withdrawn from the market by the manufacturer. Thus, while orthogonal strategies are somewhat suitable for WB, genetic strategies generate far more robust characterization data for IF.

Antibodies and reproducible science
The availability of renewable, well-characterized antibodies would be expected to increase the reproducibility of research. To assess the bibliometric impact of underperforming antibodies, we used the reagent search engine CiteAb (https://www.citeab.com/) to quantify how antibodies in our dataset have been used in the literature. We identified 2010 publications that employed one of the 180 antibodies we tested for WB. Of those, 69% used a well-performing antibody that specifically immunodetected its target protein by WB, while 31% used an antibody unsuccessful in our protocol ( Figure 4C). For IP, 105 publications employed 41 of our tested antibodies while 65% of these used a well-performing antibody but 35% employed an antibody unable to immunocapture its target protein ( Figure 4C). For IF, we found 548 publications that employed 80 . CC-BY 4.0 International license available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint this version posted August 17, 2023. ; https://doi.org/10.1101/2023.06.01.543292 doi: bioRxiv preprint of the antibodies we tested. Of these publications, 22% used an antibody unable to immunolocalize its target protein ( Figure 4C), with 88% containing no validation data ( Figure 4D).
If our results are representative, this suggests that 20-30% of figures in the literature are generated using antibodies that do not recognize their intended target, and that more effort in antibody characterization is highly justified. . CC-BY 4.0 International license available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made

Discussion
Here we present the analysis of a dataset of commercial antibodies as an assessment of the problem of antibody performance, and as a step toward a comprehensive and standardized ecosystem to validate commercial antibodies. We evaluated 614 antibodies against 65 human proteins side-by-side in WB, IP and IF. All raw data are openly available (https://ZENODO.org/communities/ycharos/), identifiable on the RRID portal and on participating antibody manufacturers' websites. Our studies provide an unbiased and scalable analytical framework for the representation and comparison of antibody performance, an estimate of the coverage of human proteins with renewable antibodies, an assessment of the scientific value of common antibody characterization methods, and they inform a strategy to identify renewable antibodies for all human proteins.
Our approach, developed in collaboration with manufacturers, and intended to be applied to entire proteomes, uses universal protocols for all tested antibodies in each application. Scientists use variants of such protocols, optimized for their protein of interest, which can have a major impact on antibody performance [21][22][23]. Nevertheless, the process robustly identifies antibodies that fail to recognize their intended target, which becomes evident when other antibodies tested in parallel perform well. At a minimum, removal of these poorly performing products from the market will have significant impact in that hundreds of published papers report the use of such antibodies.
Importantly, from a total of 409 antibodies that presented conflicting data between our characterization data and antibody supplier's recommendations, the participating companies have withdrawn 73 antibodies from the market and changed recommendations for 153 antibodies (Supplemental Figure 3). In turn, high-quality antibodies are being promoted. We expect to see additional changes and an overall improvement in the general quality of commercial reagents as more antibody characterization reports are generated.
The impacts of poorly performing antibodies are well documented [4,8,24,25]; our analyses provide insight into the magnitude of the problem. In our set of 65 proteins, we found that an average of ~12 papers per protein included use of an antibody that failed to recognize the intended protein target using our protocols. Scientists are not entirely to blame; dozens of antibodies can be used in a single study, often unrelated to the authors' protein of interest. Genetic validation of every antibody used in a study remains a difficult, if not impossible task. In addition, even with our optimized protocol, the cost of characterizing antibodies for a single protein is estimated at ~$25,000 USD. And if each investigator performs such an analysis, there will be . CC-BY 4.0 International license available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint this version posted August 17, 2023. ; 12 multiple overlapping validation of any given antibody. We estimate a cost of $50 million USD to characterize antibodies against all proteins in a proteome, considering parallelization and industrialization of the procedure. The costs mentioned exclude the expenses of antibodies and knockout cell lines. However, it should be noted that this estimated cost for validation is far below the predicted waste on bad antibodies, currently estimated at ~$1B/year [4]. Thus, independent antibody characterization with openly published data, funded by various global organizations, is an important, if not essential, initiative that is certain to save large amounts of money and increase the quality and reproducibility of the literature. This study demonstrates the feasibility of such an initiative.
Life scientists tend to focus on a small subset of human proteins, leading to an imbalance between a small percentage of well-studied proteins, and a higher percentage of poorly characterized proteins [26]. Our set of 65 funder-designated proteins is an unbiased sample, representative of the heterogeneity of knowledge of the human proteome; a search of the NIH protein database revealed that 15 proteins (23% of our protein sample) are well studied with more than 500 publications and 50 proteins (77% of our protein sample) have corresponding publications ranging from 37 to 498 (Table 1). Although we observed that there are more commercial antibodies available for the best-studied proteins (Table 1), an encouraging result of our work is that more than half of our protein targets are covered by well-performing, renewable antibodies for WB, IP and IF -including both well characterized and more poorly studied proteins. Within the antibodies we tested, we found a successful renewable antibody for WB for 77% of proteins (50/65), for IP for 75% of proteins (49/65) and for IF for 54% of proteins (30/56) examined. Extrapolation of our findings to the human proteome would suggest that it might be possible to identify well-performing renewable reagents for half the human proteome, including poorly characterized proteins, simply by mining commercial collections. Indeed, it is likely that the coverage is greater because our corporate partners only represent 27% of the antibody production worldwide.
The research market is heavily dominated by polyclonal antibodies, and their use contributes to reproducibility issues in biomedical research [2,5] and present important ethical concerns. From a scientific perspective, polyclonal antibodies suffer from batch-to-batch variation and are thus in conflict with the scientific community desire to use and provide only renewable reagents. From an ethical perspective, the generation of polyclonal antibodies requires large numbers of animals yearly [27]. While recombinant antibodies may rely on the use of animals for the initiation of an antibody generation program, animal-free in vitro molecular strategies are also used for production, and to generate new batches of these antibodies [19]. As of today, the uptake of . CC-BY 4.0 International license available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint this version posted August 17, 2023. ; https://doi.org/10.1101/2023.06.01.543292 doi: bioRxiv preprint 13 recombinant antibodies by the scientific community has not been satisfactory. For example, while leading antibody manufacturers are converting top-cited polyclonal antibodies into recombinant antibodies and removing underperforming antibodies from their catalogues, polyclonals remain the most purchased. This situation has also been acknowledged by the EU Reference Laboratory for Alternatives to Animal Testing and a lack of understanding in the use of recombinant methods has been suggested by authors of a recent correspondence to the editors of Nature Biotechnology [19]. One reason for this confusion could be the absence of large-scale performance data comparing the various antibody generation technologies. In our dataset, recombinant antibodies performed well in all applications tested, arguing there is no reason not to adopt the recombinant technology. Moreover, our study strongly supports the idea that future antibody generation programs should focus on recombinant technologies.
Our analyses also inform the characterization pipelines to use for newly generated antibodies.
Currently, it is common to use WB as the initial screen [28]. However, we find that success in IF is the best predictor of performance in WB and IP (Supplemental Figure 4). Given that it is difficult to imagine a process dependent on IF, we suggest that using KO (or knockdown in the case of an essential gene) strategies to screen antibodies for the intended application will provide the most effective approach to identify selective antibodies. Currently, one of the main barriers to large-scale production of high-quality antibodies is the lack of availability of KO lines derived from cells express detectable levels of each human protein. Creation of a broadly accessible biobank of bespoke KO cells for each human gene should be a priority for the community.
Our studies are rapidly shared via the open platform ZENODO, and selected studies were published on the F1000 publication platform (https://f1000research.com/ycharos). This data generation and dissemination is intended to benefit the global life science community, but its impact depends on the real-world uptake of the data. In addition, we recognize that antibodies are used in other protocols or in variations of our protocols that may yield important new or different outcomes. Posting of such information from users worldwide on open platforms will allow continued improvements to the data. Thus, we have partnered with the RRID Portal Community to improve our dissemination strategies. The Antibody Registry is a comprehensive repository of over 2.5 million commercial antibodies that have been assigned with RRIDs to ensure proper reagent identification [1]. Our data can be searched in the AntibodyRegistry.org and other portals that display this data such as the RRID.site portal and dkNet.org. The search term "ycharos" will return all the currently available antibodies that have been characterized and searching for the target or the catalogue number of the antibody in any of these portals will also bring back the . CC-BY 4.0 International license available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made Overall, this project provides the global life sciences community with a tremendous resource for the study of human proteins and will result in significant improvements in rigour and reproducibility in antibody-based assays and scientific discovery.
. CC-BY 4.0 International license available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made

Data analysis
Performance of each antibody was retrieved from the corresponding ZENODO report or publication (Table 1), for WB, IP and IF, and analyzed following the performance criteria described in Box 1. Antibody properties, application recommendations and antibody characterization strategies were taken from the manufacturers' datasheets. Throughout the manuscript, renewable antibodies refer to monoclonal antibodies from hybridomas and to recombinant antibodies (monoclonal and polyclonal recombinant antibodies) generated in vitro.
For Figure 2A, the analysis of the antibody coverage of human proteins was performed as previously described [1] and antibodies were divided into polyclonal and renewable categories.
To evaluate the number of citations corresponding to each tested antibody ( Figure 4C of cell a, b, c and d).
The number of articles corresponding to each human target protein was assessed by searching the NIH protein database (https://www.ncbi.nlm.nih.gov/protein/) on May 4, 2023.  . CC-BY 4.0 International license available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made

Competing interests
The authors declare no competing interests.
. CC-BY 4.0 International license available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint this version posted August 17, 2023.      (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made  . CC-BY 4.0 International license available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made  . CC-BY 4.0 International license available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint this version posted August 17, 2023. ; https://doi.org/10.1101/2023.06.01.543292 doi: bioRxiv preprint . CC-BY 4.0 International license available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made   Specific, non-selective . CC-BY 4.0 International license available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made   (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint this version posted August 17, 2023. ; https://doi.org/10.1101/2023.06.01.543292 doi: bioRxiv preprint