Conflicts of Interest Publication Disclosures: Descriptive Study

Introduction

Multiple lines of research have documented the effects that author conflicts of interest (COI) can have on the biomedical research enterprise [1-5]. Author COI have been shown to increase the likelihood of positive findings [1] and influence study design [6-9], and they may be associated with diminished product safety [5,10]. To mitigate the effects of COI, journals, universities, professional societies, and academic medical centers have adopted disclosure requirements and policies designed to support transparency around COI [2,11]. Although such transparency endeavors are essential for continued efforts to understand the nature, prevalence, and effects of author COI, research in this area remains limited by the lack of a comprehensive centralized repository of author COI disclosure data for a few reasons [12-15]. First, the vast majority of COI disclosures are not readily available for analysis. Although PubMed has included COI disclosure statements as an available data category since 2017 [16], most journals do not participate at all or do not participate in all papers [17-19]. Related research in scientometrics also indicates that these data are frequently incomplete, incorrect, or both in commercial publication databases such as the Web of Science [13-15]. In the COIPonD dataset, for example, journals deposited disclosure statements for an average of 6.5% (SD 0.21%) of papers, and 86% (3242/3769) of journals deposited no disclosure statements for any paper sought for retrieval. Similarly, the US Centers for Medicare and Medicaid Services Open Payments Database can support research into COI, but not all COIs are reflected in this resource. Second, the data that do exist may be difficult to match with individual journal authors, and third, they include only US health care providers, greatly limiting the applicability of the dataset to nonproviders, researchers, and those working in other jurisdictions. In sum, in its current state, much of the published research on COI is grounded in datasets limited to specific subspecialties, disciplines, or publication venues [20,21] due, in part, to the difficulty in accessing disclosure statements.

To address these issues, the Conflicts of Interest Publication Disclosures (COIPonD) dataset provides a comprehensive database of author-reported COI collected from published research pertaining to the most commonly used drug products in the United States. Specifically, the dataset provides author-reported COI data on over 38,000 individual papers published between 1949 and 2022. The entries include specific disclosures and, where relevant, information about the absence of disclosures. The data come from over 3500 English-language journals published across medical subspecialties. COIPonD offers a unique, extensive, and otherwise nearly impossible to obtain a collection of disclosed COIs.

The dataset offers numerous advantages for continued and rigorous research on COI. Researchers and policy makers can use these data to improve and expand efforts to assess variability in disclosure requirements and thresholds across multiple publication domains and timelines. In addition, this dataset was developed intentionally to provide a comprehensive view of PubMed-indexed literature. Therefore, COIPonD includes, by design, not only rigorous studies published in high-quality journals but also non–peer-reviewed content and opinion pieces, which have been shown to be a common vector for COI-induced biases [22], and publications in predatory journals, which increasingly influence the state of science through consultation and citation [23]. The dataset spans a 70-year publication period and includes over 38,000 papers, both broad and long-term in scale for this search context. Furthermore, an additional advantage of this dataset is that it extends beyond the currently common focus on individual disciplines or subspecialties by focusing instead on a diverse range of drug products. It can also support future research into more effective policy solutions for addressing the risks of COI. For example, existing research suggests that different types of COI involve different risks of bias, but has not yet specified the nature of those risks or how they might be mitigated in part because of the absence of a dataset such as this one [2]. Larger datasets, like this one, are necessary for appropriately powered subsample analyses. Understanding the effects of COI and how to effectively manage COI across research contexts is critical for ensuring the integrity of the biomedical research enterprise. This dataset can facilitate the development of knowledge to reach these aims.

Methods

Study Identification

As previously mentioned, research on COI generally focuses on datasets of clinical trials or systematic reviews grounded in specific subspecialties or disciplinary journals [2]. For example, evaluations of COI’s effects on research have focused on psychiatry [24], oncology [25], and plastic surgery [26]. In contrast, the aim of our work was to capture information about COI related to the most frequently prescribed drug products across all subspecialties and publication types. In adopting this approach, we sought to mirror common practices for conducting a review of the relevant literature on a specific product. Specifically, for each identified product, we conducted a targeted search of PubMed for the most relevant papers. Our search focused on the most commonly used drug products as indexed by ClinCalc [27]. ClinCalc curates an annually updated list of the Top 300 most commonly used drugs. The list lags 2-3 years behind the current year. Drug use data are derived from ClinCalc’s analysis of the Agency for Health Care Research and Quality annual Medical Expenditure Panel Survey, which surveys the US residents on medical drug use (prescription and over-the-counter). To develop the COIPonD dataset, we used ClinCalc’s Top 300 drugs lists, each for 2017 and 2018 (the latest available at the time of query), to develop a list of target products. As expected, there was significant overlap in the most commonly used drugs between these 2 years, and so the final number of target products was 319.

For each of these 319 products, we used the National Library of Medicine’s (NLM’s) MeSH (Medical Subject Headings) browser to identify the preferred or supplementary concept, as most relevant. The MeSH-controlled vocabulary was selected for this project as it represents the primary ontology used by the NLM to support topic indexing and search retrieval. While alternative vocabularies, such as RxNorm, can be useful for improving interoperability between heterogeneous datasets, those alternative vocabularies are not engineered into the MEDLINE information architecture. MeSH terms group synonymous records in categories called “concepts” [28]. For most products, the generic name is the preferred concept, and associated trade names map automatically to that generic name in the MeSH ontology. In some cases (eg, combination neomycin, polymyxin B, and dexamethasone), the generic names map to a different preferred concept, such as the trade name. Occasionally, the preferred concept does not match the actual product in use, so we identified appropriate supplementary concepts (eg, the preferred concept for levothyroxine is thyroxine). Finally, in a few cases, no preferred or supplementary concepts were available. In these cases, we developed queries that would search for corresponding trade and generic names.

Our overall approach was to target, as closely as possible, highly relevant papers for each drug product. We identified the 150 most relevant papers per product, according to the PubMed relevance algorithm [29]. The intentional use of this algorithm to order search results was to mirror what researchers would discover in a search. In April 2021, we deployed an iterative query development strategy through which we identified the relevant preferred concept. If a preferred concept was available, we searched narrowly among MeSH terms and assessed if the available results were sufficiently large to secure a sample of 150 papers. If no preferred concepts were available, we attempted to identify a relevant supplementary concept and conducted the search again. In cases where the number of returned results for MeSH terms or supplementary concepts was insufficient (fewer than 150 papers returned), we then expanded our search to the title and abstract fields. If the search was still insufficient, we expanded our search again to all PubMed fields. Finally, in cases where the generic and trade names were not mapped to each other in MeSH, we conducted independent searches for each and aggregated the results. A complete list of primary and secondary search terms is available in Multimedia Appendix 1. For all searches, we collected up to 200 articles from PubMed, sorted by relevance. In cases where there were multiple searches per product (eg, where MeSH did not map the trade name to the generic name), searches were aggregated, and the number of times each article appeared was used to resort to relevance. All searches were executed using Rentrez, an R package designed to access the National Center for Biotechnology Information’s (NCBI’s) eUtilities application programming interface (API) [30]. The final dataset includes the most relevant papers per product, up to 150 per product. Some papers in the COIPonD database were identified for inclusion multiple times as part of different product searches. In such cases, the paper’s association with multiple products is recorded in the dataset. A total of 9/319 drug products returned fewer than 50 results even after this iterative search process, and these products were excluded from further data collection.

Data Extraction

Once target papers were identified, we used the eUtilities API to extract available PubMed metadata for each paper. Our disclosure statement data collection protocol proceeded in three steps that are: (1) in cases where a PubMed-indexed disclosure statement was available, it is included in the final dataset; (2) if no PubMed disclosure statement was available in the metadata, we developed an automated tool that attempted to locate and extract the relevant data from PubMed Central; and (3) if target disclosure statements were not available on either PubMed or PubMed Central, the paper was referred for manual collection.

While PubMed indexes all available disclosure statements under the XML tag <CoiStatement>, PubMed Central uses a variety of different possible tags. Therefore, for each paper indexed in PubMed Central, we searched all available XML tags for the following terms: conflict of interest, funding, conflicts of interest, disclosure statement, disclosure, author disclosure statement, and declaration of interest. All PubMed and PubMed Central data extraction was performed also using the Rentrez R package accessing the NCBI eUtilities API. In some cases, the multi-tag query approach led to multiple results being returned for a single paper. Any paper with multiple results was manually reviewed. In cases where the results were duplicate, they were reduced to a single entry. In cases where the results were unique, the results were combined into a single statement. This aggregation was most often required when a given paper’s XML schema separated each author’s disclosure into separate statements rather than combining them.

For each paper that did not have a disclosure statement available on PubMed or PubMed Central, we sought to locate the full text and extract the relevant data. Journal conventions for displaying disclosure statements vary widely, so we adopted the following heading prioritization schema to help ensure more uniform results. Specifically, we extracted all text under any heading of “Conflict of Interests,” “Conflicts of Interest,” “Competing Interests,” or “Duality of Interests.” If no such heading was available, the data extraction team then looked to assess the appropriateness of available information under “Disclosures,” “Funding,” or “Acknowledgements” headings. We extracted all data that specifically mentioned the terms “conflict of interests,” “conflicts of interest,” “competing interests,” or “duality of interests,” and any text that identified author-level financial relationships. We did not collect data from secondary headings if those data only identified study funding sources (as opposed to author-level relationships) or provided more general acknowledgments and statements of appreciation. In some cases, disclosure statements were hyperlinks to authors’ completed ICMJE (International Committee of Medical Journal Editors) forms. In these cases, the link was collected, but we did not seek to evaluate the data in the PDFs. Finally, if no disclosure information was available, the absence of available disclosures was recorded in the dataset.

Quality Assurance

Once data collection was complete, we conducted a quality assurance protocol to evaluate data accuracy (Table 1). This protocol involved first drawing a random sample of papers (N=381). The sample size was determined using the Cochran equation to determine representativeness at 95% confidence level and a 5% margin of error [31]. Where possible, the full text of each paper in the sample was located, and relevant data were re-extracted per the same data collection protocol described above. Re-extracted data were compared against previously collected data. The results of this protocol found that data collected for 349/381 (91.6%) of the papers were correct. Among the incorrect data 16/381 were classified as incorrect due to innocuous incomplete capture that would not adversely affect subsequent analyses (4.2%). Typically, this involved disclosure statements where additional assurances were provided and data entry only captured the reported COI. Another subset of 15/381 of the errors involved cases where data would have been partially or entirely missed (3.9%), and one was a case where a null result was incorrectly classified (no COI statement instead of non-English).

Ethical Considerations

This project does not involve human subjects or animal research, and no ethics review was required.

Results

The total number of papers included in the dataset is 38,705. The final dataset includes papers indexed to 319 unique drug products, and the number of papers located per drug product ranges from 32 to 150 with an average of 139 (SD 18.99). The paper sample size is greater than 100 for 296 of the target drug products. Recall that 9/319 drug products returned fewer than 50 results and were excluded from the study and further data collection. The collected papers were published in 3769 unique journals. Each journal is represented between 1 and 488 times, with an average of 10.27 (SD 23.26) papers per journal. The most commonly represented journals are PLoS One (n=488), Scientific Reports (n=454), BMJ Case Reports (n=351), Medicine (n=319), and the International Journal of Pharmaceutics (n=309). Paper publication years range from 1949 to 2022, with an average publication date of 2016. Notably, the dataset includes 10,134 papers with no locatable disclosure statement. Missing rates vary widely by publication year (14%-100% with an average of 78%). As would be expected, we see a sharp decrease in missing rates as journal COI policies started to proliferate in the 1990s. Figure 1 lists the details (due to the timing of the query, there is only 1 article from 2022 in the dataset).

Notably, the presence of a disclosure statement does not definitively indicate that a COI was disclosed. In many cases, authors disclosed no specific relationships, typically asserting that there were no disclosures or indicating affirmatively that there was nothing to disclose. Accordingly, COIPonD contains approximately 15,500 entries that indicate no conflict existed, including many variations of “none” or “nothing to disclose.” These values are approximate because they are based on an analysis of disclosure statements that repeat exactly 2 or more times in the dataset. It is probable that a number of unique disclosure statements documenting either the absence of disclosure or the absence of COI are available in the data. These likely include author-specific phraseology such as “RV has nothing to disclose.” In terms of the repeated negative disclosures, we document over 600 variations, including “none” (n=1024), “the authors declare no competing interests” (n=430), and “no competing financial interests exist” (n=112). In addition, approximately 200 records contain URLs, and a substantial proportion of these are links to PDF disclosure forms maintained by the publishing journal. This leaves approximately 12,500 entities reporting the presence of COI of some variety.

Most statements were not available on PubMed or PubMed Central. Of the 38,705 papers for which we sought to retrieve disclosure statements, only 3184 were available on PubMed (8%). Of those not available on PubMed, approximately 10% (3871/38,705) were retrieved from PubMed Central. The vast majority of disclosure statements had to be retrieved manually.

The current release of the COIPonD dataset can be found on its Texas Data Repository page [32]. The data are provided in two data tables that include (1) paper metadata, (2) disclosures, and (3) drug look-up (Textbox 1).

‎

Discussion

COIPonD indexes COI disclosure statements for the 38,705 most relevant papers related to 319 unique drug products that were the most commonly used in the United States in 2017 and 2018. The data come from journals across medical subspecialties. The dataset is a unique, extensive collection of disclosed COI that has otherwise been very difficult, if not impossible, to obtain previously. When combined with the associated article metadata and relevant product table, these data can serve as a foundation for new research on COI and the subsequent development of evidence-based COI management policies. COIPonD affords a range of benefits to investigators and can support a variety of study designs. Primarily, it offers researchers and policy makers an assortment of data for assessing potential bias and differential effects based on the type of COI. Results from these studies might be used to develop or implement evidence-based COI policy reform in biomedical publishing. Because the dataset focuses on author-level disclosures across a diverse range of drug products, it is possible to evaluate drug safety and risk in relation to disclosed COI and by type of COI [5]. COIPonD could be used to establish an evidence-based understanding of variability in disclosure requirements and authors’ understandings of those requirements.

In addition, the data can assist government and institutional policy makers to make better-informed decisions about implementing new disclosure policies based on evidence. Much of the current research on COI policy is limited to comparisons of (1) the presence or absence of COI and (2) associated study features, such as positive results or methodological rigor. In contrast, the organization of this dataset provides opportunities for more granular and rigorous examinations of outcomes associated with different COI types. Subsequent analysis with COIPonD data could help address persistent and misleading calls for yet more evidence that COI has adverse effects on the biomedical research enterprise. As such, it can facilitate more robust COI research and reform. Associated metadata about disclosure statement provenance may also be helpful in evaluating the transparency of certain publications, publishers, and government transparency initiatives. These features of the dataset are relevant to researchers interested in developing an effective public management system for COI disclosure. For example, researchers might use it to conduct more comprehensive audits of studies comparing disclosed COI to data in the Open Payments Database. In addition, research using these data may contribute to the development of more robust transparency and reporting frameworks and data collection and aggregation systems. For example, while research has suggested the US NLM and the ORCID (Open Researcher and Contributor ID) researcher registry are particularly suited to a public registry, this dataset provides a starting point for those interested in building the necessary infrastructure and systems to support new institutional practices by making disclosures more accessible and streamlined in PubMed and PubMed Central [12,17].

These data were collected to pursue specific research questions and thus are not suitable for use in all study designs. For example, research on industry influence on the biomedical enterprise may wish to consider the effects of study sponsorship in addition to the disclosed COI. This dataset does not include information specifically related to study funding or sponsorship, but may contribute in combination with other data sources. The data are also limited to the articles related to the most commonly used drug products in the United States at a specific time (2017-2018). Data on new, less commonly used products and medical devices are not available. In particular, less frequently prescribed but more expensive biological products are not included in this dataset, even though their economic effects may be significant and are an important direction for future work. Although these are important components of broader research efforts related to industry influence on the biomedical research enterprise, they fall beyond the scope of the research effort that produced this dataset. The papers included in this dataset are limited to those available through a PubMed search at the time the query was initiated (April 2021). As millions of new papers are published each year and considering ongoing efforts of drug repurposing, we expect the COI picture to change with time, even for the same drug products.

Finally, the results of this research point to the need for more robust data management approaches for collecting information about COI disclosures. Although PubMed has included COI disclosure statements in its system since 2016, participation is optional and therefore low. PubMed Central offers some additional access to disclosure statements but is reliant on individual publisher data structures, leading to inconsistent field names for available disclosures. The fee-based Web of Science offers a further resource but has been noted for similar information gaps and low data quality with respect to COI [13-15]. These serious gaps in data availability led to substantial human labor hour requirements in order to complete the COIPonD dataset. It would be a significant benefit to the medical, bioethics, and scientometric research communities if repositories could explore ways to incentivize or even compel more complete participation from major publishers. For example, participation in citation metrics might be made contingent on the full reporting of COI data. Researchers and policy makers alike would benefit from ready access to COI disclosures as well as the ability to filter results based on the presence or absence of these relationships. However, even with greater participation in COI reporting systems, additional computational or labor investments would be required for researchers to make the most productive use of available data. We would, therefore, also endorse previous recommendations [18,33] to require COI disclosure reporting in structured formats that cleanly identify and link funders, recipients, and mechanisms of disbursement.