research methods papers

When you choose to publish with PLOS, your research makes an impact. Make your work accessible to all, without restrictions, and accelerate scientific discovery with options like preprints and published peer review that make your work more Open.

PLOS Biology
PLOS Climate
PLOS Complex Systems
PLOS Computational Biology
PLOS Digital Health
PLOS Genetics
PLOS Global Public Health
PLOS Medicine
PLOS Mental Health
PLOS Neglected Tropical Diseases
PLOS Pathogens
PLOS Sustainability and Transformation
PLOS Collections
How to Write Your Methods

Ensure understanding, reproducibility and replicability

What should you include in your methods section, and how much detail is appropriate?

Why Methods Matter

The methods section was once the most likely part of a paper to be unfairly abbreviated, overly summarized, or even relegated to hard-to-find sections of a publisher’s website. While some journals may responsibly include more detailed elements of methods in supplementary sections, the movement for increased reproducibility and rigor in science has reinstated the importance of the methods section. Methods are now viewed as a key element in establishing the credibility of the research being reported, alongside the open availability of data and results.

A clear methods section impacts editorial evaluation and readers’ understanding, and is also the backbone of transparency and replicability.

For example, the Reproducibility Project: Cancer Biology project set out in 2013 to replicate experiments from 50 high profile cancer papers, but revised their target to 18 papers once they understood how much methodological detail was not contained in the original papers.

What to include in your methods section

What you include in your methods sections depends on what field you are in and what experiments you are performing. However, the general principle in place at the majority of journals is summarized well by the guidelines at PLOS ONE : “The Materials and Methods section should provide enough detail to allow suitably skilled investigators to fully replicate your study. ” The emphases here are deliberate: the methods should enable readers to understand your paper, and replicate your study. However, there is no need to go into the level of detail that a lay-person would require—the focus is on the reader who is also trained in your field, with the suitable skills and knowledge to attempt a replication.

A constant principle of rigorous science

A methods section that enables other researchers to understand and replicate your results is a constant principle of rigorous, transparent, and Open Science. Aim to be thorough, even if a particular journal doesn’t require the same level of detail . Reproducibility is all of our responsibility. You cannot create any problems by exceeding a minimum standard of information. If a journal still has word-limits—either for the overall article or specific sections—and requires some methodological details to be in a supplemental section, that is OK as long as the extra details are searchable and findable .

Imagine replicating your own work, years in the future

As part of PLOS’ presentation on Reproducibility and Open Publishing (part of UCSF’s Reproducibility Series ) we recommend planning the level of detail in your methods section by imagining you are writing for your future self, replicating your own work. When you consider that you might be at a different institution, with different account logins, applications, resources, and access levels—you can help yourself imagine the level of specificity that you yourself would require to redo the exact experiment. Consider:

Which details would you need to be reminded of?
Which cell line, or antibody, or software, or reagent did you use, and does it have a Research Resource ID (RRID) that you can cite?
Which version of a questionnaire did you use in your survey?
Exactly which visual stimulus did you show participants, and is it publicly available?
What participants did you decide to exclude?
What process did you adjust, during your work?

Tip: Be sure to capture any changes to your protocols

You yourself would want to know about any adjustments, if you ever replicate the work, so you can surmise that anyone else would want to as well. Even if a necessary adjustment you made was not ideal, transparency is the key to ensuring this is not regarded as an issue in the future. It is far better to transparently convey any non-optimal methods, or methodological constraints, than to conceal them, which could result in reproducibility or ethical issues downstream.

Visual aids for methods help when reading the whole paper

Consider whether a visual representation of your methods could be appropriate or aid understanding your process. A visual reference readers can easily return to, like a flow-diagram, decision-tree, or checklist, can help readers to better understand the complete article, not just the methods section.

Ethical Considerations

In addition to describing what you did, it is just as important to assure readers that you also followed all relevant ethical guidelines when conducting your research. While ethical standards and reporting guidelines are often presented in a separate section of a paper, ensure that your methods and protocols actually follow these guidelines. Read more about ethics .

Existing standards, checklists, guidelines, partners

While the level of detail contained in a methods section should be guided by the universal principles of rigorous science outlined above, various disciplines, fields, and projects have worked hard to design and develop consistent standards, guidelines, and tools to help with reporting all types of experiment. Below, you’ll find some of the key initiatives. Ensure you read the submission guidelines for the specific journal you are submitting to, in order to discover any further journal- or field-specific policies to follow, or initiatives/tools to utilize.

Tip: Keep your paper moving forward by providing the proper paperwork up front

Be sure to check the journal guidelines and provide the necessary documents with your manuscript submission. Collecting the necessary documentation can greatly slow the first round of peer review, or cause delays when you submit your revision.

Randomized Controlled Trials – CONSORT The Consolidated Standards of Reporting Trials (CONSORT) project covers various initiatives intended to prevent the problems of inadequate reporting of randomized controlled trials. The primary initiative is an evidence-based minimum set of recommendations for reporting randomized trials known as the CONSORT Statement .

Systematic Reviews and Meta-Analyses – PRISMA The Preferred Reporting Items for Systematic Reviews and Meta-Analyses ( PRISMA ) is an evidence-based minimum set of items focusing on the reporting of reviews evaluating randomized trials and other types of research.

Research using Animals – ARRIVE The Animal Research: Reporting of In Vivo Experiments ( ARRIVE ) guidelines encourage maximizing the information reported in research using animals thereby minimizing unnecessary studies. (Original study and proposal , and updated guidelines , in PLOS Biology .)

Laboratory Protocols Protocols.io has developed a platform specifically for the sharing and updating of laboratory protocols , which are assigned their own DOI and can be linked from methods sections of papers to enhance reproducibility. Contextualize your protocol and improve discovery with an accompanying Lab Protocol article in PLOS ONE .

Consistent reporting of Materials, Design, and Analysis – the MDAR checklist A cross-publisher group of editors and experts have developed, tested, and rolled out a checklist to help establish and harmonize reporting standards in the Life Sciences . The checklist , which is available for use by authors to compile their methods, and editors/reviewers to check methods, establishes a minimum set of requirements in transparent reporting and is adaptable to any discipline within the Life Sciences, by covering a breadth of potentially relevant methodological items and considerations. If you are in the Life Sciences and writing up your methods section, try working through the MDAR checklist and see whether it helps you include all relevant details into your methods, and whether it reminded you of anything you might have missed otherwise.

Summary Writing tips

The main challenge you may find when writing your methods is keeping it readable AND covering all the details needed for reproducibility and replicability. While this is difficult, do not compromise on rigorous standards for credibility!

Keep in mind future replicability, alongside understanding and readability.
Follow checklists, and field- and journal-specific guidelines.
Consider a commitment to rigorous and transparent science a personal responsibility, and not just adhering to journal guidelines.
Establish whether there are persistent identifiers for any research resources you use that can be specifically cited in your methods section.
Deposit your laboratory protocols in Protocols.io, establishing a permanent link to them. You can update your protocols later if you improve on them, as can future scientists who follow your protocols.
Consider visual aids like flow-diagrams, lists, to help with reading other sections of the paper.
Be specific about all decisions made during the experiments that someone reproducing your work would need to know.

Don’t

Summarize or abbreviate methods without giving full details in a discoverable supplemental section.
Presume you will always be able to remember how you performed the experiments, or have access to private or institutional notebooks and resources.
Attempt to hide constraints or non-optimal decisions you had to make–transparency is the key to ensuring the credibility of your research.
How to Write a Great Title
How to Write an Abstract
How to Report Statistics
How to Write Discussions and Conclusions
How to Edit Your Work

The contents of the Peer Review Center are also available as a live, interactive training session, complete with slides, talking points, and activities. …

The contents of the Writing Center are also available as a live, interactive training session, complete with slides, talking points, and activities. …

There’s a lot to consider when deciding where to submit your work. Learn how to choose a journal that will help your study reach its audience, while reflecting your values as a researcher…

Locations and Hours
UCLA Library
Research Guides
Research Tips and Tools

Advanced Research Methods

Writing the research paper.

What Is Research?
Library Research
Writing a Research Proposal

Before Writing the Paper

Methods, thesis, and hypothesis, clarity, precision, and academic expression, format your paper, typical problems, a few suggestions, avoid plagiarism.

Presenting the Research Paper
Try to find a subject that really interests you.
While you explore the topic, narrow or broaden your target and focus on something that gives the most promising results.
Don't choose a huge subject if you have to write a 3 page long paper, and broaden your topic sufficiently if you have to submit at least 25 pages.
Consult your class instructor (and your classmates) about the topic.
Find primary and secondary sources in the library.
Read and critically analyse them.
Take notes.
Compile surveys, collect data, gather materials for quantitative analysis (if these are good methods to investigate the topic more deeply).
Come up with new ideas about the topic. Try to formulate your ideas in a few sentences.
Review your notes and other materials and enrich the outline.
Try to estimate how long the individual parts will be.
Do others understand what you want to say?
Do they accept it as new knowledge or relevant and important for a paper?
Do they agree that your thoughts will result in a successful paper?
Qualitative: gives answers on questions (how, why, when, who, what, etc.) by investigating an issue
Quantitative:requires data and the analysis of data as well
the essence, the point of the research paper in one or two sentences.
a statement that can be proved or disproved.
Be specific.
Avoid ambiguity.
Use predominantly the active voice, not the passive.
Deal with one issue in one paragraph.
Be accurate.
Double-check your data, references, citations and statements.

Academic Expression

Don't use familiar style or colloquial/slang expressions.
Write in full sentences.
Check the meaning of the words if you don't know exactly what they mean.
Avoid metaphors.
Almost the rough content of every paragraph.
The order of the various topics in your paper.
On the basis of the outline, start writing a part by planning the content, and then write it down.
Put a visible mark (which you will later delete) where you need to quote a source, and write in the citation when you finish writing that part or a bigger part.
Does the text make sense?
Could you explain what you wanted?
Did you write good sentences?
Is there something missing?
Check the spelling.
Complete the citations, bring them in standard format.

Use the guidelines that your instructor requires (MLA, Chicago, APA, Turabian, etc.).

Adjust margins, spacing, paragraph indentation, place of page numbers, etc.
Standardize the bibliography or footnotes according to the guidelines.

EndNote and EndNote Basic by UCLA Library Last Updated Aug 29, 2023 1995 views this year
Zotero by UCLA Library Last Updated Jan 18, 2023 1277 views this year

(Based on English Composition 2 from Illinois Valley Community College):

Weak organization
Poor support and development of ideas
Weak use of secondary sources
Excessive errors
Stylistic weakness

When collecting materials, selecting research topic, and writing the paper:

Be systematic and organized (e.g. keep your bibliography neat and organized; write your notes in a neat way, so that you can find them later on.
Use your critical thinking ability when you read.
Write down your thoughts (so that you can reconstruct them later).
Stop when you have a really good idea and think about whether you could enlarge it to a whole research paper. If yes, take much longer notes.
When you write down a quotation or summarize somebody else's thoughts in your notes or in the paper, cite the source (i.e. write down the author, title, publication place, year, page number).
If you quote or summarize a thought from the internet, cite the internet source.
Write an outline that is detailed enough to remind you about the content.
Read your paper for yourself or, preferably, somebody else.
When you finish writing, check the spelling;
Use the citation form (MLA, Chicago, or other) that your instructor requires and use it everywhere.

Plagiarism : somebody else's words or ideas presented without citation by an author

Cite your source every time when you quote a part of somebody's work.
Cite your source every time when you summarize a thought from somebody's work.
Cite your source every time when you use a source (quote or summarize) from the Internet.

Consult the Citing Sources research guide for further details.

<< Previous: Writing a Research Proposal
Next: Presenting the Research Paper >>
Last Updated: Sep 18, 2023 4:52 PM
URL: https://guides.library.ucla.edu/research-methods

USC Libraries
Research Guides

Organizing Your Social Sciences Research Paper

6. The Methodology
Purpose of Guide
Design Flaws to Avoid
Independent and Dependent Variables
Glossary of Research Terms
Reading Research Effectively
Narrowing a Topic Idea
Broadening a Topic Idea
Extending the Timeliness of a Topic Idea
Academic Writing Style
Choosing a Title
Making an Outline
Paragraph Development
Research Process Video Series
Executive Summary
The C.A.R.S. Model
Background Information
The Research Problem/Question
Theoretical Framework
Citation Tracking
Content Alert Services
Evaluating Sources
Primary Sources
Secondary Sources
Tiertiary Sources
Scholarly vs. Popular Publications
Qualitative Methods
Quantitative Methods
Insiderness
Using Non-Textual Elements
Limitations of the Study
Common Grammar Mistakes
Writing Concisely
Avoiding Plagiarism
Footnotes or Endnotes?
Further Readings
Generative AI and Writing
USC Libraries Tutorials and Other Guides
Bibliography

The methods section describes actions taken to investigate a research problem and the rationale for the application of specific procedures or techniques used to identify, select, process, and analyze information applied to understanding the problem, thereby, allowing the reader to critically evaluate a study’s overall validity and reliability. The methodology section of a research paper answers two main questions: How was the data collected or generated? And, how was it analyzed? The writing should be direct and precise and always written in the past tense.

Kallet, Richard H. "How to Write the Methods Section of a Research Paper." Respiratory Care 49 (October 2004): 1229-1232.

Importance of a Good Methodology Section

You must explain how you obtained and analyzed your results for the following reasons:

Readers need to know how the data was obtained because the method you chose affects the results and, by extension, how you interpreted their significance in the discussion section of your paper.
Methodology is crucial for any branch of scholarship because an unreliable method produces unreliable results and, as a consequence, undermines the value of your analysis of the findings.
In most cases, there are a variety of different methods you can choose to investigate a research problem. The methodology section of your paper should clearly articulate the reasons why you have chosen a particular procedure or technique.
The reader wants to know that the data was collected or generated in a way that is consistent with accepted practice in the field of study. For example, if you are using a multiple choice questionnaire, readers need to know that it offered your respondents a reasonable range of answers to choose from.
The method must be appropriate to fulfilling the overall aims of the study. For example, you need to ensure that you have a large enough sample size to be able to generalize and make recommendations based upon the findings.
The methodology should discuss the problems that were anticipated and the steps you took to prevent them from occurring. For any problems that do arise, you must describe the ways in which they were minimized or why these problems do not impact in any meaningful way your interpretation of the findings.
In the social and behavioral sciences, it is important to always provide sufficient information to allow other researchers to adopt or replicate your methodology. This information is particularly important when a new method has been developed or an innovative use of an existing method is utilized.

Bem, Daryl J. Writing the Empirical Journal Article. Psychology Writing Center. University of Washington; Denscombe, Martyn. The Good Research Guide: For Small-Scale Social Research Projects . 5th edition. Buckingham, UK: Open University Press, 2014; Lunenburg, Frederick C. Writing a Successful Thesis or Dissertation: Tips and Strategies for Students in the Social and Behavioral Sciences . Thousand Oaks, CA: Corwin Press, 2008.

Structure and Writing Style

I. Groups of Research Methods

There are two main groups of research methods in the social sciences:

The e mpirical-analytical group approaches the study of social sciences in a similar manner that researchers study the natural sciences . This type of research focuses on objective knowledge, research questions that can be answered yes or no, and operational definitions of variables to be measured. The empirical-analytical group employs deductive reasoning that uses existing theory as a foundation for formulating hypotheses that need to be tested. This approach is focused on explanation.
The i nterpretative group of methods is focused on understanding phenomenon in a comprehensive, holistic way . Interpretive methods focus on analytically disclosing the meaning-making practices of human subjects [the why, how, or by what means people do what they do], while showing how those practices arrange so that it can be used to generate observable outcomes. Interpretive methods allow you to recognize your connection to the phenomena under investigation. However, the interpretative group requires careful examination of variables because it focuses more on subjective knowledge.

II. Content

The introduction to your methodology section should begin by restating the research problem and underlying assumptions underpinning your study. This is followed by situating the methods you used to gather, analyze, and process information within the overall “tradition” of your field of study and within the particular research design you have chosen to study the problem. If the method you choose lies outside of the tradition of your field [i.e., your review of the literature demonstrates that the method is not commonly used], provide a justification for how your choice of methods specifically addresses the research problem in ways that have not been utilized in prior studies.

The remainder of your methodology section should describe the following:

Decisions made in selecting the data you have analyzed or, in the case of qualitative research, the subjects and research setting you have examined,
Tools and methods used to identify and collect information, and how you identified relevant variables,
The ways in which you processed the data and the procedures you used to analyze that data, and
The specific research tools or strategies that you utilized to study the underlying hypothesis and research questions.

In addition, an effectively written methodology section should:

Introduce the overall methodological approach for investigating your research problem . Is your study qualitative or quantitative or a combination of both (mixed method)? Are you going to take a special approach, such as action research, or a more neutral stance?
Indicate how the approach fits the overall research design . Your methods for gathering data should have a clear connection to your research problem. In other words, make sure that your methods will actually address the problem. One of the most common deficiencies found in research papers is that the proposed methodology is not suitable to achieving the stated objective of your paper.
Describe the specific methods of data collection you are going to use , such as, surveys, interviews, questionnaires, observation, archival research. If you are analyzing existing data, such as a data set or archival documents, describe how it was originally created or gathered and by whom. Also be sure to explain how older data is still relevant to investigating the current research problem.
Explain how you intend to analyze your results . Will you use statistical analysis? Will you use specific theoretical perspectives to help you analyze a text or explain observed behaviors? Describe how you plan to obtain an accurate assessment of relationships, patterns, trends, distributions, and possible contradictions found in the data.
Provide background and a rationale for methodologies that are unfamiliar for your readers . Very often in the social sciences, research problems and the methods for investigating them require more explanation/rationale than widely accepted rules governing the natural and physical sciences. Be clear and concise in your explanation.
Provide a justification for subject selection and sampling procedure . For instance, if you propose to conduct interviews, how do you intend to select the sample population? If you are analyzing texts, which texts have you chosen, and why? If you are using statistics, why is this set of data being used? If other data sources exist, explain why the data you chose is most appropriate to addressing the research problem.
Provide a justification for case study selection . A common method of analyzing research problems in the social sciences is to analyze specific cases. These can be a person, place, event, phenomenon, or other type of subject of analysis that are either examined as a singular topic of in-depth investigation or multiple topics of investigation studied for the purpose of comparing or contrasting findings. In either method, you should explain why a case or cases were chosen and how they specifically relate to the research problem.
Describe potential limitations . Are there any practical limitations that could affect your data collection? How will you attempt to control for potential confounding variables and errors? If your methodology may lead to problems you can anticipate, state this openly and show why pursuing this methodology outweighs the risk of these problems cropping up.

NOTE : Once you have written all of the elements of the methods section, subsequent revisions should focus on how to present those elements as clearly and as logically as possibly. The description of how you prepared to study the research problem, how you gathered the data, and the protocol for analyzing the data should be organized chronologically. For clarity, when a large amount of detail must be presented, information should be presented in sub-sections according to topic. If necessary, consider using appendices for raw data.

ANOTHER NOTE : If you are conducting a qualitative analysis of a research problem , the methodology section generally requires a more elaborate description of the methods used as well as an explanation of the processes applied to gathering and analyzing of data than is generally required for studies using quantitative methods. Because you are the primary instrument for generating the data [e.g., through interviews or observations], the process for collecting that data has a significantly greater impact on producing the findings. Therefore, qualitative research requires a more detailed description of the methods used.

YET ANOTHER NOTE : If your study involves interviews, observations, or other qualitative techniques involving human subjects , you may be required to obtain approval from the university's Office for the Protection of Research Subjects before beginning your research. This is not a common procedure for most undergraduate level student research assignments. However, i f your professor states you need approval, you must include a statement in your methods section that you received official endorsement and adequate informed consent from the office and that there was a clear assessment and minimization of risks to participants and to the university. This statement informs the reader that your study was conducted in an ethical and responsible manner. In some cases, the approval notice is included as an appendix to your paper.

III. Problems to Avoid

Irrelevant Detail The methodology section of your paper should be thorough but concise. Do not provide any background information that does not directly help the reader understand why a particular method was chosen, how the data was gathered or obtained, and how the data was analyzed in relation to the research problem [note: analyzed, not interpreted! Save how you interpreted the findings for the discussion section]. With this in mind, the page length of your methods section will generally be less than any other section of your paper except the conclusion.

Unnecessary Explanation of Basic Procedures Remember that you are not writing a how-to guide about a particular method. You should make the assumption that readers possess a basic understanding of how to investigate the research problem on their own and, therefore, you do not have to go into great detail about specific methodological procedures. The focus should be on how you applied a method , not on the mechanics of doing a method. An exception to this rule is if you select an unconventional methodological approach; if this is the case, be sure to explain why this approach was chosen and how it enhances the overall process of discovery.

Problem Blindness It is almost a given that you will encounter problems when collecting or generating your data, or, gaps will exist in existing data or archival materials. Do not ignore these problems or pretend they did not occur. Often, documenting how you overcame obstacles can form an interesting part of the methodology. It demonstrates to the reader that you can provide a cogent rationale for the decisions you made to minimize the impact of any problems that arose.

Literature Review Just as the literature review section of your paper provides an overview of sources you have examined while researching a particular topic, the methodology section should cite any sources that informed your choice and application of a particular method [i.e., the choice of a survey should include any citations to the works you used to help construct the survey].

It’s More than Sources of Information! A description of a research study's method should not be confused with a description of the sources of information. Such a list of sources is useful in and of itself, especially if it is accompanied by an explanation about the selection and use of the sources. The description of the project's methodology complements a list of sources in that it sets forth the organization and interpretation of information emanating from those sources.

Azevedo, L.F. et al. "How to Write a Scientific Paper: Writing the Methods Section." Revista Portuguesa de Pneumologia 17 (2011): 232-238; Blair Lorrie. “Choosing a Methodology.” In Writing a Graduate Thesis or Dissertation , Teaching Writing Series. (Rotterdam: Sense Publishers 2016), pp. 49-72; Butin, Dan W. The Education Dissertation A Guide for Practitioner Scholars . Thousand Oaks, CA: Corwin, 2010; Carter, Susan. Structuring Your Research Thesis . New York: Palgrave Macmillan, 2012; Kallet, Richard H. “How to Write the Methods Section of a Research Paper.” Respiratory Care 49 (October 2004):1229-1232; Lunenburg, Frederick C. Writing a Successful Thesis or Dissertation: Tips and Strategies for Students in the Social and Behavioral Sciences . Thousand Oaks, CA: Corwin Press, 2008. Methods Section. The Writer’s Handbook. Writing Center. University of Wisconsin, Madison; Rudestam, Kjell Erik and Rae R. Newton. “The Method Chapter: Describing Your Research Plan.” In Surviving Your Dissertation: A Comprehensive Guide to Content and Process . (Thousand Oaks, Sage Publications, 2015), pp. 87-115; What is Interpretive Research. Institute of Public and International Affairs, University of Utah; Writing the Experimental Report: Methods, Results, and Discussion. The Writing Lab and The OWL. Purdue University; Methods and Materials. The Structure, Format, Content, and Style of a Journal-Style Scientific Paper. Department of Biology. Bates College.

Writing Tip

Statistical Designs and Tests? Do Not Fear Them!

Don't avoid using a quantitative approach to analyzing your research problem just because you fear the idea of applying statistical designs and tests. A qualitative approach, such as conducting interviews or content analysis of archival texts, can yield exciting new insights about a research problem, but it should not be undertaken simply because you have a disdain for running a simple regression. A well designed quantitative research study can often be accomplished in very clear and direct ways, whereas, a similar study of a qualitative nature usually requires considerable time to analyze large volumes of data and a tremendous burden to create new paths for analysis where previously no path associated with your research problem had existed.

To locate data and statistics, GO HERE .

Another Writing Tip

Knowing the Relationship Between Theories and Methods

There can be multiple meaning associated with the term "theories" and the term "methods" in social sciences research. A helpful way to delineate between them is to understand "theories" as representing different ways of characterizing the social world when you research it and "methods" as representing different ways of generating and analyzing data about that social world. Framed in this way, all empirical social sciences research involves theories and methods, whether they are stated explicitly or not. However, while theories and methods are often related, it is important that, as a researcher, you deliberately separate them in order to avoid your theories playing a disproportionate role in shaping what outcomes your chosen methods produce.

Introspectively engage in an ongoing dialectic between the application of theories and methods to help enable you to use the outcomes from your methods to interrogate and develop new theories, or ways of framing conceptually the research problem. This is how scholarship grows and branches out into new intellectual territory.

Reynolds, R. Larry. Ways of Knowing. Alternative Microeconomics . Part 1, Chapter 3. Boise State University; The Theory-Method Relationship. S-Cool Revision. United Kingdom.

Yet Another Writing Tip

Methods and the Methodology

Do not confuse the terms "methods" and "methodology." As Schneider notes, a method refers to the technical steps taken to do research . Descriptions of methods usually include defining and stating why you have chosen specific techniques to investigate a research problem, followed by an outline of the procedures you used to systematically select, gather, and process the data [remember to always save the interpretation of data for the discussion section of your paper].

The methodology refers to a discussion of the underlying reasoning why particular methods were used . This discussion includes describing the theoretical concepts that inform the choice of methods to be applied, placing the choice of methods within the more general nature of academic work, and reviewing its relevance to examining the research problem. The methodology section also includes a thorough review of the methods other scholars have used to study the topic.

Bryman, Alan. "Of Methods and Methodology." Qualitative Research in Organizations and Management: An International Journal 3 (2008): 159-168; Schneider, Florian. “What's in a Methodology: The Difference between Method, Methodology, and Theory…and How to Get the Balance Right?” PoliticsEastAsia.com. Chinese Department, University of Leiden, Netherlands.

<< Previous: Scholarly vs. Popular Publications
Next: Qualitative Methods >>
Last Updated: Oct 10, 2023 1:30 PM
URL: https://libguides.usc.edu/writingguide

Essential components of methods papers

Affiliations.

1 In vitro Toxicology and Biomedicine, Dept inaugurated by the Doerenkamp-Zbinden Foundation, University of Konstanz, Konstanz, Germany.
2 Konstanz Research School Chemical Biology (KoRS-CB) and Co-operative research training school 'Advanced in-vitro test systems for the analysis of cell-chemical interactions in drug discovery and environmental safety (inviTe)', University of Konstanz, Konstanz, Germany.
3 CAAT-Europe, University of Konstanz, Konstanz, Germany.
4 Leibniz Research Centre for Working Environment and Human Factors (IfADo), Technical University of Dortmund, Dortmund, Germany.
5 Archives of Toxicology, editor-in-chief, Heidelberg, Germany.
PMID: 30008013
DOI: 10.14573/altex.1807031

Methods papers are important for the progress of biomedical research, as they provide the essential tools to explore new questions and help to better answer old ones. However, it is often not clear how a methods paper differs from a methods protocol. Confusion between these two very different types of publication is widespread. The resultant misunderstanding contributes to a relatively poor reputation of methods research in biology despite the fact that many Nobel prizes have been awarded specifically for method development. Here, the key components of a methods paper are summarized: (i) methods description, (ii) performance standards, (iii) applicability domains, (iv) evidence for advances compared to the state-of-the-art, (v) exemplification of the method by practical application. In addition, information domains are discussed that are desirable but may be provided on a case-by-case basis or over the course of a series of papers: (vi) method robustness, (vii) accuracy and (viii) precision measures, including various quantifications of method performance, and (ix) measures of uncertainty, including a sensitivity analysis. Finally, elements of the overall framing of the method description are highlighted. These include the scientific, technical and, e.g., toxicological rationale for the method, and also the prediction model, i.e., the procedure used to transform primary data into new information.

Keywords: BenchMarks series.

Publication types

Biomedical Research*
Publications / standards*
Research Design / standards*

Organizing Academic Research Papers: 6. The Methodology

Purpose of Guide
Design Flaws to Avoid
Glossary of Research Terms
Narrowing a Topic Idea
Broadening a Topic Idea
Extending the Timeliness of a Topic Idea
Academic Writing Style
Choosing a Title
Making an Outline
Paragraph Development
Executive Summary
Background Information
The Research Problem/Question
Theoretical Framework
Citation Tracking
Content Alert Services
Evaluating Sources
Primary Sources
Secondary Sources
Tertiary Sources
What Is Scholarly vs. Popular?
Qualitative Methods
Quantitative Methods
Using Non-Textual Elements
Limitations of the Study
Common Grammar Mistakes
Avoiding Plagiarism
Footnotes or Endnotes?
Further Readings
Annotated Bibliography
Dealing with Nervousness
Using Visual Aids
Grading Someone Else's Paper
How to Manage Group Projects
Multiple Book Review Essay
Reviewing Collected Essays
About Informed Consent
Writing Field Notes
Writing a Policy Memo
Writing a Research Proposal
Acknowledgements

The methods section of a research paper provides the information by which a study’s validity is judged. The method section answers two main questions: 1) How was the data collected or generated? 2) How was it analyzed? The writing should be direct and precise and written in the past tense.

Importance of a Good Methodology Section

You must explain how you obtained and analyzed your results for the following reasons:

Readers need to know how the data was obtained because the method you choose affects the results and, by extension, how you likely interpreted those results.
Methodology is crucial for any branch of scholarship because an unreliable method produces unreliable results and it misappropriates interpretations of findings .
In most cases, there are a variety of different methods you can choose to investigate a research problem. Your methodology section of your paper should make clear the reasons why you chose a particular method or procedure .
The reader wants to know that the data was collected or generated in a way that is consistent with accepted practice in the field of study. For example, if you are using a questionnaire, readers need to know that it offered your respondents a reasonable range of answers to choose from.
The research method must be appropriate to the objectives of the study . For example, be sure you have a large enough sample size to be able to generalize and make recommendations based upon the findings.
The methodology should discuss the problems that were anticipated and the steps you took to prevent them from occurring . For any problems that did arise, you must describe the ways in which their impact was minimized or why these problems do not affect the findings in any way that impacts your interpretation of the data.
Often in social science research, it is useful for other researchers to adapt or replicate your methodology. Therefore, it is important to always provide sufficient information to allow others to use or replicate the study . This information is particularly important when a new method had been developed or an innovative use of an existing method has been utilized.

Bem, Daryl J. Writing the Empirical Journal Article . Psychology Writing Center. University of Washington; Lunenburg, Frederick C. Writing a Successful Thesis or Dissertation: Tips and Strategies for Students in the Social and Behavioral Sciences . Thousand Oaks, CA: Corwin Press, 2008.

Structure and Writing Style

I. Groups of Research Methods

There are two main groups of research methods in the social sciences:

The empirical-analytical group approaches the study of social sciences in a similar manner that researchers study the natural sciences. This type of research focuses on objective knowledge, research questions that can be answered yes or no, and operational definitions of variables to be measured. The empirical-analytical group employs deductive reasoning that uses existing theory as a foundation for hypotheses that need to be tested. This approach is focused on explanation .
The interpretative group is focused on understanding phenomenon in a comprehensive, holistic way . This research method allows you to recognize your connection to the subject under study. Because the interpretative group focuses more on subjective knowledge, it requires careful interpretation of variables.

II. Content

An effectively written methodology section should:

Introduce the overall methodological approach for investigating your research problem . Is your study qualitative or quantitative or a combination of both (mixed method)? Are you going to take a special approach, such as action research, or a more neutral stance?
Indicate how the approach fits the overall research design . Your methods should have a clear connection with your research problem. In other words, make sure that your methods will actually address the problem. One of the most common deficiencies found in research papers is that the proposed methodology is unsuited to achieving the stated objective of your paper.
Describe the specific methods of data collection you are going to use , such as, surveys, interviews, questionnaires, observation, archival research. If you are analyzing existing data, such as a data set or archival documents, describe how it was originally created or gathered and by whom.
Explain how you intend to analyze your results . Will you use statistical analysis? Will you use specific theoretical perspectives to help you analyze a text or explain observed behaviors?
Provide background and rationale for methodologies that are unfamiliar for your readers . Very often in the social sciences, research problems and the methods for investigating them require more explanation/rationale than widely accepted rules governing the natural and physical sciences. Be clear and concise in your explanation.
Provide a rationale for subject selection and sampling procedure . For instance, if you propose to conduct interviews, how do you intend to select the sample population? If you are analyzing texts, which texts have you chosen, and why? If you are using statistics, why is this set of statisics being used? If other data sources exist, explain why the data you chose is most appropriate.
Address potential limitations . Are there any practical limitations that could affect your data collection? How will you attempt to control for potential confounding variables and errors? If your methodology may lead to problems you can anticipate, state this openly and show why pursuing this methodology outweighs the risk of these problems cropping up.

III. Problems to Avoid

Irrelevant Detail The methodology section of your paper should be thorough but to the point. Don’t provide any background information that doesn’t directly help the reader to understand why a particular method was chosen, how the data was gathered or obtained, and how it was analyzed. Unnecessary Explanation of Basic Procedures Remember that you are not writing a how-to guide about a particular method. You should make the assumption that readers possess a basic understanding of how to investigate the research problem on their own and, therefore, you do not have to go into great detail about specific methodological procedures. The focus should be on how you applied a method , not on the mechanics of doing a method. NOTE: An exception to this rule is if you select an unconventional approach to doing the method; if this is the case, be sure to explain why this approach was chosen and how it enhances the overall research process. Problem Blindness It is almost a given that you will encounter problems when collecting or generating your data. Do not ignore these problems or pretend they did not occur. Often, documenting how you overcame obstacles can form an interesting part of the methodology. It demonstrates to the reader that you can provide a cogent rationale for the decisions you made to minimize the impact of any problems that arose. Literature Review Just as the literature review section of your paper provides an overview of sources you have examined while researching a particular topic, the methodology section should cite any sources that informed your choice and application of a particular method [i.e., the choice of a survey should include any citations to the works you used to help construct the survey].

It’s More than Sources of Information! A description of a research study's method should not be confused with a description of the sources of information. Such a list of sources is useful in itself, especially if it is accompanied by an explanation about the selection and use of the sources. The description of the project's methodology complements a list of sources in that it sets forth the organization and interpretation of information emanating from those sources.

Azevedo, L.F. et al. How to Write a Scientific Paper: Writing the Methods Section. Revista Portuguesa de Pneumologia 17 (2011): 232-238; Butin, Dan W. The Education Dissertation A Guide for Practitioner Scholars . Thousand Oaks, CA: Corwin, 2010; Carter, Susan. Structuring Your Research Thesis . New York: Palgrave Macmillan, 2012; Lunenburg, Frederick C. Writing a Successful Thesis or Dissertation: Tips and Strategies for Students in the Social and Behavioral Sciences . Thousand Oaks, CA: Corwin Press, 2008. Methods Section . The Writer’s Handbook. Writing Center. University of Wisconsin, Madison; Writing the Experimental Report: Methods, Results, and Discussion . The Writing Lab and The OWL. Purdue University; Methods and Materials . The Structure, Format, Content, and Style of a Journal-Style Scientific Paper. Department of Biology. Bates College.

Writing Tip

Statistical Designs and Tests? Do Not Fear Them!

Another Writing Tip

Knowing the Relationship Between Theories and Methods

Introspectively engage in an ongoing dialectic between theories and methods to help enable you to use the outcomes from your methods to interrogate and develop new theories, or ways of framing conceptually the research problem. This is how scholarship grows and branches out into new intellectual territory.

Reynolds, R. Larry. Ways of Knowing. Alternative Microeconomics. Part 1, Chapter 3. Boise State University; The Theory-Method Relationship . S-Cool Revision. United Kingdom.

<< Previous: What Is Scholarly vs. Popular?
Next: Qualitative Methods >>
Last Updated: Jul 18, 2023 11:58 AM
URL: https://library.sacredheart.edu/c.php?g=29803
QuickSearch
Library Catalog
Databases A-Z
Publication Finder
Course Reserves
Citation Linker
Digital Commons

Research Support

Ask a Librarian
Appointments
Interlibrary Loan (ILL)
Research Guides
Databases by Subject
Citation Help

Using the Library

Reserve a Group Study Room
Renew Books
Honors Study Rooms
Off-Campus Access
Library Policies
Library Technology

User Information

Grad Students
Online Students
COVID-19 Updates
Staff Directory
News & Announcements
Library Newsletter

My Accounts

Interlibrary Loan
Staff Site Login

FIND US ON

How to write the Methods section of a research paper

Dr. Dhriti Bhattacharyya

The Methods section of a research article is like a roadmap leading to the core of the research, guiding the readers through the actual journey the authors took to reach their destination. In the manuscript, this section contains the essential details for other scientists to replicate the experiments of the study and help the common readers to understand the study better.

In this article, we will share some tips to make the Methods section of your manuscript interesting and informative. While the article uses examples mostly from the biomedical and clinical research studies, authors from other fields too would find the tips useful for preparing their next manuscript.

Break ice between the readers and the Methods section

First, let’s ponder over the issue of the perception of boredom we often associate with the Methods section of an article. It may be the names of the reagents and instruments, separated by some numbers in terms of some concentrations or the technical terminologies that make the reading a heavy-duty task. Listed below are some useful ways of breaking the ice between the Methods section and the readers:

1. Explanation : Usually, each paragraph or subsection of the Methods section talks about a specific experiment. Early in each paragraph, explain the rationale behind your choices of that particular experiment.; for example, why you used a certain compound, a specific strain of mice as the experimental model or the particular concentration of that key reagent.

For clinical research, providing a detailed rationale for selecting the exclusion or inclusion criteria can be a good idea to present early in the Methods section. If you took a conventional or widely used method, you certainly don’t need to appear stating the obvious, but for less conventional approaches sharing your reasoning of the study design instantly makes the readers curious and engaged with your paper.

2. Visual presentation : To help the readers follow the study design or methodology better, visual elements like the schematic diagram, flowchart, and table can be used in this section. They help in breaking the monotony and making the absorption of complex information easy.

The dos and don’ts of writing the Methods section

Secondly, the information in the methods section is closely scrutinized by the journal editors and peer reviewers to assess whether the most appropriate technique was used to reach your research goal. While every detail of your experiment need not be included, the essential and critical steps should be well described to receive a positive peer review.

The essential do’s and don’ts of writing a technically sound Methods section:

1. Adhere to the specific guidelines: Read the author’s instruction section of your target journal carefully and follow the specific instructions. For example, the heading of the section “Materials and Methods” may need to be changed to “Patients and the Method” to follow the guidelines of your target journal or the name of the institutes could be omitted for the journals that do not prefer open-label reporting. Also, you may be expected to follow a particular style guideline like the one published by the American Psychological Association while writing the Methods section.

Biomedical researchers would benefit from using the checklists for different study types to ensure the essential details are included in the Methods. Some of the standardized and widely referred checklists include the ones for randomized clinical trials CONSORT (Consolidated Standards of Reporting Trials), cohort, case-control, cross‐sectional studies STROBE (STrengthening the Reporting of OBservational studies in Epidemiology), diagnostic accuracy STARD (STAndards for the Reporting of Diagnostic accuracy studies), systematic reviews and meta‐analyses PRISMA (Preferred Reporting Items for Systematic reviews and Meta‐Analyses), and Case reports CARE (CAse REport).

2. Structure the section so that it tells the story of your research : All the experiments should be presented in a logical manner that helps the reader retrace the gradual and development and nuances of the study. A useful way of achieving this is to describe the methods in a chronological order of the experiments. For example: for a clinical trial, you may start with the setting and time of the study ( the beginning and termination dates of the study) , followed by the details of the patient recruitment ( Number of subjects/patients etc.) , study design (prospective, retrospective or other), randomization (if any), assigning into groups, intervention, and describing the techniques used to collect, measure, and analyse data.

3. Follow the order of the results: To improve the readability and flow of your manuscript, match the order of specific methods to the order of the results that were achieved using those methods.

4. Use subheadings: Dividing the Methods section in terms of the experiments helps the reader to follow the section better. You may write the specific objective of each experiment as a subheading. Alternatively, if applicable, the name of each experiment can also be used as subheading.

5. Provide all details meticulously: Provide the details that you considered while designing the study or collecting the data because the smallest variations in these steps may affect the results and interpretation of their significance. When employing the outcome measures, the readers would like to know the information regarding validity and reliability. The correct way of reporting the reliability and the validity depends on the specific research design. Usually, information from existing literature is presented to support for the reliability and the validity of a measure.

Carefully describe the materials, equipment (like testing instruments and technical equipment), or stimuli used in the experiment. If your study involved a survey or any psychological assessment, mention the questionnaire, scoring methods, and validation of scales with every possible detail.

Also, be careful about one common manuscript error i.e. not mentioning the sample size estimation (whenever relevant). Although the estimated sample size is computed before the actual study starts, it helps the reader assess the expected change in the outcome variables and the number of subjects needed to detect that change within a certain confidence range. Similarly, mentioning power calculation is a critical point to be mentioned in the Methods section.

6. Mention the ethical approval: If relevant, early in the Methods section mention whether your study was approved by the ethics committee or institutional review board, and whether you have received oral/ written informed consent from the patients or the guardians.

7. Specify the variables : Clearly mention not only the control variables, independent variables, dependent variables but also if there were any extraneous variables that might influence the result of your study. For example, in a tutorial on learning how to write ‘Research Methodology’, one group is provided with a traditional text while the other group is provided with an interactive online tool. However, if some participants already have prior knowledge of ‘how to write the Methods section’, this pre-knowledge will act as an extraneous variable.

8. Statistical analysis: In this section, describe all statistical tests, levels of significance, and software packages used to conduct the statistical analysis. You may also consult the biostatistician of your team to receive help to write this section . Don’t forget to indicate if the recommendations of a knowledgeable and experienced statistician were considered. Finally, it is important to provide the justification of the preferred statistical method used in the study. For example, why the author is using a one-tailed or two-tailed analysis.

1. Do not describe well-known methods in detail: For the sake of brevity, avoid listing the details of the experiments that are widely used or already published in numerous articles in your field of research. Instead, mention and cite the specific experiment and mention that the referred process was followed. However, if you have modified the standard process to meet the specific aim of your study, do describe the modifications and the reasons for those in sufficient detail.

2. Do not provide unnecessary details: Avoid unnecessary details that are not relevant to the result of the experiment. For example, you need not mention trivial details such as the color of the bucket that held the ice. Try to stick only to the details that are relevant and have an impact on your study.

3. Do not discuss the pros and cons of other methods: While it may be tempting to discuss the reasons why you did not use a particular method or how your chosen method is superior to others, save these details for the Discussion section. Utilize the Methods section only to mention the details of the methods you chose.

To summarize all the tips stated above, the Methods section of an ideal manuscript aims to share the scientific knowledge with transparency and also establishes the robustness of the study. I hope that this article helps you to reach the goal of writing a perfect manuscript!

The correct way to report p values

How to choose the research methodology best suited for your study

Planning to Write

Manuscript structure: How to convey your most important ideas through…

How to write the Methods section of a research paper 8 min read

10 Tips on how to write an effective research grant proposal 9 min read

11 Commonly confused elements of a research paper 16 min read

Manuscript structure: How to convey your most important ideas through your paper 6 min read

How to choose the research methodology best suited for your study 7 min read

Recent Searches

Review paper
Responding to reviewer comments
Predatory publishers
Scope and delimitations
Open access
Plagiarism in research
Journal selection tips
Editor assigned
Types of articles
"Reject and Resubmit" status
Decision in process
Conflict of interest

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

Knowledge Base
Research paper

Writing a Research Paper Introduction | Step-by-Step Guide

Published on September 24, 2022 by Jack Caulfield . Revised on March 27, 2023.

The introduction to a research paper is where you set up your topic and approach for the reader. It has several key goals:

Present your topic and get the reader interested
Provide background or summarize existing research
Position your own approach
Detail your specific research problem and problem statement
Give an overview of the paper’s structure

The introduction looks slightly different depending on whether your paper presents the results of original empirical research or constructs an argument by engaging with a variety of sources.

Step 1: introduce your topic, step 2: describe the background, step 3: establish your research problem, step 4: specify your objective(s), step 5: map out your paper, research paper introduction examples, frequently asked questions about the research paper introduction.

The first job of the introduction is to tell the reader what your topic is and why it’s interesting or important. This is generally accomplished with a strong opening hook.

The hook is a striking opening sentence that clearly conveys the relevance of your topic. Think of an interesting fact or statistic, a strong statement, a question, or a brief anecdote that will get the reader wondering about your topic.

For example, the following could be an effective hook for an argumentative paper about the environmental impact of cattle farming:

A more empirical paper investigating the relationship of Instagram use with body image issues in adolescent girls might use the following hook:

Don’t feel that your hook necessarily has to be deeply impressive or creative. Clarity and relevance are still more important than catchiness. The key thing is to guide the reader into your topic and situate your ideas.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

This part of the introduction differs depending on what approach your paper is taking.

In a more argumentative paper, you’ll explore some general background here. In a more empirical paper, this is the place to review previous research and establish how yours fits in.

Argumentative paper: Background information

After you’ve caught your reader’s attention, specify a bit more, providing context and narrowing down your topic.

Provide only the most relevant background information. The introduction isn’t the place to get too in-depth; if more background is essential to your paper, it can appear in the body .

Empirical paper: Describing previous research

For a paper describing original research, you’ll instead provide an overview of the most relevant research that has already been conducted. This is a sort of miniature literature review —a sketch of the current state of research into your topic, boiled down to a few sentences.

This should be informed by genuine engagement with the literature. Your search can be less extensive than in a full literature review, but a clear sense of the relevant research is crucial to inform your own work.

Begin by establishing the kinds of research that have been done, and end with limitations or gaps in the research that you intend to respond to.

The next step is to clarify how your own research fits in and what problem it addresses.

Argumentative paper: Emphasize importance

In an argumentative research paper, you can simply state the problem you intend to discuss, and what is original or important about your argument.

Empirical paper: Relate to the literature

In an empirical research paper, try to lead into the problem on the basis of your discussion of the literature. Think in terms of these questions:

What research gap is your work intended to fill?
What limitations in previous work does it address?
What contribution to knowledge does it make?

You can make the connection between your problem and the existing research using phrases like the following.

Now you’ll get into the specifics of what you intend to find out or express in your research paper.

The way you frame your research objectives varies. An argumentative paper presents a thesis statement, while an empirical paper generally poses a research question (sometimes with a hypothesis as to the answer).

Argumentative paper: Thesis statement

The thesis statement expresses the position that the rest of the paper will present evidence and arguments for. It can be presented in one or two sentences, and should state your position clearly and directly, without providing specific arguments for it at this point.

Empirical paper: Research question and hypothesis

The research question is the question you want to answer in an empirical research paper.

Present your research question clearly and directly, with a minimum of discussion at this point. The rest of the paper will be taken up with discussing and investigating this question; here you just need to express it.

A research question can be framed either directly or indirectly.

This study set out to answer the following question: What effects does daily use of Instagram have on the prevalence of body image issues among adolescent girls?
We investigated the effects of daily Instagram use on the prevalence of body image issues among adolescent girls.

If your research involved testing hypotheses , these should be stated along with your research question. They are usually presented in the past tense, since the hypothesis will already have been tested by the time you are writing up your paper.

For example, the following hypothesis might respond to the research question above:

A faster, more affordable way to improve your paper

Scribbr’s new AI Proofreader checks your document and corrects spelling, grammar, and punctuation mistakes with near-human accuracy and the efficiency of AI!

Proofread my paper

The final part of the introduction is often dedicated to a brief overview of the rest of the paper.

In a paper structured using the standard scientific “introduction, methods, results, discussion” format, this isn’t always necessary. But if your paper is structured in a less predictable way, it’s important to describe the shape of it for the reader.

If included, the overview should be concise, direct, and written in the present tense.

This paper will first discuss several examples of survey-based research into adolescent social media use, then will go on to …
This paper first discusses several examples of survey-based research into adolescent social media use, then goes on to …

Full examples of research paper introductions are shown in the tabs below: one for an argumentative paper, the other for an empirical paper.

Argumentative paper
Empirical paper

Are cows responsible for climate change? A recent study (RIVM, 2019) shows that cattle farmers account for two thirds of agricultural nitrogen emissions in the Netherlands. These emissions result from nitrogen in manure, which can degrade into ammonia and enter the atmosphere. The study’s calculations show that agriculture is the main source of nitrogen pollution, accounting for 46% of the country’s total emissions. By comparison, road traffic and households are responsible for 6.1% each, the industrial sector for 1%. While efforts are being made to mitigate these emissions, policymakers are reluctant to reckon with the scale of the problem. The approach presented here is a radical one, but commensurate with the issue. This paper argues that the Dutch government must stimulate and subsidize livestock farmers, especially cattle farmers, to transition to sustainable vegetable farming. It first establishes the inadequacy of current mitigation measures, then discusses the various advantages of the results proposed, and finally addresses potential objections to the plan on economic grounds.

The rise of social media has been accompanied by a sharp increase in the prevalence of body image issues among women and girls. This correlation has received significant academic attention: Various empirical studies have been conducted into Facebook usage among adolescent girls (Tiggermann & Slater, 2013; Meier & Gray, 2014). These studies have consistently found that the visual and interactive aspects of the platform have the greatest influence on body image issues. Despite this, highly visual social media (HVSM) such as Instagram have yet to be robustly researched. This paper sets out to address this research gap. We investigated the effects of daily Instagram use on the prevalence of body image issues among adolescent girls. It was hypothesized that daily Instagram use would be associated with an increase in body image concerns and a decrease in self-esteem ratings.

The introduction of a research paper includes several key elements:

A hook to catch the reader’s interest
Relevant background on the topic
Details of your research problem

and your problem statement

A thesis statement or research question
Sometimes an overview of the paper

Don’t feel that you have to write the introduction first. The introduction is often one of the last parts of the research paper you’ll write, along with the conclusion.

This is because it can be easier to introduce your paper once you’ve already written the body ; you may not have the clearest idea of your arguments until you’ve written them, and things can change during the writing process .

The way you present your research problem in your introduction varies depending on the nature of your research paper . A research paper that presents a sustained argument will usually encapsulate this argument in a thesis statement .

A research paper designed to present the results of empirical research tends to present a research question that it seeks to answer. It may also include a hypothesis —a prediction that will be confirmed or disproved by your research.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Caulfield, J. (2023, March 27). Writing a Research Paper Introduction | Step-by-Step Guide. Scribbr. Retrieved November 6, 2023, from https://www.scribbr.com/research-paper/research-paper-introduction/

Is this article helpful?

Jack Caulfield

Other students also liked, writing strong research questions | criteria & examples, writing a research paper conclusion | step-by-step guide, research paper format | apa, mla, & chicago templates, what is your plagiarism score.

Privacy Policy
SignUp/Login

Home » Research Methods – Types, Examples and Guide

Research Methods – Types, Examples and Guide

Table of Contents

Research Methods

Definition:

Research Methods refer to the techniques, procedures, and processes used by researchers to collect , analyze, and interpret data in order to answer research questions or test hypotheses. The methods used in research can vary depending on the research questions, the type of data that is being collected, and the research design.

Types of Research Methods

Types of Research Methods are as follows:

Qualitative research Method

Qualitative research methods are used to collect and analyze non-numerical data. This type of research is useful when the objective is to explore the meaning of phenomena, understand the experiences of individuals, or gain insights into complex social processes. Qualitative research methods include interviews, focus groups, ethnography, and content analysis.

Quantitative Research Method

Quantitative research methods are used to collect and analyze numerical data. This type of research is useful when the objective is to test a hypothesis, determine cause-and-effect relationships, and measure the prevalence of certain phenomena. Quantitative research methods include surveys, experiments, and secondary data analysis.

Mixed Method Research

Mixed Method Research refers to the combination of both qualitative and quantitative research methods in a single study. This approach aims to overcome the limitations of each individual method and to provide a more comprehensive understanding of the research topic. This approach allows researchers to gather both quantitative data, which is often used to test hypotheses and make generalizations about a population, and qualitative data, which provides a more in-depth understanding of the experiences and perspectives of individuals.

Key Differences Between Research Methods

The following Table shows the key differences between Quantitative, Qualitative and Mixed Research Methods

Examples of Research Methods

Examples of Research Methods are as follows:

Qualitative Research Example:

A researcher wants to study the experience of cancer patients during their treatment. They conduct in-depth interviews with patients to gather data on their emotional state, coping mechanisms, and support systems.

Quantitative Research Example:

A company wants to determine the effectiveness of a new advertisement campaign. They survey a large group of people, asking them to rate their awareness of the product and their likelihood of purchasing it.

Mixed Research Example:

A university wants to evaluate the effectiveness of a new teaching method in improving student performance. They collect both quantitative data (such as test scores) and qualitative data (such as feedback from students and teachers) to get a complete picture of the impact of the new method.

Applications of Research Methods

Research methods are used in various fields to investigate, analyze, and answer research questions. Here are some examples of how research methods are applied in different fields:

Psychology : Research methods are widely used in psychology to study human behavior, emotions, and mental processes. For example, researchers may use experiments, surveys, and observational studies to understand how people behave in different situations, how they respond to different stimuli, and how their brains process information.
Sociology : Sociologists use research methods to study social phenomena, such as social inequality, social change, and social relationships. Researchers may use surveys, interviews, and observational studies to collect data on social attitudes, beliefs, and behaviors.
Medicine : Research methods are essential in medical research to study diseases, test new treatments, and evaluate their effectiveness. Researchers may use clinical trials, case studies, and laboratory experiments to collect data on the efficacy and safety of different medical treatments.
Education : Research methods are used in education to understand how students learn, how teachers teach, and how educational policies affect student outcomes. Researchers may use surveys, experiments, and observational studies to collect data on student performance, teacher effectiveness, and educational programs.
Business : Research methods are used in business to understand consumer behavior, market trends, and business strategies. Researchers may use surveys, focus groups, and observational studies to collect data on consumer preferences, market trends, and industry competition.
Environmental science : Research methods are used in environmental science to study the natural world and its ecosystems. Researchers may use field studies, laboratory experiments, and observational studies to collect data on environmental factors, such as air and water quality, and the impact of human activities on the environment.
Political science : Research methods are used in political science to study political systems, institutions, and behavior. Researchers may use surveys, experiments, and observational studies to collect data on political attitudes, voting behavior, and the impact of policies on society.

Purpose of Research Methods

Research methods serve several purposes, including:

Identify research problems: Research methods are used to identify research problems or questions that need to be addressed through empirical investigation.
Develop hypotheses: Research methods help researchers develop hypotheses, which are tentative explanations for the observed phenomenon or relationship.
Collect data: Research methods enable researchers to collect data in a systematic and objective way, which is necessary to test hypotheses and draw meaningful conclusions.
Analyze data: Research methods provide tools and techniques for analyzing data, such as statistical analysis, content analysis, and discourse analysis.
Test hypotheses: Research methods allow researchers to test hypotheses by examining the relationships between variables in a systematic and controlled manner.
Draw conclusions : Research methods facilitate the drawing of conclusions based on empirical evidence and help researchers make generalizations about a population based on their sample data.
Enhance understanding: Research methods contribute to the development of knowledge and enhance our understanding of various phenomena and relationships, which can inform policy, practice, and theory.

When to Use Research Methods

Research methods are used when you need to gather information or data to answer a question or to gain insights into a particular phenomenon.

Here are some situations when research methods may be appropriate:

To investigate a problem : Research methods can be used to investigate a problem or a research question in a particular field. This can help in identifying the root cause of the problem and developing solutions.
To gather data: Research methods can be used to collect data on a particular subject. This can be done through surveys, interviews, observations, experiments, and more.
To evaluate programs : Research methods can be used to evaluate the effectiveness of a program, intervention, or policy. This can help in determining whether the program is meeting its goals and objectives.
To explore new areas : Research methods can be used to explore new areas of inquiry or to test new hypotheses. This can help in advancing knowledge in a particular field.
To make informed decisions : Research methods can be used to gather information and data to support informed decision-making. This can be useful in various fields such as healthcare, business, and education.

Advantages of Research Methods

Research methods provide several advantages, including:

Objectivity : Research methods enable researchers to gather data in a systematic and objective manner, minimizing personal biases and subjectivity. This leads to more reliable and valid results.
Replicability : A key advantage of research methods is that they allow for replication of studies by other researchers. This helps to confirm the validity of the findings and ensures that the results are not specific to the particular research team.
Generalizability : Research methods enable researchers to gather data from a representative sample of the population, allowing for generalizability of the findings to a larger population. This increases the external validity of the research.
Precision : Research methods enable researchers to gather data using standardized procedures, ensuring that the data is accurate and precise. This allows researchers to make accurate predictions and draw meaningful conclusions.
Efficiency : Research methods enable researchers to gather data efficiently, saving time and resources. This is especially important when studying large populations or complex phenomena.
Innovation : Research methods enable researchers to develop new techniques and tools for data collection and analysis, leading to innovation and advancement in the field.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

Assignment – Types, Examples and Writing Guide

References in Research – Types, Examples and...

Figures in Research Paper – Examples and Guide

Delimitations in Research – Types, Examples and...

Qualitative Research – Methods, Analysis Types...

Research Findings – Types Examples and Writing...

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

View all journals
My Account Login
Explore content
About the journal
Publish with us
Sign up for alerts
Data Descriptor
Open access
Published: 01 June 2023

SciSciNet: A large-scale open data lake for the science of science research

Zihang Lin ORCID: orcid.org/0000-0003-4262-6354 1 , 2 , 3 , 4 ,
Yian Yin ORCID: orcid.org/0000-0003-3018-4544 1 , 2 , 3 , 5 ,
Lu Liu 1 , 2 , 3 &
Dashun Wang ORCID: orcid.org/0000-0002-7054-2206 1 , 2 , 3 , 5

Scientific Data volume 10 , Article number: 315 ( 2023 ) Cite this article

8849 Accesses

2 Citations

71 Altmetric

Metrics details

Scientific community

The science of science has attracted growing research interests, partly due to the increasing availability of large-scale datasets capturing the innerworkings of science. These datasets, and the numerous linkages among them, enable researchers to ask a range of fascinating questions about how science works and where innovation occurs. Yet as datasets grow, it becomes increasingly difficult to track available sources and linkages across datasets. Here we present SciSciNet, a large-scale open data lake for the science of science research, covering over 134M scientific publications and millions of external linkages to funding and public uses. We offer detailed documentation of pre-processing steps and analytical choices in constructing the data lake. We further supplement the data lake by computing frequently used measures in the literature, illustrating how researchers may contribute collectively to enriching the data lake. Overall, this data lake serves as an initial but useful resource for the field, by lowering the barrier to entry, reducing duplication of efforts in data processing and measurements, improving the robustness and replicability of empirical claims, and broadening the diversity and representation of ideas in the field.

Background & Summary

Modern databases capturing the innerworkings of science have been growing exponentially over the past decades, offering new opportunities to study scientific production and use at larger scales and finer resolution than previously possible. Fuelled in part by the increasing availability of large-scale datasets, the science of science community turns scientific methods on science itself 1 , 2 , 3 , 4 , 5 , 6 , helping us understand in a quantitative fashion a range of important questions that are central to scientific progress—and of great interest to scientists themselves—from the evolution of individual scientific careers 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 to collaborations 19 , 20 , 21 , 22 , 23 , 24 , 25 and science institutions 26 , 27 , 28 to the evolution of science 2 , 3 , 5 , 29 , 30 , 31 , 32 , 33 , 34 to the nature of scientific progress and impact 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , – 55 .

Scholarly big data have flourished over the past decade, with several large-scale initiatives providing researchers free access to data. For example, CiteSeerX 56 , one of the earliest digital library search engines, offers a large-scale scientific library focusing on the literature in computer and information science. Building on a series of advanced data mining techniques, AMiner 57 indexes and integrates a wide range of data about academic social networks 58 . Crossref ( https://www.crossref.org/ ) 59 , as well as other initiatives in the open metadata community, have collected metadata such as Digital Object Identifier (DOI) in each publication record and linked them to a broad body of event data covering scholarly discussions. OpenAlex ( https://openalex.org/ ) 60 , based on Microsoft Academic Graph (MAG) 61 , 62 , 63 , aims to build a large-scale open catalog for the global research system, incorporating scholarly entities and their connections across multiple datasets. In addition to data on scientific publications and citations capturing within-science dynamics, researchers have also tracked interactions between science and other socioeconomic spheres by tracing, for example, how science is referenced in patented inventions 64 , 65 , 66 , regarding both front-page and in-text citations from patents to publications 67 , 68 . Table 1 summarizes several exemplary datasets commonly used in the science of science literature, with information on their coverage and accessibility.

The rapid growth of the science of science community 69 , 70 , 71 , combined with its interdisciplinary nature, raises several key challenges confronting researchers in the field. First, it becomes increasingly difficult to keep track of available datasets and their potential linkages across disparate sources, raising the question of whether there are research questions that are underexplored simply due to a lack of awareness of the data. Second, as data and their linkages become more complex, there are substantial data pre-processing steps involved prior to analyses. Many of these steps are often too detailed to document in publications, with researchers making their own analytical choices when processing the data. Third, as tools and techniques used in the science of science grow in sophistication, measurements on these datasets can be computationally involved, requiring substantial investment of time and resources to compute these measures.

All these challenges highlight the need for a common data resource designed for research purposes, which could benefit the community in several important ways. First, it provides a large-scale empirical basis for research, helping to strengthen the level of evidence supporting new findings as well as increase the replicability and robustness of these findings. Second, it helps to reduce duplication of efforts across the community in data preprocessing and common measurements. Third, by compiling various datasets, linkages, and measurements, the data resource significantly lowers the barrier to entry, hence has the potential to broaden the diversity and representation of new ideas in the field.

To support these needs in the community, we present SciSciNet, a large-scale open data lake for the science of science research. The data lake not only incorporates databases that capture scientific publications, researchers, and institutions, but also tracks their linkages to related entities, ranging from upstream funding sources like NIH and NSF to downstream public uses, including references of scientific publications in patents, clinical trials, and media and social media mentions (see Fig. 1 and Table 2 for more details of entities and their relationships). Building on this collection of linked databases, we further calculate a series of commonly used measurements in the science of science, providing benchmark measures to facilitate further investigations while illustrating how researchers can further contribute collectively to the data lake. Finally, we validate the data lake using multiple approaches, including internal data validation, cross-database verification, as well as reproducing canonical results in the literature.

The entity relationship diagram of SciSciNet. SciSciNet includes “SciSciNet_Papers” as the main data table, with linkages to other tables capturing data from a range of sources. For clarity, here we show a subset of the tables (see Data Records section for a more comprehensive view of the tables). PK represents primary key, and FK represents foreign key.

The data lake, SciSciNet, is freely available at Figshare 72 . At the core of the data lake is the Microsoft Academic Graph (MAG) dataset 61 , 62 , 63 . The MAG data is one of the largest and most comprehensive bibliometrics data in the world, and a popular dataset for the science of science research. However, MAG was sunset by Microsoft at the end of 2021. Since then, there have also been several important efforts in the community to ensure the continuity of data and services. For example, there are mirror datasets 73 available online for MAG, and the OpenAlex ( https://openalex.org ) initiative builds on the MAG data, and not only makes it open to all but also provides continuous updates 60 . While these efforts have minimized potential disruptions, the sunsetting of MAG has also accelerated the need to construct open data resources designed for research purposes. Indeed, large-scale systematic datasets for the science of science mostly come in the form of raw data, which requires further data pre-processing and filtering operations to extract fine-grained research data with high quality. It usually takes substantial efforts and expertise to clean the data, and many of these steps are often too detailed to document in publications, with researchers making their own analytical choices. It thus suggests that there is value in constructing an open data lake, which aims to continue to extend the usefulness of MAG, with substantial data pre-processing steps documented. Moreover, the data lake links together several disparate sources and pre-computed measures commonly used in the literature, serving as an open data resource for researchers interested in the quantitative studies of science and innovation.

Importantly, the curated data lake is not meant to be exhaustive; rather it represents an initial step toward a common data resource to which researchers across the community can collectively contribute. Indeed, as more data and measurements in the science of science become available, researchers can help to contribute to the continuous improvement of this data lake by adding new data, measurements, and linkages, thereby further increasing the utility of the data lake. For example, if a new paper reports a new measurement, the authors could publish a data file linking the new measurement with SciSciNet IDs, which would make it much easier for future researchers to build on their work.

Data selection and curation from MAG

The Microsoft Academic Graph (MAG) dataset 61 , 62 , 63 covers a wide range of publication records, authors, institutions, and citation records among publications. MAG has a rich set of prominent features, including the application of advanced machine learning algorithms to classify fields of study in large-scale publication records, identify paper families, and disambiguate authors and affiliations. Here we use the edition released on December 6 th , 2021 by MAG, in total covering 270,694,050 publication records.

The extensive nature of the MAG data highlights a common challenge. Indeed, using the raw data for research often requires substantial pre-processing and data-cleaning steps to arrive at a research-ready database. For example, one may need to perform a series of data selection and curation operations, including the selection of scientific publications with reliable sources, aggregation of family papers, and redistribution of citation and reference counts. After going through these steps, one may generate a curated publication data table, which serves as the primary scientific publication data table in SciSciNet (Table 3 , “SciSciNet_Papers”). However, each of these steps requires us to make specific analytical choices, but given the detailed nature of these steps, the specific choices made through these steps have remained difficult to document through research publications.

Here we document in detail the various procedures we took in constructing the data lake. From the original publication data in MAG, we use MAG Paper ID as the primary key, and consider a subset of main attributes, including DOI (Digital Object Identifier), document type and publication year. As we are mainly interested in scientific publications within MAG, we first remove paper records whose document type is marked as patent. We also remove those with neither document type nor DOI information. Each scientific publication in the database may be represented by different entities (e.g., preprint and conference), indicated as a paper “family” in MAG. To avoid duplication, we aggregate all papers in the same family into one primary paper. We also do not include retracted papers in the primary paper table in SciSciNet. Instead, we include records of retracted papers and affiliated papers in paper families in another data table “SciSciNet_PaperDetails” (Table 8 ) linked to the primary paper table, recording information of DOIs, titles, original venue names, and original counts for citations and references in MAG. Following these steps, the primary data table “SciSciNet_Papers” contains 134,129,188 publication records with unique primary paper ids, including 90,764,813 journal papers, 4,629,342 books, 3,932,366 book chapters, 5,123,597 conference papers, 145,594 datasets, 3,083,949 repositories, 5,998,509 thesis papers, and 20,451,018 other papers with DOI information.

For consistency, we recalculate the citation and reference counts within the subset of 134 M primary papers, such that each citation or reference record is also included in this subset and can be found in “SciSciNet_PaperReferences” (Table 5 ). For papers in the same family, we aggregate their citations and references into the primary paper and drop duplicated citation pairs. Building on the updated citations, we recalculate the number of references and citations for each primary paper.

MAG also contains information of authors, institutions, and fields. While author disambiguation 58 , 74 , 75 , 76 , 77 , 78 , 79 remains a major challenge, we adopt the author disambiguation method from MAG and create an author table, which offers a baseline for future studies of individual careers. We also supplement the author table with empirical name-gender associations to support gender research 80 , drawing from work by Van Buskirk et al . 80 ; this allows us to build “SciSciNet_Authors_Gender” (Table 9 ) with 134,197,162 author records including their full names.

For fields, we use the fields of study records from MAG and focus on the records related to the selected primary papers (19 Level-0 fields and 292 Level-1 fields, Table 6 ). We incorporate this information into two tables, the “SciSciNet_PaperAuthorAffiliations” (Table 4 ) and “SciSciNet_PaperFields” (Table 7 ), with 413,869,501 and 277,494,994 records, respectively.

We further use the information of “PaperExtendedAttributes” table from MAG to construct high-quality linkages between MAG Paper ID and PubMed Identifier (PMID). We drop duplicate links by only keeping the MAG primary paper record (if one PMID was linked to multiple MAG Paper IDs) or the latest updated PubMed record (if one MAG Paper ID was linked to multiple PMIDs), obtaining 31,230,206 primary MAG Paper ID-PMID linkages (95.6% of the original records) to further support linkage with external sources.

Together, the resulting SciSciNet includes 134,129,188 publications (Table 3 ), 134,197,162 authors (Table 9 ), 26,998 institutions (Table 10 ), 49,066 journals (Tables 21 ), 4,551 conference series (Tables 22 ), 19 top-level fields of study, 292 subfields (Table 6 ), and the internal links between them, including 1,588,739,703 paper-references records (Table 5 ), 413,869,501 paper-author-affiliations records (Table 4 ), and 277,494,994 paper-fields records (Table 7 ).

Linking publication data with external sources

While the main paper table captures citation relationships among scientific publications, there has been growing interest in studying how science interacts with other socioeconomic institutions 35 , 36 , 41 , 55 , 81 , 82 . Here, we further trace references of scientific publications in data sources that go beyond publication datasets, tracking the linkage between papers to their upstream funding supports and downstream uses in public domains. Specifically, here we link papers to the grants they acknowledge in NSF and NIH, as well as public uses of science by tracking references of scientific publications in patents, clinical trials, and news and social media.

NIH funding

The National Institutes of Health (NIH) is the largest public funder for biomedical research in the world. The recent decade has witnessed increasing interest in understanding the role of NIH funding for the advancement of biomedicine 81 , 82 and its impact on individual career development 83 , 84 . NIH ExPORTER provides bulk NIH RePORTER ( https://report.nih.gov/ ) data on research projects funded by the NIH and other major HHS operating divisions. The database also provides link tables (updated on May 16, 2021) that connects funded projects with resulting publications over the past four decades.

To construct the funded project-paper linkages between SciSciNet Paper ID and NIH Project Number, we use the PMID of MAG papers (from our previously curated “PaperExtendedAttributes” table based on MAG) as the intermediate key, matching more than 98.9% of the original NIH link table records to primary Paper ID in SciSciNet. After dropping duplicate records, we end up with a collection of 6,013,187 records (Table 11 ), linking 2,636,061 scientific papers (identified by primary MAG Paper IDs) to 379,014 NIH projects (identified by core NIH-funded project numbers).

NSF funding

Beyond biomedical research, the National Science Foundation (NSF) funds approximately 25% of all federally supported basic research conducted by the United States’ colleges and universities across virtually all fields of science and engineering. NSF provides downloadable information on research projects it has funded, including awardee, total award amount, investigator, and so forth, but no information on funded research publications. While Federal RePORTER offers downloadable files on NSF awards with links to supported publications (662,072 NSF award-publication records by 2019), it only covers a limited time period and has been retired by March 2022. To obtain a more comprehensive coverage of records linking NSF awards to supported papers, we crawl the webpages of all NSF awards to retrieve information on their resulting publications. In particular, we first created a comprehensive list of all NSF award numbers from https://www.nsf.gov/awardsearch/download.jsp . We then iterate over this list to download the entire webpage document of each NSF award (from the URL https://www.nsf.gov/awardsearch/showAward?AWD_ID = [Award number]), and use “Publications as a result of this research” column to identify scientific publications related to this award. We then extract paper titles and relevant information provided by using the Python library ElementTree to navigate and parse the webpage document structurally. We end up collecting 489,446 NSF awards since 1959 (Table 20 ), including linkages between 131,545 awards and 1,350,915 scientific publications.

To process information crawled from NSF.gov, which is presented as raw text strings, we design a text-based multi-level matching process to link NSF awards to SciSciNet scientific publications:

For records with DOI information in the raw texts of funded research publications, we perform an exact match with SciSciNet primary papers through DOI. If the DOI in an NSF publication record matched that of one primary paper, we create a linkage between the NSF Award Number and the primary Paper ID. We matched 458,463 records from NSF awards to SciSciNet primary papers, where each DOI appeared only once in the entire primary paper table, thus enabling association with a unique Paper ID (exact match). After dropping duplicates where the same DOI appears repeatedly in the same NSF award, we yield 350,611 records (26.0%) from NSF awards to SciSciNet primary papers.

To process the rest of the records, we then use the title information of each article for further matching. After extracting the title from NSF records and performing a standardization procedure (e.g., converting each letter into lowercase and removing punctuation marks, extra spaces, tabs, and newline characters), our exact matches between paper titles in the NSF award data and SciSciNet primary paper data yield 246,701 unique matches (18.3% in total) in this step.

We further develop a search engine for records that have not been matched in the preceding steps. Here we use Elasticsearch, a free and open search and analytics engine, to index detailed information (paper title, author, journal or conference name, and publication year) of all SciSciNet primary papers. We then feed raw texts of the crawled NSF publications into the system and obtain results with the top two highest scores associated with the indexed primary papers. Similar to a previous study 55 , we use scores of the second matched primary papers as a null model, and then identify the first matched primary paper as a match if its score is significantly higher than the right-tail cutoff of the second score distribution ( P = 0.05). Following this procedure, we match the remaining 467,159 records (34.6%) from the two previous steps with significantly higher scores (Fig. 2a ). Note that this procedure likely represents a conservative strategy that prioritizes precision over recall. Manually inspecting the rest of potential matchings, we find that those with large differences between the top two Z-scores (Fig. 2b ) are also likely to be correct matches. To this end, we also include these heuristic links, together with the difference of their Z-scores, as fuzzy matching linkages between SciSciNet papers and NSF awards.

Matching NSF reference string to MAG records. ( a ) Distribution of Z-scores for papers matched in ElasticSearch with the first and second highest scores. The vertical red line denotes the right-tail cutoff of the second score distribution ( P = 0.05). ( b ) Distribution of pairwise Z-score differences for papers matched in search engine but with the first score no higher than the right-tail cutoff of the second score distribution ( P = 0.05).

We further supplement these matchings with information from Crossref data dump, an independent dataset that links publications to over 30 K funders including NSF. We collect all paper-grant pairs where the funder is identified as NSF. We then use the raw grant number from Crossref and link paper records between Crossref and SciSciNet using DOIs. We obtain 305,314 records after cleaning, including 196,509 SciSciNet primary papers with DOIs matching to 83,162 NSF awards.

By combining records collected from all these steps, we collect 1,130,641 unique linkages with high confidence levels and 178,877 additional possible linkages from fuzzy matches (Table 12 ). Together these links connect 148,148 NSF awards and 929,258 SciSciNet primary papers.

Patent citations to science

The process in which knowledge transfers from science to marketplace applications has received much attention in science and innovation literature 35 , 41 , 85 , 86 , 87 , 88 . The United States Patent and Trademark Office (USPTO) makes patenting activity data publicly accessible, with the PatentsView platform providing extensive metadata including as related to patent assignees, inventors, and lawyers, along with patents’ internal citations and full-text information. The European Patent Office (EPO) also provides open access to patent data containing rich attributes.

Building on recent advances in linking papers to patents 35 , 67 , 68 , Marx and Fuegi developed a large-scale dataset of over 40 M citations from USPTO and EPO patents to scientific publications in MAG. Using this corpus (Version v34 as of December 24, 2021), we merge 392 K patent citation received by affiliated MAG papers to their respective primary IDs in the same paper family. Dropping possible duplicate records with the same pair of primary Paper ID and Patent ID results in 38,740,313 paper-patent citation pairs between 2,360,587 patents from USPTO and EPO and 4,627,035 primary papers in SciSciNet (Table 15 ).

Clinical trials citations to science

Understanding bench-to-bed-side translation is essential for biomedical research 81 , 89 . ClinicalTrials.gov provides publicly available clinical study records covering 50 U.S. states and 220 countries, sourced from the U.S. National Library of Medicine. The Clinical Trials Transformation Initiative (CTTI) makes available clinical trials data through a database for Aggregate Analysis of ClinicalTrials.gov (AACT), an aggregated relational database helping researchers better study drugs, policies, publications, and other related items to clinical trials.

Overall, the data covers 686,524 records linking clinical trials to background or result papers (as of January 26th, 2022). We select 480,893 records with papers as reference background supporting clinical trials, of which 451,357 records contain 63,281 unique trials matching to 345,797 reference papers with PMIDs. Similar to the process of linking scientific publications to NIH-funded projects, we again establish linkages between SciSciNet primary Paper ID and NCT Number (National Clinical Trial Number) via PMID, aided by the curated “PaperExtendedAttributes” table as the intermediary. After standardizing the data format of the intermediate index PMID to merge publications and clinical trials, we obtain 438,220 paper-clinical linkages between 61,447 NCT clinical trials and 337,430 SciSciNet primary papers (Table 13 ).

News and social mentions of science

Understanding how science is mentioned in media has been another important research direction in the science of science community 44 , 90 . The Newsfeed mentions in Crossref Event Data link scientific papers in Crossref 59 with DOIs to news articles or blog posts in RSS and Atom feeds, providing access to the latest scientific news mentions from multiple sources, including Scientific American , The Guardian , Vox , The New York Times , and others. Also, Twitter mentions in Crossref Event Data link scientific papers to tweets created by Twitter users, offering an opportunity to explore scientific mentions in Twitter.

We use the Crossref Event API to collect 947,160 records between 325,396 scientific publications and 387,578 webpages from news blogs or posts (from April 5 th , 2017 to January 16 th , 2022) and 59,593,281 records between 4,661,465 scientific publications and 58,099,519 tweets (from February 7 th , 2017 to January 17 th , 2022).

For both news media and social media mentions, we further link Crossref’s publication records to SciSciNet’s primary papers. To do so, we first normalize the DOI format of these data records and converted all alphabetic characters to lowercase. We use normalized DOI as the intermediate index, as detailed below:

For news media mentions, we construct linkages between primary Paper ID and Newsfeed Object ID (i.e., the webpage of news articles or blog posts) by inner joining normalized DOIs. We successfully link 899,323 records from scientific publications to news webpages in the Newsfeed list, accounting for 94.9% of the total records. The same news mention may be collected multiple times. After removing duplicate records, we end up with 595,241 records, linking 307,959 papers to 370,065 webpages from Newsfeed (Table 17 ).

Similarly, for social media mentions, we connect primary Paper IDs with Tweet IDs through inner joining normalized DOIs, yielding 56,121,135 records, more than 94% of the total records. After dropping duplicate records, we keep 55,846,550 records, linking 4,329,443 papers to 53,053,505 tweets (Table 16 ).

We also provide metadata of paper-news linkages, including the mention time and the detailed mention information in Newsfeed, to better support future research on this topic (Table 18 ). Similarly, we also offer the metadata of paper-tweet links, including the mention time and the original collected Tweet ID so that interested researchers can merge with further information from Twitter using the Tweet ID (Table 19 ).

Nobel Prize data from the dataset of publication records for Nobel laureates

We integrate a recent dataset by Li et al . 91 in the data lake, containing the publication records of Nobel laureates in science from 1900 to 2016, including both Nobel prize-winning works and other papers produced in their careers. After mapping affiliated MAG Paper IDs to primary ones, we obtain 87,316 publication records of Nobel laureates in SciSciNet primary paper Table (20,434 in physics, 38,133 in chemistry, and 28,749 in physiology/medicine, Table 14 ).

Calculation of commonly used measurements

Using the constructed dataset, we further calculate a range of commonly used measurements of scientific ideas, impacts, careers, and collaborations. Interested readers can find more details and validations of these measurements in the literature 15 , 19 , 20 , 46 , 47 , 48 , 92 , 93 , 94 , 95 , 96 , 97 , 98 .

Publication-level

The number of researchers and institutions in a scientific paper.

Building on team science literature 19 , 27 , we calculate the number of authors and the number of institutions for each paper as recorded in our data lake. We group papers by primary Paper ID in the selected “SciSciNet_PaperAuthorAffiliations” table and aggregate the unique counts of Author IDs and Affiliation IDs as the number of researchers (team size) and institutions, respectively.

Five-year citations ( c 5 ), ten-year citations ( c 10 ), normalized citation ( c f ), and hit paper

The number of citations of a paper evolves over time 46 , 48 , 99 , 100 . Here we calculate c 5 and c 10 , defined as the number of citations a paper received within 5 years and 10 years of publication, respectively. For the primary papers, we calculate c 5 for all papers published up to 2016 (As the last version of MAG publication data is available until 2021) by counting the number of citation pairs with time difference less than or equal to 5 years. Similarly, we calculate c 10 for all papers published up to 2011.

To compare citation counts across disciplines and time, Radicchi et al . 48 proposed the relative citation indicator c f , as the total number of citations c divided by the average number of citations c 0 in the same field and the same year. Here we calculate the normalized citation indicator for each categorized paper in both top-level fields and subfields, known as Level-0 fields (19 in total) and Level-1 fields (292 in total) categorized by MAG, respectively. Note that each paper may be associated with multiple fields, hence here we report calculated normalized citations for each paper-field pair in the “SciSciNet_PaperFields” data table.

Another citation-based measure widely used in the science of science literature 16 , 19 , 83 is “hit papers”, defined as papers in the top 5% of citations within the same field and year. Similar to our calculation of c f , we use the same grouping by fields and years, and identify all papers with citations greater than the top 5% citation threshold. We also perform similar operations for the top 1% and top 10% hit papers.

Citation dynamics

A model developed by Wang, Song, and Barabási (the WSB model) 46 captures the long-term citation dynamics of individual papers after incorporating three fundamental mechanisms, including preferential attachment, aging, and fitness. The model predicts the cumulative citations received by paper i at time t after publication: \({c}_{i}^{t}=m\left[{e}^{{{\rm{\lambda }}}_{i}\Phi \left(\frac{lnt-{{\rm{\mu }}}_{i}}{{{\rm{\sigma }}}_{i}}\right)}-1\right]\) , where Φ ( x ) is the standard cumulative normal distribution of x , m captures the average number of references per paper, and μ i , σ i , and λ i indicate the immediacy, longevity, and fitness parameters characterizing paper i , respectively.

We implement the WSB model with prior for papers published in the fields of math and physics. Following the method proposed by Shen et al . 92 , we adopt the Bayesian approach to calculate the conjugate prior, which follows a gamma distribution. The method allows us to better predict the long-term impact through the posterior estimation of λ i , while helping to avoid potential overfitting problems. Fitting this model to empirical data, we compute the immediacy μ i , the longevity σ i , and the ultimate impact \({c}_{{\rm{i}}}^{\infty }={\rm{m}}\left[{e}^{{{\rm{\lambda }}}_{i}}-1\right]\) for all math and physics papers with at least 10 citations within 10 years after publication (published no later than 2011). To facilitate research on citation dynamics across different fields 48 , we have also used the same procedure to fit the citation sequences for papers that have received at least 10 citations within 10 years across all fields of study from the 1960s to the 1990s.

Sleeping beauty coefficient

Sometimes it may take years or even decades for papers to gain attention from the scientific community, a phenomenon known as the “Sleeping Beauty” in science 93 . The sleeping beauty coefficient B is defined as \({\rm{B}}={\sum }_{t=0}^{{t}_{m}}\frac{\frac{{c}_{{t}_{m}}-{c}_{0}}{{t}_{m}}\cdot t+{c}_{0}-{c}_{t}}{{\rm{\max }}\left(1,{c}_{t}\right)}\) , where the paper receives its maximum yearly citation \({c}_{{t}_{m}}\) in year t m and c 0 in the year of publication. Here we calculate the sleeping beauty coefficient from yearly citation records of a paper. We match the publication years for each citing-cited paper pair published in journals and then aggregate yearly citations since publication for each cited paper. Next, we group the “SciSciNet_PaperReferences” table by each cited paper and compute the coefficient B , along with the awakening time. As a result, we obtain 52,699,363 records with sleeping beauty coefficients for journal articles with at least one citation.

Novelty and conventionality

Research shows that the highest-impact papers in science tend to be grounded in exceptionally conventional combinations of prior work yet simultaneously feature an intrusion of atypical combinations 47 . Here following this work 47 , we calculate the novelty and conventionality score of each paper by computing the Z-score for each combination of journal pairs. We further calculate the distribution of journal pair Z-scores by traversing all possible duos of references cited by a particular paper. A paper’s median Z-score characterizes the median conventionality of the paper, whereas a paper’s 10 th percentile Z-score captures the tail novelty of the paper’s atypical combinations.

More specifically, we first use the information of publication years for each citing-cited paper pair both published in journals and shuffle the reference records within the citing-cited year group to generate 10 randomized citation networks, while controlling the naturally skewed citation distributions. We then traverse each focal paper published in the same year. We further aggregate the frequency of reference journal pairs for papers in the real citation network and 10 randomized citation networks, calculating the Z-score of each reference journal pair for papers published in the same year. Finally, for each focal paper, we obtain its 10 th percentile and median of the Z-scores distribution, yielding 44,143,650 publication records with novelty and conventionality measures for journal papers from 1950 to 2021.

Disruption score

Disruption index quantifies the extent to which a paper disrupts or develops the existing literature 20 , 51 . Disruption, or D , is calculated through citation networks. For a given paper, one can separate its future citations into two types. One type only cites the focal paper itself while ignoring all the references that the paper builds upon, and the other is to cite both the focal paper and its references. D is expressed as: \({\rm{D}}={{\rm{p}}}_{{\rm{i}}}-{{\rm{p}}}_{{\rm{j}}}=\frac{{n}_{i}-{n}_{j}}{{n}_{i}+{n}_{j}+{n}_{k}}\) , where n i is the number of subsequent works that only cite the focal paper, n j is the number of subsequent works that cite both the focal paper and its references, and n k is the number of subsequent works that cite the references of the focal paper only. Following this definition, we calculate the disruption scores for all the papers that have at least one forward and backward citation (48,581,274 in total).

The number of NSF and NIH supporting grants

For external linkages from scientific publications to upstream supporting funding sources, we calculate the number of NSF/NIH grants associated with each primary paper in SciSciNet.

The number of patent citations, Newsfeed mentions, Twitter mentions, and clinical trial citations

For external linkages from scientific publications to downstream public uses of science, we also calculate the number of citations each primary paper in SciSciNet received from domains that go beyond science, including patents from USPTO and EPO, news and social media mentions from Newsfeed and Twitter, and clinical trials from ClinicalTrials.gov.

Individual- and Institutional-level measures

Productivity.

Scientific productivity is a widely used measure for quantifying individual careers 9 , 15 . Here we aggregate the unique primary Paper ID in SciSciNet, after grouping the records in the “SciSciNet_PaperAuthorAffiliations” data table by Author ID or Affiliation ID and calculate the number of publications produced by the same author or affiliation.

H-index is a popular metric to estimate a researcher’s career impact. The index of a scientist is h , if h of her papers have at least h citations and each of the remaining papers have less than h citations 94 , 101 . Here we compile the full publication list associated with each author, sort these papers by their total number of citations in descending order, and calculate the maximum value that satisfies the condition above as the H-index. By repeating the same procedure on each research institution, we also provide an institution-level H-index as well.

Scientific impact

Building on our c 10 measure at the paper level, here we further calculate the average c 10 (< c 10 >) for each author and affiliation, which offers a proxy to individual and institutional level scientific impact. Similarly, we calculate the average log c 10 (<log c 10 >), which is closely related to the Q parameter 15 of individual scientific impact.

Here we group by Author and Affiliation ID in the “PaperAuthorAffiliations” table, and then aggregate c 10 and log c 10 (pre-calculated at the paper level) of all papers published by the same id. Following previous works 15 , 16 , 102 , to avoid taking logarithm of zeros, we increase c 10 by one when calculating the <log c 10 >.

Name-gender associations

The availability of big data also enables a range of studies focusing on gender disparities, ranging from scientific publications and careers 17 , 103 , 104 , 105 , 106 to collaboration patterns 25 , 107 and the effects of the pandemic on women scientists 45 , 108 , 109 , 110 . Here we apply the method from a recent statistical model 80 to infer author gender based on their first names in the original author table. The method feeds unique author names into a cultural consensus model of name-gender associations incorporating 36 separate sources across over 150 countries. Note that for all the 134,197,162 authors, 23.26% of the authors (31,224,458) have only the first initials, which are excluded from the inference. By fine-tuning the annotated names from these data sources following the original method, we obtain 409,809 unique names with max uncertainty threshold set to 0.26 and 85% of the sample classified. Finally, we merge these name-gender inference records into the original SciSciNet_Authors table, resulting a SciSciNet_Authors_Gender table, which contains 86,286,037 authors with inferred probability that indicates a name belongs to an individual gendered female, denoted as P(gf), as well as the number of inference source datasets and empirical counts. Together, by combining new statistical models with our systematic authorship information, this new table provides name-gender information, useful in studying gender-related questions. It is important to note that such name-based gender inference algorithms, including the one used here as well as other popular tools such as genderize.io , have limitations and are necessarily imperfect. The limitations should be considered carefully when applying these methods 96 .

Data Records

The data lake, SciSciNet, is freely available at Figshare 72 .

Data structure

Table 2 presents the size and descriptions of these data files.

Table 3 contains information about “SciSciNet_Papers”, which is the data lake’s primary paper table, containing information on the primary scientific publications, including Paper ID, DOI, and others, along with the Journal ID or Conference Series ID, which can link papers to corresponding journals or conference series that take place regularly. The short description in each data field includes the corresponding explanation of that field.

Tables 4 – 22 include the data fields and corresponding descriptions of each data table. Each data field specified is clear from its index name. An ID of the data field in a data table can be linked, if this field has the same ID name as another field in another table. Further, the data link tables provide linkages from scientific publications to external socioeconomic institutions. For example, the paper with primary “PaperID” as “246319838”, which studied the hereditary spastic paraplegia 111 , lead to three core NIH project number “R01NS033645”, “R01NS036177”, and “R01NS038713” in the Table 11 “SciSciNet_Link_NIH”. We can not only extract detailed information and metrics of the paper in the data lake (e.g., title from Table 8 “SciSciNet_PaperDetails”, or citation counts from the primary paper Table 3 “SciSciNet_Papers”) but also obtain further information of the funded-projects, such as the total funding amount, from NIH RePORTER ( https://report.nih.gov ).

Descriptive statistics

Next, we present a set of descriptive statistics derived from the data lake. Figure 3a–c show the distribution of papers across 19 top-level fields, the exponential growth of scientific publications in SciSciNet over time, and the average team size of papers by field over time.

Summary statistics of scientific publications in SciSciNet. ( a ) The number of publications in 19 top-level fields. For clarity we aggregated the field classification into the top level (e.g., a paper is counted as a physics paper if it is associated with physics or any other subfields of physics). ( b ) The exponential growth of science over time. ( c ) Average team size by field from 1950 to 2020. The bold black line is for papers in all the 19 top-level fields. Each colored line indicates each of the 19 fields (color coded according to (a)).

Building on the external linkages we constructed, Fig. 4a–f show the distribution of paper-level upstream funding sources from NIH and NSF, and downstream applications and mentions of science, including USPTO/EPO patents, clinical trials, news mentions from Newsfeed, and social media mentions from Twitter.

Linking scientific publications with socioeconomic institutions. Panels ( a, b and d, e ) show the distribution of paper-level downstream applications ( a : Twitter mentions; b : Newsfeed mentions; d : Patents; e : Clinical trials). Panels ( c and f ) show the distribution of supporting scientific grants from NIH ( c ) and NSF ( f ).

Figure 5 presents the probability distributions of various commonly used metrics in the science of science using our data lake, which are broadly consistent with the original studies in the literature.

Commonly used metrics in SciSciNet. ( a ) The distribution of disruption score for 48,581,274 papers 20 (50,000 bins in total). ( b ) Cumulative distribution function (CDF) of 44,143,650 journal papers’ 10 th percentile and median Z-scores 47 . ( c ) Distribution of \({e}^{{\rm{\langle }}log{c}_{\mathrm{10}}{\rm{\rangle }}}\) for scholars 15 with at least 10 publications in SciSciNet. The red line corresponds to a log-normal fit with μ = 2.14 and σ = 1.14. ( d ) Survival distribution function of sleeping beauty coefficients 93 for 52,699,363 papers, with a power-law fit: exponent α = 2.40. ( e ) Data collapse for a selected subset of papers with more than 30 citations within 30 years across journals in physics in the 1960s, based on WSB model 46 . The red line corresponds to the cumulative distribution function of the standard normal distribution.

Technical Validation

Validation of publication and citation records.

As we select the primary papers from the original MAG dataset, we have re-counted the citations and references within the subset of primary papers. To test the reliability of updated citation and reference counts in SciSciNet, here we compare the two versions (i.e., raw MAG counts and redistributed SciSciNet counts), by calculating the Spearman correlation coefficients for both citations and references. The Spearman correlation coefficients are 0.991 for citations and 0.994 for references, indicating that these metrics are highly correlated before and after the redistribution process.

We also examine the coverage of our publication data through a cross-validation with an external dataset, Dimensions 112 . By using DOI as a standardized identifier, we find that the two databases contain a similar number of papers, with 106,517,016 papers in Dimensions and 98,795,857 papers in SciSciNet associated with unique DOIs. We further compare the overlap of the two databases, finding the two data sources share a vast majority of papers in common (84,936,278 papers with common DOIs, accounting for 79.74% of Dimensions and 85.97% of SciSciNet).

Further, the citation information recorded by the two datasets appears highly consistent. Within the 84.9 M papers we matched with common DOIs, SciSciNet records a similar, yet slightly higher number of citations on average (16.75), compared with Dimensions (14.64). Our comparison also reveals a high degree of consistency in paper-level citation counts between the two independent corpora, with a Spearman correlation coefficient 0.946 and a concordance coefficient 98 , 113 of 0.940. Together, these validations provide further support for the coverage of the data lake.

Validation of external data linkages

We further perform additional cross-validation to understand the reliability of data linkages from scientific publications to external data sources. Here we focus more on the NSF-SciSciNet publications linkages we created from raw data collection to final data linkage. We also use the same approach to validate the NIH-SciSciNet publications linkages.

Here we compare the distribution and coverage of paper-grants linkages between SciSciNet and Dimensions—one of the state-of-the-art commercial databases in publication-grant linkages 112 . Figure 6a,b present the distribution of the number of papers matched to each NSF award and NIH grant, showing that our open-source approach offers a comparable degree of coverage. We further perform individual grant level analysis, by comparing the number of papers matched to each grant reported by the two sources (Fig. 6c,d ), again finding high degrees of consistency (Spearman correlation coefficient: 0.973 for NIH grants and 0.714 for NSF grants).

Validation of data linkages between SciSciNet and Dimensions. Panels ( a, b ), The distribution of number of papers matched to each NIH and NSF grant, respectively. Panels ( c, d ), The number of papers matched to each NIH and NSF grant, respectively. All panels are based on data in a 20-year period (2000–2020).

We further calculate the confusion matrices of linkage from SciSciNet and Dimensions. By connecting the two datasets through paper DOIs and NSF/NIH grant project numbers, we compare their overlaps and differences in grant-paper pairs. For NSF, the confusion matrix is shown in Table 23 . The two datasets provide a similar level of coverage, with Dimensions containing 670,770 pairs and SciSciNet containing 632,568 pairs. 78.9% pairs in Dimensions (and 83.7% pairs in SciSciNet) can be found in the other dataset, documenting a high degree of consistency between the two sources. While there are data links contained in Dimensions that are not in SciSciNet, we also find that there exists a similar amount of data records in SciSciNet but not in Dimensions. Table 24 shows the confusion matrix of NIH grant-paper pairs between the two datasets. Again, the two datasets share a vast majority of grant-paper pairs in common, and 95.3% pairs in Dimensions (and 99.7% pairs in SciSciNet) can also be found in the other dataset. These validations further support the overall quality and coverage of data linkages in SciSciNet.

Validation of calculations of commonly used measurements

We also seek to validate the calculated metrics included in SciSciNet. In addition to manual inspection of independent data samples during data processing, along with presenting the corresponding distributions of indicators in the Descriptive statistics section, which capture general patterns, we further double-check the calculation results of these popular measurements in SciSciNet by reproducing canonical results in the science of science under a series of standardized and transparent processes.

For disruption scores, we plot the median disruption percentile and average citations on different team sizes for 48,581,274 publications with at least one citation and reference record in SciSciNet. As shown in Fig. 7a , when team size increases, the disruption percentile decreases while the average citations increase, which is consistent with the empirical findings that small teams disrupt whereas large teams develop 20 . In addition, the probability of being among the top 5% disruptive publications is negatively correlated with the team size, while the probability of being among the most impactful publications increases is positively correlated with the team size (Fig. 7b ). These results demonstrate the consistency with results obtained in the literature.

Calculating commonly used measurements in the science of science literature. ( a, b ), Small teams disrupt while large teams develop in SciSciNet. ( c ), The cumulative distribution functions (CDFs) of proportion of external citations for papers with high (top 10,000, B > 307.55), medium (from 10,001 st to top 2% SBs, 33< B < = 307.55); and low (B < = 33) sleeping beauty indexes. ( d ), The probability of a 5% hit paper, conditional on novelty and conventionality for all journal articles in SciSciNet from 1950 to 2000.

The combinations of conventional wisdom and atypical knowledge tend to predict a higher citation impact 47 . Here we repeat the original analysis by categorizing papers based on (1) median conventionality: whether the median score of a paper is in the upper half and (2) tail novelty: whether the paper is within the top 10 th percentile of novelty score. We then identified hit papers (within the subset of our analysis), defined as papers rank in the top 5% of ten-year citations within the same top-level field and year. The four quadrants in Fig. 7d suggest that papers with high median conventionality and high tail novelty present a higher hit rate of 7.32%, within the selection of SciSciNet papers published from 1950 to 2000. Also, papers with high median conventionality but low tail novelty show a hit rate of 4.18%, roughly similar to the baseline rate of 5%, while those with low median conventionality but high tail novelty display a hit rate of 6.48%. Meanwhile, papers with both low median conventionality and low tail novelty exhibit a hit rate of 3.55%. These results are broadly consistent with the canonical results reported in 47 .

In Fig. 5e , we select 36,802 physics papers published in the 1960s with more than 30 citations within 30 years of publication. By rescaling their citation dynamics using the fitted parameters, we find a remarkable collapse of rescaled citation dynamics which appears robust across fields and decades. We further validate the predictive power of the model with prior based on Shen et al . 92 , by calculating the out-of-sample prediction accuracy. We find that with a training period of 15 years, the predictive accuracy (defined as a strict absolute tolerance threshold of 0.1) stays above 0.65 for 10 years after the training period, and the Mean Absolute Percentage Error (MAPE) is less than 0.1. The MAPE stays less than 0.15 for 20 years after the training period.

Sleeping beauty

We first fit the distribution of the sleeping beauty coefficients in SciSciNet (Fig. 5d ) to a power-law form using maximum likelihood estimation 114 , obtaining a power-law exponent α = 2.40 and minimum value B m = 23.59. By using fine-grained subfield information provided by MAG, we further calculate the proportion of external citations. Consistent with the original study 93 , we find that papers with high B scores are more likely to have a higher proportion of external citations from other fields (Fig. 7c ).

Usage Notes

Note that, recognizing the recent surge of interest in quantitative understanding of science 95 , 97 , 98 , 115 , 116 , the measurements currently covered in the data lake are not meant to be comprehensive; rather they serve as examples to illustrate how researchers from the broader community can collectively contribute and enrich the data lake. There are also limitations of the data lake that readers should keep in mind when using the data lake. For example, our grant-publication linkage is focused on scientific papers supported by NSF and NIH; patent-publication linkage is limited to citations from USPTO and EPO patents; clinical trial-publication linkage is derived from clinitrials.gov (where the geographical distribution may be heterogenous across countries, Table 25 ); and media-publication linkage is based on sources tracked by Crossref. Further, while our data linkages are based on state-of-the-art methods of data extraction and cleaning, as with any matching, the methods are necessarily imperfect and may be further improved through integration with complementary commercial products such as Altmetric and Dimensions. Finally, our data inherently represents a static snapshot, drawing primarily from the final edition of MAG (Dec 2021 version). While this snapshot is already sufficient in answering many of the research questions that arise in the field, future work may engage in continuous improvement and update of the data lake to maximize its potential.

Overall, this data lake serves as an initial step for serving the community in studying publications, funding, and broader impact. At the same time, there are also several promising directions for future work expanding the present effort. For example, the rapid development in natural language processing (NLP) models and techniques, accompanied by the increasing availability of text information from scientific articles, offers new opportunities to collect and curate more detailed content information. For example, one can link SciSciNet to other sources such as OpenAlex or Semantic Scholar to analyze large-scale data of abstract, full-text, or text-based embeddings. Such efforts will not only enrich the metadata associated with each paper, but also enable more precise identification and linkage of bio/chemical entities studied in these papers 117 . Further, although platforms like MAG have implemented advanced algorithms for name disambiguation and topic/field classification at scale, these algorithms are inherently imperfect and not necessarily consistent across datasets, hence it is essential to further validate and improve the accuracy of name disambiguation and topic classifications 118 . Related, in this paper we primarily focus on paper-level linkages across different datasets. Using these linkages as intermediary information, one can further construct and enrich individual-level profiles, allowing us to combine professional information (e.g., education background, grants, publications, and other broad impact) of researchers with important demographic dimensions (e.g., gender, age, race, and ethnicity). Finally, the data lake could contribute to an ecosystem for the collective community of the science of science. For example, there are synergies with the development of related programming packages, such as pySciSci 119 . By making the data lake fully open, we also hope it inspires other researchers to contribute to the data lake and enrich its coverage. For example, when a research team publishes a new measure, they could put out a data file that computes their measure based on SciSciNet, effectively adding a new column to the data lake. Lastly, science forms a complex social system and often offers an insightful lens to examine broader social science questions, suggesting that the SciSciNet may see greater utility by benefiting adjacent fields such as computational social science 120 , 121 , network science 122 , 123 , complex systems 124 , and more 125 .

Code availability

The source code for data selection and curation, data linkage, and metrics calculation is available at https://github.com/kellogg-cssi/SciSciNet .

Liu, L., Jones, B. F., Uzzi, B. & Wang, D. Measurement and Empirical Methods in the Science of Science. Nature Human Behaviour , https://doi.org/10.1038/s41562-023-01562-4 (2023).

Fortunato, S. et al . Science of science. Science 359 , eaao0185 (2018).

Article PubMed PubMed Central Google Scholar

Wang, D. & Barabási, A.-L. The science of science . (Cambridge University Press, 2021).

Zeng, A. et al . The science of science: From the perspective of complex systems. Physics reports 714 , 1–73 (2017).

Article ADS MathSciNet MATH Google Scholar

Azoulay, P. et al . Toward a more scientific science. Science 361 , 1194–1197 (2018).

Article ADS PubMed Google Scholar

Clauset, A., Larremore, D. B. & Sinatra, R. Data-driven predictions in the science of science. Science 355 , 477–480 (2017).

Article ADS CAS PubMed Google Scholar

Liu, L., Dehmamy, N., Chown, J., Giles, C. L. & Wang, D. Understanding the onset of hot streaks across artistic, cultural, and scientific careers. Nature communications 12 , 1–10 (2021).

ADS Google Scholar

Jones, B. F. The burden of knowledge and the “death of the renaissance man”: Is innovation getting harder? The Review of Economic Studies 76 , 283–317 (2009).

Article MATH Google Scholar

Way, S. F., Morgan, A. C., Clauset, A. & Larremore, D. B. The misleading narrative of the canonical faculty productivity trajectory. Proceedings of the National Academy of Sciences 114 , E9216–E9223, https://doi.org/10.1073/pnas.1702121114 (2017).

Article ADS CAS Google Scholar

Jones, B. F. & Weinberg, B. A. Age dynamics in scientific creativity. Proceedings of the National Academy of Sciences 108 , 18910–18914 (2011).

Malmgren, R. D., Ottino, J. M. & Amaral, L. A. N. The role of mentorship in protege performance. Nature 465 , 622–U117 (2010).

Article ADS CAS PubMed PubMed Central Google Scholar

Liénard, J. F., Achakulvisut, T., Acuna, D. E. & David, S. V. Intellectual synthesis in mentorship determines success in academic careers. Nature communications 9 , 1–13 (2018).

Article Google Scholar

Petersen, A. M. et al . Reputation and Impact in Academic Careers. Proceedings of the National Academy of Science USA 111 , 15316–15321 (2014).

Ma, Y., Mukherjee, S. & Uzzi, B. Mentorship and protégé success in STEM fields. Proceedings of the National Academy of Sciences 117 , 14077–14083 (2020).

Sinatra, R., Wang, D., Deville, P., Song, C. M. & Barabasi, A. L. Quantifying the evolution of individual scientific impact. Science 354 (2016).

Liu, L. et al . Hot streaks in artistic, cultural, and scientific careers. Nature 559 , 396–399 (2018).

Larivière, V., Ni, C., Gingras, Y., Cronin, B. & Sugimoto, C. R. Bibliometrics: Global gender disparities in science. Nature News 504 , 211 (2013).

Sugimoto, C. R. et al . Scientists have most impact when they’re free to move. Nature 550 , 29–31 (2017).

Wuchty, S., Jones, B. F. & Uzzi, B. The increasing dominance of teams in production of knowledge. Science 316 , 1036–1039 (2007).

Wu, L., Wang, D. & Evans, J. A. Large teams develop and small teams disrupt science and technology. Nature 566 , 378–382, https://doi.org/10.1038/s41586-019-0941-9 (2019).

Milojevic, S. Principles of scientific research team formation and evolution. Proceedings of the National Academy of Sciences 111 , 3984–3989 (2014).

Newman, M. E. The structure of scientific collaboration networks. Proceedings of the National Academy of Sciences 98 , 404–409 (2001).

Article ADS MathSciNet CAS MATH Google Scholar

AlShebli, B. K., Rahwan, T. & Woon, W. L. The preeminence of ethnic diversity in scientific collaboration. Nature communications 9 , 1–10 (2018).

Article CAS Google Scholar

Shen, H.-W. & Barabási, A.-L. Collective credit allocation in science. Proceedings of the National Academy of Sciences 111 , 12325–12330 (2014).

Leahey, E. From Sole Investigator to Team Scientist: Trends in the Practice and Study of Research Collaboration. Annual Review of Sociology, Vol 42 42 , 81–100 (2016).

Clauset, A., Arbesman, S. & Larremore, D. B. Systematic inequality and hierarchy in faculty hiring networks. Science advances 1 , e1400005 (2015).

Article ADS PubMed PubMed Central Google Scholar

Jones, B. F., Wuchty, S. & Uzzi, B. Multi-university research teams: shifting impact, geography, and stratification in science. science 322 , 1259–1262 (2008).

Deville, P. et al . Career on the move: Geography, stratification, and scientific impact. Scientific reports 4 (2014).

Chu, J. S. & Evans, J. A. Slowed canonical progress in large fields of science. Proceedings of the National Academy of Sciences 118 (2021).

Azoulay, P., Fons-Rosen, C. & Graff Zivin, J. S. Does science advance one funeral at a time? American Economic Review 109 , 2889–2920 (2019).

Article PubMed Google Scholar

Jin, C., Ma, Y. & Uzzi, B. Scientific prizes and the extraordinary growth of scientific topics. Nature communications 12 , 1–11 (2021).

Nagaraj, A., Shears, E. & de Vaan, M. Improving data access democratizes and diversifies science. Proceedings of the National Academy of Sciences 117 , 23490–23498 (2020).

Evans, J. A. & Reimer, J. Open access and global participation in science. Science 323 , 1025–1025 (2009).

Peng, H., Ke, Q., Budak, C., Romero, D. M. & Ahn, Y.-Y. Neural embeddings of scholarly periodicals reveal complex disciplinary organizations. Science Advances 7 , eabb9004 (2021).

Ahmadpoor, M. & Jones, B. F. The dual frontier: Patented inventions and prior scientific advance. Science 357 , 583–587 (2017).

Yin, Y., Gao, J., Jones, B. F. & Wang, D. Coevolution of policy and science during the pandemic. Science 371 , 128–130 (2021).

Ding, W. W., Murray, F. & Stuart, T. E. Gender differences in patenting in the academic life sciences. science 313 , 665–667 (2006).

CAS PubMed Google Scholar

Bromham, L., Dinnage, R. & Hua, X. Interdisciplinary research has consistently lower funding success. Nature 534 , 684 (2016).

Larivière, V., Vignola-Gagné, E., Villeneuve, C., Gélinas, P. & Gingras, Y. Sex differences in research funding, productivity and impact: an analysis of Québec university professors. Scientometrics 87 , 483–498 (2011).

Li, D., Azoulay, P. & Sampat, B. N. The applied value of public investments in biomedical research. Science 356 , 78–81 (2017).

Fleming, L., Greene, H., Li, G., Marx, M. & Yao, D. Government-funded research increasingly fuels innovation. Science 364 , 1139–1141, https://doi.org/10.1126/science.aaw2373 (2019).

Lazer, D. M. et al . The science of fake news. Science 359 , 1094–1096 (2018).

Scheufele, D. A. & Krause, N. M. Science audiences, misinformation, and fake news. Proceedings of the National Academy of Sciences 116 , 7662–7669 (2019).

Kreps, S. E. & Kriner, D. L. Model uncertainty, political contestation, and public trust in science: Evidence from the COVID-19 pandemic. Science advances 6 , eabd4563 (2020).

Myers, K. R. et al . Unequal effects of the COVID-19 pandemic on scientists. Nature Human Behaviour https://doi.org/10.1038/s41562-020-0921-y (2020).

Wang, D. S., Song, C. M. & Barabasi, A. L. Quantifying Long-Term Scientific Impact. Science 342 , 127–132 (2013).

Uzzi, B., Mukherjee, S., Stringer, M. & Jones, B. Atypical combinations and scientific impact. Science 342 , 468–472 (2013).

Radicchi, F., Fortunato, S. & Castellano, C. Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences 105 , 17268–17272 (2008).

de Solla Price, D. J. Networks of Scientific Papers. Science 149 , 510–515 (1965).

Article ADS Google Scholar

Price, D. d. S. A general theory of bibliometric and other cumulative advantage processes. Journal of the American society for Information science 27 , 292–306 (1976).

Funk, R. J. & Owen-Smith, J. A Dynamic Network Measure of Technological Change. Management Science 63 , 791–817 (2017).

Thelwall, M., Haustein, S., Larivière, V. & Sugimoto, C. R. Do altmetrics work? Twitter and ten other social web services. PloS one 8 (2013).

Wang, R. et al . in Proceedings of the 27th ACM International Conference on Information and Knowledge Management 1487–1490 (Association for Computing Machinery, Torino, Italy, 2018).

Tan, Z. et al . in Proceedings of the 25th International Conference Companion on World Wide Web 437–442 (International World Wide Web Conferences Steering Committee, Montréal, Québec, Canada, 2016).

Yin, Y., Dong, Y., Wang, K., Wang, D. & Jones, B. F. Public use and public funding of science. Nature Human Behaviour https://doi.org/10.1038/s41562-022-01397-5 (2022).

Wu, J. et al . CiteSeerX: AI in a Digital Library Search Engine. AI Magazine 36 , 35–48, https://doi.org/10.1609/aimag.v36i3.2601 (2015).

Wan, H., Zhang, Y., Zhang, J. & Tang, J. AMiner: Search and Mining of Academic Social Networks. Data Intelligence 1 , 58–76, https://doi.org/10.1162/dint_a_00006 (2019).

Zhang, Y., Zhang, F., Yao, P. & Tang, J. in Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining . 1002–1011.

Hendricks, G., Tkaczyk, D., Lin, J. & Feeney, P. Crossref: The sustainable source of community-owned scholarly metadata. Quantitative Science Studies 1 , 414–427 (2020).

Priem, J., Piwowar, H. & Orr, R. OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts. arXiv preprint arXiv:2205.01833 (2022).

Sinha, A. et al . in Proceedings of the 24th International Conference on World Wide Web 243–246 (Association for Computing Machinery, Florence, Italy, 2015).

Wang, K. et al . A Review of Microsoft Academic Services for Science of Science Studies. Frontiers in Big Data 2 , 45 (2019).

Wang, K. et al . Microsoft Academic Graph: When experts are not enough. Quantitative Science Studies 1 , 396–413 (2020).

Pinski, G. & Narin, F. Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics. Information processing & management 12 , 297–312 (1976).

Carpenter, M. P., Cooper, M. & Narin, F. Linkage between basic research literature and patents. Research Management 23 , 30–35 (1980).

Narin, F., Hamilton, K. S. & Olivastro, D. The increasing linkage between US technology and public science. Research policy 26 , 317–330 (1997).

Marx, M. & Fuegi, A. Reliance on science: Worldwide front‐page patent citations to scientific articles. Strategic Management Journal 41 , 1572–1594 (2020).

Marx, M. & Fuegi, A. Reliance on science by inventors: Hybrid extraction of in‐text patent‐to‐article citations. Journal of Economics & Management Strategy (2020).

de Solla Price, D. Little science, big science . (Columbia University Press, 1963).

Sinatra, R., Deville, P., Szell, M., Wang, D. & Barabási, A.-L. A century of physics. Nature Physics 11 , 791–796 (2015).

de Solla Price, D. Science since babylon . (Yale University Press, 1961).

Lin, Z., Yin, Y., Liu, L. & Wang, D. SciSciNet: A large-scale open data lake for the science of science research, Figshare , https://doi.org/10.6084/m9.figshare.c.6076908.v1 (2022).

Microsoft Academic. Microsoft Academic Graph. Zenodo , https://doi.org/10.5281/zenodo.6511057 (2022).

Smalheiser, N. R. & Torvik, V. I. Author name disambiguation. Annual review of information science and technology 43 , 1–43 (2009).

Tang, J., Fong, A. C., Wang, B. & Zhang, J. A unified probabilistic framework for name disambiguation in digital library. IEEE Transactions on Knowledge and Data Engineering 24 , 975–987 (2011).

Ferreira, A. A., Gonçalves, M. A. & Laender, A. H. A brief survey of automatic methods for author name disambiguation. Acm Sigmod Record 41 , 15–26 (2012).

Sanyal, D. K., Bhowmick, P. K. & Das, P. P. A review of author name disambiguation techniques for the PubMed bibliographic database. Journal of Information Science 47 , 227–254 (2021).

Morrison, G., Riccaboni, M. & Pammolli, F. Disambiguation of patent inventors and assignees using high-resolution geolocation data. Scientific data 4 , 1–21 (2017).

Tekles, A. & Bornmann, L. Author name disambiguation of bibliometric data: A comparison of several unsupervised approaches1. Quantitative Science Studies 1 , 1510–1528, https://doi.org/10.1162/qss_a_00081 (2020).

Van Buskirk, I., Clauset, A. & Larremore, D. B. An Open-Source Cultural Consensus Approach to Name-Based Gender Classification. arXiv preprint arXiv:2208.01714 (2022).

Cleary, E. G., Beierlein, J. M., Khanuja, N. S., McNamee, L. M. & Ledley, F. D. Contribution of NIH funding to new drug approvals 2010–2016. Proceedings of the National Academy of Sciences 115 , 2329–2334 (2018).

Packalen, M. & Bhattacharya, J. NIH funding and the pursuit of edge science. Proceedings of the National Academy of Sciences 117 , 12011–12016, https://doi.org/10.1073/pnas.1910160117 (2020).

Wang, Y., Jones, B. F. & Wang, D. Early-career setback and future career impact. Nature communications 10 , 1–10 (2019).

Hechtman, L. A. et al . NIH funding longevity by gender. Proceedings of the National Academy of Sciences 115 , 7943–7948 (2018).

Agrawal, A. & Henderson, R. Putting patents in context: Exploring knowledge transfer from MIT. Management science 48 , 44–60 (2002).

Bekkers, R. & Freitas, I. M. B. Analysing knowledge transfer channels between universities and industry: To what degree do sectors also matter? Research policy 37 , 1837–1853 (2008).

Owen-Smith, J. & Powell, W. W. To patent or not: Faculty decisions and institutional success at technology transfer. The Journal of Technology Transfer 26 , 99–114 (2001).

Mowery, D. C. & Shane, S. Introduction to the special issue on university entrepreneurship and technology transfer. Management Science 48 , v–ix (2002).

Williams, R. S., Lotia, S., Holloway, A. K. & Pico, A. R. From Scientific Discovery to Cures: Bright Stars within a Galaxy. Cell 163 , 21–23, https://doi.org/10.1016/j.cell.2015.09.007 (2015).

Article CAS PubMed Google Scholar

Hmielowski, J. D., Feldman, L., Myers, T. A., Leiserowitz, A. & Maibach, E. An attack on science? Media use, trust in scientists, and perceptions of global warming. Public Understanding of Science 23 , 866–883 (2014).

Li, J., Yin, Y., Fortunato, S. & Wang, D. A dataset of publication records for Nobel laureates. Scientific data 6 , 33 (2019).

Shen, H., Wang, D., Song, C. & Barabási, A.-L. in Proceedings of the AAAI Conference on Artificial Intelligence .

Ke, Q., Ferrara, E., Radicchi, F. & Flammini, A. Defining and identifying Sleeping Beauties in science. Proceedings of the National Academy of Sciences , 201424329 (2015).

Hirsch, J. E. An index to quantify an individual’s scientific research output. Proceedings of the National academy of Sciences of the United States of America 102 , 16569–16572 (2005).

Article ADS CAS PubMed PubMed Central MATH Google Scholar

Waltman, L., Boyack, K. W., Colavizza, G. & van Eck, N. J. A principled methodology for comparing relatedness measures for clustering publications. Quantitative Science Studies 1 , 691–713, https://doi.org/10.1162/qss_a_00035 (2020).

Santamaría, L. & Mihaljević, H. Comparison and benchmark of name-to-gender inference services. PeerJ Computer Science 4 , e156 (2018).

Bornmann, L. & Williams, R. An evaluation of percentile measures of citation impact, and a proposal for making them better. Scientometrics 124 , 1457–1478, https://doi.org/10.1007/s11192-020-03512-7 (2020).

Haunschild, R., Daniels, A. D. & Bornmann, L. Scores of a specific field-normalized indicator calculated with different approaches of field-categorization: Are the scores different or similar? Journal of Informetrics 16 , 101241, https://doi.org/10.1016/j.joi.2021.101241 (2022).

Yin, Y. & Wang, D. The time dimension of science: Connecting the past to the future. Journal of Informetrics 11 , 608–621 (2017).

Stringer, M. J., Sales-Pardo, M. & Amaral, L. A. N. Statistical validation of a global model for the distribution of the ultimate number of citations accrued by papers published in a scientific journal. Journal of the American Society for Information Science and Technology 61 , 1377–1385 (2010).

Bornmann, L. & Daniel, H.-D. What do we know about the h index? Journal of the American Society for Information Science and Technology 58 , 1381–1385, https://doi.org/10.1002/asi.20609 (2007).

Li, J., Yin, Y., Fortunato, S. & Wang, D. Nobel laureates are almost the same as us. Nature Reviews Physics 1 , 301 (2019).

Abramo, G., D’Angelo, C. & Caprasecca, A. Gender differences in research productivity: A bibliometric analysis of the Italian academic system. Scientometrics 79 , 517–539 (2009).

Huang, J., Gates, A. J., Sinatra, R. & Barabási, A.-L. Historical comparison of gender inequality in scientific careers across countries and disciplines. Proceedings of the National Academy of Sciences 117 , 4609–4616 (2020).

Dworkin, J. D. et al . The extent and drivers of gender imbalance in neuroscience reference lists. Nature neuroscience 23 , 918–926 (2020).

Squazzoni, F. et al . Peer review and gender bias: A study on 145 scholarly journals. Science advances 7 , eabd0299 (2021).

Yang, Y., Tian, T. Y., Woodruff, T. K., Jones, B. F. & Uzzi, B. Gender-diverse teams produce more novel and higher-impact scientific ideas. Proceedings of the National Academy of Sciences 119 , e2200841119 (2022).

Squazzoni, F. et al . Only second-class tickets for women in the COVID-19 race. A study on manuscript submissions and reviews in 2329 Elsevier journals. A study on manuscript submissions and reviews in 2329 (2020).

Vincent-Lamarre, P., Sugimoto, C. R. & Larivière, V. The decline of women’s research production during the coronavirus pandemic. Nature index 19 (2020).

Staniscuaski, F. et al . Gender, race and parenthood impact academic productivity during the COVID-19 pandemic: from survey to action. Frontiers in psychology 12 , 663252 (2021).

Fink, J. K. Hereditary spastic paraplegia. Neurologic Clinics 20 , 711–726, https://doi.org/10.1016/S0733-8619(02)00007-5 (2002).

Herzog, C., Hook, D. & Konkiel, S. Dimensions: Bringing down barriers between scientometricians and data. Quantitative Science Studies 1 , 387–395 (2020).

Lawrence, I. & Lin, K. A concordance correlation coefficient to evaluate reproducibility. Biometrics , 255–268 (1989).

Clauset, A., Shalizi, C. R. & Newman, M. E. Power-law distributions in empirical data. SIAM review 51 , 661–703 (2009).

Bornmann, L. & Wohlrabe, K. Normalisation of citation impact in economics. Scientometrics 120 , 841–884, https://doi.org/10.1007/s11192-019-03140-w (2019).

van Eck, N. J. & Waltman, L. Citation-based clustering of publications using CitNetExplorer and VOSviewer. Scientometrics 111 , 1053–1070, https://doi.org/10.1007/s11192-017-2300-7 (2017).

Xu, J. et al . Building a PubMed knowledge graph. Scientific Data 7 , 205, https://doi.org/10.1038/s41597-020-0543-2 (2020).

Torvik, V. I. & Smalheiser, N. R. Author name disambiguation in MEDLINE. ACM Transactions on Knowledge Discovery from Data (TKDD) 3 , 1–29 (2009).

Reproducible Science of Science at scale: pySciSci Abstract Quantitative Science Studies 1-17, https://doi.org/10.1162/qss_a_00260 .

Lazer, D. M. et al . Computational social science: Obstacles and opportunities. Science 369 , 1060–1062 (2020).

Lazer, D. et al . Computational social science. Science 323 , 721–723 (2009).

Article CAS PubMed PubMed Central Google Scholar

Barabási, A.-L. Network science . (Cambridge University, 2015).

Newman, M. Networks: an introduction . (Oxford University Press, 2010).

Castellano, C., Fortunato, S. & Loreto, V. Statistical physics of social dynamics. Reviews of modern physics 81 , 591 (2009).

Dong, Y., Ma, H., Shen, Z. & Wang, K. in Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . 1437–1446 (ACM).

Download references

Acknowledgements

The authors thank Alanna Lazarowich, Krisztina Eleki, Jiazhen Liu, Huawei Shen, Benjamin F. Jones, Brian Uzzi, Alex Gates, Daniel Larremore, YY Ahn, Lutz Bornmann, Ludo Waltman, Vincent Traag, Caroline Wagner, and all members of the Center for Science of Science and Innovation (CSSI) at Northwestern University for their help. This work is supported by the Air Force Office of Scientific Research under award number FA955017-1-0089 and FA9550-19-1-0354, National Science Foundation grant SBE 1829344, the Alfred P. Sloan Foundation G-2019-12485, and Peter G. Peterson Foundation 21048. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Authors and affiliations.

Center for Science of Science and Innovation, Northwestern University, Evanston, IL, USA

Zihang Lin, Yian Yin, Lu Liu & Dashun Wang

Northwestern Institute on Complex Systems, Northwestern University, Evanston, IL, USA

Kellogg School of Management, Northwestern University, Evanston, IL, USA

School of Computer Science, Fudan University, Shanghai, China

McCormick School of Engineering, Northwestern University, Evanston, IL, USA

Yian Yin & Dashun Wang

You can also search for this author in PubMed Google Scholar

Contributions

D.W. and Y.Y. conceived the project and designed the experiments; Z.L. and Y.Y. collected the data; Z.L. performed data pre-processing, statistical analyses, and validation with help from Y.Y., L.L. and D.W.; Z.L., Y.Y. and D.W. wrote the manuscript; all authors edited the manuscript.

Corresponding author

Correspondence to Dashun Wang .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and Permissions

About this article

Cite this article.

Lin, Z., Yin, Y., Liu, L. et al. SciSciNet: A large-scale open data lake for the science of science research. Sci Data 10 , 315 (2023). https://doi.org/10.1038/s41597-023-02198-9

Download citation

Received : 13 July 2022

Accepted : 02 May 2023

Published : 01 June 2023

DOI : https://doi.org/10.1038/s41597-023-02198-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Forecasting the future of artificial intelligence with machine learning-based link prediction in an exponentially growing knowledge network.

Mario Krenn
Lorenzo Buffoni
Michael Kopp

Nature Machine Intelligence (2023)

Data, measurement and empirical methods in the science of science

Benjamin F. Jones
Dashun Wang

Nature Human Behaviour (2023)

Quick links

Explore articles by subject
Guide to authors
Editorial policies

Centerline Archive
CBE in the News
Livable Buildings Awards

CBE Unveils Renovated Controlled Environment Chamber, Expanding Future Research

David Lehrer November 6, 2023

Categories: Organization Events HVAC Research IEQ

Tags: #CBE staff #HVAC research #thermal comfort #IEQ #events

Since opening in 1988, CBE’s controlled environment chamber has been used for research that has produced hundreds of journal papers, including keystone work related to human response, indoor environments and mechanical systems in buildings and automobile cabins. A major renovation was completed this fall, updating obsolete systems and failing equipment that was hindering important research operations. This milestone was commemorated last month, with a ribbon cutting ceremony held during a happy hour before CBE’s fall Industry Advisory Board meeting. Research Specialist Charlie Huizenga , who managed the project for CBE, gave a presentation on the chamber’s history and the upgrades. Attendees also learned a less familiar history, describing how the launch of this facility led to a noteworthy romance (more on that later). The upgrade design and construction administration were done by David Heinzerling of CBE Industry Partner Taylor Engineers.

Ribbon cutting at CBE controlled environment chamber, October 2023

(L to R) David Heinzerling of Taylor Engineering, CBE Director Edward Arens, and Ippei Izuhara of Sanken at the October ribbon cutting ceremony.

With its recent upgrade, the Berkeley facility is one of the most advanced test chambers of its type. The two external walls use a double wall design that allows conditioned air to be circulated behind the inside surface, so the temperature of the interior glass and walls can be controlled independently of the room temperature. Both the ceiling and underfloor spaces can be used as supply or return plenums. The upgrade added radiant ceiling panels, donated along with additional financial support by CBE Industry Partner Sanken , to create a new range of study capabilities. The upgrade also included a humidifier, control software from Automated Logic Corporation, and controllable LED lighting dimmable to one percent, with adjustable color temperature, fixture-integrated occupancy sensors and networked controls.

While the chamber was initially being constructed, the research team led by Fred Bauman learned about the evolving underfloor air distribution (UFAD) technology, and connected with Geno Brager , who was a supplier for Tate Access Floor and eventually donated the raised floor that was installed in the chamber. At the time, Prof. Gail (Schiller) Brager had recently begun her academic career at UC Berkeley’s Building Science Group, and met Geno for the first time literally inside the chamber. To make a long story short, they had an instant connection, and have now been happily married for over 30 years.

Romances aside, many of the early activities surrounding the chamber led to far-reaching research results. Tate Access Floor later became one of CBE’s founding industrial partners, and this collaboration led to studies in the chamber, plus field studies and simulations, that produced UFAD design tools and papers, and two editions of the UFAD Design Guide . A chamber-based dissertation study with human subjects by Hui Zhang led to seminal work on thermal comfort and the development of the Advanced Berkeley Comfort Model . The chamber has been used for studies on personal comfort devices , fans of all types , automotive cabin conditioning , window views and more.

With the renovation complete, CBE is excited to be embarking on a long list of planned experiments. A new study is underway on the effects of age and sex on thermal comfort, with support and participation of CBE Industry Partner Arup . We have also begun studies to improve our understanding of the design and operation of a convection-enhanced radiant panel used in combination with a dedicated outdoor air supply (DOAS) for heating and cooling, developed by Industry Partner Sanken . These are just a few examples of research that will take advantage of this upgraded facility, and we look forward using it while pursuing new directions in collaboration with CBE consortium members and with research affiliates here at UC Berkeley and further afield.

Photo at top: Research team members Ying Li, Akihisa Nomoto and Xue Zhai configure measurement equipment in the chamber.

Faculty and Staff
Visiting Scholars
Visiting Scholar Program
Graduate Programs
Indoor Environmental Quality
HVAC Systems
Facade Systems
Human Interactions
Building Decarbonization
Industry Partners (Industry Advisory Board)
Joining CBE
Benefits of Membership
Partner Advisory Board Meetings
Partner Login
Publications
CBE Occupant Survey
Upcoming Events
Past CBE Events

This paper is in the following e-collection/theme issue:

Published on 6.11.2023 in Vol 25 (2023)

Evaluation of a Virtual Reality Platform to Train Stress Management Skills for a Defense Workforce: Multisite, Mixed Methods Feasibility Study

Authors of this article:

Original Paper

Murielle G Kluge 1, 2 , PhD ;
Steven Maltby 1, 2 , PhD ;
Caroline Kuhne 1, 3 , BSc ;
Nicole Walker 4 , MPsych ;
Neanne Bennett 5 , MPsych ;
Eugene Aidman 2, 6 , PhD ;
Eugene Nalivaiko 1, 2 , PhD ;
Frederick Rohan Walker 1, 2 , PhD

1 Centre for Advanced Training Systems, Faculty of Health, Medicine & Wellbeing, The University of Newcastle, Callaghan, Australia

2 School of Biomedical Sciences & Pharmacy, Faculty of Health, Medicine & Wellbeing, The University of Newcastle, Callaghan, Australia

3 School of Psychological Sciences, College of Engineering, Science and Environment, The University of Newcastle, Callaghan, Australia

4 Army School of Health, Australian Defence Force, Canberra, Australia

5 Joint Health Command, Department of Defence, Canberra, Australia

6 Human and Decision Sciences Division, Defence Science & Technology Group, Edinburgh, Australia

Corresponding Author:

Frederick Rohan Walker, PhD

Centre for Advanced Training Systems

Faculty of Health, Medicine & Wellbeing

The University of Newcastle

University Drive

Callaghan, 2300

Phone: 61 024921 5012

Email: [email protected]

Background: Psychological stress-related injuries within first-responder organizations have created a need for the implementation of effective stress management training. Most stress management training solutions have limitations associated with scaled adoption within the workforce. For instance, those that are effective in civilian populations often do not align with the human performance culture embedded within first-responder organizations. Programs involving expert-led instructions that are high in quality are often expensive.

Objective: This study sought to evaluate a tailored stress management training platform within the existing training schedule of the Australian Defense Force (ADF). The platform, known as Performance Edge (PE), is a novel virtual reality (VR) and biofeedback-enabled stress management skills training platform. Focusing on practical training of well-established skills and strategies, the platform was designed to take advantage of VR technology to generate an immersive and private training environment. This study aimed to assess the feasibility of delivering the VR platform within the existing group-based training context and intended training population. In this setting, the study further aimed to collect data on critical predictors of user acceptance and technology adoption in education, including perceived usability, usefulness, and engagement, while also assessing training impacts.

Methods: This study used a mixed methods, multisite approach to collect observational, self-reported, and biometric data from both training staff and trainers within a real-world “on-base” training context in the ADF. Validated scales include the Presence Questionnaire and User Engagement Scale for perceived usefulness, usability, and engagement, as well as the State Mindfulness Scale and Relaxation Inventory, to gain insights into immediate training impacts for specific training modules. Additional surveys were specifically developed to assess implementation feedback, intention to use skills, and perceived training impact and value.

Results: PE training was delivered to 189 ADF trainees over 372 training sessions. The platform was easy to use at an individual level and was feasible to deliver in a classroom setting. Trainee feedback consistently showed high levels of engagement and a sense of presence with the training content and environment. PE is overall perceived as an effective and useful training tool. Self-report and objective indices confirmed knowledge improvement, increased skill confidence, and increased competency after training. Specific training elements resulted in increased state mindfulness, increased physical relaxation, and reduced breathing rate. The ability to practice cognitive strategies in a diverse, private, and immersive training environment while in a group setting was highlighted as particularly valuable.

Conclusions: This study found the VR-based platform (PE) to be a feasible stress management training solution for group-based training delivery in a defense population. Furthermore, the intended end users, both trainers and trainees, perceive the platform to be usable, useful, engaging, and effective for training, suggesting end-user acceptance and potential for technology adoption.

Introduction

Stress management training in the workforce.

The negative impacts of unmanaged stress exposure are well documented in first-responder populations [ 1 , 2 ]. Consequences of prolonged exposure to unmanaged psychological stress can include, but are not limited to, changes in cognition, judgment, motivation, and mood [ 3 ]. With prolonged exposure to stress, disruptions in mood and cognition can transition into diagnosable pathologies such as burnout, anxiety, depression, and trauma [ 4 - 6 ].

To protect their workforce and those under their care, many first-responder organizations have sought to deliver scalable and structured stress management training to their staff. Stress management training includes several different forms and components. A useful viewpoint is the definition of stress management training as “the application of any set of techniques ( e.g. , exposure training, relaxation, biofeedback, and cognitive behavioural therapy) with the intent to improve the way people cope with stress” [ 7 ]. Furthermore, existing stress management training programs can be broadly classified into two types: (1) stress-inoculation training—repeated exposure to a stressor to develop tolerance (eg, outdoor adventure, live fire, and mission rehearsal) and (2) cognitive and psychological skills training conducted in a nonstressful setting. Stress inoculation training programs have been popular for military and first-responder training organizations when the stressor is predictable and likely (eg, physical and verbal altercation or combat [ 8 , 9 ]). Cognitive and psychological skills training, also often referred to as resilience training, can include mindfulness-based, cognitive-behavioral strategies and relaxation techniques [ 10 - 15 ]. Breathwork and mindfulness-based training interventions have documented efficacy in both clinical and nonclinical settings [ 16 , 17 ]. These strategies are being increasingly assessed in workforce and workplace contexts [ 18 - 21 ]. Cognitive strategies, including goal setting and emotional and attentional control, have also been shown to positively impact psychological well-being in military personnel [ 22 - 25 ]. Although there is growing evidence on the benefit of training cognitive stress management skills in first responders and similar occupational cohorts [ 26 - 30 ], further research is required to inform best practice strategies on how to effectively implement and scale training within and across organizations. A major challenge in this context is that many workplace and training organizations deliver training as scheduled activities and in groups.

The Australian Defence Force (ADF) has a well-established stress-management training platform (BattleSMART) based on cognitive behavioral therapy tailored to ADF members [ 31 ]. Consistent with other stress management programs, BattleSMART relies on the provision of instructional and theoretical materials on using cognitive and psychological skills [ 31 ]. Although well accepted, BattleSMART has been constrained, with respect to skill establishment, by the number of expert facilitators and the time required to deliver the program. Robustly establishing psychological skills, as is the case with other complex skills, requires extensive time for skill rehearsal and expert-led facilitation [ 18 ]. A major challenge connected to the delivery of practical stress management skills is that this type of training benefits from a private and focused training environment. Hence, it is typically facilitated in one-on-one sessions. However, like many training organizations, ADF typically organize and operate their activities via a group-based structure. A change from group-based to one-on-one training for stress management skills instruction would represent a significant burden to the organization.

The Use of Virtual Reality for Stress Management Training

Virtual reality (VR) is an interesting solution for stress management training. VR provides a technical platform to place skill development and expert-led instruction “into the headset.” Moving from a human delivered to digitally delivered instruction can mitigate many issues associated with specialist workforce limitations and allow for flexibility in when and where training can occur. Although similar things can be said about many digital software solutions, the VR headset can create a private and immersive environment that can be particularly beneficial for both group-based delivery and the nature of the subject matter. Additional benefits of VR over other conventional 2D-based platforms include increased immersion, interaction through handheld controllers, a strong sense of presence, engagement, and student motivation [ 32 - 34 ]. Thus, VR-delivered training may provide a viable solution to circumvent the major challenges associated with group-delivered implementation of stress management training for large organizations, including defense.

Several VR and biofeedback-integrated training applications have already been trialed in military and police populations, including those targeting stress inoculation, passive relaxation, and breath control [ 9 , 35 - 38 ]. To our knowledge, however, there have been no comprehensive stress management training programs that teach a diverse range of stress management skills that are appropriate for first-responder organizations, such as the Australian military.

The Performance Edge Stress Management Training Platform

To address the existing unmet needs for practical and scalable training of stress management skills for first-responder populations, we developed a new and comprehensive VR-based training platform called Performance Edge (PE) [ 39 ]. In collaboration with the ADF, PE was specifically developed to target an early career training population and be aligned with ADF values. Evaluation of the first PE module, which focused on training controlled breathing skills, demonstrated the in-principal suitability of the technology and training approach [ 39 ]. Building on the initial work, the modular PE platform was expanded to include 5 modules, each of which provided fundamental skill training for evidence-based stress management strategies adapted from cognitive-behavioral therapy and acceptance and commitment therapy. More details on the PE training concept, the framework and specific module content are provided in Multimedia Appendix 1 .

This study has the following objectives:

Evaluate the feasibility and implementation of the 5-module PE prototype training package, delivered on-site at multiple military locations to its intended ADF target population, and within existing training activities and group-based, classroom settings.
Assess critical predictors of technology adoption (eg, perceived usability, and usefulness), engagement, and training impact.

PE Training Platform and Framework

The PE software was delivered on the Oculus Quest (Meta), a freestanding VR headset with 2 handheld controllers and an inside-out tracking system. A total of 20 headsets using the Oculus for Business Enterprise platform were used, which supported fleet management, content uploading, and battery monitoring. Respiratory signals were collected and integrated into VR training using a GoDirect respiratory belt (Vernier). A custom software application was developed and installed to translate and display the breathing traces and rates (breaths per minute) from the belt into the headset in real time to facilitate biofeedback for training.

The PE software application used in this study contained a full menu management system and 5 individual training modules. Each module focused on a specific skill or cognitive strategy adapted from cognitive-behavioral therapy and acceptance and commitment therapy principles ( Table 1 and Multimedia Appendix 1 ). All PE modules included an introductory video, guided narration, personalized learning, practical training, user interactions, feedback, performance measures, and the opportunity for repetition.

The PE interface adopts a futuristic design that pays homage to space-themed computer games ( Figures 1 A-1D). Although the target audience was military, visual design features were created using a neutral pallet without specific reference to military design, situations, or terminology. This was in part due to military triservice-specific branding but also emphasized that the skills can be relevant for any stress-provoking situation, either work related or in everyday life. Alignment with ADF values was generated using a clear and directive tone for instructions, feedback, and explanations, with an emphasis on practical elements. All the exercises were designed to take advantage of the immersive nature and interactive capability of VR technology to create an engaging learning and training environment. Examples can be found in Multimedia Appendices 2 and 3 , containing walk-through videos from the user's view within the VR headset.

Study Participants and Recruitment

Participants were recruited from the initial employment training (IET) programs at 3 military bases in Australia. The IET program is the first service-specific training that military personnel undergo after basic training. As both the existing stress management training program (BattleSMART) and the PE platform were developed for military personnel early in their careers, this population represents the target training population. Participants were informed of the VR training and research study that took part as part of their regular training schedule 1 to 2 weeks prior by their commanding officers, who also provided them with Participant Information Statements. Participants were invited to attend multiple PE training sessions over a week in May 2020, June 2021, November 2021, and March 2022. On-base training staff members were also invited to attend open exploration sessions to use PE. A total of 189 participants were involved in the study across all locations (156 ADF trainees and 33 training staff members). Participant numbers for each module differed, as not all trainees were available across all delivery days at each site, and not all modules were tested at all locations (eg, trial week 2 at trial site 2 was shortened due to the COVID-19 lockdown). Furthermore, 12 trainees attended 2 separate trial weeks, and their data were only included in the analysis for their first attendance.

Ethical Considerations

Research activities were reviewed and approved by the Australian Defense Science and Technology Low Risk Ethics Panel (Protocol Land Division 17-19) and coregistered with The University of Newcastle Human Research Ethics Committee (H-2020-0020). All participants provided written informed consent to participate in this study. Participants were not reimbursed and did not receive compensation for participating in the study. The study was conducted as a scheduled activity within the IET program; however, there was clear communication that participation in the research study was voluntary and outside of their training requirements. An alternative work- or training-related activity, as specified by their commanding officer, was provided to trainees who chose not to participate. A participant ID number was used to match responses where necessary, and no identifiable information (name, rank, or ID number) was collected. Any identifiable data or information provided by a participant within the survey responses were redacted to ensure the anonymity of the participants.

Study Design

A mixed methods approach was applied to evaluate the delivery of PE as a multiday classroom-based training program across the 3 training sites. The trainees completed one to five PE modules on consecutive days within a training week. Training and subsequent assessment of each module occurred within a single 1-hour session in a group setting with 8 to 15 participants simultaneously ( Figures 1 E and 1F and Figure 2 ).

The trainees completed pre- and posttraining surveys specific to each training module. Each training session was concluded with a 10-minute group discussion focused on providing an opportunity for group feedback.

The on-base training staff attended a single unstructured exploration session, in which any module or component could be explored, and completed the posttraining trainer survey only.

Measures and Data Collection

This trial employed observational and objective biometric data, as well as self-report and focus group data related to multiple aspects of PE training delivery.

Self-Report Instruments

Training gains measures.

Each pre- and posttraining survey contained validated scales and general questions on relevant constructs and domains, informed by the technology acceptance model. This included questions to assess technology acceptance (perceived usability, perceived usefulness, and implementation feedback), relevance of content and framework, previous work experience, virtual reality, stress management skills, and perceived training impact. A full list of questions is provided in Multimedia Appendix 1 , Table S1.

The study-specific questions were adapted from those previously developed and administered in studies assessing VR training and the first PE pilot study [ 39 ]. Questions were constructed using a 3-step process (defining construct, content domains, item generation, and determining the format) from VR experts, teaching, training experts, and ADF members. Questions were drafted by investigator MK and finalized using an iterative approach, with feedback from all study investigators using multiple choice, 5-point Likert scale, or an open-ended format. A list of surveys, validated scales, validity, and relevant references are provided in Tables 2 and 3 . All surveys were distributed using QR codes and Office Forms (Microsoft Office).

a VR: virtual reality.

b ADF: Australian Defense Force.

Trainer Surveys

Training staff received a tailored trainer survey after exploring PE content. Questions were adapted from trainee surveys and included additional questions on the suitability of the training framework and technology delivery within the broader organization.

Observational Data

The research team documented quantitative data, including costs, time frames, hardware charging, and set-up requirements.

Objective Biometric Data

Respiratory signals were sampled every 0.05 seconds (20 Hz) and translated into respiratory rate (breaths per minute), calculated after every inhalation. Rates are calculated via a peak detection algorithm using the derivate of data points after smoothing low-pass filtering and are updated after each peak inhalation detection. The average respiratory rate for each 2-minute exercise period was collected and saved, time-stamped, and sent using Wi-Fi to an external cloud-based server with data linked to the serial number of the corresponding headset. Respiratory data integration and collection occurred in modules 2 and 3, respectively. All the modules collected and recorded the selection choices, interactions, and time spent in each exercise.

Data Analysis

Data from all locations were pooled for analysis unless specifically mentioned in the text, whereas data from trainees and training staff were analyzed and reported separately. Self-reported and objective data were analyzed using Prism (version 8; GraphPad) and JASP (version 0.16.3).

Self-reported data were summarized and presented as mean (SD) or absolute number of responses, and all responses were included. Validated scales were calculated per protocol and analyzed using a 1-tailed parametric test (paired t test) or nonparametric equivalent (Wilcoxon signed-rank test), where assumptions were violated.

Respiratory rate data were analyzed using one-way repeated measures ANOVA adjusted for multiple comparisons and a paired-sample t test to compare average respiratory rates during training in M2 and average respiratory rates before and after training in M3.

Both P values and Bayes factor (BF) have been reported. Values of P <.05 are reported as indicators of significance and a BF 10 >3 is considered decisive evidence in favor of a difference or effect.

Training Population and Existing Skill Level

Overall, 40% (50/126) reported previous experience with VR technology, largely in the recreational gaming context with 50% (25/50) reporting less than1 hour, 28% (14/50) less than 10 hours, and 22% (11/50) reporting over 10 hours of total exposure. Overall confidence in the set-up and use of VR technology was moderate (mean 3.4, SD 1.2; 1=not at all confident, 5=extremely confident). The remaining 60% (114/126) of trainees had not previously used VR.

Trainees indicated general awareness and theoretical knowledge of stress management, but the perceived utility of specific skills at the outset of training was modest. When questioned about day-to-day attention to their thoughts, emotions, and initial behaviors (“inner world”), trainees reported being very aware (33/64, 52%) or somewhat aware (31/64, 48%) of the degree to which their thoughts and emotions influenced each other and their behaviors. Nearly all participants (62/63, 98%) indicated that they were either very (33/63, 52%) or somewhat (29/63, 46%) aware of the impact that stress can have on their inner world. Of the specific skills taught in PE, controlled breathing was the most highly reported stress management skill used in the cohort pretraining, with 80% of trainees (64/80, 80%) previously engaging in controlled breathing. Although awareness of controlled breathing as an effective stress management skill was high (80/80, 100%; very or somewhat aware), it was primarily applied in a physical and exercise context (eg, target shooting and combat training) rather than for stress management per se. Few other skills and stress management concepts were understood to be effective strategies to reduce stress, and their use was limited. Only 15% (8/54), 23% (23/98), and 36% (15/42) of trainees reported having used progressive muscle relaxation, grounding, or an acceptance strategy to manage challenging emotions, respectively.

Engagement and Sense of Presence With PE Training

Engagement with PE training was assessed for each module using the validated User-Engagement Scale–Short Form (UES-SF; [ 40 ]) for digital domains, which includes the subdimensions of esthetic appeal, focused attention, perceived usability, and reward. All 5 modules were positively rated by the trainees ( Figure 3 A), with the highest level of user engagement reported for M2—Controlled Breathing (3.9, SD 0.54) and lowest for M5—Managing Emotions (3.5, SD 0.60; 5-point Likert scale, >3=positive), with no modules scoring below 3.0 (which would indicate a negative assessment) in any UES-SF subdimensions. Across all modules, perceived usability was consistently the highest scoring subdimension.

To assess presence within the experience, the Presence Questionnaire was administered after each training module ( Figure 3 B). All the modules were perceived as having an overall feeling of presence. The sense of presence ranged from 4.7 (SD 0.94) in M5—Managing Emotions, to 5.1 in M2—Controlled Breathing, and M3—Progressive Muscle Relaxation (PMR) (SD M2 0.67; SD M3 0.73; 7-point scale). The single item assessing perceived privacy was rated above 3.6 for each module (5-point scale, Figure 3 C).

By design, all training modules and most exercises within PE contain interactive components, including actions and response options for trainees. Trainees responded to interactions within 6 to 31 s, with response times varying by the number of response options, complexity, and familiarity with the exercise. Although answer options were never considered “correct” or “incorrect,” certain exercises and scenarios aimed to invoke a particular emotion or thought response. For example, 87% (32/38) of trainees selected either relaxed or happy as the main emotion felt during an intentionally relaxing 360° video beach environment in M5—Exercise 1.

In an open-ended response section, immersion and privacy of VR technology were named the most beneficial components. This observation was supported by additional question responses, in which both immersion and privacy were rated to be useful or extremely useful in supporting training and attention focus (immersion=3.9, SD 0.90 and privacy=3.7, SD 0.86, respectively; 1=not at all useful and 5=extremely useful; Figure 3 D). Biofeedback functionality, interactive elements, and concept visualization were also named as particularly positive features of PE training across all modules in open-ended questions. Survey responses consistently provided positive feedback on individual modules while endorsing the overall length of individual exercises.

When asked which elements of the training could be improved, open-ended responses primarily included the expansion of existing elements, escalation of provocative content and scenarios, and inclusion of ADF-specific content.

Module-Specific PE Training Outcomes

Module-specific training perceptions were assessed using the objective and qualitative outcomes relevant to the specific training objectives of each module. The central training objective within M2—Controlled Breathing, was to gain awareness and control over breath cadence, specifically to maintain a slow and steady breathing rate across escalating training exercises ( Figure 4 A). Consequently, changes in breathing rates were used to gain insights into training outcomes with the hypothesis that exercises within PE would result in a reduction in breaths per minute (BPM) compared with the initial, noninstructed breathing rates recorded at the start of the module. Initial baseline respiratory rates, without prompts, were 16.3 (SD 3.9) breaths BPM for trainees and 16.8 (SD 5.6) BPM for training staff. When prompted to actively reduce their breathing rate, initial mean breathing rates reduced to 10.3 (SD 3.4) BPM (trainees) and 10.7 (SD 2.1) BPM (staff). Live biofeedback with a respiratory trace (termed “controlled breathing assisted”) supported further reductions to 8.5 (SD 3.2) BPM (trainees) and 8.5 (SD 3.1) BPM (staff). Participants subsequently sustained reduced controlled breathing rates without live biofeedback (“controlled breathing non-assisted”), and in subsequent exercises incorporating a distracting environment (“controlled breathing concert”) and shooting tasks. Training effects for M3—Progressive Muscle Relaxation were assessed using changes in breathing rates and state relaxation before and after the 15-minute guided relaxation exercise. Mean respiratory rates were significantly reduced compared with baseline at the start of the exercise, from 19 (SD 3.8) to 6.9 (SD 2.3) BPM (mean difference −12.74, SD 3.4; t 26 =19.198, P <.001; BF 10 =1.413e+14).

The Physical Assessment Scale (PAS) and Cognitive Tension Scale (CTS) subscales of the Relaxation Inventory were administered before and after both M2 and M3 to quantify relaxation states ( Figure 4 B). Owing to the common use of controlled breathing and PMR for relaxation, we hypothesized that training would result in an increased self-report rating on the PAS and reduced ratings on the CTS. As hypothesized, physical relaxation (PAS) significantly increased after both M2—Controlled Breathing (mean increase 7.78, SD 9.9; t 30 =−4.354, P <.001; BF 10 =374.435) and M3—PMR training (mean increase 11.27, SD 14.49; w=224, Z =−5.087; P <.001, BF 10 =891.313). Similarly, cognitive tension scores (reverse scored to indicate relaxation) increased after both modules (M2 mean increase 2.53, SD 6.04; w=87, Z =−2.451 , P =.007; BF 10 =22.523; M3 mean increase 3.4, SD 5.10; w=193, Z =−4.406, P <.001; BF 10 =3543.625).

Given the training objectives for M4 and M5, we explored the assumption that training may increase mindfulness, as measured using the State Mindfulness Scale ( Figure 4 C). M4—Grounding increased mindfulness scores from 63.42 (SD 17.26) to 69.98 (SD 16.57) , a significant change (mean increase 6.14, SD 15.22; w=906.5, Z =−4.036, P <.001; BF 10 =5067.078). A trend toward increased State Mindfulness Scale scores following M5—Managing Emotions training was not statistically meaningful (mean increase 2.97, SD 12.45; w=155, Z =−1.351, P =.09, BF 10 =0.743). This was true for both the Body (mean increase 0.50, SD 5.195; w=120.000, Z =−1.410, P =.08; BF 10 =0.830), and Mind (mean increase 2.47, SD 8.97; w=186.5, Z =−1.449, P =.08; BF 10 =1.452) subscales. However, the Bayes factor for the Mind subscale (BF 10 =1.452) provided anecdotal evidence of an increase in mental-state mindfulness.

In addition to objective outcomes, self-reported data on training impact, efficacy, and perceived value were collected after each module and final training session. Following training, trainees reported a deeper understanding of the theoretical concepts taught within PE, specifically for underlying practical skills (M1: 3.9, SD 0.7; M2: 3.8, SD 0.7; M3: 3.8, SD 0.71; M4: 3.8, SD 0.7; M5: 3.8, SD 0.7; 1=strongly disagree; 5=strongly agree; Figure 5 A). Perceived skill competency improved after training for M2—Controlled Breathing, M3—PMR, M4—Grounding and M5—Managing Emotions (M2: 3.7, SD 0.8; M3: 3.8, SD 0.7; M4: 3.7, SD 0.6; M5: 3.3, SD 0.9; 1=strongly disagree; 5=strongly agree; Figure 5 C). Statistical analysis was conducted on self-report items asking about the likelihood of engaging in these skills. We hypothesized that PE training would increase the likelihood of engaging in the respective skill in a stressful context. Trainees indicated they were more likely to actively consider their thoughts and emotions before reacting to a stressful event (mean 3.8, SD 0.8 on a 5-point Likert scale; mean difference [post-pre]: 0.46, SD 0.7; Wilcoxon signed-rank test Z =−2.548, P =.002; BF 10 =27.94). Similarly, trainees were more likely to use grounding skills after PE training compared with pretraining (mean increase 0.55, SD 0.9; Wilcoxon signed-rank test Z =−2.353, P =.008; BF 10 =9.118). Although the likelihood of using other skills did not change pre-post training, trainees indicated they were likely to engage in controlled breathing (4.0, SD 0.8) and use an acceptance strategy to manage their emotions (3.5, SD 0.9) the next time they encountered a stressful or challenging event (1=not at all likely, 5=extremely likely; Figure 5 B), indicating that pretraining levels were already favorable toward these skills. The overall intention to use PMR and grounding skills after completing M3 and M4 training was only modest (mean 3.1, SD 1.2 and 3.1, SD 1.1, respectively). Overall, 66% (75/113) of participants were confident or extremely confident that PE represented a useful and effective platform to train and practice stress management skills (3.9 SD, 0.8, 1=not at all confident; 5=extremely confident, Figure 5 D), whereas only 2 of 113 trainees stated they were not confident that the platform was a useful stress management training tool.

Trainer Feedback

Most trainers agreed or strongly agreed that trainees engaged with the training content (24/29, 83%) and that the platform delivered effective practical training on stress management skills (18/28, 64%; only 2 trainers disagreed). Furthermore, 50% (15/30) of the training staff agreed that the platform provided valuable knowledge transfer, whereas 5 disagreed with that statement ( Figure 6 A). Importantly, 79% (23/29) of the trainers stated that they were confident or extremely confident in their ability to deliver VR training in the classroom. Verbal feedback and responses to an open-ended question indicated that staff members were positively surprised by the platform and how the technology supported fundamental skill development. With expected stress inoculation training, many saw great benefits in approaching cognitive and emotional skills training within the immersive and private environment of the VR headset. Engaging and interactive components were named particularly beneficial and contrasted with the traditional delivery approach using PowerPoint-based materials.

Trainer quotes from survey:

[Performance Edge modules are] really good at helping people identify their responses to situations.

It is very beneficial to include practical training related to a soldier mindset.

Although trainers praised the use of nonmilitary design, language, and introduction segments, both trainees and trainers suggested that the final exercises in each module would benefit from being placed within a relevant military context.

Alignment of PE With the Existing ADF Training Framework

The PE platform was developed as an extension of the ADF stress management framework. Compared with the existing approach, most staff and trainees indicated training added value or was superior ( Figure 6 B). Specifically, for staff, 41% (11/27) believed PE to be a valuable addition, which supports existing stress management training, and 19% indicated that PE was superior (a combined total of 60%). Of the remaining staff, 26% indicated that they were unable to recall the existing stress management training and, therefore, were unable to make a comparison, 11% felt it neither supported nor detracted from existing training, and only 1 trainer (out of 27) indicated that PE was inferior. For trainees, 57% (86/150) believed PE to be a valuable addition, supporting existing stress management training, and 28% indicated that PE was superior (combined total of 85%). Moreover, 12% felt that it neither supported nor detracted, and only 1 trainee (of 150) indicated that it was inferior to the existing approach.

Delivery of PE Training

Data on trial logistics and requirements were also collected to inform the implementation strategies and determine their feasibility. A total of 372 PE training sessions were delivered over 9 days across 4 training weeks. Average “in headset” training time (in minutes) for each module was M1=22.9 (SD 4.2), M2=17.4 (SD 2.8), M3=18.9 (SD 1.8), M4=33.9 (SD 1.3), and M5=32.6 (SD 6.2).

Content was loaded on headsets off-site before each trial week (duration approximately 20 minutes) and the initial on-site hardware set-up of 20 headsets required approximately 1 hour by a study team member. A full headset charge from 0% to 100% required approximately 3 hours. After three consecutive training sessions (approximately 1.5-hour active runtime plus standby time of approximately 5 h), the average headset battery charge was 40%. The hardware was charged overnight between training days. A silicone cover was used over the headset face foam. The silicone cover, outside the headset, controllers, and respiratory belt were wiped down before and after each session using a skin-friendly VR head mounted display cleaning wipe provided to the trainees (a typical classroom set-up is shown in Figure 1 E).

Trial weeks in 2021 occurred under COVID-19 social distancing requirements (1.5 m distancing between individuals; Figure 1 F). Although the first session each week required specific instructions and clarifications on how to set-up, use, and navigate the hardware, trainees required limited input in subsequent sessions. Additional instructions were provided for the biofeedback-enabled modules to ensure that the respiratory belt was plugged into the headset. Trainees required repeated and specific instructions on how to re-enter the VR field of view. Attendance by two instructional staff members to support the delivery was useful, but only required during the first session each week to support information and consent processes for the research component of the trial. For standard operational training delivery outside the research context, one team member would be sufficient.

In addition, 98% (363/372) of the training sessions were delivered without any issues or difficulties identified. During the initial March 2020 trial (n=42 total participants), a total of 8 technical issues were reported. All but two technical issues were resolved with assistance during the session. The remaining two issues related to automated data capture were not apparent until the completion of the training. Five technical issues related to the connection between the respiratory belt and the VR headset (specifically, the USB-C wired connection). The USB-C connection issue was resolved for subsequent trial dates by replacing the USB-C adapter with a robust model. The remaining issues included freezing of the VR headset screen, operator difficulties, and set-up of the VR “guardian area.”

During the November 2021 trial (n=37 participants), 7 trainees were unable to complete the final exercise of M1 owing to a loading issue in the software, which was subsequently addressed and resolved for the March 2022 trial. Two additional trainees accidentally exited the training module. No VR-induced motion sickness was reported directly or in the survey responses during or after training. The research team was informed of a single unanticipated response to one exercise in the M4—Grounding module, in which a trainee reported to the ADF chaplain that the memory recall component of the training triggered emotional distress. As a result, a warning was included in subsequent versions, noting the potential for an emotional response.

Principal Findings

This trial describes the delivery of the PE prototype as a VR-based practical training platform for fundamental stress management skills within a workplace setting. The outcomes demonstrated the perceived usefulness, feasibility, usability, and positive training outcomes of the technology platform, training concept, and specific training modules within the intended real-world context and training population.

PE was delivered to 189 military trainees during consecutive 1 hour in-classroom training sessions of up to 20 trainees at a time and 5 modules in total. The distribution and utility of the biofeedback-integrated VR system were portable, easy to set-up, and suitable for the needs and requirements of the training organization and the target training population. Both the intended training population and their training staff perceived the platform to be useful, easy to use, engaging, immersive, and aligned with the existing stress management training framework ( Figures 4 and 6 ). Based on the technology acceptance model, our results for perceived usefulness, immersion, and engagement suggest that future adoption of the platform is highly feasible [ 44 , 45 ]. The ability to practice cognitive strategies in a diverse, private, and immersive training environment, while in a group setting, was highlighted as particularly valuable and supported the training objectives. Training benefits were observed in both physiological and mindfulness outcomes for specific training modules ( Figure 4 ). Consistent positive feedback and self-report responses from both the training staff and trainees indicated increased knowledge, skill competency, and intention to use certain skills in the future (controlled breathing, awareness, and emotional acceptance; Figure 5 ).

PE Is a Feasible VR Training Solution for Group-Based Training

VR technology is an emerging field of research in military contexts and is predicted to improve training effectiveness in multiple domains, including combat command and decision making [ 35 , 46 , 47 ]. The use of VR as a training modality is relatively new, and very few training organizations have sustainably adopted, scaled, or integrated technology within their training continuum. This is particularly true for VR-based cognitive and psychological stress management interventions as their integration into the workforce has been challenging [ 14 , 48 , 49 ]. The relative novelty of VR-based stress management training has resulted in research focusing largely on the efficacy of applications under controlled experimental conditions [ 50 ]. However, these types of controlled studies cannot address questions about feasibility and implementation in real-world contexts, and there are limited reports on implementation challenges in the literature [ 51 ]. Factors that may impair the uptake of an otherwise effective digital tool include technology acceptance, practical usability, proficiency in use, and the existing structures required to support the technology [ 52 ]. This was due to the paucity of information on implementation issues, which we placed a particular emphasis on in this study.

After our original study evaluating PE in a controlled research trial, we found that the initial set-up and hardware was impractical, complex, and unstable outside of a research setting, particularly the research grade biometric data collection system. The platform underwent significant redesign to address challenges related to practical implementation within a group-based classroom setting [ 39 ]. We demonstrated the effective delivery of training to groups within their existing unit size, using a trainer to trainee ratio of 1:20. Seamless delivery was made possible using a freestanding headset solution and an enterprise hardware version, allowing remote fleet management, software upload, and updates. Familiarization with the technology occurred quickly despite limited previous trainee experience with VR technology. Notably, no occurrence of motion sickness, cybersickness or dizziness, a common concern among first-time VR users, has been reported [ 53 , 54 ]. Taken together, the results from this trial suggest that PE represents a VR training solution that is suitable for group-delivered training in the workplace context.

Engagement and User Acceptance of a Novel Training Solution

PE represents a novel training approach, not because of the intrinsic strategies included within the platform but rather by delivering practical training in a diverse, engaging, and immersive training environment that is private despite delivery in a group setting.

An important feature of PE is the emphasis placed on the practical development of stress management skills [ 55 ]. Consistent with BattleSMART, PE adopts the perspective that stress-management skills are central to optimal human performance. Throughout the modules, language with an overt mental health tone has been avoided. Instead, the perspective is taken that stress management skills should be considered like any other skill that contributes to healthy rounded performance in the workplace. Throughout the modules, the trainee is encouraged to consider stress management skills as general life skills that can be usefully applied not just to major stressors but also in response to small or mundane day-to-day challenges. The effort to recast validated skills within the platform was broadly received by the training audience.

In addition to the practical challenges of implementation, critical predictors for the future use and acceptance of new technologies in education are perceived usability and usefulness [ 44 ]. User engagement is particularly relevant for the transition of digital mental health interventions into real-world practice is user engagement [ 56 , 57 ]. However, transitioning from an expert-led to a digital-training format is often associated with low engagement, shallow learning, and potential frustration [ 51 , 58 ]. Given the technological and framework novelty of PE as a training solution for stress management soft skills, critical elements related to technology acceptance were investigated in this case study.

User engagement was validated for the target training population, specifically for military staff undergoing initial employment training. Feedback from both trainees and trainers suggests a general level of enjoyment and high levels of engagement for all modules ( Figures 3 A and 6A). Engagement is further supported by self-report ratings on the UES-SF and its subscales (engagement, esthetic appeal, focused attention, and perceived usability) as well as data collection and response times within the headset, which indicate that users participated in the activities, including personal reflections, as intended. Although the UES-SF is not intended to compare scores across applications or empirically classify high and low ratings, positive ratings and general training participation provide evidence of end-user acceptance of the training platform [ 40 ].

Immersion and sense of presence within PE are important to validate, as 360° video and computer-generated interface activities were intentionally designed to generate relaxing (beach and forest scenes), interesting (at an aquarium, in a sports gym), distracting (at a rock concert), or confrontational (angry men) training environments. The terms “immersion” and “presence” are often used interchangeably. However, in the scientific literature, presence refers specifically to the subjective psychological response of being within the environment, whereas immersion describes objective inputs within the digital environment (eg, interactions with the surroundings and selection items) [ 59 , 60 ]. The cognitive skills and reflections practiced within PE benefit from a sense of presence, as research suggests that presence can prompt emotional responses and interactions with digital avatars and environments despite users being fully aware of the fictitious nature of the setting [ 61 - 63 ]. A sense of presence has also been linked to improved training efficacy in digital-training applications [ 64 , 65 ]. All modules of PE generated a sense of presence (average rating of 5 on a 7-point scale; 7=high, 0=low), and the training environments triggered emotional states as intended ( Figure 3 ). In support of VR technology, immersion and privacy were rated as valuable and specifically mentioned as useful elements of the platform ( Figure 3 ). An important element connected to trainee engagement with the subject matter (specifically, emotional, thought awareness, and memory recall activities) was the affordance of privacy in a group setting provided by the VR headset. Existing research into the use of VR in education and training has shown the benefits of immersive VR over 2D screen delivery in areas of increased relaxation and arousal, motivation, engagement, and interest [ 34 , 66 - 68 ]. Although future research is required to validate the outcomes for PE platform in direct comparison to a screen-delivered training tool, it seems unlikely that trainees would feel an equivalent sense of presence, privacy, and engagement in a room and training setting of 20 trainees.

Training Outcomes

This study demonstrates the perceived usefulness of PE training, directly and indirectly, through self-report of improved skill competency and enhanced knowledge for each module ( Figure 5 ). Both training staff and trainees reported that PE provided useful and effective practical training and represented a valuable addition to the existing program, particularly via the provision of biofeedback, as well as the use of immersive and interactive training components. In pretraining surveys, ADF members indicated a relatively high level of understanding of the benefits and familiarity with concepts related to stress management skills. These findings are consistent with reports of military personnel who respond positively to the use of stress management techniques [ 69 ]. This background knowledge is likely to be, at least in part, due to previous exposure to the ADF BattleSMART program, which provides comprehensive education on optimal emotional and behavioral outcomes, resiliency, and arousal reduction skills [ 31 ]. However, despite this existing awareness, ADF staff reported minimal application of stress management skills outside of the training. Trainees had heard of grounding and progressive muscle relaxation but had very little practical experience in using them.

Modules 2 and 3 (controlled breathing and progressive muscle relaxation) aimed to develop skills that reduce the physiological effects of stress. The training outcomes included improved relaxation states and reduced respiratory rates ( Figures 4 A and 4B). Trainees were able to effectively reduce their respiration during the controlled breathing module, which is comparable with the results of our previous pilot trial [ 39 ]. Gaining conscious control, reducing breathing rates, and increasing relaxation are elements associated with effective stress management and reduced stress [ 70 , 71 ].

Modules 4 and 5 (grounded and emotional acceptance) provide training on cognitive skills; thus, skill competency and training outcomes for these modules were difficult to objectively quantify and compare with existing training approaches. There is growing literature on the relationship between mindfulness, psychological health, and stress reduction [ 72 ]. As elements of grounding overlap with mindfulness, the State Mindfulness Scale was administered, and an increase was observed following module 4 training. Although no differences between pre- and posttraining mindfulness states were observed for module 5, this may be due to the specific items used within the State Mindfulness Scale. Items within the body subdomain may not be relevant to training in emotion identification and acceptance. Although increased state mindfulness suggests a beneficial impact on stress, further research would be useful in assessing the efficacy of these training modules (M4 and M5).

Overall, the data suggest a positive immediate impact of PE training across the four stress management skill areas. Further studies will be useful to assess whether these short-term effects translate effectively to skills consolidation, the application of skills to real-world contexts, and the long-term effects of behavior changes and stress outcomes.

Study Limitations

The current trial was intended to assess the usability, feasibility, and suitability of PE training within its target training population in a real-world context. It should be noted that this is a case study within the ADF, and thus the findings may not be generalizable to other training settings, user populations, or training tools. Given the limited amount of investigation into the effective implementation of VR technology in the workplace, the results of this study provide valuable information on how to effectively integrate VR training into the workplace. Although specific to the ADF, we would view this as a useful starting point for any large organization interested in using VR within their training continuum. This work is intended to be the first step, with future studies required to document training efficacy. In particular, future research should investigate the number of iterations of VR exposure to develop skill mastery, the rate of skill degradation, how effectively these skills can be incorporated into real-world performance, and any effects on long-term mental health outcomes and workplace performance. Software and hardware were updated iteratively in response to the identified issues across the project and differed slightly between trial locations. To mitigate self-report bias, a conscious effort was made to brief participants that their input and responses were completely anonymous and that their input was being sought to improve the development of the application. Unfortunately, not all of the modules could be tested at all trial locations owing to staff availability and last-minute changes resulting from COVID-19 restrictions. As a result, trainee response numbers varied across modules (all numbers are reported). Despite this limitation, the multilocation trial approach resulted in participant numbers that far exceeded numbers generally seen within the VR literature, provided consistent findings for ADF service members across varying branches and locations, and provided balanced feedback for this level of evaluation.

Conclusions

This study found that the PE platform was feasible, implementable, and acceptable for stress management skills training within the ADF. Although many other studies have only assessed training solutions in controlled study environments, our work shows that virtual-reality and biofeedback technology can support training in real-world workplace settings. The ability of the PE platform to generate a private and immersive environment within a group setting provides a valuable proposition for the use of VR for this type of cognitive training. Engagement with the training platform is likely connected to the use of a targeted training framework as well as an approach and philosophy that is aligned with the overarching organizational values and practical requirements.

Acknowledgments

This study was funded by the Australian Department of Defense through the Defense Innovation Hub open-submission process. We would like to acknowledge all members of the Australian Defense Force who contributed to and participated in the studies, university staff members, and software developers at JumpGate who assisted with study support and implementation.

Data Availability

The datasets generated and analyzed in this study are available from the corresponding author upon reasonable request.

Conflicts of Interest

None declared.

Additional information and self-report items.

Promotional video for the Performance Edge software recorded within the headset (it is advised that the recording may result in motion sickness).

Module 1 walk-through start. This video was recorded as a screen capture within the virtual reality headset and shows the beginning of module 1 (thoughts, emotions, and behaviors). It is advised that the recording may result in motion sickness.

Waller M, Treloar SA, Sim MR, McFarlane AC, McGuire AC, Bleier J, et al. Traumatic events, other operational stressors and physical and mental health reported by Australian Defence Force personnel following peacekeeping and war-like deployments. BMC Psychiatry 2012 Jul 26;12:88 [ https://bmcpsychiatry.biomedcentral.com/articles/10.1186/1471-244X-12-88 ] [ CrossRef ] [ Medline ]
Garbarino S, Cuomo G, Chiorri C, Magnavita N. Association of work-related stress with mental health problems in a special police force unit. BMJ Open 2013 Jul 19;3(7):e002791 [ https://bmjopen.bmj.com/lookup/pmidlookup?view=long&pmid=23872288 ] [ CrossRef ] [ Medline ]
Hemi A, Sopp MR, Schäfer SK, Michael T, Levy-Gigi E. Adaptive responding to prolonged stress exposure: a binational study on the impact of flexibility on latent profiles of cognitive, emotional and behavioural responses to the COVID-19 pandemic. Health Soc Care Community 2022 Nov;30(6):e6163-e6174 [ https://europepmc.org/abstract/MED/36184793 ] [ CrossRef ] [ Medline ]
Nisar SK, Rasheed MI. Stress and performance: investigating relationship between occupational stress, career satisfaction, and job performance of police employees. J Public Aff 2019 Jun 24;20(1):e1986 [ CrossRef ]
Motowidlo SJ, Packard JS, Manning MR. Occupational stress: its causes and consequences for job performance. J Appl Psychol 1986;71(4):618-629 [ CrossRef ]
Jones S. Describing the mental health profile of first responders: a systematic review [formula: see text]. J Am Psychiatr Nurses Assoc 2017 May;23(3):200-214 [ CrossRef ] [ Medline ]
Lehrer PM, Woolfolk RL. Principles and Practice of Stress Management. New York, NY: Guilford Publications; 2021.
Bouchard S, Baus O, Bernier F, McCreary DR. Selection of key stressors to develop virtual environments for practicing stress management skills with military personnel prior to deployment. Cyberpsychol Behav Soc Netw 2010 Feb;13(1):83-94 [ CrossRef ] [ Medline ]
Rizzo A, Shilling R. Clinical virtual reality tools to advance the prevention, assessment, and treatment of PTSD. Eur J Psychotraumatol 2017 Jan 16;8(sup5):1414560 [ https://europepmc.org/abstract/MED/29372007 ] [ CrossRef ] [ Medline ]
Worthen M, Cash E. Stress management. In: StatPearls. Treasure Island, FL: StatPearls Publishing; Aug 14, 2023.
Murphy LR. Stress management in work settings: a critical review of the health effects. Am J Health Promot 1996;11(2):112-135 [ CrossRef ] [ Medline ]
Tabibnia G, Radecki D. Resilience training that can change the brain. Consult Psychol J Pract Res 2018;70(1):59-88 [ CrossRef ]
Jha AP, Morrison AB, Parker SC, Stanley EA. Practice is protective: mindfulness training promotes cognitive resilience in high-stress cohorts. Mindfulness 2016 Jan 21;8:46-58 [ CrossRef ]
Precious D, Lindsay A. Mental resilience training. J R Army Med Corps 2019 Apr;165(2):106-108 [ CrossRef ] [ Medline ]
Mattingly V, Kraiger K. Can emotional intelligence be trained? A meta-analytical investigation. Hum Resour Manag Rev 2019 Jun;29(2):140-155 [ CrossRef ]
Khoury B, Sharma M, Rush SE, Fournier C. Mindfulness-based stress reduction for healthy individuals: a meta-analysis. J Psychosom Res 2015 Jun;78(6):519-528 [ CrossRef ] [ Medline ]
Ma X, Yue ZQ, Gong ZQ, Zhang H, Duan NY, Shi YT, et al. The effect of diaphragmatic breathing on attention, negative affect and stress in healthy adults. Front Psychol 2017 Jun 06;8:874 [ https://europepmc.org/abstract/MED/28626434 ] [ CrossRef ] [ Medline ]
Crawford C, Wallerstedt DB, Khorsan R, Clausen SS, Jonas WB, Walter JAG. A systematic review of biopsychosocial training programs for the self-management of emotional stress: potential applications for the military. Evid Based Complement Alternat Med 2013;2013:747694 [ https://doi.org/10.1155/2013/747694 ] [ CrossRef ] [ Medline ]
Birrer D, Morgan G. Psychological skills training as a way to enhance an athlete's performance in high-intensity sports. Scand J Med Sci Sports 2010 Oct 14;20 Suppl 2:78-87 [ CrossRef ] [ Medline ]
Zaccaro A, Piarulli A, Laurino M, Garbella E, Menicucci D, Neri B, et al. How breath-control can change your life: a systematic review on psycho-physiological correlates of slow breathing. Front Hum Neurosci 2018 Sep 7;12:353 [ https://europepmc.org/abstract/MED/30245619 ] [ CrossRef ] [ Medline ]
Holman D, Johnson S, O'Connor E. Stress management interventions: improving subjective psychological well-being in the workplace. In: Diener E, Oishi S, Tay L, editors. Handbook of Well-Being. Salt Lake City, UT: DEF Publishers; 2018.
Adler AB, Williams J, McGurk D, Moss A, Bliese PD. Resilience training with soldiers during basic combat training: randomisation by platoon. Appl Psychol Health Well Being 2015 Mar;7(1):85-107 [ CrossRef ] [ Medline ]
Cornum R, Matthews MD, Seligman ME. Comprehensive soldier fitness: building resilience in a challenging institutional context. Am Psychol 2011 Jan;66(1):4-9 [ CrossRef ] [ Medline ]
Guo D, Sun L, Yu X, Liu T, Wu L, Sun Z, et al. Mindfulness-based stress reduction improves the general health and stress of Chinese military recruits: a pilot study. Psychiatry Res 2019 Nov;281:112571 [ CrossRef ] [ Medline ]
Taylor MK, Rolo C, Stump J, Mayo J, Hernandez LM, Gould DR. Psychological strategies during military training are linked to resilience in US navy explosive ordnance disposal operators. J Spec Oper Med 2019;19(1):61-65 [ CrossRef ] [ Medline ]
Obuobi-Donkor G, Eboreime E, Bond J, Phung N, Eyben S, Hayward J, et al. An e-mental health solution to prevent and manage posttraumatic stress injuries among first responders in Alberta: protocol for the implementation and evaluation of text messaging services (Text4PTSI and Text4Wellbeing). JMIR Res Protoc 2022 Apr 25;11(4):e30680 [ https://www.researchprotocols.org/2022/4/e30680/ ] [ CrossRef ] [ Medline ]
Aidman E, Fogarty GJ, Crampton J, Bond J, Taylor P, Heathcote A, et al. An app-enhanced cognitive fitness training program for athletes: the rationale and validation protocol. Front Psychol 2022 Aug 30;13:957551 [ https://europepmc.org/abstract/MED/36110271 ] [ CrossRef ] [ Medline ]
Aidman E. Cognitive fitness framework: towards assessing, training and augmenting individual-difference factors underpinning high-performance cognition. Front Hum Neurosci 2019 Jan 14;13:466 [ https://europepmc.org/abstract/MED/32009919 ] [ CrossRef ] [ Medline ]
Albertella L, Kirkham R, Adler AB, Crampton J, Drummond SPA, Fogarty GJ, et al. Building a transdisciplinary expert consensus on the cognitive drivers of performance under pressure: An international multi-panel Delphi study. Front Psychol 2022;13:1017675 [ https://europepmc.org/abstract/MED/36755983 ] [ CrossRef ] [ Medline ]
Fogarty GJ, Crampton J, Bond J, Zaichkowsky LD, Taylor P, Aidman E. Neurobiological foundations of cognitive fitness in high-performance applications. In: Boyle GJ, Northoff G, Barbey AK, Fregni F, Jahanshahi M, Pascual-Leone A, et al, editors. The Sage Handbook of Cognitive and Systems Neuroscience Cognitive Systems, Development and Applications. Thousand Oaks, CA: SAGE Publications; 2023.
Cohn A, Hodson S, Crane M. Resilience training in the Australian Defence Force. InPsych 2010;32(2)
Garzón J, Acevedo J, Pavón J, Baldiris S. Promoting eco-agritourism using an augmented reality-based educational resource: a case study of aquaponics. Interact Learn Environ 2020 Jan 13;30(7):1200-1214 [ CrossRef ]
Mikropoulos TA, Chalkidis A, Katsikis A, Emvalotis A. Students' attitudes towards educational virtual environments. Educ Inf Technol 1998;3:137-148 [ https://link.springer.com/article/10.1023/A:1009687025419#citeas ] [ CrossRef ]
Jensen L, Konradsen F. A review of the use of virtual reality head-mounted displays in education and training. Educ Inf Technol 2017 Nov 25;23:1515-1529 [ CrossRef ]
Pallavicini F, Argenton L, Toniazzi N, Aceti L, Mantovani F. Virtual reality applications for stress management training in the military. Aerosp Med Hum Perform 2016 Dec 01;87(12):1021-1030 [ CrossRef ] [ Medline ]
Prinsloo GE, Derman WE, Lambert MI, Laurie Rauch HG. The effect of a single session of short duration biofeedback-induced deep breathing on measures of heart rate variability during laboratory-induced cognitive stress: a pilot study. Appl Psychophysiol Biofeedback 2013 Jun;38(2):81-90 [ CrossRef ] [ Medline ]
McCraty R, Atkinson M. Resilience training program reduces physiological and psychological stress in police officers. Glob Adv Health Med 2012 Nov;1(5):44-66 [ https://europepmc.org/abstract/MED/27257532 ] [ CrossRef ] [ Medline ]
Hourani L, Tueller S, Kizakevich P, Lewis G, Strange L, Weimer B, et al. Toward preventing post-traumatic stress disorder: development and testing of a pilot predeployment stress inoculation training program. Mil Med 2016 Sep;181(9):1151-1160 [ CrossRef ] [ Medline ]
Kluge MG, Maltby S, Walker N, Bennett N, Aidman E, Nalivaiko E, et al. Development of a modular stress management platform (Performance Edge VR) and a pilot efficacy trial of a bio-feedback enhanced training module for controlled breathing. PLoS One 2021 Feb 2;16(2):e0245068 [ https://dx.plos.org/10.1371/journal.pone.0245068 ] [ CrossRef ] [ Medline ]
O’Brien HL, Cairns P, Hall M. A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form. Int J Hum Comput Stud 2018 Apr;112:28-39 [ CrossRef ]
Witmer BG, Singer MJ. Measuring presence in virtual environments: a presence questionnaire. Presence Teleoperators Virtual Environ 1998;7(3):225-240 [ https://direct.mit.edu/pvar/article-abstract/7/3/225/92643/Measuring-Presence-in-Virtual-Environments-A?redirectedFrom=fulltext ]
Tanay G, Bernstein A. State Mindfulness Scale (SMS): development and initial validation. Psychol Assess 2013 Dec;25(4):1286-1299 [ CrossRef ] [ Medline ]
Crist DA, Rickard HC, Prentice-Dunn S, Barker HR. The Relaxation Inventory: self-report scales of relaxation training effects. J Pers Assess 1989;53(4):716-726 [ CrossRef ] [ Medline ]
Granić A, Marangunić N. Technology acceptance model in educational context: a systematic literature review. Br J Educ Technol 2019 Jul 09;50(5):2572-2593 [ CrossRef ]
Marangunić N, Granić A. Technology acceptance model: a literature review from 1986 to 2013. Univ Access Inf Soc 2014 Feb 16;14:81-95 [ CrossRef ]
Ahir K, Govani K, Gajera R, Shah M. Application on virtual reality for enhanced education learning, military training and sports. Augment Hum Res 2019 Nov 29;5:7 [ CrossRef ]
Liu X, Zhang J, Hou G, Wang Z. Virtual reality and its application in military. IOP Conf Ser Earth Environ Sci 2018 Jul 17;170:032155 [ CrossRef ]
Thompson SR, Dobbins S. The applicability of resilience training to the mitigation of trauma-related mental illness in military personnel. J Am Psychiatr Nurses Assoc 2018 Nov 15;24(1):23-34 [ CrossRef ] [ Medline ]
Hourani LL, Council CL, Hubal RC, Strange LB. Approaches to the primary prevention of posttraumatic stress disorder in the military: a review of the stress control literature. Mil Med 2011 Jul;176(7):721-730 [ CrossRef ] [ Medline ]
Gaggioli A, Pallavicini F, Morganti L, Serino S, Scaratti C, Briguglio M, et al. Experiential virtual scenarios with real-time monitoring (interreality) for the management of psychological stress: a block randomized controlled trial. J Med Internet Res 2014 Jul 08;16(7):e167 [ https://www.jmir.org/2014/7/e167/ ] [ CrossRef ] [ Medline ]
Fleming T, Bavin L, Lucassen M, Stasiak K, Hopkins S, Merry S. Beyond the trial: systematic review of real-world uptake and engagement with digital self-help interventions for depression, low mood, or anxiety. J Med Internet Res 2018 Jun 06;20(6):e199 [ https://www.jmir.org/2018/6/e199/ ] [ CrossRef ] [ Medline ]
Butler DL, Sellbom M. Barriers to adopting technology for teaching and learning. Educause Q 2002;25(2):22-28 [ https://www.learntechlib.org/p/92849/ ]
Howard MC, Van Zandt EC. A meta-analysis of the virtual reality problem: unequal effects of virtual reality sickness across individual differences. Virtual Real 2021 May 11;25(4):1221-1246 [ CrossRef ]
Rebenitsch L, Owen C. Review on cybersickness in applications and visual displays. Virtual Real 2016 Apr 26;20(2):101-125 [ CrossRef ]
Varvogli L, Darviri C. Stress management techniques: evidence-based procedures that reduce stress and promote health. Health Sci J 2011 Apr;5(2):74-89 [ https://www.researchgate.net/publication/236685932_Stress_Management_Techniques_evidence-based_procedures_that_reduce_stress_and_promote_health_Varvogli_L_Darviri_C ]
Baumel A, Edan S, Kane JM. Is there a trial bias impacting user engagement with unguided e-mental health interventions? A systematic comparison of published reports and real-world usage of the same programs. Transl Behav Med 2019 Nov 25;9(6):1020-1033 [ CrossRef ] [ Medline ]
Borghouts J, Eikey E, Mark G, De Leon C, Schueller SM, Schneider M, et al. Barriers to and facilitators of user engagement with digital mental health interventions: systematic review. J Med Internet Res 2021 Mar 24;23(3):e24387 [ https://www.jmir.org/2021/3/e24387/ ] [ CrossRef ] [ Medline ]
Mrazek AJ, Mrazek MD, Cherolini CM, Cloughesy JN, Cynman DJ, Gougis LJ, et al. The future of mindfulness training is digital, and the future is now. Curr Opin Psychol 2019 Aug;28:81-86 [ CrossRef ] [ Medline ]
Bowman DA, McMahan RP. Virtual reality: how much immersion is enough? Computer 2007 Jul;40(7):36-43 [ CrossRef ]
Harris DJ, Bird JM, Smart PA, Wilson MR, Vine SJ. A framework for the testing and validation of simulated environments in experimentation and training. Front Psychol 2020 Mar 31;11:605 [ https://europepmc.org/abstract/MED/32296379 ] [ CrossRef ] [ Medline ]
Sanz FA, Olivier AH, Bruder G, Pettré J, Lécuyer A. Virtual proxemics: locomotion in the presence of obstacles in large immersive projection environments. In: Proceedings of the 2015 IEEE Virtual Reality (VR). 2015 Presented at: 2015 IEEE Virtual Reality (VR); March 23-27, 2015; Arles, France [ CrossRef ]
Meehan M, Razzaque S, Insko B, Whitton M, Brooks FPJ. Review of four studies on the use of physiological reaction as a measure of presence in stressful virtual environments. Appl Psychophysiol Biofeedback 2005 Sep;30(3):239-258 [ CrossRef ] [ Medline ]
Slater M, Antley A, Davison A, Swapp D, Guger C, Barker C, et al. A virtual reprise of the Stanley Milgram obedience experiments. PLoS One 2006 Dec 20;1(1):e39 [ https://dx.plos.org/10.1371/journal.pone.0000039 ] [ CrossRef ] [ Medline ]
Grassini S, Laumann K, Rasmussen Skogstad M. The use of virtual reality alone does not promote training performance (but sense of presence does). Front Psychol 2020 Jul 17;11:1743 [ https://europepmc.org/abstract/MED/32765384 ] [ CrossRef ] [ Medline ]
Lerner D, Mohr S, Schild J, Göring M, Luiz T. An immersive multi-user virtual reality for emergency simulation training: usability study. JMIR Serious Games 2020 Jul 31;8(3):e18822 [ https://games.jmir.org/2020/3/e18822/ ] [ CrossRef ] [ Medline ]
Cao L, Peng C, Hansberger JT. Usability and engagement study for a serious virtual reality game of lunar exploration missions. Informatics 2019 Oct 03;6(4):44 [ CrossRef ]
Riches S, Azevedo L, Bird L, Pisani S, Valmaggia L. Virtual reality relaxation for the general population: a systematic review. Soc Psychiatry Psychiatr Epidemiol 2021 Oct 13;56(10):1707-1727 [ https://europepmc.org/abstract/MED/34120220 ] [ CrossRef ] [ Medline ]
Andersen JP, Gustafsberg H. A training method to improve police use of force decision making: a randomized controlled trial. SAGE Open 2016 Apr 07;6(2) [ CrossRef ]
Stetz MC, Bouchard S, Wiederhold BK, Riva G, Folen RA. The receptiveness of stress management techniques by military personnel. Stud Health Technol Inform 2009;144:125-127 [ Medline ]
Kaspereen D. Relaxation intervention for stress reduction among teachers and staff. Int J Stress Manag 2012 Aug;19(3):238-250 [ CrossRef ]
Bouchard S, Bernier F, Boivin E, Morin B, Robillard G. Using biofeedback while immersed in a stressful videogame increases the effectiveness of stress management skills in soldiers. PLoS One 2012 Apr 27;7(4):e36169 [ https://dx.plos.org/10.1371/journal.pone.0036169 ] [ CrossRef ] [ Medline ]
Grossman P, Niemann L, Schmidt S, Walach H. Mindfulness-based stress reduction and health benefits: a meta-analysis. J Psychosomatic Res 2004 Jul;57(1):35-43 [ CrossRef ]

Abbreviations

Edited by A Mavragani; submitted 08.02.23; peer-reviewed by T Ong, T Cahill, S Cargill-Fealy; comments to author 31.07.23; revised version received 24.08.23; accepted 18.09.23; published 06.11.23

©Murielle G Kluge, Steven Maltby, Caroline Kuhne, Nicole Walker, Neanne Bennett, Eugene Aidman, Eugene Nalivaiko, Frederick Rohan Walker. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 06.11.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

Research on Air Mass Flow and Pressure Control Method for the Multi-Stack Fuel Cell System Based on Model Predictive Control 2023-01-7037

The multi-stack fuel cell system (MFCS) has the advantages of higher efficiency, stronger robustness and longer life, and could be widely used in high-power application scenarios such as automobiles, airplanes, trains, and ships. The appropriate air mass flow and air pressure have a crucial impact on the output power performance indicators of the MFCS. Considering that the designed integrated air supply system for the MFCS has significant gas supply hysteresis and strong coupling between the inlet air mass flow and air pressure of each stack, this paper identifies multiple steady-state operating points of the fuel cell system to obtain corresponding linear predictive models and establishes corresponding predictive control algorithms. The Model Predictive Control (MPC) algorithms are switched in real-time based on the current load throughout the entire C-WTVC (China World Transient Vehicle Cycle) working condition. The simulation results show that the designed MPC algorithm can control all inlet air flow and air pressure of the MFCS (20kW/70kW/120kW) within the error range of ± 2% of the expected target values, which is significantly better than the PID control algorithm.

SAE MOBILUS

Subscribers can view annotate, and download all of SAE's content. Learn More »

Access SAE MOBILUS »

Researchers Discover New Vulnerability in Large Language Models

Share on Facebook (opens in new window)
Share on Twitter (opens in new window)
Share on LinkedIn (opens in new window)
Print this page
Share by email

Large language models (LLMs) use deep-learning techniques to process and generate human-like text. The models train on vast amounts of data from books, articles, websites and other sources to generate responses, translate languages, summarize text, answer questions and perform a wide range of natural language processing tasks.

This rapidly evolving artificial intelligence technology has led to the creation of both open- and closed-source tools, such as ChatGPT, Claude and Google Bard, enabling anyone to search and find answers to a seemingly endless range of queries. While these tools offer significant benefits, there is growing concern about their ability to generate objectionable content and the resulting consequences.

Researchers at Carnegie Mellon University’s School of Computer Science (opens in new window) (SCS), the CyLab Security and Privacy Institute (opens in new window) , and the Center for AI Safety in San Francisco (opens in new window) have uncovered a new vulnerability, proposing a simple and effective attack method that causes aligned language models to generate objectionable behaviors at a high success rate.

In their latest study, ‘ Universal and Transferable Adversarial Attacks on Aligned Language Models (opens in new window) ,’ CMU Associate Professors Matt Fredrikson (opens in new window) and Zico Kolter (opens in new window) , Ph.D. student Andy Zou (opens in new window) , and alumnus Zifan Wang found a suffix that, when attached to a wide range of queries, significantly increases the likelihood that both open- and closed-source LLMs will produce affirmative responses to queries that they would otherwise refuse. Rather than relying on manual engineering, their approach automatically produces these adversarial suffixes through a combination of greedy and gradient-based search techniques.

"At the moment, the direct harms to people that could be brought about by prompting a chatbot to produce objectionable or toxic content may not be especially severe,” said Fredrikson. “The concern is that these models will play a larger role in autonomous systems that operate without human supervision. As autonomous systems become more of a reality, it will be very important to ensure that we have a reliable way to stop them from being hijacked by attacks like these.”

In 2020, Fredrikson and fellow researchers from CyLab and the Software Engineering Institute (opens in new window) discovered vulnerabilities within image classifiers, AI-based deep-learning models that automatically identify the subject of photos. By making minor changes to the images, the researchers could alter how the classifiers viewed and labeled them.

Researchers' adversarial prompts can elicit arbitrary harmful behaviors from state-of-the-art commercial LLMs with high probability, demonstrating potentials for misuse.

Using similar methods, Fredrikson, Kolter, Zou, and Wang successfully attacked Meta’s open-source chatbot, tricking the LLM into generating objectionable content. While discussing their finding, Wang decided to try the attack on ChatGPT, a much larger and more sophisticated LLM. To their surprise, it worked.

“We didn’t set out to attack proprietary large language models and chatbots,” Fredrikson said. “But our research shows that even if you have a big trillion parameter closed-source model, people can still attack it by looking at freely available, smaller and simpler open-sourced models and learning how to attack those.”

By training the attack suffix on multiple prompts and models, the researchers have also induced objectionable content in public interfaces like Google Bard and Claud and in open-source LLMs such as Llama 2 Chat, Pythia, Falcon and others.

“Right now, we simply don’t have a convincing way to stop this from happening, so the next step is to figure out how to fix these models,” Fredrikson said.

Similar attacks have existed for a decade on different types of machine learning classifiers, such as in computer vision. While these attacks still pose a challenge, many of the proposed defenses build directly on top of the attacks themselves.

“Understanding how to mount these attacks is often the first step in developing a strong defense,” he said.

— Related Content —

Sandholm Earns AAAI Award for Artificial Intelligence That Benefits Humanity

The cover of Science featuring the Zoonomia story

Machine Learning Method Developed by CMU Researchers Illuminates Fundamental Aspects of Evolution

The Piper: Campus & Community News (opens in new window)
Official Events Calendar (opens in new window)

Welcome to the new NASA website. We’re working to continuously improve your web experience. If you don't find what you are looking for, give feedback .

Humans in Space

Earth & climate, the solar system, the universe, aeronautics, learning resources, news & events.

Rita Owens: Keeper of NASA’s Digital Knowledge

New Patterns in Mars’s Clouds Revealed by Volunteers

Tundra wetlands are shown in late spring at the Yukon Delta National Wildlife Refuge in Alaska.

NASA Flights Link Methane Plumes to Tundra Fires in Western Alaska

Search All NASA Missions
A to Z List of Missions
Upcoming Launches and Landings
Spaceships and Rockets
Communicating with Missions
James Webb Space Telescope
Hubble Space Telescope
Why Go to Space
Commercial Space
Destinations
Living in Space
Explore Earth Science
Earth, Our Planet
Earth Science in Action
Earth Multimedia
Earth Science Researchers
Pluto & Dwarf Planets
Asteroids, Comets & Meteors
The Kuiper Belt
The Oort Cloud
Skywatching
The Search for Life in the Universe
Black Holes
The Big Bang
Dark Energy & Dark Matter
Earth Science
Planetary Science
Astrophysics & Space Science
The Sun & Heliophysics
Biological & Physical Sciences
Lunar Science
Citizen Science
Astromaterials
Aeronautics Research
Human Space Travel Research
Science in the Air
NASA Aircraft
Flight Innovation
Supersonic Flight
Air Traffic Solutions
Green Aviation Tech
Drones & You
Technology Transfer & Spinoffs
Space Travel Technology
Technology Living in Space
Manufacturing and Materials
Science Instruments
For Kids and Students
For Educators
For Colleges and Universities
For Professionals
Science for Everyone
Requests for Exhibits, Artifacts, or Speakers
STEM Engagement at NASA
NASA's Impacts
Centers and Facilities
Directorates
Organizations
People of NASA
Internships
Our History
Doing Business with NASA
Get Involved
Aeronáutica
Ciencias Terrestres
Sistema Solar
All NASA News
Video Series on NASA+
Newsletters
Social Media
Media Resources
Upcoming Launches & Landings
Virtual Events
Sounds and Ringtones
Interactives
STEM Multimedia

NASA’s Webb Telescope Improves Simulation Software

This enhanced image of the Jovian moon Ganymede was obtained by the JunoCam imager aboard NASA's Juno spacecraft during the mission's June 7, 2021, flyby of the icy moon on Juno's 34th pass close to Jupiter.

Salts and Organics Observed on Ganymede’s Surface by NASA’s Juno

President William J. “Bill” Clinton introduces the STS-95 crew

25 Years Ago: STS-95, John Glenn Returns to Space

Frank Rubio, un hombre de pelo y ojos oscuros y con lentes, sonrÃe y tiene los brazos cruzados. Va vestido con un polo oscuro y pantalones khaki. DetrÃ¡s suyo se ve la atmÃ³sfera de la Tierra a travÃ©s de las ventanas de observaciÃ³n de la cÃºpula.

Misión récord de astronauta ayuda a planificar viajes al espacio profundo

This image shows mixing at the interface of two shear-driven systems in a ground analog of hardware for Ring-Sheared Drop-IBP, which tests computer models for the behavior of high-concentration protein fluids.

Space Station Science Highlights: Week of May 15, 2023

image of an astronaut taking photos inside the cupola of the space station with views of Earth in backdrop

Earth Observation From the Space Station

Engineers joined the two main components of NISAR in an ISRO clean room in Bengaluru, India, in June, 2023. The payload arrived from NASA's Jet Propulsion Laboratory in March, 2023, while the bus was built at the ISRO facility.

Powerful NASA-ISRO Earth Observing Satellite Coming Together in India

Thanks to a wet winter, California saw tremendous gains in the amount of water in the San Joaquin, Sacramento, and Tulare river basins (outlined in yellow) from October 2022 to March 2023, GRACE-FO data shows.

US-German Satellites Show California Water Gains After Record Winter

NASA Delivers First Flight Hardware to ESA for Lunar Pathfinder

NASA’s Curiosity Mars rover used its black-and-white navigation cameras to capture panoramas of “Marker Band Valley” at two times of day on April 8.

NASA’s Curiosity Captures Martian Morning, Afternoon in New ‘Postcard’

NASA scientists have made the first observation of a polar cyclone on Uranus.

NASA Scientists Make First Observation of a Polar Cyclone on Uranus

View of Didymos-Dimorphos asteroid system after DART impact. Webb's view, hued in red, shows a bright center and darker streaks emanating outward from it.

Webb, DART Missions Win AIAA 2023 Premier Awards

A planet covered with volcanic outbursts and spots of water takes up most of the left-hand foreground. In the distance, against the black background, is a faint blue planet.

NASA’s Spitzer, TESS Find Potentially Volcano-Covered Earth-Size World

Ames Spaceflight Projects Office (SPO)

NASA’s Curiosity Rover Clocks 4,000 Days on Mars

Artist concept of an In-situ Resource Utilization (ISRU) demonstration on the Moon. Many technologies in six priority areas encompassed by NASA’s Lunar Surface Innovation Initiative will need testing, such as advancing ISRU technologies that could lead to future production of fuel, water, or oxygen from local materials, expanding exploration capabilities.

NASA Invites Stakeholders to STMD’s LIFT-1 Industry Forum

Joby Aviation's electric vertical takeoff and landing (eVTOL) aircraft is pictured at Edwards Air Force Base in Edwards, California.

Government and Industry Collaboration Leads to First Air Taxi Delivery 

Joe Piotrowski, Jr. smiles in front of NASA Armstrongs Airborne Science aircraft.

Joe Piotrowski, Jr.: One Engineer’s (Flight) Path to Airborne Science

Children walk around the Darkstar prop made for the Top Gun: Maverick. The prop was on display at the 2022 Aerospace Valley Open House, Air Show, & STEM Expo in Edwards, California.

Forget Movie Magic, NASA Armstrong has the Real Thing

Two people wearing blue hooded parkas, waist deep in snow, manipulate tubes holding snow samples. equipment lies on the snow surface near them and in the background a line of dark trees

NASA Tech Lets Scientists See Snow-Water Through the Trees

A wide shot of a silvery piece of space hardware about the size of a car sitting on a silver platform. Three techs in white head-to-toe suits are pushing it from behind through a very large, arched doorway that leads to a dark room lined with dim lights.

Primary Instrument for NASA’s Roman Completed, Begins Tests

A white device that looks like a TV remote with a big, red button is held a few inches above two green, potted plants in a cardboard frame.

STELLA: NASA’s DIY Educational Gadget for Measuring Plant Health

Students from the AIAA Orange Country Section team of Irvine, California, display their rocket to news media and the public during Rocket Fair – an annual showcase event of NASA’s Student Launch competition April 2023.

NASA Announces Teams for 2024 Student Launch Challenge

NASA astronaut Frank Rubio uses a tool in his right hand as he activates a space biology experiment that is studying how weightlessness affects genetic expression in microbes.

Ciencia destacada del año en el espacio del astronauta Frank Rubio

Pruebas de la NASA con maniquí de Artemis I aportan información para futuras misiones tripuladas

NASA’s DART Data Validates Kinetic Impact as Planetary Defense Method

Emily Furfaro

Since NASA’s Double Asteroid Redirection Test (DART) successfully impacted its target nearly five months ago, on Sept. 26 — altering the orbit of the asteroid moonlet Dimorphos by 33 minutes — the DART team has been hard at work analyzing the data collected from the world’s first planetary defense test mission.

The DART mission employed an asteroid-deflection technique known as a “kinetic impactor,” which in simplest terms means smashing a thing into another thing — in this case, a spacecraft into an asteroid. From the data, the DART investigation team, led by the Johns Hopkins Applied Physics Laboratory (APL) in Laurel, Maryland, found that a kinetic impactor mission like DART can be effective in altering the trajectory of an asteroid, a big step toward the goal of preventing future asteroid strikes on Earth. These findings were published in four papers in the journal Nature.

“I cheered when DART slammed head on into the asteroid for the world’s first planetary defense technology demonstration, and that was just the start,” said Nicola Fox, associate administrator for the Science Mission Directorate at NASA Headquarters in Washington. “These findings add to our fundamental understanding of asteroids and build a foundation for how humanity can defend Earth from a potentially hazardous asteroid by altering its course.”

The first paper reports DART’s successful demonstration of kinetic impactor technology in detail : reconstructing the impact itself, reporting the timeline leading up to impact, specifying in detail the location and nature of the impact site, and recording the size and shape of Dimorphos.

The authors, led by Terik Daly, Carolyn Ernst, and Olivier Barnouin of APL, note DART’s successful autonomous targeting of a small asteroid, with limited prior observations, is a critical first step on the path to developing kinetic impactor technology as a viable operational capability for planetary defense.

This image depicts the footprint of the Double Asteroid Redirection Test (DART) spacecraft and its two long solar panels over the spot where it impacted asteroid Dimorphos.

Their findings show intercepting an asteroid with a diameter of around half a mile, such as Dimorphos, can be achieved without an advance reconnaissance mission, though advance reconnaissance would give valuable information for planning and predicting the outcome. What is necessary is sufficient warning time — several years at a minimum, but preferably decades. “Nevertheless,” the authors state in the paper, DART’s success “builds optimism about humanity’s capacity to protect the Earth from an asteroid threat.”

The yellow surface is a digital terrain model of the impact site made from DART images, and the rendering of the DART spacecraft depicts its position a few tens of microseconds before impact.

The second paper uses two independent approaches based on Earth-based lightcurve and radar observations. The investigation team, led by Cristina Thomas of Northern Arizona University, arrived at two consistent measurements of the period change from the kinetic impact : 33 minutes, plus or minus one minute. This large change indicates the recoil from material excavated from the asteroid and ejected into space by the impact (known as ejecta) contributed significant momentum change to the asteroid, beyond that of the DART spacecraft itself.

The key to kinetic impact is that the push to the asteroid comes not only from colliding spacecraft, but also from this ejecta recoil. The authors conclude: “To serve as a proof-of-concept for the kinetic impactor technique of planetary defense, DART needed to demonstrate that an asteroid could be targeted during a high-speed encounter and that the target’s orbit could be changed. DART has successfully done both.”

In the third paper, the investigation team, led by Andrew Cheng of APL, calculated the momentum change transferred to the asteroid as a result of DART’s kinetic impact by studying the change in the orbital period of Dimorphos. They found the impact caused an instantaneous slowing in Dimorphos’ speed along its orbit of about 2.7 millimeters per second — again indicating the recoil from ejecta played a major role in amplifying the momentum change directly imparted to the asteroid by the spacecraft. That momentum change was amplified by a factor of 2.2 to 4.9 (depending on the mass of Dimorphos), indicating the momentum change transferred because of ejecta production significantly exceeded the momentum change from the DART spacecraft alone.

This finding “[validates] the effectiveness of kinetic impact for preventing future asteroid strikes on the Earth,” the authors conclude.

DART’s scientific value goes beyond validating kinetic impactor as a means of planetary defense. By smashing into Dimorphos, the mission has broken new ground in the study of asteroids. DART’s impact made Dimorphos an “active asteroid” — a space rock that orbits like an asteroid but has a tail of material like a comet – which is detailed in the fourth paper led by Jian-Yang Li of the Planetary Science Institute.

Although scientists had proposed that some active asteroids are the result of impact events, until now no one had ever observed the activation of an asteroid. The DART mission activated Dimorphos under precisely known and carefully observed impact conditions , enabling the detailed study of the formation of an active asteroid for the first time.

“DART, as a controlled, planetary-scale impact experiment, provides a detailed characterization of the target, the ejecta morphology, and the entire ejecta evolution process,” the authors write. “DART will continue to be the model for studies of newly discovered asteroids that show activity caused by natural impacts.”

DART’s Legacy Begins

“We are so proud of the DART team and the investigation’s latest results,” said Jason Kalirai, Civil Space Mission Area Executive at APL. “With the core analysis activities starting after the impact of Dimorphos, the results demonstrate how successful the kinetic impactor technique can be — paving the way for a bright future for planetary defense.”

Johns Hopkins APL manages the DART mission for NASA’s Planetary Defense Coordination Office as a project of the agency’s Planetary Missions Program Office. The LICIACube project is managed by ASI Robotic Exploration Mission Office, with industrial contractor Argotec S.r.I. and a scientific team from the National Institute of Astrophysics, Polytechnic University of Milan, the University of Bologna, the University of Naples Parthenope, and CNR-IFAC.

For more information about DART, visit:

https://www.nasa.gov/dartmission

Written by Ajai Raj Johns Hopkins Applied Physics Laboratory

Media Contacts:

Karen Fox / Alana Johnson Headquarters, Washington 202-358-1600 [email protected] / [email protected]

Justyna Surowiec Johns Hopkins Applied Physics Laboratory 240-302-9268 [email protected]

IMAGES

FREE 27+ Research Paper Formats in PDF
🎉 Research paper methodology example. Methology Sample for You. 2019-01-21
Example Of Thesis Methodology Chapter
😝 Sample methodology section of a research paper. Sample of Methodology
Example Method Paper / 28 Research Paper Formats
College essay: Apa format apa research paper sample

VIDEO

Research Process
Choosing A Research Topic
Econ-203A Research Methods_1 solved past Papers 2021and 22 4th semester
Research Paper Methodology
PhD Pro-Tips: Mastering The Art Of Planning Ahead!
Your Key To Publishing Success

COMMENTS

Research Methods
Research methods are specific procedures for collecting and analyzing data. Developing your research methods is an integral part of your research design. When planning your methods, there are two key decisions you will make. First, decide how you will collect data. Your methods depend on what type of data you need to answer your research question:
What Is a Research Methodology?
Home Knowledge Base Dissertation What Is a Research Methodology? | Steps & Tips What Is a Research Methodology? | Steps & Tips Published on August 25, 2022 by Shona McCombes and Tegan George. Revised on June 22, 2023. Your research methodology discusses and explains the data collection and analysis methods you used in your research.
How to Write Your Methods
A constant principle of rigorous science A methods section that enables other researchers to understand and replicate your results is a constant principle of rigorous, transparent, and Open Science. Aim to be thorough, even if a particular journal doesn't require the same level of detail . Reproducibility is all of our responsibility.
How to Write an APA Methods Section
The methods section of an APA style paper is where you report in detail how you performed your study. Research papers in the social and natural sciences often follow APA style. This article focuses on reporting quantitative research methods.
PDF Methodology Section for Research Papers
Methodology Section for Research Papers The methodology section of your paper describeshow your research was conducted. This information allows readers to check whether your approach is accurate and dependable. A good methodology can help increase the reader's trust in your findings. First, we will define and differentiate quantitative and ...
How to Write a Research Paper
A research paper is a piece of academic writing that provides analysis, interpretation, and argument based on in-depth independent research. Research papers are similar to academic essays, but they are usually longer and more detailed assignments, designed to assess not only your writing skills but also your skills in scholarly research.
Writing the Research Paper
UCLA Library Research Guides Advanced Research Methods Writing the Research Paper Advanced Research Methods Research advice for upper division undergraduates and graduates in the social sciences and humanities. Before Writing the Paper Find a topic. Try to find a subject that really interests you.
Organizing Your Social Sciences Research Paper
There are two main groups of research methods in the social sciences: The empirical-analytical group approaches the study of social sciences in a similar manner that researchers study the natural sciences.
Essential components of methods papers
Here, the key components of a methods paper are summarized: (i) methods description, (ii) performance standards, (iii) applicability domains, (iv) evidence for advances compared to the state-of-the-art, (v) exemplification of the method by practical application.
What makes a Nature Methods paper
Scope. Nature Methods' mission is to champion method and tool development research in the basic life sciences. Therefore, we consider methods papers with a primarily clinical, diagnostic or ...
How to Write Research Methodology: Overview, Tips, and Techniques
Chief Data Scientist & Head of Content The main aim of research is not merely to gather information. Instead, it goes beyond that. The true goal of research is to seek answers to previously unanswered questions to contribute to the body of knowledge in a discipline according to Goddard and Melville (2001, p.1).
Organizing Academic Research Papers: 6. The Methodology
I. Groups of Research Methods. There are two main groups of research methods in the social sciences: The empirical-analytical group approaches the study of social sciences in a similar manner that researchers study the natural sciences. This type of research focuses on objective knowledge, research questions that can be answered yes or no, and operational definitions of variables to be measured.
How to Write a Research Methodology in 4 Steps
Learn how to write a strong methodology chapter that allows readers to evaluate the reliability and validity of the research. A good methodology chapter incl...
How to write the Methods section of a research paper
1. Explanation: Usually, each paragraph or subsection of the Methods section talks about a specific experiment.
Papers on research methods: The hidden gems of the research literature
First published: 15 November 2019 https://doi.org/10.1002/nur.21995 Citations: 1 Sections PDF Tools Share As our editorial board pondered the focus for a special issue, which our journal has not had previously, we were struck by a dominant feature of the top ten list of most downloaded articles.
Writing a Research Paper Introduction
Table of contents. Step 1: Introduce your topic. Step 2: Describe the background. Step 3: Establish your research problem. Step 4: Specify your objective (s) Step 5: Map out your paper. Research paper introduction examples. Frequently asked questions about the research paper introduction.
Research Methodology
The research methodology is an important section of any research paper or thesis, as it describes the methods and procedures that will be used to conduct the research. It should include details about the research design, data collection methods, data analysis techniques, and any ethical considerations.
Literature review as a research methodology: An ...
This paper discusses literature review as a methodology for conducting research and offers an overview of different types of reviews, as well as some guidelines to how to both conduct and evaluate a literature review paper. ... but time-consuming approach. Another option could be to focus on the research method or findings, and a third option ...
Examples of Methodology in Research Papers (With Definition)
Example of a methodology in a research paper. The following example of a methodology in a research paper provides insight into the structure and content to consider when writing your own: This research article discusses the psychological and emotional impact of a mental health support program for employees. The program provided prolonged and ...
Research Methods
Definition: Research Methods refer to the techniques, procedures, and processes used by researchers to collect, analyze, and interpret data in order to answer research questions or test hypotheses. The methods used in research can vary depending on the research questions, the type of data that is being collected, and the research design.
Methodology in a Research Paper: Definition and Example
The methodology section of your research paper allows readers to evaluate the overall validity and reliability of your study and gives important insight into two key elements of your research: your data collection and analysis processes and your rationale for conducting your research.
What are the different types of research methodology?
Quantitative research methodologies refer to studies that require evidence of numerical or statistical data. Relevant fields that would use this research method, in particular, are maths-based subjects. Qualitative methodology refers to non-numerical, information-based data that often forms the basis of linguistic research papers.
Vignettes as a novel research tool in spiritual care: A methods paper
The purpose of this method paper is not to systematically review the literature on vignette research. Rather the authors aimed to introduce and discuss an example of a vignette as a novel research tool in spiritual care research based on findings of recently published reviews of vignettes in qualitative research (Murphy et al., 2021 ; Tremblay ...
SciSciNet: A large-scale open data lake for the science of science research
Research shows that the highest-impact papers in science tend to be grounded in exceptionally conventional ... The method feeds unique author names into a cultural consensus model of name-gender ...
CBE Unveils Renovated Controlled Environment Chamber, Expanding Future
CBE's controlled environment chamber has been used for research leading to hundreds of journal papers, including keystone work related to human response, indoor environments and mechanical systems. In this post we describe a recent renovation, and discuss past work and future directions.
Journal of Medical Internet Research
Background: Psychological stress-related injuries within first-responder organizations have created a need for the implementation of effective stress management training. Most stress management training solutions have limitations associated with scaled adoption within the workforce. For instance, those that are effective in civilian populations often do not align with the human performance ...
Research on Air Mass Flow and Pressure Control Method for the Multi
Research on Air Mass Flow and Pressure Control Method for the Multi-Stack Fuel Cell System Based on Model Predictive Control 2023-01-7037. ... TechSelect is a cost-effective subscription option to select and download 12-100 full-text Technical Papers per year. Find more information here. Standards & Publications. SAE MOBILUS; Standards;
Researchers Discover New Vulnerability in Large Language Models
The models train on vast amounts of data from books, articles, websites and other sources to generate responses, translate languages, summarize text, answer questions and perform a wide range of natural language processing tasks. This rapidly evolving artificial intelligence technology has led to the creation of both open- and closed-source ...
NASA's DART Data Validates Kinetic Impact as Planetary Defense Method
In the third paper, the investigation team, led by Andrew Cheng of APL, calculated the momentum change transferred to the asteroid as a result of DART's kinetic impact by studying the change in the orbital period of Dimorphos. They found the impact caused an instantaneous slowing in Dimorphos' speed along its orbit of about 2.7 millimeters ...

Ensure understanding, reproducibility and replicability

Why Methods Matter

A clear methods section impacts editorial evaluation and readers’ understanding, and is also the backbone of transparency and replicability.

What to include in your methods section

A constant principle of rigorous science

Imagine replicating your own work, years in the future

Visual aids for methods help when reading the whole paper

Ethical Considerations

Existing standards, checklists, guidelines, partners

Summary Writing tips

Don’t

Advanced Research Methods

Before Writing the Paper

Academic Expression

Organizing Your Social Sciences Research Paper

Importance of a Good Methodology Section

Structure and Writing Style

Writing Tip

Another Writing Tip

Yet Another Writing Tip

Essential components of methods papers

Publication types

Organizing Academic Research Papers: 6. The Methodology

Importance of a Good Methodology Section

Structure and Writing Style

Writing Tip

Another Writing Tip

Research Support

Using the Library

User Information

My Accounts

How to write the Methods section of a research paper

Related Reading

The correct way to report p values

How to choose the research methodology best suited for your study

Manuscript structure: How to convey your most important ideas through…

Trending Searches

Recent Searches

Have a language expert improve your writing

Writing a Research Paper Introduction | Step-by-Step Guide

Table of contents

Here's why students love Scribbr's proofreading services

Argumentative paper: Background information

Empirical paper: Describing previous research

Argumentative paper: Emphasize importance

Empirical paper: Relate to the literature

Argumentative paper: Thesis statement

Empirical paper: Research question and hypothesis

A faster, more affordable way to improve your paper

Cite this Scribbr article

Is this article helpful?

Jack Caulfield

Research Methods – Types, Examples and Guide

Research Methods

Types of Research Methods

Qualitative research Method

Quantitative Research Method

Mixed Method Research

Key Differences Between Research Methods

Examples of Research Methods

Applications of Research Methods

Purpose of Research Methods

When to Use Research Methods

Advantages of Research Methods

About the author

Muhammad Hassan

You may also like

Assignment – Types, Examples and Writing Guide

References in Research – Types, Examples and...

Figures in Research Paper – Examples and Guide

Delimitations in Research – Types, Examples and...

Qualitative Research – Methods, Analysis Types...

Research Findings – Types Examples and Writing...

SciSciNet: A large-scale open data lake for the science of science research

Background & Summary

Data selection and curation from MAG

Linking publication data with external sources

NIH funding

NSF funding

Patent citations to science