Writing screening criteria two reviewers can apply consistently - notes from Yila https://yila.ai https://yila.ai
Find a file
2026-10-22 19:33:33 +00:00
README.md Add research workflow guide 2026-10-22 19:33:33 +00:00

Writing Screening Criteria That Two Reviewers Can Apply the Same Way

Most literature reviews fail quietly at the screening stage. The search may be careful and the synthesis thoughtful, but if the rules for including or excluding a paper are vague, two people reading the same abstract will make different decisions, and the final set of studies will reflect mood and fatigue rather than a defensible question. This guide covers how to write screening criteria that are precise enough to be applied consistently, how to pilot and calibrate them, and how a source-linked assistant such as Yila can support the later reading stages without taking over the judgement that screening requires.

Start from the question, not from the papers

Screening criteria are a translation of your research question into rules. If the question is loose, the criteria will be loose. Before writing a single rule, restate the question in a structured form. Frameworks such as PICO (population, intervention, comparison, outcome), SPIDER (sample, phenomenon of interest, design, evaluation, research type) or PEO (population, exposure, outcome) exist precisely to force this clarity. You do not have to use any of them formally, but each element of your question should map to at least one inclusion rule.

For example, a question such as "Does peer tutoring help first-year students in mathematics?" leaves many decisions open. Which students count as first-year? Does tutoring delivered online count? What outcome is "help": grades, retention, confidence? Writing criteria is the moment to settle these decisions on purpose rather than letting them be settled implicitly by whichever papers you read first.

Write rules that can be checked from the record

A good criterion can be checked from the information available at that stage. At title and abstract screening, you usually have the title, the abstract, the keywords and the publication type. A rule such as "the study must use a validated outcome measure" is often impossible to apply from an abstract, so it belongs at full-text screening instead. Splitting criteria by stage avoids a common problem: reviewers guessing, and guessing differently.

Each rule should be phrased so that the answer is yes, no or unclear. "Participants are undergraduate students in their first year of study" is checkable. "Participants are relevant to higher education" is not. Where a judgement is genuinely needed, describe the boundary cases explicitly: "Foundation-year students count as first-year; postgraduate conversion courses do not."

Separate inclusion from exclusion and order them

It helps to list exclusion reasons in a fixed order and to record only the first reason that applies. A typical order might be: wrong publication type, wrong population, wrong intervention or exposure, wrong comparison, wrong outcome, wrong study design, duplicate. Recording the first applicable reason keeps the PRISMA flow diagram honest and makes disagreements easier to diagnose, because you can see exactly which rule two reviewers interpreted differently.

Keep a short definitions section beside the rules. If "intervention" includes structured study groups led by a trained peer but excludes informal study groups, write that down. If "outcome" includes end-of-term grades but not self-reported confidence, write that down too. Definitions are where most disagreements are actually resolved.

Pilot the criteria on a small, mixed sample

Before screening the full set, take a random sample of perhaps 30 to 50 records and have two reviewers screen them independently using the draft criteria. Random matters: if you pilot only on the first records returned by a database, you will see the most relevant papers and miss the awkward edge cases that cause trouble later.

Compare the decisions. Calculate simple agreement and, if you want a chance-corrected figure, Cohen's kappa. More important than the number, though, is the conversation about each disagreement. For every record where the reviewers differed, ask which rule produced the difference. Often the answer is that a rule was ambiguous, that a definition was missing, or that a criterion was being applied at the wrong stage. Revise the wording, record the change and the date, and pilot again on a fresh sample if agreement was poor.

Decide in advance how to handle "unclear"

At the title and abstract stage, the safest default is that unclear records go forward to full-text screening. Excluding on uncertainty is the most common way relevant studies disappear. The cost is a larger full-text set, but that cost is visible and manageable, whereas a wrongly excluded study is invisible.

At full-text screening, unclear should become rare. If a paper genuinely does not report enough information to decide, you can contact the authors, or you can exclude it with a recorded reason such as "insufficient information to assess population". What matters is that the decision is documented and consistent.

Resolve conflicts with a fixed procedure

Two independent reviewers will disagree on some records, and that is a feature rather than a failure. Agree beforehand how disagreements will be resolved: discussion between the two reviewers first, then a third reviewer if discussion does not settle it. Record the original decisions as well as the final one. This keeps an audit trail and lets you report agreement honestly in the methods section.

If you are working alone, as many students do, you can still introduce some independence. Re-screen a random subset of records a week later without looking at your earlier decisions, and compare. It is not the same as a second reviewer, but it will reveal whether your criteria depend on your mood on a given day.

Keep the criteria and the log together

Screening criteria change during a review more often than people admit. A rule turns out to be unworkable, a new type of study appears that nobody anticipated, or the supervisor narrows the scope. Each change should be dated and explained, and records screened before the change should be checked against the new rule. Storing the criteria document next to the search log and the screening spreadsheet makes this easy, and makes it obvious to a reader how the final set was produced.

Where an assistant helps, and where it should not

Automated tools are improving at prioritising records, flagging duplicates and suggesting likely relevance. Used carefully, they can reduce the time spent on obviously irrelevant records. They should not replace the documented human decision, especially for borderline cases, because the point of explicit criteria is that someone can check and reproduce the reasoning.

After screening, the work shifts to reading, extracting and writing, and this is where a source-linked assistant earns its place. Yila is built to turn papers and data into cited reviews, figures and slides, keeping each statement attached to the document it came from. When the included set is fixed, you can use it to draft extraction summaries or a first pass at the synthesis, then check each claim against the cited passage. Because the links stay intact, verifying the draft is a matter of following references rather than searching for them again.

A short checklist

  • Restate the research question in a structured form and map each element to a rule.
  • Split criteria into title/abstract rules and full-text rules.
  • Phrase each rule so the answer is yes, no or unclear.
  • Order exclusion reasons and record only the first that applies.
  • Add a definitions section for every term that caused hesitation.
  • Pilot on a random sample, discuss every disagreement and revise.
  • Send unclear records forward at the abstract stage.
  • Fix a conflict-resolution procedure before you start.
  • Date every change to the criteria and re-check earlier decisions.

Conclusion

Screening criteria are a small document with a large effect on the credibility of a review. When they are explicit, staged, piloted and logged, two reviewers can apply them the same way and a reader can see why each study is in or out. That consistency is what lets the later stages, including any drafting done with an assistant like Yila, rest on a set of studies chosen for good reasons.