README.Rmd

---
output: github_document
output_file: README.md
---

<!-- README.md is generated from README.Rmd. Please edit that file -->

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%",
  #eval = httr2::secret_has_key("AISCREENR_KEY")
  eval = FALSE
)
```


<a href="https://mikkelvembye.github.io/AIscreenR/"><img src="man/figures/AIscreenR_hex_light.png" align="right" width="180"/></a>

# AIscreenR: AI screening tools in R for systematic reviewing 

<!-- badges: start -->

[![R-CMD-check](https://github.com/MikkelVembye/AIscreenR/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/MikkelVembye/AIscreenR/actions/workflows/R-CMD-check.yaml) [![CRAN Version](https://www.r-pkg.org/badges/version/AIscreenR)](https://cran.r-project.org/package=AIscreenR) [![Monthly Downloads](https://cranlogs.r-pkg.org/badges/AIscreenR)](https://cranlogs.r-pkg.org/badges/AIscreenR) [![Total Downloads](https://cranlogs.r-pkg.org/badges/grand-total/AIscreenR)](https://cranlogs.r-pkg.org/badges/grand-total/AIscreenR)

<!-- badges: end -->

The goal of AIscreenR is to use AI tools to support screening processes (including title and abstract screening) in systematic reviews and related literature reviews. At the current stage, the main aim of the `AIscreenR` package is to use generative pre-trained transformer (GPT) models as second screeners of titles and abstracts or alternatively to reduce the number of references needed to be screened by humans. For validation measures and guidance on how to conduct reliable title and abstract screenings with GPT API models, see [Vembye et al. (2025)](https://psycnet.apa.org/record/2026-37236-001).

## Installation

Install the latest release from CRAN:

```{r, eval = FALSE}
install.packages("AIscreenR")
```

You can install the development version of AIscreenR from [GitHub](https://github.com/) with:

```{r}
# install.packages("remotes")
remotes::install_github("MikkelVembye/AIscreenR", build_vignettes = TRUE)
```

Installing the the development package along with its vignette may take several minutes. Using `build_vignettes = FALSE` speeds up the installation, but you cannot to access the package’s vignette in R afterward.

## Setting API key and checking rate limits

```{r, eval=FALSE}
# Find your api key via https://developers.openai.com/api/docs/quickstart#generate-an-api-key
# Thereafter, either encrypt it with the secret functions from the httr2 package
# see https://httr2.r-lib.org/reference/secrets.html or run set_api_key() 
# and then enter you key.
library(AIscreenR)
library(tibble)
library(dplyr)
library(future)

# Setting API
set_api_key()

# Obtain rate limits info (Default is "gpt-4o-mini")
rate_limits <- rate_limits_per_minute()
rate_limits
#> # A tibble: 1 × 3
#>   model       requests_per_minute tokens_per_minute
#>   <chr>                     <dbl>             <dbl>
#> 1 gpt-4o-mini               30000         150000000
```

## How to load RIS files
In this example we have downloaded the RIS files from the [EPPI-Reviewer](https://eppi.ioe.ac.uk/cms/Default.aspx?tabid=2914), and the RIS files comes from the review by Filges et al. (2015) on family-based interventions for drug abuse reduction in young people. The files are included in the package for tutorial purposes only.

```{r, eval = FALSE}

excl_path <- system.file("extdata", "excl_tutorial.ris", package = "AIscreenR")

# Loading RIS file via the read_ris_to_dataframe() function
ris_dat_excl <- read_ris_to_dataframe(excl_path) |> 
  select(studyid = eppi_id, title, abstract) |> 
  as_tibble() |> 
  mutate(
    human_code = 0 # Indicating exclusion
  )

incl_path <- system.file("extdata", "incl_tutorial.ris", package = "AIscreenR")

ris_dat_incl <- read_ris_to_dataframe(incl_path) |> 
  select(studyid = eppi_id, title, abstract) |> 
  as_tibble() |> 
  mutate(
    human_code = 1 # Indicating inclusion
  )

filges2015_dat <- 
  bind_rows(ris_dat_excl, ris_dat_incl) |> 
  # Removing references without abstracts since these can distort the accuracy of the screening
  filter_out(abstract == "NA") 

head(filges2015_dat, 10)
#> # A tibble: 10 × 4
#>    studyid title                                             abstract human_code
#>    <chr>   <chr>                                             <chr>         <dbl>
#>  1 9434957 Estimating and communicating prognosis in advanc… "Progno…          0
#>  2 9433838 Self-Directed Behavioral Family Intervention: Do… "Behavi…          0
#>  3 9431171 Frequency domain source localization shows state… "The to…          0
#>  4 9433968 A Review of: 'Kearney, C. A. (2010). Helping Chi… "The ar…          0
#>  5 9434460 Topographic differences in the adolescent matura… "STUDY …          0
#>  6 9433554 BOOK REVIEW                                       "The ar…          0
#>  7 9435130 Rapid improvement of depression and quality of l… "Backgr…          0
#>  8 9432040 Pictorial cognitive task solving and dynamics of… "AIMS: …          0
#>  9 9434093 Enhancing the Impact of Parent Training Through … "New an…          0
#> 10 9431505 EEG spectrum as information carrier               "Sponta…          0
```

```{r, echo=FALSE, include=FALSE}
head(filges2015_dat, 10)
```

Example of how to enter a prompt in R. Can also be done in Word (see vignette).

```{r, eval = FALSE}
prompt <- "Evaluate the following study based on the selection criteria
 for a systematic review on the effects of family-based interventions on drug abuse
 reduction for young people in treatment for non-opioid drug use.
 A family-based intervention (FFT) is equivalent to a behavior focused
 family therapy, where young people’s drug use is understood in relation to family
 behavior problems. Family-based interventions also includes manual-based family therapies as
 it targets young people and their families as a system throughout treatment, and thereby recognizes
 the important role of the family system in the development and treatment of young people’s drug use
 problems. FFT was developed in the late 1980s on request from the US National Institute on Drug Abuse
 (NIDA). The development of FFT was initially heavily inspired by the alcohol abuse program
 Community Reinforcement Approach (CRA), which was aimed at restructuring the environment
 to reinforce non-alcohol associated activities. FFT developed to have more emphasis on
 contingency contracting, impulse control strategies specific to drug use,
 and increased emphasis on involvement of family members in treatment.
 FFT is designed to accommodate diverse populations of youths with a variety of behavioral,
 cultural and individual preferences. FFT has evolved for use in severe behavioral disturbances
 known to co-exist with substance use and dependence, and the core interventions
 have been enhanced to address several mental health related problems commonly occurring
 as comorbid conditions in drug use treatment participant.  For each study,
 I would like you to assess:  1) Is the study about a family-based intervention,
 such as Functional Family Therapy, Multidimensional Family Therapy, or
 Behavioral Family Therapy? (Outpatient manual-based interventions of any
 duration delivered to young people and their families). If not, exclude study.
 2) Are the participants in outpatient drug treatment primarily
 for non-opioid drug use? 3) Are the participants within age 11–21?"
```

Approximate price of screening before running the screening.

```{r, eval = FALSE}
app_obj <- 
  approximate_price_gpt(
    data = filges2015_dat,
    prompt = prompt,
    studyid = studyid, # indicate the variable with the studyid in the data
    title = title, # indicate the variable with the titles in the data
    abstract = abstract, # indicate the variable with the abstracts in the data
    model = "gpt-4o-mini",
    rep = 1 
  )

app_obj
#> The approximate price of the (simple) screening will be around $0.0453.

app_obj$price_dollar
#> [1] 0.0453
app_obj$price_data
#> A tibble: 1 × 8
#>   promptid model       iterations prompt   completion_tokens input_price_dollar output_price_dollar
#>   <chr>    <chr>            <dbl> <chr>                <dbl>              <dbl>               <dbl>
#> 1 Prompt 1 gpt-4o-mini          1 Prompt 1             1706.             0.0443             0.00102
#> ℹ 1 more variable: total_price_dollar <dbl>
```

Example of how to conduct simple screening, returning `1` if a reference should be included, `0` if excluded, and `1.1` if uncertain.
The example is used for tutorial purposes only, and the results should not be used to draw conclusions about the accuracy of GPT models for screening. For a more thorough evaluation of the accuracy of GPT models for screening, see [Vembye et al. (2025)](https://psycnet.apa.org/record/2026-37236-001).

```{r, warning=FALSE, eval = FALSE}
# Subsetting the number of references to speed up the tutorial screening
plan(multisession)
test_obj <- 
  tabscreen_gpt(
    data = filges2015_dat[c(1:5, 238:242),],
    prompt = prompt, 
    studyid = studyid, # indicate the variable with the studyid in the data
    title = title, # indicate the variable with the titles in the data
    abstract = abstract, # indicate the variable with the abstracts in the data
    model = "gpt-4o-mini",
    reps = 10, # Number of times the same question is asked 
    incl_cutoff_upper = 0.1, # If GPT includes a reference in 10% or more of the iterations, it will be included
    # If the GPT model as no or little information to base its decision on, it will be more likely to include the reference. 
    # Setting overinclusive = TRUE means that if the model is uncertain about a reference, 
    # it will be included (i.e., coded as 1.1 instead of 0).
    overinclusive = TRUE 
  )
#> * The approximate price of the current (simple) screening will be around $0.0181.
#> Progress: ───────────────────────────────────────────────────────────────────────────────────── 100%
plan(sequential)
test_obj
#> 
#> Find the final result dataset via result_object$answer_data_aggregated

# Data sets in object
price_dat <- test_obj$price_data
price_dat
#> A tibble: 1 × 8
#>   promptid model       iterations prompt completion_tokens input_price_dollar output_price_dollar
#>      <int> <chr>            <dbl>  <int>             <dbl>              <dbl>               <dbl>
#> 1        1 gpt-4o-mini         10      1              7000              0.017             0.00042
#> ℹ 1 more variable: total_price_dollar <dbl>

agg_dat <- test_obj$answer_data_aggregated
agg_dat |> select(human_code, final_decision_gpt, final_decision_gpt_num)
#>  A tibble: 10 × 3
#>    human_code final_decision_gpt final_decision_gpt_num
#>         <dbl> <chr>                               <dbl>
#>  1          0 Exclude                                 0
#>  2          0 Exclude                                 0
#>  3          0 Exclude                                 0
#>  4          0 Exclude                                 0
#>  5          0 Exclude                                 0
#>  6          1 Exclude                                 0
#>  7          1 Exclude                                 0
#>  8          1 Exclude                                 0
#>  9          1 Include                                 1
#> 10          1 Include                                 1
```

## References
Filges, T., Andersen, D, & Jørgensen, A-M. K (2015). Functional Family Therapy (FFT) for Young People in Treatment for Non-opioid Drug Use: A Systematic Review. *Campbell Systematic Reviews*. <https://journals.sagepub.com/doi/full/10.4073/csr.2015.14>

Vembye, M. H., Christensen, J., Mølgaard, A. B., & Schytt, F. L. W. (2025). Generative pretrained transformer models can function as highly reliable second screeners of titles and abstracts in systematic reviews: A proof of concept and common guidelines. *Psychological Methods*. 1-20 <https://psycnet.apa.org/record/2026-37236-001>.