---
title: "Parsing Europe PMC FTP files"
author: "Chris Stubben"
date: '`r gsub("  ", " ", format(Sys.time(), "%B %e, %Y"))`'
output: rmarkdown::html_vignette
vignette: >
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteIndexEntry{Parse PMC FTP files}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "# "
)
```


The [Europe PMC FTP] includes 2.5 million open access articles separated into
files with 10K articles each.  Download and unzip a recent series of PMC ids
and load into R using the `readr` package.   A sample file with the first 10
articles is included in the `tidypmc` package.

```{r load}
library(readr)
pmcfile <- system.file("extdata/PMC6358576_PMC6358589.xml", package = "tidypmc")
pmc <- read_lines(pmcfile)
```


Find the start of the article nodes.

```{r startnode}
a1 <- grep("^<article ", pmc)
head(a1)
n <- length(a1)
n
```

Read a single article by collapsing the lines into a new line separated string.


```{r read1, echo=-1}
options(width=100)
library(xml2)
x1 <- paste(pmc[2:29], collapse="\n")
doc <- read_xml(x1)
doc
```


Loop through the articles and save the metadata and text below.
All 10K articles takes about 10 minutes to run on a Mac laptop and returns 1.7M
sentences.


```{r loop}
library(tidypmc)
a1 <- c(a1, length(pmc))
met1 <- vector("list", n)
txt1 <- vector("list", n)
for(i in seq_len(n)){
  doc <- read_xml(paste(pmc[a1[i]:(a1[i+1]-1)], collapse="\n"))
  m1 <- pmc_metadata(doc)
  id <- m1$PMCID
  message("Parsing ", i, ". ", id)
  met1[[i]] <- m1
  txt1[[i]] <- pmc_text(doc)
}
```


Combine the list of metadata and text into tables.


```{r combine, echo=-1, message=FALSE}
options(width=100)
library(dplyr)
met <- bind_rows(met1)
names(txt1) <- met$PMCID
txt <- bind_rows(txt1, .id="PMCID")
met
txt
```




[Europe PMC FTP]: https://europepmc.org/ftp/oa/