\documentclass[nojss]{jss}
\usepackage{thumbpdf}
%% need no \usepackage{Sweave}
\shortcites{exams:Blesius+Moreno-Ger+Neumann:2007,exams:Agea+Crespo-Garcia+Delgado-Kloos:2009}
%% for internal use
\newcommand{\fixme}[1]{\emph{\marginpar{FIXME} (#1)}}
\newcommand{\readme}[1]{\emph{\marginpar{README} (#1)}}
%% new environments for exams
\newenvironment{question}{\item \textbf{Problem}\newline}{}
\newenvironment{solution}{\textbf{Solution}\newline}{}
\author{Achim Zeileis\\Universit\"at Innsbruck
\And Nikolaus Umlauf\\Universit\"at Innsbruck\\
\And Friedrich Leisch\\Universit\"at f\"ur\\Bodenkultur Wien}
\Plainauthor{Achim Zeileis, Nikolaus Umlauf, Friedrich Leisch}
\title{Flexible Generation of E-Learning Exams in \proglang{R}: \pkg{Moodle} Quizzes, \pkg{OLAT} Assessments, and Beyond}
\Plaintitle{Flexible Generation of E-Learning Exams in R: Moodle Quizzes, OLAT Assessments, and Beyond}
\Shorttitle{Flexible Generation of E-Learning Exams in \proglang{R}}
\Keywords{exams, e-learning, multiple choice, arithmetic problems, \code{Sweave}, \proglang{R}, {\LaTeX}, HTML, XML, IMS QTI, \pkg{Moodle}, \pkg{OLAT}}
\Plainkeywords{exams, e-learning, multiple choice, arithmetic problems, Sweave, R, LaTeX, HTML, XML, IMS QTI, Moodle, OLAT}
\Abstract{
This introduction to version~2 of the \proglang{R}/\pkg{exams} package is a (slightly)
modified version of \cite{exams:Zeileis+Umlauf+Leisch:2014}, published in the
\emph{Journal of Statistical Software}. Since its publication, the \pkg{exams} package
has been extended in numerous ways which are not reflected in this vignette. Examples include:
Exercises in \proglang{R}/Markdown format (instead of \proglang{R}/{\LaTeX}),
the \code{exams2nops} interface for exams that can be automatically scanned and
evaluated, export to \pkg{Canvas} or \pkg{Blackboard} etc.
For a current and more general introduction to the \proglang{R}/\pkg{exams} package
see the materials at the official web page: \url{https://www.R-exams.org/}.
The capabilities of the package \pkg{exams} for automatic generation
of (statistical) exams in \proglang{R} are extended by adding support for learning
management systems: As in earlier versions of the package
exam generation is still based on separate \code{Sweave} files for each exercise --
but rather than just producing different types of PDF output
files, the package can now render the \emph{same} exercises into a wide variety of output formats.
These include HTML (with various options for displaying mathematical
content) and XML specifications for online exams in learning
management systems such as \pkg{Moodle} or \pkg{OLAT}.
This flexibility is accomplished by a new modular and extensible design of the
package that allows for reading all weaved exercises into \proglang{R} and
managing associated supplementary files (such as graphics or data files).
The manuscript discusses the readily available user interfaces,
the design of the underlying infrastructure, and how new functionality can
be built on top of the existing tools.
}
\Address{
Achim Zeileis, Nikolaus Umlauf\\
Universit\"at Innsbruck\\
Department of Statistics\\
Universit\"atsstr.~15, 6020 Innsbruck, Austria\\
E-mail: \email{Achim.Zeileis@R-project.org}, \email{Nikolaus.Umlauf@uibk.ac.at}\\
URL: \url{https://www.zeileis.org/}, \url{https://nikum.org/}\\
Friedrich Leisch\\
Universit\"at f\"ur Bodenkultur Wien\\
Institute of Applied Statistics and Computing\\
Peter Jordan-Str.~82, 1190 Wien, Austria\\
E-mail: \email{Friedrich.Leisch@R-project.org}\\
URL: \url{https://boku.ac.at/rali/stat/personen/leisch-friedrich}
}
\begin{document}
\SweaveOpts{engine = R, eps = FALSE, keep.source = TRUE}
%\VignetteIndexEntry{Flexible Generation of E-Learning Exams in R: Moodle Quizzes, OLAT Assessments, and Beyond}
%\VignetteDepends{stats, graphics, tools, utils, tth, base64enc}
%\VignetteKeywords{exams, e-learning, multiple choice, arithmetic problems, Sweave, R, LaTeX, HTML, XML, IMS QTI, Moodle, OLAT}
%\VignettePackage{exams}
<<preliminaries, echo=FALSE, results=hide>>=
options(width = 76, prompt = "R> ", continue = "+ ")
library("exams")
library("base64enc")
library("tth")
@
\section{Introduction} \label{sec:intro}
The design for version~1 of the \pkg{exams} package was conceived
eight years ago (in 2006) when the original authors
\citep{exams:Gruen+Zeileis:2009} were involved in a redesign of the
introductory statistics lecture at WU Wirtschaftsuniversit\"at Wien.
Back then the main goal was to be able to produce exams along with
associated self-study materials as PDF (portable document format)
files. Thus, the main focus was still on printable materials for
classic classroom exams. Although e-learning systems started to become
available more easily back at that time, they were still not very widely used
and, more importantly, rather few easy-to-use standards for specifying
e-learning exams were available (e.g., WU Wien used a partially
self-developed e-learning system based on \pkg{.LRN}, see \citealp{exams:Blesius+Moreno-Ger+Neumann:2007}).
However, since 2006 the situation
has clearly changed: E-learning systems are now abundant with many
universities offering one (ore more) e-learning system(s) in which
all students are readily registered. Consequently, many lecturers
routinely offer online exams (or tests, quizzes, assessments)
for large-lecture courses -- either as self-study materials or as
(part of) the main assessment of the course.
Among the more popular choices of learning management systems are the
open-source systems \pkg{Moodle} -- developed by \cite{exams:Moodle:2014}
and supported by a large world-wide user community -- or \pkg{OLAT}
(for online learning and training) -- originally developed by \cite{exams:OLAT:2012}
and with the recent fork \pkg{OpenOLAT} being developed by \cite{exams:OpenOLAT:2014}
and a support community. A popular proprietary
learning management system is \pkg{Blackboard} developed by \cite{exams:Blackboard:2010}.
Standards for specifying and exchanging e-learning exams/assessements
have also emerged \citep[see][for an overview]{exams:Agea+Crespo-Garcia+Delgado-Kloos:2009}.
While \pkg{Moodle} specifies its own \pkg{Moodle} XML format (but can
import and export several other formats), \pkg{OLAT}/\pkg{OpenOLAT} and \pkg{Blackboard}
employ certain subsets of the international QTI (question \& test interoperability)
standard, version~1.2, maintained by the \cite{exams:QTI12:2002}.
The successor formats are QTI~2.0 and the current QTI~2.1 which is
for example employed in the \pkg{ONYX} testsuite \citep{exams:BPS:2014}
that also offers interfaces to \pkg{OLAT} and \pkg{Blackboard}.
Therefore, although the PDF exams produced by version~1 of the \pkg{exams}
package as introduced by \cite{exams:Gruen+Zeileis:2009} are still
useful for many types of courses, it would also be highly desirable
to have support for generating e-learning exams \emph{from the same pool
of exercises}. In fact, this became an apparent need when the authors
of the present manuscript took over new large-lecture statistics and
mathematics courses at their respective institutions (Universit\"at Innsbruck
and Universit\"at f\"ur Bodenkultur Wien, BOKU, respectively). For example,
the new ``Mathematics~101'' lecture at Universit\"at Innsbruck is
currently attended by about 1,600~students (mostly first-year business and
economics students) and accompanied by biweekly online exams conducted in
the university's \pkg{OLAT} learning management system. This was a strong
incentive to start developing version~2 of the \pkg{exams} package that is presented
here and offers an extensible toolbox for generating e-learning exams,
including easy-to-use functions for \pkg{Moodle} quizzes and \pkg{OLAT}
assessments\footnote{Meanwhile, various configurations of exams in \pkg{OLAT}
and \pkg{OpenOLAT} are supported. See the convenience function \code{exams2openolat()}
for some more details.}.
The new version of the \pkg{exams} package for the \proglang{R} system for
statistical computing \citep{exams:R:2014} is available from the
Comprehensive \proglang{R} Archive Network at \url{https://CRAN.R-project.org/package=exams}.
Like prior versions it employs ideas and technologies
from literate programming and reproducible research
\citep[see e.g.,][]{exams:Knuth:1992,exams:deLeeuw:2001,exams:Leisch+Rossini:2003,exams:Kuhn:2014}
by using \code{Sweave()} \citep{exams:Leisch:2002a} to combine data-generating
processes in \proglang{R} with corresponding questions/solutions
in {\LaTeX} \citep{exams:Knuth:1984,exams:Lamport:1994}. But in addition
to producing exams in PDF format, the new version of \pkg{exams} includes
extensible tools for generating other output formats \emph{without} having
to modify the pool of exercises. Thus, the design principles of the \pkg{exams}
package are only somewhat extended compared to version~1:
\begin{itemize}
\item Each exercise template (also called ``exercise'' for short) is a single
\code{Sweave} file (\code{.Rnw}) interweaving \proglang{R} code for data
generation and {\LaTeX} code for describing question and solution (possibly
containing mathematical notation in {\LaTeX}).
\item Exams can be generated by randomly drawing different versions of exercises from a
pool of such \code{Sweave} exercise templates. The resulting exams can be
rendered into various formats including PDF, HTML, \pkg{Moodle} XML, or
QTI~1.2\footnote{An implementation of the newer QTI~2.1 standard is
also available in the package, see \code{?exams2qti21}. However, at the
time of writing the function still needed more testing and hence was
not discussed here.} (for \pkg{OLAT}/\pkg{OpenOLAT}).
\item Maintenance is simple as exercises are separate standalone files. Thus,
large teams can work jointly on the pool of exercises in a multi-author and
cross-platform setting because each team member can independently develop
and edit a single exercise.
\end{itemize}
The remainder of this paper consists of two major parts: First, we illustrate in
Section~\ref{sec:usage} how to use both the old and new exam-generating functions
that are readily available in the package. This serves as a first introduction
and is sufficient for getting a good overview of the available features
and how to get started. Second, Section~\ref{sec:design} provides details about
the design underlying the toolbox for the new infrastructure. This section -- as well
as the subsequent Section~\ref{sec:extensions} showing how to extend the toolbox --
may be skipped upon first reading but it contains many important details that are
likely to be required when actually starting to create course materials with the
package. Finally, a discussion in Section~\ref{sec:discussion} concludes the paper.
\section[Using the exams package]{Using the \pkg{exams} package} \label{sec:usage}
In this section we provide an overview of the most important user interfaces provided
by the \pkg{exams} package. This serves as a first introduction, assuming only
(basic) knowledge of \code{Sweave} \citep{exams:Leisch:2008a,exams:Leisch:2008}.
First, the format of the exercise \code{Sweave} files
is reviewed along with the old (version~1) \code{exams()} function. Subsequently,
the new (version~2) functions of type \code{exams2}\emph{xyz}\code{()}
are introduced: \code{exams2pdf()} and \code{exams2html()} produce one PDF or HTML
file for each exam, respectively. In case
of just a single exam, this is shown interactively in a viewer/browser.
\code{exams2moodle()} and \code{exams2qti12()} generate \pkg{Moodle} and QTI~1.2 exams,
i.e., just a single XML or ZIP file, respectively, which can be easily uploaded into
\pkg{Moodle} and \pkg{OLAT}.
\subsection[Version 1: PDF exams() from Sweave exercises]{Version~1: PDF \code{exams()} from \code{Sweave} exercises}
Exercise templates (or just ``exercises'' for short) are essentially
separate standard \code{Sweave} files \citep{exams:Leisch:2008a,exams:Leisch:2008}.
They are composed of the following elements:
\begin{itemize}
\item \proglang{R} code chunks (as usual within \verb|<<>>=| and \code{@})
for random data generation.
\item Question and solution descriptions contained in {\LaTeX}
environments of corresponding names. Both can contain \proglang{R}
code chunks again or include data via \verb|\|\verb|Sexpr{}|.
\item Metainformation about the exercise type (numeric, multiple
choice, \dots), its correct solution etc. All metainformation commands
are in {\LaTeX} style but are actually commented out and hidden in the final
output file.
\end{itemize}
The underlying ideas are eplained in more detail by
\cite{exams:Gruen+Zeileis:2009} and Section~\ref{sec:design} provides
more technical details. Here, we focus on illustrating how different output
formats can be generated from such exercises.
%% from Rnw
\begin{figure}[p!]
\centering
\makebox[\textwidth]{\hrulefill}
<<echo=FALSE, results=tex>>=
invisible(file.copy(system.file("exercises", "tstat.Rnw", package = "exams"), "tstat.Rnw"))
Rnw <- readLines("tstat.Rnw")
cat(c("\\begin{verbatim}", Rnw, "\\end{verbatim}"), sep = "\n")
@
\makebox[\textwidth]{\hrulefill}
\caption{\label{fig:Rnw} A simple \code{Sweave} exercise: \code{tstat.Rnw}.}
\end{figure}
%% to LaTeX
\begin{figure}[t!]
\centering
\makebox[\textwidth]{\hrulefill}
<<echo=FALSE, results=hide>>=
set.seed(1090)
Sweave("tstat.Rnw")
tex <- readLines("tstat.tex")
file.remove(c("tstat.Rnw", "tstat.tex"))
@
<<echo=FALSE, results=tex>>=
cat(c("\\begin{verbatim}", tex, "\\end{verbatim}"), sep = "\n")
@
\makebox[\textwidth]{\hrulefill}
\caption{\label{fig:tex} {\LaTeX} output of \code{Sweave("tstat.Rnw"}).}
\end{figure}
%% to PDF/HTML
\begin{figure}[p!]
\centering
\makebox[\textwidth]{\hrulefill}
\begin{enumerate}
<<echo=FALSE, results=tex>>=
cat(tex, sep = "\n")
@
\end{enumerate}
\makebox[\textwidth]{\hrulefill}
\caption{\label{fig:pdf} Display of a \code{tstat} exercise as PDF via \code{exams()} or \code{exams2pdf()}.}
\vspace*{0.5cm}
\includegraphics[width=\textwidth]{screenshots/exams2html-tstat.png}
\caption{\label{fig:exams2html-tstat} Display of a \code{tstat} exercise as HTML via \code{exams2html()}.
MathML is employed for mathematic equations, as rendered by a \pkg{Firefox} browser.}
\end{figure}
In Figure~\ref{fig:Rnw}, the \code{Sweave} file for a simple exercise asking
students to compute a one-sample $t$~test statistic is shown for
illustration \citep[as already used by][]{exams:Gruen+Zeileis:2009}. The
\proglang{R} chunk for the data-generating process (DGP), the \code{question} and \code{solution} environments,
and the metainformation can be easily distinguished. The {\LaTeX} file
resulting from an \code{Sweave()} call is shown in Figure~\ref{fig:tex}, and
Figure~\ref{fig:pdf} shows the final compiled PDF output generated by
<<exams1, echo=TRUE, eval=FALSE>>=
library("exams")
set.seed(1090)
exams("tstat.Rnw")
@
Here, the \code{exams()} function looks for the exercise template \code{tstat.Rnw}
first in the local working directory and then within the installed \pkg{exams} package
where this file is provided. Then it copies the exercise \code{.Rnw} to a temporary
directory, calls \code{Sweave()} to generate the \code{.tex}, and includes this
in the default {\LaTeX} template for exams before producing the \code{.pdf}. As,
by default, just a single \code{.pdf} exam is produced and no output directory
is specified, a PDF viewer pops up and displays the resulting exam (as in
Figure~\ref{fig:pdf}).
\begin{table}[t!]
\centering
\begin{tabular}{lp{13cm}} \hline
Argument & Description \\ \hline
\code{file} & A (list of) character vector(s) specifying the (base) names of
the \code{Sweave} exercise files. \\
\code{n} & The number of exams to be generated from the list of exercises. Default: 1.\\
\code{nsamp} & The number of exercise files sampled from each list element of \code{file}.
Default: One for each list element.\\ \hline
\code{dir} & Path to output directory. Default: Single PDF or HTML files are shown directly
in a viewer/browser (i.e., \code{exams}/\code{exams2pdf}/\code{exams2html}
with \texttt{n~=~1}). In all other cases the current working directory is used by default.\\
\code{edir} & Path to the directory in which the exercises in \code{file} are stored.
Default: Working directory (or within the \pkg{exams} installation). \\
\code{tdir} & Path to a temporary directory in which \code{Sweave()} is carried out.
Default: New \code{tempfile()}. \\
\code{sdir} & Path to the directory in which supplementary files (e.g., graphics or
data files) are stored (except for \code{exams()}). Default: New \code{tempfile()}. \\ \hline
\code{name} & Name prefix for the resulting exam files.\\
\code{template} & Character specifying the (base) names of a {\LaTeX}, HTML, or XML file
template for the exam (except for \code{exams2moodle()}). Default:
A function-specific template provided within the \pkg{exams} installation.\\
\code{encoding} & Character specifying the encoding to be used (in version~2 interfaces only).\\
\code{verbose} & Should progress information be displayed (in version~2 interfaces only)?\\\hline
\end{tabular}
\caption{\label{tab:args} Common arguments of the main user interfaces for generating exams:
\texttt{exams()}, \texttt{exams2pdf()}, \texttt{exams2html()}, \texttt{exams2moodle()}, \texttt{exams2qti12()}.
The first group of arguments pertains to the specification of the exam(s), the second
group to the handling of input/temporary/output directories, and the last group to name and setup
for the resulting files.
For further function-specific arguments and more details/examples, see the corresponding manual pages.}
\end{table}
While applying \code{exams()} to just a single exercise is very useful while
writing/programming an exercise, a full exam will typically encompass several different
exercises. Also, it may require suppressing the solutions, including a title page
with a questionnaire form, etc. The former can be achieved by supplying a
(list of) vector(s) of exercises while the latter can be accomodated by
using different templates:
<<myexam1>>=
myexam <- list(
"boxplots",
c("confint", "ttest", "tstat"),
c("anova", "regression"),
"scatterplot",
"relfreq")
odir <- tempfile()
set.seed(1090)
exams(myexam, n = 3, dir = odir, template = c("exam", "solution"))
@
The \code{myexam} list contains five exercises: the first one is always \code{boxplots.Rnw}
while the second exercise is randomly drawn from \code{confint.Rnw}, \code{ttest.Rnw},
\code{tstat.Rnw}, and so on for the remaining exercises. Then, \code{exams()} is used to draw
\code{n = 3} random exams and produce one exam and one solution PDF for each. The \code{template}
argument takes names of {\LaTeX} files which provide the {\LaTeX} headers and footers.
These templates can be used to create a title page with a questionnaire form (for student
name, id, signature, etc.), show or suppress solutions, and set further formatting details.
All involved \code{.Rnw} files (with exercises) and \code{.tex} templates employed in
the example above are provided in the \pkg{exams} source package and its installed versions.
The resulting output files are stored along with the extracted metainformation in the output directory:
<<myexam1-odir>>=
dir(odir)
@
More details on basic usage and more advanced customization of this function are
provided by the vignette \code{vignette("exams", package = "exams")} which is an updated
version of \cite{exams:Gruen+Zeileis:2009}. The vignette actually uses the new and somewhat more flexible
\code{exams2pdf()} function rather than \code{exams()} and discusses the enhancements
made for the former (see also below).
\subsection[Version 2: Producing PDF, HTML, or XML for Moodle or OLAT]{Version~2: Producing PDF, HTML, or XML for \pkg{Moodle} or \pkg{OLAT}}
The new infrastructure added to the \pkg{exams} package on the road to version~2 is
providing more flexibility and enables a much broader variety of output formats while keeping the
specification of the exercise templates fully backward compatible and only slightly extended.
While the design of the underlying workhorse functions is rather different (see Section~\ref{sec:design}),
the new user interfaces are very similar to the old one, sharing most of its arguments (see Table~\ref{tab:args}).
Hence, for users of the previous version of the package, it is easy and straightforward to
adapt to the new facilities.
\subsubsection[Producing PDF documents: exams2pdf()]{Producing PDF documents: \code{exams2pdf()}}
As mentioned above, the function \code{exams2pdf()} is a more flexible reimplementation of \code{exams()}
using the new extensible infrastructure of the \pkg{exams} package. For the user virtually nothing
changes:
<<exams2pdf, eval=FALSE>>=
set.seed(1090)
exams2pdf("tstat.Rnw")
@
pops up the same PDF as shown in Figure~\ref{fig:pdf}. We refrain from further discussion
of customization of the PDF output because this is discussed in \code{vignette("exams", package = "exams")}
with details about {\LaTeX} master templates, additional auxiliary files, showing/hiding solutions
etc. Here we only point out the main difference between the old \code{exams()} function and the
new \code{exams2pdf()}: The latter not only returns the metainformation from the exercise but additionally
also the {\LaTeX} code for the question and solution environments as well as paths to supplementary
materials (such as graphics or data files). Section~\ref{sec:design} explains
the structure of the return values in more detail and illustrates how this can be used.\footnote{To
obtain the same type of return value as from the \texttt{exams()} function, \texttt{exams\_metainfo(exams2}\emph{xyz}\texttt{(...))}
can be used.}
\subsubsection[Producing HTML documents: exams2html()]{Producing HTML documents: \code{exams2html()}}
As a first step towards including exams generated from \code{Sweave} files into e-learning
exams, it is typically necessary to be able to generate an HTML version of the exams. Hence, the
function \code{exams2html()} is designed analogously to \code{exams()}/\code{exams2pdf()} but
produces HTML files. In case of just a single generated exam, this is displayed in a browser
using base \proglang{R}'s \code{browseURL()} function\footnote{In \pkg{RStudio}
\citep{exams:RStudio:2014}, versions prior to~0.97.133, the \code{"browser"} option is set
to a function that cannot browse local HTML files on some platforms. Recent versions of
\pkg{RStudio} have resolved this problem and \code{?exams2html} also provides
workarounds for older \pkg{RStudio} versions.}. Again, this is particularly useful while
writing/programming a new exercise template.
For example,
<<exams2html, eval=FALSE>>=
set.seed(1090)
exams2html("tstat.Rnw")
@
generates the HTML file shown in Figure~\ref{fig:exams2html-tstat} which corresponds directly to the PDF
file from Figure~\ref{fig:pdf}. Note that for properly viewing the formulas in this HTML file,
a browser with MathML support is required. This is discussed in more detail in Section~\ref{sec:transform_html}.
Here, the \pkg{Firefox} browser is used (in Debian Linux's rebranded \pkg{Iceweasel} version)
which has native MathML support.
To transform the {\LaTeX} questions/solutions to something that a web browser can
render, three options are available: translation of the {\LaTeX} to (1)~plain HTML,
(2)~HTML plus MathML for mathematical formulas (default), or conversion of the corresponding PDF
to (3)~HTML with one embedded raster images for the whole question and solution, respectively.
The former two options are considerably faster and more elegant -- they just require the
\proglang{R}~package \pkg{tth} \citep{exams:Hutchinson+Leisch+Zeileis:2013} that makes
the `{\TeX}-to-HTML' converter \pkg{TtH} \citep{exams:Hutchinson:2012} easily available
in~\proglang{R}. Also, by default, the \pkg{base64enc} package \citep{exams:Urbanek:2012}
is employed for embedding graphics in Base64 encoding. More details on this approach
are provided in Section~\ref{sec:transform_html}.
The HTML files produced with approaches~(1) and~(2) can also easily contain hyperlinks to
supplementary files. For example, if the \proglang{R} code in the \code{Sweave} file
generates a file \code{mydata.rda}, say, then simply including \verb|\url{mydata.rda}|
in the question/solution will result in a suitable hyperlink. The supplementary data
files for each random replication of the exercise is managed fully automatically and
a copy of the data is created in an (exam-specific) sub-directory of the output directory.
Run \code{exams2html("boxhist.Rnw")} for such an example.
Just like \code{exams()}/\code{exams2pdf()}, \code{exams2html()} can also generate
multiple replications of randomly drawn exams via \code{exams2html(myexam, n = 3, dir = odir)}.
Also multiple versions of the same replications can be generated by providing
several templates, e.g., for showing/suppressing solutions.
\subsubsection[Producing Moodle XML: exams2moodle()]{Producing \pkg{Moodle} XML: \code{exams2moodle()}}
To incorporate exams generated from \code{Sweave} exercises into learning management
systems, such as \pkg{Moodle}, two building blocks are typically required: (1)~questions/solutions
are available in plain text or HTML format, and (2)~questions/solutions can be embedded
along with the metainformation about the possible and correct solutions into some
exam description format. (1)~can be accomplished as outlined in the
previous subsection for \code{exams2html()} and for \pkg{Moodle} (2)~requires
embedding everything into \pkg{Moodle} XML format. Both steps can be easily carried
out using the \code{exams2moodle()} function\footnote{Meanwhile, the default \code{converter}
for Moodle is \code{"pandoc-mathjax"}. Here, we use \code{"ttm"} instead which was
the default at the time of writing.}:
<<exams2moodle, eval=TRUE>>=
set.seed(1090)
exams2moodle(myexam, n = 3, dir = odir, converter = "ttm")
@
This draws the same three random exams from the \code{myexam} list that were already
generated in PDF format above. The output file, stored again in \code{odir}, is a
single XML file.
<<myexam2-odir>>=
dir(odir)
@
This XML file \code{moodlequiz.xml} can be easily imported into a
\pkg{Moodle} quiz and then further customized: First, the XML file is
imported into the question bank in \pkg{Moodle}. Then, all
replications of each exercise can be added as ``random'' questions
into a quiz (and potentially further
customized). Figure~\ref{fig:exams2moodle-boxplots} shows the first
random draw of the \code{boxplots} exercise in the resulting
\pkg{Moodle} quiz (again rendered by a \pkg{Firefox} browser). More
details on how \pkg{exams}-generated questions can be integrated in
\pkg{Moodle} are provided in Section~\ref{sec:discussion}.
\begin{figure}[t!]
\centering
\includegraphics[width=0.95\textwidth]{screenshots/exams2moodle-boxplots.png}
\caption{\label{fig:exams2moodle-boxplots} Display of exercise~1 (\code{boxplots}) from
\code{myexam} in \pkg{Moodle} (as rendered by a \pkg{Firefox} browser).}
\end{figure}
The corresponding solutions are displayed upon completion of the exam in \pkg{Moodle}.
As before, selected supplementary files are automatically managed and can easily be
included using \verb|\url{}| in the underlying {\LaTeX} code. To be able to include
all these supplements in a single XML file, Base64 encoding is employed for all
supplements. See the manual page for the list of all supported supplement file
formats.
\subsubsection[Producing QTI 1.2 XML (for OLAT): exams2qti12()]{Producing QTI 1.2 XML (for \pkg{OLAT}): \code{exams2qti12()}}
\begin{figure}[t!]
\centering
\includegraphics[width=0.95\textwidth]{screenshots/exams2qti12-boxplots.png}
\caption{\label{fig:exams2qti12-boxplots} Display of exercise~1 (\code{boxplots}) from
\code{myexam} in \pkg{OLAT} (as rendered by a \pkg{Firefox} browser).}
\end{figure}
The generation of QTI~1.2 assessments (for \pkg{OLAT}) proceeds essentially in the
same way as for the \pkg{Moodle} quizzes, by default using \code{ttm} for transformation
of the text to HTML\footnote{It may be of interest to \pkg{OLAT} users that we
experienced problems with the display of MathML matrices in \pkg{OLAT}. The columns
were not separated by spaces and we have not been able to adapt our \pkg{OLAT} installation
to avoide this problem. Hence, if we want to display matrices in \pkg{OLAT}, we
generate them with extra empty columns. The \code{cholesky.Rnw} exercise template has
code that can automatically do this, if enabled.}.
The same three random draws of exams from \code{myexam} can be prepared in QTI~1.2 format via:
<<exams2qti12, eval=TRUE>>=
set.seed(1090)
exams2qti12(myexam, n = 3, dir = odir)
@
This produces a single ZIP file \code{qti.zip}, again written to \code{odir}.
<<myexam3-odir>>=
dir(odir)
@
The ZIP file can again be easily imported into an \pkg{OLAT} test configuration where
further customization can be performed\footnote{While customization of the
features of the overall assessment was always possible for us, \pkg{OLAT}
typically did not allow for modification of the individual exercise items.
We were not able to track down which part of the QTI~1.2 XML specification causes this.}.
The first \code{boxplots} exercise from the exam generated above is shown in
\pkg{OLAT} in Figure~\ref{fig:exams2qti12-boxplots}
(again as rendered by a \pkg{Firefox} browser). The corresponding solutions
are displayed in \pkg{OLAT} immediately after \emph{incorrectly} completing an
individual exercise. The display of solutions can also be suppressed completely
by setting \code{solutionswitch = FALSE} in \code{exams2qti12()}.\footnote{In our
e-learning exams, we typically employ these default settings (i.e.,
\code{maxattemps = 1} and \code{solutionswitch = TRUE}). Alternatively,
we give the students an unlimited number of attempts to solve an exercise
(\code{maxattempts = Inf}) but then suppress solutions completely (\code{solutionswitch = FALSE})
because otherwise the correct solution would be displayed after the first incorrect attempt.}
The main difference of the generated ZIP file for QTI~1.2, compared to the \pkg{Moodle} XML output,
is that in addition to the \code{qti.xml} file further supplementary files can be included.
Hence, supplements in all potential formats can be easily included and uploaded in one
go into \pkg{OLAT}. Therefore, by default, Base64 is employed only for graphics but
not for other files (such as data sets etc.) but either approach can be used optionally
for all types of supplements.
The QTI~1.2 standard allows for rather fine control of the properties
of the exercises (also known as items in QTI~1.2) and the exams (also known as assessments).
Hence, \code{exams2qti12()} provides a variety of options for controlling
the appearance of exam/exercises, see the manual page \code{?exams2qti12} for details. Also, the
underlying XML template can be adapted.
\subsection{Creating the first exam} \label{sec:skeleton}
When creating the first ``real'' exam with \pkg{exams}, i.e., when starting to
prepare course materials with the help of the package, it is our experience
that it works best to start (almost) from scratch with some simple examples.
The package provides a wide range of examples for typical exercises (see
Table~\ref{tab:exercises} in the appendix for an overview) which can serve as a
starting point and it is often useful to copy parts of these exercises to
create new ones. In particular, we recommend to keep the formatting as simple
as possible for two reasons: (1)~The resulting exercises are typically more
robust and work well with different \code{exams2}\emph{xyz}\code{()} interfaces
(especially both in {\LaTeX} and in HTML). (2)~Some formatting issues require
attention to technical details, e.g., as discussed in Section~\ref{sec:design}.
Thus, we recommend to start with exercises taking inspiration from the available
examples and only using basic {\LaTeX} markup for mathematical notation
and formatting. To aid this process, \pkg{exams} provides the function
\code{exams_skeleton()} (or equivalently \code{exams.skeleton()}) which
creates a directory with a copy of the exercises from Table~\ref{tab:exercises}
(in the appendix) and the required templates (e.g., for {\LaTeX}, HTML, or XML) along with
a `\code{demo.R}' script illustrating the use of the various
\code{exams2}\emph{xyz}\code{()} interfaces.
As an illustration, assume
that we are interested in using \code{exams2moodle()} and \code{exams2pdf()}
for generating both \pkg{Moodle} and PDF exams (for printout). To test that the
{\LaTeX}-to-HTML conversion for \pkg{Moodle} actually works for all exercises,
we additionally want to inspect the results of \code{exams2html()}. Hence,
the code below calls \code{exams_skeleton()} specifying these three writers.
Here, we employ a temporary directory but users may set the \code{dir} argument
to something like \code{"C:/myexam/"} or \code{"~/myexam/"} etc.
<<exams_skeleton>>=
dir.create(mydir <- tempfile())
exams_skeleton(dir = mydir, absolute = TRUE,
writer = c("exams2html", "exams2pdf", "exams2moodle"))
dir(mydir)
@
The directory then contains the files `\code{demo-all.R}' which can be opened in
any editor for \proglang{R}~scripts -- as well as separate `\code{demo-*.R}' scripts
for the different interfaces. These illustrate how to create
various kinds of output using the \code{exams2html()}, \code{exams2pdf()},
and \code{exams2moodle()} functions based on the exercises and templates
in the subdirectories of the same name. Absolute paths are employed in the
script to refer to these subdirectories (while the default \code{absolute = FALSE}
would result in relative paths being used).
The function \code{exams_skeleton()} always copies all exercise files to the
directory but the `\code{demo-*.R}' scripts only employ some selected files
for each exercise type. It is easy to modify the `\code{demo-*.R}' scripts, omitting or adding
exercises that are readily available in the subdirectory.
Finally, to illustrate how different encodings can be used, \code{exams_skeleton()}
can also be called with an \code{encoding} argument, e.g., setting
\code{encoding = "UTF-8"}. This modifies the \code{demo-*.R} scripts as well
as the HTML and {\LaTeX} templates accordingly. The \code{encoding}s
\code{"latin1"} (or \code{"ISO-8859-1"}) and \code{"latin9"} (or \code{"ISO-8859-15"})
have also been tested. As usual, employing \code{Sweave} files in a particular encoding
can be very convenient for special characters (such as accents or umlauts) but might
also lead to problems if they are used in different locales (e.g., on different
operating systems). An alternative route (employed by the authors of the \pkg{exams}
package) is to employ \code{Sweave} ASCII files only, using {\LaTeX} commands for
special characters.
\section{Design} \label{sec:design}
All the new \code{exams2}\emph{xyz}\code{()} interfaces for generating exams in
different formats (with currently \emph{xyz} $\in$ \{\code{pdf}, \code{html}, \code{moodle}, \code{qti12}\})
are built by combining the modular building blocks provided by version~2 of \pkg{exams}.
All functions have the same goal, i.e., to write exam files in a certain format
to the hard disk.
The approach is that the \code{Sweave} exercises are first \emph{weaved} to {\LaTeX},
\emph{read} into \proglang{R}, potentially \emph{transformed} (e.g., to HTML), and
then \emph{written} to suitable output file formats on the disk. Different customizable
driver functions (or even driver-generating functions) for performing the
weave/read/transform/write steps are available in \pkg{exams}.
Internally, all the \code{exams2}\emph{xyz}\code{()} interfaces choose certain
drivers and then call the new function \code{xexams()} (for extensible exams) that handles
all temporary files/directories and suitably executes the drivers. In the following subsections,
all these building blocks are introduced in detail.
\subsection{Extended specification of exercises} \label{sec:exercises}
As discussed in Section~\ref{sec:usage} and illustrated in Figure~\ref{fig:Rnw}, each exercise
is simply an \code{Sweave} file containing \proglang{R} code for data generation,
\code{{question}}/\code{{solution}} environments with {\LaTeX} text, and
metainformation about the type of exercise and the correct solution etc. This
design was introduced by \cite{exams:Gruen+Zeileis:2009} but is slightly extended
in the new version to provide some more options for the generation of e-learning exams.
See Table~\ref{tab:metainfo} for an overview for a list of exercise types and
corresponding metainformation commands.
\begin{table}[t!]
\centering
\begin{tabular}{lp{11.8cm}} \hline
Command & Description \\ \hline
\verb|\extype{}| & Specification of the type of exercise (required): \code{num}
for questions with a numeric answer, \code{mchoice} for questions
with multiple-choice answers, \code{schoice} for questions with
single-choice answers (i.e., multiple-choice with exactly one correct
solution), \code{string} for questions with a (short) text answer, or
\code{cloze} for cloze solutions (i.e., combinations of the above). \\
\verb|\exname{}| & Short name/description (to be used for printing within \proglang{R}). \\
\verb|\extitle{}| & Pretty longer title (for \pkg{Moodle}). \\
\verb|\exsection{}| & Section of the exercise (for \pkg{Moodle}, with slashes for subsections as in a URL). \\
\verb|\exversion{}| & Version of the exercise. \\ \hline
\verb|\exsolution{}| & Correct solution (required). It must contain a numeric solution for \code{num},
a string of zeros/ones for \code{mchoice}/\code{schoice}, or a character
string of \code{string}. For \code{cloze} a combination of these can
be specified, e.g., \verb+\exsolution{1.23|001|glm}+. \\
\verb|\extol{}| & Tolerance for \code{num} solutions or a vector of tolerances (expanded if
necessary) for \code{cloze} solutions. If unspecified the tolerance is 0. \\
\verb|\exclozetype{}| & List of types for the elements of a \code{cloze} exercise, e.g.,
\verb+\exclozetype{num|schoice|string}+ for the example above. \\
\verb|\expoints{}| & Points for (fully) correct solution. Default is 1.\\ \hline
\verb|\exextra[]{}| & Additional metainformation to be read and stored, e.g., for new custom
interfaces. The default storage type is character, e.g.,
\verb|\exextra[myinfo]{1.23}| yields a metainformation element
\code{myinfo} of \code{"1.23"}. The type can also be numeric or logical,
e.g., \verb+\exextra[myinfo,logical]{FALSE|FALSE|TRUE}+. \\
\hline
\end{tabular}
\caption{\label{tab:metainfo} Overview of metainformation commands in exercises. The commands in the
first section allow for a general description, those in the second section for question/answer
specification.
%%and those in the last section for customization in e-learning environments.
Only \texttt{extype} and \texttt{exsolution} are always required (but \texttt{exname} is recommended
additionally for nice printing in \proglang{R}).}
\end{table}
Each exercise must specify at least an \verb|\extype{}| and an \verb|\exsolution{}| and
should typically also have a short \verb|\exname{}|. There are now five different \code{extype}s.
Two types that have a single question and answer:
\begin{itemize}
\item \code{num} for questions with a numeric answer, e.g., \verb|\exsolution{1.23}|.
\item \code{string} for questions with a (short) text answer, e.g., \verb|\exsolution{glm}|.
\end{itemize}
Three types have a list of questions (or statements):
\begin{itemize}
\item \code{mchoice} for multiple-choice questions where each element of the
question/statement can either be true or false, e.g., \verb|\exsolution{01011}|.
\item \code{schoice} for single-choice questions where exactly one of the
questions/statements is true and all others are false, e.g., \verb|\exsolution{01000}|.
\item \code{cloze} for a combination of questions/statements with \code{num},
\code{string}, or \code{mchoice} answers. Thus, each element of the question
has either a numeric, short text, or single/multiple-choice answer, e.g., \verb+\exsolution{1.23|001|glm}+.
To specify the individual \code{cloze} types, a \code{clozetype} has to be given, e.g.,
\verb+\exclozetype{num|schoice|string}+.
\end{itemize}
The types \code{schoice} and \code{cloze} have been newly introduced. The purpose
of the former is mainly to allow for different processing of options (e.g., for
assigning points to correct/wrong results) between \code{mchoice} and \code{schoice}.
The \code{cloze} type was introduced because both \pkg{Moodle} and QTI~1.2 have
support for it (albeit in slightly different ways, for details see below). Possible
evaluation strategies (with/without partial credits and/or with/without negative points)
are discussed below for \code{exams2moodle()} and \code{exams2qti12()} and in
Appendix~\ref{sec:eval} for further functionality within \proglang{R}.
For the three types with lists of questions (\code{mchoice}, \code{schoice}, \code{cloze}),
the \code{{question}} and \code{{solution}} environments should each contain at the end
an \code{{answerlist}} environment. In the \code{{question}} this should list an \verb|\item|
for each question/statement and in the \code{{solution}} the corresponding answers/explanations
can be provided (if any). The \code{{answerlist}} environment can either be written as usual ``by hand''
or by using the \code{answerlist()} function provided
by the \pkg{exams} package. For illustration, we set up a multiple-choice question with
three statements about Switzerland. First, we generate an \code{{answerlist}} with statements
for the \code{{question}}.
<<answerlist-question>>=
qu <- c("Zurich is the capital of Switzerland.",
"Italian is an official language in Switzerland.",
"Switzerland is part of the European Union (EU).")
answerlist(qu)
@
Then the corresponding \code{{answerlist}} for the \code{{solution}} is set up.
<<answerlist-solution>>=
sol <- c(FALSE, TRUE, FALSE)
ex <- c("The capital of Switzerland is Bern.",
"The official languages are: German, French, Italian, Romansh.",
"Switzerland is part of the Schengen Area but not the EU.")
answerlist(ifelse(sol, "True", "False"), ex)
@
For more examples see the exercise files in the \code{inst/exercises} directoy
of the \pkg{exams} source package. There are various multiple-choice questions
with and without figures and/or verbatim \proglang{R} output (e.g., \code{anova},
\code{boxplots}, \code{cholesky}, among others). The files \code{tstat} and
\code{tstat2} illustrate how the same type of exercise can be coded as a
\code{num} or \code{schoice} question, respectively. The \code{cloze} type
is employed in, e.g., \code{boxhist} or \code{fourfold} (with more flexible formatting specifically
for \pkg{Moodle} in \code{boxhist2} and \code{fourfold2}, respectively).
See also Table~\ref{tab:exercises} in the appendix for an overview and
Appendix~\ref{sec:cloze} for more details on cloze exercises.
\subsection[The xexams() wrapper function]{The \code{xexams()} wrapper function}
To avoid recoding certain tedious tasks -- such as copying/reading files and handling
temporary directories -- for each of the user interfaces introduced
in Section~\ref{sec:usage}, the new \pkg{exams} package provides a modular and
extensible framework for building new exam-generating functions. This framework
is tied together by the \code{xexams()} function which is typically not called by
users directly but forms the basis for all new \code{exams2}\emph{xyz} interfaces.
To accomplish this, \code{xexams()} also takes the arguments listed in Table~\ref{tab:args}
(except \code{name} and \code{template}), draws exams from the exercise \code{file}
list, and does all the necessary file/directory handling. Furthermore,
it takes a \code{driver} argument that needs to be a list of four functions
\code{driver = list(sweave, read, transform, write)}. These are utilized as follows:
\begin{enumerate}
\item \emph{Weave:} For each of the selected exercise files (within all \code{n} exams)
\code{driver$sweave(file)} is run to weave the \code{.Rnw} file into a \code{.tex} file.
If \code{sweave = NULL} (the default), the standard \code{Sweave()} function is used.
If \code{sweave = list(...)} is a list, e.g., \code{list(pdf = FALSE, png = TRUE)},
this is passed as arguments to \code{Sweave()}.
\item \emph{Read:} Each resulting \code{.tex} file is read into \proglang{R} using
\code{driver$read(file)}. By default (\code{read = NULL}), the function
\code{read_exercise()} is used (see below), resulting in a list of character
vectors with the {\LaTeX} code for question/solution plus metainformation.
\item \emph{Transform:} Each of these exercise-wise list objects can subsequently be transformed by
\code{driver$transform(object)} which can be leveraged for transformations from
{\LaTeX} to HTML etc. By default (\code{transform = NULL}), no transformation is applied.
\item \emph{Write:} The (possibly transformed) lists of exercises, read into \proglang{R}
for each exam object, can be written out to one ore more files per exam in an
output directory via \code{driver$write(object, dir, info = list(id, n))}.
By default (\code{write = NULL}), no files are written.
\end{enumerate}
After performing each of the driver functions, \code{xexams()} returns invisibly
a nested list object (currently unclassed) as illustrated in Figure~\ref{fig:xexam}.
It is a list of \emph{exams} (of length \code{n}),
each of which is a list of \emph{exercises} (whose length depends on the length of \code{file} and \code{nsamp}),
each of which is a list (whose length/contents depends on \code{driver$read}). When
used with the default \code{read_exercise()}, each exercise is a list of length~6
containing the question/solution texts, metainformation, and paths to supplementary
files. These will be introduced in more detail in the next section.
\begin{figure}[t!]
\hrulefill
\begin{Code}
list( ## of 'exams', length: n
list( ## of 'exercises', length: k
list( ## of exercise content, length: 6
question,
questionlist,
solution,
solutionlist,
metainfo,
supplements
)
)
)
\end{Code}
\hrulefill
\caption{\label{fig:xexam} Structure of the return value of \texttt{xexams()}, when used with the
default \code{read} driver \texttt{read\_exercises()}.}
\end{figure}
All of the interfaces introduced in Section~\ref{sec:usage} employ the standard
\code{Sweave()} function for the weaving step (possibly with custom arguments)
and the \code{read_exercise()} function for the reading step. They mainly differ
in the transformation and writing step. \code{exams2pdf()} needs no transformation
and the writer first sets up a \code{.tex} file for each exam, calls \code{texi2dvi(pdf = TRUE)},
and then copies the resulting \code{.pdf} file to the output \code{dir}. \code{exams2html()}
on the other hand uses a {\TeX}-to-HTML transformation and the writer then sets
up a \code{.html} file for each exam and copies it to the output \code{dir}. Finally,
\code{exams2moodle()} and \code{exams2qti12()} both also use a transformation to
HTML but have no writer. The reason for this is that they do not write one file
per exam (i.e., with only one replication per exercise) but rather need to produce
XML files that include all different replications of each exercise. Hence, they take
the list returned by \code{xexams()} and process it subsequently in different ways. The details for all
these steps are explained in the subsequent subsections.
\subsection[The read driver: read_exercise() and read_metainfo()]{The \code{read} driver: \code{read\_exercise()} and \code{read\_metainfo()}}
The function \code{read_exercise()} reads the weaved exercises, i.e.,
files like that shown in Figure~\ref{fig:tex}. It simply extracts the text lines
from the \code{{question}} and \code{{solution}} environments and stores them
in vectors of the same name. If these environments contain \code{{answerlist}}
environments, these are extracted and stored separately in \code{questionlist} and \code{solutionlist}
vectors, respectively. Finally, the metainformation is extracted using
\code{read_metainfo()} which not only stores character vectors but also transforms
them to suitable types (depending on the \code{extype}) and performs some
sanity checks. The resulting metainformation is a list with elements essentially
corresponding to the commands from Table~\ref{tab:metainfo}.
For illustration, we run \code{xexams()} to select the same three exams as
used in the \pkg{Moodle} and \pkg{OLAT} examples above. However, using the
default \code{driver} specification, \code{xexams()} just performs the weaving
and reading steps (and has no transformation or writing step):
<<myexam4-xexams1>>=
set.seed(1090)
x <- xexams(myexam, n = 3)
@
The resulting object is a nested list as shown in Figure~\ref{fig:xexam} with
\Sexpr{length(x)}~exams of \Sexpr{length(x[[1]])}~exercises each (drawn from
the \code{myexam} list). Using \code{x[[i]][[j]]}, the \code{j}-th exercise
of the \code{i}-th exam can be accessed. Here, we explore the first exercise
(\code{boxplots}, a multiple-choice question) from the first exam that is also
shown in Figures~\ref{fig:exams2moodle-boxplots} and~\ref{fig:exams2qti12-boxplots}.
Its general question text (in {\LaTeX}) is printed below -- it requires
a graphic which is stored in a supplementary file in a temporary directory.
<<myexam4-xexams2>>=
writeLines(x[[1]][[1]]$question)
x[[1]][[1]]$supplements
@
The corresponding list of statements about the graphic is stored separately.
It is shown below along with the most important metainformation elements.
<<myexam4-xexams3>>=
x[[1]][[1]]$questionlist
x[[1]][[1]]$metainfo[c("file", "type", "solution")]
@
In summary, \code{xexams()} combined with the default readers is relatively
straightforward to use in other progams (such as the \code{exams2}\emph{xyz}
functions). The return value is somewhat ``raw'' as it is not classed and
has no dedicated methods for subsetting etc. However, we refrained from
using a more elaborate structure as this function is not meant to be
called by end-users while we expected other developers to find the current
structure sufficiently simple to use in their programs.
\subsection[LaTeX-to-HTML transform driver generator]{{\LaTeX}-to-HTML \code{transform} driver generator} \label{sec:transform_html}
When embedding statistical/mathematical exercises into web pages or learning management systems,
the exercises' {\LaTeX} text -- typically containing mathematical notation -- has to be transformed
in some way so that it can be rendered by a browser. Until relatively recently, this posed the notorious problem of how
to display the mathematical formulas and often the only good answer was to
embed raster images of the formulas. However, this
situation has clearly changed \citep[see e.g.,][]{exams:Vismor:2012} and there are now various
convenient options: e.g., using the mathematical markup language MathML \citep{exams:W3C+MathML:2014,exams:Wiki+MathML:2014}
or keeping {\LaTeX} formulas in the web page and embedding some \proglang{JavaScript} for rendering them.
Especially the display of MathML in web pages has become very easy: \pkg{Firefox} long had native support for it and
for the Microsoft \pkg{Internet Explorer} the \pkg{MathPlayer} plugin \citep{exams:MathPlayer:2013} has
long been available. More recently, other major browsers like \pkg{Opera} or \pkg{Safari}
also added support for MathML \citep[see][Section~1.2]{exams:Vismor:2012}.
Google \pkg{Chrome} briefly enabled MathML support but disabled it again due to instabilities.
Furthermore, \pkg{MathJax} \citep{exams:MathJax:2013}, an open-source \proglang{JavaScript}
engine, can be used to render MathML (or {\LaTeX}) formulas on a server rather than in the local browser.
Therefore, the new \pkg{exams} package offers functionality for automatically transforming
the {\LaTeX} exercises to HTML within \proglang{R} and by default employs MathML for all
mathematical notation (e.g., as demonstrated in Figure~\ref{fig:exams2html-tstat}).
More specifically, the package provides the driver generator
\code{make_exercise_transform_html()}. It returns a function suitable for plug-in into the
\code{transform} driver in \code{xexams()} which then replaces the {\LaTeX} code in
\code{question}/\code{questionlist} and \code{solution}/\code{solutionlist} with HTML code.
For illustration, we set up a particular function \code{trafo()} below and apply it to
the first exercise in the first exam within the object \code{x} that we had considered before:
<<myexam4-xexams4>>=
trafo <- make_exercise_transform_html(converter = "ttm", base64 = FALSE)
writeLines(trafo(x[[1]][[1]])$question)
@
It can be seen that the resulting exercise employs HTML text, e.g., uses \code{<em>} instead
of \verb|\emph| or \code{<img>} instead of \verb|\includegraphics|.\footnote{It may be noteworthy to
that the conversion (a)~assumes the graphics to be in \texttt{.png} format and (b)~does not
resolve the figure reference at the beginning of the text correctly. For (a), we just need
to make sure that the \texttt{sweave} driver in \texttt{xexams()} has \texttt{png = TRUE} (and
\texttt{pdf = FALSE}) which is accounted for in \texttt{exams2html()} etc. Issue (b), however,
needs to be avoided by formulating the underlying \texttt{.Rnw} differently (or by tolerating the
missing number).}
Internally, \code{make_exercise_transform_html()} can leverage three different \code{converter}s:
\code{ttm} (default), \code{tth}, or \code{tex2image}. The
former two come from the \proglang{R} package \pkg{tth} \citep{exams:Hutchinson+Leisch+Zeileis:2013}
and internally call the two \proglang{C}~functions \code{tth} ({\TeX} to HTML) and \code{ttm} ({\TeX} to HTML/MathML)
taken from the \pkg{TtH} suite of \cite{exams:Hutchinson:2012}. The last option, \code{tex2image},
is a function provided by the \pkg{exams} package itself. It proceeds by first running \code{texi2dvi(pdf = TRUE)}
from the base \proglang{R} package \pkg{tools} and subsequently converting the resulting PDF to a raster
image in a \code{system()} call to \pkg{ImageMagick}'s \code{convert} function \citep{exams:ImageMagick:2014}.
Thus, for this function \pkg{ImageMagick} is assumed to be installed and in the search path.
All three converters have their benefits and drawbacks:
\begin{itemize}
\item \code{tth} is typically preferable if there is no or only very simple mathematical notation.
The resulting HTML can then be rendered in any modern browser.
\item \code{ttm} is preferable if there is some standard mathematical notation
(e.g., fractions or equation arrays etc.). As argued above this can still be easily
displayed in suitable browsers or by employing \pkg{MathJax} in the web page.
\item \code{tex2image} is the ``last resort'' if neither of the two previous approaches work,
e.g., if more complex {\LaTeX} commands/packages need to be used which are not
supported by \code{tth}/\code{ttm}. It is fairly slow while \code{tth}/\code{ttm} are
typically even faster than calling {\LaTeX}.
\end{itemize}
To explore the differences of the results, the converters can also be called directly
on character strings containing {\LaTeX}. Below we use two simple code lines for which
\code{tth()} would probably be sufficient:
<<ttm-tth>>=
tex <- c("This is \\textbf{bold} and this \\textit{italic}.",
"Points on the unit circle: $x^2 + y^2 = 1$.")
ttm(tex)
tth(tex)
@
%% avoid evaluation of tex2image() in order to have no system dependency on ImageMagick
<<tex2image, eval=FALSE>>=
(tex2image(tex, dir = odir, show = FALSE))
@
<<tex2image-out, echo=FALSE>>=
t2i <- file.path(odir, "tex2image_1.png")
foo <- file.create(t2i)
print(t2i)
@
Note that \code{tex2image(tex)} returns the path to a raster image file which by default
is also shown directly in the browser.
Finally, our illustration of \code{make_exercise_transform_html()} also employed a second
option, \code{base64 = FALSE}, which still deserves more detailed explanation. After converting
an exercise from {\LaTeX} to HTML code (using either of the three converters above), the
HTML code may contain references to supplementary files (e.g., in \code{<img>} tags). Optionally,
by using the default \code{base64 = TRUE}, these images can be embedded directly into the
HTML code in Base64 encoding \citep[via the \pkg{base64enc} package in \proglang{R},][]{exams:Urbanek:2012}
and thus waiving the need for having them as supplementary files.
\subsection[PDF and HTML write driver generators]{PDF and HTML \code{write} driver generators}
In the first three steps of \code{xexams()}, exams are randomly drawn and weaved, read into \proglang{R},
and potentially transformed from {\LaTeX} to HTML (or some other format). However, so far,
no output files have been generated. The original idea of \cite{exams:Gruen+Zeileis:2009}
was to produce one or more output files for each of the \code{n} generated exams. To do so
in \code{xexams()} a \code{write} driver can be specified. The package provides several
generating functions for suitable drivers, especially for generating PDF and HTML files.
As before, the idea is to pass customization options to the driver generator which can
then be plugged into \code{xexams()}.
For PDF output files, the following driver generator is available:
\begin{Code}
make_exams_write_pdf(template = "plain", name = NULL,
inputs = NULL, header = list(Date = Sys.Date()), quiet = TRUE,
control = NULL)
\end{Code}
This is employed in \code{exams2pdf()} and proceeds in the same way as described
by \cite{exams:Gruen+Zeileis:2009} for the \code{exams()} function. It includes the \code{question}/\code{questionlist}
and \code{solution}/\code{solutionlist} in a {\LaTeX} \code{template}, then runs \code{texi2dvi(pdf = TRUE)}
from the base \pkg{tools} package, and finally copies the resulting PDF files to a desired output
directory. The default \code{plain.tex} template is provided within the \pkg{exams} package and
also more than one template can be employed as illustrated in Section~\ref{sec:usage}. Details
about the remaining customization arguments are provided on the manual page and in \cite{exams:Gruen+Zeileis:2009}.
\begin{figure}[t!]
\hrulefill
\begin{Code}
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<title>Exam ##ID##</title>
<style type="text/css">
body{font-family:Arial;}
</style>
</head>
<body>
<h2>Exam ##ID##</h2>
##\exinput{exercises}##
</body>
</html>
\end{Code}
\hrulefill
\caption{\label{fig:plain.html} Default HTML \code{template} file \texttt{plain.html} employed in
\texttt{make\_exams\_write\_html()}. Elements marked by \texttt{\#\#...\#\#} are being replaced in
each replication of the exam.}
\end{figure}
For HTML output files, a similar driver generator is available:
\begin{Code}
make_exams_write_html(template = "plain", name = NULL,
question = "<h4>Question</h4>", solution = "<h4>Solution</h4>",
mathjax = FALSE)
\end{Code}
This is employed in \code{exams2html()} and is also based on a \code{template}. By default the \code{plain.html}
file is used that is provided within \pkg{exams} and shown in Figure~\ref{fig:plain.html}. This contains placeholders marked with
\verb|##...##| that are to be replaced in each randomly drawn exam. The \verb|##ID##| is simply replaced with a
numeric ID ($1, \dots, \mathtt{n}$) and \verb|##\exinput{exercises}##| is replaced by an ordered list (\code{<ol>}) containing
the question/solution. If the \code{question} and \code{solution} arguments to \code{make_exams_write_html()}
are character strings, these are added as titles in the list. Alternatively, either argument can also
be set to \code{FALSE} which avoids inclusion of the corresponding element of the exercise in the
resulting HTML file.
As an additional convenience setting \code{mathjax = TRUE} includes the \code{<script>} tag for loading
the \pkg{MathJax} \proglang{JavaScript}. Then, \pkg{MathJax} (rather than the browser) handles the
rendering of the MathML formulas (if any) in the HTML file. To experiment with this option, one
can simply use examples like \code{exams2html("tstat", mathjax = TRUE)}.
\subsection[Further functions for processing xexams return values]{Further functions for processing \code{xexams} return values}
The interfaces \code{exams2moodle()} and \code{exams2qti12()} work somewhat differently
compared to \code{exams2pdf()} and \code{exams2html()}. They produce a single XML file
containing all \code{n} replications of all exercises rather than separate files
per exam. The reason is that learning management systems such as \pkg{Moodle} or \pkg{OLAT}
provide their own functionality for randomly drawing questions from a pool
stored in the system. Hence, \code{exams2moodle()} and \code{exams2qti12()} do not
really select \code{n} separate exams but supply a set of \code{n} replications
(either from identical or differing templates) that can be uploaded into the systems'
question pools.
Therefore, both interfaces do call \code{xexams()} with the standard weave/read
drivers and the HTML transformer introduced above but without a write driver.
Instead, the whole \proglang{R} list of exercise replications returned by
\code{xexams()} is processed subsequently in one go and embedded into a suitable XML file.
For the \pkg{Moodle} interface, the function has the following arguments:
\begin{Code}
exams2moodle(file, n = 1L, nsamp = NULL, dir = ".", verbose = FALSE,
name = NULL, edir = NULL, tdir = NULL, sdir = NULL,
quiet = TRUE, resolution = 100, width = 4, height = 4,
encoding = "", iname = TRUE, stitle = NULL, testid = FALSE, zip = FALSE,
num = NULL, mchoice = NULL, schoice = mchoice, string = NULL,
cloze = NULL, points = NULL, rule = NULL, pluginfile = TRUE, ...)
\end{Code}
Thus, in addition to the usual arguments from the first two lines (see Table~\ref{tab:args}),
the third line has those arguments passed to \code{Sweave}, and lines~4--6 have
the arguments responsible for the XML formatting. These are employed in the following steps:
\begin{itemize}
\item A character vector with the XML code for the \code{<moodlequiz>} is generated.
\item For each question a title text is included (in suitable XML tags),
where \code{iname}, \code{stitle}, and \code{testid} can be used for the fine-tuning.
\item The XML code for each question/exercise is inserted. It is generated by
the transformation functions \code{num}, \code{mchoice}, \code{schoice}, \code{string},
and \code{cloze}. For example, if \code{x[[i]][[j]]} is a multiple-choice
exercise, then \code{mchoice(x[[i]][[j]])} is employed to generate the XML
character string.
\end{itemize}
Thus, users can supply custom functions that handle the XML question generation.
By default, the package has a flexible generator \code{make_question_moodle()}
that returns a suitable function. Analogously to other generators employed previously,
this can be easily adapted. For example, the user could set
\code{mchoice = list(solution = FALSE, shuffle = TRUE)} and then the
\code{mchoice} XML driver would be \code{make_question_moodle(solution = FALSE, shuffle = TRUE)}.
Hence, while only a single generator function is available, one can easily set
different argument lists for numeric or multiple-choice exercises etc.
Furthmore, other fine-control options are available, e.g., for
setting the \code{points} for each exercise (overruling the \verb|\expoints| metainformation)
or the \code{rule} used for partial credits in multiple-choice exercises
(see also Appendix~\ref{sec:eval}).
The approach taken in \code{exams2qti12()} is essentially analogous to that
of the \pkg{Moodle} interface. It also has separate \code{num}, \code{mchoice},
\code{schoice}, \code{string}, and \code{cloze} XML transformation functions,
each of which is by default generated by \code{make_itembody_qti12()} (as exercises
are called items in \pkg{OLAT}), possibly supplying further arguments for
customization.
%
\begin{Code}
exams2qti12(file, n = 1L, nsamp = NULL, dir = ".", verbose = FALSE,
template = "qti12", name = NULL, edir = NULL, tdir = NULL, sdir = NULL,
quiet = TRUE, resolution = 100, width = 4, height = 4,
encoding = "", num = NULL, mchoice = NULL, schoice = mchoice,
string = NULL, cloze = NULL, duration = NULL,
stitle = "Exercise", ititle = "Question",
adescription = "Please solve the following exercises.",
sdescription = "Please answer the following question.",
maxattempts = 1, cutvalue = 0, solutionswitch = TRUE, zip = TRUE,
points = NULL, eval = list(partial = TRUE, negative = FALSE), ...)
make_itembody_qti12(rtiming = FALSE, shuffle = FALSE, rshuffle = shuffle,
minnumber = NULL, maxnumber = NULL, defaultval = NULL, minvalue = NULL,
maxvalue = NULL, cutvalue = NULL, enumerate = TRUE, digits = NULL,
tolerance = is.null(digits), maxchars = 12,
eval = list(partial = TRUE, negative = FALSE), fix_num = TRUE)
\end{Code}
%
For details about the arguments see \code{?exams2qti12}.
The main difference between the \pkg{Moodle} XML and QTI~1.2 XML specifications is
that the former just provides some control over the individual exercises (or questions, items)
whereas the latter also has control options for the whole exam (or assessment).
Therefore, the XML specification is somewhat more complex. Hence, \code{exams2qti12()}
also takes a \code{template} argument that is by default set to the \code{qti12.xml}
file provided within \pkg{exams}. The template must contain exactly one \code{<section>}
with exactly one \code{<item>} with a placeholder \verb|##ItemBody##|. Then,
\code{exams2qti12()} reads the template, replicates the \code{<section>} for each exercise,
replicates the \code{<item>} \code{n} times within each \code{<section>}, and then
inserts the \verb|##ItemBody##| with the XML transformation functions for \code{num},
\code{mchoice}, etc.
Similar to \code{exams2moodle()}, one can specify the \code{points} for each
exercise (again overruling the \verb|\expoints| metainformation) or specify an
\code{eval} argument that describes the evaluation policy employed (see
Appendix~\ref{sec:eval} for details).
One notable feature of the QTI~1.2 interfaces should be briefly explained: The \code{fix_num}
argument is provided to work around an error in \pkg{OLAT}. While numeric exercises/items
are not officially supported, they actually work correctly except for the display of the
correct answer in the final summary. This is fixed (by default) by extending the processing
of the answer (without changing the result). Another optional route for numeric exercises
is to process them up to a certain number of \code{digits} only (e.g., \code{digits = 2})
and store them as a string. However, in that case, the \code{tolerance} cannot be employed.
In summary, most end users should just have to call the main interfaces \code{exams2moodle()}
or \code{exams2qti12()} and customize by setting options for \code{num}, \code{mchoice}, etc.\
as some \code{list(...)}. If this is not sufficient, though, the users could program their
own XML transformation functions for \code{num}, \code{mchoice}, etc. And finally,
for QTI~1.2, a different template could be used.
\section[Extending the exams toolbox and writing new drivers]{Extending the \pkg{exams} toolbox and writing new drivers} \label{sec:extensions}
In some cases it is not sufficient to use the arguments of the existing \code{exams2}\emph{xyz}\code{()} functions
or to provide alternative \code{template}s to them. In particular, when a completely different output format
is required (e.g., a different XML format), it might be necessary to develop new drivers for the \code{xexams()}
toolbox. One example for such a situation is the software product that is currently employed for generating
\emph{printed} large-lecture exams at Universit\"at Innsbruck. This allows for
\begin{itemize}
\item specification of (static) single/multiple-choice exercises in a browser interface,
\item production of so-called ``scrambled'' PDF exams from it (where the static questions and solutions are simply shuffled),
\item optical character recognition (OCR) of scans from the exams' title pages,
\item computation of the points/marks achieved by the students.
\end{itemize}
Although, the \pkg{exams} package can also generate PDF exams directly, some courses still
prefer the interface that have been used previously.
Fortunately, this so-called LOPS exam server (developed by a spin-off company of WU Wien) also
employs an XML specification for importing/exporting its exams. Therefore, it was easily
possible for us to establish a new \code{exams2lops()} interface that produces one ZIP file
for each exam, including the XML plus supplementary graphics. A corresponding \code{write} driver
generator \code{make_exams_write_lops()} is also supplied in the package. Its details are not
discussed here because the XML format adopted is specific to this WU-developed software which
is not widely used. The \code{exams2lops()} interface then essentially proceeds in the following manner:
\begin{Code}
htmltransform <- make_exercise_transform_html(converter = "tex2image",
base64 = FALSE)
lopswrite <- make_exams_write_lops(...)
xexams(file, n, nsamp, driver = list(
sweave = list(quiet = TRUE, pdf = FALSE, png = TRUE, ...),
read = NULL,
transform = htmltransform,
write = lopswrite),
...)
\end{Code}
First, an HTML \code{transform} driver is set up which uses the \code{"tex2image"} converter
because the LOPS server does not support MathML. Then, it sets up the custom
\code{write} driver using a couple of extra arguments (\code{...}) whose details are suppressed here
for simplicity. Finally, \code{xexams()} is called with (1)~the default \code{sweave} driver
\code{Sweave()} with options set to producing PNG but not PDF graphics,
(2)~the default \code{read} driver, (3)~the \code{tex2image}-based {\TeX}-to-HTML \code{transform}
driver, (4)~the custom \code{write} driver.
Of course, the part that involves a certain amount of coding is to program the \code{write}
driver (or driver generator, as here). However, the building blocks for the weave/read/transform
steps can be easily recycled. Also, if readers of this manuscript need to code their own driver
generator, we recommend to use the drivers from the \pkg{exams} package for inspiration. Last
but not least, the \pkg{exams} package is hosted and \proglang{R}-Forge \citep{exams:Theussl+Zeileis:2009}
and also provides a forum for support and discussions of e-learning exams in \proglang{R} at
\url{https://R-Forge.R-project.org/forum/?group_id=1337}.
\section{Summary and discussion} \label{sec:discussion}
\subsubsection*{Summary}
Motivated by the need for automatic generation of exams (or quizzes, tests, assessments) for
learning management systems, the \pkg{exams} package is turned into an extensible toolbox
for exam generation. While previous versions of the package just supported generation of
random replications of exams in PDF format, the new version of the package provides
interfaces for various output formats, such as PDF, HTML, or XML specifications for
\pkg{Moodle} or \pkg{OLAT}. All exam output formats are based on the \emph{same}
specification of exercise \code{Sweave} files whose format was only slightly extended
compared to previous versions. The flexibility of producing different output formats
is accomplished by adopting a new extensible framework that consists of the following
modular steps: (1)~\emph{weaving} a single exercise, (2)~\emph{reading} the resulting
{\LaTeX} text and metainformation into \proglang{R}, (3)~\emph{transforming} the text
(if necessary, e.g., from {\LaTeX} to HTML), (4)~\emph{writing} the text into output
files such as {\LaTeX}, HTML, or XML templates. Flexible building blocks are
available for each of the steps that can either be customized for the existing output
formats or reused for generating new output formats.
\subsubsection*{Infrastructure vs.\ content}
As emphasized in the discussion of version~1 of \pkg{exams} \citep{exams:Gruen+Zeileis:2009},
the objective of the package is to provide the technological infrastructure for automatic
generation of exams, especially for large-lecture courses. Thus, users of \pkg{exams}
should not have to worry about implementation details and can focus on the content
of their exams when they build up a pool of exercises accompanying a particular course.
Creating ``good'' exercises from an educational (rather than computational) point
of view is not a trivial task but guidelines for this are beyond the scope of
the \pkg{exams} package and this manuscript. Hence, we just provide a few references
to the relevant literature on statistical education and assessment:
\cite{exams:Gal+Garfield:1997} and \cite{exams:Garfield+Chance:2000}
discuss issues such as topics covered and skills developed in
statistics courses as well as suitable ways of assessment. Strategies for
good multiple-choice questions, especially if they are also used for
self-study materials, are suggested by \cite{exams:Klinke:2004}.
\subsubsection*{Strategies for setting up exercises}
When switching a course to the \pkg{exams} infrastructure, clearly the most
work has to go into the generation of the content, i.e., the \code{Sweave} exercises.
However, by the modular design of the package it is easy to distribute the
workload among a large team of contributors. Each person can just work on
stand-alone \code{.Rnw} files, e.g., for a particular exercises type or for
the exercises pertaining to a specific chapter from the lecture etc. Depending
on the output formats, it is typically a good idea to make sure that
the exercise, \code{foo.Rnw} say, works as desired by running \code{exams2pdf("foo.Rnw")}
and \code{exams2html("foo.Rnw")} to make sure that it can be appropriately rendered
in both PDF and HTML. To check that the solution is correctly entered in the
metainformation, it helps to run \code{exams_metainfo(exams2html("foo.Rnw"))} (or analogously
for \code{exams2pdf()}).
When the pool of exercises is ready, then it is typically useful to set up
a convenience wrapper function that (a)~selects the desired exercises from this
pool and (b)~produces the desired output format(s) for them. For the latter
step, it may just be necessary to set the arguments of one of the \code{exams2}\emph{xyz}\code{()}
functions appropriately or maybe to write a custom \code{template} that can be
plugged into the function. However, the customization of such a wrapper function
is typically not a lot of work and can be performed by a single person, e.g.,
the team member with some more experience in the technologies involved (\proglang{R}, HTML, XML, \dots).
A useful starting point for setting up such a wrapper can be generated with
the \code{exams_skeleton()} function, based on which different interfaces
and templates can be easily explored.
\subsubsection*{Experiences at Universit\"at Innsbruck}
In 2012, the Department of Statistics at Universit\"at Innsbruck built
up infrastructure for a new ``Mathematics 101'' course. The team included
seven professors and lecturers, and six student assistants. All professors
and lecturers were previously familiar with \proglang{R} and {\LaTeX}
(but not necessarily with HTML or XML) while several of the student
assistants had experience in neither. The workload was then split up
so that the professors and lecturers designed the content of the
exercises and programmed prototypes. The student assistants then
typically performed tasks such as checking the correctness of the
exercises, testing out the random data generation or making it more
flexible, and creating variations of existing exercises by making
small modifications in the underlying ``stories'' or changing the
data generating process. Even though many of the student assistants
had no prior knowledge of \proglang{R} and {\LaTeX}, they were
rather quickly able to work on the exercise \code{Sweave} files (with
all the usual small problems that often occur when learning \proglang{R}/{\LaTeX}).
The resulting pool of exercises is maintained in a \pkg{Subversion} repository
\citep[SVN,][]{exams:Pilato+Collins-Sussman+Fitzpatrick:2004} for
version control so that all team members can easily obtain the latest
version or contribute fixes/improvements. In combination with the
\pkg{exams} package this approach proved to be rather successful in
addressing the needs of multi-author and cross-platform development.
After having the pool of exercises established, just one team member
is concerned with running \code{exams2qti12()} and uploading the resulting
ZIP file into \pkg{OLAT} for the biweekly online tests. And
for creating the printed tests at the end of the semester the
\code{exams2lops()} or \code{exams2pdf()} interfaces are employed.\footnote{Meanwhile,
further OCR infrastructure for scanning multiple/single-choice exams
within \proglang{R} has been developed, see \code{exams2nops()}.}
\subsubsection*{Experiences at BOKU Wien}
At BOKU a web-based online exercise system had been in place for
several years \citep{exams:Moder:2011}. The old system used
\proglang{Fortran} to generate data and a standalone web-interface for
students to enter results and get feedback. The storylines of the old
system were transfered into \code{Sweave} files, the \proglang{Fortran} code
was re-programmed in \proglang{R} for more flexibility in random
number generation. Workload management was similar as in Innsbruck:
Professors and lecturers supervised the effort, but programming of
examples was done by a team of PhD students, two of which were new at
the department and had only limited prior experience with \proglang{R} (but all had
some experience in {\LaTeX}).
There are three types of \pkg{exams}-generated exercises used
throughout three ``Statistics 101'' courses totalling more than 1200
students:
\begin{itemize}
\item Online exercises where students get immediate feedback on
correctness of results,
\item homework exercises where students create a small report as PDF
and upload this to the server, and
\item pen-and-paper multiple-choice exams as in package \pkg{exams}
version~1.
\end{itemize}
For the pen-and-paper multiple-choice tests a mixture of
\code{exams2pdf()} and \code{exams2moodle()} is currently used: Moodle
in principle generates exercise and answer sheets for multiple-choice
exams. However, in the current implementation the question text may
not contain figures and mathematical equations are really ugly. So
question sheets are generated directly as PDF, but the XML for Moodle
is also created, which generates then in turn the answer sheets and is
used for automatic scanning and grading. A much more detailed
description of contents and example types has been written as a
seperate manuscript and is currently under review. The focus of this
paper is to be a technical manual of the new features of package
\pkg{exams}.
\subsubsection*{Outlook}
In the current version, \pkg{exams} already provides a wide variety of
different output formats, some additional formats may be desirable for future
developments though. For example, QTI~2.0/2.1 is likely to become more
widely adopted -- and is already currently employed by some programs
such as \pkg{ONYX}. The current version of \pkg{exams} already provides
a new \code{exams2qti21()} function which so far has only been tested
with \pkg{ONYX}. Hence, the function will probably be changed and
improved somewhat in future versions of the package.
Furthermore, we have started to explore support for the proprietary
\pkg{Blackboard} system. While this is, in principle, based on QTI~1.2,
\pkg{Blackboard} employs its own and somewhat special flavor of QTI~1.2.
Finally, users may be interested in extensions/adaptions of existing
e-learning formats. A forum for support and discussions of such issues
is available on \proglang{R}-Forge at
\url{https://R-Forge.R-project.org/forum/?group_id=1337}.
\section*{Acknowledgments}
We are indebted to all our colleagues in the ``Mathematics'' team at the Department of Statistics
at Universit\"at Innsbruck for testing and challenging the code and making suggestions for improvement.
Furthermore, code patches, discussions, and feedback by Dominik Ernst (Universit\"at f\"ur Bodenkultur Wien)
and Niels Smits (Vrije Universiteit Amsterdam) are gratefully acknowledged.
This project was also partially supported by an e-learning grant (\#2011.241)
of the Universit\"at Innsbruck.
\bibliography{exams}
\newpage
\begin{appendix}
\section[List of Sweave exercises in exams]{List of \texttt{Sweave} exercises in \pkg{exams}}
Table~\ref{tab:exercises} lists all exercises that are currently provided as example \code{Sweave}
files within the \pkg{exams} package. All of these exercises (except \code{confint.Rnw} that is
not compatible with the version~2 interfaces) are copied when setting up an
\code{exams_skeleton()} (see Section~\ref{sec:skeleton}).
\begin{table}[h!]
\centering
\begin{tabular}{llp{10.9cm}}
\hline
File & Type & Description \\ \hline
\code{confint} & \code{num} & Confidence interval for one-sample mean, i.e., result of length two
(for version~1 interface; for version~2 see \code{confint2} below).\\
\code{dist} & \code{num} & Very simple numeric exercise (for introductory illustrations).\\
\code{lagrange} & \code{num} & Lagrange optimzation under constraint. Result is randomly selected to be one of three potential quantities. \\
\code{regression} & \code{num} & Prediction in simple linear regression.\\
\code{tstat} & \code{num} & Computation of 1-sample $t$~statistic.\\ \hline
\code{anova} & \code{mchoice} & Interpretation of \code{anova()} table and corresponding boxplots.\\
\code{boxplots} & \code{mchoice} & Interpretation of two parallel boxplots with potentially varying location, scatter, skewness, and outliers.\\
\code{cholesky} & \code{mchoice} & Computation of Cholesky decomposition, result is checked by randomly
constructed statements about different matrix elements.\\
\code{relfreq} & \code{mchoice} & Interpretation of 2-way contingency table.\\
\code{scatterplot} & \code{mchoice} & Interpretation of scatterplot.\\
\code{ttest} & \code{mchoice} & Interpretation of \code{t.test()} output.\\ \hline
\code{tstat2} & \code{schoice} & Single-choice version constructed from the numeric \code{tstat}.\\ \hline
\code{function} & \code{string} & Simple string exercise asking for the name of specific \proglang{R}~functions.\\ \hline
\code{boxhist} & \code{cloze} & Based on randomly generated data as (exercise-specific) \code{.csv} files
some quantiles have to be computed (\code{num}) and interpretations of a boxplot
and histogram have to be made (\code{mchoice}). \\
\code{boxhist2} & \code{cloze} & Same exercise but with custom layout of the answer fields (for \code{exams2moodle()} only). \\
\code{confint2} & \code{cloze} & Cloze version (for all version~2 interfaces) constructed from the numeric
exercise \code{confint}.\\
\code{dist2} & \code{cloze} & Extended cloze version of the numeric \code{dist}.\\
\code{fourfold} & \code{cloze} & Computation of joint probabilities in a fourfold table
based on given (conditional) probabilities.\\
\code{fourfold2} & \code{cloze} & Same exercise but with table layout of the answer fields (for \code{exams2moodle()} only). \\ \hline
\code{currency8} & \code{num} & Exercise in UTF-8 encoding (with Euro and Pound symbol
and German umlaut).\\
\code{currency9} & \code{num} & Same exercise in Latin-9 (or ISO-8859-15) encoding.\\
\code{currency1} & \code{num} & Same exercise in Latin-1 (or ISO-8859-1) encoding (omitting
the Euro symbol).\\ \hline
\end{tabular}
\caption{\label{tab:exercises} List of \texttt{Sweave} exercises provided as examples
in \code{exams/inst/exercises}.}
\end{table}
\pagebreak
\section{Evaluation policies} \label{sec:eval}
Evaluation of many exercise types generated by \pkg{exams} is relatively
straightforward: \code{num}, \code{schoice}, and \code{string} answers
can either be correct or wrong (possibly allowing for some tolerance
in \code{num} answers). However, for \code{mchoice} and \code{cloze}
exercises there is more flexibility: Either all parts of an answer
have to be exactly correct or partial credits can be assigned.
Furthermore, even if answers can only be correct or wrong, there is
an additional degree of freedom: Either wrong answers are not
penalized (and thus do not differ from unanswered exercises) or
negative points can be assigned to wrong answers. In this case, one
needs to distinguish between exercises that were answered incorrectly
and not attempted at all.
To conceptualize these different evaluation policies and provide some
auxiliary functions for evaluating exam results within \proglang{R},
\pkg{exams} implements the function
\begin{Code}
exams_eval(partial = TRUE, negative = FALSE,
rule = c("false2", "false", "true", "all", "none"))
\end{Code}
where \code{partial} signals whether partial credits should be employed
in \code{mchoice} exercises, \code{negative} indicates whether negative
points are possible or not, and \code{rule} specifies the strategy
for partial credits.
The function \code{exams_eval()} returns a list of its arguments along
with several auxiliary \proglang{R} functions that can compare a given
\code{answer} with the corresponding \code{correct} answer and assign
point percentages. The details of these functions are illustrated with
many examples on the corresponding manual page \code{?exams_eval}.
The function \code{exams_eval()} itself will be most useful for
\pkg{exams} users that obtain exam results themselves (as opposed
to a learning management system), e.g., through optical character recognition
or through their own custom web form. In such a situation,
\code{exams_eval()} will provide useful building blocks for a
custom evaluation policy.
More importantly, the same ``vocabulary'' for describing evaluation
policies can be used in the \code{exams2qti12} and \code{exams2moodle}
interfaces:
\begin{itemize}
\item \pkg{Moodle} only provides partial evaluation of multiple-choice
exercises and the user can only choose how to assign partial credits.
Every selected correct choice will always yield the fraction 1/\#correct
of points. When an incorrect choice is selected, it should lead to
negative points. Five strategies are currently implemented: \code{"false"}
uses 1/\#wrong while \code{"false2"} uses $1/\max(\#\mathrm{wrong}, 2)$;
\code{"true"} uses 1/\#correct (so that each wrong selection cancels
one correct selection); \code{"all"} uses 1 (so that a single wrong
selection cancels all correct selections); and \code{"none"} uses 0
(so that wrong selections have no effect at all). Finally, the overall
points of an exercise can never become negative.
\item In QTI~1.2/\pkg{OLAT}, there is some more flexibility. In
principle partial credits can be switched off for multiple-choice
questions and also the points assigned to an exercise can become
negative.
\end{itemize}
Hence, \code{exams2moodle} only implements the \code{rule} argument
while \code{exams2qti12} implements all three arguments (and would also
allow to set them differently for different exercise types).
In both interfaces, \code{cloze} exercises always use partial credits
without negative points for wrong answers in one of its components.
\section{Cloze exercises} \label{sec:cloze}
\begin{figure}[b!]
\centering
\includegraphics[width=0.95\textwidth]{screenshots/exams2qti12-boxhist.png}
\caption{\label{fig:exams2qti12-boxhist} Display of \code{boxhist} exercise in
\pkg{OpenOLAT} (as rendered by a \pkg{Firefox} browser).}
\end{figure}
Many large-scale exams focus on multiple- or single-choice exercise because these
are easy to evaluate, especially when conducted in classical printed form. However,
e-learning exams offer many more possibilities beyond the multiple/single-choice
format by combining different types of questions in a single exercise. The
\pkg{exams} package provides \code{cloze} exercises that can combine numeric,
multiple/single-choice, and string questions, see Section~\ref{sec:exercises}
for an overview. As this type of exercises is very flexible, it can be both,
very useful especially for statistics exercises but also somewhat more challenging
to set up. Hence, we provide some additional explanations for cloze exercises here.
Figure~\ref{fig:exams2qti12-boxhist} shows the \code{boxhist} exercise (see also
Table~\ref{tab:exercises}) as a typical cloze in \pkg{OpenOLAT}. It was produced by
<<exams2qti12-boxhist, eval=FALSE>>=
set.seed(1090)
exams2qti12("boxhist", n = 1, name = "boxhist")
@
It provides the link to a data file with comma-separated values (CSV) that can be
downloaded by the test takers and analyzed. There are six questions: three
single-choice questions (about modality, skewness, and outliers) and three numeric questions
about quartiles. In \pkg{Moodle} the display is similar but drow-down menus are used
for the single-choice questions.
To set up such a \code{cloze} exercise, separate solutions have to be provided for all questions,
in this particular random version as
\verb+\exsolution{10|001|10|2.54|3.9|3.72}+ along with the corresponding types as
\verb+\exclozetype{schoice|schoice|schoice|num|num|num}+. The individual questions
shown in the exercise are taken from the \code{answerlist} within the \code{question}
which has to consist of $10$ ($= 2 + 3 + 2 + 1 + 1 + 1$) items. Hence, if a
corresponding \code{answerlist} is also provided in the \code{solution}, it also has to
consist of 10~items. (Alternatively, the \code{solution} can also have no \code{answerlist} at all.)
See the \code{boxhist} (or the \code{fourfold}) exercise in the \pkg{exams} package
for examples how to dynamically produce all metainformation commands in the correct format.
\begin{figure}[t!]
\centering
\includegraphics[width=0.7\textwidth]{screenshots/exams2moodle-fourfold.png}
\vspace*{0.2cm}
\includegraphics[width=0.7\textwidth]{screenshots/exams2moodle-fourfold2.png}
\caption{\label{fig:exams2moodle-fourfold} Display of \code{fourfold} (top) and \code{fourfold2} (bottom)
exercise in \pkg{Moodle} (as rendered by a \pkg{Firefox} browser).}
\end{figure}
Furthermore, the \pkg{Moodle} XML format allows for finer control of the arrangement of the
answer fields in the question. By default, all questions are collected at the end of the
exercise (based on the corresponding \code{answerlist}). Alternatively, the answer fields
for the individual questions can be placed anywhere within the text, arranged in a particular
layout (e.g., a matrix or a table) etc. To do so, one simply has to place fields
\verb+##ANSWER1##+, \verb+##ANSWER2##+, \dots within the \code{question} text. Then,
\code{exams2moodle()} will replace the placeholders by the corresponding answer fields (i.e.,
drop-down menus or numeric entry fields). The \code{boxhist2} exercise in \pkg{exams} shows
how to do this as an alternative to \code{boxhist}.
Another illustration for a cloze exercise is given by \code{fourfold} and \code{fourfold2}, respectively.
The exercise provides some (conditional) probabilities and asks about the corresponding
joint probabilities in the associated fourfold table. In \code{fourfold} (see the upper panel of
Figure~\ref{fig:exams2moodle-fourfold}) the standard layout as a list of question fields
is used while \code{fourfold2} employs the custom \pkg{Moodle} layout (see the lower panel of
Figure~\ref{fig:exams2moodle-fourfold}), actually displaying a fourfold table (plus marginal
sums). Both versions have again been created after \code{set.seed(1090)}. Users of the
\pkg{exams} infrastructure thus have to decide whether they need more flexibility through
setting up clozes that can be processed by all \code{exams2}\emph{xyz}\code{()} interfaces
(such as \code{boxhist} or \code{fourfold}) or whether they prefer to have a nicer layout
(as in \code{boxhist2} or \code{fourfold2}). If the latter is chosen, then we recommend
to always include a \code{answerlist} in the \code{question} even in cases where it is not necessary
(as in \code{fourfold2}) so that the exercise can be processed without error in
\code{exams2html()} or \code{exams2pdf()} when programming/checking the exercise.
\end{appendix}
\end{document}