Andrew M. Raim

Vim as a Simple (R) IDE without Plugins II

Fri, 09 Jan 2026 05:00:00 GMT

1 Introduction

This post is a sequel to Vim as a Simple IDE without Plugins. There we showed how to send text from an open file in Vim to a console running another buffer in the same Vim session. This lets us step through scripts in R, Python, Julia, etc interactively without having to start a full IDE like Rstudio, or even necessarily be in a graphical environment - we just need a terminal session.

In this post, we make a simple popup menu to work with R packages. If our current working directory is within an R package, the menu will allow us to build & install the package, generate its documentation, run a CRAN check, etc. That will take place in a terminal buffer running in the same Vim session. This post is specific to R, but the pattern might extend to other languages.

The complete code is given in a vimrc file; this also includes the material from the prequel. As before, you can copy or import the contents into your own vimrc file to make use of them.

The final result is demonstrated in Figure 1, where the following sequence is shown.

Initially start Vim in a folder which is not an R package and see an alert when we try to bring up the menu.
Exit Vim and change to another folder that is part of a package.
Start Vim, try to build the package documentation, but get informed that no R terminal is running.
Start an R terminal in Vim and exchange windows so that terminal is on the right side of the screen.
Build package documentation and PDF manual using the menu.
Run a line from the script window to view a manual page from the package.
Switch focus to the terminal, close the manual page, and exit the session.

Figure 1: Demonstration.

2 Are we in an R Package?

Suppose our current working directory is /path/to/folder. To determine whether we are in an R package, we check for the existence of a DESCRIPTION file in the current directory; if found, that will be considered the root of the project. Otherwise, we repeat the check in the parent directory /path/to. We proceed this way until we reach / where there are no more parents to check. This follows the idea from the find_package_root function in the desc R package.

The following Vimscript function carries this out.

function! RProjectRoot(path) abort
    let path = a:path
    let done = v:false

    while done == v:false
        let ff = expand(path) . "/" . 'DESCRIPTION'

        if filereadable(ff)
            return path
        endif

        let newpath = fnamemodify(path, ':h')
        let done = (newpath == path)
        let path = newpath
    endwhile

    throw "Not within an R project: " . a:path
endfunction

The fnamemodify function is used to obtain the parent directory of the current path. If the argument path is the root, the result newpath will also be the root, and we have nothing else to check. An exception is thrown if we cannot find any DESCRIPTION file when we reach the root directory.

3 Menu Operations and R Functions

If we determine that we are in an R package, our menu will carry out selected actions via corresponding calls to the R devtools package. This mapping is shown in Table 1.

Table 1: Menu operations and corresponding devtools functions.

Action	Function
`Load`	`load_all`
`Install`	`install_local`
`Document`	`document`
`Test`	`test`
`Clean`	`clean_dll`
`Manual`	`build_manual`
`Vignettes`	`build_vignettes`
`Build Source`	`build` with `binary = FALSE`
`Build Binary`	`build` with `binary = TRUE`
`Check`	`check`

4 Menu in Vim

For the Vim menu, we will use the popup_menu function. It takes an array of menu options and a dictionary of additional named arguments. We will set two named arguments: the title of the menu and a handler. The handler is another function that is called when an item is selected in the menu; its job is to take the selection and carry out the desired action. Both of these functions are defined within an RProjectMenu function, is invoked to bring up the menu.

The following snippet is an abridged version of RProjectMenu, which can be seen in vimrc. The MenuHandler function is somewhat long, so the contents are summarized as comments.

function! RProjectMenu() abort
    function! MenuHandler(id, idx)
        " 1. Get the selected menu item
        " 2. Call RProjectRoot to see if we are in an R project; if so,
        "    get the path to the root. If not, print a message to the
        "    user and return.
        " 3. Form a string with the corresponding devtools command.
        " 4. See if there is a terminal running R. If not, alert the
        "    user and return.
        " 5. If we made it this far, send command to the terminal
        "    running R.
    endfunction

    try
        let projroot = RProjectRoot(getcwd())
    catch /.*/
        echom v:exception
        return
    endtry

    let items = [ 'Load', 'Install', 'Document', 'Test', 'Clean', 'Manual',
    \ 'Vignettes', 'Build Source', 'Build Binary', 'Check' ]

    let args = #{
    \ title: 'R Project: ' . pathshorten(expand(projroot)),
    \ callback: 'MenuHandler'
    \ }

    call popup_menu(items, args)
endfunction

Note

In our implementation, we look for a terminal by name using the function call bufnr('R'). The 'R' name will be set if we start the terminal using the method developed in the prequel post, or if we start it manually via :term('R') or :vert term('R'). These actions start a dedicated terminal for R which will exit when the R session ends.

It is also possible to repurpose a running terminal by changing its buffer name. To do this, take the following steps.

Change focus to the terminal window.
Enter terminal-normal mode using keystrokes (Ctrl-\ then Ctrl-n).
Enter the command :file R'.
Type i to return to terminal-insert mode

5 Keybinding

We can set a keybinding to easily call the menu. In the prequel post, we set the key to space bar. Here we set r (i.e., then r) as the binding to invoke the menu.

nnoremap  r :call RProjectMenu()

We should now be able to use the menu as in Figure 1.

Customized Bash Prompt

Tue, 06 Jan 2026 05:00:00 GMT

1 Introduction

I have used Bash for the vast majority of time as a Unix / Linux user. It is always the default shell and sometimes I don’t have access to change it. Modern shells like Zsh and Fish appear to offer better aesthetics and more information; however, we can also customize the Bash prompt. Here I will show some preferred customizations; Figure 1 displays the results after implementing them.

Figure 1: Prompt after customizations.

The complete script is here. To activate it in a Bash session, use the source command as follows.

source /path/to/prompt

To apply it to new sessions, include the line above in your ~/.bashrc file.

2 Environment Variables for the Prompt

The prompt can be set by manipulating the PS1 and PS2 environment variables. PS1 is the primary prompt and PS2 is the “continuation” prompt which is shown when a command spans more than one line. There are also PS0, PS3 and PS4 prompts used in other particular situations, discussed in the bash manual page.

The defaults for PS1 and PS2 are usually something like the following. There may also be codes added to make the display colorful, but we will omit them for now.

PS1="[\u@\h:\w]\$ "
PS2="> "
[araim@localhost:~/.config/systemd/user]$

The prompts appear as follows.

[araim@localhost:~/.config/systemd/user]$ echo "hello world"
hello world
[araim@localhost:~/.config/systemd/user]$ for i in 1 2 3
> do
> echo "hello world $i"
> done
hello world 1
hello world 2
hello world 3
[araim@localhost:~/.config/systemd/user]$

3 Placeholders and Aesthetic Prompt Symbol

The strings \u, \h, and \w are placeholders for the username, hostname, and current working directory. See the PROMPTING section of the Bash manual page for the list of available placeholders.

The typing area can start to feel cramped if there is a lot of information in front of it, so let’s include a newline after the information. For aesthetics, let’s also replace the ascii > with the unicode glyph ❱ “Heavy right-pointing angle bracket ornament” which has hex code 2771.

PS1='\u@\h'
PS1+=':'
PS1+='\w'
PS1+='\$'
PS1+='\n'
PS1+='❱'
PS1+=' '

PS2="❱ "

This is how it looks now.

araim@localhost:~/.config/systemd/user$
❱ echo "hello world"
hello world
araim@localhost:~/.config/systemd/user$
❱ for i in 1 2 3
❱ do
❱ echo "hello world $i"
❱ done
hello world 1
hello world 2
hello world 3
araim@localhost:~/.config/systemd/user$

The \h element can become very long for deeply nested paths. We can limit the number of parent directories in the display using the PROMPT_DIRTRIM environment variable.

araim@localhost:~/.config/systemd/user$
❱ mkdir -p very/long/path
araim@localhost:~/.config/systemd/user$
❱ cd very/long/path
araim@localhost:~/.config/systemd/user/very/long/path$
❱ PROMPT_DIRTRIM=3
araim@localhost:~/.../very/long/path$
❱

4 Custom Information with Function Calls

If our current working directory is within a Git repo, it could be useful to have the repo name and current branch displayed on the prompt. Let’s define a Bash function to gather that information.

prompt_git() {
    basedir=$(git rev-parse --show-toplevel 2> /dev/null) || return
    repo=$(basename ${basedir})
    branch=$(git rev-parse --abbrev-ref HEAD)
    echo -ne " ➜${repo}:${branch}"
}

The arrow glyph ➜ is the unicode “Heavy round-tipped rightward arrow” character with hex code 279C. Include a call to prompt_git in our primary prompt.

PS1='\u@\h'
PS1+=':'
PS1+='\w'
PS1+='\$'
PS1+='$(prompt_git)'
PS1+='\n'
PS1+='❱'
PS1+=' '

❱ cd ~/.config/systemd/user
araim@localhost:~/.config/systemd/user$                ## Not a git repo
❱ cd vimrc/
araim@localhost:~/.../systemd/user/vimrc$ ➜vimrc:main  ## A git repo
❱

Another useful piece of information would be the return code $? which was set by from the previous command. But let’s only display if is something other than zero.

prompt_ecode() {
    [[ "$1" == "0" ]] || echo -ne "[$1] "
    return $1
}

The function prompt_ecode takes one argument which is expected to be the value of $? set by the previous command. The caller must ensure we get this information before $? is reset by another operation. We return $1 to put the variable $? back to the same state, perhaps for another function to use it.

Note

There is also a PROMPT_COMMAND environment variable that can be used to take some action before PS1 is displayed. This can also be used to check the last command’s return code. However, if PROMPT_COMMAND is doing something to set the return code in $?, it may interfere with prompt_ecode.

PS1='$(prompt_ecode $?)'
PS1+='\u@\h'
PS1+=':'
PS1+='\w'
PS1+='\$'
PS1+='$(prompt_git)'
PS1+='\n'
PS1+='❱'
PS1+=' '

Here is how it looks.

araim@localhost:~/.config/systemd/user$
❱ cd vimrc
araim@localhost:~/.../systemd/user/vimrc$ ➜vimrc:main
❱ ech "hello world"
bash: ech: command not found
[127] araim@localhost:~/.../systemd/user/vimrc$ ➜vimrc:main
araim@localhost:~/.../systemd/user/vimrc$ ➜vimrc:main
❱ echo "hello world"
hello world
❱

5 Colors

Let’s apply some basic colors. This will help to distinguish the prompt from other activity in the terminal. Define environment variables for three colors and insert them into the primary prompt.

COL_FG='\033[38;5;016m'
COL_BG='\033[48;5;002m'
COL_OFF='\033[0m'
PS1="\[${COL_FG}${COL_BG}\]"
PS1+='$(prompt_ecode $?)'
PS1+='\u@\h'
PS1+=':'
PS1+='\w'
PS1+='\$'
PS1+='$(prompt_git)'
PS1+="\[${COL_OFF}\]"
PS1+='\n'
PS1+='❱'
PS1+=' '

Here, \033[ is an ASCII control sequence, 38;5;016m sets the foreground color (38;5) to black (016), and 48;5;002m sets the background color (48;5) to green (002). The charater m signifies the end of the sequence. Color codes are wrapped in \[ ... \] to mark them as nonprintable characters: if this is omitted, the terminal will be glitchy when we are editing commands and traversing the command history. The website colors.sh provides a nice interface to browse terminal colors find the corresponding codes. Figure 2 shows the result so far.

Figure 2: Prompt with colors.

6 Informative Colors

It may be useful to change the color when there is a non-zero return code from the previous command, to serve as a visual indicator. We add another function to handle the colors as follows. Again, argument $1 should be the return code from the previous command; we return it in case we need it subsequently.

prompt_color() {
    COL_FG="\033[38;5;016m"
    COL_CLR_BG="\033[48;5;002m"
    COL_SET_BG="\033[48;5;003m"

    if [ "$1" == "0" ]; then
        echo -ne "${COL_FG}${COL_CLR_BG}"
    else
        echo -ne "${COL_FG}${COL_SET_BG}"
    fi

    return $1
}

Now the prompt background changes to olive / yellow if a nonzero status code was set by the last command.

Remove the hard-coded colors from the beginning of PS1 and call prompt_color instead.

PS1='\[$(prompt_color $?)\]'
PS1+='$(prompt_ecode $?)'
PS1+='\u@\h'
PS1+=':'
PS1+='\w'
PS1+='\$'
PS1+='$(prompt_git)'
PS1+="\[${COL_OFF}\]"
PS1+='\n'
PS1+='❱'
PS1+=' '

This should result in the prompt from Figure 1.

Vim as a Simple IDE without Plugins

Sat, 03 Jan 2026 05:00:00 GMT

1 Introduction

There are a vast number of options for text editors, but one compelling reason to stick with the classic Vim is that it comes packaged with many Unix-based operating systems such as Linux. It is free, open source, and available almost everywhere. Despite its humble appearance, Vim is a powerful tool with extensive features and can be heavily customized.

This post demonstrates some configuration changes that will let us use Vim as a simple integrated development environment (IDE), without the need for any additional plugins or a graphical environment. Here, one Vim window can be used to edit and run code and a second Vim window can be running a console. While editing the code, we can use a few keystrokes to run snippets in the console. We also make it easy to launch the console when we are editing a file. A small demonstration of the result is given in Figure 1; here we launch an R console using one of our keybindings, issue some R commands from the script using several other bindings, and then close Vim.

Figure 1: A quick demonstration of Vim as an IDE.

A file with the all of the functions and keybindings is given here. The easiest way to use it is to paste the contents into your own vimrc file; however, it is also possible to use a more modular configuration with multiple files.

This material was developed using version 9.1.1882 of Vim. The embedded terminal that we will need was introduced in version 8.1.

If you would prefer more sophisticated IDE capabilities than DIYing, see vim-slime, R.nvim for R within Neovim, julia-repl-vim for Julia. There are probably also some good options for tmux if you’d like persistant terminal sessions or are not primarily focused on Vim.

Caution

The code shown in this post is written in the Vimscript scripting language. I am not very familiar with Vimscript, but was able to get things working with some help from AI and various forum posts. There are certainly opportunities for improvement in this code.

Notice, for example in Figure 1, some of the content appears out of order as it is sent to the console. This is only a visual artifact: the code actually runs correctly, but we would prefer the display to be correct as well.

2 Embedded Terminal

We first note that Vim has its own internal terminal. We will make use of this to run an intrepreter for our language(s) of choice. To start a terminal from Vim, enter :term from normal mode. This starts the terminal in a horizontal split. See the documentation on windows on how to change focus between windows, change their positions, resize them, etc. Exiting the terminal closes the split. We can also start a terminal with a vertical split using :vert term from normal mode.

Suppose we have a vertical split with an R script on the left and a terminal running the R console on the right. We would like to be able to send text from the editor to be execute in the R console without too many keystrokes. We would also like the ability to quickly spawn that R console when we are working on an R script.

We will now define two functions.

3 Send Text to the Terminal

Our first function takes a string content and sends it to the terminal. There may be multiple terminals; in this simple implementation - and because I have never used more than one terminal in the same session - we will always send our content to the first terminal in the list.

function! SendToTerm(content)
    let tl = term_list()

    if len(tl) == 0
        echom "No terminal buffer found"
    else
        call term_sendkeys(tl[0], a:content)
    endif
endfunction

The function term_list() returns an array of terminal buffer numbers. If there are no terminal buffers, a notification to the user is printed with echom. Otherwise, the argument content is sent to the selected terminal using term_sendkeys.

4 Spawn a Terminal with a Console Program

The second function spawns a terminal with a suitable program to handle the type of file we are currently editing. If we don’t have a specific program in mind, we will spawn the terminal and just leave it at the command line.

The function takes a boolean argument vert; the terminal will be started with a vertical split if vert is true, otherwise a horizontal split will be used. After the terminal is spawned, we exchange positions with the previous window. In vertical mode, this puts the script on the left and the terminal on the right. In horizontal mode, the script will be on top and the terminal will be underneath. This will also shift focus back to the script.

function! FileTypeToTerm(vert)
    if &filetype == 'r'
        let cmd = "R\n"
    elseif &filetype == 'julia'
        let cmd = "julia\n"
    elseif &filetype == 'python'
        let cmd = "python\n"
    else
        let cmd = ""
    endif

    if a:vert
        execute "vertical terminal! " . cmd
    else
        execute "terminal! " . cmd
    endif

    execute "normal! \x"
endfunction

I have defined programs for three specific languages. Of course, you can modify this for your own cases.

Tip

The strings 'r', 'julia', and 'python' represent designated file types in Vim. To display the file type for the currently open buffer, use the command :echo(&filetype) in normal mode.

Tip

When cmd is a non-empty command, the terminal will halt when the command exits. However, the Vim window will remain open in the halted state until we exit (e.g., via :q). This allows us to note any error messages from our code, in case the console crashed or the exit was otherwise not intentional.

5 Keybindings

We will set several keybindings to interact with our functions. First, let us define the leader key, assuming it has not been defined already. A space is commonly used for this purpose.

let mapleader = " "

The following to send content from a script to the console.

nnoremap   :call SendToTerm(getline('.')."\n")g$
\:call search('\S', 'W'):nohl
xnoremap    mode() ==# 'V' ?
\ 'y :call SendToTerm(@")`>' :
\ 'y :call SendToTerm(@" . "\n")`>'

The nnoremap binding is used in normal mode. It sends the current line under the cursor to the console and moves cursor to the next non-whitespace character after the line. The move is accomplished by pattern-matching for \S, with option W to avoid wrapping back to the beginning when we have reached the last line of non-whitespace. The command :nohl is used to suppress highlighting during pattern searching.

The xnoremap binding handles two distinct cases. The first case handles visual line mode, where entire lines are selected at a time. Here we send the entire selection - which already includes a newline at the end - to the console, then move the cursor at the end of the selection using then >.

The second case handles regular visual mode which is character-by-character. Here the selection ends with a non-newline character, so we explicitly send a newline at the end.

The normal mode binding allows us to type repeatedly to step through the script and run commands line-by-line. The visual mode bindings allow us to highlight text in visual mode and type to send it to the console.

The next set of keybindings initializes our console: th opens a console with a horizontal split and tv opens a console with a vertical split.

nnoremap  th :call FileTypeToTerm(v:false)
nnoremap  tv :call FileTypeToTerm(v:true)

vws: Vertical Weighted Strips in R using C++

Wed, 05 Nov 2025 05:00:00 GMT

Raim et al. (2024+) consider a method of proposal construction for rejection sampling referred to as vertical weighted strips (VWS). Here, the target density is regarded as a weighted density; i.e., the product of a weight function and a simpler (“base”) density function. The practitioner majorizes the weight function by providing another function that bounds it from above. The majorizing function is combined with the base density to form a proposal distribution. In many cases, it is useful (and also more convenient) to partition the support of the target and use a piecewise majorizer. In this case, the proposal distribution can be regarded as a finite mixture whose components are correspond to the regions in the partition.

The vws package has been developed to support development of such proposals and their use in rejection sampling. Coding is primarily done in C++ using the provided API. Sampling functions may be then exposed in R via Rcpp for use in applications. A detailed guide to vws programming - including several worked examples - is provided in the package vignette.

vws package

Deployed on CRAN
Source on Github

References

Raim, Andrew M., James A. Livsey, and Kyle M. Irimata. 2024+. “Rejection Sampling with Vertical Weighted Strips.” 2024+. https://arxiv.org/abs/2401.09696.

Self-Tuned VWS within Gibbs and Small Area Estimation

Tue, 23 Sep 2025 04:00:00 GMT

Raim et al. (2025+) consider an application of the vertical weighted strips (VWS) method (Raim et al. 2024+) to small area estimation (SAE). SAE is used in official statistics to augment estimates from sample surveys (“direct estimates”) with a model. “Small areas” are cross-sections of a population - based on geography or other characteristics - where the sample size from the survey is small.

In particular, You (2021) presents an SAE model with regressions on both direct point estimates and corresponding variance estimates. A Gibbs sampler is proposed for Bayesian analysis, with one family of conditionals being an unfamiliar distribution that arises from assuming a lognormal regression in the variance model. VWS may be used to generate from this family for each small area. We find that some of the latent variances mix poorly using the independent Metropolis-Hastings step considered in You (2021), which is a routine choice for use within Gibbs samplers. In these cases, mixing is seen to be greatly improved by taking exact draws from the conditionals with rejection sampling.

It is computationally burdensome to construct new VWS proposals for each small area over the course of many Gibbs sampling iterations. To address this, we consider a rule-of-thumb to adjust the proposals as the chain evolves. In particular, knots are added using rejected draws when the probability of rejection is too large. Knots are removed when the probability of rejection is sufficiently small and their corresponding regions are found to have a low contribution. The number of adjustments is seen to diminish as the chain moves to the target posterior distribution.

References

Raim, Andrew M., Kyle M. Irimata, and James A. Livsey. 2025+. “Self-Tuned Rejection Sampling Within Gibbs and a Case Study in Small Area Estimation.” 2025+. https://arxiv.org/abs/2509.17155.

Raim, Andrew M., James A. Livsey, and Kyle M. Irimata. 2024+. “Rejection Sampling with Vertical Weighted Strips.” 2024+. https://arxiv.org/abs/2401.09696.

You, Yong. 2021. “Small Area Estimation Using Fay-Herriot Area Level Model with Sampling Variance Smoothing and Modeling.” Survey Methodology 47 (2). http://www.statcan.gc.ca/pub/12-001-x/2021002/article/00007-eng.htm.

Numerical Maximum Likelihood with the stats4 Package

Wed, 11 Dec 2024 05:00:00 GMT

1 Introduction

The stats4 package included in R provides a layer of usability on top of optim for numerial maximum likelihood (ML). For example, the user can request an estimate of the covariance matrix associated with the ML estimates rather than computing it manually. According to Henningsen and Toomet (2011), stats4 has been bundled into R since 2003, but I only found out about it recently. In this post, we will present some brief examples using stats4. Note that the maxLik package presented by Henningsen and Toomet (2011) covers similar ground with additional functionality; therefore it may also be of interest.

2 Poisson Regression Example

Consider a Poisson regression setting with , independently distributed for where for covariate and unknown coefficient .

Let us simulate a dataset from this setting.

set.seed(1235)
n = 200
x = rnorm(n)
X = cbind(1, x)
d = ncol(X)
beta_true = c(1, -0.25)
lambda_true = exp(X %*% beta_true)
y = rpois(n, lambda_true)

Of course we can use the glm function to obtain the MLE in this setting. Let us do that as a point of comparison.

glm_out = glm(y ~ x, family = poisson)
summary(glm_out)


Call:
glm(formula = y ~ x, family = poisson)

Coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept)  0.98109    0.04392  22.338  < 2e-16 ***
x           -0.25893    0.03908  -6.626 3.44e-11 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for poisson family taken to be 1)

    Null deviance: 272.04  on 199  degrees of freedom
Residual deviance: 228.85  on 198  degrees of freedom
AIC: 749.28

Number of Fisher Scoring iterations: 5

3 Using the `mle` Function in `stats4`

Recall that the loglikelihood for this model is If this were a nonstandard likelihood, we would need to be able to code and optimize it ourselves. The stats4 package allows us to do this. Here are the arguments of the stats4::mle function which may be used to fit the MLE.

str(mle, give.attr = F)

function (minuslogl, start, optim = stats::optim, method = if (!useLim) "BFGS" else "L-BFGS-B", 
    fixed = list(), nobs, lower, upper, ...)

Note that the first argument expects the negative loglikelihood; the optimization is then carried out as a minimization. Let us code it.

nloglik = function(beta = numeric(d)) {
    -sum(dpois(y, lambda = exp(X %*% beta), log = TRUE))
}

mle_out = mle(nloglik, method = "L-BFGS-B", nobs = length(y))

A variety of included accessors can be used to work with the result.

summary(mle_out)

Maximum likelihood estimation

Call:
mle(minuslogl = nloglik, method = "L-BFGS-B", nobs = length(y))

Coefficients:
        Estimate Std. Error
beta1  0.9810935 0.04392105
beta2 -0.2589335 0.03907591

-2 log L: 745.2773

coef(mle_out)

     beta1      beta2 
 0.9810935 -0.2589335

logLik(mle_out)

'log Lik.' -372.6387 (df=2)

AIC(mle_out)

[1] 749.2773

BIC(mle_out)

[1] 755.874

confint(mle_out, level = 0.90)

Profiling...

             5 %      95 %
beta1  0.9079239  1.052437
beta2 -0.3230551 -0.194514

vcov(mle_out)

             beta1        beta2
beta1 0.0019290582 0.0004114616
beta2 0.0004114616 0.0015269270

profile(mle_out)

An object of class "profile.mle"
Slot "profile":
$beta1
            z par.vals.beta1 par.vals.beta2
1  -3.0192356      0.8453337     -0.2894697
2  -2.5258203      0.8679603     -0.2841584
3  -2.0285449      0.8905870     -0.2789367
4  -1.5273678      0.9132136     -0.2738039
5  -1.0222468      0.9358402     -0.2687596
6  -0.5131396      0.9584668     -0.2638030
7   0.0000000      0.9810935     -0.2589335
8   0.5172066      1.0037201     -0.2541504
9   1.0385329      1.0263467     -0.2494528
10  1.5640206      1.0489733     -0.2448400
11  2.0937147      1.0716000     -0.2403111
12  2.6276606      1.0942266     -0.2358653

$beta2
            z par.vals.beta1 par.vals.beta2
1  -2.5855783      0.9478930     -0.3595864
2  -2.0668516      0.9555116     -0.3394558
3  -1.5489482      0.9626386     -0.3193253
4  -1.0318503      0.9692764     -0.2991947
5  -0.5155395      0.9754273     -0.2790641
6   0.0000000      0.9810935     -0.2589335
7   0.5147983      0.9862771     -0.2388029
8   1.0288654      0.9909803     -0.2186724
9   1.5422257      0.9952050     -0.1985418
10  2.0549007      0.9989531     -0.1784112
11  2.5669123      1.0022264     -0.1582806
12  3.0782832      1.0050269     -0.1381501


Slot "summary":
Maximum likelihood estimation

Call:
mle(minuslogl = nloglik, method = "L-BFGS-B", nobs = length(y))

Coefficients:
        Estimate Std. Error
beta1  0.9810935 0.04392105
beta2 -0.2589335 0.03907591

-2 log L: 745.2773

4 Fixed Parameters

An argument of mle that can be useful is fixed, where we can specify some parameters to be held fixed during the optimization. In our example with let us consider another fit with the slope coefficient fixed at .

mle0_out = mle(nloglik, method = "L-BFGS-B", nobs = length(y),
    fixed = list(beta = c(NA, 0)))
print(mle0_out)


Call:
mle(minuslogl = nloglik, method = "L-BFGS-B", fixed = list(beta = c(NA, 
    0)), nobs = length(y))

Coefficients:
   beta1    beta2 
1.011601 0.000000

Note that NA was associated with to indicate that it should not remain fixed. We can compute a likelihood ratio test of the hypothesis versus using the two fits mle_out and mle0_out.

lrt = 2 * (logLik(mle_out) - logLik(mle0_out))
pval = pchisq(lrt, df = 1, lower.tail = F)
cat("LRT was computed with test statistic", lrt, "and p-value", pval, ".\n")

LRT was computed with test statistic 43.19441 and p-value 4.956214e-11 .

5 Transforming Parameters

A convenient property of ML estimates (MLEs) is the invariance property: suppose the MLE of is , then the MLE of a function is . An associated variance estimate is given by where is the Jacobian of the transformation. Here is an R function to compute the transformed variance. Note that we use the numDeriv package.

vcov_tx = function(object, tx) {
    J = numDeriv::jacobian(func = tx, x = coef(object))
    V = vcov(object)
    J %*% tcrossprod(V, J)
}

As an example, let us estimate each and its variance, and construct an associated confidence interval with level 90% nominal coverage level.

alpha = 0.10
tx = function(beta) { exp(X %*% beta) }
tx_hat = tx(coef(mle_out))
sd_tx_hat = sqrt(diag(vcov_tx(mle_out, tx)))
tx_lo = tx_hat - qnorm(1 - alpha/2) * sd_tx_hat
tx_hi = tx_hat + qnorm(1 - alpha/2) * sd_tx_hat

Here are the results in a data frame.

library(tidyverse)
library(kableExtra)
data.frame(EST = tx_hat, SD = sd_tx_hat, LO = tx_lo, HI = tx_hi) %>%
    mutate(TRUTH = lambda_true) %>%
    head() %>%
    kbl(booktabs = T)

EST	SD	LO	HI	TRUTH
3.195763	0.1463983	2.954960	3.436567	3.236515
3.720229	0.2166837	3.363816	4.076642	3.747967
2.064235	0.1344154	1.843142	2.285329	2.122321
2.591277	0.1170420	2.398760	2.783794	2.643374
2.589646	0.1170497	2.397117	2.782176	2.641767
1.718365	0.1510794	1.469862	1.966869	1.777932

6 References

Henningsen, Arne, and Ott Toomet. 2011. “maxLik: A Package for Maximum Likelihood Estimation in R.” Computational Statistics 26 (3): 443–58. https://doi.org/10.1007/s00180-010-0217-1.

fntl: Numerical Tools for Rcpp and Lambda Functions

Tue, 16 Jul 2024 04:00:00 GMT

Lambda functions were added to C++ in the C++11 specification. They are functions which can be defined on the fly in the course of a program, can make use of variables in the environment within their body, and be passed as arguments to other functions. Use of lambdas may be especially appealing for Rcpp programmers who are accustomed to working with function objects in R but occasionally need C++ when performance becomes a concern.

The objective of the R package fntl (Raim 2024a) is to facilitate programming in Rcpp with lambda functions by providing an API to routinely needed numerical tools such as integration, root-finding, and optimization. Such functions require one or more functions as a primary argument; these are supplied as lambdas to fntl. Where possible, the fntl API utilizes methods exposed from the R API so that the same numerical methods are used as in R. In cases where methods are not exposed from the R API, fntl implements methods that are intended to be comparable. A detailed guide to the fntl API is provided in the package vignette (Raim 2024b).

fntl package

Deployed on CRAN
Source on Github

References

Raim, Andrew M. 2024a. fntl: Numerical Tools for Rcpp and Lambda Functions. https://github.com/andrewraim/fntl.

Raim, Andrew M. 2024b. fntl: Numerical Tools for Rcpp and Lambda Functions. Research Report Series: Computing #2024-01. Center for Statistical Research; Methodology, U.S. Census Bureau. https://www.census.gov/library/working-papers/2024/adrm/RRC2024-01.html.

Rejection Sampling with Vertical Weighted Strips

Fri, 19 Jan 2024 05:00:00 GMT

Rejection sampling is a classical algorithm (Neumann 1951) to generate variates from a target distribution where a constructive sampling method—e.g., by composing other random variables—may not be apparent. A benefit of rejection sampling is that accepted draws follow the target distribution exactly. However, coming up with a good proposal for rejection sampling can take some creativity. A poor choice can result in extremely low acceptance rates where practically no draws are accepted from a very large number of candidates.

Raim et al. (2024+) explore an approach to construct proposals by regarding the target as a weighted density and majorizing the weight function. This yields another weighted density whose unnormalized form can be used as an envelope in the acceptance ratio. If designed consciously, the normalized form can also be used to draw candidate variates to complete the rejection sampler. An improved proposal can be obtained by partitioning the support and majorizing within each region. Here the proposal is a finite mixture. The method is referred to as “vertical weighted strips” because it can be regarded as an extension of the vertical strips method (Devroye 1986, VIII; Martino et al. 2018, sec. 3.6), with the weighted form introducing an additional degree of flexibility for algorithm development.

References

Devroye, Luc. 1986. Non-Uniform Random Variate Generation. Springer.

Martino, Luca, David Luengo, and Joaquín Míguez. 2018. Independent Random Sampling Methods. Springer. https://doi.org/https://dx.doi.org/10.1007/978-3-319-72634-2.

Neumann, John von. 1951. “Various Techniques in Connection with Random Digits.” In Monte Carlo Methods, edited by A. S. Householder, G. E. Forsythe, and H. H. Germond. National Bureau of Standards Applied Mathematics Series. U.S. Government Printing Office, Washington, DC.

Raim, Andrew M., James A. Livsey, and Kyle M. Irimata. 2024+. “Rejection Sampling with Vertical Weighted Strips.” 2024+. https://arxiv.org/abs/2401.09696.

Sending Mail from R

Mon, 01 May 2023 04:00:00 GMT

Here is a quick note on sending mail from R. For example, this could be useful in a long running job to alert yourself when it completes or halts because of an error. The sendmailR package provides a general interface to mail.

Here is an example where we use sendmailR to send mail to the local mailbox of our user (araim) on the host machine. This assumes we have a sendmail server set up to relay local mail and that araim has a mailbox.

The following call in R sends a message.

result = sendmail(
    from = "sendmailR",
    to = "araim",
    subject = "Hello from R",
    msg = "This is an email from R using the sendmailR package.",
    control = list(smtpServer = "localhost")
)

Here is how the message appears when viewed with mutt.

i:Exit  -:PrevPg  :NextPg  v:View Attachm.  d:Del  r:Reply  j:Next  ?:Help
Date: Tue, 02 May 2023 09:24:31 -0000
From: sendmailR@localhost
To: araim@localhost
Subject: Hello from R

[-- Attachment #1 --]
[-- Type: text/plain; charset=us-ascii, Encoding: 7bit, Size: 0.1K --]

This is an email from R using the sendmailR package.

-N +- 1155/1155: sendmailR@localhost   Hello from R                      -- (all)

Here is an example of sending mail on an exception using withCallingHandlers. Here we use sys.calls to give some context about the call that led to the error.

f = function(x) { stop("An exception!") }
g = function(x) { f(x) }

withCallingHandlers({
    g(1:10)
}, error = function(e) {
    result = sendmail(
        from = "sendmailR",
        to = "araim",
        subject = "Exception from R",
        msg = sprintf("%d: %s", seq_along(sys.calls()), sys.calls()),
        control = list(smtpServer = "localhost")
    )
})

And here is how the message appears when viewed with mutt.

i:Exit  -:PrevPg  :NextPg  v:View Attachm.  d:Del  r:Reply  j:Next  ?:Help
Date: Tue, 02 May 2023 10:06:56 -0000
From: sendmailR@localhost
To: araim@localhost
Subject: Exception from R

[-- Attachment #1 --]
[-- Type: text/plain; charset=us-ascii, Encoding: 7bit, Size: 0.6K --]

1: withCallingHandlers({
   g(1:10)
}, error = function(e) {
   result = sendmail(from = "sendmailR", to = "araim", subject = "Exception from
+R", msg = sprintf("%d: %s", seq_along(sys.calls()), sys.calls()), control =
+list(smtpServer = "localhost"))
})
2: g(1:10)
3: f(x)
4: stop("An exception!")
5: .handleSimpleError(function (e) {
   result = sendmail(from = "sendmailR", to = "araim", subject = "Exception from
+R", msg = sprintf("%d: %s", seq_along(sys.calls()), sys.calls()), control =
+list(smtpServer = "localhost"))
}, "An exception!", base::quote(f(x)))
6: h(simpleError(msg, call))

-N +- 1162/1162: sendmailR@localhost   Exception from R                  -- (all)

Comparison of Usability Preferences with Bivariate Ordinal Regression

Thu, 20 Apr 2023 04:00:00 GMT

Suppose respondents are asked to rate several alternatives on a scale from, say, 1 to 5, with 1 being least preferred and 5 being most preferred. How can we determine whether one alternative is significantly more preferred to the others? The data may include covariates which are associated with the preference of interest.

Elizabeth Nichols and I carry out this kind of analysis on map usability in a recent report (Raim and Nichols 2023). With ordinal data recorded in a survey setting, we compare preferences for satellite maps - which display details such as landmarks and terrain - to those of simplified road maps. We make use of a bivariate ordinal regression model to carry out the analysis. Computations are carried out using the mvord R package by Hirk et al. (2020), which can also support multivariate models with more than two alternatives.

References

Hirk, Rainer, Kurt Hornik, and Laura Vana. 2020. “mvord: An R Package for Fitting Multivariate Ordinal Regression Models.” Journal of Statistical Software 93 (4): 1–41. https://doi.org/10.18637/jss.v093.i04.

Raim, Andrew M., and Elizabeth Nichols. 2023. A Comparison of Map Usability via Bivariate Ordinal Analysis. Study Series: Statistics #2023-01. Center for Statistical Research; Methodology, U.S. Census Bureau. https://www.census.gov/library/working-papers/2023/adrm/SSS2023-01.html.

Interactive Map with Shiny and Leaflet

Mon, 27 Feb 2023 05:00:00 GMT

There are powerful tools for working with geographical data in R. For example, the ggplot and sf packages can be used together to display such data. But what about interactive maps which support panning, zooming, and clicks to trigger actions (e.g., highlighting an area)?

One way to accomplish this is through Shiny and Leaflet. With this combination of tools, it is relatively easy to stand up an graphical interface. Shiny supports constructs beyond routine R programming which allow variables, displays, and computations to react to user inputs. Leaflet provides widgets for interactive maps.

There are already a number of interesting examples on the web, but here is something I put together.

It displays counties in Maryland along with some randomly placed markers, and displays the count of markers within each county as a choropleth. Users can remove existing markers or add new ones, which in turn triggers the counts to be recomputed and the choropleth to be updated.

Worker Tool for Computational Studies

Sun, 12 Feb 2023 05:00:00 GMT

Large statistical simulations and other computational studies often consist of a collection of independent tasks to be processed. For example, to study properties of an estimator, we may wish to vary the parameters of an assumed data-generating mechanism along with the sample size. The estimator requires substantial amount of computation because it makes use of a Markov chain Monte Carlo (MCMC) procedure. We generate 500 datasets in each setting, evaluate the estimator, and save the results to be summarized and later. From a parallel computing perspective, such tasks are described as embarrassingly parallel because they do not need to interact while they run.

There are usually many more tasks than available processors. Suppose we are limited to the use of processors; we would like to keep these processors busy with our workload without using additional processors. Ideally, we would like to do this without the need for manual intervention. Worker is a lightweight tool to help automate this process.

Suppose our study consists of a one parameter with 1000 levels. Suppose that we have set up 1000 corresponding directories, named as follows.

$ ls -1
run000
run001
...
run999

In each folder, suppose there is a script launch.R that runs the corresponding level of the simulation. An output file output.csv is placed in the directory upon successful completion of launch.R.

$ ls run000
launch.R    output.csv

We would like to run of the launch.R scripts at a time until the collection of 1000 tasks is complete. The worker script attempts to automate this in a way that is agnostic of the underlying simulation. It loops through tasks and runs them sequentially, but multiple workers can be run simultaneously on the same study to achieve “embarrassingly parallel” parallel computing. Workers make use of lock files to avoid stepping on each other’s toes: each task is taken up by only one worker. This paradigm can be useful in shared computing environments where users may be asked to limit the number of processors requested at any given time. We can start, say, ten workers to ensure ten simultaneous tasks rather than having to continually monitor a scheduler’s queue and submit more tasks.

Here is an example of the syntax used to invoke each worker in this example.

$ worker.sh -p 'run???' -c 'Rscript launch.R'

The worker looks for folders whose names match pattern run???. When such a folder is found, it enters the folder and first checks to see if the task has previously been claimed by a worker (including itself in a previous iteration). If not, it claims the task and launches it via the specified command Rscript launch.R. The task is run in the worker’s foreground until completion. The worker terminates when it is no longer actively engaged and there are no unclaimed jobs remaining.

After all of the runs are completed, we will likely want to load each of the output.csv files and construct tables and plots to summarize the results. This post may be helpful if you are looking to construct Latex tables from a computational study.

Generating Latex Tables in R

Sat, 11 Feb 2023 05:00:00 GMT

Introduction

Preparing Latex tables from results of a simulation study or other program can be tedious and time-consuming when done by hand. Especially when the process must be repeated many tables as results are updated to correct mistakes or to investigate unexpected findings. Hours spent manually adjusting formatting of table entries and typing ampersands between them could be spent in better ways. Fortunately, there are some excellent tools in R to help. Taking time to learn the tools and automate your table generation may be worth the investment. We will give a brief example in this post. This is only based on my experience; there are likely even better tools and methods that I have yet to learn.

Objective

Our objective will be to generate a Latex table for the first six rows of the airquality dataset.

R> head(airquality, n = 6)
  Ozone Solar.R Wind Temp Month Day
1    41     190  7.4   67     5   1
2    36     118  8.0   72     5   2
3    12     149 12.6   74     5   3
4    18     313 11.5   62     5   4
5    NA      NA 14.3   56     5   5
6    28      NA 14.9   66     5   6

We would also like to meet the following criteria.

Criteria for formatting of the rendered Latex table.
1. Use booktabs and make all separators horizontal lines.
2. Use multiline columns to render column names.
3. Add several headers which group together columns in the display.
4. Specify a caption and a label which can be used in Latex references.
5. Request the table to be placed “here”.
Criteria for table content.
1. Include row labels (observation number) in the table as a column.
2. Format numbers in a purposeful way.
3. Display the Month and Day columns together as a date.
4. Customize column names to be more descriptive.
Ensure that the generated Latex code is legible.
1. Pad strings in a given column so that they are nicely aligned in the code.

To do this, we will make use of the following tools.

The Tidyverse framework to manipulate tables.
The knitr package for reproducible research in R. In particular, we will use the kable function. to generate Latex from data frames.
The kableExtra package to handle some additional Latex work.
The built-in sprintf function to format table entries.

Preparing the Data Frame

First let us load the packages we will use.

R> library(knitr)
R> library(dplyr)
R> library(tibble)
R> library(stringr)
R> library(kableExtra)

Before converting to Latex, let us format the entries and column names as we would like them to appear.

tbl = head(airquality, n = 6) %>%
    rownames_to_column(var = "Num") %>%
    mutate(Date = sprintf("%04d-%02d-%02d", 1973, Month, Day)) %>%
    mutate(Solar.R = sprintf("%0.2e", Solar.R)) %>%
    mutate(TempC = sprintf("%0.2f", 5/9 * (Temp - 32))) %>%
    select(Num, Date, Ozone, Solar.R, Wind, TempC)

We have done the following.

Assemble Month and Day into Date, where 1973 is the year which all observations were taken (according to the manual page for the dataset).
Convert Solar.R to a string with the original value in scientific notation.
Convert temperature Temp from Farenheit to Celcius and call the result TempC.
Use tibble::rownames_to_column to include row labels in the table as the Obs column.

This produces the following table.

R> print(tbl)
  Num       Date Ozone  Solar.R Wind TempC
1   1 1973-05-01    41 1.90e+02  7.4 19.44
2   2 1973-05-02    36 1.18e+02  8.0 22.22
3   3 1973-05-03    12 1.49e+02 12.6 23.33
4   4 1973-05-04    18 3.13e+02 11.5 16.67
5   5 1973-05-05    NA       NA 14.3 13.33
6   6 1973-05-06    28       NA 14.9 18.89

Generating Latex

We can now generate code to display out formatted table as Latex code.

out = tbl %>%
    mutate(Solar.R = str_pad(Solar.R, width = 10, side = "left", pad = "\u00A0")) %>%
    mutate(Wind = str_pad(Wind, width = 6, side = "left", pad = "\u00A0")) %>%
    kable(format = "latex", booktabs = TRUE, linesep = "", align = c("rlrrrr"),
        caption = "My formatted airquality table.", label = "airquality",
        col.names = c("Number", "Date", "Ozone (PPB)", "Radiation (Ly)",
            "Wind (MPH)", "Temp (C)")) %>%
    kable_styling(latex_options = c("hold_position")) %>%
    add_header_above(c("Observation" = 2, "Solar" = 2, "Weather" = 2))

We have done the following.

Left-pad the Solar.R and Wind fields with the unicode “nbsp” character, which will render as a space in our resulting code rather than be ignored, as a regular space would.
Use kable to generate a Latex table from our data frame. We have specified options such as cell alignments, the caption, and the label, which should be familiar to Latex users. We have also specified descriptive column names here.
The option hold_position specifies the option !h for how Latex should place the table.
The function add_header_above specifies a layer of headers above the column names. These respectively have text “Observation”, “Solar” and “Weather”, and are each two columns wide.

Printing the result yields the following Latex code.

R> print(out)
\begin{table}[!h]
\centering
\caption{\label{tab:airquality}My formatted airquality table.}
\centering
\begin{tabular}[t]{rlrrrr}
\toprule
\multicolumn{2}{c}{Observation} & \multicolumn{2}{c}{Solar} & \multicolumn{2}{c}{Weather} \\
\cmidrule(l{3pt}r{3pt}){1-2} \cmidrule(l{3pt}r{3pt}){3-4} \cmidrule(l{3pt}r{3pt}){5-6}
Number & Date & Ozone (PPB) & Radiation (Ly) & Wind (MPH) & Temp (C)\\
\midrule
1 & 1973-05-01 & 41 &   1.90e+02 &    7.4 & 19.44\\
2 & 1973-05-02 & 36 &   1.18e+02 &      8 & 22.22\\
3 & 1973-05-03 & 12 &   1.49e+02 &   12.6 & 23.33\\
4 & 1973-05-04 & 18 &   3.13e+02 &   11.5 & 16.67\\
5 & 1973-05-05 & NA &         NA &   14.3 & 13.33\\
6 & 1973-05-06 & 28 &         NA &   14.9 & 18.89\\
\bottomrule
\end{tabular}
\end{table}

We now have a Latex table environment - which is fairly human-readable - that can be copy/pasted into a Latex document, or even included programmatically using Sweave.

COMPoissonReg: Usage, the Normalizing Constant, and Other Computational Details

Tue, 15 Nov 2022 05:00:00 GMT

I have been maintaining the COMPoissonReg package, which facilitates use of the Conway-Maxwell Poisson (CMP) distribution. The normalizing constant of the CMP distribution is central to many computations in the package. Users have occasionally reported strange results that can be traced back to problems computing the normalizing constant. Early versions computed the constant by truncating the series to an finite sum with a fixed number of terms; however, this may yield inaccurate results. We addressed this to some degree in more recent versions in an ad hoc way. In the latest version (0.8.0) of the package, these computations have been revisited in a more principled way. Kimberly Sellers and I recently completed a report describing the updated method, which also demonstrates general use of the package. A vignette version of this report will be included in the package and updated along with any changes in the software.

Browsing summary files from the decennial census with R

Mon, 15 Aug 2022 04:00:00 GMT

If you have needed to use decennial census data without access to the internet, you may have encountered the raw data form of the summary files. These may be downloaded from the Census Bureau website for offline use; e.g., here. The format of the files is compact but can be difficult to navigate. James Livsey, Kyle Irimata, and I prepared a technical report and a package to assist R users with this via the tidyverse. These materials focus on the 2010 SF2 summary file, but support for others may be added in the future as needed.

knitr and Latex Documents

Mon, 27 Jun 2022 04:00:00 GMT

Rmarkdown seems to be the most standard way to embed R code and results into a document. Markdown has many benefits compared to Latex: it is much easier to get started with and the source code is closer to a plain readable text document. It is also possible to use Latex within Markdown when an occasional equation is needed. But sometimes you really want to work in Latex without going through Markdown.

I have been aware that Sweave allows embedded R for Latex documents, but it seemed archaic compared to more modern R tools for reproducible research, and I avoided using it. Recently, I was made aware that Overleaf supports authoring Sweave documents which require very little special markup and are almost like working in regular Latex. After some additional searching, I found that this is also possible without Overleaf. knitr has long been able to do this.

Examples of Sweave with knitr can be found online, but I thought it would be worthwhile to post several more. See the repo https://github.com/andrewraim/sweave-templates for an example in article format (pdf) and one in Beamer slide format (pdf).

Presentation for 2020 Joint Statistical Meetings

Mon, 27 Jul 2020 04:00:00 GMT

For this year’s virtual JSM, I am presenting in the session listed here. Here are links to download the poster PDF and voiceover MP4. Note that the files are hosted on Google Drive: if this is not accessible to you, email me for copies of the files.

R Workshop in Kerala

Fri, 05 Jan 2018 05:00:00 GMT

1 About the Workshop

George Ostrouchov, Nagaraj Neerchal and I are giving a two-day workshop High Performance Statistical Computing using R at the International Conference on Recent Advances in Statistical Methodology with Applications in Clinical and Official Statistics (ICSA 2018), Department of Statistics, St. Thomas College, Pala, Kerala, India. An abstract and brief outline for the workshop are given here.

The workshop builds up to parallel and distributed computing in R, but begins by introducing basics of working in R. Previous experience with R and Rstudio will be helpful, but much of the material should be accessible to new R users with experience in another technical computing language such as Matlab, Python, or Julia. Applications involving statistical and machine learning methods will be presented throughout the workshop. A bachelors degree in a discipline such as statistics, computer science, or mathematics - OR equivalent work experience - should be sufficient to understand the methodology.

Attendees are encouraged to bring their laptops. Many example codes will be provided, and there will be opportunities at some points during the workshop for users to follow along. If you will be bringing your laptop, the remainder of this page discusses how to prepare it for the workshop.

It is possible to manually install R, Rstudio, and all packages that will be used in the workshop. But this is quite impractical, especially for the material on parallel and distributed programming which varies somewhat based on the computing platform (Windows, Mac, Linux). Therefore, we highly suggest a second option…
Instead, we have prepared a Docker container, which is like a virtual machine. This will provide all attendees with the same programming environment which contains all the software needed for the workshop.

Instructions to obtain and run the Docker container are given below. For attendees who cannot install Docker, alternative instructions are provided further below.

Workshop contents such as slides, programs, and dataset will be provided at the start of the workshop.

2 Installing Docker

Follow instructions to install Docker based on your computing platform (Windows, Mac, or Linux). Look for the Community Edition, for which there is no cost to use. Also, we recommend the Stable release rather than the Edge release. Here are some brief notes specific to platform.

2.1 Windows

Note that Windows users who do not have Windows 10 Pro or Enterprise - which will probably be most of us - will need a version of Docker called Docker Toolbox. Once it is installed, open the Docker Quickstart Terminal to issue commands, which should appear as follows.

Logging into Rstudio Server

Take note of the IP address listed in the output; we will use it later when connecting to Rstudio.

I initially had some trouble installing Docker Toolbox on Windows 10 Home. I believe it was due to low disk space - I suggest having at least a 5-10 GB available before installing. For me, a solution was to remove Docker Toolbox, and the Oracle VirtualBox program installed with it, remove the .docker and .virtualbox subdirectories in C:\Users\Andrew, and rerun the Docker Toolbox installer. It may also be helpful to temporarily turn off Windows Firewall, or your antivirus program’s firewall if you have one, during the installation.

2.2 Linux

Linux users should follow the installation instructions specific to their Linux distribution (Ubuntu, CentOS, etc). Once it is installed, Docker can be controlled through the system terminal.

Note that Docker commands on Linux need administrator-level access, and must be prefixed with sudo. Because we anticipate mostly Windows users in attendance, the sudo has been left out of the commands shown on the remainder of this page.

2.3 Mac

Mac users should follow their set of installation instructions. As with Linux, once Docker is installed on a Mac, it can be controlled through the Terminal application. sudo is also needed for Mac when executing Docker commands.

2.4 A Quick Test of your Docker Setup

In your terminal, issue the following command.

$ docker run hello-world

Here, $ represents the prompt, which is not part of the command. If your Docker setup is working, you should get the following result.

$ docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
 https://cloud.docker.com/
For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

3 Downloading Workshop Config Files

Download the workshop’s Dockerfile and accompanying start.sh. Save the files to a folder where you will keep workshop materials. Let us call this directory /path/to/workshop.

In Windows, my web browser renamed Dockerfile to Dockerfile.txt when it was downloaded. If this happens, rename the file and remove the .txt extension.

After downloading start.sh, Linux and Mac users may need to modify its permissions before the Docker container will run. Ensure that the script can be executed via the following command.

$ chmod +x start.sh

4 Preparing and Running the Workshop Container

The following “build” command downloads and builds all of the prerequisites used in the container. It may take a while to run, especially on a slower network or a slower computer. For me it typically takes 20 or 30 minutes (roughly).

$ docker build -t rworkshop /path/to/workshop

The build command does not need to be run again unless the Dockerfile is changed, or unless your deployment changes. If the build was successful, the last few lines of output should be something like the following.

Successfully built f5055a2787e5
Successfully tagged rworkshop:latest

The container can now be run using the following “run” command (which is proceeded by defining two environment variables for our convenience).

$ export XSOCK=/tmp/.X11-unix/X0
$ export EXT=/path/to/workshop
$ docker run -v $XSOCK:$XSOCK -v $EXT:/home/rstudio/ext \
    -p 8787:8787 -i -t rworkshop

Note that: * The option -v $XSOCK:$XSOCK allows the container to display graphics. This is useful for packages such as Rcmdr. * The option -v $EXT:/home/rstudio/ext makes the directory $EXT on your computer available inside the container as /home/rstudio/ext. This will allow you to load files from your local machine into the container, and to save work from the container back to your local machine. * The option -p 8787:8787 exposes port 8787 from the container on your computer. This is needed to use Rstudio in the container. * The flags -i and -t are specified because the container is interactive. * The \ character is used to break up a long command into multiple lines.

5 Using R in the Container

Once the container is successfully started, you will encounter a Linux command prompt like the following.

rstudio@a8f9a900c791:~$

5.1 R on the Command Line

To start R on the command line, issue the R command.

rstudio@a8f9a900c791:~$ R
> cat("Hello world\n")
Hello world

5.2 Rstudio

To connect to Rstudio server, open a web browser on your laptop.

Mac and Linux users should navigate to the URL http://localhost:8787.
Windows users should recall the IP address they noted back in section 2.1. Suppose your assigned IP was 192.168.99.100 (which is the one I have in the screenshot); navigate to the URL http://192.168.99.100:8787. If you forgot the IP address, start up another Docker Terminal and run the following.

$ docker-machine ls

A login prompt should appear in your browser. Enter rstudio as both the username and the password.

Logging into Rstudio Server

Now you should be ready to use Rstudio in your web browser.

Rstudio Server in browser

5.3 R Commander

To launch the R Commander GUI, first start R via the command line or Rstudio. Then issue the following command.

> library(Rcmdr)

5.4 Running MPI Jobs

To demonstrate running an MPI job, let us use a simple Hello World example. Open a text editor on your laptop and save the following code to the file /path/to/workshop/hello.R

library(pbdMPI, quiet = TRUE)

msg <- sprintf("Hello world from process %d\n", comm.rank(), comm.size())
comm.cat("Say hello:\n", quiet = TRUE)
comm.cat(msg, all.rank = TRUE)

finalize()

Recall that this file will be accessible inside the container via the path /home/rstudio/ext/hello.R. Inside the container, you should be able to run the script in parallel via the mpirun command.

rstudio@a8f9a900c791:~$ mpirun -np 4 Rscript ~/ext/hello.R
Say hello:
COMM.RANK = 0
Hello world from process 0
COMM.RANK = 1
COMM.RANK = 2
Hello world from process 1
Hello world from process 2
COMM.RANK = 3
Hello world from process 3

Outputs from the container should be saved to /home/rstudio/ext. This will allow you to view images, PDFs, etc using the tools already installed on your laptop.

If you made it this far, congratulations - your laptop is ready!

6 More on Docker

For a list of Docker commands, see https://docs.docker.com/engine/reference/commandline/docker. Here we will mention a few specific ones. If you would like a video tutorial, there appear to be several good ones on YouTube such as this one.

List active Docker containers.

$ [araim@araim-inspiron docker]$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                    NAMES
f117630f47b8        rworkshop           "/etc/start.sh"     8 seconds ago       Up 7 seconds        0.0.0.0:8787->8787/tcp   hopeful_elion

Normally, our rworkshop container will stop running when you log out of its shell. In case something went wrong with the container listed above and we wanted to kill it, we could use the following command from another terminal window.

$ docker kill f117630f47b8
f117630f47b8

List Docker images that have been deployed onto your machine.

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
rworkshop           latest              d658ef03dc24        13 hours ago        1.98GB
rocker/rstudio      3.4.3               f206114fe549        3 weeks ago         1.08GB
hello-world         latest              1815c82652c0        6 months ago        1.84kB

The rworkshop and rocker/rstudio images are rather large, so you may need to remove them after the workshop. Here we will remove the hello-world image, for demonstration purposes.

$ docker image rm 1815c82652c0
Untagged: hello-world:latest
Untagged: hello-world@sha256:f3b3b28a45160805bb16542c9531888519430e9e6d6ffc09d72261b0d26ff74f
Deleted: sha256:1815c82652c03bfd8644afda26fb184f2ed891d921b20a0703b46768f9755c57
Deleted: sha256:45761469c965421a92a69cc50e92c01e0cfa94fe026cdd1233445ea00e96289a

7 If You Cannot Install Docker

Installing and running Docker requires administrator-level access to your computer. For example, if using a laptop issued by your employer, you may not have administrator access but may have R and Rstudio already installed. For the benefit of these attendees, we will list individual components that we have used in preparing the workshop.

The major disadvantage of installing the components individually is that you may not be able to run some of the parallel and distributed computing examples. For example, the mclapply function depends on forking, and therefore cannot be used directly in Windows. Also, our distributed computing examples rely on MPI, but we (the presenters) do not have experience installing and configuring MPI libraries in Windows.

Check the version numbers of components which you may have already installed - especially R, Rstudio, and any R packages. Some parts of the workshop may not be accessible with older versions.

7.1 Install R

Install R from CRAN. Windows users should navigate to “Download R for Windows” at the top of the page. We are using version 3.4.3.

7.2 Install Rstudio

Install Rstudio. Click Products => RStudio on the top menu, and look for a button labeled Download Rstudio Desktop. Select RStudio Desktop Open Source License among the available versions. We are using version 1.1.383.

7.3 Install the Rcpp package

See https://support.rstudio.com/hc/en-us/articles/200486498-Package-Development-Prerequisites for instructions to install the prerequisites (compilers, important libraries, etc). Once the prerequisites are installed, simply install Rcpp as you would any other package.

To test your Rcpp environment, try the following in RStudio. 1. Select File => New File => C++ File from the main menu. This will create a simple Rcpp program.
2. Save the program, say, as example.cpp.
3. Click the Source button, which is located in the upper-right corner of the panel with the example.cpp source code. This should produce the following output.

> timesTwo(42)
[1] 84

7.4 Install MPI

There are several MPI implementations available, such as OpenMPI and MVAPICH2. The instructions will vary based on your choice of implementation, as well as your hardware and operating system. You are on your own here; good luck! If your installation is successful, you should be able to run the following minimal example.

First, create a new file called hello_mpi.c with the following contents.

#include 
#include 

int main (int argc, char *argv[])
{
    int id, np;
    char processor_name[MPI_MAX_PROCESSOR_NAME];
    int processor_name_len;

    MPI_Init(&argc, &argv);

    MPI_Comm_size(MPI_COMM_WORLD, &np);
    MPI_Comm_rank(MPI_COMM_WORLD, &id);
    MPI_Get_processor_name(processor_name, &processor_name_len);

    printf("Hello world from process %03d out of %03d, processor name %s\n",
        id, np, processor_name);

    MPI_Finalize();
    return 0;
}

Now compile and run the program with MPI.

$ mpicc hello-mpi.c -o hello-mpi
$ mpirun -np 4 hello-mpi
Hello world from process 000 out of 004, processor name localhost
Hello world from process 001 out of 004, processor name localhost
Hello world from process 002 out of 004, processor name localhost
Hello world from process 003 out of 004, processor name localhost

Depending on your platform and choice of MPI implementation, your commands may be different. The string localhost in the output will be replaced with your machine’s hostname.

7.5 Install R Packages

Download the script install.R and run it in R to obtain packages which will be used in the workshop. Suppose we have placed it into the directory /path/to/workshop. Run the following command in R.

> source("/path/to/workshop/install.R")

To make sure your package versions are up to date, open Rstudio and click Tools => Check for Package Updates on the main menu.

Rcpp Workshop at UMBC

Mon, 07 Aug 2017 04:00:00 GMT

Note

Please note that the workshop date has been moved from Friday Sept 8, 2017 to Friday Sept 22, 2017. See below for details.

Iris Gauran and I are planning to give a half-day workshop on Rcpp at UMBC. Tentative details are given below. Part of the workshop will be a “quick start”, where we will demonstrate accessing C++ code from R via toy examples. Attendees are encouraged to prepare their laptops for Rcpp programming, and bring them to the workshop to follow along with this portion.

We will focus on Rcpp programming in RStudio, which provides a common interface across all major platforms (Windows, Mac, and Linux), and helps to automate some procedural tasks.

Install R (http://www.r-project.org). The current version as of this writing is 3.4.1 “Single Candle”.
Install RStudio Desktop (http://www.rstudio.com). The current version is 1.0.153.
Install the Rcpp, RcppArmadillo, and RcppGSL packages.
Some additional libraries may be needed to compile C++ programs on your computer. These differ depending on your operating system. See http://support.rstudio.com/hc/en-us/articles/200486498-Package-Development-Prerequisites.

This may be a good time to upgrade if you have older versions of R, RStudio, or any of the listed packages.

To test your Rcpp environment, try the following in RStudio.

Select File => New File => C++ File from the main menu. This will create a simple Rcpp program.
Save the program, say, as example.cpp.
Click the Source button, which is located in the upper-right corner of the panel with the example.cpp source code. This should produce the output

R> timesTwo(42)
[1] 84

Location: University of Maryland, Baltimore County.
Room: MP 401 (Math/Stat Seminar Room).
Time: Friday Sept 22, 1 - 4:30pm.
Cost: This workshop is free for the UMBC community.
Prerequisites: intermediate knowledge in a high level language like R, Python, or Matlab. Familiarity with R and RStudio is highly recommended.
Snacks: Light refreshments (coffee and bagels) will be provided.

Abstract
R is the preferred computing environment for many statisticians, and is used both in research and applied problems. R has achieved tremendous popularity because it is free, open source, and available on all modern platforms (Windows, Mac, and Linux). The R programming language is simple and intuitive, and well-suited for fast prototyping of complicated algorithms. However, R users can often find the performance of their programs to be lacking.

This workshop will demonstrate Rcpp, an extension of R that facilitates interoperability between R and C++ (Eddelbuettel, 2013). With Rcpp, computationally intensive parts of your program can be written in C++ and seamlessly called from R, potentially giving dramatic performance improvements within the familiar R user environment. We will provide a quick start for new Rcpp users through simple examples, and also examine several larger-scale statistical applications. C++ programming is usually more burdensome than R programming, so we will discuss tradeoffs between optimizing R code and migrating to C++, and ways to minimize suffering while making the transition.

References

Dirk Eddelbuettel. Seamless R and C++ Integration with Rcpp. Springer, 2013.
Also see http://www.rcpp.org.

Web Page Migration

Wed, 15 Mar 2017 04:00:00 GMT

I am now using GitHub Pages to host my website, along with the static site generator Jekyll. My previous website was hosted at UMBC.

Andrew M. Raim

Vim as a Simple (R) IDE without Plugins II

1 Introduction

2 Are we in an R Package?

3 Menu Operations and R Functions

4 Menu in Vim

5 Keybinding

Customized Bash Prompt

1 Introduction

2 Environment Variables for the Prompt

3 Placeholders and Aesthetic Prompt Symbol

4 Custom Information with Function Calls

5 Colors

6 Informative Colors

Vim as a Simple IDE without Plugins

1 Introduction

2 Embedded Terminal

3 Send Text to the Terminal

4 Spawn a Terminal with a Console Program

5 Keybindings

vws: Vertical Weighted Strips in R using C++

References

Self-Tuned VWS within Gibbs and Small Area Estimation

References

Numerical Maximum Likelihood with the stats4 Package

1 Introduction

2 Poisson Regression Example

3 Using the mle Function in stats4

4 Fixed Parameters

5 Transforming Parameters

6 References

fntl: Numerical Tools for Rcpp and Lambda Functions

References

Rejection Sampling with Vertical Weighted Strips

References

Sending Mail from R

Comparison of Usability Preferences with Bivariate Ordinal Regression

References

Interactive Map with Shiny and Leaflet

Worker Tool for Computational Studies

Generating Latex Tables in R

Introduction

Objective

Preparing the Data Frame

Generating Latex

COMPoissonReg: Usage, the Normalizing Constant, and Other Computational Details

Browsing summary files from the decennial census with R

knitr and Latex Documents

Presentation for 2020 Joint Statistical Meetings

R Workshop in Kerala

1 About the Workshop

2 Installing Docker

2.1 Windows

2.2 Linux

2.3 Mac

2.4 A Quick Test of your Docker Setup

3 Downloading Workshop Config Files

4 Preparing and Running the Workshop Container

5 Using R in the Container

5.1 R on the Command Line

5.2 Rstudio

5.3 R Commander

5.4 Running MPI Jobs

6 More on Docker

7 If You Cannot Install Docker

7.1 Install R

7.2 Install Rstudio

7.3 Install the Rcpp package

7.4 Install MPI

7.5 Install R Packages

Rcpp Workshop at UMBC

Web Page Migration

3 Using the `mle` Function in `stats4`