Importantly: Understand that hash-2.0.0, breaks backward compatibility; code written with previous versions of the hash package are not guaranteed to work with this or future versions. This is due to changes made in order to achieve much higher performance.  Assignments and look-ups are achieved more quickly through direct inheritance of environments, stripping of non-essential customizations and reliance on core and primitive functions.

Here is a summary of major changes:

• Coercion of keys to valid R names ( i.e. non-blank character values) is not the responsibility of the user.  The four accessor functions: [, [[, $, values, no longer do this automatically.  An error results if a proper R name is not provided.

• The default for missing keys has changed from NA to NULL. This is to match the behavior lists in trying to access non-existing objects in R.  ( For a more complete, discussion, see my previous blog post discussing the differences between NA and NULL. )
  
  • Custom behavior for accessing non-existent keys has been removed.  Access to non-existing keys will always yield NULL.  Consistency is often better than customization.

ChangeLog and TODO track many technical details; here I will discuss only the more important changes:

Performance

Included in this version is a demo script that runs benchmarks (demo(hash-benchmarks).  One of the questions that has been repeatedly posed, often in the context of look-up, is:  how does this compare to native R named lists and vectors? In other words, how much quicker is accessing a value on a hash / environment as opposed to a list (or vector)?  This is a difficult questions, and generally depends on the size of the hash or list.  My rule of thumb is that it is quicker to look-up elements on lists and vectors less than about 500 elements.  After ~500 elements, hashes and environments greatly outperform lists.  The difference increases relative to the size of the object.  However, look-ups for all these objects are very fast if objects are small  ( >120,000 / sec ).  So unless you are doing many serial look-ups, hashes are likely the better option.

I have written previously about hashes in R [1] [2], and will continue to  discuss the evolution of R hashes on this blog.  Additionally I will be speaking on this and related work at useR!2010 (July 20-23.)

It is common for programming languages to have a NULL value.  What often leads to confusion is the fact NULL can have two distinct meanings.  In the first, NULL is used to represent missing or undefined values.  This is well appreciated in SQL. In the second case, NULL is the logical representation a statement that is neither TRUE nor FALSE.  This indeterminacy is the basis for ternary logic.  While these meanings are distinct, they are very often related.  When missing values (the first meaning) are evaluated, the desired result is often an ambiguous result (the second).  That is, the former implies the latter.  In programming, the distinction is often unnecessary and glossed over and the concepts become confounded.

The R language has two closely related NULL-like values, NA and NULL.  Both are fully support in the language by core functions (e.g, is.na, is.null, as.null, etc.). And, while NA is used exclusively in the logical sense, both are used to represent missing or undefined values.  This has lead to much confusion.  Here’s what the R documentation has to say:

NULL represents the null object in R: it is a reserved word.
NULL is often returned by expressions and functions whose values are
undefined.

NA is a logical constant of length 1 which contains a missing
value indicator. NA can be freely coerced to any other vector
type except raw. There are also constants NA_integer_,
NA_real_, NA_complex_ and NA_character_ of the other atomic
vector types which support missing values: all of these are
reserved words in the R language.

There is a lot of subtlety in the treatment of these values.  A good way to understand the distinction between  NA and NULL is through some examples:

 > NA
 [1] NA

 > class(NA)
 [1]   "logical"

 > NA > 1
 [1] NA

 > NULL
 NULL 

 > class(NULL)
 [1]   "NULL"

 > NULL > 1
 logical(0)

The important distinction is that NA is a ‘logical’ value that when evaluated in an expression, yields NA.  This is the expected behavior of a value that handles logical indeterminacy.   NULL is its own thing and does not yield any response when evaluated in an expression, which is not how we would want or expect NA to work.

To delve deeper into the behavior we must look at how R’s basic data structures, vectors (including matrices and arrays) and lists (including data.frames) behave.  Vectors and lists are similar structures, both allow for multiple values with similar accessors.  There are subtle differences in the treatment of NA and NULL.  Let’s take a look at how they compare:

 > v <-  c( 1, NA, NULL)
 > v 
 [1]  1 NA

 

 > list(1, NA, NULL)
 [[1]]
 [1] 1

 [[2]]
 [1] NA

 [[3]]
 NULL

What happened?  NULL is not allowed in a vector.  When you attempt to set it as a value in a vector, it is it is quietly ignored.  This is because NULL is an object and type of its own.  NULL does not have various types such as NULL_integer_.  There is just NULL. By contrast, NA has NA_integer, etc. and happily coexists with any of the basic vector types vector.  So for any vector (matrix or array), NA represents a missing value.  NULL does not.

Now, let’s look at the lists example. This is interesting! Unlike the vector, the list can hold objects and values other than the basic types.  This includes the NULL value/object.  Perhaps a little inconsistent and not what we would expect.  But from here, things get a little quirky, let’s try value assignment:

  > v[[1]] <- NULL
   Error in v[[1]] <- NULL :
    more elements supplied than there are to replace

 

 > li <- list( 1, 2, 3 )
 > li[[1]] <- NULL
 > li
 [[1]]
 [1] 2

 [[2]]
 [1] 3

Sure enough NULL cannot be assigned to a vector.  So for all purposes, NA with respect to the basic vector behaves like NULL in other languages.  NULL is almost never what you want.  On the list side, however, we see an idiom of NULL. Assigning NULL to list items, removes them.  This behavior is a bit unexpected, but it is the idiom.

There is one final idiom to know about NULL and lists. Namely, that trying to access a list element by a non-existing name yields a NULL value.

> li$aa
NULL
> li[['aa']]
NULL

 

( Note: the same is true for trying to access non-existing objects on an environment )

R does not have a consistent or intuitive way of dealing with missing and logically ambiguous values, i.e. addressing the two meanings from the beginning of this post.  For vectors and basic variables, R mimics other languages and uses NA.  For lists however, the syntax is more idiomatic.  It is this latter case that presents difficulty.  R has other quirks too.  But all languages have quirks, and given R’s strength for statistical analysis, I have found no better tool for this.

Earlier today, hash-1.99.x was released to CRAN. This is a stable release and adds some more functions to an already full-featured hash implementation. This version fixes some bugs, adds some features, improves performance and stability. You can read about the hash package in my previous blog post, The hash package: hashes come to R. All changes were responsible from users who wrote in and contributed, thoughts, ideas and use cases. Keep the good ideas coming.  Two of the major changes are summarized below.

Matthias Buch-Kromann of the Copenhagen Business School recommended the ability to access multiple keys from a single call and even access the same key multiple times. This was previously allowed using the [[ method, but was deprecated. By convention, the [[ method returns only one value. ( You can read about the conventions of this and other R accessors in my previous blog post, R Accessors Explained. ) This behavior has returned to hash-1.99.x the use of the values method and the and optional keys argument:


h <- hash( c('a','b','c'), 1:3 )
values(h)
values(h, keys=c('a','b','c','a','b','c' ) )


Matthias suggested calling the method mget, but there was some disparity with the mget function in base. The generic function that I needed just wouldn't play nice with base::mget.

Another change in the behavior was prompted by Mohammad Fahim of the Department of Computer Engineering and Computer Science at the University of Louisville. He wrote me to ask if there is a way to suppress warnings when trying to access non-existent keys. When accessing hashes hundreds of thousands of times, it becomes a drag to continually see:

key: xxxx not found in the hash : hash_table_name

I have refactored the behavior to be more R-like by following na.action-type conventions. Now the default behavior is to return NA when trying to access non-existing keys.


> library(hash)
>h <- hash( c('a','b','c'), 1:3 )
> h h[ letters[1:5] ]
containing 6 key-value pair(s).
a : 1
b : 2
c : 3
d : NA
e : NA


The behavior is also controllable by na.action.hash option. The functions are provided for most use cases:

• na.default.hash (default) returns NA silently ,
• na.fail.hash (old default) errors on non-existing keys
• na.warn.hash returns NA but issues a warning.

Behaviors can be set by setting the na.hash.action option. For example, to get the default behavior:


> options( na.hash.action = na.fail.hash )
> h$d
Error: key, d, not found in hash.
> h[[ 'd' ]]
Error: key, d, not found in hash.


And , for the [ and [[ methods, this behavior can be declared at access time:


> h[[ 'd', na.action=na.warn.hash ]]
Warning: key, d, not found in hash.
d
NA
> h[[ 'd', na.action=na.fail.hash ]]
Error: key, d, not found in hash.
> h[[ 'd', na.action=na.default.hash ]]
d
NA


If you don’t like these hash-key-miss behaviors, you are free to write your own. Functions should minimally accept arguments of the hash and the key.

Thanks to both Matthias and Mohammed for your feedback.

New features are on their way. Notably, the ability to use any object as keys and to preserve the order of the hash. These are sometimes called Indexed Hashes. Look for that in the hash-2.00.x release. If you would like to see features added contact me at cbrown -at- opendatagroup.com

References:

• The hash package: hashes come to R
• R Accessors Explained

R Accessors

For more than ten years, I have been teaching R both formally and informally. One thing that I find often trips up students is the use of R’s accessors and mutators. ( For those readers not from a formal computer science background, an accessor is a method for accessing data in an object usually an attribute of that object.) A simple example is taking a subset of a vector:


letters[1:3]
[1] "a" "b" "c"


As you can see, the result is a character vector containing the first three letters of letters vector.

Good programming languages have a standard pattern for accessor and mutators. For R, there are three: [, [[, and $. This confuses beginners coming from other programming languages. Java and Python have one: '.'. Why does R need three?

The reason derives from R's data centric view of the world. R natively provides vectors, lists, data frames, matrices, etc. In truth, one can get by using only [ to extract information from these structures, but the others are handy in certain scenarios. So much so that after a while, they feel indispensible. I will explain each and hopefully by the end of this article you will understand why each exists, what to remember and, more importantly, when to each should be used.

Subset with [

When you want a subset of an object use [. Remember that when you take a subset of an the object you get the same type of thing. Thus, the subset of a vector will be a vector, the subset of a list will be a list and the subset of a data.frame will be a data.frame.

There is one inconsistency, however. The default in R is to reduce the results to the lowest dimension, so if your subset contains only result, you will only get that one item which may be something of a different type. Thus, taking a subset of the iris data frame with only one column


class( iris[ , "Petal.Length" ] )
[1] numeric


returns a numeric vector and not a data frame. You can override this behavior with the little publicized drop parameter, which indicates not to reduce the result. Taking the subset of iris with drop = FALSE


iris[ , "Petal.Length", drop=FALSE ]


is a proper data frame.

Things to Remember:

• Most often, a subset is the same type as the original object.
• Both indices and names can be used to extract the subset. ( In order to use names, object must have a name type attribute such as names, rownames, colnames, etc. )
• You can use negative integers to indicate exclusion.
• Unquoted variables are interpolated within the brackets.

Extract one item with [[

The double square brackets are used to extract one element from potentially many. For vectors yield vectors with a single value; data frames give a column vector; for list, one element:


letters[[3]]
iris[["Petal.Length"]]


The mnemonic device, here is that the double square bracket look as if you are asking for something deep within a container. You are not taking a slice but reaching to get at the one thing at the core.

Three important things to remember:

• You can return only one item.
• The result is not (necessarily) the same type of object as the container.
• The dimension will be the dimension of the one item which is not necessarily 1.
• And, as before:
  • Names or indices can both be used.
  • Variables are interpolated.

Interact with $

Interestingly enough, the accessor that provides the least unique utility is also probably used the most often used. $ is a special case of [[ in which you access a single item by actual name. The following are equivalent:


iris$Petal.Length
iris[["Petal.Length"]]


The appeal of this accessor is nothing more than brevity. One character, $, replaces six, [[""]]. This accessor is handiest when doing interactive programming but should be discouraged for more production oriented code because of its limitations, namely the inability to interpolate the names or use integer indices.

Things to Remember:

• You cannot use integer indices
• The name will not be interpolated.
• Returns only one item.
• If the name contains special characters, the name must be enclosed in backticks: ``

That is really all there is to it. [ - for subsets, [[ - for extracting items, and $ - for extracting by name.

As much as I like R, on occasion I will find myself lost in some dark corner. Sometimes, I find light. Sometimes I am gnashing teeth and wringing hands. Frustrated. In a recent foray, I found myself trying to do something that I thought exceedingly trivial: expanding character and factor vectors to dummy variables. There must be some function, but what? Trying ?dummy didn’t turn up anything. Surely some else must have encountered this and provided a package. I went to the Internet and sure enough the R-wiki was here to save me. And looking even harder, I found some who had treaded before me on the R-Help archives. It turns out, it’s simple. Expanding a variable as a dummy variable can be done like so:


x <- c(2, 2, 5, 3, 6, 5, NA)
xf <- factor(x, levels = 2:6)
model.matrix( ~ xf - 1)


Two problems. The first problem is that without an external source (Google), I would have never stumbled upon what I wanted. ( Thanks Google!) I understand it now, but for what I wanted to do, I would never have thought, “oh, model.matrix.”

The second problem is the arcane syntax, wtf <- ~ xf - 1. I get it now, but it took me some time to figure out what was going on. I get it, but why not just dummy(var)? This is what I want to do.

The solution on the wiki wasn’t quite what I was looking for. For instance, you can’t say:

model.matrix( ~ xf1 + xf2 + xf3- 1)

It turns out, you can only expand one variable at a time. Well, this is not good. I know that you could solve this with some sapply’s and some tests, but next time I might forgot about how to do it. So with a couple of spare hours, I decided that the next guy, wouldn’t have to think about it. He could just use my dummies package.

Like the R-wiki solution, the dummies package provides a nice interface for encoding a single variable. You can pass a variable -or- a variable name with a data frame. These are equivalent:


dummy( df$var )
dummy( "var", df )


Moreover, you can choose the style of the dummy names, whether to include unused factor level, to have verbose output, etc.

But more than the R-wiki solution, dummy.data.frame offers to something similar to data.frames. You can specify which columns to expand by name or class and whether to return non-expanded columns.

The package dummies-1.04 is available in CRAN. Comments and questions are always appreciated.