The fastest way forward is to use a public cloud, external experts, and to do some quick experiments and prototyping. At this point, for many companies, there is a problem. It is quite common these days for companies to have policies that prohibit placing proprietary data, or data that contains information that can identify customers, on public clouds. Providing access to this data to third parties is also usually quite difficult.

One practical approach is to replace actual data with simulated data, and, instead of using public clouds, to use instead private clouds operated by third parties. This requires using data simulators that produce realistic data. For example, large data is rarely normally distributed, but more often follows power laws or similar types of distributions.

As a reminder, a private cloud is a cloud that is used exclusively by a single organization. It may be managed by the organization or by a third party; and, it may exist on premise (an in-house private cloud) or off premise (a third-party private cloud). In contrast, in a public cloud, the cloud infrastructure is made available to the general public, or a large group, and is owned by an organization selling cloud services (a cloud service provider). In this post, we assume that private third party clouds are also single tenant clouds; that is, only one client’s data is on the cloud at a time and the cloud is sanitized between use by different clients.

In more detail, one approach for moving your analytics to clouds is:

• use simulated data following realistic simulations, instead of actual data;
• supplement in-house expertise with third party experts who specialize in analytics and cloud computing;
• use third party private clouds instead of public clouds to decrease risk or perceived risk;
• experiment with different analytic approaches and different analytic infrastructures;
• agree on APIs up front and transfer technology by transferring code that uses these APIs.

We have found this approach works well. We would be interested in hearing your experiences.

Full disclosure: Open data operates private clouds, has developed software that provides simulated data for a variety of industries, including financial services, and provides consulting services using simulated data on private clouds so that companies can rapidly explore the use of cloud computing to develop innovative cloud computing applications, especially analytic applications.

Importantly: Understand that hash-2.0.0, breaks backward compatibility; code written with previous versions of the hash package are not guaranteed to work with this or future versions. This is due to changes made in order to achieve much higher performance.  Assignments and look-ups are achieved more quickly through direct inheritance of environments, stripping of non-essential customizations and reliance on core and primitive functions.

Here is a summary of major changes:

• Coercion of keys to valid R names ( i.e. non-blank character values) is not the responsibility of the user.  The four accessor functions: [, [[, $, values, no longer do this automatically.  An error results if a proper R name is not provided.

• The default for missing keys has changed from NA to NULL. This is to match the behavior lists in trying to access non-existing objects in R.  ( For a more complete, discussion, see my previous blog post discussing the differences between NA and NULL. )
  
  • Custom behavior for accessing non-existent keys has been removed.  Access to non-existing keys will always yield NULL.  Consistency is often better than customization.

ChangeLog and TODO track many technical details; here I will discuss only the more important changes:

Performance

Included in this version is a demo script that runs benchmarks (demo(hash-benchmarks).  One of the questions that has been repeatedly posed, often in the context of look-up, is:  how does this compare to native R named lists and vectors? In other words, how much quicker is accessing a value on a hash / environment as opposed to a list (or vector)?  This is a difficult questions, and generally depends on the size of the hash or list.  My rule of thumb is that it is quicker to look-up elements on lists and vectors less than about 500 elements.  After ~500 elements, hashes and environments greatly outperform lists.  The difference increases relative to the size of the object.  However, look-ups for all these objects are very fast if objects are small  ( >120,000 / sec ).  So unless you are doing many serial look-ups, hashes are likely the better option.

I have written previously about hashes in R [1] [2], and will continue to  discuss the evolution of R hashes on this blog.  Additionally I will be speaking on this and related work at useR!2010 (July 20-23.)

It is common for programming languages to have a NULL value.  What often leads to confusion is the fact NULL can have two distinct meanings.  In the first, NULL is used to represent missing or undefined values.  This is well appreciated in SQL. In the second case, NULL is the logical representation a statement that is neither TRUE nor FALSE.  This indeterminacy is the basis for ternary logic.  While these meanings are distinct, they are very often related.  When missing values (the first meaning) are evaluated, the desired result is often an ambiguous result (the second).  That is, the former implies the latter.  In programming, the distinction is often unnecessary and glossed over and the concepts become confounded.

The R language has two closely related NULL-like values, NA and NULL.  Both are fully support in the language by core functions (e.g, is.na, is.null, as.null, etc.). And, while NA is used exclusively in the logical sense, both are used to represent missing or undefined values.  This has lead to much confusion.  Here’s what the R documentation has to say:

NULL represents the null object in R: it is a reserved word.
NULL is often returned by expressions and functions whose values are
undefined.

NA is a logical constant of length 1 which contains a missing
value indicator. NA can be freely coerced to any other vector
type except raw. There are also constants NA_integer_,
NA_real_, NA_complex_ and NA_character_ of the other atomic
vector types which support missing values: all of these are
reserved words in the R language.

There is a lot of subtlety in the treatment of these values.  A good way to understand the distinction between  NA and NULL is through some examples:

 > NA
 [1] NA

 > class(NA)
 [1]   "logical"

 > NA > 1
 [1] NA

 > NULL
 NULL 

 > class(NULL)
 [1]   "NULL"

 > NULL > 1
 logical(0)

The important distinction is that NA is a ‘logical’ value that when evaluated in an expression, yields NA.  This is the expected behavior of a value that handles logical indeterminacy.   NULL is its own thing and does not yield any response when evaluated in an expression, which is not how we would want or expect NA to work.

To delve deeper into the behavior we must look at how R’s basic data structures, vectors (including matrices and arrays) and lists (including data.frames) behave.  Vectors and lists are similar structures, both allow for multiple values with similar accessors.  There are subtle differences in the treatment of NA and NULL.  Let’s take a look at how they compare:

 > v <-  c( 1, NA, NULL)
 > v 
 [1]  1 NA

 

 > list(1, NA, NULL)
 [[1]]
 [1] 1

 [[2]]
 [1] NA

 [[3]]
 NULL

What happened?  NULL is not allowed in a vector.  When you attempt to set it as a value in a vector, it is it is quietly ignored.  This is because NULL is an object and type of its own.  NULL does not have various types such as NULL_integer_.  There is just NULL. By contrast, NA has NA_integer, etc. and happily coexists with any of the basic vector types vector.  So for any vector (matrix or array), NA represents a missing value.  NULL does not.

Now, let’s look at the lists example. This is interesting! Unlike the vector, the list can hold objects and values other than the basic types.  This includes the NULL value/object.  Perhaps a little inconsistent and not what we would expect.  But from here, things get a little quirky, let’s try value assignment:

  > v[[1]] <- NULL
   Error in v[[1]] <- NULL :
    more elements supplied than there are to replace

 

 > li <- list( 1, 2, 3 )
 > li[[1]] <- NULL
 > li
 [[1]]
 [1] 2

 [[2]]
 [1] 3

Sure enough NULL cannot be assigned to a vector.  So for all purposes, NA with respect to the basic vector behaves like NULL in other languages.  NULL is almost never what you want.  On the list side, however, we see an idiom of NULL. Assigning NULL to list items, removes them.  This behavior is a bit unexpected, but it is the idiom.

There is one final idiom to know about NULL and lists. Namely, that trying to access a list element by a non-existing name yields a NULL value.

> li$aa
NULL
> li[['aa']]
NULL

 

( Note: the same is true for trying to access non-existing objects on an environment )

R does not have a consistent or intuitive way of dealing with missing and logically ambiguous values, i.e. addressing the two meanings from the beginning of this post.  For vectors and basic variables, R mimics other languages and uses NA.  For lists however, the syntax is more idiomatic.  It is this latter case that presents difficulty.  R has other quirks too.  But all languages have quirks, and given R’s strength for statistical analysis, I have found no better tool for this.

Despite an emphasis toward gaming and video — the primary market and impetus for technology — the high performance computing enthusiast should take away a better understanding the technology and what hardware and software toys are on the horizon.

Earlier today, hash-1.99.x was released to CRAN. This is a stable release and adds some more functions to an already full-featured hash implementation. This version fixes some bugs, adds some features, improves performance and stability. You can read about the hash package in my previous blog post, The hash package: hashes come to R. All changes were responsible from users who wrote in and contributed, thoughts, ideas and use cases. Keep the good ideas coming.  Two of the major changes are summarized below.

Matthias Buch-Kromann of the Copenhagen Business School recommended the ability to access multiple keys from a single call and even access the same key multiple times. This was previously allowed using the [[ method, but was deprecated. By convention, the [[ method returns only one value. ( You can read about the conventions of this and other R accessors in my previous blog post, R Accessors Explained. ) This behavior has returned to hash-1.99.x the use of the values method and the and optional keys argument:


h <- hash( c('a','b','c'), 1:3 )
values(h)
values(h, keys=c('a','b','c','a','b','c' ) )


Matthias suggested calling the method mget, but there was some disparity with the mget function in base. The generic function that I needed just wouldn't play nice with base::mget.

Another change in the behavior was prompted by Mohammad Fahim of the Department of Computer Engineering and Computer Science at the University of Louisville. He wrote me to ask if there is a way to suppress warnings when trying to access non-existent keys. When accessing hashes hundreds of thousands of times, it becomes a drag to continually see:

key: xxxx not found in the hash : hash_table_name

I have refactored the behavior to be more R-like by following na.action-type conventions. Now the default behavior is to return NA when trying to access non-existing keys.


> library(hash)
>h <- hash( c('a','b','c'), 1:3 )
> h h[ letters[1:5] ]
containing 6 key-value pair(s).
a : 1
b : 2
c : 3
d : NA
e : NA


The behavior is also controllable by na.action.hash option. The functions are provided for most use cases:

• na.default.hash (default) returns NA silently ,
• na.fail.hash (old default) errors on non-existing keys
• na.warn.hash returns NA but issues a warning.

Behaviors can be set by setting the na.hash.action option. For example, to get the default behavior:


> options( na.hash.action = na.fail.hash )
> h$d
Error: key, d, not found in hash.
> h[[ 'd' ]]
Error: key, d, not found in hash.


And , for the [ and [[ methods, this behavior can be declared at access time:


> h[[ 'd', na.action=na.warn.hash ]]
Warning: key, d, not found in hash.
d
NA
> h[[ 'd', na.action=na.fail.hash ]]
Error: key, d, not found in hash.
> h[[ 'd', na.action=na.default.hash ]]
d
NA


If you don’t like these hash-key-miss behaviors, you are free to write your own. Functions should minimally accept arguments of the hash and the key.

Thanks to both Matthias and Mohammed for your feedback.

New features are on their way. Notably, the ability to use any object as keys and to preserve the order of the hash. These are sometimes called Indexed Hashes. Look for that in the hash-2.00.x release. If you would like to see features added contact me at cbrown -at- opendatagroup.com

References:

• The hash package: hashes come to R
• R Accessors Explained