Persistence Library

Contents

The components provided are:

Dependencies

The persistence library has a soft dependency on the containers and portability libraries. This means that it only requires these two libraries in order to make their contents persistent. If there is no need for that functionality, then it is possible to compile the persistence library to exclude them.

To compile the persistence library in stand-alone mode, you need to add the following pre-processor switch to the compiler setup. If you are building using the provided makefile system, then this can be added to the file persistence/Makefile.

CPPFLAGS += -DNO_STLPLUS_CONTAINERS
CPPFLAGS += -DNO_STLPLUS_INF

In fact, these lines are already present in the makefile, but commented out.

You will also need to provide these switches in any program that uses the persistence library, since it controls what's visible in the headers that are included into your code.

You will also need to comment out the dependencies:

#LIBRARIES += ../portability
#LIBRARIES += ../containers

Note the # comment at the start of the line to comment out these dependencies.

Introduction

Persistence is the ability to dump a data structure to a "serialised" form and then restore it later either in the same run of the program, a later run of a program, or even in a different program. It is an easy way to save a program's state or to communicate information in a structural form between programs.

Persistence is not limited to disk dumps, since the same idea can be used to transfer information from one program to another down a pipe or even an Internet connection. In effect you can use persistence to communicate a data structure of any complexity between two programs even if they are running on different computers under different operating systems.

At a more basic level, persistence does away with the need to design file formats. Instead, just design a data structure and then make that data structure persistent. The file format is designed for you by the persistence subsystem.

The persistent format chosen for the STLplus is a binary format so is extremely efficient both in data size and in CPU time required. For example, text formats such as XML tend to be dominated by the processing required to convert integer values between their machine form (2's-complement binary) and the text form (sign-magnitude decimal). This problem doesn't occur with the persistence format which dumps and restores in the native binary form.

The data format is platform-independent so that there are no problems in transferring data between different computers, regardless of what operating system is running, what compiler was used, what processor is being used, what convention is used to store numeric types (i.e. little-endian versus big-endian) or even what size of word is being used (e.g. 32-bit versus 64-bit).

So, for example, a program compiled with Gnu's gcc on Gnu/Linux running on a 32-bit big-endian processor can transfer data either as a file or over the internet to a program compiled with Microsoft's Visual Studio, running on a 64-bit little-endian processor.

The approach taken with the persistence subsystem is to provide a toolkit which makes it easy if not trivial to make a data structure persistent. However, it is not totally automatic - C++ is too flexible a language to be able to take any data structure and just dump it. This is why the approach has been to provide a toolkit out of which persistence routines can be written.

The toolkit provides a set of functions for dumping and restoring a wide range of types. All the basic C types are made persistent, as are C++ types like std::string and std::complex. However, the real power of the persistence functions is that template functions are provided for making all of the STL and STLplus container classes persistent.

Overview

The idea is that a container is made persistent by dumping its contents using a dump routine for the contained data type. For example, a std::vector of std::string is dumped by dumping vector-specific information and then repeatedly calling the dump routine for string. The restore function restores the vector and then repeatedly calls the restore function for string to restore the vector's contents.

The same concept is applied to all the container classes. Therefore, to make a container persistent, all you have to do is supply dump and restore functions for the contained type. If the contained type is a basic C or C++ type, then these functions are already provided and the data structure is already persistent.

The core of the persistence system are the persistence contexts, which act like I/O objects. Indeed, the persistence contexts take an IOStream object to perform the I/O part of the dump/restore operation.

The sequence of events is:

  1. Create an IOStream device for the source/target of the dump/restore
  2. Create a dump_context/restore_context for the dump/restore
  3. Use the Persistence Functions to perform the dump/restore

The persistence functions are used to build up layers of recursive function calls which parallel the data structure being made persistent. So for example, a list of vectors of strings will use the persistence functions for list, calling the functions for vector, which in turn calls the functions for string. Two sets of functions are written, one for dumping and one for restoring.

Persistence functions for simple types are non-templates and take two arguments. For example, here are the dump/restore functions for std::string:

void stlplus::dump_string(stlplus::dump_context&, const std::string& data)
  throw(stlplus::persistent_dump_failed);

void stlplus::restore_string(stlplus::restore_context&, std::string& data) 
  throw(stlplus::persistent_restore_failed);

The persistence functions for container templates take an extra parameter - the name of a function to dump/restore the element type of the container. For example, here are the dump/restore functions for vector:

template<typename T, typename D>
void stlplus::dump_vector(stlplus::dump_context&, const std::vector<T>& data, D dump_fn)
  throw(stlplus::persistent_dump_failed);

template<typename T, typename R>
void stlplus::restore_vector(stlplus::restore_context&, std::vector<T>& data, R restore_fn)
  throw(stlplus::persistent_restore_failed);

Note how there is a final parameter dump_fn/restore_fn. In use, these must be provided with the name of the function to dump/restore the element type.

For example, to dump a vector of string, combine the above two functions:

stlplus::dump_vector(context, data, stlplus::dump_string);

To create a list of vectors of strings, simply add the next layer. The list functions are:

template<typename T, typename D>
void stlplus::dump_list(stlplus::dump_context&, const std::list<T>& data, D dump_fn)
  throw(stlplus::persistent_dump_failed);

template<typename T, typename R>
void stlplus::restore_list(stlplus::restore_context&, std::list<T>& data, R restore_fn)
  throw(stlplus::persistent_restore_failed);

However, the last parameter - the dump_fn and restore_fn - must be only two-parameter functions. This means you need to wrap the existing dump_vector call into a specialisation function:

void dump_string_vector(stlplus::dump_context&, const std::vector<std::string>& data) throw(stlplus::persistent_dump_failed)
{
  stlplus::dump_vector(context, data, stlplus::dump_string);
}

void restore_string_vector(stlplus::restore_context&, std::vector<std::string>& data) throw(stlplus::persistent_restore_failed)
{
  stlplus::restore_vector(context, data, stlplus::restore_string);
}

It is now possible to implement the list routines. For example, to dump a list of vectors of string, combine the above two functions:

stlplus::dump_list(context, data, dump_string_vector);

And of course, this can then be wrapped into a two-parameter function to be used in yet another level of data structure.

void dump_string_vector_list(stlplus::dump_context&, const std::list<std::vector<std::string> >& data) throw(stlplus::persistent_dump_failed)
{
  stlplus::dump_list(context, data, stlplus::dump_string_vector);
}

void restore_string_vector_list(stlplus::restore_context&, std::list<std::vector<std::string> >& data) throw(stlplus::persistent_restore_failed)
{
  stlplus::restore_list(context, data, stlplus::restore_string_vector);
}

There are also some shortcut functions that perform the whole three step sequence of creating an IOStream device, creating a context and performing the dump/restore in a single step. For example, to dump to a file called data.pst:

stlplus::dump_to_file(data, "data.pst", dump_string_vector_list, 0);

Subsystems in the Persistence Library

The rest of the explanation is split up into separate pages for each of the main subsystems within the persistence library:

Include Files

This section lists the types that are already made persistent by the built-in functions in the persistence library. In each case, the table below classifies the type and includes a link to the relevant page describing that subset of types. For example, all the basic (c-style) types are described on one page, the STL types on another and so on. Click on the "Library" column of the type you are interested in to go to the page for that type and others in the same category.

Persistence Functions
TypeLibraryIncludeFunction Names
bool C persistent_bool.hpp dump/restore_bool
char C persistent_int.hpp dump/restore_char
signed char C persistent_int.hpp dump/restore_signed_char
unsigned char C persistent_int.hpp dump/restore_unsigned_char
short C persistent_int.hpp dump/restore_short
unsigned short C persistent_int.hpp dump/restore_unsigned_short
int C persistent_int.hpp dump/restore_int
unsigned C persistent_int.hpp dump/restore_unsigned
long C persistent_int.hpp dump/restore_long
unsigned long C persistent_int.hpp dump/restore_unsigned_long
inf STLpluspersistent_inf.hpp dump/restore_inf
enum{} C persistent_enum.hpp dump/restore_enum
float C persistent_float.hpp dump/restore_float
double C persistent_float.hpp dump/restore_double
T* (simple) C/C++ persistent_pointer.hpp dump/restore_pointer
T* (polymorphic, interface)C++ persistent_interface.hpp dump/restore_interface
T* (polymorphic, callbacks)C++ persistent_callback.hpp dump/restore_callback
shared_ptr<T> (simple)STLpersistent_shared_ptr.hpp dump/restore_shared_ptr
shared_ptr<T> (polymophic, interface)STLpersistent_shared_ptr.hppdump/restore_shared_ptr_interface
shared_ptr<T> (polymorphic, callbacks)STLpersistent_shared_ptr.hppdump/restore_shared_ptr_callback
smart_ptr<T> (simple)STLpluspersistent_smart_ptr.hpp dump/restore_smart_ptr
smart_ptr_clone<T> (polymophic, interface)STLpluspersistent_smart_ptr.hppdump/restore_smart_ptr_clone_interface
smart_ptr_clone<T> (polymorphic, callbacks)STLpluspersistent_smart_ptr.hppdump/restore_smart_ptr_clone_callback
simple_ptr<T> (simple)STLpluspersistent_simple_ptr.hpp dump/restore_simple_ptr
simple_ptr_clone<T> (polymophic, interface)STLpluspersistent_simple_ptr.hppdump/restore_simple_ptr_clone_interface
simple_ptr_clone<T> (polymorphic, callbacks)STLpluspersistent_simple_ptr.hppdump/restore_simple_ptr_clone_callback
char* C persistent_cstring.hpp dump/restore_cstring
string STL persistent_string.hpp dump/restore_string
basic_string<T> STL persistent_string.hpp dump/restore_basic_string
bitset<N> STL persistent_bitset.hpp dump/restore_bitset
complex<T> STL persistent_complex.hpp dump/restore_complex
deque<T> STL persistent_deque.hpp dump/restore_deque
list<T> STL persistent_list.hpp dump/restore_list
vector<T> STL persistent_vector.hpp dump/restore_vector
pair<T1,T2> STL persistent_pair.hpp dump/restore_pair
triple<T1,T2,T3> STLpluspersistent_triple.hpp dump/restore_triple
foursome<T1,T2,T3,T4>STLpluspersistent_foursome.hpp dump/restore_foursome
hash<K,T,H,E> STLpluspersistent_hash.hpp dump/restore_hash
map<K,T> STL persistent_map.hpp dump/restore_map
multimap<K,T> STL persistent_multimap.hpp dump/restore_multimap
set<T> STL persistent_set.hpp dump/restore_set
multiset<T> STL persistent_multiset.hpp dump/restore_multiset
digraph<N,A> STLpluspersistent_digraph.hpp dump/restore_digraph
matrix<T> STLpluspersistent_matrix.hpp dump/restore_matrix
ntree<T> STLpluspersistent_ntree.hpp dump/restore_ntree