Chapter  13:  Data Serialization (ASN.1, XML)

Format Specific Streams: The CObject[IO]Stream classes

The reading and writing of serialized data objects entails satisfying two independent sets of constraints and specifications: (1) format-specific parsing and encoding schemes, and (2) object-specific internal structures and rules of composition. The NCBI C++ Toolkit implements serial IO processes by combining a set of object stream classes with an independently defined set of data object classes. These classes are implemented in the serial and objects directories respectively.

The base classes for the object stream classes are CObjectIStream and CObjectOStream. Each of these base classes has derived subclasses which specialize in different formats, including XML, binary ASN.1, and text ASN.1. A simple example program, xml2asn.cpp (see Box 1), described in Processing serial data, uses these object stream classes in conjunction with a CBiostruct object to translate a file from XML encoding to ASN.1 formats. In this chapter, we consider in more detail the class definitions for object streams, and how the type information associated with the data is used to implement serial input and output.

Each object stream specializes in a serial data format and a direction (in/out). It is not until the input and output operators are applied to these streams, in conjunction with a specified serializable object, that the object-specific type information comes into play. For example, if instr is a CObjectIStream, the statement: instr >> myObject invokes a Read() method associated with the input stream, whose sole argument is a CObjectInfo for myObject.

Similarly, the output operators, when applied to a CObjectOstream in conjunction with a serializable object, will invoke a Write() method on the output stream which accesses the object's type information. The object's type information defines what tag names and value types should be encountered on the stream, while the CObject[IO]Stream subclasses specialize the data serialization format.

The input and output operators (<< and >>) are declared in serial/serial.hpp header.

The CObjectIStream (*) classes

CObjectIStream is a virtual base class for the CObjectIStreamXml, CObjectIStreamAsn, and CObjectIStreamAsnBinary classes. As such, it has no public constructors, and its user interface includes the following methods:

• Open()

• Close()

• GetDataFormat()

• ReadFileHeader()

• Read()

• ReadObject()

• ReadSeparateObject()

• Skip()

• SkipObject()

There are several Open() methods; most of these are static class methods that return a pointer to a newly created CObjectIStream. Typically, these methods are used with an auto_ptr, as in:

Here, an XML format is specified by the enumerated value eSerial_Xml, defined in ESerialDataFormat. Because these methods are static, they can be used to create a new instance of a CObjectIStream subclass, and open it with one statement. In this example, a CObjectIStreamXml is created and opened on the file filename.

An additional non-static Open() method is provided, which can only be invoked as a member function of a previously instantiated object stream (whose format type is of course, implicit to its class). This method takes a CNcbiIstream and a Boolean argument, specifying whether or not the CNcbiIstream should also be deleted when the object stream is closed:

The next three methods have the following definitions. Close() closes the stream. GetDataFormat() returns the enumerated ESerialDataFormat for the stream. ReadFileHeader() reads the first line from the file, and returns it in a string. This might be used for example, in the following context:

Several Read*() methods are provided for usage in different contexts. CObjectIStream::Read() should be used for reading a top-level "root" object from a data file. For convenience, the input operator >>, as described above, indirectly invokes this method on the input stream, using a CObjectTypeInfo object derived from myObject. By default, the Read() method first calls ReadFileHeader(), and then calls ReadObject(). Accordingly, calls to Read() which follow the usage of ReadFileHeader()must include the optional eNoFileHeader argument.

Most data objects also contain embedded objects, and the default behavior of Read() is to load the top-level object, along with all of its contained subobjects into memory. In some cases this may require significant memory allocation, and it may be only the top-level object which is needed by the application. The next two methods, ReadObject() and ReadSeparateObject(), can be used to load subobjects as either persistent data members of the root object or as temporary local objects. In contrast to Read(), these methods assume that there is no file header on the stream.

As a result of executing ReadObject(member), the newly created subobject will be instantiated as a member of its parent object. In contrast, ReadSeparateObject(local), instantiates the subobject in the local temporary variable only, and the corresponding data member in the parent object is set to an appropriate null representation for that data type. In this case, an attempt to reference that subobject after exiting the scope where it was created generates an error.

The Skip() and SkipObject() methods allow entire top-level objects and subobjects to be "skipped". In this case the input is still read from the stream and validated, but no object representation for that data is generated. Instead, the data is stored in a delay buffer associated with the object input stream, where it can be accessed as needed. Skip() should only be applied to top-level objects. As with the Read() method, the optional ENoFileHeader argument can be included if the file header has already been extracted from the data stream. SkipObject(member) may be applied to subobjects of the root object.

All of the Read and Skip methods are like wrapper functions, which define what activities take place immediately before and after the data is actually read. How and when the data is then loaded into memory is determined by the object itself. Each of the above methods ultimately calls objTypeInfo->ReadData() or objTypeInfo->SkipData(), where objTypeInto is the static type information object associated with the data object. This scheme allows the user to install type-specific read, write, and copy hooks, which are described below. For example, the default behavior of loading all subobjects of the top-level object can be modified by installing appropriate read hooks which use the ReadSeparateObject() and SkipObject() methods where needed.

The CObjectOStream (*) classes

The output object stream classes mirror the CObjectIStream classes. The CObjectOStream base class is used to derive the CObjectOStreamXml, CObjectOStreamAsn, and CObjectOStreamAsnBinary classes. There are no public constructors, and the user interface includes the following methods:

• Open()

• Close()

• GetDataFormat()

• WriteFileHeader()

• Write()

• WriteObject()

• WriteSeparateObject()

• Flush()

• FlushBuffer()

Again, there are several Open() methods, which are static class methods that return a pointer to a newly created CObjectOstream:

The Write*() methods correspond to the Read*() methods defined for the input streams. Write() first calls WriteFileHeader(), and then calls WriteObject(). WriteSeparateObject() can be used to write a temporary object (and all of its children) to the output stream. It is also possible to install type-specific write hooks. Like the Read() methods, these Write() methods serve as wrapper functions that define what occurs immediately before and after the data is actually written.

The CObjectStreamCopier (*) classes

The CObjectStreamCopier class is neither an input nor an output stream class, but a helper class, which allows one to "pass data through" without storing the intermediate objects in memory. Its sole constructor is:

and its most important method is the Copy(CObjectTypeInfo&) method, which, given an object's description, reads that object from the input stream and writes it to the output stream. The serial formats of both the input and output object streams are implicit, and thus the translation between two different formats is performed automatically.

In keeping with the Read and Write methods of the CObjectIStream and CObjectOStream classes, the Copy method takes an optional ENoFileHeader argument, to indicate that the file header is not present in the input and should not be generated on the output. The CopyObject() method corresponds to the ReadObject() and WriteObject() methods.

As an example, consider how the Run() method in xml2asn.cpp might be implemented differently using the CObjectStreamCopier class:

It is also possible to install type-specific Copy hooks. Like the Read and Write methods, the Copy methods serve as wrapper functions that define what occurs immediately before and after the data is actually copied.

Type-specific I/O routines

Much of the functionality needed to read and write serializable objects may be type-specific yet application-driven. Because the specializations may vary with the application, it does not make sense to implement fixed methods, yet we would like to achieve a similar kind of object-specific behavior.

To address these needs, the C++ Toolkit provides hook mechanisms, whereby the needed functionality can be installed with the object's static class type information object. Such hooks can be installed globally, where they will be applied on all streams where these events occur, or locally, where they will only be applied to a selected stream.

For any given object and specific stream, at most one read hook and one write hook is "active". If myObject has a locally installed read hook as well as a global read hook, then the locally installed hook will override the global hook when a read occurs on the "local" stream. Read events on all of the other "non-local" streams will of course, trigger the globally installed hook. Designating multiple read/write hooks (both local and global) for a selected object does not generate an error. Older or less specific hooks are simply overridden by the more specific or most recently installed hook.

The Read hook classes

All of the different contexts in which an object might be encountered on an input stream can be reduced to three cases:

as a stand-alone object

as a data member of a containing object

as a variant of a choice object

Hooks can be installed for each of these contexts, depending on the desired level of specificity. Corresponding to these contexts, three abstract base classes provide the foundations for deriving new Read hooks:

• CReadObjectHook

• CReadClassMemberHook

• CReadChoiceVariantHook

Each of these base hook classes exists only to define a pure virtual Read method, which can then be implemented (in a derived subclass) to install the desired type of read hook. If the goal is to apply the new Read method in all contexts, then the new hook should be derived from the CReadObjectHook class, and registered with the object's static type information object. For example, to install a new CReadObjectHook for a CBioseq, one might use:

Another way of installing hooks of any type (read/write/copy, object/member/variant) is provided by CObjectHookGuard class described below.

Alternatively, if the desired behavior is to trigger the specialized Read method only when the object occurs as a data member of a particular containing class, then the new hook should be derived from the CReadClassMemberHook, and registered with that member's type information object:

Similarly, one can install a read hook that will only be triggered when the object occurs as a choice variant:

The new hook classes for these examples should be derived from CReadObjectHook, CReadClassMemberHook, and CReadChoiceVariantHook, respectively. In the first case, all occurrences of CBioseq on any input stream will trigger the new Read method. In contrast, the last case installs this new Read method to be triggered only when the CBioseq occurs as a choice variant in a CSeq_entry object.

All of the virtual Read methods take two arguments: a CObjectIStream and a reference to a CObjectInfo. For example, the CReadObjectHook class declares the ReadObject() method as:

The ReadClassMember and ReadChoiceVariant hooks differ from the ReadObject hook class, in that the second argument to the virtual Read method is an iterator, pointing to the object type information for a data member or choice variant respectively.

In summary, to install a read hook for an object type:

derive a new class from the appropriate hook class:

• if the target object occurs in any context, use the CReadObjectHook class.

• if the target object occurs as a data member, use the CReadClassMemberHook class.

• if the target object occurs as a choice variant, use the CReadChoiceVariant Hook class.

implement the virtual Read method for the new class.

install the hook, using the SetGlobalReadHook() or SetLocalReadHook() method defined in

• CObjectTypeInfo for a CReadObjectHook

• CMemberInfo for a CReadClassMemberHook

• CVariantInfo for a CReadChoiceVariantHook

or use CObjectHookGuard class to install any of these hooks.

In many cases you will need to read the hooked object and do some special processing, or to skip the entire object. To simplify object reading or skipping all base hook classes have DefaultRead() and DefaultSkip() methods taking the same arguments as the user provided ReadXXXX() methods. Thus, to read a bioseq object from a hook:

Note that from a choice variant hook you can not skip stream data -- this could leave the choice object in an uninitialized state. For this reason the CReadChoiceVariantHook class has no DefaultSkip() method.

For a good example of using a CReadClassMemberHook object, see the asn2asn.cpp and test_serial.cpp demo programs.

The Write hook classes

The Write hook classes parallel the Read hook classes, and again, we have three base classes:

• CWriteObjectHook

• CWriteClassMemberHook

• CWriteChoiceVariantHook

These classes define the pure virtual methods:

Like the read hooks, your derived write hooks can be installed by invoking the SetGlobalWriteObjectHook() or SetLocalWriteObjectHook() methods for the appropriate type information objects. Corresponding to the examples for read hooks then, we would have:

CObjectHookGuard class provides is a simple way to install write hooks.

The asn2asn.cpp and test_serial.cpp demo programs also demonstrate the usage of the CWriteClassMemberHook class.

The Copy hook classes

As with the Read and Write hook classes, there are three base classes which define the following Copy methods:

Newly derived copy hooks can be installed by invoking the SetGlobalCopyObjectHook() or SetLocalCopyObjectHook() methods for the appropriate type information objects. The other way of installing hooks is described below in the CObjectHookGuard section.

To do default copying of an object in the overloaded hook method each of the base copy hook classes has DefaultCopy() method.

The CObjectHookGuard class

To simplify hooks usage CObjectHookGuard class may be used. It's a template class: the template parameter is the class to be hooked (in case of member or choice variant hooks it's the parent class of the member).

The CObjectHookGuard class has several constructors for installing different hook types. The last argument to all constructors is a stream pointer. By default the pointer is NULL and the hook is intalled as a global one. To make the hook stream-local pass the stream to the guard constructor.

• Object read/write hooks:
  CObjectHookGuard(CReadObjectHook& hook,
  CObjectIStream* in = 0);
  CObjectHookGuard(CWriteObjectHook& hook,
  CObjectOStream* out = 0);

• Class member read/write hooks:
  CObjectHookGuard(string id,
  CReadClassMemberHook& hook,
  CObjectIStream* in = 0);
  CObjectHookGuard(string id,
  CWriteClassMemberHook& hook,
  CObjectOStream* out = 0);

The string "id" argument is the name of the member in ASN.1 specification for generated classes.

• Choice variant read/write hooks:
  CObjectHookGuard(string id,
  CReadChoiceVariantHook& hook,
  CObjectIStream* in = 0);
  CObjectHookGuard(string id,
  CWriteChoiceVariantHook& hook,
  CObjectOStream* out = 0);

The string "id" argument is the name of the variant in ASN.1 specification for generated classes.

The guard's destructor will uninstall the hook. Since all hook classes are derived from CObject and stored as CRef<>-s, the hooks are destroyed automatically when uninstalled. For this reason it's recommended to create hook objects on heap.

Stack Path Hooks

When using serialization hooks one might want to specify a more specific context when such hook should be triggered. For example, "I want to hook the reading of object A when and only when it is a member of object B, not all occurrences of object A", or "I want to hook the reading of all members named 'Title' in all objects, not only in a specific one". The serial library makes it possible to set serialization hooks by string that describes a place (or stack path), for example:

The format of the string is as follows:

Where TypeName and MemberName are strings, '.' is a separator. Wildcard is defined as

Here the question mark means "one member with any name", while the asterisk means "one or more members with any names".

As with regular serialization hooks, it is possible to install a path hookfor a specific object:

a data member of an object:

or a variant of a choice object:

Here in is a pointer to an input object stream. If it is equal to zero, the hook will be installed globally, otherwise - for that particular stream. In addition to that, it is possible to install such hooks in object streams. So, for example to install a read hook on all string data members and choice variants named LastName, one could use either the following code:

Or this one:

Setting path hooks directly in streams also makes it possible to differentiate between LastName being a data member and choice variant. So, for example

will catch all data members and skip choice variants; while

will trigger for all variants and skip data members.

Stream Iterators

When working with a stream, it is sometimes convenient to be able to read or write data elements directly, bypassing the standard data storage mechanism. For example, when reading a large container object, the purpose could be to process its elements. It is possible to read everything at once, but this could require a lot of memory to store the data in. An alternative approach, which greatly reduces the amount of required memory, could be to read elements one by one, process them as they arrive, and then discard. Or, when writing a container, one could construct it in memory only partially, and then add missing elements 'on the fly' - where appropriate. To make it possible, the SERIAL library introduces stream iterators. Needless to say, the most convenient way of using this mechanism is in read/write hooks.

SERIAL library defines the following stream iterator classes: CIStreamClassMemberIterator and CIStreamContainerIterator for input streams, and COStreamClassMember and COStreamContainer for output ones.

Reading a container could look like this:

Writing - like this:

For more examples of using stream iterators please refer to asn2asn sample application.

The ByteBlock and CharBlock classes

CObject[IO]Stream::ByteBlock class may be used for non-standard processing of an OCTET STRING data, e.g. from a read/write hooks. The CObject[IO]Stream::CharBlock class has almost the same functionality, but may be used for VisibleString data processing.

An example of using ByteBlock or CharBlock classes is generating data on-the-fly in a write hook. To use block classes:

Initialize the block variable with an i/o stream and, in case of output stream, the length of the block.

Use Read()/Write() functions to process block data

Close the block with the End() function

Below is an example of using CObjectOStream::ByteBlock in an object write hook for non-standard data processing. Note, that ByteBlock and CharBlock classes read/write data only. You should also provide some code for writing class' and members' tags.

Since OCTET STRING and VisibleString in the NCBI C++ Toolkit are implemented as vector<char> and string classes, which have no serailization type info, you can not install a read or write hook for these classes. The example also demonstrates how to process members of these types using the containing class hook. Another example of using CharBlock with write hooks can be found in test_serial.cpp application.

NCBI C++ Toolkit Network Service (RPC) Clients

The following topics are discussed in this section:

• Introduction and Use

• Implementation Details

Verification of Class Member Initialization

When serializing an object, it is important to verify that all mandatory primitive data members (e.g. strings, integers) are given a value. The NCBI C++ Toolkit implements this through a data initialization verification mechanism. In this mechanism, the value itself is not validated; that is, it still could be semantically incorrect. The purpose of the verification is only to make sure that the member has been assigned some value. The verification also provides for a possibility to check whether the object data member has been initialized or not. This could be useful when constructing such objects in memory.

From this perspective, each data member (XXX) of a serial object generated by DATATOOL from an ASN or XML specification has the IsSetXXX() and CanGetXXX() methods. Also, input and output streams have SetVerifyData() and GetVerifyData() methods. The purpose of CanGetXXX() method is to answer the question whether it is safe or not to call the corresponding GetXXX(). The meaning of IsSetXXX() is whether the data member has been assigned a value explicitly (using assignment function call, or as a result of reading from a stream) or not. The stream's SetVerifyData() method defines a stream behavior in case it comes across an uninitialized data member.

There are three kinds of object data members:

• optional ones,

• mandatory with a default value,

• mandatory with no default value.

Optional members and mandatory ones with no default have "no value" initially. As such, they are "ungetatable"; that is, GetXXX() throws an exception (this is also configurable though). Mandatory members with a default are always getable, but not always set. It is possible to assign a default value to a mandatory member with a default value. In this case it becomes set, and as such will be written into an output stream.

The discussion above refers only to primitive data members, such as strings, or integers. The behavior of containers is somewhat different. All containers are pre-created on the parent object construction, so for container data members CanGetXXX() always returns TRUE. This can be justified by the fact that containers have a sort of "natural default value" - empty. Also, IsSetXXX() will return TRUE if the container is either mandatory, or has been read (even if empty) from the input stream, or SetXXX() was called for it.

The following additional topics are discussed in this section:

• Initialization Verification in CSerialObject Classes

• Initialization Verification in Object Streams

Simplified Serialization Interface

The reading and writing of serial object requires creation of special object streams which encode and decode data. While such streams provide with a greater flexibility in setting the formatting parameters, in some cases it is not needed - the default behavior is quite enough. NCBI C++ toolkit library makes it possible to use the standard I/O streams in this case, thus hiding the creation of object streams. So, the serialization would look like this:

The only information that is always needed is the output format. It is defined by the following stream manipulators:

• MSerial_AsnText

• MSerial_AsnBinary

• MSerial_Xml

Few additional manipulators define the handling of un-initialized object data members:

• MSerial_VerifyDefault

• MSerial_VerifyNo

• MSerial_VerifyYes

• MSerial_VerifyDefValue

Finding in input stream objects of a specific type

When processing serialized data, it is pretty often that one has to find all objects of a specific type, with this type not being a root one. To make it easier, serial library defines a helper template function Serial_FilterObjects. The idea is to be able to define a special hook class with a single virtual function Process with a single parameter: object of the requested type. Input stream is being scanned then, and, when an object of the requested type is encountered, the user-supplied function is being called.

Say of instance, an input stream contains Bioseq objects, and you need to find and process all Seq-inst objects in it. First, you need to define a class that will process them:

Second, you just call filtering function specifying the root object type:

Another variant of this function – Serial_FilterStdObjects – finds objects of standard type, not derived from CSerialObject – strings, for example. The usage is similar. First, define a hook class that will process data:

Then, call the filtering function:

Even more sophisticated, yet easier to use mechanism relies on multi-threading. It puts data reading into a separate thread and hides synchronization issues from client application. There are two template classes, which make it possible: CIStreamObjectIterator and CIStreamStdIterator. The former finds objects of CSerialObject type:

The latter – objects of standard type:

STL generic iterators

Iterators are an important cornerstone in the generic programming paradigm - they serve as intermediaries between generic containers and generic algorithms. Different containers have different access properties, and the interface to a generic algorithm must account for this.

The vector class allows input, output, bidirectional, and random access iterators. In contrast, the list container class does not allow random access to its elements. This is depicted graphically by one less strand in the ribbon connector. In addition to the iterators, the generic algorithms may require function objects such as less<T> to support the template implementations.

The STL standard iterators are designed to iterate through any STL container of homogeneous elements, e.g., vectors, lists, deques, stacks, maps, multimaps, sets, multisets, etc. A prerequisite however, is that the container must have begin() and end() functions defined on it as start and end points for the iteration.

But while these standard iterators are powerful tools for generic programming, they are of no help in iterating over the elements of aggregate objects - e.g., over the heterogeneous data members of a class object. As this is an essential operation in processing serialized data structures, the NCBI C++ Toolkit provides additional types of iterators for just this purpose. In the section on Runtime object type information, we described the CMemberIterator and CVariantIterator classes, which provide access to the instance and type information for all of the data members and choice variants of a class or choice object. In some cases however, we may wish to visit only those data members which are of a certain type, and do not require any type information. The iterators described in this section are of this type.

CTypeIterator (*) and CTypeConstIterator (*)

The CTypeIterator and CTypeConstIterator can be used to traverse a structured object, stopping at all data members of a specified type. For example, it is very common to represent a linked list of objects by encoding a next field that embeds an object of the same type. One way to traverse the linked list then, would be to "iterate" over all objects of that type, beginning at the head of the list. For example, suppose you have a CPersonclass defined as:

Given this definition, one might then define a neighborhood using a single CPerson. Assuming a function FullerBrushMan(CPerson&) must now be applied to each person in the neighborhood, this could be implemented using a CTypeIterator as follows:

In this example, the data members visited by the iterator are of the same type as the top-level aggregate object, since neighbor is an instance of CPerson. Thus, the first "member" visited is the top-level object itself. This is not always the case however. The top-level object is only included in the iteration when it is an instance of the type specified in the template argument (CPerson in this case).

All of the NCBI C++ Toolkit type iterators are recursive. Thus, since neighborhood has CPerson data members, which in turn contain objects of type CPerson, all of the nested data members will also be visited by the above iterator. More generally, given a hierarchically structured object containing data elements of a given type nested several levels deep, the NCBI C++ Toolkit type iterators effectively generate a "flat" list of all these elements.

It is not difficult to imagine situations where recursive iterators such as the CTypeIterator could lead to infinite loops. An obvious example of this would be a doubly-linked list. For example, suppose CPerson had both previous and next data members, where x->next->previous == x. In this case, visiting x followed by x->next would lead back to x with no terminating condition. To address this issue, the Begin() function accepts an optional second argument, eDetectLoops. eDetectLoops is an enum value which, if included, specifies that the iterator should detect and avoid infinite loops. The resulting iterator will be somewhat slower but can be safely used on objects whose references might create loops.

Let's compare the syntax of this new iterator class to the standard iterators:

The standard iterator begins by pointing to the first item in the container x.begin(), and with each iteration, visits subsequent items until the end of the container x.end() is reached. Similarly, the CTypeIterator begins by pointing to the first data member of ObjectName that is of type T, and with each iteration, visits subsequent data members of type T until the end of the top-level object is reached.

A lot of code actually uses = Begin(...) instead of (Begin(...)) to initialize iterators; although the alternate syntax is somewhat more readable and often works, some compilers can mis-handle it and give you link errors. As such, direct initialization as shown above generally works better. Also, note that this issue only applies to construction; you should (and must) continue to use = to reset existing iterators.

How are generic iterators such as these implemented? The Begin() expression returns an object containing a pointer to the input object ObjectName, as well as a pointer to a CTypeInfo object containing type information about that object. On each iteration, the ++ operator examines the current type information to find the next data member which is of type T. The current object, its type information, and the state of iteration is pushed onto a local stack, and the iterator is then reset with a pointer to the next object found, and in turn, a pointer to its type information. Each data member of type T (or derived from type T) must be capable of providing its own type information as needed. This allows the iterator to recursively visit all data members of the specified type at all levels of nesting.

More specifically, each object included in the iteration, as well as the initial argument to Begin(), must have a statically implemented GetTypeInfo() class member function to provide the needed type information. For example, all of the serializable objects generated by datatool in the src/objects subtrees have GetTypeInfo() member functions. In order to apply type iterators to user-defined classes (as in the above example), these classes must also make their type information explicit. A set of macros described in the section on User-defined Type Information are provided to simplify the implementation of the GetTypeInfo() methods for user-defined classes. The example included at the end of this section (see Additional Information) uses several of the C++ Toolkit type iterators and demonstrates how to apply some of these macros.

The CTypeConstIterator parallels the CTypeIterator, and is intended for use with const objects (i.e. when you want to prohibit modifications to the objects you are iterating over). For const iterators, the ConstBegin() function should be used in place of Begin().

Class hierarchies, embedded objects, and the NCBI C++ type iterators

As emphasized above, all of the objects visited by an iterator must have the GetTypeInfo() member function defined in order for the iterators to work properly. For an iterator that visits objects of type T, the type information provided by GetTypeInfo() is used to identify:

• data members of type T

• data members containing objects of type T

• data members derived from type T

• data members containing objects derived from type T

Explicit encoding of the class hierarchy via the GetTypeInfo() methods allows the user to deploy a type iterator over a single specified type which may in practice include a set of types via inheritance. The section Additional Information a simple example of this feature. The preprocessor macros used in this example which support the encoding of hierarchical class relations are described in the User-defined Type Information section. A further generalization of this idea is implemented by the CTypesIterator described later.

CObjectIterator (*) and CObjectConstIterator (*)

Because the CObject class is so central to the Toolkit, a special iterator is also defined, which can automatically distinguish CObjects from other class types. The syntax of a CObjectIterator is:

Note that there is no need to specify the object type to iterate over, as the type CObject is built into the iterator itself. This iterator will recursively visit all CObjects contained or referenced in ObjectName. The CObjectConstIterator is identical to the CObjectIterator but is designed to operate on const elements and uses the ConstBegin() function.

User-defined classes that are derived from CObject can also be iterated over (assuming their GetTypeInfo() methods have been implemented). In general however, care should be used in applying this type of iterator, as not all of the NCBI C++ Toolkit classes derived from CObject have implementations of the GetTypeInfo() method. All of the generated serializable objects in include/objectsdo have a defined GetTypeInfo() member function however, and thus can be iterated over using either a CObjectIterator or a CTypeIterator with an appropriate template argument.

CStdTypeIterator (*) and CStdTypeConstIterator (*)

All of the type iterators described thus far require that each object visited must provide its own type information. Hence, none of these can be applied to standard types such as int, float, double or the STL type string. The CStdTypeIterator and CStdTypeConstIterator classes selectively iterate over data members of a specified type. But for these iterators, the type must be a simple C type (int, double, char*, etc.) or an STL type string. For example, to iterate over all the string data members in a CPerson object, we could use:

The CStdTypeConstIterator is identical to the CStdTypeIterator but is designed to operate on const elements and requires the ConstBegin() function.

Code examples using the CTypeIterator and CStdTypeIterator are given in ctypeiter.cpp (see Box 2; for ctypeiter.hpp, see Box 3).

CTypesIterator (*)

Sometimes it is necessary to iterate over a set of types contained inside an object. The CTypesIterator, as its name suggests, is designed for this purpose. For example, suppose you have loaded a gene sequence into memory as a CBioseq (named seq), and want to iterate over all of its references to genes and organisms. The following sequence of statements defines an iterator that will step through all of seq's data members (recursively), stopping only at references to gene and organism citations:

Here, CType is a helper template class that simplifies the syntax required to use the multiple types iterator:

• CType<TypeName>::AddTo(i) specifies that iterator i should stop at type TypeName.

• CType<TypeName>::Match(i) returns true if the specified type TypeName is the type currently pointed to by iterator i.

• CType<TypeName>::Get(i) retrieves the object currently pointed to by iterator iif there is a type match to TypeName, and otherwise returns 0. In the event there is a type match, the retrieved object is type cast to TypeName before it is returned.

The Begin() expression is as described for the above CTypeIterator and CTypeConstIterator classes. The CTypesConstIterator is the const implementation of this type of iterator, and requires the ConstBegin() function.

Context Filtering In Type Iterators

In addition to traversing object of a specific type one might want to specify the context in which such objects should appear. For example, you could wish to iterate over string data members, but only those of them that are called title. This could be done using context filtering. Such filter is a string with the format identical to the one used in Stack Path Hooks and is specified as an additional parameter of a type iterator. So, for example, the declaration of string data member iterator with context filtering could look like this:

Additional Information

The following example demonstrates how the class hierarchy determines which data members will be included in a type iterator. The example uses five simple classes:

• Class CA contains a single int data member and is used as a target object type for the type iterators demonstrated.

• class CB contains an auto_ptr to a CA object.

• Class CC is derived from CA and is used to demonstrate the usage of class hierarchy information.

• Class CD contains an auto_ptr to a CC object, and, since it is derived from CObject, can be used as the object pointed to by a CRef.

• Class CX contains both pointers-to and instances-of CA, CB, CC, and CD objects, and is used as the argument to Begin() for the demonstrated type iterators.

The preprocessor macros used in this example implement the GetTypeInfo() methods for the classes, and are described in the section on User-defined type information.

Figure 1 illustrates the paths traversed by CTypeIterator<CA> and CTypeIterator<CC>, where both iterators are initialized with Begin(a). The data members visited by the iterator are indicated by enclosing boxes. See Figure 1.

For additional examples of using the type iterators described in this section, see ctypeiter.cpp.

Accessing the object header files and serialization libraries

Reading and writing serialized data is implemented by an integrated set of streams, filters, and object types. An application that reads encoded data files will require the object header files and libraries which define how these serial streams of data should be loaded into memory. This entails #include statements in your source files, as well as the associated library specifications in your makefiles. The object header and implementation files are located in the include/objects and src/objects subtrees of the C++ tree, respectively. The header and implementation files for serialized streams and type information are in the include/serial and src/serial directories.

If you have checked out the objects directories, but not explicitly run the datatool code generator, then you will find that your include/objects subdirectories are (almost) empty, and the source subdirectories contain only makefiles and ASN.1 specifications. These makefiles and ASN.1 specifications can be used to build your own copies of the objects' header and implementation files, using make all_r (if you configured using the --with-objects flag), or running datatool explicitly.

However, building your own local copies of these header and implementation files is neither necessary nor recommended, as it is simpler to use the pre-generated header files and prebuilt libraries. The pre-built header and implementation files can be found in $NCBI/c++/include/objects/ and $NCBI/c++/src/objects/, respectively. Assuming your makefile defines an include path to $NCBI/c++/include, selected object header files such as Date.hpp, can be included as:

This header file (along with its implementations in the accompanying src directory) was generated by datatool using the specifications from src/objects/general/general.asn. In order to use the classes defined in the objects directories, your source code should begin with the statements:

All of the objects' header and implementation files are generated by datatool, as specified in the ASN.1 specification files. The resulting object definitions however, are not in any way dependent on ASN.1 format, as they simply specify the in-memory representation of the defined data types. Accordingly, the objects themselves can be used to read, interpret, and write any type of serialized data. Format specializations on the input stream are implemented via CObjectIStream objects, which extract the required tags and values from the input data according to the format specified. Similarly, Format specializations on an output stream are implemented via CObjectOStream objects.

Reading and writing serial data

Let's consider a program xml2asn.cpp that translates an XML data file containing an object of type Biostruc, to ASN.1 text and binary formats. In main(), we begin by initializing the diagnostic stream to write errors to a local file called xml2asn.log. (Exception handling, program tracing, and error logging are described in the Diagnostic Streams section).

An instance of the CTestAsn class is then created, and its member function AppMain() is invoked. This function in turn calls CTestAsn::Run(). The first three lines of code there define the XML input and ASN.1 output streams, using auto_ptrs, to ensure automatic destruction of these objects.

Each stream is associated with data serialization mechanisms appropriate to the ESerialDataFormat provided to the constructor:

CObjectIStream and CObjectOStream are base classes which provide generic interfaces between the specific type information of a serializable object and an I/O stream. The object stream classes that will actually be instantiated by this application, CObjectIStreamXml, CObjectOStreamAsn, and CObjectOStreamAsnBinary, are descendants of these base classes.

Finally, a variable for the object type that will be generated from the input stream (in this case a CBiostruc) is defined, and the CObject[I/O]Stream operators "<<" and ">>" are used to read and write the serialized data to and from the object. (Note that it is not possible to simply "pass the data through", from the input stream to the output stream, using a construct like: *inObject >> *outObject). The CObject[I/O]Streams know nothing about the structure of the specific object - they have knowledge only of the serialization format (text ASN, binary ASN, XML, etc.). In contrast, the CBiostruc knows nothing about I/O and serialization formats, but it contains explicit type information about itself. Thus, the CObject[I/O]Streams can apply their specialized serialization methods to the data members of CBiostruc using the type information associated with that object's class.

Determining Which Header Files to Include

As always, we include the corelib header files, ncbistd.hpp and ncbiapp.hpp. In addition, the serial header files that define the generic CObject[IO]Stream objects are included, along with serial.hpp, which defines generalized serialization mechanisms. Finally, we need to include the header file for the object type we will be using.

There are two source browsers that can be used to locate the appropriate header file for a particular object type. All class names in the NCBI C++ Toolkit begin with the letter "C". Using the class hierarchy browser, we find CBiostruc, derived from CBiostruc_Base, which is in turn derived from CObject. Following the CBiostruc link, we can then use the locate button to move to the LXR source code navigator, and there, find the name of the header file. In this case, we find CBiostruc.hpp is located in include/objects/mmdb1. Alternatively, if we know the name of the C++ class, the source code navigator's identifier search tool can be used directly. In summary, the following #include statements appear at the top of xml2asn.cpp:

Determining Which Libraries to Link To

Determining which libraries must be linked to requires a bit more work and may involve some trial and error. The list of available libraries currently includes:

access biblio cdd featdef general medlars medline mmdb1 mmdb2 mmdb3 ncbimime objprt proj pub pubmed seq seqalign seqblock seqcode seqfeat seqloc seqres seqset submit xcgi xconnect xfcgi xhtml xncbi xser

It should be clear that we will need to link to the core library, xncbi, as well as to the serial library, xser. In addition, we will need to link to whatever object libraries are entailed by using a CBiostruc object. Minimally, one would expect to link to the mmdb libraries. This in itself is insufficient however, as the CBiostruc class embeds other types of objects, including PubMed citations, features, and sequences, which in turn embed additional objects such as Date. The makefile for xml2asn.cpp, Makefile.xml2asn.app lists the libraries required for linking in the make variable LIB.

See also the example program, asn2asn.cpp which demonstrates more generalized translation of Seq-entry and Bioseq-set (defined in seqset.asn).

Introduction

Object type information, as it is used in the NCBI C++ Toolkit, is defined in the section on Runtime Object Type Information. As described there, all of the classes and constructs defined in the serial include and src directories have a static implementation of a GetTypeInfo() function that yields a CTypeInfo for the object of interest. In this section, we describe how type information can also be generated and accessed for user-defined types. We begin with a review of some of the basic notions introduced in the previous discussion.

The type information for a class is stored outside any instances of that class, in a statically created CTypeInfo object. A class's type information includes the class layout, inheritance relations, external alias, and various other attributes that are independent of specific instances. In addition, the type information object provides an interface to the class's data members.

Limited type information is also available for primitive data types, enumerations, containers, and pointers. The type information for a primitive type specifies that it is an int, float, or char, etc., and whether or not that element is signed. Enumerations are a special kind of primitive type, whose type information specifies its enumeration values and named elements. Type information for containers can specify both the type of container and the type of elements. The type information for a pointer provides convenient methods of access to the type information for the type pointed to.

For all types, the type information is encoded in a static CTypeInfo object, which is then accessed by all instances of a given type using a GetTypeInfo() function. For class types, this function is implemented as a static method for the class. For non class types, GetTypeInfoXxx() is implemented as a static global function, where Xxx is a unique suffix generated from the type's name. With the first invocation of GetTypeInfo() for a given type, the static CTypeInfo object is created, which then persists (local to the function GetTypeInfo()) throughout execution. Subsequent calls to GetTypeInfo() simply return a pointer to this statically created local object.

In order to make type information about user-defined classes accessible to your application, the user-defined classes must also implement a static GetTypeInfo() method. A set of preprocessor macros is available, which greatly simplifies this effort. A pre-requisite to using these macros however, is that the class definition must include the following line:

This pre-processor macro will generate the following in-line statement in the class definition:

As with class objects, there must be some means of declaring the type information function for an enumeration prior to using the macros which implement that function. Given an enumeration named EMyEnum, DECLARE_ENUM_INFO(EMyEnum) will generate the following declaration:

The DECLARE_ENUM_INFO() macro should appear in the header file where the enumeration is defined, immediately following the definition. The DECLARE_INTERNAL_ENUM_INFO macro is intended for usage with internal class definitions, as in:

The C++ Toolkit also allows one to provide type information for legacy C style struct and choice elements defined in the C Toolkit. The mechanisms used to implement this are mentioned but not described in detail here, as it is not likely that newly-defined types will be in these categories.

Installing a GetTypeInfo() function: the BEGIN_/END_macros

Several pre-processor macros are available for the installation of the GetTypeInfo() functions for different types. Table 2 lists six BEGIN_NAMED_*_INFO macros, along with a description of the type of object each can be applied to and its expected arguments. Each macro in Table 2 has a corresponding END_*_INFO macro definition.

The first four macros in Table 2 apply to C++ objects. The DECLARE_INTERNAL_TYPE_INFO() macro must appear in the class definition's public section. These macros take two string arguments:

• an external alias for the type, and

• the internal C++ symbolic class name

The next two macros implement global, uniquely named functions which provide access to type information for C++ enumerations; the resulting functions are named GetTypeInfo_enum_[EnumName]. The DECLARE_ENUM_INFO() or DECLARE_ENUM_INFO_IN() macro should be used in these cases to declare the GetTypeInfo*() functions.

The usage of these six macros generally takes the following form:

That is, the BEGIN/END macros are used to generate the function's signature and enclosing block, and various ADD_* macros are applied to add information about internal members and class relations.

Specifying internal structure and class inheritance: the ADD_ macros

Information about internal class structure and inheritance is specified using the ADD_* macros (see Table 3). Again, each macro has both a "named" and "unnamed" implementation. The arguments to all of the ADD_NAMED_* macros begin with the external alias and C++ name of the item to be added.

The ADD_* macros that take only an alias and a name require that the type being added must be either a built-in type or a type defined by the name argument. When adding a CRef data member to a class or choice object however, the class referenced by the CRef must be made explicit with the RefClass argument, which is the C++ class name for the type pointed to.

Similarly, when adding an enumerated data member to a class, the enumeration itself must be explicitly named. For example, if class CMyClass contains a data member m_MyEnumVal of type EMyEnum, then the BEGIN_NAMED_CLASS_INFO macro for CMyClass should contain the statement:

or, equivalently:

or, to define a "custom" (non-default) external alias:

Here, EMyEnum is defined in the same namespace and scope as CMyClass. Alternatively, if the enumeration is defined in a different class or namespace (and therefore, then the ADD_ENUM_IN_MEMBER macro must be used:

In this example, EMyEnum is defined in a class named COtherClassName. The CppContext argument (defined here as COtherClassName::) acts as a scope operator, and can also be used to specify an alternative namespace. The ADD_NAMED_ENUM_CHOICE_VARIANT and ADD_NAMED_ENUM_IN_CHOICE_VARIANT macros are used similarly to provide information about enumerated choice options. The ADD_ENUM_VALUE macro is used to add enumerated values to the enumeration itself, as demonstrated in the above example of the BEGIN_NAMED_ENUM_INFO macro.

The most complex macros by far are those which use the TypeMacro and TypeMacroArgs arguments: ADD(_NAMED)_MEMBER and ADD(_NAMED)_CHOICE_VARIANT. These macros are more open-ended and allow for more complex specifications. We have already seen one example of using a macro of this type, in the implementation of the GetTypeInfo() method for CPerson:

The ADD_MEMBER and ADD_CHOICE_VARIANT macros always take at least two arguments:

the internal member (variant) name

the definition of the member's (variant's) type

Depending on the (second) TypeMacro argument, additional arguments may or may not be needed. In this example, the TypeMacro is POINTER, which does require additional arguments. The TypeMacroArgs here specify that m_NextDoor is a pointer to a class type whose C++ name is CPerson.

More generally, the remaining arguments depend on the value of TypeMacro, as these parameters complete the type definition. The possible strings which can occur as TypeMacro, along with the additional arguments required for that type, are given in Table 4.

The ADD_MEMBER macro generates a call to the corresponding ADD_NAMED_MEMBER macro as follows:

Some examples of using the ADD_MEMBER macro are:

Similarly, the ADD_CHOICE_VARIANT macro generates a call to the corresponding ADD_NAMED_CHOICE_VARIANT macro. These macros add type information for the choice object's variants.

Introduction

Run-time information about data types is necessary in several contexts, including:

When reading, writing, and processing serialized data, where runtime information about a type's internal structure is needed

When reading from an arbitrary data source, where data members' external aliases must be used to locate the corresponding class data members (e.g.MyXxx may be aliased as my-xxx in the input data file)

When using a generalized C++ type iterator to traverse the data members of an object

When accessing the object type information per se (without regard to any particular object instance), e.g. to dump it to a file as ASN.1 or DTD specifications (not data)

In the first three cases above, it is necessary to have both the object itself as well as its runtime type information. This is because in these contexts, the object is usually passed inside a generic function, as a pointer to its most base parent type CObject. The runtime type information is needed here, as there is no other way to ascertain the actual object's data members. In addition to providing this information, a runtime type information object provides an interface for accessing and modifying these data members.

In the last case (4) above, the type information is used independent of any actual object instances.

Motivation

We use a simple example to help motivate the use of this type and object information model. Let us suppose that we would like to have a generic function LoadObject(), which can populate an object using data read from a flat file. For example, we might like to have:

where myObj is an instance of some subclass of Object. Assuming that the text in the file is of the form:

we would like to find the corresponding data member in myObj for each MemberName, and set that data member's value accordingly. Unfortunately, myObj cannot directly supply any useful type information, as the member names we seek are for a specific subclass of Object. Now suppose that we have an appropriate type information object available for myObj, and consider how this might be used:

Here, we assume that our type information object, info, stores information about the memory offset of each data member in myObj, and that such information can be retrieved using some sort of identifying member name such as myName. This is not too difficult to imagine, and indeed, this is exactly the type of information and facility provided by the C++ Toolkit's type information classes. The FindMember() function just needs to return a void pointer to the appropriate location in memory. The AssignValue() function presents a much greater challenge however, as its two sole arguments are a void pointer and a string. This would be fine if the data member was indeed a void pointer, and a string value was acceptable. In general this is not the case, and stronger methods are clearly needed.

In particular, for each data member encountered, we need to retrieve the type of that member as well as its location in memory, so as to process myValue appropriately before assigning it. In addition, we need safer mechanisms for making such "untyped" assignments. Ideally, we would like a FindMember() function that returns a correctly cast pointer to that data member, along with its associated type information. This is what the object information classes provide - a pointer to the object instance as well as a pointer to its static type information. The interface to the object information class also provides a number of methods such as GetClassMember(), GetTypeFamily(), SetPrimitiveValue(), etc., to support the type of activity described above.

Object Information Classes

The following topics are discussed in this section:

• CObjectTypeInfo (*)

• CConstObjectInfo (*)

• CObjectInfo (*)

Usage of object type information

We can now reconsider how our LoadObject() function might be implemented using the CObjectInfo class:

Here, info contains pointers to the CObject itself as well as to its associated CTypeInfo object. For each member alias read from the file, we apply FindClassMember(alias), and dereference the returned iterator to retrieve a CObjectInfo object for that member. We then use the operator () to verify that the member was located, and if so, use the member's CObjectInfo to set a value in the function SetValue():

In this example, SetValue() can only assign primitive types. More generally however, the CObjectInfo class allows the assignment of more complex types that are simply not implemented here. Note also that the arguments to SetValue() are const, even though the function does modify the value of the data instance pointed to. In particular, the type const CObjectInfo should not be confused with the type CConstObjectInfo. The former specifies that object information construct is non-mutable, although the instance it points to can be modified. The latter specifies that the instance itself is non-mutable.

In addition to user-specific applications of the type demonstrated in this example, the generic implementations of the C++ type iterators and the CObject[IO]Streamclass methods provide excellent examples of how runtime object type information can be deployed.

As a final example of how type information might be used, we consider an application whose simple task is to translate a data file on an input stream to a different format on an output stream. One important use of the object classes defined in include/objects is the hooks and parsing mechanisms available to applications utilizing CObject[IO]Streams. The stream objects specialize in different formats (such as XML or ASN.1), and must work in concert with these type-specific object classes to interpret or generate serialized data. In some cases however, the dynamic memory allocation required for large objects may be substantial, and it is preferable to avoid actually instantiating a whole object all at once.

Instead, it is possible to use the CObjectStreamCopier class, described in CObject[IO]Streams. Briefly, this class holds two CObject[IO]Stream data members pointing to the input and output streams, and a set of Copy methods which take a CTypeInfo argument. Using this class, it is easy to translate files between different formats; for example:

copies a CBioseq_set encoded in XML to a new file, reformatted in ASN.1 binary format.

Introduction

The datatool program processes the ASN.1 specification files (*.asn) in the src/objects/ directories to generate the associated C++ class definitions. The corresponding program implemented in the C Toolkit, asntool, used the ASN.1 specifications to generate C enums, structs, and functions. In contrast, datatool must generate C++ enums, classes and methods. In addition, for each defined object type, datatool must also generate the associated type information method or function.

There is a significant difference in how these two tools implement ASN.1 choice elements. As an example, consider the following ASN.1 specification:

The ASN.1 choice element specifies that the corresponding object may be any one of the listed types. In this case, the possible types are an integer and a string. The approach used in asntool was to implement all choice objects as ValNodes, which were in turn defined as:

The DataVal field is a union, which may directly store numerical values, or alternatively, hold a void pointer to a character string or C struct. Thus, to process a choice element in the C Toolkit, one could first retrieve the choice field to determine how the data should be interpreted, and subsequently, retrieve the data via the DataVal field. In particular, no explicit implementation of individual choice objects was used, and it was left to functions which manipulate these elements to enforce logical consistency and error checking for legitimate values. A C struct which included a choice element as one of its fields merely had to declare that element as type ValNode. This design was further complicated by the use of a void pointer to store non-primitive types such as structs or character strings.

In contrast, the C++ datatool implementation of choice elements defines a class with built-in, automatic error checking for each choice object. The usage of CObject class hierarchy (and the associated type information methods) solves many of the problems associated with working with void pointers.

C++ choice objects

The classes generated by datatool for choice elements all have the following general structure:

For the above ASN.1 specification, datatool generates a class named CObject_id, which is derived from CObject. For each choice variant in the specification, an enumerated value (in E_Choice), and an internal typedef are defined, and a declaration in the union data member is made. For this example then, we would have:

In this case both of the choice variants are C++ built-in types. More generally however, the choice variant types may refer to any type of object. For convenience, we refer to their C++ type names here as "CXxx",

Two private data members store information about the currently selected choice variant: m_choice holds the enum value, and m_Xxx holds (or points to a CObject containing) the variant's data. The choice object's member functions provide access to these two data members. Which() returns the currently selected variant's E_Choice enum value. Each choice variant has its own Get() and Set() methods. Each GetXxx() method throws an exception if the variant type for that method does not correspond to the current selection type. Thus, it is not possible to unknowingly retrieve the incorrect type of choice variant.

Select(e_Xxx) uses a switch(e_Xxx) statement to initialize m_Xxx appropriately, sets m_choice to e_Xxx, and returns. Two SetXxx() methods are defined, and both use this Select() method. SetXxx() with no arguments calls Select(e_Xxx) and returns m_Xxx (as initialized by Select()). SetXxx(TXxx& value) also calls Select(e_Xxx) but resets m_Xxx to value before returning.

Some example choice objects in the C++ Toolkit are:

• CDate

• CInt_fuzz

• CObject_id

• CPerson_id

• CAnnotdesc

• CSeq_annot

Locating the Class Definitions

In general, traversing through a class object requires that you first become familiar with the internal class structure and member access functions for that object. In this section we consider how you can access this information in the source files, and apply it. The example provided here involves a Biostruc type which is implemented by class CBiostruc, and its base (parent) class, CBiostruc_Base.

The first question is: how do I locate the class definitions implementing the object to be traversed? There are now two source browsers which you can use. To obtain a synopsis of the class, you can search the index or the class hierarchy of the Doc++ browser and follow a link to the class. For example, a synopsis of the CBiostruc class is readily available. From this page, you can also access the relevant source files archived by theLXR browser, by following the Locate CBiostruc link. Alternatively, you may want to access the LXR engine directly by using the Identifier search tool.

Because we wish to determine which headers to include, the synopsis displayed by the Identifier search tool is most useful. There we find a single header file, Biostruc.hpp, listed as defining the class. Accordingly, this is the header file we must include. The CBiostruc class inherits from the CBiostruc_Base class however, and we will need to consult that file as well to understand the internal structure of the CBiostruc class. Following a link to the parent class from the class hierarchy browser, we find the definition of the CBiostruc_Base class.

This is where we must look for the definitions and access functions we will be using. However, it is the derived user class (CBiostruc) whose header should be #include'd in your source files, and which should be instantiated by your local program variable. For a more general discussion of the relationship between the base parent objects and their derived user classes, see Working with the serializable object classes.

Accessing and Referencing Data Members

Omitting some of the low-level details of the base class, we find the CBiostruc_Base class has essentially the following structure:

With the exception of the structure's chemical graph, each of the class's private data members is actually a list of references (pointers), as specified by the type definitions. For example, TId is a list of CRef objects, where each CRef object points to a CBiostruc_id. The CRef class is a type of smart pointer used to hold a pointer to a reference-counted object. The dereferencing operator, when applied to a (dereferenced) iterator pointing to an element of CBiostruc::TId, e.g. **CRef_i, will return a CBiostruc_id. Thus, the call to GetId() returns a list which must then be iterated over and dereferenced to get the individual CBiostruc_id objects. In contrast, the function GetChemicalGraph() returns the object directly, as it does not involve a list or a CRef.

NOTE: It is strongly recommended that you use type names defined in the generated classes (e.g. TId, TDescr) rather than generic container names (list< CRef<CBiostruc_id> > etc.). The real container class may change occasionally and you will have to modify the code using generic container types every time it happens. When iterating over a container it's recommended to use ITERATE and NON_CONST_ITERATE macros.

The GetXxx() and SetXxx() member functions define the user interface to the class, providing methods to access and modify ("mutate") private data. In addition, most classes, including CBiostruc, have IsSetXxx() and ResetXxx() methods to validate and clear the data members, respectively.

Traversing a Biostruc

The program traverseBS.cpp (see Box 4) demonstrates how one might load a serial data file and iterate over the components of the resulting object. This example reads from a text ASN.1 Biostruc file and stores the information into a CBiostruc object in memory. The overloaded Visit() function is then used to recursively examine the object CBiostruc bs and its components.

Visit(bs) simply calls Visit() on each of the CBiostruc data members, which are accessed using bs.GetXxx(). The information needed to write each of these functions - the data member types and member function signatures - is contained in the respective header files. For example, looking at Biostruc_.hpp, we learn that the structure's descriptor list can be accessed using GetDescr(), and that the type returned is a list of pointers to descriptors:

Consulting the base class for CBiostruc_desc in turn, we learn that this class has a choice state defining the type of value stored there as well as the method that should be used to access that value. This leads to an implementation of Visit(CBiostruc::TDescr DescrList) that uses an iterator over its list argument and a switch statement over the current descriptor's choice state.

Iterating Over Containers

Most of the Visit() functions implemented here rely on standard STL iterators to walk through a list of objects. The general syntax for using an iterator is:

Dereferencing the iterator is required, as the iterator behaves like a pointer that traverses consecutive elements of the container. For example, to iterate over the list of descriptors in the Biostruc, we use a container of type CBiostruc::TDescr, and an iterator of type const_iterator to ensure that the data is not mutated in the body of the loop. Because the descriptor list contains pointers (CRefs) to objects, we will actually need to dereference twice to get to the objects themselves.

In traversing the descriptor list in this example, we handled each type of descriptor with an explicit case statement. In fact, however, we really only visit those descriptors whose types have string representations: TName, TPdb_comment, and TOther_comment. The other two descriptor types, THistory and TAttribute, are objects that are "visited" recursively, but the associated visit functions are not actually implemented (see Box 5, traverseBS.hpp).

The NCBI C++ Toolkit provides a rich and powerful set of iterators for various application needs. An alternative to using the above switch statement to visit elements of the descriptor list would have been to use an NCBI CStdTypeIterator that only visits strings. For example, we could implement the Visit function on a CBiostruc::TDescr as follows:

In this example, the iterator will skip over all but the string data members.

The CStdTypeIterator is one of several iterators which makes use of an object's type information to implement the desired functionality. We began this section by positing that the traversal of an object requires an a priori knowledge of that object's internal structure. This is not strictly true however, if type information for the object is also available. An object's type information specifies the class layout, inheritance relations, data member names, and various other attributes such as size, which are independent of specific instances. All of the C++ type iterators described in The NCBI C++ Toolkit Iterators section utilize type information, which is the topic of the next section: Runtime Object Type Information.

SOAP message

The core section of the SOAP specification is the messaging framework. The client sends a request and receives a response in a form of a SOAP message. A SOAP message is a one-way transmission between SOAP nodes: from a SOAP sender to a SOAP receiver. The root element of a SOAP message is the Envelope. The Envelope contains an optional Header element followed by a mandatory Body element. The Body element represents the message payload - it is a generic container that can contain any number of elements from any namespace.

In the Toolkit, the CSoapMessage class defines Header and Body containers. Serializable objects (derived from the CSerialObject class) can be added into these containers using AddObject() method. Such message object can then be sent to a message receiver. The response will also come in a form of an object of CSoapMessage class. At this time, it is possible to investigate its contents using GetContent() method; or ask directly for an object of a specific type using SOAP_GetKnownObject() template function.

SOAP client (CSoapHttpClient)

SOAP client is the initial SOAP sender - a node that originates a SOAP message. Knowing SOAP receiver's URL, it sends a SOAP message request to it and receives a response using Invoke() method.

Internally, data objects in the Toolkit SOAP library are serialized and de-serialized using serializable objects streams. Since each serial data object also provides access to its type information, writing such objects is a straightforward operation. Reading the response is not that transparent. Before actually parsing incoming data, SOAP processor should decide what object type information to use. Hence, a client application should tell the SOAP processor what types of data objects it might encounter in the incoming data. If the processor recognizes the object's type, it will parse it and store as a correct data object of this specific type. Otherwise, the processor will parse the data into an object of CAnyContentObject class.

So, a SOAP client must define the server's URL and register (using RegisterObjectType() method), object types which might be present in incoming data. Other functions encapsulate valid operations for a given Web server.

SOAP server (CSoapServerApplication)

SOAP server is an ultimate SOAP receiver. It receives SOAP messages from a client, and is responsible for processing the contents of the SOAP Body and SOAP Header.

The processing of incoming requests is being done with the help of "message listeners" -the server methods which analyze requests in the form of objects of CSoapMessage class and create a response. It is possible to have more than one listener for each message. When such listener returns TRUE, the SOAP server base class object passes the request to the next listener, if it exists, and so on.

To give server the ability to return WSDL specification the name of the specification file should be provided in the SOAP server constructor, and the file should be deployed alongside the server.

Sample SOAP server and client

The Toolkit contains a simple example of SOAP client and server in its src/app/sample/soap folder.

The sample SOAP server supports the following operations:

GetDescription() - server receives an empty object of type Description, and it sends back a single string;

GetVersion() - server receives a string, and it sends back two integer numbers and a string;

DoMath() - server receives a list of Operand objects (two integers and an enumerated value), and it sends back a list of integers

The starting point is the WSDL specification - src\app\sample\soap\server\soap_server_sample.wsdl

Both client and server use data objects whose types are described in the message types section of WSDL specification. So, we extract the XML schema part of the specification into a separate file, and create a static library - soap_dataobj. All code in this library is generated automatically by .