Library usage

This page should give you a high-level introduction to the LinXmart Envelope Builder library and how it can be used to transform raw datasets into a format that can be used by LinXmart for linkage.

High level requirements for the Envelope Builder integration:

• Conversion of named identifiers to privacy-preserving identifiers
• Privacy-preserving transformations defined by an externally provided configuration (Project Definition File)
• Creation of a LinXmart supported Envelope format (directly or indirectly)

Technical requirements:

• Source data is streamed in
• Output data is streamed out
• Provided as a NuGet package
• Target multiple .NET frameworks
• Support for 32-bit and 64-bit machines
• Minimal memory usage (keep as low as possible)

Overview of the API

There is one main class that forms the high-level API with the Envelope Builder library: EnvelopeDataItemBuilder. This class will take in an IDataReader for the source data, together with a Project Definition and a key to use for cryptographic hashing. It also requires an implementation of the IEnvelopeDataItemStreamFactory interface - this is used to obtain an output stream for each data item in the final "Envelope".

A simple example usage is shown below:

try
{
    using (var streamFactory = new ZipFileStreamFactory(@"D:\Temp\new-envelope.zip"))
    {
        IDataReader dataReader = new DataTableReader(...); // define

        var builder = new EnvelopeDataItemBuilder(streamFactory)
        {
            DataSourceReader = dataReader,
            ProjectDefinition = File.ReadAllText(@"D:\Temp\Synth-config.json"),
            HmacKey = "My secret key"
        };

        builder.Transform();
    }
}
catch (Exception ex)
{
    // Handle known exceptions here
    throw;
}

The FileStreamFactory class used above is an example implementation of the IEnvelopeDataItemStreamFactory interface that writes to an external file.

The files that are created during this process must be combined into a zip file for compatibility with LinXmart.

Field mapping

The Project Definition file contains a list of field transformations that are executed during the Transform process. Each of these fields map to a source field, denoted as a string; it is this value that is used to map to a particular column within the IDataReader's schema.

For example, the implementation of IDataReader.GetSchemaTable() returns a DataTable with a number of columns (DataColumn). The ColumnName property on each DataColumn should correspond to one or more field transforms within the Project Definition. If one of the field transforms in the Project Definition file has no corresponding DataColumn in the source data, the output of that value will be empty for every source row.

EnvelopeDataItemBuilder class

Namespace: CDL.NLS.Envelope.Builder

The main high-level API class used to transform named identifiers into a privacy-preserved "Envelope" for use with the LinXmart system.

public class EnvelopeDataItemBuilder

Constructors

Constructor	Description
EnvelopeDataItemBuilder(IEnvelopeDataItemStreamFactory)	Initialises a new instance of EnvelopeDataItemBuilder with an implementation of IEnvelopeDataItemStreamFactory.

Properties

Property	Description
DataSourceReader	An IDataReader instance that provides the source data stream. This must implement the GetSchemaTable() method with all data columns specified.
ProjectDefinition	A JSON formatted string containing the project definition, as provided from the LinXmart system. Configuration and transformation specifics are detailed within this document.
HmacKey	A key used by the privacy-preserving transformation process. For two privacy-preserved Envelopes to be linked together, they must use the same key.
IncludeSourceFieldMetadata	Default value = false. A flag that indicates whether metadata on the source fields should be gathered and returned as a data item. Note: this will require the source data to be fully loaded into memory for the metadata calculation.
IncludeEnvelopeDefinition	Default value = true. A flag that indicates whether the envelope definition is to be included as a data item.
IncludeSystemInfo	Default value = true. A flag that indicates whether system information is to be included as a data item.

Methods

Method	Description
Transform	Reads the data stream from the DataSourceReader property, and transforms the data according to the ProjectDefinition. Calling this method will trigger calls to the instance of the IEnvelopeDataItemStreamFactory passed in the constructor to obtain output streams.

Remarks

The EnvelopeDataItemBuilder class is the main class for the high-level API of the Envelope Builder library. It will take a source data stream and a Project Definition (configuration) and transform the source data into a series of files that make up an "Envelope" for use in LinXmart.

Supported types for the IDataReader

The following system types (including nullable versions) are supported for the column types in the IDataReader's schema:

• short
• int
• long
• float
• double
• decimal
• bool
• string
• char
• DateTime

If a column has a type not in the list above, a NotSupportedException will be thrown.

Exceptions

The Transform method may throw exceptions if there are issues with the object's property values.

Exception	Cause
InvalidProjectDefinitionException	The project definition JSON is incorrectly formatted or the wrong version.
InvalidOperationException	Thrown if the object is not in the correct state, which may be due to: a missing or empty property; or a transformation that specifies a type that does not exist or cannot be instantiated.
InvalidDataException	Something is invalid with the IDataReader: it is closed; there is no schema; or the ordinal, column name, or data type columns cannot be found in the schema.
NotSupportedException	One of the columns in the IDataReader is of a type that is not supported.

IEnvelopeDataItemStreamFactory interface

Namespace: CDL.NLS.Envelope.Builder

A means for the client of the Envelope Builder library to provide output stream instances for every Envelope data item created during the transformation process.

public interface IEnvelopeDataItemStreamFactory

Methods

Method	Description
CreateDataItemStream(string)	Returns an instance of Stream for a data item with the specified name. The cleanup of the stream will be managed by the envelope builder.

Example 1 - Save to folder

The following class implements the IEnvelopeDataItemStreamFactory interface to create file-based output streams to a designated file path.

class FileStreamFactory : IEnvelopeDataItemStreamFactory
{
    private readonly string _outputFolder;

    public FileStreamFactory(string outputFolder)
    {
        _outputFolder = outputFolder;
    }

    public Stream CreateDataItemStream(string itemName)
    {
        var filePath = Path.Combine(_outputFolder, itemName);
        return new FileStream(filePath, FileMode.CreateNew);
    }
}

Example 2 - Save to zip

The following class implements the IEnvelopeDataItemStreamFactory interface to create a zip file at a specific location. The System.IO.Compression.ZipArchive class is used for this.

class ZipFileStreamFactory : IEnvelopeDataItemStreamFactory, IDisposable
{
    private readonly FileStream _zipFileStream;
    private readonly ZipArchive _zipArchive;

    public ZipFileStreamFactory(string outputPath)
    {
        _zipFileStream = new FileStream(outputPath, FileMode.Create,
                                        FileAccess.Write);
        _zipArchive = new ZipArchive(_zipFileStream, ZipArchiveMode.Create);
    }

    public Stream CreateDataItemStream(string itemName)
    {
        var dataEntry = _zipArchive.CreateEntry(itemName);
        return dataEntry.Open();
    }

    public void Dispose()
    {
        _zipArchive.Dispose();
        _zipFileStream.Dispose();
    }
}

Envelope Builder package

The Envelope Builder API is packaged and made available through NuGet packages.

Target frameworks:

• .NET Framework 4.8
• .NET 8.0
• .NET 10.0
• .NET Standard 2.0

The following CDL packages are not publicly available and are provided directly under license:

• CDL.NLS.Envelope.Builder (main package)
• CDL.NLS.Envelope
• CDL.Privacy

There are also some third party packages required (depending on the target framework), available via the public NuGet source (https://www.nuget.org/packages):

• Portable.BouncyCastle
• Newtonsoft.Json