Cryptographic methodology

For PPRL linkage projects, two related but distinct encoding mechanisms are used, depending on how a field is configured in the project definition file:

Hash fields

Fields configured as a simple hash are transformed using a keyed hash — this is an HMAC using the SHA-2 or SHA-3 family. A keyed hash (as opposed to an unkeyed hash) means that the output cannot be matched against a rainbow table or dictionary of raw values without possession of the key, and different key holders using different keys cannot compare hash outputs directly — this is deliberate, since only data for the same intended linkage project should be able to produce comparable outputs.

Bloom filter fields

Fields configured for Bloom filter encoding (typically higher-value identifiers such as name, address and suburb, where approximate/fuzzy matching is needed) are encoded using n-gram (bigram) based Bloom filters, where each bigram of the field value is hashed (again using a keyed HMAC) into a set of bit positions within a fixed-length bit vector. This allows two records to be compared for similarity without either party seeing the original values. Parameters such as filter length and hashes-per-bigram are configurable per linkage project.

Key file and key management

Both hash and Bloom filter transforms require a symmetric key, which is distributed via a key file. This key file is password protected and is distributed to all parties preparing data for the same linkage project. The key file is never distributed to the linkage unit.

Key file property	Description
KeyType	Describes what the file is (e.g. "LinXmart Envelope Builder Key File").
Version	Version of the key file format.
Project	The LinXmart-defined code identifying the linkage project the key is for.
Data	AES-GCM encrypted random byte array (the underlying key material), decrypted at runtime using a PBKDF2-derived key from the operator's password.

The mechanics are as follows:

1. The key file is a JSON text file containing a KeyType label, a format Version, a Project code, and a Data field.
2. The Data field is a randomly generated array of bytes, encrypted at rest using AES-GCM, with the encryption key derived from an operator-supplied password using PBKDF2 (password-based key derivation).
3. The password itself is never stored in the key file. It must be communicated and stored separately by the operator, using their own secure channel/password management practice — the security of the key file at rest is entirely dependent on the strength and protection of this password.
4. At runtime, the operator supplies the password; Envelope Builder uses it to decrypt the Data field in memory. This decrypted value, combined with each field's name, derives a unique per-field key used for that field's hash or Bloom filter transform — meaning every field gets a distinct effective key, but the same key file/password combination will reproduce the same result deterministically across all datasets in the project, which is required for records to be linkable.
5. A non-reversible fingerprint of the HMAC hash key in use is included in the Envelope's metadata. This allows a downstream quality check to confirm the same key was used across datasets without the key or password ever being exposed.

Key file creation and distribution

The key file is created by one of the participating data providers and shared with all other participating data providers, together with the password required to unlock it with Envelope Builder.

warning

The key file and password are never shared with the linkage unit running the LinXmart record linkage system that will link the datasets together.

The Envelope Builder App provides a mechanism to create a new key file, using a randomly generated array of bytes and a user chosen password. Alternatively, an existing key file can be loaded from disk using the appropriate password.

Creation of the key file is tasked to one of the data providers. Distribution of this key file should only occur via secure file transfer mechanisms to the other data providers.