Bulk load of multiple data sets
Triggering the load of many data sets at the same time can be done from the Bulk Load Data link on the Linkage Project page, below the list of Event Types.
Creating a Workflow
This operation is always run as a Worfklow, which can be run immediately or scheduled to occur periodically.
The bulk load of data can only be used for linkages and extractions! Other types of data (such as deletions) must be loaded individually.
This function gives you the ability to not just load multiple datasets, but optionally extract linkage results when linkage has completed.
You are presented with a form where you will need to choose:
- whether to combine all of the data sources into a single linkage
- the name of Envelope(s) to put the linkage(s) in
- one or more data sources to load data from
- (optionally) an Extraction Project to extract linkage results
- (optionally) a recurring schedule in which to run this workflow
If you select the Combine For Linkage option, only one Envelope will be created; all data sources selected will be loaded first, before the matching and grouping of the data will occur.
The Combine For Linkage option is great for small, periodic and incremental loads of data.
An single Extraction Project can also be selected; this determines if linkage results should be extracted on completion of the linkages. Data will be extracted for every Event Type included in the list of selected Data Sources.
Only Extraction Projects that are configured for enduring linkage keys will be available to choose from.
The data loads (with the subsequent linkages) and optional linkage key extractions will all be run within the context of a single Worfklow.
Clicking on the Load button without checking the Run on Schedule option will create a new Link and Extract Workflow to be queued to run immediately.
Scheduling
Instead of running the workflow immediately, it can be scheduled to run periodically:
Check the Run on Schedule option, give the schedule a name and choose the recurrence required. Frequency options are:
- Daily
- Weekly
- Monthly
- Custom
Daily, weekly and monthly options cover most scenarios. However, the custom option allows you to specify a standard 5 part Cron Expression, giving you complete control over when the workflow is run.
If you select to run on a schedule, clicking on the Load button will create a new schedule to for the bulk load selection. Otherwise, a new Worfklow will be queued to run immediately.