Module 1 - De-duplication of a single file
In this Module, you will learn how to create a simple linkage project in LinXmart and how to undertake a de-duplication of a single (small) dataset. The Module will introduce you to some simple concepts in linkage and will help you gain confidence in operating the LinXmart software.
This module is part of a series of training modules that can be used with any LinXmart deployment. However, the user will require permissions to create Linkage Projects, load data for linkage extract linkage map results.
All required data files are available from the main self guideded training page.
Getting started
Step 1: Connect to the LinXmart web interface by typing in the LinXmart web address. You will see a screen similar to this:
Step 2: Ensure that your installation of LinXmart has a Data Provider called TRAINDP. To do this, click on the CATALOG tab at the top of the page.
If this Data Provider is not displayed, please follow the setup instructions for this series of training modules before continuing.
Creating a Linkage Project
Before data can be linked, we first define our project and include some metadata in the system – i.e. information about the data to be used and the linkages we wish to conduct.
Step 3: Click on the PROJECTS tab at the top of the web page.
This screen lists all the Linkage Projects that have been created in the system. We will create a new Linkage Project.
Step 4: Click on the Add Project link at the bottom of the page.
Enter the Linkage Project details in the fields provided. For the purposes of this Module, we will use XX_DEATH_DEDUP.
As the value for Code must be unique in the system, change XX_ to something else (e.g. your initials) to create your Linkage Project. Click on SAVE when you have finished (see below).
Adding data to a Linkage Project
Step 5: Your XX_DEATH_DEDUP project details page should be displayed.
You are presented with an empty project (as shown below).
Before data can be loaded, one or more Event Types need to be attached to the Project.
Step 6: Click on the Attach some event types link presented to you below the record stats for the empty project.
Click on the Event Types field.
You will be presented with a list of Event Types that are already known to (defined in) LinXmart.
Check the tickbox next to Data Provider TRAINDP and Event Type TDP_DEATH (TDP_DEATH) and then click OK. If there are a lot of Event Types in the system, use the Search field to filter the list.
Click the Attach button to add the selected Event Type to the project.
The list of Event Types for your project should now look like this:
Linking your first file
We have now added sufficient information into LinXmart to perform a simple linkage – a de-duplication or internal linkage of a single file.
We will upload the TDP_DEATH_clear.csv dataset found in the Training Data folder called TRAINDP Clear text datasets. You may wish to view this file in a text editor to see its contents. It is a comma delimited file containing personal identifiers and a unique record identifier – all records are synthetic. The format of this file (called it’s Import Format) has already been defined within the system.
Step 7: If you are not already on your Project dashboard, navigate to it (PROJECTS --> select XX_DEATH_DEDUP).
Then click on Settings. Then Data Sources. This should take you to your Event Type Data Sources page (see below).
Click the Load icon on the TDP_DEATH row on this page.
Step 8: Click on the Choose File icon and select the TDP_DEATH_clear.csv file from the relevant Data folder.
Click Load. This may take a minute while the dataset is uploaded to the LinXmart server from your local machine.
You will be taken to a Load Data Confirmation page. Please ignore the Group by JLK option for now and select Load. This will begin the linkage process; its process can be monitored in real-time through the ACTIVITY tab.
Monitoring your linkage job
Step 9: To see how the linkage job is progressing, navigate to the Job Monitoring screen by clicking on the ACTIVITY tab at the top of the screen. Next select the ENVELOPES sub-tab. Information regarding the current job's status will be appear under ‘All Envelopes’.
The data in this file was internally linked or de-duplicated, identifying records within the file belonging to the same person. As there were no other datasets in the Project, this is the only linkage that took place. A green tick is the Status column indicates that the job successfully ran to completion (green is good).
Step 10: Click on the PROJECTS tab and select XX_DEATH_DEDUP Project to go back to your Linkage Project dashboard.
The dashboard for the XX_DEATH_DEDUP Project will now reflect the new data, showing you the records that were added, pairs created and groups (unique individuals) that were found (shown below).
Step 11: In the Data Load History section, click on the report icon in the Options column for the linkage you just ran (shown below)
The Linkage Report details information on the changes made to the system for this particular Envelope, including storing a snapshot of the configuration used.
You can also click on the Notifications icon to the left of the Report icon to view other notifications that were sent out as part of the process.
This concludes the first Module. Well done!
Extension Exercise
As an additional step, navigate to the Quality Review screen to look at any quality issues that the system may have found with your linkage.
To do this, navigate to your project on the Project dashboard, and click on the Quality Review icon on the right-hand side (shown below).
Did any records break the Authoritative rule of Death records? What might you do next?
Take your time to consider the information provided.
Module 3 may assist you with the additional steps in the Quality Review stage.