Manage OLSTF imports

Reference data validation

Note this feature is an optional component for data imports.

In Data Exchange> Import Batch, in modify mode, this button appears below the Files to Import list. The function extracts all of the reference data tags from the OLSTF file and populates the Reference Data Mappings list below, with the best guess mappings where it can.

This results in a list such as:

Tag, Data	Association	Domain	Matched Value	Notes
CCLOAN	Import Batch	Copy Categories		58 occurrences
CC REF	Import Batch	Copy Categories	Resell Copy	43 occurrences; Longest Match COPYCAT
CCRESL	Import Batch	Copy Categories	Resell Copy	18 occurrences; Longest Match COPYCAT
CC STAND	Import Batch	Copy Categories	Standard	3108 occurrences; Uppercased COPYCAT
CZ AVL	Import Batch	Copy Status	Available	2 occurrences; Uppercased COPYST
CTTP ST	Import Batch	Currencies	Swiss franc	1 occurrences; Longest Match CURRENCY

The matching process is carried out as follows:

For tags that map to a reference data key ID, the process first tests the reference data table by matching the upper cased incoming data with the key ID (eg: MAIN for System Location). This is what the normal OLSTF import does and is therefore used as the first match to minimise the potential for introducing unnecessary changes to the import as a whole. If this matches, then that reference data record is chosen and no further lookups are carried out for this value. This can be seen above for CZ AVL. Also shown for this data value is that it was found twice in the data (2 occurrences) and confirmation that it matched the COPYST key value when uppercased.

If this does not match, then the process continues as follows

For description tags (and unmatched key IDs), the process first checks for an exact match on the reference data Long Description field. If this does not match, it then looks for a match without regard to upper or lower case (eg: “system location” will match “System Location”).

If there is still no match, then the process will look for the longest match it can find in the reference data, without regard to upper or lower case.

If there is still no match, then the process will repeat the above with the Short Description field and finally the key value.

CC REF	Import Batch	Copy Categories	Resell Copy	43 occurrences; Longest Match COPYCAT
CCRESL	Import Batch	Copy Categories	Resell Copy	18 occurrences; Longest Match COPYCAT

The screenshot above is for the incoming data values CC REF and CC RESL. It shows that the process did not match by an upper case key value for any of the Copy Categories, but instead matched the first 2 and 3 characters of a long description, respectively.

Where the reference data matches more than one value equally, the alternate values are listed in the Notes.

Click on the record Tag link, through to the Reference Data Mappings detail to demonstrate this more clearly:

Key ID	1291
Tag and Data	CPTP	ST
Domain	Currencies
Change Mapping
Mapped To	CHF	Swiss franc
Description/Notes	1 occurrences; Longest Match CURRENCYLD: 1; South Korean Won; Sri Lanka Rupee; Saudi Riya
Import Batch	4

In this example the incoming value ST, having not matched a currency code, continued to match the longest description available. In this case it matched only the first character of several currencies. To remedy this for example:

Switch the Reference Data Mapping record into modify to manually edit.
From Change Mapping select a new currency from the drop down, e.g. GBP Pound Sterling.

Without this process, if the reference data did not match, then the import will not generate valid data. This could present in a variety of ways depending on the tag in question, for example:

An invalid Location may prevent a copy from being created
An invalid Copy Category will import the copy, but it cannot be shown in the application as, amongst other things, the application cannot determine which layout to utilise
An invalid currency will not present any errors, but will simply assign the price without showing any currency

Saving the above record will mean that, for this import, the data value ST in the CPTP tag will be changed to GBP as the import is carried out.

Scope of the mapping: the Association column

This applies to both OLSTF and MARC Import Batches.

A mapping is picked up if it is assigned to either the Import Batch or the Record Source associated with the Import Batch. In this fashion, a mapping can be made more generic so that it will apply to other Import Batches that use the same record source.

The mapping can be made more generic by either removing the Import Batch number from the mapping record manually on its edit layout or by selecting several mappings and selecting the Make Generic action.

Copy Result action

The Copy Result action is used as a shortcut to simplify the mapping of several values to a single result.

In order to use this action, select one mapping with the desired result and one or more mappings for the same domain that show no result, for example:

CCLOAN	Import Batch	Copy Categories		58 occurrences
CC STAND	Import Batch	Copy Categories	Standard	3108 occurrences; Uppercased COPYCAT

Using the Copy Result action with the selection above will map the unmapped tag to Standard.

Handle several files on the Import Batch

The process effectively handles one file at a time. However, the mappings can remain present and and therefore in effect for all future files processed for this Import Batch and/or Record source.

Reference data domains

A list of the tags and the associated reference data domains are provided below. Tags in bold, by default, match on Key IDs as opposed to descriptions.

OLSTF Tag and Domain

ADTP	Address Types
ATP	Address Types
TT	Alternative Title Types
TTLD	Alternative Title Types
BTC	Binding Types
BTD	Binding Types
CS	Classmark Schemas
XS	Control No. Types
CC	Copy Categories
CCLD	Copy Categories
CZ	Copy Statuses
COURSE	Courses
COURSED	Courses
CPTP	Currencies
PRTP	Currencies
DEPT	Departments
DEPTD	Departments
EXST	Extraction Statuses
FMC	Formats
FMD	Formats
TFRLD	Frequencies
INST	Institutions
ISTP	ISxN Types
LA	Languages
LALD	Languages
LATP	Languages
TLATP	Languages
LT	Link Types
LTLD	Link Types
LS	Loan Status Reference Data
LL	Locations
LO	Locations
LOC	Locations
LOLD	Locations
PICKLOC	Locations
ME	Media Sub-Types
CMTLD	Media Types
MT	Media Types
MTLD	Media Types
MTTP	Media Types
SEMT	Media Types
NT	Note Types
NTD	Note Types
OST	Object Statuses
OT	Object Types
OTD	Object Types
OCCUP	Occupations
OG	Organisation Types
SUPTP	Publisher/Supplier Types
RESRC	Record Source Codes
SURST	[ Record Statuses ]
AR	Responsibility Types
ARLD	Responsibility Types
MR	Responsibility Types
PUBP1	Serial Publication Patterns
PUBP2	Serial Publication Patterns
STP	Series Types
SA	Shelves
SC	Shelves
SCLD	Shelves
SUBTP	Subject Types
SUBTPD	Subject Types
SUPST	Supplier Statuses
SERST	Title/Series Statuses
SHST	Title/Series Statuses
TISER	Title/Series Statuses
CAT	User Categories
TP	User Types
UTP	User Types

Check progress

Once the data import batch has completed a confirmation message is displayed
During the dataload the status of the batch will be: Loading
When the batch load has been completed the status of the batch will change to: Loaded
If not, the import process displays: Failed

During the import, if you have SQL*Plus installed on the pc and know how to use it, the following SQL may be run for a Titles import. Re-running on a regular basis will give the status of the import.

select mediatp, count(*) from titles
group by mediatp;

When the batch has loaded, you can see a summary of the load:

In the Import Batch record click the link in Files to Import.
This loads the Object record with the details of the file.
Click Related Items and Control Data to view the number of records Processed and the number of Records Loaded.

Import Log

In the Data Exchange> Import Log domain you can search by the Batch No. This is useful for example to check over when you have loaded a small trial batch. You can clear the records out manually prior to running subsequent batches.

If the import fails, entries are made in this domain to indicate the reasons for the failure. Search for import log records by batch number, review the reasons for failure, rectify the problems and repeat the whole process to re-import the records. It is important to check the import log as even batches which load correctly may have some records that have failed.

Note: data imports pick up where they left off in the event of failure.

Import Batch domain

This domain lists all record imports regardless of the type of import. You can search for the batch record by description, e.g. 'w' would retrieve WorldCat record download: automated use only. Searches filtered by Batch Type are also useful.

The Import Batch record shows the details of each batch loaded in Files to Import. You can also make amendments for any future batches as required. The second sheet of the Import Batch layout is the MARC Summary. This is useful in identifying how data is being handled in this batch.

Restart a dataload

If the OLIB database is shut down, then the import will pick up from where it left off automatically. If however, the Normalise Server Programme was stopped (e.g. the application server was rebooted), then the import will stop. This could also happen if the database and application server are on the same machine and it is rebooted, because the database might restart the import before the Normalise Server has started up.

When a batch import is set to run and the server closes down overnight, the import may not automatically restart. The File to Import record (the Object) shows how many records were processed. If there are more records to import, but the import process has stopped, you can restart it by simply re-running the Load Data action for the same Import Batch Object record:

Go to the Import Batch domain and search on the Batch No to display the batch record.
Click the check box on the File to Import and in the Actions box select Load Data.

It is possible that a problem is caused by one specific record, rather than a restart of the server. In this situation you can adjust the value in Records Loaded and rerun the Load Data action for the same Import Batch Object record to skip over the problem record(s):

Go to the Import Batch domain and search on the Batch No to display the batch record.
Click on the File to Import to display the Object details.
Go to the Related Items and Control Data sheet. Note the number of records already processed and modify record.
In order to restart the dataload from the right place, enter the number of records it should start off from again in Records Loaded.
Save and Close the Object record and go back to the Import Batch record.
Click the check box on the File to Import and in the Actions box select Load Data.

The import starts again from the point you specified.

Import replace and update

Existing bibliographic records in the catalogue can be updated or replaced by running an OLSTF import. This requires the system manager to configure 2 additional attributes on the Import Batch domain for the OLSTF layout:

Match Flag
Modify Flag

Match flag

Use this drop down to select the key that you want to match the records on. Currently you can match on the following OLSTF tags:

BC	Copy Barcode
IS	ISxN
INVIS	Invalid ISxN value
RN	Record Number. Data in this tag imports to / exports from the Record Control Number field in OLIB. Sometimes referred to as “Import RCN” or just “RCN” .
TNUPD	Title import record control number. Refers to the Title Number. It is used for imports where the incoming data is intended to supplement the existing records. For example, when there are additional copies.
XC	Title control number

Modify flag

Use this drop down to choose whether you want to the OLSTF import to replace matching records or update matching records:

OLSTF: Add Copies for matching records	This will match an existing title record and process only the copy data, adding the copy to the title (provided there is no clash with the barcode). If an incoming record does not match an existing title, nothing of the incoming record will be processed
OLSTF: Add new; Ignore matching records	This option will only add new title and copy records. If the incoming record matches an existing title, nothing of the incoming record will be processed
OLSTF: Add new; Add Copies for matching record	This option will add new title and copy records, but will also add copies for matching records as per the first option above OLSTF: Add new: Replace matching records
OLSTF: Add new; Update matching records

Replace or update records:

Use the procedure as normal for an OLSTF data import.
When creating the Import Batch record choose the relevant field to match records on using Match Flag.
Choose whether to replace or update records using the Modify Flag.
Complete the data import as normal.

View imported and matched records

Before running record imports it is possible to configure an Import Batch record so that when records are imported through it, the imported records are automatically added to a Folder to facilitate any review/amendment processing.

The Import Batch record can also be configured so that matched items are automatically added to a different Folder for viewing.
Procedure

The System Manager should add one or both of these attributes to the Import Batch layout using Layout Manager:

New Items Folder (mb_new_folder)
Matched Items Folder (mb_match_folder)

Create one or both new Folders in advance for viewing the imported records and the matched records.
These should be created by the same System Manager user who goes on to create the Import Batch record and whose name is present in the Import Batch: Data Entered By field.
When setting up the Import Batch record it should include the relevant Folder name(s) in the New Items Folder drop down and the Matched Items Folder drop down.

When the import is run, OLIB will populate the relevant Folder(s) with imported MARC records.

Matches on Author

During the MARC import, OLIB attempts to match on the author records. If the import process produces too many author duplicate records, you can run the de-duplication process on the relevant records.

The author matching process helps to achieve accurate author matching during imports and to prevent duplicate author records if possible.

During normal usage, you do not normally need to visit or maintain this information. However, it could be useful to ascertain why an imported record has mismatched an existing author. You can then amend the match value to avoid future mismatches.

Author initials generation

During the import, Author initials are not generated if:

your OLIB system does not include the initials field on any authors layout
the OLIB Defaults> Prevent Generation of Author Initials = Yes

Match values

During imports, OLIB generates and maintains a list of match values against each author. You can view these on the Names layout: Controls sheet - Import Matching.

Author match process

The author matching process first checks to see if ORCid, VIAF or ISNI are present on the incoming record. If one of these match an existing author, any further processing is ignored.
If not present, the import will generate a Match Value from the incoming data and attempt to match this with a Match Value for an existing author.
If no match is found by Match Value, the import will attempt to match on the surname, forename, initials etc... in the author record itself.

If a match is found, the process adds Match Values to the author it has matched with
If no match is found, the process creates a new author record in OLIB, with the generated Match Value

If this causes a duplicate record, you can edit the Match Values in the Authority Control Details: Import Matching field, for your preferred author record. You can then run de-duplication (the match values will be copied over to ensure that future imports with the same incoming data will match accordingly).

View a match value

In Import Matching, you can view any existing match values - if there has been a previous import since OLIB 9 Service Pack 12:

Go to Cataloguing> Names and search for the author.
Go to Authority Control Details> Import Matching. For the current author, this will show information such as:

Match Value, Source, First Use, Last Use, Modified By

For any future imports, the Match Value is the value that will be used in preference for matching. The import processes will first attempt to match exactly and then attempt a case insensitive match on just the letters, numbers and spaces in the field.

You can see who the new value was generated by and when. In Source where an OCLC#, Import RCN or Batch number is shown, this match value was generated by an import.

Value generated by existing author record

If the import author matching does not find a successful match for an author, then OLIB matches on a value generated from the existing author record. To view what this would be:

Go to Cataloguing> Names and search for the author.
In Authority Control Details> Import Matching, click New. OLIB generates a Match Value based on the author record.
You can manually edit the Match Value in the list box, or it can be removed if you check the box and click the Delete link below the list. You can also click New to create an additional match value. (The import process will try to match with the first value and if not an exact match, then try the next value).

Note: for MARC imports, the match value will be generated from all but the Relator Term and Relator Code subfields. The subfield codes other than $a will be included in the match value.

Note: for OLSTF imports, the match value will be generated as Forenames Initials Surname for personal names, as seen when generated by New above. In addition, there is an OLSTF import setting that permits the dates associated with the author to be ignored as part of the matching. If this is then enabled, the search will be carried out as a secondary match after the generation of the match value with dates, and a match value including dates will be added to the new or existing author record.