►Data Exchange> Import Batch
►Data Exchange> Import Log
Note this feature is an optional component for data imports.
In Data Exchange> Import Batch, in modify mode, this button appears below the Files to Import list. The function extracts all of the reference data tags from the OLSTF file and populates the Reference Data Mappings list below, with the best guess mappings where it can.
This results in a list such as:
Tag, Data | Association | Domain | Matched Value | Notes |
---|---|---|---|---|
CCLOAN | Import Batch | Copy Categories | 58 occurrences | |
CC REF | Import Batch | Copy Categories | Resell Copy | 43 occurrences; Longest Match COPYCAT |
CCRESL | Import Batch | Copy Categories | Resell Copy | 18 occurrences; Longest Match COPYCAT |
CC STAND | Import Batch | Copy Categories | Standard | 3108 occurrences; Uppercased COPYCAT |
CZ AVL | Import Batch | Copy Status | Available | 2 occurrences; Uppercased COPYST |
CTTP ST | Import Batch | Currencies | Swiss franc | 1 occurrences; Longest Match CURRENCY |
The matching process is carried out as follows:
For tags that map to a reference data key ID, the process first tests the reference data table by matching the upper cased incoming data with the key ID (eg: MAIN for System Location). This is what the normal OLSTF import does and is therefore used as the first match to minimise the potential for introducing unnecessary changes to the import as a whole. If this matches, then that reference data record is chosen and no further lookups are carried out for this value. This can be seen above for CZ AVL. Also shown for this data value is that it was found twice in the data (2 occurrences) and confirmation that it matched the COPYST key value when uppercased.
If this does not match, then the process continues as follows
For description tags (and unmatched key IDs), the process first checks for an exact match on the reference data Long Description field. If this does not match, it then looks for a match without regard to upper or lower case (eg: “system location” will match “System Location”).
If there is still no match, then the process will look for the longest match it can find in the reference data, without regard to upper or lower case.
If there is still no match, then the process will repeat the above with the Short Description field and finally the key value.
CC REF | Import Batch | Copy Categories | Resell Copy | 43 occurrences; Longest Match COPYCAT |
CCRESL | Import Batch | Copy Categories | Resell Copy | 18 occurrences; Longest Match COPYCAT |
The screenshot above is for the incoming data values CC REF and CC RESL. It shows that the process did not match by an upper case key value for any of the Copy Categories, but instead matched the first 2 and 3 characters of a long description, respectively.
Where the reference data matches more than one value equally, the alternate values are listed in the Notes.
Key ID | 1291 | |
---|---|---|
Tag and Data | CPTP | ST |
Domain | Currencies | |
Change Mapping | ||
Mapped To | CHF | Swiss franc |
Description/Notes | 1 occurrences; Longest Match CURRENCYLD: 1; South Korean Won; Sri Lanka Rupee; Saudi Riya | |
Import Batch | 4 |
In this example the incoming value ST, having not matched a currency code, continued to match the longest description available. In this case it matched only the first character of several currencies. To remedy this for example:
Without this process, if the reference data did not match, then the import will not generate valid data. This could present in a variety of ways depending on the tag in question, for example:
Saving the above record will mean that, for this import, the data value ST in the CPTP tag will be changed to GBP as the import is carried out.
This applies to both OLSTF and MARC Import Batches.
A mapping is picked up if it is assigned to either the Import Batch or the Record Source associated with the Import Batch. In this fashion, a mapping can be made more generic so that it will apply to other Import Batches that use the same record source.
The mapping can be made more generic by either removing the Import Batch number from the mapping record manually on its edit layout or by selecting several mappings and selecting the Make Generic action.
The Copy Result action is used as a shortcut to simplify the mapping of several values to a single result.
In order to use this action, select one mapping with the desired result and one or more mappings for the same domain that show no result, for example:
CCLOAN | Import Batch | Copy Categories | 58 occurrences | |
CC STAND | Import Batch | Copy Categories | Standard | 3108 occurrences; Uppercased COPYCAT |
Using the Copy Result action with the selection above will map the unmapped tag to Standard.
The process effectively handles one file at a time. However, the mappings can remain present and and therefore in effect for all future files processed for this Import Batch and/or Record source.
A list of the tags and the associated reference data domains are provided below. Tags in bold, by default, match on Key IDs as opposed to descriptions.
ADTP | Address Types |
ATP | Address Types |
TT | Alternative Title Types |
TTLD | Alternative Title Types |
BTC | Binding Types |
BTD | Binding Types |
CS | Classmark Schemas |
XS | Control No. Types |
CC | Copy Categories |
CCLD | Copy Categories |
CZ | Copy Statuses |
COURSE | Courses |
COURSED | Courses |
CPTP | Currencies |
PRTP | Currencies |
DEPT | Departments |
DEPTD | Departments |
EXST | Extraction Statuses |
FMC | Formats |
FMD | Formats |
TFRLD | Frequencies |
INST | Institutions |
ISTP | ISxN Types |
LA | Languages |
LALD | Languages |
LATP | Languages |
TLATP | Languages |
LT | Link Types |
LTLD | Link Types |
LS | Loan Status Reference Data |
LL | Locations |
LO | Locations |
LOC | Locations |
LOLD | Locations |
PICKLOC | Locations |
ME | Media Sub-Types |
CMTLD | Media Types |
MT | Media Types |
MTLD | Media Types |
MTTP | Media Types |
SEMT | Media Types |
NT | Note Types |
NTD | Note Types |
OST | Object Statuses |
OT | Object Types |
OTD | Object Types |
OCCUP | Occupations |
OG | Organisation Types |
SUPTP | Publisher/Supplier Types |
RESRC | Record Source Codes |
SURST | [ Record Statuses ] |
AR | Responsibility Types |
ARLD | Responsibility Types |
MR | Responsibility Types |
PUBP1 | Serial Publication Patterns |
PUBP2 | Serial Publication Patterns |
STP | Series Types |
SA | Shelves |
SC | Shelves |
SCLD | Shelves |
SUBTP | Subject Types |
SUBTPD | Subject Types |
SUPST | Supplier Statuses |
SERST | Title/Series Statuses |
SHST | Title/Series Statuses |
TISER | Title/Series Statuses |
CAT | User Categories |
TP | User Types |
UTP | User Types |
During the import, if you have SQL*Plus installed on the pc and know how to use it, the following SQL may be run for a Titles import. Re-running on a regular basis will give the status of the import.
select mediatp, count(*) from titles group by mediatp;
When the batch has loaded, you can see a summary of the load:
In the Data Exchange> Import Log domain you can search by the Batch No. This is useful for example to check over when you have loaded a small trial batch. You can clear the records out manually prior to running subsequent batches.
If the import fails, entries are made in this domain to indicate the reasons for the failure. Search for import log records by batch number, review the reasons for failure, rectify the problems and repeat the whole process to re-import the records. It is important to check the import log as even batches which load correctly may have some records that have failed.
Note: data imports pick up where they left off in the event of failure.
This domain lists all record imports regardless of the type of import. You can search for the batch record by description, e.g. 'w' would retrieve WorldCat record download: automated use only. Searches filtered by Batch Type are also useful.
The Import Batch record shows the details of each batch loaded in Files to Import. You can also make amendments for any future batches as required. The second sheet of the Import Batch layout is the MARC Summary. This is useful in identifying how data is being handled in this batch.
If the OLIB database is shut down, then the import will pick up from where it left off automatically. If however, the Normalise Server Programme was stopped (e.g. the application server was rebooted), then the import will stop. This could also happen if the database and application server are on the same machine and it is rebooted, because the database might restart the import before the Normalise Server has started up.
When a batch import is set to run and the server closes down overnight, the import may not automatically restart. The File to Import record (the Object) shows how many records were processed. If there are more records to import, but the import process has stopped, you can restart it by simply re-running the Load Data action for the same Import Batch Object record:
It is possible that a problem is caused by one specific record, rather than a restart of the server. In this situation you can adjust the value in Records Loaded and rerun the Load Data action for the same Import Batch Object record to skip over the problem record(s):
The import starts again from the point you specified.
Existing bibliographic records in the catalogue can be updated or replaced by running an OLSTF import. This requires the system manager to configure 2 additional attributes on the Import Batch domain for the OLSTF layout:
Use this drop down to select the key that you want to match the records on. Currently you can match on the following OLSTF tags:
BC | Copy Barcode |
---|---|
IS | ISxN |
INVIS | Invalid ISxN value |
RN | Record Number. Data in this tag imports to / exports from the Record Control Number field in OLIB. Sometimes referred to as “Import RCN” or just “RCN” . |
TNUPD | Title import record control number. Refers to the Title Number. It is used for imports where the incoming data is intended to supplement the existing records. For example, when there are additional copies. |
XC | Title control number |
Use this drop down to choose whether you want to the OLSTF import to replace matching records or update matching records:
OLSTF: Add Copies for matching records | This will match an existing title record and process only the copy data, adding the copy to the title (provided there is no clash with the barcode). If an incoming record does not match an existing title, nothing of the incoming record will be processed |
---|---|
OLSTF: Add new; Ignore matching records | This option will only add new title and copy records. If the incoming record matches an existing title, nothing of the incoming record will be processed |
OLSTF: Add new; Add Copies for matching record | This option will add new title and copy records, but will also add copies for matching records as per the first option above OLSTF: Add new: Replace matching records |
OLSTF: Add new; Update matching records |
Replace or update records:
Before running record imports it is possible to configure an Import Batch record so that when records are imported through it, the imported records are automatically added to a Folder to facilitate any review/amendment processing.
The Import Batch record can also be configured so that matched items are automatically added to a different Folder for viewing.
Procedure
The System Manager should add one or both of these attributes to the Import Batch layout using Layout Manager:
When the import is run, OLIB will populate the relevant Folder(s) with imported MARC records.
During the MARC import, OLIB attempts to match on the author records. If the import process produces too many author duplicate records, you can run the de-duplication process on the relevant records.
The author matching process helps to achieve accurate author matching during imports and to prevent duplicate author records if possible.
During normal usage, you do not normally need to visit or maintain this information. However, it could be useful to ascertain why an imported record has mismatched an existing author. You can then amend the match value to avoid future mismatches.
During the import, Author initials are not generated if:
During imports, OLIB generates and maintains a list of match values against each author. You can view these on the Names layout: Controls sheet - Import Matching.
In Import Matching, you can view any existing match values - if there has been a previous import since OLIB 9 Service Pack 12:
Match Value, Source, First Use, Last Use, Modified By
For any future imports, the Match Value is the value that will be used in preference for matching. The import processes will first attempt to match exactly and then attempt a case insensitive match on just the letters, numbers and spaces in the field.
You can see who the new value was generated by and when. In Source where an OCLC#, Import RCN or Batch number is shown, this match value was generated by an import.
If the import author matching does not find a successful match for an author, then OLIB matches on a value generated from the existing author record. To view what this would be:
Note: for MARC imports, the match value will be generated from all but the Relator Term and Relator Code subfields. The subfield codes other than $a will be included in the match value.
Note: for OLSTF imports, the match value will be generated as Forenames Initials Surname for personal names, as seen when generated by New above. In addition, there is an OLSTF import setting that permits the dates associated with the author to be ignored as part of the matching. If this is then enabled, the search will be carried out as a secondary match after the generation of the match value with dates, and a match value including dates will be added to the new or existing author record.