Batch Importing MARC Records

Introduction

The cataloging module includes a MARC Batch Import interface for loading MARC (and MARCXML) records. In general, it can handle batches up to 5,000 records without a problem. This interface allows you to specify matchpoints between incoming and existing records, to specify MARC fields that should be overlaid or preserved, and to only overlay records if the incoming record is of higher quality than the existing record. Records are added to a queue where you can apply filters that enable you to generate any errors that may have occurred during import. You can print, email, or export your queue as a CSV file.

You can control whether you are presented with the options for Bibliographic import, Acquisitions Import, or both by setting the Library Setting ACQ: Which import tab(s) display in general Import/Export?. Accepted setting values are: "cat" for Import for Cataloging, "acq" for Import for Acquisitions, "both" or unset to display both. The examples below show the Import for Cataloging option. The Import for Acquisitions option embeds the Load MARC Order Records interface, which is separately documented.

Permissions

Permissions to use this interface include:

  • IMPORT_MARC - Using batch importer to create new bib records requires the IMPORT_MARC permission. If the permission fails, the queued record will fail import and be stamped with a 'import.record.perm_failure' import error.

  • IMPORT_ACQ_LINEITEM_BIB_RECORD_UPLOAD - This allows interfaces leveraging the batch importer, such as Acquisitions, to create a higher barrier to entry. This permission prevents users from creating new bib records directly from the ACQ vendor MARC file upload interface.

  • CREATE_BIB_IMPORT_QUEUE - Permits creating an import queue for bibliographic records.

  • CREATE_AUTHORITY_IMPORT_QUEUE - Permits creating an import queue for authority records.

  • ADMIN_IMPORT_MATCH_SET - Permits the use of match sents in record import.

Record Display Attributes

This feature enables you to specify the tags and subfields that will display in records that appear in the import queue. You can set different attributes for bibliographic records and authority records.

Evergreen is configured with standard defaults for these attributes.

Record Display Attributes

Record Match Sets

This feature enables you to create custom match points that you can use to accurately match incoming records with existing catalog records.

Creating a Match Set

In this example, to demonstrate matching on record attributes and MARC tags and subfields, we will create a record match set that defines a match based on the title of the record, in either the 240 or 245, and the fixed field, Lang. You can add multiple record attributes and MARC tags to customize a record match set.

  1. Navigate to Cataloging → MARC Batch Import/Export.

  2. Create a new record match set. Select Record Match Sets → New Match Set.

  3. Enter a name for the record match set.

  4. Select an Owning Library from the drop down menu. Staff with permissions at this location will be able to use this record match set.

  5. Select a Match Set Type from the drop down menu. You can create a match set for authority records or bibliographic records.

  6. Click Save.

    New Record Match Set

    The screen will refresh to list the record match set that you created. From the Name column, select the link to your record match set.

Next, create an expression that will define the match points for the incoming record. You can choose from two areas to create a match: Record Attribute or MARC Tag and Subfield. You can use the Boolean operators AND and OR to combine these elements to create a match set.

To use Record Attribute match points:

  1. Select a Record Attribute from the drop-down menu.

  2. Enter a Match Score.

    • The match score indicates the relative importance of that match point as Evergreen evaluates an incoming record against an existing record.

    • You can enter any integer into this field. The number that you enter is only important as it relates to other match points.

    • Recommended practice is that you create a match score of one (1) for the least important match point and assign increasing match points by powers of 2 to indicate match points with increasing importance (e.g., scores of 1, 2, 4, 8).

  3. Select the Negate? checkbox if you want to negate the match point. Selecting this checkbox would be the equivalent of applying a Boolean operator of NOT to the match point.

    Record Attribute

  4. Select an operator from the expression tree to the right.

  5. Select Add to Selected Node. Once this is selected, the tree will refresh and update with the new expression.

If you wish to add another expression, enter another Boolean operator to further refine your match set.

  1. Select Boolean Operator*.

  2. Select the OR operator from the drop down menu.

  3. Select Add to Selected Node. The operator will appear on the tree.

    Operators

    To use MARC Tag and Subfield match points:

  4. Select MARC Tag and Subfield

  5. Enter a MARC tag on which you want the records to match.

  6. Enter a Subfield on which you want the records to match.

  7. Enter a Match Score.

    • The match score indicates the relative importance of that match point as Evergreen evaluates an incoming record against an existing record.

    • You can enter any integer into this field. The number that you enter is only important as it relates to other match points.

    • Recommended practice is that you create a match score of one (1) for the least important match point and assign increasing match points by powers of 2 to indicate match points with increasing importance (e.g., scores of 1, 2, 4, 8).

  8. Select the Negate? checkbox if you want to negate the match point. Selecting this checkbox would be the equivalent of applying a Boolean operator of NOT to the match point.

  9. Select Add to Selected Node. Once this is selected, the tree will refresh and update with the new expression.

    MARC Tag and Subfield

    === Replace Mode ===

Replace Mode enables you to replace an existing part of the expression tree with a new record attribute, MARC tag, or Boolean operator. For example, if the top of the tree is AND, in Replace Mode, you could change that to an OR.

  1. Create a working match point.

  2. Select an expression in the tree to edit.

  3. Make any changes in the form.

  4. Select Replace Selected Node.

    Replacing Nodes

  5. The updated change will display in the tree.

Quality Metrics

Quality metrics are used to determine the overall quality of a record. Each metric is given a weight and the total quality value for a record is equal to the sum of all metrics that apply to that record. For example, a record that has been cataloged thoroughly and contains accurate data would be more valuable than one of poor quality. You may want to ensure that the incoming record is of the same or better quality than the record that currently exists in your catalog; otherwise, you may want the match to fail. The quality metric is optional.

You can create quality metrics based on the record attribute or the MARC Tag and Subfield.

  1. From within the Record Match Sets interface, select the Match Set Quality Metrics tab.

  2. Select the Record Attribute button.

  3. Select a Record Attribute from the drop down menu.

  4. Enter a Value for the attribute.

  5. Enter a Match Score.

    • You can enter any integer into this field. The number that you enter is only important as it relates to other quality values for the current configuration. Higher scores would indicate increasing quality of incoming records.

    • Recommended practice is that you create a match score of one (1) for the least important match point and assign increasing match points by powers of 2 to indicate match points with increasing importance (e.g., scores of 1, 2, 4, 8).

  6. Select Add.

    Match Set Quality

    You can add quality metrics based on data in MARC fields and subfields as well.

Merge/Overlay Profiles

If Evergreen finds a match for an incoming record in the database, you need to identify which fields should be replaced, which should be preserved, and which should be added to the record.

Select Merge/Overlay Profiles to create a profile that contains this information.

You can use these profiles when importing records through the MARC Batch Importer or Acquisitions Load MARC Order Records interface.

You can create a new profile by selecting the New Bib Import Merge Profile button. Available options for handling the fields include:

  • Preserve specification - fields in the existing record that should be preserved.

  • Replace specification - fields in existing record that should be replaced by those in the incoming record.

  • Add specification - fields from incoming record that should be appended to existing record.

  • Remove specification - fields that should be removed from incoming record.

  • Update bib source - If this value is false, just the bibliographic data will be updated when you overlay a new MARC record. If it is true, then Evergreen will also update the record’s bib source to the one you select on import; the last edit date to the date the new record is imported; and the last editor to the person who imported the new record.

You can add multiple tags to the specification options, separating each tag with a comma.

Import Item Attributes

If you are importing items with your records, you will need to map the data in your MARC holdings tag to fields in the item record. This interface allows you to specify subfield codes or XPATH queries to define where item data lives.

Select Holdings Import Profiles to map this information.

  1. Select the New Import Item Attribute Definition button to create a new mapping for the holdings tag.

  2. Enter a Name for the definition.

  3. Use the Tag field to identify the MARC tag that contains your holdings information.

  4. Add the subfields that contain specific item information to the corresponding item field.

    • At a minimum, you should add the subfields that identify the Circulating Library, the Owning Library, the Call Number, and the Barcode.

    • See Table 1, below, for more information about Item Attributes.

All fields (except for Name and Tag) can contain a MARC subfield code (such as "a") or an XPATH query. You can also use the related library settings to set defaults for some of these fields.
New Import Item Attribute Definition
Table 1. Holdings Item Attribute Definitions
Field Recommended Description

Alert Message

No

This field is deprecated

Barcode

Yes

Call Number

Yes

Circulate

Circulate As MARC Type

Copy Number

Definition ID

Deposit

Deposit Amount

Holdable

Name

Yes

Name you will choose from the MARC Batch Import screen

OPAC Visible

Overlay Match ID

The copy ID of an existing item to overlay

Owner

Owning Library

Yes

Parts Data

Of the format PART LABEL 1|PART LABEL 2.

Price

Private Note

Public Note

Reference

Shelving Location

Stat Cat Data

Of the format CATEGORY 1|VALUE 1||CATEGORY 2|VALUE 2. If you are overlaying existing items which already have stat cats attached to them, the overlay process will keep those values unless the incoming items contain updated values for matching categories.

Status

Tag

Yes

MARC Holdings Tag/Field (e.g. 949). Use the Tag field to identify the MARC tag that contains your holdings information.

Additionally, Evergreen can create an item-level record for each copy. If an item barcode, call number, shelving location, or circulation modifier is not set in the embedded holdings, Evergreen will apply a default value based on the configured Library Settings. A default prefix can be applied to the auto-generated call numbers and item barcodes.

See the section below on Default Values for Item Import.

Import Records

The Import Records interface incorporates record match sets, quality metrics, more merging options, and improved ways to manage your queue.

Fields below are generally listed in the order of top to bottom, left to right as they appear in the interface.

Importing Records interface

First, set your import parameters in the MARC File Upload section of the interface.

  1. Select a Record Type from the drop down menu.

  2. Select a Record Source from the drop down menu.

  3. Select a Record Match Set from the drop down menu.

  4. Select a specific bucket in Limit Matches to Bucket if desired.

  5. Select a Holdings Import Profile if you want to import holdings that are attached to your records.

  6. Select Import Non-Matching Records if you want records to be imported from your file even if they don’t have a match.

  7. Select a Merge Profile. Merge profiles enable you to specify which tags should be removed or preserved in incoming records.

  8. Select an Insufficient Quality Fall-Through Profile if desired.

    • This field enables you to indicate that if the inbound record does not meet the configured quality standards, then you may still import the record using an alternate merge profile.

    • This field is typically used for selecting a merge profile that allows the user to import holdings attached to a lower quality record without replacing the existing (target) record with the incoming record.

  9. Enter a Best/Single Match Minimum Quality Ratio.

    • Divide the incoming record quality score by the record quality score of the best match that might exist in the catalog. By default, Evergreen will assign any record a quality score of 1 (one).

    • If you want to ensure that the inbound record is only imported when it has a higher quality than the best match, then you must enter a ratio that is higher than 1. For example, if you want the incoming record to have twice the quality of an existing record, then you should enter a 2 (two) in this field. If you want to bypass all quality restraints, enter a 0 (zero) in this field.

      Quality ratio affects only the Merge on Single Match and Merge on Best Match options.

  10. Choose one of the following import options if you want to auto-import records:

    1. Merge on Exact Match (901c) - Use if you’re reloading exported records from the system to match on the record ID.

    2. Merge on Single Match - Using the Record Match Set, Evergreen will attempt to perform the merge/overlay action if one and only one match was found in the catalog.

    3. Merge on Best Match - If more than one match is found in the catalog for a given record, Evergreen will attempt to perform the merge/overlay action with the best match as defined by the match score and quality metric.

  11. Make an Optional Session Name if desired.

  12. Select Auto-overlay In-process Acquisitions Items if you want to overlay temporary copies that were created by the Acquisitions module. The system will attempt to overlay copies where:

    • Copies have associated line item details (i.e., they were created by the acquisitions process); AND

    • Said line item detail has the same owning_lib as the incoming copy’s owning_lib; AND

    • The current copy associated with that line item detail is in the status In process.

  13. Select a Remove MARC Field Groups set, if applicable.

    • Select Auto-overlay On-order Cataloging Items if you wish to overlay your items in On Order status with the items in your import file.

    • Select Use Org Unit Matching in Copy to Determine Best Match if you want the importer to match incoming copies to on order copies in the same Org Unit. This is used in conjunction with Auto-overlay On-order Cataloging Items.

Once you have set your import parameters, you can set your upload parameters.

  1. Enter the name of a new or existing queue to which you can upload your records. Queues are linked to match sets and a holdings import profile. You cannot change a holdings import or record match set for a queue.

  2. Select a file to upload to the queue from your computer using File to Upload. The file can be in either MARC (.mrc) or MARCXML (.xml)format.

  3. If you wish to have the import processed as a background import, select the Request Background Import checkbox and enter an email address for import notifications. See more about background import below.

    Background import options

  4. Select the Upload button to import your file to youe selected queue.

  5. You will see three progress bars: Upload Progress, Enqueue Progress, and Import Progress. Once all three have completed, you will see a button labeled Go To Queue.

The Go To Queue button will take you to the Inspect Queue interface which will show you records that have been uploaded to your queue.

Inspect Queue Interface

Above the grid there are two sections, as well as a set of quick filter options:

  • Queue Summary shows a brief summary of the records and items included in the queue, including counts of import successes and failures for both records and items in the queue.

  • Queue Actions lists common actions for the queue.

    • Export Non-Imported Records will export a MARC file of records that failed to import, allowing those records to be edited as needed and imported separately. These records can be viewed by selecting the Limit to Non-Imported Records filter.)

  • The three checkboxes above the grid are Queue Filters which provide options to limit which records display in the grid.

If Evergreen indicates that matching records exist for a particular record, you will see a hyperlinked numeral in the Matches dolumn. If this record was also not imported, take the following steps to resolve the match:

  1. Select the hyperlinked numeral to view the matching records.

  2. If the record in the queue is not yet imported, lick in the Merge Target column to instruct Evergreen that you want to merge the existing record and the incoming record.

    Select Merge Target

  3. Select Return to Queue, and repeat as necessary for other matched non-imported records.

  4. Once complete, check the boxes of the records that you want to import, and select Import Selected Records

    In this case, it is recommended that you apply the Non-Imported Records filter.

    === Background Import Feature ===

As of VERSION, Evergreen supports a feature to let imports be processed in the background.

If you selected Request Background Import in your queue options, after upload you will see a notification saying Upload Complete for Background Import and a link labeled Background Import Management. This will open a new tab with a grid showing details of all Background Imports associated with your username.

You can also access the Background Import Management page at Cataloging → MARC Barch Import/Export → Background Imports.

image::batch_importing_MARC/background_import_review.png[Background Import Reiew].

This interface shows a grid that includes:

  • A link to inspect the queue(s) marked for background import

  • The owner, workstation, and notification email for the background import.

  • The request time, complete time, and state (new, running, or complete) of the background import.

  • The type of background import - bib = bibliographic import and acq = acquisitions import.

  • Statistics about the number of records and items in the background import, and successes and failures for each.

  • A Parameters column, that includes the import parameter information in the form of a JSON blob.

Default Values for Item Import

The following Library Settings can be configured to apply these default values to imported items:

  • Vandelay: Generate Default Barcodes - Auto-generate default item barcodes when no item barcode is present

  • Vandelay: Default Barcode Prefix - Apply this prefix to any auto-generated item barcodes

  • Vandelay: Generate Default Call Numbers - Auto-generate default item call numbers when no item call number is present

  • Vandelay: Default Call Number Prefix - Apply this prefix to any auto-generated item call numbers

  • Vandelay: Default Copy Location - Default copy location value for imported items

  • Vandelay: Default Circulation Modifier - Default circulation modifier value for imported items

Previous
Next
To report a problem with this documentation or provide feedback, please contact the DIG mailing list.