RESTful API in Evergreen

Access is permitted to the OpenAPI server for those Evergreen user accounts that have been promoted to an Integrator account, and have had an additional password with the new api password type associated with them. These accounts do not need to have a known main password, nor do they need to have login type permissions other than API_LOGIN. Such accounts are effectively API-only accounts, and while they may be given some set of staff-like permissions in order to access necessary patron and transaction data, they do not need to have staff client or other elevated access to Evergreen.

Today, promotion of an account to be an Integrator is managed using the api_ctl tool. The api_ctl command path to add a user as an Integrator is integrator → add → USERNAME and the command path to set the password for Integrator API login is integrator → load → USERNAME → password. This can be accomplished in a single command directly from the command line:

$ api_ctl --command integrator add USERNAME password

which will then prompt for the API password to assign to the newly promoted user. A newly added Integrator is enabled by default. However, an Integrator promotion can be disable, re-enabled, or removed similarly:

$ api_ctl --command integrator disable USERNAME

$ api_ctl --command integrator enable USERNAME

$ api_ctl --command integrator remove USERNAME

More details on the integrator control functionality of the api_ctl tool can be found below.

While it is possible to use any valid authentication token to request methods through the OpenAPI server, including those created through OPAC and Staff Client login, logging in through the OpenAPI server itself using the /self/auth path is additionally protected and the user must meet the following minimum requirements:

These restrictions are imposed by the open-ils.auth OpenSRF application, and are intended to prevent accidentally allowing OpenAPI access. A caveat to the foregoing is noted below, to facilitate convenient patron access to the OpenAPI server.

Rate of access to each endpoint can be restricted by the use of Rate limit definitions. These simple rules consist of a limit interval and a limit count, where there must be no more than limit count requests over the preceding limit interval time.

Each endpoint request is evaluated in the context of the user making the request and their IP address, the Rate limit definition attached to the endpoint or the default fallback limit, any Rate limit definitions attached to the endpoint set into which the endpoint is placed, and any Rate limit definitions that combine the user and IP address with the endpoint or relevant endpoint set.

If any user or IP address related Rate limit definitions are configured for the request, the most restrictive of those is applied before allowing access. If no user or IP address related Rate limit definitions are configured, the most restrictive of the general endpoint and endpoint Set limits is used.

Access to each API method is restricted based on the Evergreen permissions assigned to the user that has been promoted to Integrator. For flexibility and ease of configuration, these access permissions can be grouped into permission sets. These permission sets are then assigned to endpoints and endpoint set, described below, and the Integrator must have all of the permissions in at least one of the permission sets in effect for an endpoint in order to successfully call the API method.

For example, there are two stock permission sets associated with the self endpoint set, which includes the primary authentication API method located at the /self/auth path and methods that allow user self-service functionality.

One set, the Self - standard permissions permission set, includes just the OPAC_LOGIN permission. This allows normal patron-type users that have been promoted to Integrator and have been given an API password to log into the OpenAPI server using Basic authentication. This user will log in with that separate password to access the self-service methods, or use the URL parameter login to supply a login type of "opac" with their main username and password credentials. Another set, the Self - API only permission set, contains the new API_LOGIN and REST.api permissions and allows login and self-service requests only for Integrator users that have both of those new permission. They do not need to have the OPAC_LOGIN permission.

As a more complicated example, there exists a stock API method at the path /patron/:userid which returns detailed user information on the user with an ID specified at the :userid placeholder. This method is in the patron-oriented patron endpoint set which is protected by default using two permission sets, one containing the API_LOGIN, REST.api, and REST.api.patrons permissions, and the other containing the STAFF_LOGIN and VIEW_USER permissions. Additionally, an administrator may associate the endpoint with an endpoint-specific permission set, named Custom patron retrieval for this example, containing REST.api and REST.api.patrons.detail.read permissions.

Thus, there are three permission sets in effect for the /patron/:userid endpoint:

The first permission set would be congruent with a normal Evergreen staff account accessing the OpenAPI server with a Staff Client authentication token, the second with an Integrator account that has been granted general patron-related API access, and the third with an Integrator account that has been granted specific access to that one API endpoint by having the REST.api.patrons.detail.read permission, but does not have general patron-related REST.api.patrons permission.

If an authenticated account accessing this API has all of the permissions in any of those sets, then the API’s logic is allowed to execute. Users are assigned permissions in the normal Evergreen way, by having the permission added to their account directly or by being made a member of a primary or secondary permission group which has the permission assigned to it.

Once the permission to execute the API’s logic has been established, the requesting account’s normal permissions must still be applied to any underlying action. In this example, if the backend method that retrieves patron record detail requires the VIEW_USER permission be granted to the requestor at the home library of the patron record, that must still hold in order for the API to return the requested data.

Evergreen’s OpenAPI Server exposes specific business logic functions and data access pathways as API endpoints.

Each of these endpoints present a single backend function, usually a way to request particular data or initiate a specific action, by translating OpenAPI paths, HTTP methods, and HTTP headers and parameters to handler functions and their necessary parameters.

These handler functions may be either OpenSRF service and method pairs, or when some amount of pre- or post-processing of an OpenSRF method call is required, a Perl module and optional sub name that supplies any necessary additional logic.

Once the handler has run, the requested output is translated to an OpenAPI response, normally as JSON or XML formatted data. The requests may make use of parameter data passed via HTTP path components or query parameters, HTTP cookies or headers, or the HTTP request body.

Endpoints can be thought of as OpenAPI wrappers to existing Evergreen functionality implemented in OpenSRF.

Structurally, each endpoint consists of the following information:

These endpoints can then be gathered together into logical groups, called endpoint sets, to simplify access configuration and documentation. Endpoint sets can supply permission sets and rate limit definitions for endpoints that they contain. For permission sets, if an endpoint is in more than one endpoint set, all mapped permission sets are applied to the endpoint and successful authorization against any of the permission sets will allow access.

Rate limit definition selection is more complicated, but fully predictable.

First, all definitions related to the relevant endpoint sets (and the specific endpoint) that are attached with the context of the user or their IP address, if any, are compared and the strictest rate limit definition is applied. If there are no user or IP address contextual rate limit definitions in place, the strictest rate limit definition attached to the endpoint or any of its endpoint set is applied to the request, first against the user for the endpoint if the user is known, then against the IP address for the endpoint if the IP address is known, or finally, globally against the endpoint.

The content of these settings is a comma-separated list of values, which are made up of the Fieldmapper class hint and the specific Fieldmapper property to be addressed. For example, to change the visibility of the Date of Birth column from the actor.usr table, one would add au.dob to the appropriate setting value, where "au" is the Fieldmapper class hint for the class representing the actor.usr table in the database, and "dob" is the Fieldmapper property representing, and column name, which holds the date of birth data.

Whitelisting one or more properties on a Fieldmapper class will cause all other properties to be redacted by setting their values to null. Blacklisting one or more properties will cause only those named properties to be redacted in this way.

Library settings are evaluated in the context of the Integrator user’s home library, and are inheritable from parent locations in the same manner as all other Library settings. Two Library settings are for API-global use:

User settings can also be used to add Integrator-specific whitelist and blacklist rules. To apply API-global, Integrator-specific properties restrictions, two new stock User Setting Types are available, with the same name and function as the Library settings above. Because these are user-specific, administrators can provide specific Integrator accounts access to more data with more whitelisted properties, or further restrict access by adding additional blacklist properties.

In addition to the API-global Library and User settings, administrators can create new Organizational Unit Setting Types (AKA Library Setting Types) and User Setting Types through the Server Administration interface. Setting types must be named following a specific pattern. The setting types must start with the string REST.api.whitelist_properties. or REST.api.blacklist_properties. for whitelisting or blacklisting respectively — note the period — and end with the endpoint’s Operation ID. For example, if an administrator would like to restrict the output of the searchPatron endpoint so that it only returns the idof the patron, and the client is expected to retrieve the full patron record separately, they could configure following:

User Setting Types can be created through the Server Administration interface, as well. However, because there is no Staff Client interface for general User Setting maintenance, the api_ctl tool, discussed below, must be used to configure Integrator-specific whitelist and blacklist rules. Integrator-specific setting types are created automatically by api_ctl if they do not already exist for a particular endpoint, so administrators are not required to create User Setting Types by hand when using the api_ctl tool.

Both whitelist and blacklist settings can be used at the same time, though the one primary case for this would be to add a property to a more specific blacklist when it is already present on a general whitelist. This may be useful in the case where general API access should allow retrieval of a particular piece of information, but a specific integration should not provide this information because it is likely to be visible through some third party interface if it is delivered to the client application.

This output property restriction mechanism depends upon Evergreen’s knowledge of the Integrator account making the request. In practice this means that all endpoints with the exception of the primary authentication endpoint, which only delivers an authentication token, will be protected using this feature. However, if local, custom endpoints are configured for an Evergreen instance that do not use the built-in security mechanisms, they cannot make use of this property restriction feature.

There are several standard options available at most levels of the menu hierarchy:

There are common commands available for many objects that can be controlled through the api_ctl interface. Generally, objects can be listed, added, edited, and removed with menu options that are:

You can see all options available at the current level of the option hierarchy by pressing the tab key.

The layout of the menu hierarchy when started without command line options is as follows:

Obtaining an Evergreen auth token is accomplished by sending an HTTP GET request to the /self/auth path with the appropriate credentials.

The credential parameter names are described in the API specification document. They can be passed using either

Using the standard HTTP content negotiation Accept header, the client can ask for the token to be delivered as either plain text, which is useful for tasks like direct testing and high-level shell scripting, or as JSON data, which is the default and is usually better for use by actual client applications.

The output of this request, which is a standard Evergreen auth token used by all authenticated client code in Evergreen, is then used in a standard HTTP Bearer authentication header to identify the session in later requests. For ease of testing and some added flexibility, the auth token may also be passed in the URL as a query parameter called ses, or in either of the modern Evergreen-standard cookies called eg.auth.token and ses, or the new, Evergreen OpenAPI-specific cookie called eg.api.token.

Administrators can install Swagger-UI visualization tools so that developers can see the list of endpoints. These tools make use of the API specification document to assist with authorization and help the user authenticate with the API.

The web-based Swagger-UI editor and visualization tools are not robust or sophisticated enough to handle such a large and complex component schema. This is a limitation of the basic demo Swagger tools. True client applications do not try to render the full JSON Schema, are written to be robust and correct, and are not expected to have these sorts of issues if they are designed well.

Evergreen’s OpenAPI support establishes a framework for clients to access and manipulate Evergreen data as well as a set of predefined endpoints. Consequently, additional endpoints are expected to be added over time. Like any API, the definitions of endpoints are subject to change, especially as the number of clients using the endpoints grows. An API version stamp in the path component of the base URL will be used to signal whether breaking changes have been introduced in the API. The initial release of the API sets the version as v0 (i.e., https://<hostname>/openapi3/v0). This will get incremented whenever backwards-incompatible changes get made to existing endpoint or if changes to core Fieldmapper IDL classes could break clients.

Using the Evergreen API Explorer, we can see that the open-ils.actor application provides several methods that may suit our purpose here.

The parameter documentation is not great for any of these, but we can see where to find the underlying code by expanding the block.

The implementation for open-ils.actor.user.retrieve_id_by_barcode_or_username is in the OpenILS::Application::Actor Perl module, in the sub named retrieve_usr_id_via_barcode_or_usrname. Using this information, we can see that the parameters expected by the OpenSRF method are an authentication token, an optional barcode, and an optional username.

We also see that the method will return either an numeric user id, or an ILS Event object upon error or permission restriction.

Using this information, we can immediately provide RESTful OpenAPI endpoints to return a user id by either barcode or username simply by wrapping the OpenSRF method directly. We can either insert the necessary endpoint configuration directly into the database, or use the api_ctl tool to configure the new endpoints.

Likewise, we can use just a small amount of additional code to create an endpoint to return the full user object in the same format as is returned by the /patron/:id endpoint path. As this new method will be a stock endpoint, we will add it to the built-in OpenILS::OpenAPI::Controller::patron Perl module, but it can live anywhere that the Perl interpreter can find modules.

The OpenILS::OpenAPI::Controller module namespace contains many endpoint examples and helper methods that are useful for the creation of OpenAPI endpoints.

The parameters passed to the handler functions are exactly those that are defined for the OpenAPI endpoint, via its parameter list, following an invocant passed in the first parameter position. The effective invocant is the active Mojolicious Controller object. Extensive [documentation on the Mojolicious Controller is available with the Mojolicious Perl module.

And, as before, we can then register the new endpoints with the OpenAPI server either using direct SQL or the api_ctl tool.

The OpenAPI server must be restarted once any new endpoint configuration is applied. This allows the OpenAPI server to read the new endpoint configuration and add the appropriate routes and handlers.