SIP Server
About the SIP Protocol
SIP, standing for Standard Interchange Protocol, was developed by the 3M corporation to be a common protocol for data transfer between ILS' (referred to in SIP as an ACS, or Automated Circulation System) and a third party device. Originally, the protocol was developed for use with 3M SelfCheck (often abbreviated SC, not to be confused with Staff Client) systems, but has since expanded to other companies and devices. It is now common to find SIP in use in several other vendors' SelfCheck systems, as well as other non-SelfCheck devices. Some examples include:
-
Patron Authentication (computer access, subscription databases)
-
Automated Material Handling (AMH)
-
The automated sorting of items, often to bins or book carts, based on shelving location or other programmable criteria
-
Installing the SIP Server
This is a rough intro to installing the SIP server for Evergreen.
Getting the code
Current SIP server code lives at in the Evergreen git repository:
cd /opt git clone git://git.evergreen-ils.org/SIPServer.git SIPServer
Configuring the Server
-
Type the following commands from the command prompt:
$ sudo su opensrf $ cd /openils/conf $ cp oils_sip.xml.example oils_sip.xml
-
Edit oils_sip.xml. Change the commented out <server-params> section to this:
<server-params min_spare_servers='1' max_spare_servers='2' min_servers='3' max_servers='25' />
-
max_servers will directly correspond to the number of allowed SIP clients. Set the number accordingly, but bear in mind that too many connections can exhaust memory. On a 4G RAM/4 CPU server (that is also running evergreen), it is not recommended to exceed 100 SIP client connections.
Setting the encoding
SIPServer looks for the encoding in the following places:
-
An encoding attribute on the account element for the currently active SIP account.
-
The encoding element that is a child of the institution element of the currently active SIP account.
-
The encoding element that is a child of the implementation_config element that is itself a child of the institution element of the currently active SIP account.
-
If none of the above exist, then the default encoding (ASCII) is used.
Option 3 is a legacy option. It is recommended that you alter your configuration to move this element out of the implementation_config element and into its parent institution element. Ideally, SIPServer should not look into the implementation config, and this check may be removed at some time in the future.
Datatypes
The msg64_hold_datatype
setting is similar to msg64_summary_datatype
, but affects holds instead of circulations.
When set to barcode
, holds information will be delivered as a set of copy barcodes instead of title strings for
patron info requests. With barcodes, SIP clients can both find the title strings for display (via item info requests)
and make subsequent hold-related action requests, like holds cancellation.
Adding SIP Users
-
Type the following commands from the command prompt:
$ sudo su opensrf $ cd /openils/conf
-
In the <accounts> section, add SIP client login information. Make sure that all <logins> use the same institution attribute, and make sure the institution is listed in <institutions>. All attributes in the <login> section will be used by the SIP client.
-
In Evergreen, create a new profile group called SIP. This group should be a sub-group of Users (not Staff or Patrons). Set Editing Permission as group_application.user.sip_client and give the group the following permissions:
COPY_CHECKIN COPY_CHECKOUT CREATE_PAYMENT RENEW_CIRC VIEW_CIRCULATIONS VIEW_COPY_CHECKOUT_HISTORY VIEW_PERMIT_CHECKOUT VIEW_USER VIEW_USER_FINES_SUMMARY VIEW_USER_TRANSACTIONS
OR use SQL like:
INSERT INTO permission.grp_tree (name,parent,description,application_perm) VALUES ('SIP', 1, 'SIP2 Client Systems', 'group_application.user.sip_client');
INSERT INTO permission.grp_perm_map (grp, perm, depth, grantable) SELECT g.id, p.id, 0, FALSE FROM permission.grp_tree g, permission.perm_list p WHERE g.name = 'SIP' AND p.code IN ( 'COPY_CHECKIN', 'COPY_CHECKOUT', 'RENEW_CIRC', 'VIEW_CIRCULATIONS', 'VIEW_COPY_CHECKOUT_HISTORY', 'VIEW_PERMIT_CHECKOUT', 'VIEW_USER', 'VIEW_USER_FINES_SUMMARY', 'VIEW_USER_TRANSACTIONS' );
Verify:
SELECT * FROM permission.grp_perm_map pgpm INNER JOIN permission.perm_list ppl ON pgpm.perm = ppl.id INNER JOIN permission.grp_tree pgt ON pgt.id = pgpm.grp WHERE pgt.name = 'SIP';
-
For each account created in the <login> section of oils_sip.xml, create a user (via the staff client user editor) that has the same username and password and put that user into the SIP group.
The expiration date will affect the SIP users' connection so you might want to make a note of this somewhere. |
Running the server
To start the SIP server type the following commands from the command prompt:
$ sudo su opensrf
$ oils_ctl.sh -a [start|stop|restart]_sip
Logging-SIP
Syslog
It is useful to log SIP requests to a separate file especially during initial setup by modifying your syslog config file.
-
Edit syslog.conf.
$ sudo vi /etc/syslog.conf # maybe /etc/rsyslog.conf
-
Add this:
local6.* -/var/log/SIP_evergreen.log
-
Syslog expects the logfile to exist so create the file.
$ sudo touch /var/log/SIP_evergreen.log
-
Restart sysklogd.
$ sudo /etc/init.d/sysklogd restart
Syslog-NG
-
Edit logging config.
sudo vi /etc/syslog-ng/syslog-ng.conf
-
Add:
# +SIP2+ for Evergreen filter f_eg_sip { level(warn, err, crit) and facility(local6); }; destination eg_sip { file("var/log/SIP_evergreen.log"); }; log { source(s_all); filter(f_eg_sip); destination(eg_sip); };
-
Syslog-ng expects the logfile to exist so create the file.
$ sudo touch /var/log/SIP_evergreen.log
-
Restart syslog-ng
$ sudo /etc/init.d/syslog-ng restart
Testing Your SIP Connection
-
In the root directory of the SIPServer code:
$ cd SIPServer/t
-
Edit SIPtest.pm, change the $instid, $server, $username, and $password variables. This will be enough to test connectivity. To run all tests, you’ll need to change all the variables in the Configuration section.
$ PERL5LIB=../ perl 00sc_status.t
This should produce something like:
1..4 ok 1 - Invalid username ok 2 - Invalid username ok 3 - login ok 4 - SC status
-
Don’t be dismayed at Invalid Username. That’s just one of the many tests that are run.
More Testing
Once you have opened up either the SIP OR SIP2 ports to be accessible from outside you can do some testing via telnet. In the following tests:
-
Replace $server with your server hostname (or localhost if you want to skip testing external access for now);
-
Replace $username, $password, and $instid with the corresponding values in the <accounts> section of your SIP configuration file;
-
Replace the $user_barcode and $user_password variables with the values for a valid user.
-
Replace the $item_barcode variable with the values for a valid item.
-
Start by testing your ability to log into the SIP server:
We are using 6001 here which is associated with SIP2 as per our configuration.
$ telnet $server 6001 Connected to $server. Escape character is '^]'. 9300CN$username|CO$password|CP$instid
If successful, the SIP server returns a 941 result. A result of 940, however, indicates an unsuccessful login attempt. Check the <accounts> section of your SIP configuration and try again.
-
Once you have logged in successfully, replace the variables in the following line and paste it into the telnet session:
2300120080623 172148AO$instid|AA$user_barcode|AC$password|AD$user_password
If successful, the SIP server returns the patron information for $user_barcode, similar to the following:
24 Y 00120100113 170738AEFirstName MiddleName LastName|AA$user_barcode|BLY|CQY |BHUSD|BV0.00|AFOK|AO$instid|
The response declares it is a valid patron BLY with a valid password CQY and shows the user’s $name.
-
To test the SIP server’s item information response, issue the following request:
1700120080623 172148AO$instid|AB$item_barcode|AC$password
If successful, the SIP server returns the item information for $item_barcode, similar to the following:
1803020120160923 190132AB30007003601852|AJRégion de Kamouraska|CK001|AQOSUL|APOSUL|BHCAD |BV0.00|BGOSUL|CSCA2 PQ NR46 73R
The response declares it is a valid item, with the title, owning library, permanent and current locations, and call number.
SIP Communication
SIP generally communicates over a TCP connection (either raw sockets or over telnet), but can also communicate via serial connections and other methods. In Evergreen, the most common deployment is a RAW socket connection on port 6001.
SIP communication consists of strings of messages, each message request and response begin with a 2-digit ``command'' - Requests usually being an odd number and responses usually increased by 1 to be an even number. The combination numbers for the request command and response is often referred to as a Message Pair (for example, a 23 command is a request for patron status, a 24 response is a patron status, and the message pair 23/24 is patron status message pair). The table in the next section shows the message pairs and a description of them.
For clarification, the ``Request'' is from the device (selfcheck or otherwise) to the ILS/ACS. The response is… the response to the request ;).
Within each request and response, a number of fields (either a fixed width or separated with a | [pipe symbol] and preceded with a 2-character field identifier) are used. The fields vary between message pairs.
Pair |
Name |
Supported? |
Details |
01 |
Block Patron |
Yes |
01/Block_Patron - ACS responds with 24 Patron Status Response |
09-10 |
Checkin |
Yes (with extensions) |
|
11-12 |
Checkout |
Yes (no renewals) |
|
15-16 |
Hold |
Partially supported |
|
17-18 |
Item Information |
Yes (no extensions) |
|
19-20 |
Item Status Update |
No |
19/20_Item_Status_Update - Returns Patron Enable response, but doesn’t make any changes in EG |
23-24 |
Patron Status |
Yes |
23/24_Patron_Status - 63/64 ``Patron Information'' preferred |
25-26 |
Patron Enable |
No |
25/26_Patron_Enable - Used during system testing and validation |
29-30 |
Renew |
Yes |
|
35-36 |
End Session |
Yes |
|
37-38 |
Fee Paid |
Yes |
|
63-64 |
Patron Information |
Yes (no extensions) |
|
65-66 |
Renew All |
Yes |
|
93-94 |
Login |
Yes |
93/94_Login - Must be first command to Evergreen ACS (via socket) or SIP will terminate |
97-96 |
Resend last message |
Yes |
|
99-98 |
SC-ACS Status |
Yes |
01 Block Patron
A selfcheck will issue a Block Patron command if a patron leaves their card in a selfcheck machine or if the selfcheck detects tampering (such as attempts to disable multiple items during a single item checkout, multiple failed pin entries, etc).
In Evergreen, this command does the following:
-
User alert message: CARD BLOCKED BY SELF-CHECK MACHINE (this is independent of the AL Blocked Card Message field).
-
Card is marked inactive.
The request looks like:
01<card retained><date>[fields AO, AL, AA, AC]
Card Retained: A single character field of Y or N - tells the ACS whether the SC has retained the card (ex: left in the machine) or not.
Date: An 18 character field for the date/time when the block occurred.
Format: YYYYMMDDZZZZHHMMSS (ZZZZ being zone - 4 blanks when local time, ``Z'' (3 blanks and a Z) represents UTC(GMT/Zulu)
Fields: See Fields for more details.
The response is a 24 ``Patron Status Response'' with the following:
-
Charge privileges denied
-
Renewal privileges denied
-
Recall privileges denied (hard-coded in every 24 or 64 response)
-
hold privileges denied
-
Screen Message 1 (AF): blocked
-
Patron
09/10 Checkin
~The request looks like:
09<No block (Offline)><xact date><return date>[Fields AP,AO,AB,AC,CH,BI]
No Block (Offline): A single character field of Y or N - Offline transactions are not currently supported so send N.
xact date: an 18 character field for the date/time when the checkin occurred. Format: YYYYMMDDZZZZHHMMSS (ZZZZ being zone - 4 blanks when local time, ``Z'' (3 blanks and a Z) represents UTC(GMT/Zulu)
Fields: See Fields for more details.
The response is a 10 ``Checkin Response'' with the following:
10<resensitize><magnetic media><alert><xact date>[Fields AO,AB,AQ,AJ,CL,AA,CK,CH,CR,CS,CT,CV,CY,DA,AF,AG]
Example (with a remote hold):
09N20100507 16593720100507 165937APCheckin Bin 5|AOBR1|AB1565921879|ACsip_01|
101YNY20100623 165731AOBR1|AB1565921879|AQBR1|AJPerl 5 desktop reference|CK001|CSQA76.73.P33V76 1996 |CTBR3|CY373827|DANicholas Richard Woodard|CV02|
Here you can see a hold alert for patron CY 373827, named DA Nicholas Richard Woodard, to be picked up at CT
BR3''. Since the transaction is happening at AO
BR1'', the alert type CV is 02 for hold at remote library. The
possible values for CV are:
-
00: unknown
-
01: local hold
-
02: remote hold
-
03: ILL transfer (not used by EG)
-
04: transfer
-
99: other
The logic for Evergreen to determine whether the content is magnetic_media comes from or search_config_circ_modifier. The default is non-magnetic. The same is true for media_type (default 001). Evergreen does not populate the collection_code because it does not really have any, but it will provide the call_number where available. Unlike the item_id (barcode), the title_id is actually a title string, unless the configuration forces the return of the bib ID. Don’t be confused by the different branches that can show up in the same response line.
|
15/16 Hold
Evergreen supports the Hold message for the purpose of canceling holds. It does not currently support creating hold requests via SIP2.
17/18 Item Information
The request looks like:
17<xact_date>[fields: AO,AB,AC]
The request is very terse. AC is optional.
The following response structure is for SIP2. (Version 1 of the protocol had only 6 total fields.)
18<circulation_status><security_marker><fee_type><xact_date> [fields: CF,AH,CJ,CM,AB,AJ,BG,BH,BV,CK,AQ,AP,CH,AF,AG,+CT,+CS]
Example:
1720060110 215612AOBR1|ABno_such_barcode|
1801010120100609 162510ABno_such_barcode|AJ|
1720060110 215612AOBR1|AB1565921879|
1810020120100623 171415AB1565921879|AJPerl 5 desktop reference|CK001|AQBR1|APBR1|BGBR1 |CTBR3|CSQA76.73.P33V76 1996|
The first case is with a bogus barcode. The latter shows an item with a circulation_status of 10 for in transit between libraries. The known values of circulation_status are enumerated in the spec.
EXTENSIONS: The CT field for destination location and CS call number are used by Automated Material Handling systems.
23/24 Patron Status
Example:
2300120060101 084235AOUWOLS|AAbad_barcode|ACsip_01|ADbad_password|
24YYYY 00120100507 013934AE|AAbad_barcode|BLN|AOUWOLS|
2300120060101 084235AOCONS|AA999999|ACsip_01|ADbad_password|
24 Y 00120100507 022318AEDoug Fiander|AA999999|BLY|CQN|BHUSD|BV0.00|AFOK|AOCONS|
2300120060101 084235AOCONS|AA999999|ACsip_01|ADuserpassword|LY|CQN|BHUSD|BV0.00|AFOK|AOCONS|
24 Y 00120100507 022803AEDoug Fiander|AA999999|BLY|CQY|BHUSD|BV0.00|AFOK|AOCONS|
-
The BL field (SIP2, optional) is valid patron, so the N value means bad_barcode doesn’t match a patron, the Y value means 999999 does.
-
The CQ field (SIP2, optional) is valid password, so the N value means bad_password doesn’t match 999999’s password, the Y means userpassword does.
So if you were building the most basic SIP2 authentication client, you would check for |CQY| in the response to know the user’s barcode and password are correct (|CQY| implies |BLY|, since you cannot check the password unless the barcode exists). However, in practice, depending on the application, there are other factors to consider in authentication, like whether the user is blocked from checkout, owes excessive fines, reported their card lost, etc. These limitations are reflected in the 14-character patron status string immediately following the 24 code. See the field definitions in your copy of the spec.
29/30 Renew
Evergreen supports the Renew message. Evergreen checks whether a penalty is specifically configured to block renewals before blocking any SIP renewal.
35/36 End Session
3520100505 115901AOBR1|AA999999|
36Y20100507 161213AOCONS|AA999999|AFThank you!|
The Y/N code immediately after the 36 indicates success/failure. Failure is not particularly meaningful or important in this context, and for evergreen it is hardcoded Y.
63/64 Patron Information
Attempting to retrieve patron info with a bad barcode:
6300020060329 201700 AOBR1|AAbad_barcode|
64YYYY 00020100623 141130000000000000000000000000AE|AAbad_barcode|BLN|AOBR1|
Attempting to retrieve patron info with a good barcode (but bad patron password):
6300020060329 201700 AOBR1|AA999999|ADbadpwd|
64 Y 00020100623 141130000000000000000000000000AA999999|AEDavid J. Fiander|BHUSD|BV0.00 |BD2 Meadowvale Dr. St Thomas, ON Canada
90210|BEdjfiander@somemail.com|BF(519) 555 1234|AQBR1|BLY|CQN|PB19640925|PCPatrons |PIUnfiltered|AFOK|AOBR1|
See 23/24 Patron Status for info on BL and CQ fields.
93/94 Login
Example:
9300CNsip_01|CObad_value|CPBR1|
[Connection closed by foreign host.] ...
9300CNsip_01|COsip_01|CPBR1|
941
941 means successful terminal login. 940 or getting dropped means failure.
When using a version of SIPServer that supports the feature, the Location (CP) field of the Login (93) message will be used as the workstation name if supplied. Blank or missing location fields will be ignored. This allows users or reports to determine which selfcheck performed a circulation.
99/98 SC and ACS Status
99<status code><max print width><protocol version>
All 3 fields are required:
-
0: SC is OK
-
1: SC is out of paper
-
2: SC shutting down
-
status code - 1 character
-
max print width - 3 characters - the integer number of characters the client can print
-
protocol version - 4 characters - x.xx
98<on-line status><checkin ok><checkout ok><ACS renewal policy> <status update ok><offline ok><timeout period>
<retries allowed><date/time sync><protocol version><institution id> <library name><supported messages><terminal
location><screen message><print line>
Example:
9910302.00
98YYYYNN60000320100510 1717202.00AOCONS|BXYYYYYYYYYNYNNNYN|
The Supported Messages field BX appears only in SIP2, and specifies whether 16 different SIP commands are supported by the ACS or not.
Fields
All fixed-length fields in a communication will appear before the first variable-length field. This allows for simple parsing. Variable-length fields are by definition delimited, though there will not necessarily be an initial delimiter between the last fixed-length field and the first variable-length one. It would be unnecessary, since you should know the exact position where that field begins already.
Patron privacy and the SIP protocol
SIP traffic includes a lot of patron information, and is not encrypted by default. It is strongly recommended that you encrypt any SIP traffic.
SIP server configuration
On the SIP server, use iptables
or etc/hosts
to allow SSH connections on port 22 from the SIP client machine. You will probably want to have very restrictive rules
on which IP addresses can connect to this server.
SSH tunnels on SIP clients
SSH tunnels are a good fit for use cases like self-check machines, because it is relatively easy to automatically open the connection. Using a VPN is another option, but many VPN clients require manual steps to open the VPN connection.
-
If the SIP client will be on a Windows machine, install cygwin on the SIP client.
-
On the SIP client, use
ssh-keygen
to generate an SSH key. -
Add the public key to /home/my_sip_user/.ssh/authorized_keys on your SIP server to enable logins without using the UNIX password.
-
Configure an SSH tunnel to open before every connection. You can do this in several ways:
-
If the SIP client software allows you to run an arbitrary command before each SIP connection, use something like this:
ssh -f -L 6001:localhost:6001 my_sip_user@my_sip_server.com sleep 10
-
If you feel confident that the connection won’t get interrupted, you can have something like this run at startup:
ssh -f -N -L 6001:localhost:6001 my_sip_user@my_sip_server.com
-
If you want to constantly poll to make sure that the connection is still running, you can do something like this as a cron job or scheduled task on the SIP client machine:
-
#!/bin/bash instances=`/bin/ps -ef | /bin/grep ssh | /bin/grep -v grep | /bin/wc -l` if [ $instances -eq 0 ]; then echo "Restarting ssh tunnel" /usr/bin/ssh -L 6001:localhost:6001 my_sip_user@my_sip_server.com -f -N fi