Widgets and web services

Behind (most) widgets you’ll find one or more data sources – whether a relational database, a bunch of linked data or even a text file. The widget is basically an interface to this data. The relationships between widgets and their data sources fall into two main types:

• Retrieval: the widget asks the data source a question and uses the answer for some purpose (usually display)
  
• Alteration: the widget causes a change to be made in the data source

So, in our Library widget, an example of retrieval would be to show a list of the items a user has on loan (no change to underlying data). And an example of alteration would be renewing one of these items (change made in underlying data).

Which means that widgets need a way of talking to their data sources.

Most widgets talk to their data sources through web services (also known as APIs) - little scripts which sit on your web server and mediate between your widget and your data.

They work something like this - your widget makes a request to a web service - either that some data be retrieved, or that a change be made in the data. The web service translates the request into data-speak and sends it on to the data source. The data source processes the request, makes any alterations, and sends back a response. The web service translates the response into widget-speak and sends it back to the widget.

So, why can’t widgets talk to their data sources directly? Why the need for the middle-man? There are four main reasons:

• Security – if your widget is talking to the data source directly, it is likely to contain passwords. Because widgets are usually written in html/javascript, pretty much anyone can access the source code and so access your passwords.
• Efficiency – web services are reusable. Say you have three widgets that need the same bit of data. You can write one web service to make the request and do any processing on the response, and all three widgets can use it.
• Ease of use – most widgets are written in html and javascript. Basically, it’s pretty easy to make calls to web services from javascript. Making calls to data sources directly from widgets is a bit of a nightmare.
• Innovation – if you have a bunch of web services sitting on your server, you can choose to publish them. This means that other developers can build systems and functionality around your data. Which is not only pretty exciting, but also saves you having to do everything yourself.

Many systems now come with their own set of proprietary web services for talking to their data source. These should be written up in your system’s documentation. If the system has no web services or it doesn’t have a web service for what you want to do, it’s pretty simple to write your own.

EXCEPT - reflecting the two types of request you can make to your data source, there are two types of web service - those that retrieve data and those that make alterations to your data. Most systems will allow you to write your own web services to retrieve data BUT most systems will insist that you use their own proprietary web services to alter data in the system.

So, going back to our original example, we have a locally written web service which retrieves a list of a user’s loans, but we use a proprietary system web service to renew one of those loans.

Our widgets use a mixture of locally-written and proprietary web services to relate to data sources, and (unless your system’s proprietary web services are truly excellent) you’ll probably find yourself doing the same. I’ll stop here before this post turns into an essay, but in future posts I’ll outline how to write web services, and how to use them.

Where we are

The University Library (UL) and CARET have already dipped a mutual toe into widgets with the development of the Cambridge Libraries Widget. Before we immerse ourselves further, here’s an update on where we are and how we got there.

The UL became interested in widgets (portable web interfaces) as a quick and easy way of making library functionality and content available outside the traditional library catalogue. We were particularly interested in CamTools, the University’s VLE.

CamTools is written and administered by CARET, who have a wealth of experience in developing web interfaces. The UL and CARET were casting around for opportunities to collaborate, and the development of a library widget seemed like the perfect opportunity.

We found we were able to work very well together, and after various struggles (mainly to do with authentication), the Cambridge Library Widget was launched in CamTools, iGoogle and Facebook in January 2010, with an embeddable version following soon afterwards. The Widget is deployed in our production VLE, with usage climbing steadily.

It allows library users to search library collections, view their library profile (which can be fairly complex in Cambridge), and manage loans and requests. Widget functionality will underlie the user interfaces for our new resource discovery platform to be launched at the start of the next academic year.

So, what have we achieved so far? We have taken library functions out of the library catalogue and freed them to wander in our VLE and social networking sites. We have taken control of the presentation of library services and functionality in a way we haven’t been able to do before. And we’ve written a few snazzy extra features. But essentially we have been in the (admittedly important) business of recreating existing functionality in a new environment.

Which is all well and good, but we were left feeling “there must be more to widgets than this”. Hence this project, in which we aim to unlock the potential of widgets to do entirely new things, and to bring together some of the disconnected parts of academic life for our users.

This means building widgets which talk to more than one system at a time, linking people to integrated services and content. We want our interfaces to connect not just to our library management system, but also to our institutional repository and student registry.

Of course, we are still interested in new environments, and one strand of the project will tackle the challenges of deploying widget-like interfaces on mobile devices. And we’re also hoping to have some fun with a few one-off widgets which explore the odd and innovative possibilities of tying disparate data together.

Over the coming weeks, as well as keeping you up-to-date with new developments, we’ll be delving into the past to illuminate what widgets are, how they work, and how to get them working for you.

More setting up

We've got our Google code site set up now, along with Google analytics for both the blog and Google code.

I was delighted to meet our programme manager, David Flanders, for the first time at OKCon on Saturday, and look forward to seeing him in Cambridge soon!

Hello

We're just setting up the project! Here's a picture of the project team (comprising people from CARET and the UL).


From left to right:

• Katy Cherry (user testing)
• Raymond Chan (developer)
• Bert Pareyn (intern developer)
• Verity Allan (QA)
• Huw Jones (developer and librarian)
• Dan Sheppard (project manager)
• Laura James (temporary stand-in project manager)

Dan is currently away being an Arcadia Fellow at the library, and I'm holding the fort until his return.