"Final Post"

I'm in a slightly odd position writing a final post for this project. There's two reasons for that.

The first is that we're still working on future versions of all of the things we produced in this project, and will for some time. This blog won't be stopping, please don't unsubscribe! You'll miss out on all our future widgets.

The second is that I'm used to writing formal final reports, where we detail what the project has achieved, this time it's already on the blog. It's also very difficult to get the tone right in such a different environment.

I don't think it's worthwhile duplicating all that detail, but I'll summarize it here, and you can find the detail if you're interested.

First, you'll have seen the financial statement. The project management side of things seemed to work well, and we've also forged some good relationships within the University, for example getting a feed from our Student Information System.

The widgets we're proudest of, probably because they're the most advanced, are the "Library Widget" which provides the functionality you'd expect from a library (catalogue, books on loan, etc), in a single widget, embeddable in our VLE, in Facebook and in iGoogle. A related widget provides the same functionality on a mobile. These were half-done at the start of the project, through institutional commitment, but it was great to spend the time and effort to put them into production as a proper University service. The uptake was dramatic, and a little scary (see previous posts). If you want hard numbers for impact, that's where you'll find some pretty impressive ones!

Whilst Huw was working on productionising more developed widgets, like the general purpose library widget, and the mobile interface, I was responsible for getting the new widgets off the ground and into alpha. You can see inside the laboratory here, to see what's coming. But a laboratory can be dangerous, these widgets really are all in "alpha", and will almost certainly break at some point in the future, on their path to production. So have a play, but don't be surprised if something odd happens!

More advanced widgets are here on our demo page, and our forthcoming "storefront" is here. With three more widgets due for release in the next couple of weeks, we'll do a big announcement of the storefront to the University at that point.

I'm personally particularly excited about "sputnik" which isn't so much a widget in the traditional sense, but a Firefox plugin (Chrome to follow) which advertises how your library (or University more widely) can cope whilst you surf the web. In some ways it resembles Libx, and was inspired by the iCite life sciences plugin, but it's much less shy than either of them and, we think, has massive potential. For example, if you can't use the OPAC, find it in Amazon, and we'll tell you where it is in the library. Commercial vendors are not going to be shy about their offerings, they're not going to hide their information one or two clicks away, so why should we be reticent about our alternatives on commercial pages, particularly as the user will have to actively elect to install the plugin. The large area set aside also gives room for us to go beyond holdings in the future.

The exam paper widget (and associated uploader) helps students find exams and knows their subjects from our student information system. That's pretty much the first time we've integrated with our SIS, and we now have a feed which can help contextualise information. We have to work to the natural student year for this widget, of course, we can't move exams, and also the natural round of meetings of committees, but we've already got many papers from one of our subjects and approval to include more, so watch out for future posts on this subject.

Cambridge has many libraries and which library a student is a member of is not trivial to work out. We did it in the end, though, and now you'll see a widget which allows a student only to search libraries which they actually have access to. We think this is the first time this has been possible!

We've widgetised a reading list for a course (data entered via google spreadsheet) and are working on genericising it to other data sources including the possibility of linking it to Zotero or some other reading list manager.

We mustn't forget what might be the most valuable output of the project, the services for others to use, and our advice pages for widget writers which summarize much better than I am doing here the advantages of the approach.

Thanks to JISC for their support, and keep on looking for future updates. I've probably missed something out which I'll remember over the weekend, so I'll continue this shortly. In the meantime, here's a screenshot of my (personal) favourite "Sputnik" plugin, which I've already personally found invaluable (and see also Huw's usage graphs in other posts!). All the code for these widgets, even the experimental ones, should be in our googlecode svn (seach for culwidgets), and we'll make sure they're all complete at release of each widget.

In the spirit of the brave new world of blog, if you'd like further information on this project, or would like to steer any updates as to outcomes, just comment below and I'll happily answer.

CamLib Mobile interface Report

The CamLib mobile interface is entirely a product of the project. We decided that as well as pursuing students into social networking sites, we also wanted to inveigle ourselves onto their phones. There is no escape from the Library!

We initially set out to simply recreate the library widget for smartphones, but as we went along we did some thinking about what people would want to do on their phones (as opposed to waiting until they were at a 'proper' computer)

So the focus was on finding out if something existed, where it was, and how to get your hands on it. We did quite a lot of integration with maps and request functionality, and added a new "bookbag" feature to allow users to build lists of books from the catalogue and email it to themselves. Plus we kept in all the renewal, loans, libraries stuff as well.

How is it doing so far? Let's take a look (vertical axis figs are 35 and 70)

Well, just a bit better than expected. We knew that only a minority of students have smartphones, and only a minority of them use smartphones for work. So 540 unique users in a month isn't bad.

But we're expecting a tipping point sometime in this academic year, much as happened with laptops a few years ago. And when they come we will be ready!

Verdict: B-
Good effort, great things expected

Library Widget Report

This is the first of a series of end-of-term reports on each of our widgets. Who made the grade? Who must try harder? Who are the troublemakers? Who are the swots?

We'll start with the Library Widget. We've done a fair amount of development on what was already a fairly established platform. We added in Library pics to help with navigation, we beefed up security, we integrated it into our new resource discovery layer, and we did some behind-the-scenes jiggery-pokery to make everything work more smoothly.

For the Library Widget, the project was more about launching it and seeing if it floated. So let's take a look at this month's Google Analytics and see how it's doing (vertical axis figs 200 and 400)


Not bad at all! Plenty of users, plenty coming back for more, and with 2,168 unique users we are already exceeding our end of year target of 2,000 users per month. Obviously, there is more we can do with the Library Widget, but it's essentially a stable product with an established and growing user base. The future looks bright!

Verdict: B+
Steady progress, will go far.

Financial Report

Here's our project's final financial report. I hope the format is ok. If you want any more info, just ask me. It's fun to work on a project so inline with our aims, as it takes no effort at all to get people involved. The difficulty can be getting them to do their other work!

Cost	Apr 10 - Jul 10	Aug 10 - Oct 10	Total Budget	Total Spend	Variance (+ve=over)
Total Directly Incurred Staff (A)	£12,725	£9,620	£22,345	£24,988a	£2,643
Total Directly Incurred Non-Staff (B)	£1,200	£300	£1,500	£800b	-£700
Directly Incurred Total (A+B=C)	£13,925	£9,920	£23,845	£25,788	£1,943
Directly Allocated Total (D)	£5,477	£3,305	£8,782	£8,782	£0
Indirect Costs (E)	£12,657	£9,250	£21,907	£21,907	£0
Total Project Cost (C+D+E)	£32,059	£22,475	£54,534	£56,477	£1,943
Amount Requested From JISC	£26,400	£13,600	£40,000	£40,000	£0
Institutional Contributions	£5,659	£8,875	£14,534	£16,477	£1,943

a. 195hr, 643hr, 57hr, 64hr. Spine points redacted for DPA.

b. £300 travel (-£700 variance), £500 contingency.

Contributions over life of the project:

	Budget	Actual
JISC	73%	71%
Partners	27%	29%

Those Advice Documents Now Online

The documents which started out as our final post and kind of grew, are now online.

So tomorrow probably marks the formal end of the project, with our actual final post, followed by the financial report, and so on. But don't discard this blog: there should be a fair few "going live" announcements over the next couple of weeks. I'm hoping to get the Library Companion, which is a reimagining of LibX, available in alpha within the next few hours.

Update

Sorry for the delay in updating, everyone. We're in the process of putting together our closing remarks and have decided that there would be enough value in putting what we have into a number of permanent pages on our website, to guide future widget authors. We're just in the process of tidying and CSS-ing them all, and they should be up in the next couple of days.

In the meantime, we'd welcome comments. Please forgive the formatting.

We will create three pages: a top level page for high-level comments, and one each for writing services and widgets.

Here's the Services page, already formatted and complete: http://www.lib.cam.ac.uk/api/web_services.html

And below is what we have for the top level page, and widget writing page, awaiting that formatting goodness. Comments greatfully received: though these are static pages, they're intended to be living documents.

We're building a little app store for all our widgets, which we can reveal in the next couple of days (well, we can reveal it now, but you'll find it confusing until we fix a couple of things). I'll announce that ASAP, along with the locations of these pages when they're up (ETA Tuesday). We're also finalising the project accounts.

INTRODUCTION
============

This document describes why you might want to write widgets and services to solve some of the problems in your library.

The other documents in this collection are aimed at software developers. However, this introduction is for a more general audience of technically literate people, and describes motivation and general shape.

Whilst the remainder of these pages require software development skills, they do not require expensive engineers: a new graduate or a librarian with a sideline in software development should easily be able to get going with them to produce something useful.

Why bother with widgets?
========================

Often, the biggest obstacle to overcome when explaining widgets is to overcome the frivolity associated with the concept: the silly name; their ubiquitous use on social networking sites for producing pointless distractions, and so on. We encourage you to set such preconceptions aside, at least temporarily.

Reason 1: Widgets reduce the risk associated with experimentation
-----------------------------------------------------------------

Widgets are small (we're talking person-hours or -days) and easily rewritten. This allows you to take risks in creating them. If some users have indicated that a widget might be useful, you can spend half a day creating one: much less time than it would take to perform a formal survey.

Think of how many projects there have been at your institution concerning integration of various computer systems. And think, nationally, how many projects there have been only to integrate catalogue and circulation information between VLEs with LMSes. Consider, particularly, their budgets. We have such functionality already, in production, against our enterprise sites, in use by academics and students, as a result of a small project, created by Huw Jones, a librarian. And that's just one widget.

Particularly at larger HE institutions, we are used to inter-departmental collaborations involving endless meetings, vague strategy, and nobody to arbitrate or direct the project below vice-chancellor level. So, what we offer to students simply ends up being disjointed.

Developing software with such a low overhead allows integrations to occur "below the radar", and "under the budget" of large, difficult projects.

Reason 2: No more "you'll need to login to another system"
----------------------------------------------------------

Remembering URLs, usernames and passwords is difficult, and we seem to spend so much of our time logging into systems and finding our way from the front page to where we want to be.

As a user, widgets allow you to do things "from where you are". There's no "that's in another system" issues. A user can, effectively, design their own screens to fit the way they work. There is no need to login again (auth permitting), to remember a URL, or to navigate to the page you actually want through a maze of initial pages. A screen of widgets might contain some widgets derived from the LMS, some from the VLE, some from teacher-provided data (such as reading lists), and so on.

Many of these containers (for example, in a VLE) have rich knowledge of the current user, too. That means that you can present "zero-click" search results, for example for exam papers: there is no need for a student even to search for the papers they are taking because the results displayed by default are usually what the user is after. (However the opportunity is presented to change this, if the user wishes).

Both of these mean that the user spends more time interacting with the real content, and less time navigating.

Reason 3: Moving development towards specialists
------------------------------------------------

Writing widgets, and particularly services, does require some development expertise, but it doesn't require much, and certainly less than most other approaches. This makes it much easier to keep control of the development. You might have a librarian who is handy with software development. Your library might have a website maintainer who can do it. You might be able to hire someone directly into your library on a short term basis. There's certainly no need to engage a big external organisation.

Recommendation
==============

We'd particularly recommend widget writing to institutions which have struggled, technically or socially, with integration and development in the past, and where the students are faced with a confusing array of different systems to use.

Despite their silly name.

Widgets document:

Introduction
============

Widgets are small packages of code which put information from external sources onto regions of webpages. Typically they comprise small collections of HTML, CSS, Javascript and (sometimes) XML.

In their simplest form widgets contain only an "iframe" which is a window onto a website hosted on another machine. More complex widgets contain code which is to be embedded and run actually within the outer page.

Widgets can usually be written in a day in a text editor, including the time spent to learn a new widget technology, if you have the appropriate services feeding you data. Widgets use services, detailed here to perform queries on underlying systems. A widget may, or may not, have access to further information from the container such as the current user, their permissions and so on.

Basic Principles
================

* Widgets sit inside a widget container (aka portal). This can be as simple as a static webpage or as complex as Facebook, iGoogle, or a VLE. For most of the different containers which you will target, you will need to write a separate widget. However, writing a widget should be quick, and you can reuse a lot of your code.

* Widgets execute entirely in the browser of the user. Any interactions which they need to make with your own systems takes place through the use of services.

* Widgets can either use iframes or not (not using them is usually called using HTML fragments). You need to decide how yours are to be implemented (see below). But changing your mind later on isn't painful.

* If you eschew iframes and use HTML Fragments, you write HTML and CSS to lay the widget out as you intend. Then use AJAX in your javascript to call services and then update the webpage.

* Using iframes is simpler. You write a small fragment of code to be hosted on the container, containing the iframe "window", and then write the rest of your code on your own server.

* If using HTML Fragments, a container will typically help you with the javascript by providing you with libraries to perform common tasks (such as calling a remote URL for services). These may well include the jQuery or trimpath libraries of useful javascript methods. Do check particularly if jQuery is installed as, if so, jQuery can save a lot of bother. If using iframes, we strongly recommend using jQuery in your pages.

* A container may provide you with a means of storing a user's preferences, or you could use client-side cookies. This is useful for things like preferred layout and usernames.

* Because widgets run entirely in the user's browser, you cannot rely on what is returned for security purposes, as a user can intercept and change messages between widgets and services. (see below for help on security).

To iframe or not to iframe?
===========================

As described above, an iframe tag is like a window in one web page onto another, often on another site altogether. One use of iframes is for widgets. By placing only an iframe into the widget code for the container, the content of the widget can come from your own server, and will be put inside the iframe by the browser.

Some widget containers require the use of iframes. Others don't allow iframes, your widget must hosted on the site of the widget container as HTML fragments. Many containers allow both.

If your container only allows one approach or the other, then you have no choice.

If both are available, I recommend choosing HTML fragments first, and move to iframes if you find something easier to achieve using them. The exception is Facebook, where iframes are generally easier. This blog post is a good discussion of an alternative perspective on the iframe vs fragments issue.

Typically, access to services will become the deciding factor in whether or not to use iframes.

An iframe widget is running on your server, and so can be given complete access to whatever is running on your server with little difficulty through your own programming. However, the widget is at some remove from the container, and so you will find it harder to access facilities provided by the container (for example, it may be harder to determine which user is using the widget).

The opposite is true of HTML fragments: they are closely embedded in the container, but further from your own systems.

Trying, and being prepared to change if you run into trouble, is probably the best approach, until you get an instinct for this decision.

Writing Your Own
================

If you are interested in widgets in general, but have no specific target in mind, try iGoogle Gadgets. These will help clarify the issues discussed in this document ready for when you write widgets in anger.

To write an iGoogle Gadget start here. This describes writing an iGoogle gadget as an HTML fragment. The "Include Gadget" on this page is an example of an iframe-based iGoogle gadget.

A Camtools start page widget is like an iGoogle Gadget, and can be written and tested as such. Extra facilities are available within camtools, however. For details contact the Camtools team.

To write a Facebook application start here. See here for a discussion of iframe vs fragment issues in relation to Facebook in particular.

To embed a widget into a static HTML page, all you need to understand is how the iframe tag works. Write your widget as for any other iframe widget and then write the relevant iframe tag and encourage others to embed it on their page. One way to encourage this is to make the code copy-and-pastable from your site.

If you want more info, talks.cam views could be valuable, as they show how such a technique is used in a live site.

If at all possible, use a library like jQuery to help you write your javascript. This is always possible with iframe widgets, and often provided with HTML fragments. jQuery helps you write javascript by providing common methods for manipulating a webpage which have been extensively tested and debugged. Whilst the javascript language itself is quite simple (adding two numbers, calling a function, and so on), interacting with browsers (making something green, implementing tabs), whilst superficially simple, is very subtle if you want it to work on most browsers, and best left to the experts.

Integrating With Services
=========================

As described in our document on services, services are presented at a particular URL. To consume a service you need to connect to that service as if a web client. You can do this from javascript running on a user's web browser. This is known (formally erroneously) as AJAX.

However, AJAX can only issue a request to the page's originating server. In the case of HTML Fragment widgets this will be the widget container's domain (eg google for an iGoogle widget). This would make HTML Fragment widgets almost worthless were it not for the additional ability to "proxy" your request through the container: a javascript function is provided by the container to allow it to issue any requests on behalf of the widget. For iGoogle these methods are detailed here.

For iframe widgets, it will be the server from which the content of the frame was served, which may be more useful. Also, with an iframe widget you can, of course, implement server-side code to issue HTTP requests to any site of your choice.

Security
========

Computer security is very difficult. The difficulty is not only in the coding, but in reasoning correctly. Almost everyone believes their concepts are watertight, only to be surprised by an unanticipated attack. But you will probably need to think about security when writing widgets.

There are three main approaches to say safe.

First, don't use security, unless you have to, and then use it as little as possible. This may sound counterintuitive at first, but the fewer things there are to go wrong, the less vulnerable your system is. Do you really need to protect this data? Also, do you really need to be so fine-grained in your control? Keeping a simple undifferentiated circle of trusted individuals is likely to lead to better security than complex fine-grained restriction in many cases. The chances of one of the trusted team betraying the others is usually less likely, or has less of an impact than, the risk of a faulty complex system revealing details to the whole world.

Secondly, if you do use security, use a known technique. The best are those you copy from APIs at well-regarded, popular sites. Discover the techniques they use and copy them.

Thirdly, apply strength in depth. Implement as many techniques as make sense, even if they seem redundant. However, beware of unnecessarily complex systems (see first point).

Useful toolkits in your armoury include: shared secrets, salts, hashes, HMACs, OAuth, Raven, Shibboleth, IP-range restriction. Google for these for details.

Remember, a user can change anything which is sent from their browser.

Guidelines
==========

* Keep the widget as simple as possible: make it do one thing well. Don't dump lots of functionality into a single widget, users can install multiple widgets matching functions they choose to use.

* Let users customise their widget. Your widget will be a part of their page. Issues as simple as title bars, multiple branding, and different colour schemes can be irritating in a page containing many widgets and, more importantly, needlessly consume a lot of screen space.

* Consider Context. Where will your widget go? Make sure you include enough information for the user to understand what the widget does but to reduce the amount that they will already know from surrounding context. If it is to be installed in Facebook don't say "Library Widget" say which library. On your libraries internet, however, say nothing.

This site has excellent general advice about designing widgets.