Proprietary web services and proxies

A couple of weeks ago I blogged about writing local web services to extract information and embed functionality. However, many data sources come with their own range of ready-made proprietary web services. The big advantage of using proprietary web services is that you don't have to write your own. The disadvantage is that (usually) you can't alter them. Unless you have a very good relationship with your system supplier, you're stuck with what you're given.

Most data sources will allow you to write web services which extract data from the system, but the majority won't allow you to write web services which alter data in the system. If you want to use functionality in your widgets rather than simply displaying information (i.e. renew books, rather than just say what you have on loan) then you may find yourself having to use proprietary web services.

Basically, if the existing proprietary web services do exactly what you want, use them. If not, you might want to consider writing your own to extract data. But as far as altering data in the system goes, you may well be stuck with the proprietary ones.

[In practice, we find ourselves using a mixture of local and proprietary web services for data retrieval, and proprietary web services for data alteration]

Somewhere in your system's documentation you should find a list of existing web services and some notes on how to use them. This should include the service address, input parameters and return values. Which translates as how to find the service, how to ask it to do something, and how it will respond. Pretty much the same as using locally written web services.

But there is one potential complication with proprietary web services. If you make a request to a web service from a browser and then try to read the response, the browser will first establish what the response is and where it comes from. If the response is in XML and the domain of the web service is different from the domain of the page calling it, the browser gets worried about security and the whole enterprise collapses.

There are two ways round this. If the proprietary web service has the capability to return its response in json then you don't have worry about where it's coming from. If not, you will have to write a proxy.

A proxy is an intermediate web service which can sit alongside your local web services, in your own domain. You access it just like any local web service, and your browser will treat the xml it returns as if it had come from a local web service.

When you send a request to a proxy it passes it on to the proprietary web service, reads the response and then passes it back to you. Because the proxy is not a browser it doesn't care where the proprietary web service is, and because the proxy is in your domain, your browser is cool with it too.

An additional advantage of going through a proxy is that if your system's proprietary web services return responses which are over-long, oddly structured or confusing, then you can talior that response on the way back so it gives you what you actually need.

We use one proxy web service to connect to all of our library management system's proprietary web services. So we pass the service name, and the request to the proxy. The proxy calls the appropriate service with the request. Then it processes it and returns it to the browser.

Writing Web Services for widgets

In a previous post I outlined the importance of web services to widget development, and pointed to two kinds of web service - proprietary ones which you can't alter, and locally written ones which you can. Today, I'm going to concentrate on locally written web services, and talk about some of the issues involved in writing your own web services. We're going to be making our code available towards the end of the project, so you should be able to pick up some complete examples soon. In the meantime, here are some of the basic ingredients of a home-baked web service.

1. Getting started

• Programming language: we use Perl (and occasionally PHP).
• Data source: the database or dataset you want to interact with (i.e. Library Management System).
• Query language: the language you will use to retrieve or alter the data in this system. Usually SQL.

2. Finding a home

• Your web services will need to live somewhere on your web server. In theory they can go anywhere, but it's best to have a separate directory for web services in general, and then distinct sub-directories for each system or dataset you want to interact with.
  
• Putting all your web services into one overall directory has the added benefit of making it easier for your system administrator (or you) to keep on top of the permissions and settings needed for web service files. Basically, if you're writing web services in Perl, the directory needs to be CGI enabled, and the files need to be readable and executable by anybody (usual octal permissions 755).

3. Do a bit of thinking

• If you're doing a lot of interaction with a dataset you'll probably have a range of questions/alterations you want to throw at it. Most "complete" transactions with datasets are made up of multiple smaller transactions (i.e. to book a train ticket, you might need to find the current time, find the next train, and book the ticket.)
• Instead of trying to write one huge web service to do everything, break each activity into its smallest distinct parts. This will leave you with lots of small, simple web services, each of which does only one thing.
• Now you can chain these simple services together into more complex transactions - but you can also reuse them in other transactions. This saves you asking the same question (i.e. "what's the time?") again and again in multiple web services.
  

4. What input do you need?

• Web services are triggered by requests. These requests (like web services themselves) can be of two types - questions for services that retrieve data or instructions for services that alter data.
• Requests are made up of the location of the web service, plus as many parameters as are necessary to either frame the question or perform the instruction. These parameters make up the input for the web service.
• The smallest amount of parameters needed is none. Think about a question like "what's the time?". If you had a web service which just gave back the current time when triggered, no further parameters would be needed.
• Most web services take at least one parameter. Taking the "what's the time?" service, if you introduced a parameter of "city" the web service could alter the query depending on the value of the parameter. So the service could ask "what's the time - in London?", "what's the time - in New York?", depending on the value of the parameter.
• The rule of thumb with parameters is that you should use as few of them as possible in order to ask the question/perform the alteration. You can always add more if required. Again, you should be thinking of small, simple services, each related to a specific task.

5. Checking input

• There are two things you need to check your parameters for - existence and content.
• For each parameter, you need to decide if it is compulsory or optional. Compulsory parameters are necessary for the web service to perform its function. For example, a service to book a train will need the train time in order to make the booking.
• But you might make the "city" parameter of "what's the time?" optional - if it exists, return the time in that city - if not, just return the current time in the UK.
• All parameters, compulsory or optional, need to be checked for content. Either the value will have to be in a limited range (i.e. if the parameter value isn't "Y" or "N" then it is an invalid value). Or it might need to be a type of value (i.e. a number, or a string). Even in the case of parameters which allow "free text" values, you will still have to check for "dangerous" characters. Web services (and all programs executable by the web server) are vulnerable to security attacks, and you will need to make sure that incoming parameters don't contain anything dodgy.
• A good rule is to decide what values or kind of values you will allow in parameter values and build from there, rather than trying to disallow all the possibly dangerous characters (you may miss one). So a rule like "the value must be a mixture of letters, numbers, commas and whitespace" is better (or more secure) than "the value must not contain semicolons).
  

6. Building the query

• Having gathered and checked parameters (if any) the web service now has to perform its function. To do this, it needs to talk to the dataset - and the usual language for doing this is SQL.
• Web services which retrieve data will use SQL to ask the dataset a question - such as "What's the time in New York?"
• Web services which alter data will use SQL to form an instruction. For instance "Book a seat on the 9.15 from Cambridge to London"
• If your web services use parameters for input, these will (usually) be interpolated into the SQL. So, for the train booking example above, "9.15" would have been the value of the parameter "time", "Cambridge" the value of the parameter "from" and "London" the value of the parameter "to". Parameters are a means of altering the question or instruction which gets sent to the dataset
  

7. Returning the response

• All web services should send a response back to whoever made the request.
  
• For web services which retrieve data, this will be the answer to the question - i.e. for question "What is the time in New York" the response might be "The time in New York is 10.30".
• For web services which alter data, the response will tell you what the web service has done. So for the instruction "Book me a seat on the 9.15 from Cambridge to London" the response might be "Seat booked" or even the more verbose "Seat no. XX booked on the 9.15 from Cambridge to London".
  
• The response should be tailored to how it's going to be used. Sometimes, you'll just want to know if the request was processed ok, sometimes you'll need much more detailed information.
• Your web services should always send a response, even if the request fails. A response of "your request has failed" is far more useful than nothing at all!

8. Chaining web services together

• A common way of using web services is to chain them together - which means using the response from one web service as part of the input for the next.
• Taking the (very simplified) example of booking a seat on the next train, the chain might go something like this:
• Request to "what's the time?" returns value TIME
• Request to "what's the next train?" with parameter TIME returns next train after time as value TRAIN
• Request to "book seat" with parameter TRAIN returns value SEAT BOOKED

Widgets and web services

Behind (most) widgets you’ll find one or more data sources – whether a relational database, a bunch of linked data or even a text file. The widget is basically an interface to this data. The relationships between widgets and their data sources fall into two main types:

• Retrieval: the widget asks the data source a question and uses the answer for some purpose (usually display)
  
• Alteration: the widget causes a change to be made in the data source

So, in our Library widget, an example of retrieval would be to show a list of the items a user has on loan (no change to underlying data). And an example of alteration would be renewing one of these items (change made in underlying data).

Which means that widgets need a way of talking to their data sources.

Most widgets talk to their data sources through web services (also known as APIs) - little scripts which sit on your web server and mediate between your widget and your data.

They work something like this - your widget makes a request to a web service - either that some data be retrieved, or that a change be made in the data. The web service translates the request into data-speak and sends it on to the data source. The data source processes the request, makes any alterations, and sends back a response. The web service translates the response into widget-speak and sends it back to the widget.

So, why can’t widgets talk to their data sources directly? Why the need for the middle-man? There are four main reasons:

• Security – if your widget is talking to the data source directly, it is likely to contain passwords. Because widgets are usually written in html/javascript, pretty much anyone can access the source code and so access your passwords.
• Efficiency – web services are reusable. Say you have three widgets that need the same bit of data. You can write one web service to make the request and do any processing on the response, and all three widgets can use it.
• Ease of use – most widgets are written in html and javascript. Basically, it’s pretty easy to make calls to web services from javascript. Making calls to data sources directly from widgets is a bit of a nightmare.
• Innovation – if you have a bunch of web services sitting on your server, you can choose to publish them. This means that other developers can build systems and functionality around your data. Which is not only pretty exciting, but also saves you having to do everything yourself.

Many systems now come with their own set of proprietary web services for talking to their data source. These should be written up in your system’s documentation. If the system has no web services or it doesn’t have a web service for what you want to do, it’s pretty simple to write your own.

EXCEPT - reflecting the two types of request you can make to your data source, there are two types of web service - those that retrieve data and those that make alterations to your data. Most systems will allow you to write your own web services to retrieve data BUT most systems will insist that you use their own proprietary web services to alter data in the system.

So, going back to our original example, we have a locally written web service which retrieves a list of a user’s loans, but we use a proprietary system web service to renew one of those loans.

Our widgets use a mixture of locally-written and proprietary web services to relate to data sources, and (unless your system’s proprietary web services are truly excellent) you’ll probably find yourself doing the same. I’ll stop here before this post turns into an essay, but in future posts I’ll outline how to write web services, and how to use them.