[Pharo-project] Towards better HTTP client usage

Sept. 4, 2011

      Towards better HTTP client usage

At the last Pharo Sprint in Lille (July 8th), StÃ©phane and I were at one point trying to remove some old HTTPSocket usage (which indirectly uses Zn) and replace it with direct and clean Zn usage. We hadn't much time left and didn't get very far. But I realized afterwards that although technically everything was there to write good HTTP client code, it was way too difficult (it required to much code on the user's behalf).

Hence I decided to try write a new client that would support this usage much better and encourage better HTTP client usage in general. Another goal of this new client is to ultimately replace all other clients currently in Zn (these different client are creating some confusion among users).

ZnNeoClient (a temporary name, to be renamed to ZnClient in the future) is a single object with a lot of (convenience) API to build, execute and process HTTP client requests. It is somewhat similar to Gofer like classes. It is also somewhat comparable to ZnHttpClient but it contains even more functionality. 

The new tests in ZnNeoClientTests show some of the ways the client can be used. In the current Zn version all ZnClient class functionality as well as all ZnHTTPSocketFacade (and thus HTTPSocket and thus the rest of the image) has already been reimplemented using ZnNeoClient. These are good examples to look at.

At one point I plan to write some proper documentation, when the design settles down a bit, but here are some examples:

The simplest possible usage:

ZnNeoClient new get: 'http://zn.stxfx.eu/zn/small.html'.

This is shorthand for:

ZnNeoClient new
	url: 'http://zn.stxfx.eu/zn/small.html';
	get.

Which is actually shorthand for:

ZnNeoClient new
	url: 'http://zn.stxfx.eu/zn/small.html';
	get;
	contents.

If you know upfront that you will do only one request, you can help conserve resources by doing:

ZnNeoClient new
	beOneShot; 
	get: 'http://zn.stxfx.eu/zn/small.html'.

Connections are kept open whenever possible unless #beOneShot is chosen. Multiple requests (possibly to the same host) can be issued using the same client instance. At the end, #close should be send to the client, but garbage collection cleans up as well.

Specifying URLs as strings can sometimes be tricky when special characters such as spaces are involved, for this ZnNeoClient has a whole URL and request construction API. Here is an example:

ZnNeoClient new
	http;
	host: 'www.google.com';
	addPath: 'search';
	queryAt: 'q' put: 'Pharo Smalltalk';
	get.

There is also API for specifying headers, forms, multipart uploads and so on. See the unit tests, the ZnClient class side as well as the ZnHTTPSocketFacade for examples.

The result of an HTTP request is an HTTP response, modelled by ZnResponse. Hence, you can ask a client for the last #response, #entity, #contents or test for #isSuccess.

Which brings us to the problem that it is not easy to properly set up and handle the various things that can go wrong. It is here that ZnNeoClient aims to make a big contribution.

Assume the following example: at some URL there is a list of numbers as lines in a text file that we want to download and use. Let's start simple:

^ ZnNeoClient new
	get: 'http://www.example.com/numbers.txt'.

When all is well, this will return the text file as string. Let's add parsing:

^ ZnNeoClient new
	contentReader: [ :entity | 
		(entity contents lines do: [ :each |
			Integer readFrom: each ifFail: [ nil ] ])
			select: [ :each | each notNil ] ];
	get: 'http://www.example.com/numbers.txt'.

This will make sure we get a (possibly empty) list of numbers. The problem is, when number.txt is not found by the server, we don't deal properly with that situation. We can fix that:

^ ZnNeoClient new
	enforceHttpSuccess: true;
	contentReader: [ :entity | 
		(entity contents lines do: [ :each |
			Integer readFrom: each ifFail: [ nil ] ])
			select: [ :each | each notNil ] ];
	get: 'http://www.example.com/numbers.txt'.

Now, a non-success code will throw an error. We should also make sure that we do get a text/plain document back, and we want a uniform error handler reaction:

^ ZnNeoClient new
	enforceHttpSuccess: true;
	enforceAcceptContentType: true;
	accept: ZnMimeType textPlain;
	contentReader: [ :entity | 
		(entity contents lines do: [ :each |
			Integer readFrom: each ifFail: [ nil ] ])
			select: [ :each | each notNil ] ];
	ifFail: [ :exception |
		self log: exception printString, ' while fetching numbers'.
		^ #() ];
	get: 'http://www.example.com/numbers.txt'.

The failBlock will be execute on any exception, including ZnHttpUnsuccessful and ZnUnexpectedContentType. 

What about unreliable networking in general: we need a timeout, but better still, we could retry the request once or twice to cover for (possibly transient) networking/server problems.

^ ZnNeoClient new
	timeout: 15;
	numberOfRetries: 1;
	retryDelay: 2;
	enforceHttpSuccess: true;
	enforceAcceptContentType: true;
	accept: ZnMimeType textPlain;
	contentReader: [ :entity | 
		(entity contents lines do: [ :each |
			Integer readFrom: each ifFail: [ nil ] ])
			select: [ :each | each notNil ] ];
	ifFail: [ :exception |
		self log: exception printString, ' while fetching numbers'.
		^ #() ];
	get: 'http://www.example.com/numbers.txt'.

If anything goes wrong, there will be one retry after a delay of 2 seconds. 

Although there are sensible defaults for all options, it will probably make sense to group the options, like this:

^ ZnNeoClient new
	systemPolicy;
	accept: ZnMimeType textPlain;
	contentReader: [ :entity | 
		(entity contents lines do: [ :each |
			Integer readFrom: each ifFail: [ nil ] ])
			select: [ :each | each notNil ] ];
	ifFail: [ :exception |
		self log: exception printString, ' while fetching numbers'.
		^ #() ];
	get: 'http://www.example.com/numbers.txt'.

Finding the proper defaults and policies will take some trial and error. Another strategy is factory method like ZnClient class>>#client or ZnHTTPSocketFacade class>>#client. 

Here is another real-world example, invoking a REST service that returns JSON (you need the JSJsonParser class that comes with Seaside):

^ ZnNeoClient new
	systemPolicy;
	beOneShot;
	url: 'http://easy.t3-platform.net/rest/geo-ip';
	queryAt: 'address' put: '81.83.7.35';
	accept: ZnMimeType applicationJson;
	contentReader: [ :entity |
		(JSJsonParser parse: entity contents) at: #country ];
	ifFail: [ nil ];
	get.

Or as a utility method:

XYZUtils class>>#countryForIpAddress: ipAddressString ifFail: failBlock
	^ ZnNeoClient new
		systemPolicy;
		beOneShot;
		url: 'http://easy.t3-platform.net/rest/geo-ip';
		queryAt: 'address' put: ipAddressString;
		accept: ZnMimeType applicationJson;
		contentReader: [ :entity |
			(JSJsonParser parse: entity contents) at: #country ];
		ifFail: failBlock;
		get

That's it for now. There is still some implementation and testing work to be done. As always, all feedback is welcome.

Sven

[Pharo-project] Towards better HTTP client usage

Sven Van Caekenberghe