Electrifying APIs With Websockets: WAMP + REST
Websockets is a fantastic real-time technology which has struggled to catch on because of the significant knowledge gap between fast-moving frontend developers, and more traditional server-side programmers. To understand how to bridge the divide, there's a significant investment of time and energy required to ascend the learning curve.
What's All This Trendy Socket Business?
If you're not familiar with the concept of a Unix socket, it's a persistent two-way connection between computers and/or software programs which occurs over a numbered port. In the most well-known paradigm, clients connect to servers, and ask them for things. For example, an HTTP server typically runs over the TCP protocol on ports 80 and 443. A database server might provide a local file on the OS hard disk to serve the same purpose inside the computer (e.g. /var/run/database.sock).
10 years ago, in 2008, Websockets was introduced into the HTML5 specification. Around 2011 they were working in major browsers. The full specification lives here: https://html.spec.whatwg.org/multipage/web-sockets.html
In typical HTTP transactions, the client (i.e. your browser) completely forgets who you are, unless it uses session management to remember you. Each request is its own entity and lifecycle. What's different with Websockets is the connection is two-way ("full-duplex") and persists, like a tunnel. There is one connection, at the beginning; then the client and server are permanently connected together, swapping data and communicating. That is, until one of them disconnects.
Perhaps the best way to conceptualise it is as a connection which is "always open".
The most obvious application for it was online chat. In the past, you would have to make an HTTP request every second to get the latest information ("long-polling", as exemplified by Comet: https://en.wikipedia.org/wiki/Comet_(programming)), which began to be superseded by servers like Openfire (https://igniterealtime.org/projects/openfire/) and eJabberD (https://www.ejabberd.im/) which used protocols like Extensive Messaging and Presence Protocol (XMPP: https://xmpp.org/).
Applications like Slack are a prime example of how early adopters embraced the technology to revolutionise how browsers communicate over the web.
Not Everything Requires the Cool New Thing
In most situations, AJAX polling and REST APIs work just fine without any need for the cool new thing. But in a subset of circumstances, Websockets is a quantum leap of efficiency and usability. Almost all of them have a need for real-time information which can't wait for an API call to complete, or benefit from millisecond-style response times.
Some of these are:
- Social feeds
- Chat
- Multiplayer games
- Telemetry
- Collaborative writing
- Event logging
- Financial feeds
- e-Sports
- Location tracking
- Teaching chalkboards
In a typical development team, most backend and devops people are focused on building REST APIs with JSON. XML became too cumbersome, and standardised verbs like PUT and DELETE provide an simple abstract protocol which has a "neutrality" to insulate them from over-zealous innovators.
Conversely, most UI developers have been focusing on Javascript as a universal language, due to the momentum behind the fantastic NodeJS. Using frameworks like Axios (https://github.com/axios/axios), it's trivial to separate your Javascript-based application - which can be deployed across web, desktop, and mobile - from your REST backend. When they are separate, less problems are prone to occur, and development is simplified.
Inverting that idea is also a fun idea, and describes the concept of isomorphic applications (https://en.wikipedia.org/wiki/Isomorphic_JavaScript), which use the same language on the front and back ends.
The question then becomes how the systems talk to one another. Websockets is merely a transport, and leaves the implementation up to engineers. Launching a connection in Javascript, for example, is simple:
var connection = new WebSocket('ws://domain.com/socket', ['soap', 'xmpp']);
// When the connection is open, send some data to the server
connection.onopen = function () {
connection.send('Ping'); // Send the message 'Ping' to the server
};
// Log errors
connection.onerror = function (error) {
console.log('WebSocket Error ' + error);
};
// Log messages from the server
connection.onmessage = function (e) {
console.log('Server: ' + e.data);
};Any engineer's immediate thought is the next step has to be sending JSON, instead of plain text.
- CORBA (https://en.wikipedia.org/wiki/Common_Object_Request_Broker_Architecture) , for example, was already doing this in 1991.
- Financial markets have been using AMQP (http://www.amqp.org/) for the same purpose for a while. When they are not making everyone's lives hell with SOAP.
- In more recent times, the Internet of Things has been relying more and more on lightweight, old-school MQTT (https://mqtt.org/).
You can also send binary data, so you're thinking WebRTC audio and video. Websockets doesn't care. You can do as you wish. All it does is provide a two-way socket connection over HTTP; what you send and receive is not its concern.
That is where the problem lies: rolling your own protocol. More on that later.
One-Way Satellite Transmission
Before we dive into two-way communication in real-time, we have to be judicious enough to pull back and ask the same question again: is it actually necessary? What if all you need to do is push information sporadically, such as an update on an e-commerce order, or the next TV show/song coming up?
HTML5 introduces another interesting technology in its specification which can do a similiar job: Server-Sent Events (SSEs).
SSEs are inherently one-way communication, and can be thought of, in some broad sense, as an analogue of a satellite broadcast. If you are only looking to broadcast in one direction to an entire audience, a group, or individual users, and have no need for the client to talk back, they can be much simpler to implement.
However, they come at a cost: they are computationally expensive. The idea is slightly deceptive at first look, because it implies one source broadcasting (multicasting) to multiple clients. However, this is not the case. Each client makes its own connection to an open-ended CPU process.
If you have 40,000 clients, you will have 40,000 subscriptions polling your event stream. If each one is a personalised event stream, the load is increased further. But if you need no input from the client at all, they may just be a faster option.
An example of providing an event stream from the server side in Laravel using Symfony's StreamedResponse class:
$response = new StreamedResponse(function() use ($request) {
while(true) {
echo 'data: ' . json_encode(Model::all()) . "\n\n";
ob_flush();
flush();
usleep(200000);
}
});
$response->headers->set('Content-Type', 'text/event-stream');
$response->headers->set('X-Accel-Buffering', 'no');
$response->headers->set('Cache-Control', 'no-cache');
return $response;On the client side, a sequence of lines of text are received by the browser, which can subscribe natively to them:
if (!window.EventSource) {
// IE or an old browser
alert("The browser doesn't support EventSource.");
return;
}
let eventSource = new EventSource("https://site.com/user/1/event-feed", {
withCredentials: true
});
eventSource.onopen = function(e) {
console.log("Event: open");
};
eventSource.onmessage = function(e) {
console.log("Event: message, data: " + e.data);
};
eventSource.onerror = function(e) {
console.log("Event: error");
if (this.readyState == EventSource.CONNECTING) {
console.log(`Reconnecting (readyState=${this.readyState})...`);
} else {
console.log(e);
}
};
eventSource.addEventListener('some_custom_event', function(e) {
console.log("Event: some_custom_event, data: " + e.data);
});
eventSource.close();Messages might look this:
data: {"user":"Jane Doe","message":"Some important text or data"}
data: This is an event message
data: This is another message
data: Here is a third messageMore info at the official spec: https://html.spec.whatwg.org/multipage/server-sent-events.html#the-eventsource-interface
What that may also mean as a developer is you set out to implement a real-time data feed (e.g. stock prices, social activity, sports scores etc), and then figure out, along the line, the client needs to provide the server with some instructions.
Your viewer/user might not want all the scores for all the sports teams in every league in their area, and the sheer volume of data in the satellite broadcast might be getting cumbersome. The information might need to be filtered or updated through their input.
That's where Websockets typically come in.
How Do You Test Websockets?
Well, if you're working with a REST API, you'd typically be working with Postman (https://www.postman.com/) or Insomnia (https://insomnia.rest/). You need a Websocket Client app for development.
There are a good few out there. A pretty fun one similar to Postman is WebSocket King Client (https://websocketking.com/) which can be added to any Chromium browser (Chrome, Brave etc) as an extension.
App: https://chrome.google.com/webstore/detail/websocket-king-client/cbcbkhdmedgianpaifchdaddpnmgnknn
Blocking Backends & Legacy Languages
With a title like that, it doesn't really sound too appealing; however, the simple reason it's hard to deal with Websockets in legacy applications is the stateless nature of the HTTP protocol in itself.
In almost all languages, the fulfillment of an HTTP request is a synchronous, round-trip affair. The client has to wait for it, and has the result delivered in one go: it supplies an "order" to the server, and the server produces a reply - usually by consulting a database or other components. The response is blocked until it has finished. A non-blocking process can complete without carrying out all its functions immediately by delegating some of its work to an asynchronous background helper.
In languages such as C/C++, Java, C#, Perl, PHP, Python, Ruby, and so on, a single server process typically handles a single request; it builds the response (typically an HTML page) and sends it back once it has finished its job.
You can try proxy apps like Websocketd (http://websocketd.com/) but they still leave you with the same protocol problem.
To implement Websockets, most server-side languages need to run in a loop and relay messages. This is the foundation of event-driven programming: https://en.wikipedia.org/wiki/Event-driven_programming. Also see: https://en.wikipedia.org/wiki/Event-driven_architecture.
Python has Tornado (https://www.tornadoweb.org/en/stable/index.html), Java has Spring (https://spring.io/), Ruby has Resugan (https://github.com/jedld/resugan), and Go has Watermill (https://watermill.io/)
It's at this point where you will hear the protests and smug commentary from beanie-wearing "artisans" who want to explain, in great detail, why everyone should just use Node. Node's a fantastic framework, and is explosive when it comes to innovation - so there's some truth to that claim. Sadly, not everyone can suddenly uproot millions of dollars of infrastructure just to appear cool. Many of us have to contend with previous choices made at a business level, as Facebook so infamously experienced.
The most renowned implementation of Websockets in PHP, - the most widely-spread language, if the most hated -, is Ratchet (http://socketo.me/), which is also the basis of the popular Laravel package laravel-websockets (https://docs.beyondco.de/laravel-websockets/). It based on ReactPHP (https://reactphp.org/) which, like Swoole (https://www.swoole.co.uk/), somewhat emulates Node.
$server = IoServer::factory(
new HttpServer(
new WsServer(
new Chat()
)
),
8080
);
$server->run();For the most part, it does a good job. Services like Pusher (https://pusher.com/) do an amazing job of making the "broadcast" element simple to use, allowing servers to talk to raw Javascript clients.
In most instances, Pusher or Ratchet can work fine for sending basic messages down the wire to clients in any format you like - XML, JSON etc. A browser simply makes a connection to the service on the specified port, and upgrades the connection; the server manages a pool of clients to broadcast between. The browser just needs to know what it's receiving and how to work with it. The negotiation is done between the bearded IT guy and the hoodie-wearing designer down the hall.
2 interesting points of contention are related to security, and are worth knowing:
- Websockets can run over SSL, using the
wss://prefix; - Websockets supports the Basic Authentication syntax, e.g.
ws://username:[email protected]:6001
The problem comes when the Javascript client needs to talk back. Things get very complicated, very quickly. And particularly, when you're on the 2nd project and have to do it again.
Introducing Web Application Messaging Protocol (WAMP)
The most crucial thing to understand about WebSockets is it is layers of protocols: the Websockets protocol sits on top of the Hypertext Protocol (HTTP), which in turn sits on top of its transport (TCP). It become apparent you need to choose one if you're going to work with different projects and want to keep your sanity.
Now, on top of this, Websockets supports one or many sub-protocols. They can arbitrarily defined and many of them are added to registries, such as IANA's: https://www.iana.org/assignments/websocket/websocket.xml
A standardised protocol is useful because it relieves the need to define your own exchanges, and both client and server automatically know how to talk to each other.
In any journey to understanding Ratchet, you'll come across WAMP: Web Application Messaging Protocol (v1) which was created in 2012, and aims to compete with Meteor: https://en.wikipedia.org/wiki/Meteor_(web_framework)
- http://socketo.me/docs/wamp
- https://en.wikipedia.org/wiki/Web_Application_Messaging_Protocol
- https://wamp-proto.org/
WAMP is a huge mission in and of itself, but it implements the Pub/Sub pattern (https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern) as also used by programs like Redis. It also helpfully include remote procedure calls (RPCs). As the introduction describes it:
"Publish & Subscribe (PubSub) is an established messaging pattern where a component, the Subscriber, informs the router that it wants to receive information on a topic (i.e., it subscribes to a topic). Another component, a Publisher, can then publish to this topic, and the router distributes events to all Subscribers."
"Routed Remote Procedure Calls (rRPCs) rely on the same sort of decoupling that is used by the Publish & Subscribe pattern. A component, the Callee, announces to the router that it provides a certain procedure, identified by a procedure name. Other components, Callers, can then call the procedure, with the router invoking the procedure on the Callee, receiving the procedure’s result, and then forwarding this result back to the Caller. Routed RPCs differ from traditional client-server RPCs in that the router serves as an intermediary between the Caller and the Callee."
To save yourself a lot of heartache, understand this one simple issue: Ratchet and the packages built on top of it only support WAMP v1. The current version of WAMP is v2. If you want to use that, you'll need earlier'10-year-old legacy JS libraries.
Sadly, there's more trouble on the way. The only 2 PHP frameworks which support version 2 are Thruway (https://github.com/voryx/Thruway) and Minion (https://github.com/Vinelab/minion).
The additional problem? Almost none of these have been updated for years, and they rely on Symfony 4, which arrived in 2017. Symfony is currently on version 5.1, and its changes to the event-dispatcher component completely break Thruway. More modern frameworks using Symfony 5 are not compatible with the existing WAMP2 libraries.
Symfony 5 support isn't coming any time soon: https://github.com/voryx/Thruway/pull/331
Introducing Crossbar & Autobahn To Save The Day
Thankfully, there is a smarter way to get the usefulness of WAMP2 connected into your REST application. The creators of it have a Python-based router/broker named Crossbar (https://crossbar.io/) and a corresponding out-of-the-box Javascript library named Autobahn (https://crossbar.io/autobahn/).
Technically Crossbar is a "router" in WAMP-speak, but you can think of it as an ESB, a broker, a "traffic junction" or a "Websocket server" to make things easier. All you need to know it is sits in front of your web app, and allows Javascript (or other) clients to talk Websockets to something you've already built which may not speak real-time wiring. It can do a lot more on top, but for now, we'll start simply.
Crossbar doesn't really have a UI (other than the console debug), but the managing organisation are expanding the ecosystem to provide graphical network monitoring:
First up, it's (literally) a snap to install:
sudo snap install crossbarIf you're on a VPS, or anything which does weird things with mounted drives, you'll need to pull the Git repo and install with Pip, because Snap works in a sandbox which doesn't play nicely with them (e.g. permission denied errors everywhere). Make sure you're using Python 3.6 with the equivalent updated Pip (v20 or similar).
git clone https://github.com/crossbario/crossbar.git
cd crossbar
pip install ..
If you're keen on Systemd, set up a .service file for it.
Next, we need to set up our first instance ("node"):
cd /path/to/my/chosen/dir
crossbar initThis will create a directory named .crossbar wherever you run it, which includes a file named config.json. Inside this, we tell the server what to do and how to run.
To run it:
crossbar start
Done. Your Websocket server is up and running with WAMP v2.
Broadcasting To Subscribed Clients
To get started, we can run unauthenticated connections in a local environment as we test. Within config.json, we can add these blocks to define some Crossbar workers. A worker is anything doing something inside Crossbar.
A topic can be anything: a "channel", a "room", a "feed", a "pool" - anything which has a defined identifier which the JS client knows about. A user's ID might be a topic; a news category might be one; a group's channel in our chat app.
Important point here: WAMP and Pub/Sub in general is whole motherload of terminology and conceptual jargon. It's a dry read, but it's worth getting your head around these ideas first. Anyone can publish or subscribe to a topic (channel), and anyone can register or call procedures. It's more akin to a peer-to-peer (P2P) model than server/client, and peers can be browser clients, or machine-to-machine. Pro-Tip know what you want to do first.
Our first decisions are:
- Topics and calls reside within a "realm". Ours will be our company "Acme".
- Our websocket server will run on port 7000 (ws://127.0.0.1:7000)
- Our app will need to "broadcast", which means it needs to publish to Crossbar topics (aka "channels", "accounts" or such).
- Our JS clients will need to call remote procedures, so Crossbar will need to make external calls to the server (or other APIs) and get a result.
/* Snipped for brevity */
"workers": [
{
"type": "router",
"realms": [
{
"name": "acme-company",
"roles": [
{
"name": "anonymous",
"permissions": [
{
"uri": "",
"match": "prefix",
"allow": {
"call": true,
"register": true,
"publish": true,
"subscribe": true
},
"disclose": {
"caller": false,
"publisher": false
},
"cache": true
}
]
}
]
}
],
"transports": [
{
"type": "websocket",
"endpoint": {
"type": "tcp",
"port": 7000
}
},
{
"type": "web",
"endpoint": {
"type": "tcp",
"port": 8000,
"backlog": 1024
},
"paths": {
"publish": {
"type": "publisher",
"realm": "acme-company",
"role": "anonymous",
"options": {
"debug": true
}
},
"info": {
"type": "nodeinfo"
},
"ws": {
"type": "websocket"
},
"call": {
"type": "caller",
"realm": "acme-company",
"role": "anonymous",
"options": {
"debug": true
}
}
}
}
]
}
]In the above example:
We have created a realm named "acme-company" with one anonymous role which has permissions to both register and call procedures, and to both subscribe and publish to topics. Obviously we will need to change this if it runs outside a local environment.
Then we have registered 2 HTTP services:
- A Websocket interface on ws://127.0.0.1:7000;
- An internal Crossbar web server on port 8000 (http://127.0.0.1:8000), which has 4 routes:
- /publish (http://127.0.0.1:8000/publish)
- /info (http://127.0.0.1:8000/info)
- /ws (ws://127.0.0.1:7000/ws)
- /call (http://127.0.0.1:8000/call)
Our server therefore has a publisher running at /publish, an internal Websocket connection on /ws, and a caller at /call.
We have created an HTTP REST Bridge. Crossbar's documentation isn't great on this, but it has some useful examples:
When we want to broadcast an update to JS clients who are subscribed, our legacy REST app can perform an internal "outbound" POST to the local Crossbar web server with a simple JSON body:
public function broadcast(array $channels, $event, array $payload = [])
{
if ( empty($channels) ) {
return;
}
foreach (self::formatChannels($channels) as $channel)
{
$this->response = Http::withHeaders ([
'Content-Type' => 'application/json',
])->post ('http://127.0.0.1:8000/publish', [
'topic' => $channel,
'args' => [$payload],
]);
}
}The equivalent requesting cURL would be:
curl -H "Content-Type: application/json" \
-d '{"topic": "some-channel", "args": [{ "foo" : "bar"}]}' \
http://127.0.0.1:8000/publishJS clients subscribed to "some-channel" will receive the JSON payload [{ "foo" : "bar"}], i.e. an array containing an object.
Crossbar will respond with a basic 200 response code and a simple JSON object containing the ID of the published message.
On the other end, the JS client needs to connect and subscribe to the topics/channels it wants if it is going to receive the associated notifications and messages:
var connection = new autobahn.Connection({url: 'wss://127.0.0.1:9000/', realm: 'acme-company'});
connection.onopen = function (session) {
session.subscribe('activity_feed_all_users', function (args) {
console.log("Event:", args[0]);
});
session.subscribe('weather', function (args) {
console.log("Event:", args[0]);
});
session.subscribe('channel_5', function (args) {
console.log("Event:", args[0]);
});
};
connection.open();That's great if anyone can subscribe to channels for any reason, but what if we need clients to authenticate before connecting, and be authorized to see and do certain things?
Crossbar provides 7 different methods:
A "ticket" is analogous to an API token, but if our IDAM is contained within the legacy app, we need to use dynamic challenge-response authentication (dynamic WAMP-CRA). The full details of how this works is out of the scope here, but the gist is you register a remote procedure call to an "authenticator" of your choice, and allow/deny the connection based on its result:
"transports": [
{
"type": "web",
...
"paths": {
...
"ws": {
"type": "websocket",
"auth": {
"wampcra": {
"type": "dynamic",
"authenticator": "api.auth"
}
}
}
}
}Receiving Client Commands Over The Wire
Broadcasting is quite easy, as POST requests to an internal server are inherent in any server-side language. As long as your app can find the Crossbar server, it can publish a message into a topic all the connected clients subscribed to that topic will receive.
Crossbar has other components which allow apps to interact with topics and clients:
If we, for example, want our JS clients to be able to send simple Websocket messages to our REST server, we can set it up to be a subscriber to topics (channels). Crossbar will forward messages it receives as JSON posts in the other direction. Doing so is trivial:
{
"type": "container",
"options": {
"pythonpath": [
".."
]
},
"components": [
{
"type": "class",
"classname": "crossbar.bridge.rest.MessageForwarder",
"realm": "acme-company",
"extra": {
"subscriptions": [
{
"url": "http://myapp.local/path/to/post/receiver",
"topic": "breaking-news.",
"match": "prefix"
},
{
"url": "http://myapp.local/some/other/uri",
"topic": "user..orders",
"match": "wildcard"
}
],
"method": "POST",
"expectedcode": 200,
"debug": true
},
"transport": {
"type": "websocket",
"endpoint": {
"type": "tcp",
"host": "127.0.0.1",
"port": 7000
},
"url": "ws://127.0.0.1:7000/ws"
}
}
]
}What we've done above is instruct Crossbar to forward any messages sent to any topic starting with "breaking-news.*" to http://myapp.local/path/to/post/receiver in a POST, and any messages to any topic matching "user.*.orders" to http://myapp.local/some/other/uri.
These would match:
- breaking-news.politics
- breaking-news.technology
and:
- users.12.orders
- users.myusername.orders
- users.8595968474.orders
Each will receive a JSON payload such as:
{
"args": [
{
"foo": "bar"
}
],
"kwargs": [
{
"foo": "bar"
}
]
}But what if the client needs to talk to the server and call some kind of logic there once it's successfully connected? This is where Remote Procedure Calls come in. The security implications here are again obvious, so this is theory-only.
In WAMP2 and Crossbar teminology, the component which performs an outgoing request to a legacy app or external app is known as a Callee, and exists inside a Python container. It makes a REST call on the Websocket client's behalf and returns the result. It functions as a Callee middleman, processing a request for a Caller.
Callee docs: https://crossbar.io/docs/HTTP-Bridge-Callee/
Inversely, and confusingly, a Caller in WAMP2/Crossbar means a "callable" REST service you can make to the server. It's not immensely helpful when you're trying to get your head around things.
In our scenario, we want our Websocket clients to be able to remotely call 3 APIs: our internal API, Stripe, and Twilio.
After our realm and transports definitions, we add 3 containers:
"workers": [
/* SNIPPED FOR BREVITY */
{
"type": "container",
"options": {
"pythonpath": [
".."
]
},
"components": [
{
"type": "class",
"classname": "crossbar.bridge.rest.RESTCallee",
"realm": "acme-company",
"extra": {
"procedure": "webapp.api",
"baseurl": "http://api.myapp.local"
},
"transport": {
"type": "websocket",
"endpoint": {
"type": "tcp",
"host": "127.0.0.1",
"port": 7000
},
"url": "ws://127.0.0.1:7000/ws"
}
}
]
},
{
"type": "container",
"options": {
"pythonpath": [
".."
]
},
"components": [
{
"type": "class",
"classname": "crossbar.bridge.rest.RESTCallee",
"realm": "acme-company",
"extra": {
"procedure": "stripe.api",
"baseurl": "https://api.stripe.com"
},
"transport": {
"type": "websocket",
"endpoint": {
"type": "tcp",
"host": "127.0.0.1",
"port": 7000
},
"url": "ws://127.0.0.1:7000/ws"
}
}
]
},
{
"type": "container",
"options": {
"pythonpath": [
".."
]
},
"components": [
{
"type": "class",
"classname": "crossbar.bridge.rest.RESTCallee",
"realm": "acme-company",
"extra": {
"procedure": "twiio.api",
"baseurl": "https://api.twilio.com/2010-04-01/"
},
"transport": {
"type": "websocket",
"endpoint": {
"type": "tcp",
"host": "127.0.0.1",
"port": 7000
},
"url": "ws://127.0.0.1:7000/ws"
}
}
]
},
]Notice all of these are the same, and most don't contain any authentication information for the APIs themselves. We are asking Crossbar to create three internal containers for each REST service, and defining them as Callees who will make external calls on behalf of the user (Websocket client) and communicate the results internally back to the Websocket service on port 7000.
We define 3 remotes - which are referred to as "procedures". It's best to thing of these as "external services". Most is self-explanatory, apart from one thing:
What is useful to understand here is the way Crossbar talks to itself internally here is by using websockets. The transport part of the definition refers to our initial internal setup:
"paths": {
"publish": {
"type": "publisher",
"realm": "acme-company",
"role": "anonymous",
"options": {
"debug": true
}
},
"info": {
"type": "nodeinfo"
},
"ws": {
"type": "websocket" <----- HERE
}
}It's helpful to think of this as a "loopback" of sorts. It is telling the REST bridge where to send back the result of the API call.
Once Crossbar knows where to send the external REST call, the Websocket client can configure that call by specifying headers and parameters. We can use the internal REST Caller to invoke the internal Callee:
POST http://127.0.0.1:8000/call
{
"procedure" : "webapp.api",
"kwargs" : {
"url" : "someresource/123456/subresource",
"headers" : {
"Accept" : ["application\/json"],
"Authorization" : ["Basic cnBjOnNlY3JldA=="]
},
"method" : "POST",
"params" : [
{
"foo" : "bar"
}
]
}
}This will POST to http://api.myapp.local/someresource/123456/subresource with Basic (or Bearer) headers.
Crossbar will return the full HTTP response from the external server - always with a 200 response, even if the reply is the call is unauthorized because it doesn't have a JWT Token. The JSON reply is in the content field.
{
"args": [
{
"code": 401,
"content": "{\"message\":\"Unauthenticated.\"}",
"headers": {
"Server": [
"nginx/1.14.0 (Ubuntu)"
],
"Content-Type": [
"application/json"
],
"Cache-Control": [
"no-cache, private"
],
"Date": [
"Fri, 22 May 2020 22:21:38 GMT"
],
"Content-Encoding": [
""
]
}
}
]
}From our JS client, the call is similar: using session.call (procedure, args, kwargs) we can embed the same kwargs from the REST call:
session.call('webapp.api', [somearg1, somearg1], {method: 'POST', url: 'resource/3/test', headers: {}, params: {foo: 'bar'}}).then(
function (res) {
console.log("api call result:", res);
},
function (err) {
console.log("print api call error:", err);
}
);More RPC examples: https://github.com/crossbario/autobahn-python/blob/master/examples/twisted/wamp/rpc/arguments/frontend.js
Putting It Together: Nginx + Autobahn
You can run websockets in the clear using port 80, or the TLS version on port 443, in typical HTTP style. The chances are you will run it next to an existing app, so you'll need to proxy the socket connection with Nginx.
Thankfully, it's simple, but remember to ramp up your Nginx workers to match the additional connections and to factor in CORS:
location / {
proxy_pass http://127.0.0.1:7000;
proxy_read_timeout 60;
proxy_connect_timeout 60;
proxy_redirect off;
proxy_http_version 1.1; # DON'T set this 1.0 as Nginx recommends
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
if ($request_method = 'OPTIONS') {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow_Credentials' 'true';
add_header 'Access-Control-Allow-Headers' 'Authorization,Accept,Origin,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
add_header 'Access-Control-Allow-Methods' 'GET,POST,OPTIONS,PUT,DELETE,PATCH';
add_header 'Access-Control-Max-Age' 1728000;
add_header 'Content-Type' 'text/plain charset=UTF-8';
add_header 'Content-Length' 0;
return 204;
}
}If you want to run them side by side, you can switch how the request is handled by looking at the upgrade header being received:
map $http_upgrade $type {
default "web";
websocket "ws";
}
server {
location @web {
try_files $uri $uri/ /index.php?$query_string;
}
location @ws {
proxy_pass http://127.0.0.1:6001;
proxy_set_header Host $host;
proxy_read_timeout 60;
proxy_connect_timeout 60;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}More: https://www.nginx.com/blog/websocket-nginx/
A simple first step is to connect to the WebSocket server when you have it running, to see the WAMP handshake at work. In Chromium's DevTools, the Network tab has a selector named "WS" which records WebSockets traffic.
A "startup" handshake goes like this in Autobahn:
GET wss://sock.somedomain.dev/ HTTP/1.1
Host: somedomain.dev
Connection: Upgrade
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket
Origin: http://www.someui.com
Sec-WebSocket-Version: 13
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Sec-WebSocket-Key: lSxBD/bnPgwe8Ly1xA7Q0Q==
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Sec-WebSocket-Protocol: wamp.2.json, wamp.2.msgpackThe response, when going through Cloudflare, looks as so:
HTTP/1.1 101 Switching Protocols
Date: Sun, 24 May 2020 01:12:56 GMT
Connection: upgrade
Upgrade: WebSocket
Sec-WebSocket-Protocol: wamp.2.json
Sec-WebSocket-Accept: xrWBLU0ruSThcTikTej5hj7mhoA=
CF-Cache-Status: DYNAMIC
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 598326b9eb72e3ca-ATL
cf-request-id: 02e5d688310000e3cab43ee200000001A green dot with the "Switching Protocols" 101 status is good news.
The key part to notice is this:
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Sec-WebSocket-Protocol: wamp.2.json, wamp.2.msgpackYou must specify you want to use the wamp.2.json sub-protocol. If you don't, the Crossbar server won't connect. The first incarnation listed at IANA (and within Ratchet) is simply named wamp.
Like usual, we can import AutobahnJS from a CDN, and maybe if we've done OAuth2 authentication in the browser, we can place our JWT token in the DOM or localStorage for easy access:
<meta name="webapp-access-token" content="eyJ0eXAiOiJKV1QiLCJ....">
<meta name="stripe-access-token" content="eyJ0eXAiOiJKV1QiLCJ....">
<meta name="twilio-access-token" content="eyJ0eXAiOiJKV1QiLCJ....">
<script src="https://cdn.jsdelivr.net/npm/[email protected]/autobahn.min.js"></script>When the page loads, we want to subscribe to a few channels (Slack-style), and call a few procedures as we go along:
var connection = new autobahn.Connection ({
url: 'wss://realtime.somedomain.dev',
realm: 'acme-company',
authmethods: ["wampcra"],
authid: "name-or-token",
onchallenge: function (session, method, extra) {
if (method === "wampcra") {
return autobahn.auth_cra.sign("secretpassword", extra.challenge);
}
}
});
connection.onopen = function (session) {
session.subscribe('#general', function (args) {
// do something when an event happens
});
session.subscribe('#random', function (args) {
// do something when an event happens
});
session.subscribe('@myusername', function (args) {
// do something when an event happens
});
};
// Call webapp API
session.call('webapp.api', [somearg1, somearg1], {method: 'POST', url: 'endpoint/ID/updates', headers: {"Authorization" : ["Basic base64encoded=="]}, params: {foo: 'bar'}}).then(
function (res) {
console.log("api call result:", res);
},
function (err) {
console.log("api call error:", err);
}
);
// Call Stripe API
session.call('stripe.api', [somearg1, somearg1], {method: 'POST', url: 'v1/charges', headers: {"Authorization" : ["Bearer stripe-api-token-from-meta-tag"]}, params: {amount: 100}}).then(
function (res) {
console.log("api call result:", res);
},
function (err) {
console.log("api call error:", err);
}
);
// Call Twilio API
session.call('twilio.api', [somearg1, somearg1], {method: 'POST', url: 'Accounts/ACXXXXXX/Messages.json', headers: {"Authorization" : ["Bearer twilio-api-token-from-meta-tag"]}, params: {Body: 'here is an SMS'}}).then(
function (res) {
console.log("api call result:", res);
},
function (err) {
console.log("api call error:", err);
}
);
connection.open();In this example, our client doesn't talk to other clients. It doesn't publish to topics (channels). Of course, it can.
session.publish('#general', ['{message:"Hello!"}']);If you were really feeling funky, the client could offer a calculator function to other clients who are subscribed. First, it needs to register the procedure so others can call it, then it needs to respond to requests.
function calculator(args) {
return args[0] + args[1];
}
session.register('my_cool_calculator', calculator);Docs: https://github.com/crossbario/autobahn-js/blob/master/doc/reference.md
Breathe Out, Then In, Then Out Again
WebSockets don't have to be painful. They are. But the benefits to using them for the right situation are innumerable. The important thing is to know what you need to do, and be clear about how you want it to work. WebSockets aren't a Golden Hammer to fix any nail; they are, however, a powerful way to create a cabled live "wire" which is real-time and able to react more quickly to what's happening on the system connecting with it.
What it gives you comes at a price. If you want the benefits, it's going to hurt more to develop it.
Key takeaways:
- There is no reason to just to use WebSockets for the sake of replacing REST calls. There is arguably not a lot of point to using them to do REST over the socket either. If your request can work synchronously, don't fix what's not broken.
- Consider Server-Sent-Events if you're sure you're never going to need input from the client and the broadcast is one-way.
- Async event-driven apps are best done in Node. If you can't migrate, use a broker like Crossbar with a REST bridge.
- Websocketd, Pusher, Ratchet etc all work well doing a "push" until there's a crash, or your clients need to interact with the "brain" of things.
- Rolling your own Websocket protocol in XML or JSON is a bad idea.
- Authentication and authorization are complex. The form occurs once at the beginning, not on each request.
- Understand the Pub/Sub pattern as found in Redis, RabbitMQ, and others.
- Study the sub-protocols which work over WebSockets. It might be that MQTT or some other variant is what you need.
If you get it right, it's genuinely... magic.