Realtime Differential Synchronization Protocol
Real-time collaborative editing requires solving some unique problems. That's why we've built our own protocol for synchronizing data across editors. Our Realtime Differential Synchronization Protocol (RDSP) is designed to efficiently ship changes between clients.
Right now, RDSP is best thought of as a beta. Many parts could be improved. This is because we optimized for time-to-implement, not overall quality. There are definitely some bone-headed parts.
Network connection
By default, RDSP connects on TCP port 3448. The connection is TLS-encrypted. Certificate and hostname verification follow that of browsers. Other transports are possible. For example, our web editor communicates over a web socket.
Data format
Currently, all messages are sent as newline-delimited JSON. To make them easier to read, example messages on this page are pretty-printed.
In the future, a different hierarchical key-value serialization format may be chosen. BSON is a promising candidate.
Implementation note: Currently, messages are limited to 30MB in length.
Common message attributes
Messages sent to the server should have a req_id
attribute. Its value can be any integer from 1 to 2147483648. The only limitation is that it cannot be the same as an outstanding request. All messages with a req_id
will get a response from the server containing a res_id
. This response may be an error message...
{
"name": "error",
"res_id": 81,
"msg": "User doesn't exist.",
"flash": true
}
...a "reflected" message...
{
"name": "create_buf",
"res_id": 10,
"path": "example.txt",
"id": 4,
"encoding": "utf8"
"md5": "ddce269a1e3d054cae349621c198dd52",
"buf": "example\n"
}
...or just an acknowledgement.
{
"name": "ack",
"res_id": 13
}
If a req_id
is not specified, responses may still contain a res_id
. These will be increasing integers.
The server may also send messages that do not correspond to a client's requests. These will not contain a res_id
.
Message types
Authentication
Upon connecting, clients must send an authentication message. This consists of a username or API key, an API secret, and a path to join. Optional components include: supported encodings (currently base64 and utf8), protocol version, and client info such as the operating system. A typical auth message looks like this:
{
"api_key": "user_1234",
"username": "ggreer",
"secret": "apisecret1234",
"path": "ggreer/ag",
"client": "Atom",
"platform": "darwin",
"supported_encodings": ["utf8", "base64"],
"version": "0.11"
}
The response to an auth message is either room_info
or an error message followed by a disconnect. The contents of room_info
include the directory tree for the workspace, a list of users, your permissions, and the checksums of all buffers in the workspace.
{
"anon_perms": [],
"bufs": {
"1": {
"path": "FLOOBITS_README.md",
"id": 1,
"md5": "d80d9f9ffbc6f6a50bd6c9d8e3170f72",
"encoding": "utf8"
}
},
"created_at": 1405639844470,
"updated_at": 1447967922578,
"max_size": 209715200,
"owner": "ggreer",
"room_name": "ag",
"secret": false,
"temp_data": {},
"terms": {},
"tree": {
"FLOOBITS_README.md": 1
},
"users": {
"80000": {
"client": "Chrome",
"user_id": 80000,
"gravatar": "https:\/\/secure.gravatar.com\/avatar\/17bb58f556392c750c6c4140a9d35a56",
"is_anon": false,
"perms": [
"get_buf", "request_perms", "kick", "pull_repo", "perms", "solicit", "ping", "pong", "webrtc", "patch", "set_buf", "create_buf", "delete_buf", "rename_buf", "set_temp_data", "delete_temp_data", "highlight", "msg", "datamsg", "create_term", "term_stdin", "delete_term", "update_term", "term_stdout", "saved"
],
"platform": "darwin",
"color": "fuchsia",
"username": "kansface",
"version": 0.11,
"video_chatting": false
},
"80844": {
"client": "Chrome",
"user_id": 80844,
"gravatar": "https:\/\/secure.gravatar.com\/avatar\/293bdb3d59889fb65474fe6f37fd5c8d",
"is_anon": false,
"perms": [
"get_buf", "request_perms", "kick", "pull_repo", "perms", "solicit", "ping", "pong", "webrtc", "patch", "set_buf", "create_buf", "delete_buf", "rename_buf", "set_temp_data", "delete_temp_data", "highlight", "msg", "datamsg", "create_term", "term_stdin", "delete_term", "update_term", "term_stdout", "saved"
],
"platform": "darwin",
"color": "lime",
"username": "ggreer",
"version": 0.11,
"video_chatting": false
}
},
"name": "room_info",
"perms": [
"get_buf", "request_perms", "kick", "pull_repo", "perms", "solicit", "ping", "pong", "webrtc", "patch", "set_buf", "create_buf", "delete_buf", "rename_buf", "set_temp_data", "delete_temp_data", "highlight", "msg", "datamsg", "create_term", "term_stdin", "delete_term", "update_term", "term_stdout", "saved"
],
"user_id": 80844,
"res_id": 1
}
Permissions
The room_info
message will contain your permissions under the perms
attribute. Permissions are a list of message types you are allowed to send. If any other message type is sent, you will be disconnected with an error message.
Buffers
Buffers are the center of RDSP. The basic operations on them are: create_buf
, delete_buf
, get_buf
, set_buf
, and rename_buf
.
Patches
CRUD for buffers is useful, but the real power of our protocol lies in the patch
message. Patches make replicating changes much more efficient. Instead of sending the entire buffer state, only the changed parts and a few bytes of context are transmitted.
Here's an example. Let's say we have the following buffer:
testing1234
1234example
...and we want to add "hello" to the middle of it:
testing1234
hello
1234example
The patch
event would look like this:
{
"md5_before": "cd11cfac8509ee274080ebc9ab69b591",
"md5_after": "37181ede6d363fbdee5504fe16102585",
"id": 1313,
"patch": "@@ -5,16 +5,21 @@\n ing1234%0A\n+hello\n 1234exam\n",
"name": "patch",
"req_id": 63
}
Example code
We've implemented RDSP clients in several languages. Since all of our plugins are open source, you're free to use these for your own purposes.
- Python: Sublime Text plugin
- JavaScript: Atom plugin
- Java: IntelliJ IDEA plugin