OpenSync

A Synchronization Framework

White Paper

© 2004 – 2005 Armin Bauer

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License".

Table of Contents

1. Overview	3

1.1 Abstract	3

1.2 History	3

1.3 The Big Picture	3

1.3.1 OpenSync	4

1.3.2 MultiSync	4

1.4 The Goals	5

2. Technical Details	5

2.1 Sync Plugins	5

2.2 The Sync Engine	7

2.2.1 Thread Model	7

2.2.2 Conflict Detection	8

2.2.3 Resolving Conflict	8

2.3 LUID Mapping	9

2.4 Format Conversion	9

2.4.1 Plugins	10

2.4.2 Converters	10

2.4.3 Encapsulators / Decapsulators	10

2.4.4 Detectors	10

2.4.5 Conversion	11

2.5 Hashtables	12

2.6 Anchors	13

2.7 SlowSync	13

2.8 Initial Mapping	13

2.9 Sync Alerts	14

2.10 Filters	14

3. Further Readings	14

3.1 API Examples	14

3.2 Plugin Examples	14

3.3 Code Documentation	14

4. Glossary	15

Overview

Abstract

This document will give you a overview about how OpenSync and MultiSync work and how they are related. It will also give some explanations about the technical details of the SyncEngine and how OpenSync and MultiSync can be used.

History

MultiSync was originally started by Bo Lincoln. It was intended as a matter to synchronize various handhelds, cellulars and PIM application. Its focus was entirely on PIM data (contacts, calendar and todo items). MultiSync was a single application with dependencies on various gnome and GTK libraries. The connection to the devices was handled by plugins, which were loaded at the start of MultiSync.

However it became apparent soon that this approach was not flexible enough, since some people wanted to use MultiSync without a GUI (Graphical User Interface), synchronize data besides PIM data etc. At this time a new branch of MultiSync was created, labeled internally MultiSync-0.9 (as opposed to the currently stable 0.8X branch) which is supposed to offer these new features.

The Big Picture

Synchronization consist of several different parts:

Access to the data on the devices/applications, which can be separated into connecting/disconnecting, reading the data (only the changes since the last synchronization and all objects for slow-synchronization), writing data and some more “maintenance” function as well as defining which devices should be synchronized.

The LUID mapping, which describes which object on device A belongs to which object on device B, so that if one object gets changed/deleted, the correct corresponding object gets updated.

A conversion system. Sometimes the devices report objects in different formats (a contact could be stored in a VCard as well as some other format). To be able to synchronize, these formats have to be converted.

The SyncEngine, which takes care of the synchronization, conflict handling etc.

A UI (user interface) that presents everything to the user.

These parts are separated among OpenSync and MultiSync:

OpenSync implements to low level functions, like synchronization plugins that can be used for connecting to devices, format conversion, the SyncEngine, storage of configurations, etc. It also provides some helper functions.

MultiSync which implements the User Interfaces like a normal GUI, a CLI etc.

OpenSync

OpenSync implements the low level functions of the synchronization, which include the SyncEngine, conversion system, mapping tables, hashtables, anchor storage, configuration API etc.

The idea behind providing this framework is to make it possible for other developers of applications that are in need of synchronization to reuse the OpenSync framework and save work and get instant access to the available plugins. Another advantage for developers is that they can use the OpenSync Plugin API standard to use different functions from plugins to access devices and applications in a uniform way.

The advantage for the user is that once more applications start to use the framework he will be able to reuse SyncGroups he configured between these different applications since the configuration is stored in OpenSync.

MultiSync

MultiSync is one application that uses OpenSync to provide device synchronization to the user. It includes a Graphical UI and a Command Line Client.

The Goals

The goal is to create a universal synchronization Framework which has the following capabilities:

Reusability. The framework should be usable by many other applications

Speed. Synchronization should be as fast as possible to give the user the best experience.

Flexibility. We cannot predict what formats / devices the future will bring. Therefore OpenSync is built as flexible and modular as possible.

Integrity. Data must never be lost, no matter what happens.

Portability. The framework should run on as many platforms as possible (Linux, Windows, Mac OS, BSD, etc)

Some of this properties are mutually exclusive, but we try to get as close to these goals as possible.

Technical Details

Sync Plugins

A SyncPlugin is a module that provides access to a certain device / application / protocol. The basic functions that it needs to provide are:

An “initialize” function. In this function, the plugin has to malloc a struct it needs to track its internal state, load its configuration and start the listening server if it has one. The return value is the pointer to the struct.
Example: static void *fs_initialize(OSyncMember *member, OsyncError **error)

A “finalize” function which stops a listening server and frees the allocated struct
Example: static void fs_finalize(void *data)

These 2 functions are the only ones that are called synchronously on the plugins. All the other functions are called asynchronous. To be able to track them correctly, they get passed an OSyncContext struct, which they can use to answer.

It is not important at what time the following functions return, the only important thing is that they use one of the osync_context_report_* functions. Since each plugin runs in its own thread, they may block as long as they want.

The “connect” function which is called in the beginning of the synchronization. Here the plugin should connect to the device, open anything it needs etc.
Example: static void fs_connect(OSyncContext *ctx)

The “disconnect” function which is called to disconnect.
Example: static void fs_disconnect(OSyncContext *ctx)

The “get_changes” function. It is used by the SyncEngine to request the changes or all (in the case of slow-synchronization) objects from a device.
Example: static void fs_get_changeinfo(OSyncContext *ctx)

The “commit_change” function. This function is called once for each object that the engine want to write to the plugin (the second parameter).
Example: static osync_bool fs_commit_change(OSyncContext *ctx, OSyncChange *change)

The “synchronization_done” function which is called once all objects have been sent to the plugin. It is only called if the synchronization was successful (some commits may still have failed due to access or conversion errors)
Example: static void fs_synchronization_done(OSyncContext *ctx)

There is another special function on each plugin: The “get_info” function. This is the only function on the plugin that is actually read via dlsym. It will get passed a OsyncPluginInfo struct which the plugin has to fill with values. The things it has to set there are (among others): its name, version, the pointers to the above mentioned functions and the object-type and formats it accepts.

void get_info(OSyncPluginInfo *info) {	info->name = "file-synchronization";	...	info->functions.initialize = fs_initialize;	info->functions.connect = fs_connect;	info->functions.synchronization_done = fs_synchronization_done;	info->functions.disconnect = fs_disconnect;	info->functions.finalize = fs_finalize;	info->functions.get_changeinfo = fs_get_changeinfo;	...	osync_plugin_accept_objtype(info, "data");	osync_plugin_accept_objformat(info, "data", "file");	osync_plugin_set_commit_objformat(info, "data", "file", fs_commit_change);	... }

This example shows how the functions are called on a plugin. Note that there might be quite some time between certain calls to the plugin. The get_info() function gets called in the very beginning, the connect function only when a synchronization gets requested.

The Sync Engine

The SyncEngine is responsible for deciding what exactly needs to be done to synchronize the connected device. This includes initialization, connection, reading and writing changes, LUID (Local unique identifier) mapping and keeping the log. It utilizes the helper functions provided by OpenSync.

Thread Model

The SyncEngine is completely multi-threaded. One thread is responsible for the synchronization itself. For each member in the SyncGroup, a new thread is spawned also, so that all member may access their devices at the same time and block. The communication between the different threads is handled via asynchronous message queue on which a message bus has been implemented that supports messages with answers, payloads and timeouts.

Conflict Detection

There are different types of conflicts that might occur:

Differences in the change-type. For example, a contact could have been deleted on one side, but modified on the other side.

Differences in data. The data of 2 sides could have been altered in different ways.

Each time a conflict is detected, the SyncEngine calls a callback handler set by the controlling application. This function gets passed a pointer the mapping with the conflict.

void (* conflict_function) (MSyncEngine *, OSyncMapping *)

Each time this callback function gets called, the user has to resolve the conflict.

Resolving Conflict

The user can decide in different ways how this conflict should be handled. The first solution is to pick a winning side, whose change overwrites the other sides.

osync_mapping_set_masterentry(mapping, change);

The second solution is to keep all changes as separate entries (Duplication).

Another possibility is that the user wants his choice to be valid for all future conflicts (e.g. “Duplicate all conflicts”).

msync_mapping_duplicate(engine, mapping);

LUID Mapping

The LUID (Local unique identifier) mapping is one functions that is provided optionally by OpenSync. It can be used to save / load information about synchronization objects and their mappings. The changes are saved in a database in the current configuration directory of the SyncGroup. The things stored are: the LUID, the object-type, the format, the id of the member which reported this object, the id of the mapping.

This stored information is read in the beginning and can be used to map objects from different sides.

The functions provided are: Functions for creating/deleting mappings, adding/removing objects from a mapping, searching for objects in a mapping by the member that reported it, and searching for an object on a member by LUID.

Format Conversion

A lot of times it happens that different devices report the same object in different formats (a contact could be encoded in a VCard, a Palm format etc). To be able to synchronize one needs to be able to convert the formats. The format conversion takes place during the actual synchronization and is described in detail later in this chapter.

Each object has 2 attribute types assigned to it:

The object-type. This attribute classifies the abstract information the object carries. It does not say anything about the format in which the object is represented. One example for an object-type would be “Contact”.

The object-format. This attribute shows in which format the object-type is currently encoded. One example for such a format for the object-type “Contact” would be “VCard”. The formats are stackable which means that you can wrap one format into another format (which is referred to as “encapsulation”). The “Contact” object can have its information formated in a “VCard” format, which is then wrapped up into the “file” format (which adds information about the modes, owners etc of the file).

Plugins

To be able to support almost any object-type / format we choose to implement the conversion system with plugins. These plugins work similar to SyncPlugins. They have a “get_info” function, in which they register the object-types / formats they support. This function gets a pointer to a “OSyncEnv” struct which represent the environment in which to register.

void get_info(OSyncEnv *env) {	osync_env_register_objtype(env, "data");	...	osync_env_register_objformat(env, "data", "file");	osync_env_format_set_compare_func(env, "file", compare_file);	osync_env_format_set_detect_func(env, "file", detect_file);	osync_env_format_set_duplicate_func(env, "file", duplicate_file);	...	osync_env_register_converter(env, CONVERTER_CONV, "file", "VCard", conv_file_to_vcard);	osync_env_register_converter(env, CONVERTER_CONV, "VCard", "file", conv_vcard_to_file); }

Converters

One thing a plugin can register is a converter. A converter converts the data of an object to another format, therefore replacing the current format with another. The converter gets passed a pointer to some data. It then has to parse this data and return a pointer to a newly allocated struct with the data in the new format.

Encapsulators / Decapsulators

A decapsulator takes the data it gets passed and just removes the current format layer, therefore only return the data of the object. The file decapsulator for example removes all information about the mode, owner etc of the file and just returns the content of the file.

Encapsulators do just the opposite, they wrap a format into another one.

Detectors

At some point during the conversion, someone has to label the object has belonging to a certain object-type / format. This can be done while reading the change (when you connect to the evolution address book you know that you will always get “Contacts” of format “VCard”). But some plugins cannot do this (the file-synchronization plugin for example has no idea if the file is actually some data or a saved “VCard”). Therefore we have data detectors that try to parse and identify the input they get. (The “VCard” detector for example looks if it finds the “BEGIN:VCARD” in the data).

Conversion

This section will now explain how a format conversion would look like.

There are 2 occasions where a conversion might be necessary:

To compare the data of objects to find conflicts

To commit the data to another side

For the first case the data has to be converted to a “common” format to compare, which might or might not be the format that the member understands.

For the second case the data always has to be converted to a format that the member can accept. These formats are set via the accept_objtype and accept_objformat functions in the “get_info” function of the plugin.

Sometimes it is not directly possible to convert to a given format since no direct converter exists but a detour over another format has to be taken. This is why the conversion “path” is detected using a Shortest Path algorithm. It will always try to convert with as little conversions as possible. One special thing in detecting this conversion path are the encapsulators. Since decapsulating always is a loss of information, the path is searched without the decapsulators first. If no path is found then the decapsulators are used.

The next example shows how a conversion environment for a synchronization between a Palm device and a file-synchronization plugin could look like. Here also a intermediate XML Format is used.

Hashtables

Hashtables are a optional feature of OpenSync. They can be used by the plugins to detect changes in the objects. A lot of devices do not implement a change database which is used to track if an object has been synchronized. For these devices the hashtable can be used.

The plugin has to compute a hash for each object on the plugin (which might be a timestamp or something like a md5). Each time the engine queries for the changes, the plugin uses the hashtables function to compare the old hash with the new hash. If they are not the same, the plugin knows what has changed and can report this change to the SyncEngine.

A example usage of the hashtable could look like this:

Initial Sync:
Hashtable compares new timestamp “1103060445-110306044” with old timestamp NULL and reports the change as being “Added”.

Hashtable updates to current hash in the database to “1103060445-110306044”

Now the object gets modified

Second Sync:
Hashtable compares new timestamp “1103061332-110306123” with old timestamp “1103060445-110306044” and reports the change as being “Modified”.

Hashtable updates to current hash in the database to “1103061332-110306123”

Anchors

OpenSync also provides a “anchor” storage. A anchor is some kind of data that is stored on the device and is updated once during each synchronization. The anchor is also stored locally using the functions provided by OpenSync. If the anchors do not match the next time a synchronization is initiated, a SlowSync is requested.

SlowSync

If a client detects that its database has been reset or the last synchronization was not successful, a SlowSync is performed, where each client sends all its objects (as opposed to just the changed objects) to the SyncEngine. This is also used for the initial synchronization.

Initial Mapping

If a SyncGroup has never been synchronized before, it needs to detect the initial mappings on the first synchronization. It might happen that the devices, were synchronized before using another application and therefore might contain the same objects on both sides (or the user added the same object to both sides by hand). The initial mapping happens like this:

All unmapped objects from both sides are compared. The comparison might return 3 answers:

MISMATCH: The objects were different. The search continues until there are no unmapped objects left on the other side. In this case the object is being reported as “Added” and will propagate to the other synchronization during the synchronization.

SIMILAR: The objects were not exactly the same but some key properties are the same (like the name for a VCard). In this case a mapping is generated for the objects and a conflict is raised. If the connection between these objects where not correct (there might be contacts with the same name but different persons), the user can still choose to keep both object as separate entries (Duplication).

SAME: The objects were the same. A mapping is made for the objects and no conflict is raised.

Sync Alerts

Some plugins support SyncAlerts. A SyncAlert is when the plugin has the capability to detect that an object has just been changed and reports this to the engine which can then decide to initiate a synchronization.

Filters

OpenSync supports filtering objects based on certain criteria and on the direction of the object. You can, for example, filter on an object of format “file” going from member A to member B.

Once a filter triggers on an object passing through OpenSync, you can invoke certain actions. The standard actions are to either let it pass, or to drop it.

But sometimes more sophisticated filters are needed. Thats why you can define custom filters using a callback that will get called with the object that triggered the filter. You can then return if the object should be allowed to continue or if it should be dropped.

But you can not only allow/deny an object, you can also manipulate the object you caught. This way you can alter the object while it passed through OpenSync.

Filters can also be loaded through plugins.

Further Readings

API Examples

You can find several examples of how to use the API of OpenSync and the SyncEngine online at:

http://www.opensync.org/wiki/Examples

Plugin Examples

You can find a example plugin at:

http://www.opensync.org/browser/trunk/plugins/example-plugin

Code Documentation

The functions OpenSync provides are documented using comments in the code. You can find the latest documentation at:

http://www.opensync.org/docs

Glossary

LUID

The LUID (Local unique identifier) uniquely identifies an object on a member. Most of the time it is only valid and unique on one member.

SyncGroup

A SyncGroup is a group of members which are supposed to be synchronized. This group can contain any number of members.

Member

A member is part of a SyncGroup. It is a configured instance of a certain SyncPlugin.

SyncPlugin

A SyncPlugin has the code that is needed to connect to a certain device / application /server.

SyncEngine

The part of OpenSync which is responsible of deciding what to do next to achieve a synchronized state.

FormatPlugin

A FormatPlugin has the code that describes a certain format (“VCard” for example) and provides functions for manipulating this format.

2004-12-14T08:43:10