Synchronize Product Module added

A new module for setting up the product models including the dependencies  for synching is now available at http://hg.holtzberg.de/synchronize_product

After installing the module it is needed to add a remote server in the admin menu. That server has to be set in the root entry of the added product models.

Push support added

The communication with the local and remote systems is now handled through connection objects that implement a common interface. This connection interface can be implemented also for other systems for which synching is desired.

The two connections to synch between are changed if synching in the other direction is wanted. The synchjob needs its own user to distinguish between real local changes and changes that result of a previous pull operation. The create_many method for speed improvement is deferred for now because of simplicity.

The user interface for configuration has also been changed to a more easy one.

The next thing is to create a module that adds the product model as a synching model. With this reviewers only have to install two modules and add a remote server before testing a synchronization.

Code cleaned and sqlite support added

Determining ids from a sequence with postgresql is not a problem but sqlite doesnt use such sequence objects, but for adding data into the database it is required to have the ids before insert. In sqlite the sequences are handled in a special system table sqlite_sequence which stores the last used value of each sequence. By manipulating this table it is possible to get unique sequence ids before inserting the records.

Another issue is that sqlite handles sorting of null fields the other way around than postgresql and does not support ORDER BY xxx DESC NULLS FIRST. Thats the reason why at the moment it isn't possible to synch with a sqlite based Tryton-server. (The slave can be sqlite based)

Updating of existing records is now possible. Furthermore a short screen cast is available on youtube to demonstrate the generic way of synching with Tryton servers. (http://www.youtube.com/watch?v=iAGP5fpdOAM)

Handling of function fields and default values is not implemented yet. Also MySQL has not been tested yet but there will also be the sequence issue.

For updating records prepared statements are not used but can be implemented if performance increase is needed. By now the write_many method only wraps the normal write method.

Design restructured

The synchable class has been removed, the translation between local an foreign ids is now available in a translation model. With this the module can easily installed without editing the ModelStorage.

The organization of the models to synch has also been changed. When a model is added into "synchronize.model" the model will be searched recursively for dependencies. All dependencies are added as children to the proper parent. For each model the fields that are required are automatically added, additional fields can be added manually. With this unneeded fields wont be fetched to save traffic.

The next steps will be:

• Add handling for function fields to create_many method
• Add updating for records (not much work)
• Thinking about synching models with other systems for migration. (Thats not part of the GSoC but I really would like to have this functionality)
• Testing if the system works correctly

The repository is now available under http://hg.holtzberg.de/synchronize

Speed improvement

After profiling the syching procedure I found out that the bottleneck was not the foreign_id to local id relation but the ModelSql.create method which only allows to create single datasets. I added a create_many method to the Synchable class to create datasets with prepared statements which improved the speed very well. The RAM consumption is now even less than before.

Now I  can synch with another Tryton server over the Internet 2400 product.product datasets with all needed dependencies in kind of exactly one minute.

Now the bottleneck has moved to the fetching of records from the master server, but there are lots of possibilities for further optimizations.

The next days I will clean up the code and push it to a public repository for reviewing.

First try of synching with remote Trytond servers

I implemented a synching system for tryton models for testing.
The communication is based on the normal NET-RPC which is also used by the GTK-client. The advantage over a separate solution is that a "master" server needs not to be modified before another server can synch with it.

My first tests have shown that this way of synching is possible.

The next steps will be:

• Improve synching speed with some kind of caching
• Preventing creation of system records with foreign ids like the uom model
• Possibility of excluding fields from synching