[SyncEvolution] local sync
by Patrick Ohly
Hello!
Let me write down some thoughts on local sync. With that I mean
synchronization of databases which are accessed locally with
SyncEvolution backends. This is in contrast to "normal" sync, where one
side is covered by a backend and the other side is a SyncML peer (client
or server).
Right now, the only way to do this is to setup syncevo-dbus-server
+syncevo-http-server on one side and syncevolution --daemon=no on the
other. Two peers need to be configured.
We could simplify the configuration as follows. In the normal, @default
context we define the main databases to be used, for example Evolution.
No change compared to what is done so far. For those not familiar with
the concept, a "context" holds a set of peer-independent source settings
(like which backend and database are used for each source like
"addressbook").
In another context, say @xmlrpc, we define a sources that accesses
different data via some other backend, like the XMLRPC interface. Let's
assume we have one source defined, like "calendar".
Now we configure one peer "xmlrpc" in the @default context:
syncURL = local
peerIsClient = 1
[calendar]
URI = calendar@xmlrpc
When we run a sync, we treat one side as the "SyncML server" and the
other as "SyncML client". For performance reasons (more operations done
there) it is desirable to have the truly local data being used by the
server side.
In terms of logging, one option is to use one session directory like
this:
xmlrpc-2010-06-17-13-00:
calendar.after
calendar.before
calendar(a)xmlrpc.after
calendar(a)xmlrpc.before
syncevolution-log.html
syncevolution-log_trm001_001_outgoing.xml
...
syncevolution-log(a)xmlrpc_trm001_001_outgoing.xml
status.ini
status(a)xmlrpc.ini
This requires a lot of changes, both for writing it like this and for
accessing the information about the @xmlrpc status and backups. It
probably cannot be done without D-Bus API extensions and new command
line options.
The simpler solution is to have two session directories, one for the
server (xmlrpc-2010-06-17-13-00) and one for the client
(@xmlrpc-2010-06-17-13-00).
When running a sync with the @xmlrpc calendar source as client, some way
of storing persistent state is needed. Normally this is done via the
per-peer .internal.ini file. The @xmlrpc context doesn't have per-peer
directories, but we could put the file into the @default configuration
tree as peers/xmlrpc/sources/calendar/.internal-calendar(a)xmlrpc.ini,
where calendar@xmlrpc is the URI of the pseudo-peer. That allows
changing that parameter without accidentally running a sync with the
wrong change tracking state. Removing the xmlrpc@default config would
also remove this file.
Now regarding actually running two sync sessions. One option is to put
the server into the syncevo-dbus-server (as it is now) and the other
into a forked process. The advantages of this solution are:
* allows using backends with conflicting library requirements
(think KDE and Evolution with different setup of glib/libical,
etc)
* a crash on one side can be detected by the other
* no need to get rid of global variables (there are a few, related
to logging and finding the context inside Synthesis plugins)
* no need to change signal and event handling
Drawbacks:
* we are forced to serialize messages and exchange it via IPC
mechanisms (sockets, shared memory); inside the same address
space we at least have the option of skipping SyncML message
encoding/decoding if and when the libsynthesis engine gets
refactored to pass changes in its internal format directly back
and forth
* cannot write one common log file
The other option is to run everything inside the same process and do a
global context switch between the two sides.
Currently I favor the idea of using two processes, mostly because it
requires less rewriting of code.
Loading conflicting backends would require further changes, because
right now *all* backends are loaded to register them. Splitting backends
into a general-purpose registry library and the actual implementation
would be possible, or we could go for the more traditional method of
defining backends in text files.
I'll let this idea sit for a while, but might come back to it soon.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 2 months
[SyncEvolution] Google Calendar sync
by Patrick Ohly
Hello!
As those who have tried the "google" config template know, only contact
synchronization over SyncML works, because that is all that Google
supports. Those who want to synchronize their Google calendar have to
use third-party bridges to that data (ScheduleWorld, Goosync).
http://bugzilla.meego.com/show_bug.cgi?id=319
A "native" solution would be to access the Google calendar via CalDAV. I
gave that a try, using the Evolution Data Server CalDAV/Google backend
as implementation of the protocol. In theory, this should have worked
out-of-the-box with SyncEvolution, although setting it up would be
fairly complex:
* set up syncevo-http-server to access Google calendar
* set up a client config to access the normal local calendar and
syncevo-http-server
* run command line with "--daemon=no", because that is currently
the only way how we can have SyncML client and server on the
same host
In practice I ran into several problems which makes this approach
unusable right now. For those who are interested in details, below is a
description of the problems.
My conclusion right now is:
1. We need a better solution for backend A<->backend B syncs (I
sometimes call that "local sync").
2. We should better write a CalDAV backend instead of relying on
EDS. Not only will that avoid the issues below (which could be
fixed), such a backend would also be useful for users not
interested in running EDS and perform better. I'm assuming here
that CalDAV has functionality that would helps us (like
efficient listing of all items) which we can't use when going
through EDS because it doesn't have corresponding APIs.
Any CalDAV experts or volunteers around? The second task could be
started already with the current code base. If no-one volunteers, then
it'll end up on the backlog of feature requests.
-------------------------------------------
Here's how I set up testing:
* create two test calendars in Google
* create the same calendars in Evolution, using Google_ical20_1
and Google_ical20_2 as names (naming expected by client-test,
setting the evolutionsource property to the right caldav:// URL
should work without creating the calendars in Evolution)
* run CLIENT_TEST_EVOLUTION_PASSWORD=<password>
CLIENT_TEST_EVOLUTION_PREFIX=Google_ ./client-test
Client::Source::ical20
First problem: our test data uses a UID with special characters, because
that happens to break "dumb" XML encoders/decoders and we want to find
such issues. It also breaks the EDS backend. It can create such an
event, but then cannot access it because the UID becomes of a URL used
by WebDAV, and these special characters are not escaped there (seen in
evolution-data-server console output).
As a workaround, I simplified the test data to not use such a
complicated UID when CLIENT_TEST_SIMPLE_UID=1 is set.
Still no luck. client-test can create an item, but then retrieving it
fails. The root cause for that is the backend's implementation of
e_cal_create_object(). It should return the ID assigned to the new
object, but for the WebDAV EDS backend it returns an empty string
instead of the correct ID. Iterating over all objects later shows that
the ID is something like http://www.google.com/calendar/feeds/<unique ID
of calendar>%
40group.calendar.google.com/private/full/jjs14jjurojsoi7koo3op1vq3g
Even if e_cal_create_object() worked, we wouldn't trust it in
SyncEvolution, because it was broken in the past. From the code:
// creating new objects works for normal events and detached occurrences alike
if(e_cal_create_object(m_calendar, subcomp, (char **)&uid, &gerror)) {
// Evolution workaround: don't rely on uid being set if we already had
// one. In Evolution 2.12.1 it was set to garbage. The recurrence ID
// shouldn't have changed either.
ItemID newid(!id.m_uid.empty() ? id.m_uid : uid, id.m_rid);
newluid = newid.getLUID();
modTime = getItemModTime(newid);
m_allLUIDs.insert(newluid);
}
In other words, when a VEVENT has a UID, we use that as local ID. That's
the best we can do, because the "garbage" really is a random pointer as
far as I remember, so we cannot use it at all.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 2 months
[SyncEvolution] updating documentation, part II (MBC #690)
by Patrick Ohly
Hello!
For SyncEvolution 1.0 we already revamped the documentation and added a
man page, all generated from the same README.rst. The output of
"syncevolution --help" was the last duplicated piece of documentation
material.
I have experimented with removing that duplication. The result is on the
pohly branch. I'd like to get some feedback on the approach taken:
* README.rst contains SYNOPSIS, USAGE, OPTIONS.
* SYNOPSIS introduces the command line syntax.
* USAGE introduces features most operations.
* OPTIONS then adds a more formal definition of the command line
parameters.
* The SYNOPSIS and OPTIONS part appear again in the --help output.
I had some problems fitting --restore into that concept. Does the
current state make sense? Suggestions for improvements?
Current README.rst and --help output are attached.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 3 months
[SyncEvolution] new backends: QtContacts, KCalExtended
by Patrick Ohly
Hello!
While studying the QtContacts [1] and KCalExtended [2] APIs, I spent
some time writing backends for them. Current state is that item
import/export and change tracking work.
QtContacts passes all Client::Source::qt_vcard30 tests except for
testImport. That's because its set of supported properties is different
from the Evolution test cases. For example, PHOTO does not seem to be
stored at the moment.
KCalExtended is still in rapid development right now. Just getting it to
compile and link requires fiddling with the KCalExtended source. It
fails some tests and is overall in a less stable state.
Both backends are not meant to be used for actual syncing and are not
supported - you have been warned ;-)
But I'd like to merge the code into the "master" branch anyway.
Yongsheng, can you do a normal code review of the "pohly" branch
(contains the some unrelated fixes for master) and a quick check of the
"qt" branch (based on "pohly", contains the new backends)?
[1] http://doc.qt.nokia.com/qtmobility-1.0/contacts.html
[2] http://maemo.gitorious.org/extendedkcal
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 3 months
[SyncEvolution] Lessons from Genesis development: Enhancement proposals
by Frederik Elwert
Hello!
I am just about to release the first version of Genesis after the
rewrite. There were some points I noticed where there is room for
improvement regarding the interplay between SyncEvolution, sync-ui and
Genesis. Some points have been mentioned before, so please excuse some
duplications, but I thought it would make sense to list all these issues
here.
API issues
Problem:
Watching for configuration changes (e.g., for refreshing the
list of servers in the GUI) currently is a bit cumbersome. It
requires watching for SessionChanged, request GetConfigs, and
look if the configs have changed. This is not really a problem,
but it might be more convenient to get notifications about
config changes directly.
Proposed solution:
Introduce a Server.ConfigChanged signal.
Problem:
GUIs cannot tell for which server a session was started if the
GUI didn’t start the session itself. So for example when Genesis
sees that a sync session ended and displays the results, it
cannot tell (and therefore cannot show to the user) which server
was actually affected. So e.g. notifying about the results of
automated syncs lacks some rather important information.
Proposed Solution:
Introduce a Session.GetServerName() method.
Problem:
The syncevo-dbus-server handles user notifications itself (e.g.
for automated syncs.) A GUI might want to notify users about
syncs itself. E.g. Genesis shows a spinning status icon, which
makes the sync started/sync ended notifications of
syncevo-dbus-server redundant.
Proposed Solution:
Clients should be able to tell syncevo-dbus-server not to
display sync notifications. I propose a method
Server.InhibitNotification(bool) which could be used to inhibit
notifications when Genesis is started and allow notifications
before it terminates.
Sync-UI
Genesis no longer ships its own configuration facilities, but instead
uses sync-ui for this. In general, sync-ui is more powerful than
Genesis’ old setup assistant. But if I didn’t oversee something, it is
currently not possible to set evolutionsource. There might be situations
where one wants to sync other sources than the default ones, e.g. when
syncing the private calendar with one service, and the work calendar
with another one.
It would be nice if one could choose the sources in sync-uis settings
dialog. Maybe it would make sense to rename the “show server settings”
expander to something like “show extended settings” and allow to select
the data sources there.
syncevolution commandline
I just noticed that syncevolution commandline commands (like
--print-config or --print-sessions) are treated as syncs in the GUIs
(sync-ui, Genesis). This is because the GUIs watch for
Session.StatusChanged and expect that a sync was performed with
status==done. Even commandline errors (like not giving a server name
where required) is treated like a sync error in the GUIs.
I don’t know at which point this could be solved. Are there means for
the GUIs to tell if a session was a sync session or something else? If
not, should the API be extended to cover this? Or could simple the
commandline client be changed not to trigger SessionChanged signals?
I don’t know if these changes are possible, but I’d be happy to hear
your feedback to these issues.
Regards,
Frederik
12 years, 3 months
[SyncEvolution] Synchronising with Thunderbird
by h2oz7v
Using syncEvo's HTTP server, I have managed to synch using the file
backend. Now I am looking for options to synch that data with
Thunderbird. TB doesn't have direct support for SyncML, though
Funambol have an extension (compatible with TB3) that claims to be
compliant with any SyncML device:
<https://mozilla-plugin.forge.funambol.org/>
I thought I'd give it a go, the results being:
syncevo-dbus-server:
[ERROR] message type not supported for starting a sync
syncevo-http-server:
POST from 127.0.0.1 config type application/vnd.syncml+xml;
charset=UTF-8 session None args {} length 4865
requesting new session
connection /org/syncevolution/Connection/14611594851277639198 went down
Funambol extension:
Error: code 2052
12 years, 3 months
[SyncEvolution] syncevo-phone-config destroying data
by h2oz7v
I have a Sony w890i phone that I will be using for synching, which
unfortunately, there is not an existing template for.
Following <http://syncevolution.org/development/sync-phone> seems
straightforward, except for the rather innocent looking...
> Be aware that when invoked like this, the tool needs to wipe out the
> contacts, events, tasks and memos on the phone.
(p.s. this should be made much more prevalent on the wiki i.e. bold
and red font and above the command (so that people don't run the
command first only to notice it after it's too late!))
... warning. This seems very counter-intuitive: I need to *delete* all
my contacts, events, tasks etc. before I can synch my phone?! Is there
absolutely no way of avoiding this?
Meanwhile, I will try to configure my phone manually (and referring to
another Sony template in the wiki).
12 years, 3 months
[SyncEvolution] Some HTTP server confusion
by Colin Shea
I'm looking for a SyncML server, preferably a self-hosted one so that
not only can I keep my phone & computers in sync, but also maintain
privacy, features, etc. I had been trying to use libsyncml's server and
client, but the client does not work with some hosts. (i.e. GooSync)
I found out that SyncEvolution has a server, so I tried setting it up,
but I'm stuck. Following the guide at the wiki[1], I can get to step 6.
First off, the Arch Linux AUR package doesn't seem to install
syncevo-dbus-server, but it is compiled. (The packaging script just calls
./configure, make, & make install.) Running it manually always produces
an error - if the system (not session) dbus server is running, it
returns "server already running?" and if the system dbus server is not
running, it says it is "ready to run" but also says the connection setup
for connman failed. Even running it as root causes the same error.
For step 7, the script is already installed globally to /usr/bin, I
guess that doesn't matter too much.
For step 8, how do I get a client to sync? Especially a test client. I
am trying to test the configurations by doing a SyncEvolution <->
SyncEvolution sync. (I plan on 'stretching' that so that my server &
desktop are in sync instead of just on the local machine.)
The later steps, it seems, would make more sense after I understand the
above points, as the `syncevolution --configure` calls are simple
enough. I would like a series of commands that configure a client, as
one would to configure SyncEvolution for ScheduleWorld (but I don't use
ScheduleWorld anyways).
Thanks a lot!
[1]: http://syncevolution.org/development/http-server-howto
--
Regards,
Colin 'Evaryont' Shea
HTTP: http://evaryont.me
XMPP: colin(a)evaryont.me
12 years, 3 months
[SyncEvolution] deviceID and syncEvolution server
by h2oz7v
After a bit of a struggle, I've managed to successfully synch my w890i
(config here: <http://syncevolution.org/wiki/sony-ericsson-w890i>).
However, there was one point specifically that I had trouble with: the
deviceID.
I noticed in the other Sony phone report
(<http://syncevolution.org/wiki/phone-compatibility-sony-ericsson-w810i>)
that the ID should be populated automatically on the first synch.
However, this did not occur for me. The HTTP howto mentions: "search
the output for the line [ERROR] no configuration found for deviceID
XYZ123", which again I could not find. The server logs contain:
[2010-06-26 00:50:37.265] Connecting Bluetooth device with address
xx:xx:xx:xx:xx:xx and channel 10
[2010-06-26 00:50:37.680] OBEX progress
[2010-06-26 00:50:37.735] OBEX Transport: get header who from connect
response with value SYNCML-SYNC
[2010-06-26 00:50:37.735] Server sending SAN
[2010-06-26 00:50:37.735] ObexTransport send is called
[2010-06-26 00:50:37.736] OBEX progress
[2010-06-26 00:50:59.485] Cancel disconncting process
[2010-06-26 00:50:59.774] Server Alerted Sync init with SANFormat 12
failed, trying with legacy format
[2010-06-26 00:51:04.913] ObexTransport: Transport Exception in sdp_source_cb
[2010-06-26 00:51:04.914] TransportException thrown at
/work/runtests/head/syncevolution/src/syncevo/ObexTransportAgent.cpp:376
[2010-06-26 00:51:04.914] ObexTransprotAgent: Underlying transport error
Also, at this point (and found only by trial and error), the phone
would not connect again, with attempts showing:
[ERROR] ObexTransport: Transport Exception in sdp_callback_impl
[ERROR] ObexTransportAgent: Bluetooth service search failed
[ERROR] ObexTransprotAgent: Underlying transport error
... with only restarting the phone completely to fix it (sdptool
search --bdaddr ... did not work for me).
I essentially "guessed" the deviceID by entering *#06# on the phone
and noting down the IMEI code (minus the last pair of digits and the
dashes... 15 digits in all) and configured syncEvolution using the
attached script (yes, it took that many tries I had to write a script
to do it!).
Is it really this hard or did I miss something?
12 years, 3 months