[SyncEvolution] SyncML + OBEX
by Patrick Ohly
Hello Forrest!
I've just had a chat with Marcel about OBEX/SyncML integration. I'd like
to continue the discussion by email. Note that the public SyncEvolution
list is on CC, please group reply, I'll approve posts.
What we (= SyncEvolution) have to implement is the SyncML OBEX binding:
http://www.openmobilealliance.org/Technical/release_program/docs/copyrigh...
As explained in the design document for the future SyncML server, we
want to establish a central D-Bus service as the single component in the
system which runs SyncML sessions:
http://moblin.org/documentation/syncevolution/direct-synchronization-aka-...
This service has to interact with peers over OBEX in two ways:
1. actively contact them (SyncML server = OBEX client)
2. wait for connection requests (SyncML client = OBEX server)
These are the common use cases. The spec allows a SyncML client (phone)
to contact the server (desktop), but typically the UI of a phone only
allows triggering a sync via HTTP, not via OBEX. Instead, the software
on the desktop is expected to contact the phone.
The first point could be implemented via libopenobex. libsyncml could be
used as example for this. We might even copy the code, the license is
compatible. However, we don't have SyncML server support yet. Whether
that is really simpler than using obexd remains to be seen, which is why
I'd like to look at the second point first.
The second point is best solved via an obexd plugin. This is were help
would be very welcome. Forrest, do you think you could point us in the
right direction? My hope that this is trivial for someone who knows the
obexd source code. We could use help:
1. addding a new SyncML plugin to obexd
2. adding the SyncML service description (OBEX binding document,
5.2.1)
3. writing stubs for Connect/Disconnect/Put/Get/Abort operations:
* CONNECT: use a dummy connection ID
* PUT: discard incoming data
* GET: send dummy data
4. how the data could be exchanged with a D-Bus service as part of
the obexd event loop
These stubs then need to be extended so that they actually talk to the
sync D-Bus service via its (still to be defined) API.
The OBEXD binding allows splitting of one SyncML message into smaller
PUT/GET chunks. I'm undecided whether the D-Bus API should also support
this, because the maximum size of a SyncML message could already be
chosen so that we don't overload the system.
Speaking of D-Bus communication: Marcel pointed out that this additional
traffic and the amount of data can be problematic and mentioned that
D-Bus 0.4 will support fd exchange to hook up two peers directly. We
should keep that in mind, but start with data exchange via plain D-Bus
before tuning it.
We are not the only ones who transmit the data over D-Bus anyway, EDS
also does it. So our use of D-Bus is not a qualitative difference
(completely different way of using it), only a quantitative one (twice
the traffic).
To complete the OBEX support and server-initiated syncs, we (= the
SyncEvolution team and/or Synthesis) also needs to add encoding and
decoding of the SyncML Notification message (OMA DS 1.2.2, chapter 12).
This is non-trivial because the URIs of the device must be known in
advance. To get started we should make this configurable and later
decide how to simplify that for users (global device database?).
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 4 months
Re: [SyncEvolution] SyncML Tests 2009-07-27: head-evolution-testing
by Patrick Ohly
Hello!
Analysis of latest nightly run below.
On Tue, 2009-07-28 at 04:47 +0100, Ohly, Patrick wrote:
> http://runtests.syncevolution.org/2009-07-27-22-00/head-evolution-testing
>
> libsynthesis successful
> syncevolution successful
> compile successful
> dist skipped: disabled in configuration
> evolution successful
This worked on all of the other platforms, too. The problem with not
finding the default memo in earlier runs was a local install problem:
I'm pretty sure I had started Evolution once in each platform and opened
addressbook, calendar, tasks, memos, but despite that, the system memo
lists didn't exist on all platforms. It seems it is necessary to create
an entry in Evolution before it creates the database.
> scheduleworld: failed
Fails because of the new testSlowSyncSemantic test (#4703). I have added
it to the list of know failures and will document it in a
test/README.scheduleworld if it doesn't get resolved.
> memotoo skipped: disabled in configuration
> egroupware skipped: disabled in configuration
> synthesis: failed
Only one failure:
ClientTest.cpp:3851:Assertion
Test name: Client::Sync::text_vcard21::Suspend::testUserSuspendClientRemove
equality assertion failed
- Expected: 0
- Actual : 500
This was because of a timeout, according to the log at the start of the
sync session. Because the "timestampall" option is not set in the
Synthesis log, it is hard to tell how long that session ran. I have
enabled that option now.
There's also not enough debug information to identify where it got stuck
- communication with EDS perhaps?
> funambol:
One failure:
ClientTest.cpp:3851:Assertion
Test name: Client::Sync::itodo20::testCopy
equality assertion failed
- Expected: 0
- Actual : 20043
[ERROR] http://my.funambol.com/sync via libsoup: Bad Gateway
Message resending might have avoided this failure...
> zyb: failed
Still no word from ZYB. David, is anyone looking at the anchor problem?
http://bugzilla.moblin.org/show_bug.cgi?id=2424#c28
> google successful
Yeah! Congwu's script for cleaning the server data works.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 5 months
Re: [SyncEvolution] D-Bus API + server mode
by Patrick Ohly
Hi Frederik!
I suppose you wanted to keep this on the list, so I'm copying the list
again.
On Tue, 2009-07-28 at 20:39 +0100, Frederik Elwert wrote:
> Am Montag, den 27.07.2009, 09:13 +0200 schrieb Patrick Ohly:
> > Agreed so far, except that I think the logic should be inside the D-Bus
> > server.
>
> Ok, I see. Then we should probably think about an API for automated
> syncs. Just one question: Automated syncs require also some kind of
> autostart. Once a user sets up, let’s say, a sync interval of 60
> minutes, the syncs should run in the background right after booting the
> computer (or at least after logging in).
The user has to be logged in. The user's session bus must exist for
reliable access to the EDS. There are workarounds for headless setups
(server), like running dbus-launch manually, but if we do that on a
machine where the user might log in later there's no guarantee that we
won't end up with two EDS instances.
To make it even more obvious, in some setups (laptop) the home directory
might encrypted and only become available when the user logs in.
Unfortunately, there is no "per user session" cron. For permanently
monitoring the local state I think we could get a D-Bus service started
as part of a user session - but I'm not entirely sure. If anyone knows
more, please chime in...
> Does the D-Bus server get started on login? Or is it just started the
> first time a client requests it?
Currently it is only started on demand. This is good for startup time
(power on to usable desktop), an important feature in Moblin, so we have
to be careful not to slow down the startup.
> > > * The information about local changes could be used not only for
> > > automated syncs. For example, Genesis could show the information
> > > about changes to the data in the notification area, leaving the
> > > start of the sync to the user.
> >
> > Yes, this might be useful, but we should design it so that the server
> > does not needlessly monitor for changes. Monitoring would require that
> > the server runs permanently, and monitoring itself could be costly - not
> > so much for EDS, but for other backends (file, for example).
>
> What about inotify?
Yes, that works. But I have heard mixed opinions about it, so I'm not
entirely sure how well it really works.
> If the server doesn’t permanently monitor for changes, how would one get
> this information (regardless if for event-based automated syncs or for
> user notification)?
There are no good alternatives, so the server really should do it as
good as it can.
> > > The problem with that is that Ubuntu removed actions from the
> > > notifications.
> >
> > I suppose that applies to libnotify as well?
>
> Ubuntu introduced a new implementation of the notification demon called
> notify-osd. It conforms to the freedesktop spec, but doesn’t implement
> everything the old demon did (which isn’t required by the spec). So one
> can still use libnotify, but not everything will work with notify-osd,
> especially actions.
I see.
> But to come back to the D-Bus API: I am currently trying to port Genesis
> over to that API. Could somebody help me with understanding the API?
Jussi would be the prime candidate for that, but he is on vacation at
least this and next week. Let me try to help out.
> What I think I understand so far is:
>
> 1. The client starts a Sync with StartSync(). (I didn’t manage to
> call it with an empty array for a full sync, so I read the modes
> per source from the config and pushed this to the method call.)
> 2. The clients registers a handler for the Progress signal. I think
> I don’t need all the details here, so I simply wait for
> success/failure notices? Could someone please tell me, what the
> type values mean and what extra values are passed?
The values are a direct copy of the Synthesis progress events, using the
same integer codes and parameters as inside the Synthesis C/C++ SDK. The
definitions are in libsynthesis src/sysync_SDK/Sources/engine_defs.h.
Clearly this is not useful for a Python D-Bus client, but we weren't
sure about how this could be abstracted in D-Bus. We might have to
duplicate the whole set of progress events and document them as part of
our D-Bus API.
Regarding end of sync, sync_progress_cb() checks for PEV_SYNCEND.
> 3. Once the sync completed, I get the details by calling
> GetSyncReports(). But what do I actually get? What do the values
> mean?
This is intended to change. My proposal in another email in this thread
was to use a flat key/value hash with special encoding of the keys
instead.
Given that you would end up depending on an under-documented, almost
obsolete D-Bus API in 0.9, would it make more sense to take your
feedback and work on a better API for 1.0 and then release that as soon
as possible, without releasing Genesis for 0.9?
> I read through src/dbus/interfaces/syncevo-full.xml, but not everything
> is documented in detail. And I really have a hard time reading the C
> sources. ;-)
Yeah, I can imagine that it'll look nicer in Python. One more reason to
get Genesis ported ;-)
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 5 months
[SyncEvolution] hello ,every one.
by Qi Meng
Hi guys,
Very nice to find such a project about SyncML. I have been working on
the SyncML server in Access China for about 3 years. Could some body tell me
how to join this project develop group? I am very interested in taking part
time work on this project.
BR,
Smith
12 years, 5 months
Re: [SyncEvolution] SyncML Tests 2009-07-30: head-evolution-testing
by Patrick Ohly
Hello!
I've tagged this as 20090730 snapshot and consider it a release
candidate for 0.9.
On Fri, 2009-07-31 at 04:48 +0100, Ohly, Patrick wrote:
> http://runtests.syncevolution.org/2009-07-30-22-00/head-evolution-testing
>
> libsynthesis successful
> syncevolution successful
> compile successful
> dist skipped: disabled in configuration
> evolution successful
No surprises on the other platforms, they all compiled fine.
> scheduleworld: failed
Mark fixed some issues (twinning for text::testManyItems). He also
identified the root cause of the sync failures after a slow sync (the
server expected the client's last anchor to be non-empty during a slow
sync) which was resolved both in the server (no longer makes that
assumption) and the Synthesis library (sends a dummy string).
However, twinning of notes still fails in one case:
text::testSlowSyncSemantic.
Item send during regular sync:
http://runtests.syncevolution.org/2009-07-30-22-00/head-evolution-testing...
Item send again during slow sync and comes back as duplicate:
http://runtests.syncevolution.org/2009-07-30-22-00/head-evolution-testing...
This test passes with all other servers.
> memotoo skipped: disabled in configuration
> egroupware skipped: disabled in configuration
> synthesis: failed
Two failures:
ClientTest.cpp:3887:Assertion
Test name: Client::Sync::text_vcard21::Retry::testInterruptResumeServerRemove
equality assertion failed
- Expected: 0
- Actual : 20043
ClientTest.cpp:3887:Assertion
Test name: Client::Sync::text_vcard21::Retry::testInterruptResumeServerUpdate
equality assertion failed
- Expected: 0
- Actual : 20043
[ERROR] http://www.synthesis.ch/sync2 via libsoup: Connection terminated unexpectedly
> funambol successful
> zyb: failed
The known anchor problem.
> google successful
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 5 months
Re: [SyncEvolution] EXDATE in iCalendar 2.0
by Patrick Ohly
Hello!
I've started to look at a workaround for this inside the Synthesis XML
configuration. My expectation is that if DTSTART is relative to a
timezone, then EXDATEs should also be relative to that same timezone.
UTC or floating do not make sense, because if the timezone definition
changes, those exceptions might no longer match.
Following that logic I came up with this snipped for the incoming
calendar event script:
// Make sure that all EXDATE times are in the same timezone as the start
// time. Some servers send them as UTC, which is all fine and well, but
// only if the timezone definition doesn't change. Also, libical does not
// handle such UTC EXDATEs, so let's convert it while the UTC and
// time zone definition (hopefully) are in sync.
if (TIMEZONE(DTSTART) != "UTZ" && !ISFLOATING(DTSTART)) {
i = 0;
while (i<SIZE(EXDATES)) {
if (TIMEZONE(EXDATES[i]) == "UTC" || ISFLOATING(EXDATES[i])) {
// "unfloat" floating time stamps: not sure whether that occcurs
// in practice, but it looks as wrong as UTC EXDATEs
EXDATES[i] = CONVERTTOZONE(EXDATES[i],DTSTART,TRUE);
}
i=i+1;
}
}
Does that look right?
I think it would work if it wasn't for the libical VTIMEZONE parsing
problem on my test machine (see other email). Because of that problem,
the CONVERTTOZONE() macro uses the incorrect GMT offset of 0 and the
EXDATE still doesn't match.
Bye, Patrick
On Fri, 2009-07-03 at 09:07 -0700, Patrick Ohly wrote:
> Hello!
>
> Yanshuang found that Evolution and ScheduleWorld do not quite agree on
> the interpretation of EXDATE:
> http://bugzilla.moblin.org/show_bug.cgi?id=1805#c4
>
> The server sends a recurring meeting relative to TZID=Europe/Berlin with
> EXDATE in UTC time:
>
> DTSTART;TZID=/softwarestudio.org/Tzfile/Europe/Berlin:20090630T190000
> RRULE:FREQ=DAILY;UNTIL=20090702T170000Z;INTERVAL=1
> EXDATE:20090630T170000Z
>
> Evolution (or, more likely, libical) does not recognize that the first
> instance of the event was cancelled and continues to show it.
>
> Evolution itself creates an EXDATE relative to the time zone:
> EXDATE:20090630T190000
>
> Lukas, Mark, do you have any thoughts on this?
>
> Yongsheng might have found a similar issue with the Funambol server:
> http://bugzilla.moblin.org/show_bug.cgi?id=2422#c34
>
12 years, 5 months
[SyncEvolution] 0.9 final: when and what?
by Patrick Ohly
Hello!
So when do we want to release 0.9? I think we should give it at least
another week, so that users have a chance to download the source and/or
binaries and provide feedback.
That's a lower bound. If we want to include support for ZYB.com and
mobical.net, then it might take longer. ScheduleWorld sync has known
issues which Mark is slow to comment on and there's one open Funambol
issue.
Yongsheng, what's your estimate how long the testing with mobical.net
might take until we are at the usual level (testing passes without a
hitch, known issues documented)? And for ZYB.com (assuming that ZYB
comments on the anchor problem)?
A rough figure is enough, and no need to hurry: I don't consider these
critical for 0.9.
Any comments whether additional servers are worth waiting with the
release? Are there deadlines (distros?) that we should know about?
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 5 months
[SyncEvolution] status 27.07.2009: more release work...
by Patrick Ohly
Hi!
Progress since last week:
SyncEvolution 0.9 beta 3 was released. Google contact sync is now
officially supported. This required some last-minute (and post release,
duh) tweaks because Google only supports https, but SSL certificate
checking with libsoup/libgnutls of that particular certificate fails,
despite having the CA certs in Moblin now. A patch for libsoup was
submitted and accepted, but because it is not in Moblin, certificate
checking had to be disabled for Google.
We are giving 0.9 some more time for testing, in particular for server
interoperability (ZYB.com, mobical.net, some issues with ScheduleWorld).
To encourage feedback from users, binaries for non-Moblin Linux desktops
were also prepared as part of the beta 3.
Development:
* Congwu: analysis of Google problems (ORG #4631, image size
limitation #4638, MaxObjSize behavior #4635), implemented first
revision of message resending (#3427)
* Yongsheng: mobical.net analysis and discussion with Tactel
(#3009)
* Patrick: 0.9 beta 3 release, added test cases for ScheduleWorld
problem with slow syncs (#4703), added ZYB.com to nightly
testing so that the logs showing the anchor problem are
available (#2424), fixed regression in printing of command line
error message (#4676), analyzed libsoup/gnutls issue with
Google's SSL certificate and added workaround (#4551)
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 5 months
[SyncEvolution] D-Bus API + server mode
by Patrick Ohly
Hello!
I promised Jussi and Frederik that I would down some thoughts about the
next incarnation of the D-Bus API. As you can imagine, I was busy with
0.9 beta 3, and therefore haven't had the time yet.
To get started, let's summarize the current status: the D-Bus API is a
private API between syncevo-dbus-server and the sync-ui. It has the
features that were needed, but is not complete yet. Anyone using it has
to be aware that it can (and will change).
We need to stabilize the API and make it more generally useful. In
addition, it needs new functionality to implement the server stubs
mentioned in the design of the next-generation SyncEvolution:
http://moblin.org/documentation/syncevolution/direct-synchronization-aka-...
Jussi already filed some issues about the D-Bus API:
dbus api should expose sync reports as hashtables
http://bugzilla.moblin.org/show_bug.cgi?id=2061
dbus api modifications , GetSyncStatus() and signal
http://bugzilla.moblin.org/show_bug.cgi?id=2412
dbus api modifications, move progress logic to server
http://bugzilla.moblin.org/show_bug.cgi?id=2413
The first one is an implementation detail. The C++ SyncReport class
covers basically everything which can be said about a sync session,
ranging from statistics to success/failure information. New information
may get added, so we need a backwards-compatible way of sending it over
D-Bus.
The other two are about a core principle of the D-Bus design: the D-Bus
server runs sessions, D-Bus clients view results and/or control the
sessions. We are not there yet, currently the sync-ui contains quite a
bit of logic.
I've mentioned sync sessions. We should design the API so that more than
one session can be active at a time even though right now, the server
cannot run more than one at a time. Changing that would be both hard
(there are some global instances, for example logging; the
EvolutionSyncClient API takes control while a session runs) and provide
little benefit (only one session should modify local data at a time).
The server has to enforce that only one session runs at a time, which
could be done by queuing requests.
For simple clients it makes sense to limit the user interface also to
just one session. However, a client should be able to find the active
session and adapt to the one which is currently running.
The information about a session must include:
* configuration, using well-known names that can be presented to
the user (current usage model)
* peer information, for server configurations that define local
databases that other devices connect to (new server mode)
* a flag to distinguish between these two different configurations
* executing/waiting for IO/completed state, including signals when
the state changes
* session can be started/suspended: it should not be possible to
start a session unless the transport is up and/or the peer is
present (Bluetooth). We might want to disallow suspending if we
know that the peer doesn't support it.
* progress information
I think the difference between "actively doing something" and "waiting"
is relevant. A sync session can hang for a long time without any
progress update when a HTTP server or the network are slow. As I said in
"UI needs a progress
spinner" (http://bugzilla.moblin.org/show_bug.cgi?id=2229), the spinner
should only be active when the client is waiting, not when it is doing
something. If we want to be really fancy, we could spin a "waiting for
IO" animation while waiting and something else while processing (turning
gears?).
In the design for server I speculated that a session might be identified
with the configuration name. I think I would prefer a more abstract
handle. That way we can use the handle to refer to historic sessions of
a particular configuration.
Another update on the design is the handling of "server configurations".
Originally I wanted to keep the current concept of "one peer, one
configuration" because it allows us to keep the current file layout
(config.ini + .synthesis + .internal.ini). But this is problematic in
several ways.
First, it requires that the peer is known before we initialize sources
and the Synthesis Engine. The peer is identified inside the SyncML
message and only becomes available to SyncEvolution as part of the
session initialization inside the Synthesis Engine. This is a
contradiction which would have required peeking into the message.
Second, it duplicates the configuration information. Third, database
dumps become harder to find (but that is also a problem with one server
configuration and one client configuration, albeit perhaps less
common/severe).
Therefore I think we'll have to change the file layout and the internal
APIs so that SyncSources can be instantiated before the engine, but then
get access to per-peer files only later after the session has started.
The file layout has to be changed accordingly. The .synthesis directory
can be shared between peers, the Synthesis Engine takes care of tracking
which information belongs to which peer.
Coming back to the D-Bus client/server communication. I think it would
be useful to pass a stream of log messages from the server to the
client(s). These log messages are by design not translated, so they
should only be shown to advanced users. With this in place we could
change the command line client so that it uses the server instead of
calling libsyncevolution directly.
For normal users, the information provided by the server in addition to
this raw text must be rich enough to generate localized messages. This
can be difficult ("Can not do sync from GUI, and error info is
inconsistent with commands", #4660, "Synthesis error codes and
explanation", #2069). Right now, the D-Bus API directly passes through
Synthesis event codes. We might have to add and document our own
intermediate translation of these events.
In addition to reading and presenting information, there is also a
variety of operations that controllers need to do:
* start a session
* create and update configurations
* browse and restore data from the backups
* parse and execute command line parameters (so that the
"syncevolution" command line tool really is just a main()
function with some D-Bus code, unless it is compiled in the
traditional "call local class" mode)
This is all that I can think of for the UI<->D-Bus server interaction.
Did I forget anything? Any comments?
Jussi, do you think you can take over this part of the discussion,
compare it against the current D-Bus API and propose the next revision
of it? That doesn't mean that you have to implement all of it. We could
split it so that you concentrate on the GUI and someone else does the
server side.
In the meantime, I'll go back to the server design and write more about
the D-Bus API required for that after clarifying some more aspects (in
particular around OBEX).
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 6 months
Re: [SyncEvolution] compiling more than one gui
by Patrick Ohly
On Sat, 2009-07-25 at 17:20 +0100, Peter Robinson wrote:
> I want to be able to have them both as sub packages on Fedora so if
> your running a gnome desktop you install the syncevolution-gtk package
> and if you're running a moblin interface you can install teh
> syncevolution-moblin package and get the moblin interface.
I see. I created an issue for this and will implement that before the
final 0.9:
http://bugzilla.moblin.org/show_bug.cgi?id=4753
As noted in the issue, the content of the desktop files would be
identical, so in the (unlikely?) case that both are installed, the user
wouldn't be able to distinguish between them.
In later versions we could change the names, but currently we are in a
string freeze.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 6 months
