Re: [SyncEvolution] syncevolution and a radicale caldev server
by Patrick Ohly
On Fri, 2012-02-17 at 15:14 +0100, Gregor Horvath wrote:
> Am Fri, 17 Feb 2012 11:12:52 +0100
> schrieb Patrick Ohly <patrick.ohly(a)intel.com>:
>
> > On Tue, 2012-02-14 at 15:57 +0100, Patrick Ohly wrote:
> > > But as Radicale is an example where multiple databases are possible
> > > (in
> > > contrast to Google), it's worth spelling out explicitly how such a
> > > config can be created:
> >
> > [...]
> >
> > William, did this work?
> >
> > Perhaps you (or someone else) has the time to turn the instructions
> > into a proper HOWTO? I created a place holder Wiki page for it here:
> > http://syncevolution.org/wiki/synchronizing-radicale
> >
> >
William sent me some debug logs, but then ran out of time. So I've
installed Radicale myself and found an issue that'll break
SyncEvolution: items reported by the server include double slashes in
the path.
<multistatus xmlns="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav">
<response>
<href>/public_user/calendar/calendar_1//1234567890(a)dummy.ics</href>
^^
SyncEvolution will then use /1234567890(a)dummy.ics as path in future
requests, which Radicale rejects with either 404 or 401 errors.
I'll work around that in SyncEvolution. I'm not sure whether there is a
solution with 1.2.2.
Have you considered Calypso (http://keithp.com/blogs/calypso/)? I don't
know yet whether it works with SyncEvolution; if there is interest, then
I will try it.
> I am trying to do the same thing (with one calender to start with)
> N900 <-> radicale but it did not work, because it seems the Maemo
> calendar is not found:
>
> http://pastebin.com/huMXDsEx
>
> What I am doing wrong?
You have additional spaces after your \ character. The slash must be at
the end of each line.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
5 years, 7 months
[SyncEvolution] SyncEvolution 1.4.99.4
by Patrick Ohly
About 1.4.99.4
==============
This is the first release candidate for 1.5. No further changes are
planned except for fixing yet-to-be-discovered bugs - so find them
now! :-)
One focus in this release was on minimizing CPU consumption and disk
writes. The most common case, a two-way sync with no changes on either
side, no longer rewrites any meta data files. CPU consumption during
local sync was reduced to one third by exchanging messages via shared
memory instead of internal D-Bus. Redundant vCard decode/encode on the
sending side of PBAP and too agressive flushing of meta data during a
normal sync were removed. Altogether, sending 1000 contacts with photo
data in a refresh-from-server local sync takes only one sixth of the
CPU cycles compared to 1.3.99.3 (measured with valgrind's callgrind on
x86_64).
Based on community feedback and discussions, the terminology used in
SyncEvolution for configuration, local sync and database access was
revised. Some usability issues with setting up access to databases
were addressed. For Google, the obsolete SyncML config template was
removed and CalDAV/CardDAV were merged into a single "Google"
template.
Using Google Calendar/Contacts with OAuth2 authentication on a
headless server becomes a bit easier: it is possible to set up access
on one system with a GUI using either gSSO or GNOME Online Accounts,
then take the OAuth2 refresh token and use it in SyncEvolution on a
different system. See
http://cgit.freedesktop.org/SyncEvolution/syncevolution/tree/src/backends...
Some issues accessing Apple iCloud were fixed such that CardDAV works
by just giving SyncEvolution username=foobar(a)icloud.com and password. No
throrough testing was done, so iCloud support is still experimental.
The PIM Manager API also supports Google Contact syncing. Some
problems with suspending a PBAP sync were fixed. Suspend/abort can
be tested with the sync.py example.
The EDS memo backend is able to switch between syncing in plain
text and iCalendar 2.0 VJOURNAL automatically.
Details:
* oauth2: new backend using libsoup/libcurl
New backend implements identity provider for obtaining OAuth2 access
token for systems without HMI support.
Access token is obtained by making direct HTTP request to OAuth2 server
and using refresh token obtained by user in some other way.
New provider automatically updates stored refresh token when OAuth2
server is issuing new one.
* PBAP: use raw text items
This avoids the redundant parse/generate step on the sending
side of the PBAP sync.
* datatypes: raw text items with minimal conversion (FDO #52791)
When using "raw/text/calendar" or "raw/text/vcard" as SyncEvolution
"databaseFormat", all parsing and conversion is skipped. The backend's
data is identical to the item data in the engine. Finding duplicates
in a slow sync is very limited when using these types because the entire
item data must match exactly.
This is useful for the file backend when the goal is to store an exact copy
of what a peer has or for limited, read-only backends (PBAP). The downside
of using the raw types is that the peer is not given accurate information
about which vCard or iCalendar properties are supported, which may cause
some peers to not send all data.
* engine: flush map items less frequently
The Synthesis API does not say explicitly, but in practice all map
items get updated in a tight loop. Rewriting the m_mappingNode (case
insensitive string comparisons) and serialization to disk
(std::ostrstream) consume a significant amount of CPU cycles and cause
extra disk writes that can be avoided by making some assumptions about
the sequence of API calls and flushing only once.
* SoupTransport: drop CA file check
It used to be necessary to specify a CA file for libsoup to enable SSL
certificate checking. Nowadays libsoup uses the default CA store
unless told otherwise, so the check in SyncEvolution became
obsolete. However, now there is a certain risk that no SSL checking is
done although the user asked for it (when libsoup is not recent enough
or compiled correctly).
* local sync: exchange SyncML messages via shared memory
Encoding/decoding of the uint8_t array in D-Bus took a surprisingly
large amount of CPU cycles relative to the rest of the SyncML message
processing. Now the actual data resides in memory-mapped temporary
files and the D-Bus messages only contain offset and size inside these
files. Both sides use memory mapping to read and write directly.
For caching 1000 contacts with photos on a fast laptop, total sync
time roughly drops from 6s to 3s.
To eliminate memory copies, memory handling in libsynthesis or rather,
libsmltk is tweaked such that it allocates the buffer used for SyncML
message data in the shared memory buffer directly. This relies on
knowledge of libsmltk internals, but those shouldn't change and if they
do, SyncEvolution will notice ("unexpected send buffer").
* local sync: avoid updating meta data when nothing changed
The sync meta data (sync anchors, client change log) get updated after
a sync even if nothing changed and the existing meta data could be
used again. This can be skipped for local sync, because then
SyncEvolution can ensure that both sides skip updating the meta
data. With a remote SyncML server that is not possible and thus
SyncEvolution has to update its data.
This optimization is only used for local syncs with one source. It is
based on the observation that when the server side calls
SaveAdminData, the client has sent its last message and the sync is
complete. At that point, SyncEvolution can check whether anything has
changed and if not, skip saving the server's admin data and stop the
sync without sending the real reply to the client.
Instead the client gets an empty message with "quitsync" as content
type. Then it takes shortcuts to close down without finalizing the
sync engine, because that would trigger writing of meta data
changes. The server continues its shutdown normally.
This optimization is limited to syncs with a single source, because
the assumption about when aborting is possible is harder to verify
when multiple sources are involved.
* PIM: include CardDAV in CreatePeer()
This adds "protocol: CardDAV" as a valid value, with corresponding
changes to the interpretation of some existing properties and some new
ones. The API itself is not changed.
Suspending a CardDAV sync is possible. This freezes the internal
SyncML message exchange, so data exchange with the CardDAV server may
continue for a while after SuspendPeer().
Photo data is always downloaded immediately. The "pbap-sync" flag
in SyncPeerWithFlags() has no effect.
Syncing can be configured to be one-way (local side is read-only
cache) or two-way (local side is read/write). Meta data must be
written either way, to speed up caching or allow two-way syncing. The
most common case (no changes on either side) will have to be optimized
such that existing meta data is not touched and thus no disk writes
occur.
* PIM: handle SuspendPeer() before and after transfer (FDO #82863)
A SuspendPeer() only succeeded while the underlying Bluetooth transfer
was active. Outside of that, Bluez errors caused SyncEvolution to
attempt a cancelation of the transfer and stopped the sync.
When the transfer was still queueing, obexd returns
org.bluez.obex.Error.NotInProgress. This is difficult to handle for
SyncEvolution: it cannot prevent the transfer from starting and has to
let it become active before it can suspend the transfer. Canceling
would lead to difficult to handle error cases (like partially parsed
data) and therefore is not done.
The Bluez team was asked to implement suspending of queued transfers
(see "org.bluez.obex.Transfer1 Suspend/Resume in queued state" on
linux-bluetooth(a)vger.kernel.org), so this case might not happen
anymore with future Bluez.
When the transfer completes before obexd processes the Suspend(),
org.freedesktop.DBus.Error.UnknownObject gets returned by
obexd. SyncEvolution can ignore errors which occur after the active
transfer completed. In addition, it should prevent starting the next
one. This may be relevant for transfer in chunks, although the sync
engine will also stop asking for data and thus typically no new
transfer gets triggered anyway.
* PIM: add suspend/resume/abort to sync.py
CTRL-C while waiting for the end of a sync causes an interactive
prompt to appear where one can choose been suspend/resume/abort and
continuing to wait. CTRL-C again in the prompt aborts the script.
* PIM: fix sync.py --sync-flags
The help text used single quotes for the JSON example instead of
the required double quotes. Running without --sync-flags was broken
because of trying to parse the empty string as JSON.
* command line: revise usability checking of datastores
When configuring a new sync config, the command line checks whether a
datastore is usable before enabling it. If no datastores were listed
explicitly, only the usable ones get enabled. If unusable datastores
were explicitly listed, the entire configure operation fails.
This check was based on listing databases, which turned out to be too
unspecific for the WebDAV backend: when "database" was set to some URL
which is good enough to list databases, but not a database URL itself,
the sources where configured with that bad URL.
Now a new SyncSource::isUsable() operation is used, which by default
just falls back to calling the existing Operations::m_isEmpty. In
practice, all sources either check their config in open() or the
m_isEmpty operation, so the source is usable if no error is
enountered.
For WebDAV, the usability check is skipped because it would require
contacting a remote server, which is both confusing (why does a local
configure operation need the server?) and could fail even for valid
configs (server temporarily down). The check was incomplete anyway
because listing databases gave a fixed help text response when no
credentials were given. For usability checking that should have
resulted in "not usable" and didn't.
The output during the check was confusing: it always said "listing
databases" without giving a reason why that was done. The intention
was to give some feedback while a potentially expensive operation
ran. Now the isUsable() method itself prints "checking usability" if
(and only if!) such a check is really done.
Sometimes datastores were checked even when they were about to be
configure as "disabled" already. Now checking such datastores is
skipped.
* EDS: memo syncing as iCalendar 2.0 (FDO #52714)
When syncing memos with a peer which also supports iCalendar 2.0 as
data format, the engine will now pick iCalendar 2.0 instead of
converting to/from plain text. The advantage is that some additional
properties like start date and categories can also be synchronized.
The code is a lot simpler, too, because the EDS specific iCalendar 2.0
<-> text conversion code can be removed.
* datatypes: text/calendar+plain revised heuristic
When sending a VEVENT, DESCRIPTION was set to the SUMMARY if empty. This may
have been necessary for some peers, but for notes (= VJOURNAL) we don't know
that (hasn't been used in the past) and don't want to alter the item
unnecessarily, so skip that part and allow empty DESCRIPTION.
When receiving a plain text note, the "text/calendar+plain" type
used to store the first line as summary and the rest as description.
This may be correct in some cases and wrong in others.
The EDS backend implemented a different heuristic: there the first
line is copied into the summary and stays in the description. This
makes a bit more sense (the description alone is always enough to
understand the note). Therefore and to avoid behavioral changes
for EDS users when switching the EDS backend to use text/calendar+plain,
the engine now uses the same approach.
* source -> datastore rename, improved terminology
The word "source" implies reading, while in fact access is read/write.
"datastore" avoids that misconception. Writing it in one word emphasizes
that it is single entity.
While renaming, also remove references to explicit --*-property
parameters. The only necessary use today is "--sync-property ?"
and "--datastore-property ?".
--datastore-property was used instead of the short --store-property
because "store" might be mistaken for the verb. It doesn't matter
that it is longer because it doesn't get typed often.
--source-property must remain valid for backward compatility.
As many user-visible instances of "source" as possible got replaced in
text strings by the newer term "datastore". Debug messages were left
unchanged unless some regex happened to match it.
The source code will continue to use the old variable and class names
based on "source".
Various documentation enhancements:
Better explain what local sync is and how it involves two sync
configs. "originating config" gets introduces instead of just
"sync config".
Better explain the relationship between contexts, sync configs,
and source configs ("a sync config can use the datastore configs in
the same context").
An entire section on config properties in the terminology
section. "item" added (Todd Wilson correctly pointed out that it was
missing).
Less focus on conflict resolution, as suggested by Graham Cobb.
Fix examples that became invalid when fixing the password
storage/lookup mechanism for GNOME keyring in 1.4.
The "command line conventions", "Synchronization beyond SyncML" and
"CalDAV and CardDAV" sections were updated. It's possible that the
other sections also contain slightly incorrect usage of the
terminology or are simply out-dated.
* local sync: allow config name in syncURL=local://
Previously, only syncURL=local://@<context name> was allowed and used
the "target-config@context name" config as target side in the local
sync.
Now "local://config-name@context-name" or simply "local://config-name"
are also allowed. "target-config" is still the fallback if only a
context is give.
It also has one more special meaning: "--configure
target-config@google" will pick the "Google" template automatically
because it knows that the intention is to configure the target side
of a local sync. It does not know that when using some other name
for the config, in which case the template (if needed) must be
specified explicitly.
The process name in output from the target side now also includes the
configuration name if it is not the default "target-config".
* Google: remove SyncML template, combine CalDAV/CardDAV
Google has turned off their SyncML server, so the corresponding
"Google Contacts" template became useless and needs to be removed. It
gets replaced by a "Google" template which combines the three
different URLs currently used by Google for CalDAV/CardDAV.
This new template can be used to configure a "target-config@google"
with default calendar and address book database already enabled. The
actual URL of these databases will be determined during the first
sync using them.
The template relies on the WebDAV backend's new capability to search
multiple different entries in the syncURL property for databases. To
avoid listing each calendar twice (once for the legacy URL, once with
the new one) when using basic username/password authentication, the
backend needs a special case for Google and detect that the legacy URL
does not need to be checked.
* config: allow storing credentials for email address
When configuring a WebDAV server with username = email address and no
URL (which is possible if the server supports service discovery via
the domain in the email address), then storing the credentials in the
GNOME keyring used to fail with "cannot store password in GNOME
keyring, not enough attributes".
That is because GNOME keyring seemed to get confused when a network
login has no server name and some extra safeguards were added to
SyncEvolution to avoid this.
To store the credentials in the case above, the email address now gets
split into user and domain part and together get used to look up the
password.
Upgrading from releases <= 1.3.99.4:
------------------------------------
If the value of "username/databaseUser/proxyUser" contains a colon,
the "user:" prefix must be added to the value, to continue treating it
like a plain user name and not some reference to an unknown identity
provider (like "id:", "goa:", "signon:", etc.).
The lookup of passwords in GNOME Keyring was updated slightly in
1.3.99.5. It may be necessary to set passwords anew if the old one is
no longer found.
Upgrading from release 1.2.x:
-----------------------------
The sync format of existing configurations for Mobical (aka Everdroid)
must be updated manually, because the server has encoding problems when
using vCard 3.0 (now the default for Evolution contacts):
syncevolution --configure \
syncFormat=text/x-vcard \
mobical addressbook
The Funambol template explicitly enables usage of the
"refresh-from-server" sync mode to avoid getting throttled with 417
'retry later' errors. The same must be added to existing configs
manually:
syncevolution --configure \
enableRefreshSync=TRUE \
funambol
Upgrading from releases before 1.2:
-----------------------------------
Old configurations can still be read. But writing, as it happens
during a sync, must migrate the configuration first. Releases >= 1.2
automatically migrates configurations. The old configurations
will still be available (see "syncevolution --print-configs") but must
be renamed manually to use them again under their original names with
older SyncEvolution releases.
Source, Installation, Further information
=========================================
http://syncevolution.org/blogs/pohly/2014/syncevolution-14994-released
Source code bundles for users are available in
http://downloads.syncevolution.org/syncevolution/sources
and the original source is in the git repositories
http://cgit.freedesktop.org/SyncEvolution/
i386, lpia and amd64 binaries for Debian-based distributions are
available via the "unstable" syncevolution.org repository. Add the
following entry to your /apt/source.list:
deb http://downloads.syncevolution.org/apt unstable main
Then install "syncevolution-evolution", "syncevolution-kde" and/or
"syncevolution-activesync".
These binaries include the "sync-ui" GTK GUI and were compiled for
Ubuntu 10.04 LTS (Lucid), except for ActiveSync binaries which were
compiled for Debian Wheezy, Ubuntu Saucy and Ubuntu Trusty. A backend
for Ubuntu Online Accounts was compiled on Ubuntu Saucy. The packages
mentioned above are meta-packages which pull in suitable packages
matching the distro during installation.
Older distributions like Debian 4.0 (Etch) can no longer be supported
with precompiled binaries because of missing libraries, but the source
still compiles when not enabling the GUI (the default).
The same binaries are also available as .tar.gz and .rpm archives in
http://downloads.syncevolution.org/syncevolution/. In contrast
to 0.8.x archives, the 1.x .tar.gz archives have to be unpacked and the
content must be moved to /usr, because several files would not be found
otherwise.
After installation, follow the
http://syncevolution.org/documentation/getting-started steps.
--
Patrick Ohly, on behalf of everyone who has helped
to make SyncEvolution possible:
http://syncevolution.org/about/contributors
5 years, 10 months
[SyncEvolution] one-off request for paid support (korganizer + phone)
by Johannes Meier
Hi, I'm at my wit's end with syncevolution-akonadi. Maybe somebody can
configure it for me if I pay them, please?
The goal is to get the phone, the desktop calendars, and a few shared
calendars to sync in all directions.
* Desktop: Ubuntu 14.03LTS Linux. Any necessary package can be installed
(e.g. a syncml server).
* Korganizer/Kontact with 8 local calendars, 1 shared one (Google cal), 5
different subscribed ones, and the default birthday calendar. The logical
union of most of these should be synchronized to the phone. It's enough if
the past 2 weeks and future 6 months get synced because not everything will
fit. (A grand total of some 3000 calendar entries.) Appointments and
contacts are necessary, todo tasks would be nice, but I don't need notes.
* Handset: For now, it's a Sony K510i with a Feisar plugin. The
Sony_Ericsson_K750i template should do the trick. Syncing works with iSync
on Mac OS X 10.5.8. However, I'm fine with buying an inexpensive replacement
(smart)phone as long as it has an actual qwertz keyboard and real keys -
touchscreen doesn't count.
* Restrictions: (a) The phone and the desktop must synchronize without
access to the public internet. Bluetooth is fine (or with a different
handset, wireless LAN). The reason is partly availability on the road,
partly bad experience with things going out of business, partly this:
http://www.heise.de/ct/schlagseite/2014/16/gross.jpg
* Restrictions: (b) Compensation will be paid after syncing works and a
step-by-step setup guide is provided (because all software will eventually
break).
* Tools: ssh access is provided, tell me if you need anything else.
* Pay: 75EUR (around $95). I have no idea what tech support should cost,
this is what the plumber billed.
(Sorry, I might have doubleposted earlier with a wrong email address.)
Thank you!
5 years, 11 months
[SyncEvolution] N900/ownCloud - Segmentation Faults
by Christopher Calver
I have been successfully running syncevolution on my N900 (Maemo 5
Version 21.2011.38-1) for several years initially synchronising with
Horde3 and then with Radicale. Recently, I wanted to move the
synchronisation to ownCloud version 7. I have not managed to get this to
work: I have been having endless problems with the Segmentation Fault
error, both with configuration as root and configuration as user.
I thought I would clear my N900 and re-install syncevolution taking
great care at each step.
To be sure that I eliminate as many problems as possible which
architecture/version of syncevolution should I install?
Is an apt-get install with the extras-devel repository active sufficent?
Or should I download the package and dpkg -i from
https://people.debian.org/~ovek/maemo/
To clear the previous installation I have run the following commands
# apt-get purge –remove syncevolution
and then run
# find / -name syncevolution
to check there are no files left, OK, followed by
# rm -rf /root/.cache/syncevolution
# rm -rf /root/.config/syncevolution
# rm -rf /home/user/.cache/syncevolution
# rm -rf /home/user/.config/syncevolution
Is this sufficient or do I need to flash the N900?
Thanks in advance
5 years, 11 months
[SyncEvolution] FYI: Microsoft drop Nokia Sync
by Daniel CLEMENT
Hello,
Just a word to confirm what could be suspected from a recent attempt I
made at using the OVI service (AKA Nokia Sync). Like many Nokia
customers, shortly after that, I have received an e-mail from Microsoft
which reads:
[Quote] We are planning to discontinue the Nokia Sync service on 5
December 2014. After 5 December 2014, you will not be able to access
your data through the Nokia Sync service. We strongly encourage you to
export and/or migrate your data from the service before this date.
[/Quote]
So... it looks like OVI support can safely be dropped from
Syncevolution :-(
Best regards,
--
Daniel CLEMENT
5 years, 11 months
[SyncEvolution] Parse error
by Gregor Horvath
Hi,
I use SyncEvolution 1.2.2 on a N900 version in conjunction with a
Radicale server and Icedove clients on the Desktop.
It seems that syncevolution does not like some entries created from a
Debian Icedove Client.
I get the following error, although this snippet is validating fine
with http://severinghaus.org/projects/icv/
From the syncevolution log:
First ERROR encountered: error code from SyncEvolution fatal error
(local, status 10500): @default/calendar5: parsing ical:
BEGIN:VCALENDAR VERSION:2.0 PRODID:-//Synthesis AG//NONSGML SyncML
Engine V3.4.0.27//EN BEGIN:VTIMEZONE
TZID:Europe/Vienna
BEGIN:STANDARD
DTSTART:19671029T020000
RRULE:FREQ=MONTHLY;INTERVAL=12;BYDAY=-1SU
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:19870329T030000
RRULE:FREQ=MONTHLY;INTERVAL=12;BYDAY=-1SU
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
LAST-MODIFIED:20141005T065925Z
DTSTAMP:20141008T072729Z
CREATED:20141005T065735Z
UID:66348ec4-f535-4cca-8f8e-a4d5e9aa1edc
SUMMARY:Probe Ringwald
DESCRIPTION:Für 8.11
LOCATION:le Festsaal
DTSTART;TZID=Europe/Vienna:20141022T190000
END:VEVENT
END:VCALENDAR
Do you have an idea what is wrong?
lg
gregor
5 years, 11 months
[SyncEvolution] GOA credentials not found
by Todd Wilson
I may be overlooking something obvious here, but when using the latest
version (1.4.99.4-1) on Ubuntu 14.04, my GOA credentials are not being found:
% syncevolution --sync refresh-from-local google
[ERROR] goa:account_nnnn: need username+password as credentials
The account number I'm redacting as "nnnn" does exist (I can see it with
seahorse) and is authorized.
I've also tried deleting the account in seahorse and Online Accounts,
starting over with a new account, and replacing the old account number in my
google target-config with the new number, but I still get the same error.
5 years, 11 months
