About 1.4.99.2 ============== 1.4.99.2 is a new development snapshot. It enhances interoperability with CardDAV servers and in particular Google Contacts considerably. Contact data gets converted to and from the format typically used by CardDAV servers, so now anniversary, spouse, manager, assistant and instant message information are exchanged properly. Categories are not supported by Google CardDAV and thus still get lost. Custom labels get stored in EDS as extensions and no longer get lost when updating some other aspects of a contact. However, Evolution does not show custom labels and removes them when editing a property which has a custom label (BGO #730636). Scanning for CardDAV/CalDAV resources was enhanced. It now finds additional calendars with Google CalDAV and works with iCloud. However, syncing with iCloud ran into a server bug (reported as 17001498 "CalDAV REPORT drops calendar data") and needs further work. The Ubuntu Online Accounts backend was added to syncevolution.org binaries, targeting Ubuntu Saucy and later. Details: * vcard profile: avoid data loss during merging When resolving a merge conflict, repeating properties were taken wholesale from the winning side (for example, all email addresses). If a new email address had been added on the loosing side, it got lost. Arguably it is better to preserve as much data as possible during a conflict. SyncEvolution now does that in a merge script by checking which properties in the loosing side do not exist in the winning side and copying those entries. Typically only the main value (email address, phone number) is checked and not the additional meta data (like the type). Otherwise minor differences (for example, both sides have same email address, but with different types) would lead to duplicates. Only addresses are treated differently: for them all attributes (street, country, city, etc.) are compared, because there is no single main value. * engine: UID support in contact data Before, the UID property in a vCard was ignored by the engine. Backends were responsible for ensuring that the property is set if required by the underlying storage. This turned out to be handled incompletely in the WebDAV backend. This change moves this into the engine: - UID is now field. It does not get used for matching because the engine cannot rely on it being stored by both sides. - It gets parsed if present, but only generated if explicitly enabled (because that is the traditional behavior). - It is never shown in the DevInf's CtCap because the Synthesis engine would always show it regardless whether a rule enabled the property. That's because rules normally only get triggered after exchanging DevInf and thus DevInf has to be rule-independent. We don't want it shown because then merging the incoming item during a local sync would use the incoming UID, even if it is empty. - Before writing, ensure that UID is set. When updating an existing item, the Synthesis engine reads the existing item, preserves the existing UID unless the peer claims to support UID, and then updates with the existing UID. This works for local sync (where SyncEvolution never claims to support UID when talking to the other side). It will break with peers which have UID in their CtCap although they rewrite the UID and backends whose underlying storage cannot handle UID changes during an update (for example, CardDAV). * CardDAV: use Apple/Google/CardDAV vCard flavor In principle, CardDAV servers support arbitrary vCard 3.0 data. Extensions can be different and need to be preserved. However, when multiple different clients or the server's Web UI interpret the vCards, they need to agree on the semantic of these vCard extensions. In practice, CardDAV was pushed by Apple and Apple clients are probably the most common clients of CardDAV services. When the Google Contacts Web UI creates or edits a contact, Google CardDAV will send that data using the vCard flavor used by Apple. Therefore it makes sense to exchange contacts with *all* CardDAV servers using that format. This format could be made configurable in SyncEvolution on a case-by-case basis; at the moment, it is hard-coded. During syncing, SyncEvolution takes care to translate between the vCard flavor used internally (based on Evolution) and the CardDAV vCard flavor. This mapping includes: X-AIM/JABBER/... <-> IMPP + X-SERVICE-TYPE Any IMPP property declared as X-SERVICE-TYPE=AIM will get mapped to X-AIM. Same for others. Some IMPP service types have no known X- property extension; they are stored in EDS as IMPP. X- property extensions without a known X-SERVICE-TYPE (for example, GaduGadu and Groupwise) are stored with X-SERVICE-TYPE values chosen by SyncEvolution so that Google CardDAV preserves them (GroupWise with mixed case got translated by Google into Groupwise, so the latter is used). Google always sends an X-ABLabel:Other for IMPP. This is ignored because the service type overrides it. The value itself also gets transformed during the mapping. IMPP uses an URI as value, with a chat protocol (like "aim" or "xmpp") and some protocol specific identifier. For each X- extension the protocol is determined by the property name and the value is the protocol specific identifier without URL encoding. X-SPOUSE/MANAGER/ASSISTANT <-> X-ABRELATEDNAMES + X-ABLabel The mapping is based on the X-ABLabel property attached to the X-ABRELATEDNAMES property. This depends on the English words "Spouse", "Manager", "Assistant" that Google CardDAV and Apple devices seem to use regardless of the configured language. As with IMPP, only the subset of related names which have a corresponding X- property extension get mapped. The rest is stored in EDS using the X-ABRELATEDNAMES property. X-ANNIVERSARY <-> X-ABDATE Same here, with X-ABLabel:Anniversary as the special case which gets mapped. X-ABLabel parameter <-> property CardDAV vCards have labels attached to arbitrary other properties (TEL, ADR, X-ABDATE, X-ABRELATEDNAMES, ...) via vCard group tags: item1.X-ABDATE:2010-01-01 item1.X-ABLabel:Anniversary The advantage is that property values can contain arbitrary characters, including line breaks and double quotation marks, which is not possible in property parameters. Neither EDS nor KDE (judging from the lack of responses on the KDE-PIM mailing list) support custom labels. SyncEvolution could have used grouping as it is done in CardDAV, but grouping is not used much (not at all?) by the UIs working with the vCards in EDS and KDE. It seemed easier to use a new X-ABLabel parameter. Characters which cannot be stored in a parameter get converted (double space to single space, line break to space, etc.) during syncing. In practice, these characters don't appear in X-ABLabel properties anyway because neither Apple nor Google UIs allow entering them for custom labels. The "Other" label is used by Google even in case where it adds no information. For example, all XMPP properties have an associated X-ABLabel=Other although the Web UI does not provide a means to edit or show such a label. Editing the text before the value in the UI changes the X-SERVICE-TYPE parameter value, not the X-ABLabel as for other fields. Therefore the "Other" label is ignored by removing it during syncing. X-EVOLUTION-UI-SLOT (the parameter used in Evolution to determine the order of properties in the UI) gets stored in CardDAV. The only exception is Google CardDAV which got confused when an IMPP property had both X-SERVICE-TYPE and X-EVOLUTION-UI-SLOT parameters set. For Google, X-EVOLUTION-UI-SLOT is only sent on other properties and thus ordering of chat information can get lost when syncing with Google. * synccompare: support grouping and quoted parameter strings Grouped properties are sorted first according to the actual property name, then related properties are moved to the place where their group tag appears first. The first grouped property gets a "- " prefix, all following ones are just indended with " ". The actual group tag is not part of the normalized output, because its value is irrelevant: BDAY:19701230 - EMAIL:john@custom.com X-ABLabel:custom-label2 ... FN:Mr. John 1 Doe Sr. - IMPP;X-SERVICE-TYPE=AIM:aim:aim X-ABLabel:Other ... - X-ABDATE:19710101 X-ABLabel:Anniversary Redundant tags (those set for only a single property, X-ABLabel:Other) get removed as part of normalizing an item. * WebDAV: use server's order when listing collections When doing a recursive scan of the home set, preserve the order of entries as reported by the server and check the first one first. The server knows better which entries are more relevant for the user (and thus should be the default) or may have some other relevant order. Previously, SyncEvolution replaced that order with sorting by URL, which led to a predictable, but rather meaningless order. For example, Google lists the users own calendar first, followed by the shared calendars sorted alphabetical by their name. Now SyncEvolution picks the main calendar as default correctly when scanning from https://www.google.com/calendar/dav/. * WebDAV: improved database search (Google, Zimbra) Zimbra has a principal URL that also serves as home set. When using it as start URL, SyncEvolution only looked the URL once, without listing its content, and thus did not find the databases. When following the Zimbra principal URL indirectly, SyncEvolution did check all of the collections there recursively. Unfortunately that also includes many mail folders, causing the scan to abort after checking 1000 collections (an internal safe guard). The solution for both includes tracking what to do with a URL. For the initial URL, only meta data about the URL itself gets checked. Recursive scanning is only done for the home set. If that home set contains many collections, scanning is still slow and may run into the internal safe guard limit. This cannot be avoided because the CalDAV spec explicitly states that the home set may contain normal collections which contain other collections, so a client has to do the recursive scan. When looking at a specific calendar, Google CalDAV does not report what the current principal or the home set is and therefore SyncEvolution stopped after finding just the initial calendar. Now it detects the lack of meta information and adds all parents also as candidates that need to be looked at. The downside of this is that it doesn't know anything about which parents are relevant, so it ends up checking https://www.google.com/calendar/ and https://www.google.com/. In both cases Basic Auth gets rejected with a temporary redirect to the Google login page, which is something that SyncEvolution must ignore immediately during scanning without applying the resend workaround for "temporary rejection of valid credentials" that can happen for valid Google CalDAV URLs. * WebDAV: enhanced database search (Google Calendar) Additional databases where not found for several reasons. SyncEvolution ignored all shared calendars (http://calendarserver.org/ns/shared) and Google marks the additional calendars that way. The other problem was that the check for leaf collections (= collections which cannot contain other desired collections) incorrectly excluded those collections instead of only preventing listing of their content. With this change, https://www.google.com/calendar/dav/?SyncEvolution=Google can be used as starting point for Google Calendar. * WebDAV: fix database scan on iCloud The calendar home set URL on iCloud (the one ending in /calendars/) is declared as containing calendar data. That was enough for SyncEvolution to accept it incorrectly as calendar. However, the home set only contains calendar data indirectly. * WebDAV: support redirects between hosts and DNS SRV lookup based on URL When finding a new URL, we must be prepared to reinitialize the Neon session with the new host settings. iCloud does not have .well-known support on its www.icloud.com server. To support lookup with a non-icloudd.com email address, we must do DNS SRV lookup when access to .well-known URLs fails. We do this without a www prefix on the host first, because that is what happens to work for icloud.com. With these changes it becomes possible to do database scans on Apple iCloud, using syncURL=https://www.icloud.com or syncURL=https://icloud.com. Giving the syncURL like this is only necessary for a username that does not end in @icloud.com. When the syncURL is not set, the domain for DNS SRV lookup is taken from the username. * WebDAV: more efficient item creation PUT has the disadvantage that a client needs to choose a name and then figure out what the real name on the server is. With Google CardDAV that requires sending another request and only works because the server happens to remember the original name (which is not guaranteed!). POST works for new items without a name and happens to be implemented by Google such that the response already includes all required information (new name and revision string). POST is checked for as described in RFC 5995 once before creating a new item. Servers which don't support it continue to get a PUT. * WebDAV: send "User-Agent: SyncEvolution" Apple iCloud servers reject requests unless they contain a User-Agent header. The exact value doesn't seem to matter. Making the string configurable might be better, but can still be done later when it is more certain whether and for what it is needed. * WebDAV: refactor and fix DNS SRV lookup The syncevo-webdav-lookup script was not packaged. It did not report "not found" DNS results correctly and the caller did not check for this either, so when looking up the information for a domain which does not have DNS SRV entries, SyncEvolution ended up retrying for while as if there had been a temporary lookup problem. * signon: make Accounts optional The new "signon" provider only depends on lib[g]signon-glib. It uses gSSO if found, else UOA. Instead of pulling parameters and the identity via libaccounts-glib, the user of SyncEvolution now has to ensure that the identity exists and pass all relevant parameters in the "signon:" username. * gSSO: adapt to gSSO >= 2.0 * config templates: Funambol URLs Funambol turned of the URL redirect from my.funambol.com to onemedia.com. The Funambol template now uses the current URL. Users with existing Funambol configs must updated the syncURL property manually to https://onemediahub.com/sync Kudos to Daniel Clement for reporting the change. * command line: fix --update from directory The "--update <dir name>" operation was supposed to take the item luids from the file names inside the directory. That part had not been implemented, turning the operation accidentally into an "--import". Also missing was the escaping/unescaping of luids. Now the same escaping is done as in command line output and command line parsing to make the luids safe for use as file name. * testing: added server-specific tests for CardDAV covering remote item formats and edit conflicts. Upgrading from releases <= 1.3.99.4: ------------------------------------ If the value of "username/databaseUser/proxyUser" contains a colon, the "user:" prefix must be added to the value, to continue treating it like a plain user name and not some reference to an unknown identity provider (like "id:", "goa:", "signon:", etc.). The lookup of passwords in GNOME Keyring was updated slightly in 1.3.99.5. It may be necessary to set passwords anew if the old one is no longer found. Upgrading from release 1.2.x: ----------------------------- The sync format of existing configurations for Mobical (aka Everdroid) must be updated manually, because the server has encoding problems when using vCard 3.0 (now the default for Evolution contacts): syncevolution --configure \ syncFormat=text/x-vcard \ mobical addressbook The Funambol template explicitly enables usage of the "refresh-from-server" sync mode to avoid getting throttled with 417 'retry later' errors. The same must be added to existing configs manually: syncevolution --configure \ enableRefreshSync=TRUE \ funambol Upgrading from releases before 1.2: ----------------------------------- Old configurations can still be read. But writing, as it happens during a sync, must migrate the configuration first. Releases >= 1.2 automatically migrates configurations. The old configurations will still be available (see "syncevolution --print-configs") but must be renamed manually to use them again under their original names with older SyncEvolution releases. Source, Installation, Further information ========================================= http://syncevolution.org/blogs/pohly/2014/syncevolution-14992-released Source code bundles for users are available in http://downloads.syncevolution.org/syncevolution/sources and the original source is in the git repositories http://cgit.freedesktop.org/SyncEvolution/ i386, lpia and amd64 binaries for Debian-based distributions are available via the "unstable" syncevolution.org repository. Add the following entry to your /apt/source.list: deb http://downloads.syncevolution.org/apt unstable main Then install "syncevolution-evolution", "syncevolution-kde" and/or "syncevolution-activesync". These binaries include the "sync-ui" GTK GUI and were compiled for Ubuntu 10.04 LTS (Lucid), except for ActiveSync binaries which were compiled for Debian Wheezy, Ubuntu Saucy and Ubuntu Trusty. A backend for Ubuntu Online Accounts was compiled on Ubuntu Saucy. The packages mentioned above are meta-packages which pull in suitable packages matching the distro during installation. Older distributions like Debian 4.0 (Etch) can no longer be supported with precompiled binaries because of missing libraries, but the source still compiles when not enabling the GUI (the default). The same binaries are also available as .tar.gz and .rpm archives in http://downloads.syncevolution.org/syncevolution/. In contrast to 0.8.x archives, the 1.x .tar.gz archives have to be unpacked and the content must be moved to /usr, because several files would not be found otherwise. After installation, follow the http://syncevolution.org/documentation/getting-started steps. -- Patrick Ohly, on behalf of everyone who has helped to make SyncEvolution possible: http://syncevolution.org/about/contributors