[SyncEvolution] dealing with PHOTO:file:// (BMC #19661)
by Patrick Ohly
Hello!
I'm currently working on https://bugs.meego.com/show_bug.cgi?id=19661:
like N900/Maemo 5 before, MeeGo apps prefer to store URIs of local photo
files in the PIM storage instead of storing the photo data in EDS
(better for performance).
When such a contact is synced to a peer, the photo data must be included
in the vCard. Ove solved that in his N900 port by inlining the data when
reading from EDS, before handing the data to SyncEvolution/Synthesis.
This has the downside that the data must be loaded in all cases,
including those where it is not needed (slow sync comparison of the
other properties in server mode) and remains in memory much longer.
I'd like to propose a more efficient solution that'll work with all
backends. Ove, if this goes in, then you might be able to remove the
special handling of photos in your port.
The idea is that a) the data model gets extended to allow both URI and
data in PHOTO and b) a file URI gets replaced by the actual file content
right before sending (but not sooner).
Lukas, can you review the libsynthesis changes? See below. I pushed to
the bmc19661 branch in meego.gitorious.org. Some other fixes are also
included.
-----------------------
$ git log --reverse -p 92d2f367..bmc19661
commit 01c6ff4f7136d2c72b520818ee1ba89dc53c71f0
Author: Patrick Ohly <patrick.ohly(a)intel.com>
Date: Fri Jul 22 08:12:05 2011 +0200
SMLTK: fixed g++ 4.6 compiler warning
g++ 4.6 warns that the "rc" variable is getting assigned but never
used. smlEndEvaluation() must have been meant to return that error
code instead of always returning SML_ERR_OK - fixed.
diff --git a/src/syncml_tk/src/sml/mgr/all/mgrcmdbuilder.c b/src/syncml_tk/src/sml/mgr/all/mgrcmdbuilder.c
index ae040a4..601b530 100755
--- a/src/syncml_tk/src/sml/mgr/all/mgrcmdbuilder.c
+++ b/src/syncml_tk/src/sml/mgr/all/mgrcmdbuilder.c
@@ -698,7 +698,7 @@ SML_API Ret_t smlEndEvaluation(InstanceID_t id, MemSize_t *freemem)
return SML_ERR_WRONG_USAGE;
rc = xltEndEvaluation(id, (XltEncoderPtr_t)(pInstanceInfo->encoderState), freemem);
- return SML_ERR_OK;
+ return rc;
}
#endif
commit 8d5cce896dcc5dba028d1cfa18f08e31adcc6e73
Author: Patrick Ohly <patrick.ohly(a)intel.com>
Date: Fri Jul 22 08:36:22 2011 +0200
"blob" fields: avoid binary encoding if possible
This change is meant for the PHOTO value, which can contain both
binary data and plain text URIs. Other binary data fields might also
benefit when their content turns out to be plain text (shorter
encoding).
The change is that base64 encoding is not enforced if all characters
are ASCII and printable. That allows special characters like colon,
comma, and semicolon to appear unchanged in the content.
Regardless whether the check succeeds, the result is guaranteed to
contain only ASCII characters, either because it only contains those
to start with or because of the base64 encoding.
diff --git a/src/sysync/mimedirprofile.cpp b/src/sysync/mimedirprofile.cpp
index 4105d03..1499876 100644
--- a/src/sysync/mimedirprofile.cpp
+++ b/src/sysync/mimedirprofile.cpp
@@ -23,6 +23,7 @@
#include "syncagent.h"
+#include <ctype.h>
using namespace sysync;
@@ -2274,8 +2275,18 @@ sInt16 TMimeDirProfileHandler::generateValue(
}
// append to existing string
fldP->appendToString(outval,maxSiz);
- // force B64 encoding
- aEncoding=enc_base64;
+ // force B64 encoding if non-printable or non-ASCII characters
+ // are in the value
+ size_t len = outval.size();
+ for (size_t i = 0; i < len; i++) {
+ char c = outval[i];
+ if (!isascii(c) || !isprint(c)) {
+ aEncoding=enc_base64;
+ break;
+ }
+ }
+ // only ASCII in value: either because it contains only
+ // those to start with or because they will be encoded
aNonASCII=false;
}
else {
commit b69d0aecf612d0f009903179619a983706f3b8f7
Author: Patrick Ohly <patrick.ohly(a)intel.com>
Date: Fri Jul 22 08:44:21 2011 +0200
script error messages: fixed invalid memory access
If the text goes through macro expansion, then "aScriptText" is not
the chunk of memory which holds the script and "text" doesn't point
into it anymore. Therefore "text-aScriptText" calculates the wrong
offset.
Fixed by storing the real start of memory in a different variable
and using that instead of aScriptText.
Found when enclosing a string with single quotes instead of double
quotes. The resulting syntax error message contained garbled
characters instead of the real script line.
diff --git a/src/sysync/scriptcontext.cpp b/src/sysync/scriptcontext.cpp
index 35dff88..f21641e 100755
--- a/src/sysync/scriptcontext.cpp
+++ b/src/sysync/scriptcontext.cpp
@@ -2464,6 +2464,7 @@ void TScriptContext::Tokenize(TSyncAppBase *aAppBaseP, cAppCharP aScriptName, sI
text = itext.c_str();
}
// actual tokenisation
+ cAppCharP textstart = text;
SYSYNC_TRY {
// process text
while (*text) {
@@ -2540,7 +2541,7 @@ void TScriptContext::Tokenize(TSyncAppBase *aAppBaseP, cAppCharP aScriptName, sI
else if (StrToEnum(ItemFieldTypeNames,numFieldTypes,enu,p,il)) {
// check if declaration and if allowed
if (aNoDeclarations && lasttoken!=TK_OPEN_PARANTHESIS)
- SYSYNC_THROW(TTokenizeException(aScriptName, "no local variable declarations allowed in this script",aScriptText,text-aScriptText,line));
+ SYSYNC_THROW(TTokenizeException(aScriptName, "no local variable declarations allowed in this script",textstart,text-textstart,line));
// code type into token
aTScript+=TK_TYPEDEF; // token
aTScript+=1; // length of additional data
@@ -2616,7 +2617,7 @@ void TScriptContext::Tokenize(TSyncAppBase *aAppBaseP, cAppCharP aScriptName, sI
else if (strucmp(p,"WINNING",il)==0) objidx=OBJ_TARGET;
else if (strucmp(p,"TARGET",il)==0) objidx=OBJ_TARGET;
else
- SYSYNC_THROW(TTokenizeException(aScriptName,"unknown object name",aScriptText,text-aScriptText,line));
+ SYSYNC_THROW(TTokenizeException(aScriptName,"unknown object name",textstart,text-textstart,line));
text++; // skip object qualifier
aTScript+=TK_OBJECT; // token
aTScript+=1; // length of additional data
@@ -2641,13 +2642,13 @@ void TScriptContext::Tokenize(TSyncAppBase *aAppBaseP, cAppCharP aScriptName, sI
p=text;
while (isidentchar(*text)) text++;
if (text==p)
- SYSYNC_THROW(TTokenizeException(aScriptName,"missing macro name after $",aScriptText,text-aScriptText,line));
+ SYSYNC_THROW(TTokenizeException(aScriptName,"missing macro name after $",textstart,text-textstart,line));
itm.assign(p,text-p);
// see if we have such a macro
TScriptConfig *cfgP = aAppBaseP->getRootConfig()->fScriptConfigP;
TStringToStringMap::iterator pos = cfgP->fScriptMacros.find(itm);
if (pos==cfgP->fScriptMacros.end())
- SYSYNC_THROW(TTokenizeException(aScriptName,"unknown macro",aScriptText,p-1-aScriptText,line));
+ SYSYNC_THROW(TTokenizeException(aScriptName,"unknown macro",textstart,p-1-textstart,line));
TMacroArgsArray macroArgs;
// check for macro arguments
if (*text=='(') {
@@ -2772,7 +2773,7 @@ void TScriptContext::Tokenize(TSyncAppBase *aAppBaseP, cAppCharP aScriptName, sI
else token=TK_BITWISEOR; // |
break;
default:
- SYSYNC_THROW(TTokenizeException(aScriptName,"Syntax Error",aScriptText,text-aScriptText,line));
+ SYSYNC_THROW(TTokenizeException(aScriptName,"Syntax Error",textstart,text-textstart,line));
}
}
// add token if simple token found
commit e3fdd5ca811f24b2f80e598f9d00d2e134aa85e1
Author: Patrick Ohly <patrick.ohly(a)intel.com>
Date: Fri Jul 22 09:04:01 2011 +0200
scripting: added READ() method
The READ(filename) method returns the content of the file identified
with "filename". Relative paths are interpreted relative to the current
directory. On failures, an error messages is logged and UNASSIGNED
is returned.
This method is useful for inlining the photo data referenced with
local file:// URIs shortly before sending to a remote peer. SyncEvolution
uses the method in its outgoing vcard script as follows:
Field list:
<!-- Photo -->
<field name="PHOTO" type="blob" compare="never" merge="fillempty"/>
<field name="PHOTO_TYPE" type="string" compare="never" merge="fillempty"/>
<field name="PHOTO_VALUE" type="string" compare="never" merge="fillempty"/>
Profile:
<property name="PHOTO" filter="no">
<value field="PHOTO" conversion="BLOB_B64"/>
<parameter name="TYPE" default="no" show="yes">
<value field="PHOTO_TYPE"/>
</parameter>
<parameter name="VALUE" default="no" show="yes">
<value field="PHOTO_VALUE"/>
</parameter>
</property>
Script:
if (PHOTO_VALUE == "uri" &&
SUBSTR(PHOTO, 0, 7) == "file://") {
// inline the photo data
string data;
data = READ(SUBSTR(PHOTO, 7));
if (data != UNASSIGNED) {
PHOTO = data;
PHOTO_VALUE = "binary";
}
}
Test cases for inlining, not inlining because of non-file URI and
failed inling (file not found) were added to SyncEvolution.
diff --git a/src/sysync/scriptcontext.cpp b/src/sysync/scriptcontext.cpp
index f21641e..e6124c9 100755
--- a/src/sysync/scriptcontext.cpp
+++ b/src/sysync/scriptcontext.cpp
@@ -27,6 +27,7 @@
#include "pcre.h" // for RegEx functions
#endif
+#include <stdio.h>
// script debug messages
#ifdef SYDEBUG
@@ -869,6 +870,55 @@ public:
aTermP->setAsInteger(exitcode);
}; // func_Shellexecute
+ // string READ(string file)
+ // reads the file and returns its content or UNASSIGNED in case of failure;
+ // errors are logged
+ static void func_Read(TItemField *&aTermP, TScriptContext *aFuncContextP)
+ {
+ // get params
+ string file;
+ aFuncContextP->getLocalVar(0)->getAsString(file);
+
+ // execute now
+ string content;
+ FILE *in;
+ in = fopen(file.c_str(), "rb");
+ if (in) {
+ long size = fseek(in, 0, SEEK_END);
+ if (size >= 0) {
+ // managed to obtain size, use it to pre-allocate result
+ content.reserve(size);
+ fseek(in, 0, SEEK_SET);
+ } else {
+ // ignore seek error, might not be a plain file
+ clearerr(in);
+ }
+
+ if (!ferror(in)) {
+ char buf[8192];
+ size_t read;
+ while ((read = fread(buf, 1, sizeof(buf), in)) > 0) {
+ content.append(buf, read);
+ }
+ }
+ }
+
+ if (in && !ferror(in)) {
+ // return content as string
+ aTermP->setAsString(content);
+ } else {
+ PLOGDEBUGPRINTFX(aFuncContextP->getDbgLogger(),
+ DBG_ERROR,(
+ "IO error in READ(\"%s\"): %s ",
+ file.c_str(),
+ strerror(errno)));
+ }
+
+ if (in) {
+ fclose(in);
+ }
+ } // func_Read
+
// string REMOTERULENAME()
// returns name of the LAST matched remote rule (or subrule), empty if none
@@ -2220,6 +2270,7 @@ const TBuiltInFuncDef BuiltInFuncDefs[] = {
{ "REQUESTMAXTIME", TBuiltinStdFuncs::func_RequestMaxTime, fty_none, 1, param_oneInteger },
{ "REQUESTMINTIME", TBuiltinStdFuncs::func_RequestMinTime, fty_none, 1, param_oneInteger },
{ "SHELLEXECUTE", TBuiltinStdFuncs::func_Shellexecute, fty_integer, 3, param_Shellexecute },
+ { "READ", TBuiltinStdFuncs::func_Read, fty_string, 1, param_oneString },
{ "SESSIONVAR", TBuiltinStdFuncs::func_SessionVar, fty_none, 1, param_oneString },
{ "SETSESSIONVAR", TBuiltinStdFuncs::func_SetSessionVar, fty_none, 2, param_SetSessionVar },
{ "ABORTSESSION", TBuiltinStdFuncs::func_AbortSession, fty_none, 1, param_oneInteger },
-----------------------
9 years, 6 months
[SyncEvolution] event time not synchronising (out one hour)
by sean
When I enter a time in my appointment agenda on my computer and then synchronise it is changed to one hour later on my
phone (Nokia N95). The time on my phone and computer are the same time. Can anyone suggest why this may be happening and how to correct it?
Thanks
Sean
9 years, 6 months
Re: [SyncEvolution] vcalendar-sqlite
by Patrick Ohly
On Mi, 2011-07-20 at 16:56 +0100, Roger KeIrad wrote:
> all i need is a non-evolution backend.
> How can i activate the different level debug traces ?
Set the "loglevel" property. "4" is enough to get information about data
conversion and communication.
Please keep the mailing list involved by group-replying.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
9 years, 6 months
[SyncEvolution] vcalendar-sqlite
by Roger KeIrad
hello,
Even when changing the type to text/vcalendar in config file the
synchronization type observed is icalendar .
How can i synchronize using vcalendar1.0 ? is there any additional
configuration ??
i want to compile only the code of syncml client , i mean that i don't need
the server part of syncevolution. should i remove it ? is there another way
to do it ?
how can i activate sqlite part of syncevolution ?
THX.
9 years, 6 months
[SyncEvolution] automated testing of ActiveSync
by Patrick Ohly
Hello!
I just pushed some changes into activesyncd master which (finally) allow
running the SyncEvolution backend and sync testing. Sorry for the delay.
Instructions in the README, quoted below.
In contrast to what I said in the folder thread, the default database is
enough for testing because I added some code to the backend which
ensures that that database is used for testing, instead of a named
database that the test framework normally would use.
I am seeing some issues with sync key handling. In some cases, the
activesyncd debug output shows random characters. That looks very much
like an invalid memory access.
I had a brief look into eas_sync_handler_get_items(), but only found a
memory leak:
sync_key_in = g_strdup("0");
has no corresponding g_free().
Is suspect that part of the problem is that Client::eas_event::Source
needs to "simulate" access by two different clients. Both use the
ActiveSync account defined by
~/.config/syncevolution/client-test/peers/target-config/config.ini
so essentially this looks to the Exchange server like one client.
Is it absolutely necessary that the account ID is a valid email address?
If not, then I could create two accounts in gconf, one called
SyncEvolution_Test_1 and another SyncEvolution_Test_2, then switch
between them as needed. Both accounts would use exactly the same
username, password and server URL.
One test which should already work but fails (even with the time zone
fix, commit cb4046) is the Client::Source::eas_event::testImport test.
It fails with:
error 100 returned from wbxml_conv_xml2wbxml_run
See Client_Source_eas_event_testImport.log after running the test.
I have libwbxml2-0.11.beta6-7.1.i586 installed in MeeGo.
-----------------------------------------
Backend Testing
===============
Configure "local" testing (backend is covered, no syncing involved):
./syncevolution --configure username=<email adddress identifying the
account> \
--template SyncEvolution target-config@client-test
On MeeGo:
- install syncevolution-test and syncevolution-synccompare
- create an empty directory, enter it
- copy /usr/share/doc/syncevolution/testcases
- CLIENT_TEST_UNIQUE_UID=1 \
CLIENT_TEST_SERVER=exchange \
client-test Client::Source::eas_event
When compiling SyncEvolution:
- enter the src directory
- CLIENT_TEST_UNIQUE_UID=1 \
CLIENT_TEST_SERVER=exchange \
PATH=.:$PATH \
./client-test Client::Source::eas_event
CLIENT_TEST_UNIQUE_UID is necessary to avoid a UID conflict in a
simple-minded insertion test. The proper solution is to add an "item
exists" error to activesyncd and handle that in
ActiveSyncSource::insertItem().
The test data that is used for testImport is in:
testcases/eds_event.ics
This data was meant for Evolution. Because it is standard iCalendar 2.0,
it is a suitable starting point for Exchange. If it turns out that this
data doesn't work with Exchange, then a simplified/adapter version of
the file can be put into:
testcases/eds_event.ics.exchange.tem
Sync Testing
============
- Configure target-config@exchange with source 'calendar'.
- Create two sync configs:
syncevolution --configure username= password= \
syncURL=local://@exchange uri=calendar \
backend=evolution-calendar \
--template SyncEvolution_Client exchange_1@client-test-1
eds_event
syncevolution --configure username= password= \
syncURL=local://@exchange uri=calendar \
backend=evolution-calendar \
--template SyncEvolution_Client exchange_2@client-test-2
eds_event
- Create calendars named as follows in Evolution:
SyncEvolution_Test_eds_event_1
SyncEvolution_Test_eds_event_2
- run test (see above for invoking client-test and setting up
"testcases"):
CLIENT_TEST_UNIQUE_UID=1 CLIENT_TEST_SERVER=exchange \
client-test Client::Sync::eds_event::testCopy
---------------------------------------------------------
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
9 years, 6 months
[SyncEvolution] XML error after migration from 1.1.1 to 1.99.5
by Tino Keitel
Hi,
I upgraded from 1.1.1 to 1.99.5 on 2 computers running Debian sid and
now I get the following error:
invalid tag <overridedevinf> at line 1721 col 10
Fatal: Undefined function 'COMPARISONMODE' in script at line 3
Fatal error 20010, no valid configuration could be read from XML file
[ERROR] internal error, invalid XML configuration (without datastores):
Following is XML which looks like
src/syncevo/configs/remoterules/client/02google-contacts.xml.
Regards,
Tino
9 years, 6 months
[SyncEvolution] SyncEvolution D-Bus policies
by Patrick Ohly
Hello!
Last week Gabriel and I had a lot of discussions around how the MeeGo UX
Settings UI for SyncEvolution should (and should not) use the D-Bus API,
in particular in combination with configuring CalDAV + target configs.
Jussi, we identified some cases where we had to deviate from the Python
prototype that you created. Please check whether the GTK sync-UI works
as intended.
Gabriel, in one case I am uncertain whether we picked the right
solution. See "ignoring target config" below.
Use right context for templates
===============================
Templates contain the current values of shared properties. For example,
the @default context might have been configured to use
backend=KDE-Contacts for "addressbook" and
database=not-the-default-addressbook. Any template requested without
explicit context in the config name of GetConfig() will have these
values instead of the ones set in the underlying template
(backend=addressbook for SyncML peers, backend=CardDAV for WebDAV
peers).
When configuring a target-config, request the template as "<template
name>@<template name>", for example "google-calendar@google-calendar".
The command line also supports "target-config@google-calendar". The
D-Bus API probably should also recognize that as a request for the
"google-calendar" template, but right now doesn't.
The intention of this mechanism is that clients are allowed to update
just the fields they know and care about (like "sync") and then write
back the config. See item "Copy from templates..." below.
Use "SyncEvolution_Client" template for sync config in local sync
=================================================================
Instead of creating a config from scratch for the sync config in a local
sync, request the "SyncEvolution_Client" template. This ensures that,
for example, "backend" and "database" are taken from the potentially
existing and modified @default context.
Changes which need to be made:
* "SyncEvolution_Client" defines username/password=test. Unset
them when creating the config.
* Set "ConsumerReady" to the same value as in the template of the
service for which the sync config is created.
* Also copy the PeerName.
* Only configure the sources which are enabled in the template.
Open question: shall "peerType" be copied from target config into sync
config?
Pro: Setting peerType=WebDAV will hide the sync config in those
UIs which don't know how to configure them. UIs which need to
know the type don't have to retrieve the target config to find
it.
Con: not setting it will allow to run syncs in all UIs, because
that part is the same.
I'm undecided. Comments?
Copy from templates into configs inside the same session
========================================================
The content of templates is partly populated with existing values on
disk. When reading the template and then later writing it after
obtaining a session, a race condition exists:
1. client A reads template
2. client B modifies shared properties
3. client A writes back old values of these properties
Ignoring target config
======================
The "target-config@<context>" is (almost) never a config that the user
can interact with directly. In particular triggering a sync with it is
not possible.
Gabriel and I settled on setting "ConsumerReady=0" in it, but I now
believe that this is wrong. A target config as created via the command
line will have "ConsumerReady=1", as in the template, and we actually
may want to distinguish between ready and not ready also for target
configs.
So instead I suggest to add a hard-coded check for "does config name
start with target-config@" and ignore such configs.
Note sure whether this is the complete list. Gabriel, anything else?
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
9 years, 6 months
[SyncEvolution] Memotoo: HTML entity regression
by Patrick Ohly
Hello Thomas!
Some of SyncEvolution's test cases include characters that are special
in HTML, including ampersand and HTML entities. HTML entities are not
(or no longer?) handled correctly by Memotoo.
The entities are sent like this:
<Data><![CDATA[BEGIN:VCARD
VERSION:2.1
REV:20110711T140323Z
N:xml;entities;;;
FN:xml entities
X-EVOLUTION-FILE-AS:xml, entities
X-GENDER:
NICKNAME:user12
TITLE:
ORG:;;;
ROLE:
TEL:
EMAIL:
URL:
X-MOZILLA-HTML:
ADR:;;;;;;
BDAY:
NOTE:ampersand entity & less-than entity <
PHOTO:
END:VCARD
]]></Data>
[XML dump - the actual message was WBXML, where CDATA is not necessary]
http://runtests.syncevolution.org/2011-07-11-21-30_all/head-lucid-amd64/1...
This contact comes back as:
<Data><![CDATA[BEGIN:VCARD
VERSION:3.0
REV:20110711T140330Z
N:xml;entities;;;
FN:xml entities
NICKNAME:user12
NOTE:ampersand entity & less-than entity <
END:VCARD
]]></Data>
http://runtests.syncevolution.org/2011-07-11-21-30_all/head-lucid-amd64/1...
Note that the HTML entities were replaced with the corresponding
characters, which is not the same.
I'm going to disable this test case for Memotoo. But you might want to
look into this and fix it.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
9 years, 6 months
[SyncEvolution] gmane.org
by Patrick Ohly
Hello!
If this email gets through, then the gmane.org<->SyncEvolution mailing
list works again. For those who never heard of it, at
http://news.gmane.org/gmane.comp.mobile.syncevolution
there's a complete mailing list archive with the possibility to post
replies or new messages via the web interface, without having to
subscribe.
This happened to use the old syncevolution(a)moblin.org email address, which
stopped working recently. Now gmane uses the current address.
Bye, Patrick
9 years, 6 months
