[SyncEvolution] command line: item operations (BMC #23783)
by Patrick Ohly
Hello!
I've extended the command line and would like to get feedback on the
patch before merging into the command line. If you are a user of the
command line, does the following change make sense to you?
Author: Patrick Ohly <patrick.ohly(a)intel.com> 2011-11-21 17:38:34
Committer: Patrick Ohly <patrick.ohly(a)intel.com> 2011-11-21 17:38:34
Parent: 2d591e780dcfb8470d16e28aa87b52d0b754b8de (config: return value + "was set" for each config property)
Branch: pohly
Follows: syncevolution-1-2
Precedes:
command line: config and source names are optional (BMC #23783)
The need to add "foo" and "bar" pseudo config and source names to the
command line even when all parameters for the operation where
explicitly specified on the command line was confusing.
Now it is possible to invoke item operations without the config and
source name. Names which refer to non-existent configs are still
accepted, as in previous releases. Typos are handled better by
producing a detailed error report which includes (as applicable):
- config doesn't exist
- source doesn't exist or not selected
- backend property not set
This error report is created only if SyncSource was unable to create a
SyncSource, which is detected by catching the status exception with
the right error code. Trying to anticipate the error at the Cmdline
level would duplicate code found in the SyncSource and/or impose
limitations which might not hold in all cases.
Because luids used to be positional arguments after <config> and
<source>, a new --luids keyword is necessary to indicate that the
following parameters are luids and not <config> and <source>.
Added a CmdlineTest::testItemOperations for this new command line
syntax. It uses the file backend to run some real operations (missing
earlier).
Error reporting in the command line uses the same mix of writing error
messages and returning early (Cmdline) and throwing exceptions (rest
of SyncEvolution). Perhaps the Cmdline class should also use
exceptions - but not now.
---------------------------------- README.rst ----------------------------------
index 16bb95c..b6de21f 100644
@@ -42,20 +42,23 @@ Create, update or remove a configuration:
syncevolution --remove|--migrate <options> [--] <config>
List items:
- syncevolution --print-items [--] <config> <source>
+ syncevolution --print-items [--] [<config> [<source>]]
Export item(s):
- syncevolution [--delimiter <string>] --export <dir>|<file>|- [--] <config> <source> [<luid> ...]
+ syncevolution [--delimiter <string>] --export <dir>|<file>|- [--] [<config> [<source> [<luid> ...]]]
+ --luids <luid> ...
Add item(s):
- syncevolution [--delimiter <string>|none] --import <dir>|<file>|- [--] <config> <source>
+ syncevolution [--delimiter <string>|none] --import <dir>|<file>|- [--] [<config> [<source>]]
+ --luids <luid> ...
Update item(s):
syncevolution --update <dir> [--] <config> <source>
syncevolution [--delimiter <string>|none] --update <file>|- [--] <config> <source> <luid> ...
+ --luids <luid> ...
Remove item(s):
syncevolution --delete-items [--] <config> <source> (<luid> ... | '*')
@@ -419,10 +422,10 @@ the peer during the next synchronization. If the peer somehow
needs to get a clean copy of all local items, then use ``--sync
refresh-from-local`` in the next run. ::
syncevolution --print-items <config> <source>
- syncevolution [--delimiter <string>] --export <dir>|<file>|- <config> <source> [<luid> ...]
- syncevolution [--delimiter <string>|none] --import <dir>|<file>|- <config> <source>
+ syncevolution [--delimiter <string>] --export <dir>|<file>|- [<config> [<source> [<luid> ...]]]
+ syncevolution [--delimiter <string>|none] --import <dir>|<file>|- [<config> <source>]
syncevolution --update <dir> <config> <source>
syncevolution [--delimiter <string>|none] --update <file>|- <config> <source> <luid> ...
syncevolution --delete-items <config> <source> (<luid> ... | *)
@@ -430,17 +433,24 @@ Restore depends on the specific format of the automatic backups
created by SyncEvolution. Arbitrary access to item data is provided
with additional options. <luid> here is the unique local identifier
assigned to each item in the source, transformed so that it contains
only alphanumeric characters, dash and underscore. A star * in
---delete-items selects all items for deletion.
+--delete-items selects all items for deletion. There are two ways
+of specifying luids: either as additional parameters after the
+config and source parameters (which may be empty in this case, but
+must be given) or after the ``--luids`` keyword.
-<config> and <source> must be given, but they do not have to refer to
-existing configurations. In that case, the desired backend and must be
-give via the ``backend`` property, like this::
+<config> and <source> may be given to define the database which is to
+be used. If not given or not refering to an existing configuration
+(which is not an error, due to historic reasons), the desired backend
+must be given via the ``backend`` property, like this::
- syncevolution --print-items backend=evolution-contacts dummy-config dummy-source
+ syncevolution --print-items backend=evolution-contacts
+ syncevolution --export - backend=evolution-contacts \
+ --luids pas-id-4E33F24300000006 pas-id-4E36DD7B00000007
The desired backend database can be chosen via ``database=<identifier>``.
+See ``--print-databases``.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
9 years, 1 month
[SyncEvolution] regular expression library: boost Xpressive?
by Patrick Ohly
Hello!
SyncEvolution does not make use of regular expressions because it wants
to avoid a dependency on libraries which are not common enough to be
relied upon on all target platforms. That rules out Qt RegEx.
It also avoids libraries whose ABI varies too often, because it would
complicate delivering libraries via syncevolution.org. That has ruled
out anything from Boost which is more than a set of header files, like
boost::regex (libboost_regex.so.1.42.0 in Debian Stable, still in
Testing, but bound to be replaced by libboost_regex.so.1.46.0).
Another aspect is C++ support for std::string and the desire to do
search/replace. Otherwise POSIX regex support in GNU libc would be an
option.
I've just come across another alternative: boost::Xpressive.
http://www.boost.org/doc/libs/1_47_0/doc/html/xpressive/user_s_guide.html...
This is old enough (introduced in Boost 1.34.0) to not limit compilation
on older Linux distros and it is header-only.
It supports regular expression syntax, parsed at runtime. It also touts
its own C++ "domain language" for statically compiling regular
expressions as an alternative.
I am less certain about the later. While the approach looks neat, it is
also not going to be familiar for many developers, which makes the code
less approachable.
For example:
sregex re = '$' >> +_d >> '.' >> _d >> _d;
That's the same as:
sregex re = sregex::compile("$\\d\\.\d\d");
Any thoughts about using Boost Xpressive in SyncEvolution from other
developers? Any alternatives?
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
9 years, 1 month
[SyncEvolution] syncevo-dbus-server + forking processes
by Patrick Ohly
Hello!
Chris is going to work on running syncs inside a process forked from the
main syncevo-dbus-server. Before diving into the planning and
implementation, let me outline the background.
At the moment, syncevo-dbus-server is one process (when ignoring local
sync for a second) which handles D-Bus calls and syncs at the same time.
A sync involves operations which can block for a long time, so keeping
the D-Bus part alive and responsive is challenging. It involves hacks
like entering and leaving the main event loop, checking it inside
libsoup, etc.
Local sync already uses a second forked process so that it can run
client and server engines concurrently.
Getting rid of the current hacks is one motivation for running syncs in
a forked process. The main process would enter the main loop once and
never return unless it was asked to shut down.
The other is that (eventually) it'll allow to run a "clean" main process
which never loads any of the backend modules. That may help to keep the
permanent memory footprint smaller than it is today. Right now this is
not possible because merely registering a backend requires loading it.
Fixing that is outside of the current scope.
Resource leaks inside a sync become less problematic when the process
running the sync can shut down.
Here are some requirements for "sync in forked process":
* should be disabled when running outside of the D-Bus daemon
(syncevolution command line in non-daemon mode, client-test
testing)
* interactive password requests through D-Bus must continue to
work
* abort/suspend requests via D-Bus must work
* the forked process should shut down cleanly, so that "valgrind
--follow-child=yes --leak-check=yes" provides sane results (the
forked process in a local sync currently fails to do that,
because it is forked in a less controlled environment and
therefore just calls _exit())
* logging:
* the main syncevo-dbus-server process should use syslog
logging for its own messages, without redirecting
stdout/stderr
* the forked process should start redirecting
stdout/stderr into the normal logging mechanism (so that
library error messages are visible when running the
syncevolution command line as client of the D-Bus server
and, later, they appear in log files) and switch to a
session log when the session is ready
Somewhat related to this work is support for push sync. I suspect that
this will also use forked processes to monitor remote and local
databases without interfering with the main process. It would be good if
the mechanism for forking and communication between process was generic
and not tied to syncing alone.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
9 years, 1 month
[SyncEvolution] testing syncevo-dbus-server under valgrind
by Patrick Ohly
Hello!
I've changed the automatic testing so that syncevo-dbus-server is
started under valgrindcheck.sh by test-dbus.py. This revealed some minor
memory leaks and broken memory accesses (typically read after free). I
I've fixed all of that, see master branch.
The last remaining issue seems to be a leak in the error path of GIO
GDBus: when a service is not available,
g_dbus_connection_new_for_address_sync() leaks some memory.
http://syncev.meego.com/2011-12-01-10-06_syncevolution_dbus_gio-gdbus/gio...
==10608== 60 (16 direct, 44 indirect) bytes in 1 blocks are definitely lost in loss record 911 of 1,653
==10608== at 0x4C27673: malloc (vg_replace_malloc.c:263)
==10608== by 0x7FABC02: g_malloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3000.2)
==10608== by 0x7FC0976: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3000.2)
==10608== by 0x7F9045F: g_error_copy (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3000.2)
==10608== by 0x76700B1: ??? (in /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.3000.2)
==10608== by 0x761465E: g_initable_new_valist (in /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.3000.2)
==10608== by 0x7614748: g_initable_new (in /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.3000.2)
==10608== by 0x766EB1E: g_dbus_connection_new_for_address_sync (in /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.3000.2)
==10608== by 0x597E08: GDBusCXX::dbus_get_bus_connection(char const*, char const*, bool, GDBusCXX::DBusErrorCXX*) (gdbus-cxx-bridge.cpp:58)
==10608== by 0x583CB8: SyncEvo::ConnmanClient::ConnmanClient(SyncEvo::Server&) (connman-client.cpp:30)
==10608== by 0x541730: SyncEvo::Server::Server(_GMainLoop*, bool&, boost::shared_ptr<SyncEvo::Restart>&, GDBusCXX::DBusConnectionPtr const&, int) (server.cpp:239)
==10608== by 0x52141B: main (main.cpp:113)
==10608== 60 (16 direct, 44 indirect) bytes in 1 blocks are definitely lost in loss record 912 of 1,653
==10608== at 0x4C27673: malloc (vg_replace_malloc.c:263)
==10608== by 0x7FABC02: g_malloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3000.2)
==10608== by 0x7FC0976: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3000.2)
==10608== by 0x7F9045F: g_error_copy (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3000.2)
==10608== by 0x76700B1: ??? (in /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.3000.2)
==10608== by 0x761465E: g_initable_new_valist (in /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.3000.2)
==10608== by 0x7614748: g_initable_new (in /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.3000.2)
==10608== by 0x766EB1E: g_dbus_connection_new_for_address_sync (in /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.3000.2)
==10608== by 0x597E08: GDBusCXX::dbus_get_bus_connection(char const*, char const*, bool, GDBusCXX::DBusErrorCXX*) (gdbus-cxx-bridge.cpp:58)
==10608== by 0x58A9EE: SyncEvo::NetworkManagerClient::NetworkManagerClient(SyncEvo::Server&) (network-manager-client.cpp:31)
==10608== by 0x541747: SyncEvo::Server::Server(_GMainLoop*, bool&, boost::shared_ptr<SyncEvo::Restart>&, GDBusCXX::DBusConnectionPtr const&, int) (server.cpp:239)
==10608== by 0x52141B: main (main.cpp:113)
Chris, do you agree that this isn't something caused by our code?
If yes, then I'll suppress this particular problem.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
9 years, 1 month
Re: [SyncEvolution] SE_2011-12-02-16-36_dist: lucid-amd64
by Patrick Ohly
Good news!
Automatic integration testing is working and catches problems that are
hard to detect without the automated tests. From a recent run:
On Fri, 2011-12-02 at 09:18 -0800, syncevo-do-not-reply(a)meego.com wrote:
> syncevolution
[...]
> origin/for-master/make-notification-file-names-conformant: okay
> Patches:
[...]
> =?UTF-8?q?Chris=20K=C3=BChl?= - dbus-server: Make dbus-server
> notification file names conformant
Note to self: check UTF-8 handling in the scripts.
> prepare
> Value
> libsynthesis-fetch-config
> okay
> syncevolution-fetch-config
> okay
> compile
> okay
> dist
> failed
> Total passed cases (all: 4)
> 3
"make dist" failed because the renamed files were still referenced in
the "po" directory. Fixed.
Tip: use "git grep" to find all occurrences of a string in a project.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
9 years, 1 month
[SyncEvolution] problem with egroupware
by benoit
Hello,
I want to synchronize my evolution with my egroupware server.
My client is Ubuntu 11.10
My server is debian 6 with egroupware 1.8.002
The error message indicate a wrong login, but I'm sure that my login is correct and M session log
indicates a success login.
Thanks for ideas
The message error:
Start of log - Synthesis SyncML Engine 3.4.0.27
[-- collapse all --][++ expand all ++]
+
[2011-12-04 18:10:38.612] 'loadSystemZoneDefinitions' - Linux system time zones
[--][++] [->end]
[2011-12-04 18:10:39.177] SyncML server account: amandine
[2011-12-04 18:10:39.177] client: SyncEvolution 1.2.1 for workstation
[2011-12-04 18:10:39.178] device ID: syncevolution-5694a4ec-d7e7-47b2-92ce-36f5958be8d9
[2011-12-04 18:10:39.178] using libedataserver-1.2.so.15 - might not be compatible!
using libebook-1.2.so.12 - might not be compatible!
using libecal-1.2.so.10 - might not be compatible!
using libecal-1.2.so.10 - might not be compatible!
using libbluetooth.so.3
sdp_extract_pdu_safe not found
sdp_extract_seqtype_safe not found
[2011-12-04 18:10:39.178] Scanning backend libraries in /usr/lib/syncevolution/backends/
Loading backend library synckcalextended.so
Loading backend library syncakonadi.so
Loading backend library syncdav.so
Loading backend library syncsqlite.so
Loading backend library syncmaemocal.so
Loading backend library syncxmlrpc.so
Loading backend library syncaddressbook.so
Loading backend library syncebook.so
Loading backend library syncqtcontacts.so
Loading backend library syncfile.so
Loading backend library syncecal.so
[2011-12-04 18:10:39.181] memo: inactive
[2011-12-04 18:10:39.193] todo: inactive
[2011-12-04 18:10:39.541] CreateContext SyncEvolution//addressbook => 0
[2011-12-04 18:10:39.541] Module_Version = 01080000
[2011-12-04 18:10:39.542] Module_Capabilities:
PLATFORM:Linux
DLL:true
MINVERSION:V1.0.6.0
MANUFACTURER:SyncEvolution
DESCRIPTION:SyncEvolution Synthesis DB Plugin
plugin_datastore_str:no
plugin_datastore_key:yes
ITEM_AS_KEY:yes
plugin_datablob:no
[2011-12-04 18:10:39.543] Module_PluginParams
Engine=01080000
[2011-12-04 18:10:39.544] Module_Capabilities:
PLATFORM:Linux
DLL:true
MINVERSION:V1.0.6.0
MANUFACTURER:SyncEvolution
DESCRIPTION:SyncEvolution Synthesis DB Plugin
plugin_datastore_str:no
plugin_datastore_key:yes
ITEM_AS_KEY:yes
plugin_datablob:no
[2011-12-04 18:10:39.544] CreateContext SyncEvolution//calendar => 0
[2011-12-04 18:10:39.545] Module_Version = 01080000
[2011-12-04 18:10:39.545] Module_Capabilities:
PLATFORM:Linux
DLL:true
MINVERSION:V1.0.6.0
MANUFACTURER:SyncEvolution
DESCRIPTION:SyncEvolution Synthesis DB Plugin
plugin_datastore_str:no
plugin_datastore_key:yes
ITEM_AS_KEY:yes
plugin_datablob:no
[2011-12-04 18:10:39.547] Module_PluginParams
Engine=01080000
[2011-12-04 18:10:39.547] Module_Capabilities:
PLATFORM:Linux
DLL:true
MINVERSION:V1.0.6.0
MANUFACTURER:SyncEvolution
DESCRIPTION:SyncEvolution Synthesis DB Plugin
plugin_datastore_str:no
plugin_datastore_key:yes
ITEM_AS_KEY:yes
plugin_datablob:no
[2011-12-04 18:10:40.402] RespURI set to = 'https://192.168.5.173/egroupware/rpc.php?syncml_sessionid=syncml-af1b15d9
1abd68fe066cf52e62a1c192';
–
[2011-12-04 18:10:40.402] 'issue' - issuing command, Cmd=Status [--][++] [->end]
[->enclosing]
–[2011-12-04 18:10:40.402] End of 'issue' [->top] [->enclosing]
[2011-12-04 18:10:40.402] Incoming SyncHdr processed, incomingMsgID=1, SyncMLVers=1.2
[2011-12-04 18:10:40.403] - Session ID='65'
[2011-12-04 18:10:40.403] - Source (Remote party): URI='https://192.168.5.173/egroupware/rpc.php';
DisplayName=''
[2011-12-04 18:10:40.403] - Response to be sent to URI='https://192.168.5.173/egroupware/rpc.php?syncml_sessionid=syncml-af1b15d9
1abd68fe066cf52e62a1c192';
[2011-12-04 18:10:40.403] - Target (Local party) :
URI='syncevolution-5694a4ec-d7e7-47b2-92ce-36f5958be8d9' DisplayName='Linux
PC'
[2011-12-04 18:10:40.403] Deleted command 'SyncHdr' (incoming MsgID=1, CmdID=0)
–[2011-12-04 18:10:40.403] End of 'SyncHdr' [->top] [->enclosing]
–[2011-12-04 18:10:40.403] End of 'processHdr' [->top] [->enclosing]
[2011-12-04 18:10:40.403] Created command 'Status' (incoming)
–
[2011-12-04 18:10:40.403] 'processStatus' - Processing incoming Status [--][++]
[->end] [->enclosing]
[2011-12-04 18:10:40.403] Started processing Command 'Status' (incoming MsgID=1,
CmdID=1)
[2011-12-04 18:10:40.403] WARNING: RECEIVED NON-OK STATUS 401 for command 'SyncHdr'
(outgoing MsgID=1, CmdID=0)
[2011-12-04 18:10:40.403] - TargetRef (remoteID) = 'https://192.168.5.173/egroupware/rpc.php';
[2011-12-04 18:10:40.404] - SourceRef (localID) =
'syncevolution-5694a4ec-d7e7-47b2-92ce-36f5958be8d9'
[2011-12-04 18:10:40.404] Found matching command 'SyncHdr' for Status
[2011-12-04 18:10:40.404] Remote requests auth type='syncml:auth-basic'
[2011-12-04 18:10:40.404] Next Cred will have type='basic' and
format='base64' and use nonce=''
[2011-12-04 18:10:40.404] Authentication failed (status=401) with remote server
–[2011-12-04 18:10:40.404] End of 'processStatus' [->top] [->enclosing]
[2011-12-04 18:10:40.404] =================> Retrying Client Session Start
–[2011-12-04 18:10:40.404] End of 'SyncML_Incoming' [->top]
[2011-12-04 18:10:40.404] Deleted command 'Status' (outgoing MsgID=0, CmdID=0)
–
[2011-12-04 18:10:40.404] 'SyncML_Outgoing' - start of new outgoing message [--][++]
[->end]
9 years, 1 month