Feature Request: Database Merge

s4zando
KPX user
Posts: 4
Joined: 29 Jul 2008, 04:39

Feature Request: Database Merge

Postby s4zando » 13 Jan 2009, 16:14

In addition to Import and Export, it would be nice to have a Merge function so that two db files can be merged with the most recently changed and newest additions.

In my case I use a USB thumbdrive and store my 'primary' db file on that, but also keep copies of the db on my home PC and laptop. Peridoically I find myself making changes not to the primary, but to one of the backup files (b/c the thumbdrive isn't handy). It would be nice to be able to merge/synch the db files. I think a simple merge based on the Last Change field would probably be sufficient in most circumstances. Overwriting the older records with the more recent and adding new records would probably keep most people happy. Handling record Deletions would be nice, but not a 'must have'.

Thanks.

(this Feature Req added to Project page)

Boris_Brodski
KPX user
Posts: 23
Joined: 03 Mar 2009, 18:17

Re: Feature Request: Database Merge

Postby Boris_Brodski » 03 Mar 2009, 18:25

Hi!

I already implemented synchronization of two databases of my self.
I will post patch soon. I you want to give it a try, I could post current version here. (diff-file)

Boris

Boris_Brodski
KPX user
Posts: 23
Joined: 03 Mar 2009, 18:17

Re: Feature Request: Database Merge

Postby Boris_Brodski » 06 Mar 2009, 14:52

I posted a patch with some screenshots: https://sourceforge.net/tracker2/?func= ... tid=839781

magnoliasouth
KPX user
Posts: 2
Joined: 03 Nov 2009, 22:16

Re: Feature Request: Database Merge

Postby magnoliasouth » 03 Nov 2009, 22:22

I am seriously needing to do this because Vista has really screwed up my file and made "conflicted copies" it says. Problem is that I don't know a thing about diff files. How exactly do I install this patch and do I install both or just one or two?

I tried Googling it but nothing explained it well. Please explain it like I'm two. ;)

althea06
KPX user
Posts: 2
Joined: 11 Nov 2009, 05:51

Re: Feature Request: Database Merge

Postby althea06 » 11 Nov 2009, 07:08

I am searching for it, but I was really confused on the explanation that was being posted on the net.Can anyone would help me to explain this well?I also need this information to solve my problem.

magnoliasouth
KPX user
Posts: 2
Joined: 03 Nov 2009, 22:16

Re: Feature Request: Database Merge

Postby magnoliasouth » 15 Nov 2009, 16:53

Sadly, I don't think Boris is planning on responding. I was (and am) disappointed though that no one else would explain it either.

alanfranz
KPX user
Posts: 1
Joined: 09 Mar 2010, 00:33

Re: Feature Request: Database Merge

Postby alanfranz » 09 Mar 2010, 00:45

Just to let everybody know, I created a small script that permits merging databases once exported in xml format. It might not be as powerful as windows keepass synchronize function, but it does its work. It requires python >= 2.5 and should work on any platform.

I don't seem to be able to add an attachment, so I'm just pasting the code here:

Code: Select all

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# vim: set fileencoding=utf8

# by Alan Franzoni.
# this code is released in the public domain.

import sys
from time import strptime
from xml.etree.ElementTree import parse, ElementTree

class KPMerger(object):
    """
    Merge keepassx xml-format dbs.

    Some details:
    entry title + username is considered the "primary key". entries from the
    two dbs are considered the same if such couplet is the same.
    keepassx supports a sort of entry history, which saves "old" versions of an
    entry. in order to perform the merge, the latest entry, i.e. the one with
    the more recent lastmod, is compared.
    the lastmod value is central to the comparison; the most recent entry
    overwrites the older one.

    Possible todos/improvements:
    - check for conflicts: if an entry appears in a db but it's not in the
      histroy of another, it might be a conflict that should be manually
      solved.
    - check for deletion: if an entry is available in the Backup group but it's
      not in any other main group, it has was probably deleted and should not
      be merged back.

    Warning:
     use something like a ramdisk and/or shred/safely delete xml files after
     use, because they're NOT encrypted.
    """

    SEP = "$$$" # mustn't appear in title nor usernames
    KP_TIMEFORMAT = "%Y-%m-%dT%H:%M:%S"

    def get_entry_lastmod_time(self, entry):
        return strptime(entry.find("lastmod").text, self.KP_TIMEFORMAT)

    def get_entry_key(self, entry):
        """
        the key for an entry is title + username combo. $$$ is used as a sort of
        escape.
        """
        return (entry.find("title").text or "") + self.SEP + (
                entry.find("username").text or "")

    def sorted_dict_items(self, d):
        # sort by key lexicographical order.
        return sorted(d.iteritems(), key=lambda x:x[0])

    def get_group_entries(self, elem):
        """
        for comparison's sake, take the most recent entry.
        """
        d = {}
        for entry in elem.findall("entry"):
            key = self.get_entry_key(entry)
            old_entry = d.get(key, None)
            if old_entry:
                # check the lastmod attribute
                if self.get_entry_lastmod_time(entry) > self.get_entry_lastmod_time(
                        old_entry):
                        # overwrite with the most recent.           
                        d[key] = entry
            else:
                d[key] = entry

        return d

    def get_subgroup_names(self, elem):
        # don't modify backups
        return dict([(group.find("title").text, group) for group in
            elem.findall("group") if group.find("title").text != "Backup"])


    def add_missing_groups(self, source, dest):
        """
        if there's a group in source but not in dest, add it to dest.
        renames will cause duplication.
        """
        dest_groups = self.get_subgroup_names(dest)
        for source_group_name, source_group_obj in self.get_subgroup_names(
                source).iteritems():
            if source_group_name in dest_groups:
                # check and add potentially missing subgroups.
                self.add_missing_groups(source_group_obj,
                        dest_groups[source_group_name])
            else:
                dest.append(source_group_obj)

    def merge_entry(self, first, second, d_group, key):
        if self.get_entry_lastmod_time(first) > self.get_entry_lastmod_time(second):
            print "Entry %s/%s was changed." % tuple(key.split(self.SEP))
            second_entry_index = list(d_group).index(second)
            d_group.insert(second_entry_index+1, first)

    def merge_entries(self, source, dest):
        """
        # rules:
        # if an entry is in source but not in dest, it is added.
        # if an entry is both in source and dest:
        # if the content is the same, do nothing and be happy.
        # if the content differs, fully take the one with the most recent lastmod.
        """
        s_groups = self.get_subgroup_names(source)
        for group_name, s_group in s_groups.items():
            s_entries = self.get_group_entries(s_group)

            d_groups = self.get_subgroup_names(dest)
            d_group = d_groups[group_name]
            d_entries = self.get_group_entries(d_group)

            for s_entry_key, s_entry in s_entries.items():
                if not (s_entry_key in d_entries):
                    print "missing %s/%s in dest, appending" % tuple(
                            s_entry_key.split(self.SEP))
                    d_group.append(s_entry)
                else:
                    d_entry = d_entries[s_entry_key]
                    self.merge_entry(s_entry, d_entry, d_group, s_entry_key)

            self.merge_entries(s_group, d_group)

    def merge_databases(self, first_xml_db_filename, second_xml_db_filename,
            destination_filename):
        first_xml_db = parse(first_xml_db_filename).getroot()
        second_xml_db = parse(second_xml_db_filename).getroot()

        self.add_missing_groups(first_xml_db, second_xml_db)
        self.merge_entries(first_xml_db, second_xml_db)
        ElementTree(second_xml_db).write(destination_filename)

if __name__ == "__main__":
    if len(sys.argv) != 4:
        print ("Usage:\n%s first_xml_db second_xml_db destination_xml_db\n" %
                sys.argv[0])
        sys.exit(1)

    m = KPMerger()
    m.merge_databases(*sys.argv[1:])
   

pocoO
KPX user
Posts: 1
Joined: 25 Mar 2010, 17:05

Re: Feature Request: Database Merge

Postby pocoO » 25 Mar 2010, 17:08

Ok thank you,
it works perfect.

Tommy_B
KPX user
Posts: 9
Joined: 09 Oct 2009, 17:20

Re: Feature Request: Database Merge

Postby Tommy_B » 29 Mar 2010, 10:14

this is a great script, thanks for sharing!

if you're daunted by python or can't seem to make it work, there is a program called DiffMerge available for free for Linux, OS X, and Windows at http://sourcegear.com/diffmerge/ that I had been using when I had conflicts in my database and it worked pretty well. You export as XML and load both files into the program and you can see the files side by side - differences between the two are highlighted for easy navigating. You then reimport the XML file back into KeePassX.

I always went through and manually figured out what I wanted to merge into the final file; I am not sure but this app might have a more automated way of doing that. But while the basic functionality is easy to figure out for manual use, I found the rest of the app a little too technical for me so never really looked at other functionality.


Return to “Open Discussion”

Who is online

Users browsing this forum: No registered users and 1 guest