Sunday, February 12, 2012

Conquirere joins Nepomuks future

Ok the title might be rather cryptic, but today I like to blog about some of the changing details I have been working on since the last time.

As some of you might now, Nepomuk is going to change. Sebastian already blogged about the new Data Management Service and why it is a good thing. Now Conquirere is ready for this bright future as it relies nearly completely on the new API.

What does this change now? Not much for the end-user to be honest. But it is a great change in the background that allows for some functions that were very hard to do beforehand.
  • Automatic merging of duplicate entries
  • Merging of user selected entries
  • Automatic type checking of the inserted entries
  • Identify what changes are done with Conquirere
Especially the last part is great. Currently, if you tested the program it produced a lot of entries in Nepomuk and often not all of them are removed when you remove some publications (the chapters or the websites connected to it for example). This leaves a lot of junk in the database you don't need anymore.

From now on, you can test the program without any fear. You can always simply clean the Nepomuk database from all entries created or changed by Nepomuk without altering the other parts of the Nepomuk storage. One of the greatest things that come with the new dms api.

So the dms change took some time, but wasn't the only thing I've changed. I did fix a few bugs and produced a few more while doing the transition. Also I've changed a few other parts I didn't like.

UI Changes
The most noticeable change is the refactoring of the ui. I have replace the QDockwidgets by a QSplitter layout. As it seems there is no need to freely rearrange the components of the ui this seems to be a better solution. Now the ui has a fixed layout, but you can simply hide parts of it by collapsing it to the side.
To automate this process a little bit further, I added 3 buttons to change the "mode" of the ui view.

Now there is a "full view mode" that displays all parts of the ui at once. When you like to scroll trough your library there is the "project view mode" which simply hides/collapse the document preview panel and when you want to work with your documents, you can switch to the "document view mode" which hides/collapse the library/resource table and let you concentrate on the document. If you need even more space, you can hide/collapse the right side panel too and you would end up with okular (as this is used as kpart to show your files).

Document view
Project/Library view
Full view
I have also simplified the way you can add your sources (files, remote files, websites).
Additionally you can now add "cited sources" so you can "quickly" check what other papers might be interesting regarding the one you are looking at. Currently you have to add this information manually. In the future I hope such information can be retrieved automatically via pdf parsing or from the web.

Also there is now no extra fields for the note content anymore, instead any kind of note is created as a pimo:Note which is a sub-resource of the publication (or document / email / event / reference / series / webpage). So from now on, bibtex files that have keys like note-1, note 2 etc are handled correctly too.

The "multiply selection" widget is another big change.
Beforehand it was a pain to to any kind of action on several files, especially deleting them. Now you can simply select several resources and do some actions with them, including merging them into 1 resource (thanks to the new dms api this was really easy to implement).

Nepomuk auto-completion is now done based on a live Nepomuk search (same way krunner does it). So at any time there is data from the database available to help you to complete your text.

Multi-selection widget 
Nepomuk auto-completion

Manage cited sources




The imported keywords for the publications are not handled as simple tags anymore, but will be imported as pimo:Topic. This will reduce the clutter in other parts of the system. You can still tag your publication though. But now a publication will have topics such as "face recognition, fantastic math solution, solution to world peace" and you can add tags like "important, reviewed, needs attention"

Zotero changes
The other bigger part I have changed is the Zotero integration. Now I can handle Zotero groups with more than 50 items (forgot to add this beforehand). Also child items (notes) are downloaded and added to the publication, as well as files, that can be downloaded into a specified directory. Sadly uploading files isn't working at the moment, as I have  no idea how to do that via the Zotero API.

New is also the merge dialog. As soon as the item you want to sync with changed on the server, you get a neat little dialog (unless you selected to use always the server/local version) that allows you to specify what changes you really want in your database.

Zotero merge dialog
That's it for now.
There is still  a lot to do and I still do not recommend to use Conquirere in a productive environment, but now I feel comfortable to recommend proper testing. Conquirere won't mess with your system in a way you can't simply wipe all the changes it did anymore. Also as long as you don't upload anything to Zotero, it can't mess with your Zotero data too, but even the upload should work (apart from the bugs that might still be there).

For the next step I like to add the right magic. Sebastian already showed how great automatic meta data fetching will be for the system and I think this should be expanded. Not only tv-shows, but also any other media file and of course text document should have a service that fetches all kind of meta data in the background. This will lead to a magic system that knows more about your files than you do and allows to bring Nepomuk to its full potential. 

Lets hope we can make this dream come true, as soon as possible.

20 comments:

  1. Hi, that looks very promising, thank you a lot. As the pdf is a kpart it will have the same functionality as okular (review bar for instance), is that right? And i wonder if it is possible to do some online research within conquiere like in the Web of Knowledge or other libraries. Are you planning to add support for endnoteweb? Thanks for your effort. Cheers

    ReplyDelete
    Replies
    1. Yes the review bar and all other features from okular are also available.

      In the long run I hope to extract the notes and other review markers also to nepomuk.

      Searching the web for new publications is also implemented (I reuse KBibTeX nice search features for this), you can also directly search for the pdf files when you have the publication meta data.

      Endnoteweb goes on my todo-list, but this will have to wait a while.

      Delete
    2. Really nice work! The notes will be especially useful to me, since I extract highlights from PDF files using Zotfile (http://www.columbia.edu/~jpl2136/zotfile.html). Maybe Conquirere could do the same - look if there are any associated Okular annotations (saved as XML files I think), and also extract highlights with Poppler (you could take a look at Zotfile).

      Delete
  2. for the love of god CHANGE THE NAME!!

    ReplyDelete
    Replies
    1. Try: "ereyna"
      It is an anglicized version of "έρευνα" the Greek word for research.

      Delete
    2. I like it. :)
      (I guess it could also be "erevna", right? The former sounds better to me though.)

      Delete
    3. How about Librarian or KDE Librarian?

      Delete
    4. mhh "ereyna" is agood start but not really better than what I have before.

      don't get me all wrong I know the name isn't the best one as I wrote it wrong the first few month before I get used to it ;)

      I will gladly change it as soon as someone really found something better I do like too.

      @gsarkis: I don't really like the "KDE" addition or and Kname stuff.

      Delete
    5. What kind of name are you looking for? One that sounds "exotic" (e.g. Zotero), explains what it does, or a combination of both?

      I was going to suggest something "modern", Biblify, but it seems to be taken already.
      ... Liblify? ;)

      You could create a topic at http://forum.kde.org and ask for suggestions - I guess Discussions & Opinions would be a good place.

      Delete
  3. Looking great, thanks a lot for your work!

    ReplyDelete
  4. "Lets hope we can make this dream come true, as soon as possible." Horay!

    I don't want to take a minute off your contribution. But I want to test it and I have make problems. One warning in the begining
    Invalid escape sequence \1 in conquirere/build/version.cmake:6
    Lots of warnings like
    conquirere/src/nbibio/pipe/bibtextonepomukpipe.cpp:2445:56: warning: ‘QString Nepomuk::Resource::uri() const’ is deprecated
    and finally
    make[2]: *** [src/nbibio/CMakeFiles/nbibio.dir/pipe/bibtextonepomukpipe.o] Error 1
    make[1]: *** [src/nbibio/CMakeFiles/nbibio.dir/all] Error 2
    make: *** [all] Error 2
    If it's not easy to help me here, no problem. I'll try again with a later version.

    Once again, thank you for this project!

    ReplyDelete
    Replies
    1. could it be that you didn't check out the latest sources from git?
      I have fixed all the deprecated uri() warninngs already (all the other property/class/entity warnings will go away when compiled against nepomuk.core master or as soon as frameworks 5 is out)

      It shouldn't abort too (and there is some output missing to know what went wrong)
      In case you have the latest sources and still get the error and you did build conquirere beforehand, try deleting the build directory and run cmake/make again

      Delete
    2. I... have the latest sources, but not too much experience compiling source. I still get the uri() warnings with the latest source. I'll try again with a later version. I have kubuntu kde 4.7.4.

      Delete
    3. All the remaining deprecated warnings are "normal" and go away once you would compile against nepomuk.core rather than nepomuk from kdelibs. Still this is not the reason your compiling failed.

      I assume you have somewhere an error complaining htmlContent is not part of Nepomuk::Vocabulary::NIE

      which simply means you need to update your shared-desktop-ontology package.
      Sadly installing the package alone from the website isn't enough and I have no clue how to generate the correct classes from these ontologies.

      So unless your distro offers a full package of the shared-desktop-ontology that is 0.8.1 or higher, you might be unlucky and have to wait.

      In case you are using opensuse there is an obs repo for it at:
      https://build.opensuse.org/package/show?package=conquirere&project=KDE%3AUnstable%3APlayground

      Delete
    4. Although I don't see any htmlContent in the output of make (http://pastebin.com/vPw367f6), kubuntu oneiric indeed has shared-desktop-ontologies 0.7.0. There is an experimental repository to kde 4.8 and ontologies 0.8.1, but I don't want to risk my desktop. I will have to wait :(

      Delete
  5. I wasn't able to compile from the sources in the repository. Which version of kbibtex are you using? I tried kbibtex-0.4 and it looks you are using some methods not available there. I.e
    FileImporterBibTeX *importer = new FileImporterBibTeX;
    importer->setKeysForPersonDetection(keysForPersonDetection);

    Thanks

    ReplyDelete
    Replies
    1. You need current trunk of KBibTeX to compile it.

      Delete
    2. I compiled the sources using qt-copy and kde from sources (4.8) and I was able to run the software. Nice job!

      I'm having a problem when trying to add a Publication into the Library. I.e., when a press the button to create a new Article nothing happens and I get this error:

      conquirere(2165) PublicationWidget::newButtonClicked: could not new default series " has a rdfs:domain of . <_:d> only has the following types <_:cx>, "

      Any ideas? BTW, the process is working fine for other items such as notes

      Thanks

      Delete
    3. Did you copy the ontology parts into the /usr/share/ontology folder?

      The exact progress is explained in the README at the end.

      Delete