Monday, January 23, 2012

Hello Planet & Conquirere 0.1

Hello Planet! Most of you don't know me yet, so let me introduce myself: My name is Jörg Ehrichs and I'm the developer of the KDE Wacom tablet module as well as the upcoming Conquirere.

In my first of hopefully many blog posts I like to introduce you to my approach to combine bibliographic data with your files and thoughts via the Nepomuk framework.

Conquirere allows you to add bibliographic data such as journals, books, proceedings papers, articles and many more to Nepomuk. Combine this data with the documents on your harddisk or some online storage and helps you to organize your data to quickly find some parts of your research again.

Why use Nepomuk when the same could be archived via KBibTeX and the good old .bib files? Because we can! Even though KBibTeX does gain file support in its latest versions it will still only operate on the flat database from the .bib files. The same is true for any other bibliographic manager. With the help of Nepomuk we can give our data a semantic meaning.

Connect the publications with the contacts from Akonadi, attach notes to them or mark the article as part of an event from your Akonadi calendar. Group your research papers together without duplicating the data or copying files around on your harddisk and search through your papers and data via Nepomuk to find exactly what you are looking for.

Even if many people seem to dislike Nepomuk, their real problems are within the feeders (which are getting better and better now). The Nepomuk database itself was always working great and very fast.

Nepomuks only problem is the lack of programs making use of the great features offered by this system or better yet, most people don't know that Nepomuk is used in the background by several programs (Searching, tagging adding notes in KDEPim, dolphin or the krunner).

Now there is another program based completely on the idea of this semantic database.


Enough introduction for the moment, now lets see whats working already.
In its current state Conquirere will list all your documents, (pdf, odt etc) as indexed by Nepomuk. Allow you to tag and comment them and tell what kind of publication this file represents. Like you would add data to your usual bibtex file you can add authors, editors, title, publication date and so on.


Because this would be a lot of work, you can also import existing bibliographic data. Thanks to KBibTeX approach to make its functions available for others you can also import any kind of bibliographic data and create the Nepomuk resources from it.

Furthermore, if you want to look for new information search through the several online services and import their data directly. Or use Zotero to add the data while you are surfing for some research papers and import/sync them later on.


Got lost in all your data and can't find the right file with some content you need? Nepomuk offers a great way for full text search and you never have to spend hours opening and reading your pdf files again.

But don't stop with files, connect emails, events or notes together so you have always all the data you need.

Finally all the research is done and you want to start to write and cite. Export your references again. Thanks again to KBibTeX this is as easy as 2 clicks and you can create a .bib file with the references you need. Or pipe the data directly to LyX.

As you can see Conquirere already offers a lot and hopefully one day it will offer a lot more.
If you like to have a look at the current progress, feel free to check it out from playground. Just follow the instructions from the README and you're ready to go.

But please keep in mind: It is in an early state. So expect it to crash and be buggy for now.
Also not all parts are working, especially the zotero sync isn't finished completely. So i wouldn't recommend to use it in a productive environment at the moment.

Enjoy the first preview of Conquirere.

35 comments:

  1. This sounds very promising. Maybe you should talk to the Kile guys as well? Keep us updated! :-)

    ReplyDelete
  2. As Kile supports the LyX pipe as well. So it does work already. Just forgot to mention it.

    ReplyDelete
  3. Hey, nice idea. I'm currently using mendeley for citation management. Would you see conquirere as a competing or a supplementary program to it?

    ReplyDelete
  4. Unless mendelay opens their sever api so I can fetch/sync references/publications from their online service, I see Conquirere as a competition in combination with Zotero (as soon as the Zotero support is fully working)

    As Medeleys strength is mostly the online service, I hope that someday I can simply integrate it.

    My personal intention is to create something that enhances the local management of all the necessary publications/references for your current research topic. Regardless where you store it online.

    So sync with Zotero/refBase/ maybe ownCloud some day or other services is my main goal. Conquirere will only be the last step to manage all things while you write the current paper. And if you add/change something on the local site, update it to any storage you/your research group prefers to keep it.

    ReplyDelete
  5. and while I've said that, I discovered Mendeley does infact offer a REST Api for their service. So expect support for it in the future. But first I like to have Zotero/plain file sync fully working and bugfree before I start something new

    ReplyDelete
  6. Does this integrate with the Nepomuk Project Integration [1] which was started as part of last year's GSoC, or does it introduce its own concept for a "project"?

    It would be nice to see the idea of a unified (Nepomuk-wide) concept of "projects" come to life, with different KDE applications each allowing to connect their specific resources to any project, and hence the KDE desktop becoming a convenient integrated project management tool (for those who want to use this feature).

    [1] http://trueg.wordpress.com/2011/05/13/nepomuk-projects-in-gsoc-2011/

    ReplyDelete
  7. Yes it does integrate with it. As I use the pimo:Project resource to store "my kind of research project" and the pimo:isRelated to connect the resources (notes, publications, documents etc) to it.

    In fact if you use the Rekonq bookmarking tool you can already see the projects created with Conquirere in it to relate webpages to it.

    I've just disabled the list of all related bookmarks/webpages connected to a project for now due to some problems with proper defined types on how a bookmark/webpage is written into Nepomuk.

    Nonetheless, Conquirere will play nicely with any program that uses Nepomuk and the Project concept.

    ReplyDelete
    Replies
    1. "Yes it does integrate with it."

      Cool!

      "I've just disabled the list of all related bookmarks/webpages connected to a project for now"

      I'm not even sure this would be necessary. Conquirere could just focus on connecting *its* resources to any pimo:Project, and Rekonq focus on connecting *its* resources to any pimo:Project, and so on, and leave the task of producing a central overview of all things connected to a specific pimo:Project to a dedicated Project Explorer to be written in the future (maybe as a Dolphin plugin?).

      On the other hand, it probably couldn't hurt to also selectively show some foreign (but conceptually related) resources like web links in Conquirere, for convenience...

      Delete
    2. I'm not sure either if the "bookmark/Webpage" approach is necessary. Was just one thing that came into my mind when I started to select resources I could list in the program.

      Same is true for the Notes and Emails. They are not really related to a specific publication and as such not in the scope of this program. Still I'll let them in for the moment and see what the future might reveal.

      A dedicated "Project Explorer" that connects project content from all the programs that support such an feature would be a great thing. But as long as there not other projects creating and working with a pimo:Project concept I won't start to think about it.

      Delete
  8. Thank you sir! Being in academia myself, this is the *very first application* I though of when I saw what Nepomuk was!

    Now, of course, on feature requests :)

    You almost certainly have already considered this, but how about cross-references? Add to the Nepomuk metadata of an article every reference it makes (and eventually making links between articles in the local database)? Of course, this information would have to be filled by hand, unless you would be willing to parse information from databases such as Citeseer.

    Oh yes, and any chance of a Kile integration? Of course, Kile is just a sophisticated front-end for LaTeX/BibTex, but think of the following:

    1. Auto-filling the database entries for articles produced by Kile (perhaps even with the cross-reference data mentioned above by parsing the .aux files).
    2. Auto-generation of the .bib file for the article.

    Anyway, again, congratulations for the project, I think it's very promising.

    ReplyDelete
    Replies
    1. The "cross-ref" part will be considered later on. But for the moment I concentrate on the "very basics" to get the program integrated into the often used search<->zotero<->latex toolchain.

      Kile integration will be coming too. piping the references to Kile does already work. Automatic .bibfile generation more or less too (set aside the current bugs ;)

      The .aux fiel parsing is something I have to investigate, but this will be done later on.

      Delete
  9. Zotero sync. Wow that sounds fantastic :-)

    Please pay attention that Conquirere could be port to plasma active easily at a later time. Such a program would be awesome for using tablets in an academic sphere

    ReplyDelete
    Replies
    1. Currently I don't pay much attention towards plasma active. as the code-base is split between background parts and the ui, I assume writing a new simple frontend that is touch friendly in mind will be easy later on.

      On the other hand, the ui is currently more some kind of "prototype" to get all the background work to display. If I decide to change a lot on the ui side I'll investiga what is necessary for the plasma-part.

      Delete
  10. Wonderful! As an academic, I have always missed having a KDE application to manage my references, and Conquirere is exactly what I was hoping for. I currently use Mendeley, which kills me because it is not free software. You said that Mendeley's strength is its online service, but honestly the reason I use it is because of how it fits my workflow. I spend most of my time reading pdfs of journal articles and writing notes, so being able to do that process efficiently is the most important aspect of this kind of software for me. In Mendeley, in the same window I have an excellent pdf viewer and the 'notes' field on the right side, and thus I can do my work without having to switch back and forth between applications, or with the window divided horizontally as in Zotero, which is very annoying when you try to read papers in small-ish widescreen laptop screens. From what I see in the screenshots above, the document viewer in Conquirere would allow me to have the same setup that I use now. The screenshot, however, only shows a webpage. Is it possible to view pdf files as well? If so, what viewer do you use? Is it possible to copy text from the pdf and paste it to the notes field, or even better, is there a way to make highlighted text in the pdf viewer to be automatically imported to the notes field? Sorry for all the questions, I only have the laptop in which I do my work so I really cannot try Conquirere here and risk losing all my data...

    In any case, thank you for working on this and good luck!

    ReplyDelete
    Replies
    1. pdf files can already be shown as well as any other file where a KPart exist for. Pdf files will be shown via Okular. Later on Conquirere will have a "similar" but not identical look and feel as mendeley has. This will also be part of a "2nd part change" as soon as the current parts are all bug free and working.

      Currently you're better off sticking to your workflow. Hopefully this will change someday :)

      Delete
  11. Finally! Something like this would be a killer application for people in academia. Have you read the discussion in the forums about how people envision a KDE bibliographic system? http://forum.kde.org/viewtopic.php?f=199&t=90272

    Currently I'm using Zotero to organize my references, hearing that sync with Zotero is already planned is wonderful. I'm curious about what additional benefits Conquirere will offer compared to Zotero, i.e., why should I use Conquirere instead of just sticking with Zotero?

    (OT: Don't know if it's just me, but Comment as: WordPress doesn't seem to work.)

    ReplyDelete
    Replies
    1. Yes I've seen this post and parts of it will be available in the future.

      As this program started because I had some free time and was curious if such a system would somehow workout, it adds no advantages over the pure Zotero approach currently.

      If I manage to get flawless syncing between Zotero and Nepomuk I consider my try successful.

      Once you have all your research data available in a semantic database you can start to use the information together with other parts of your system.

      * Like integrated search for all the papers cited by another paper.
      * If you import some references from a colleague it will be merged with your information from akonadi and as such you can easily find the contact details and all emails send between you and your colleague.
      * The KDE activity concept is based on Nepomuk as well so you can easily restrict a big library of references to a specific activity and concentrate on the thing you really need.
      * Sync between several provider (as soon as it is implemented, sync between zotero/mendely/refBase and others) or automatic im/export with the EndNote files forthe windows users out there.
      * And then of course at the end we can simple a program that integrates a lot better into the KDE desktop via KParts and such.
      * Maybe even reliable automatic keyword extraction via the nepomuk feeders and a nice way to use all the semantic data available to nepomuk to find more suggested papers and so on.

      Most ideas I have are a theoretical improvement and only the time will tell if "we all" really gain from it. But the only way to find out is to have such a tool available ;)

      In the beginning I only aim to replace the local zotero program by a proper sync with the online storage and proper export to .bib files or direct pipe to some of the LaTeX programs.

      Afterwards I try to add more and more functionality and see where it leads to.

      Delete
  12. Thanks für that nice review,
    As i'm currently struggling which programs to use for me soone to begin phd as a long-time kubuntu user am keeping an eye on you.
    So - for my diploma thesis i used kile + jabref.
    But some month ago i found mendeley to be nice - but even better the program paperpile. This one has a really nice approach - but seems to be out of development before it left beta status. Anyway - you should take a look at some features which would fit great on any bibliographic software.

    So - i also have some questions:
    What would you compare Conquirere (hard to keep in mind that name) to - Jabref? Zotero? Endnote? XYZ?

    Will it handle ORCID identifiers?

    Besides the mendeley integration thought - maybe resarchgate would be a nice choice (also i'm not sure how close they're linked to facebook)

    And also - As i think this programm shouldn't get a lot of attention - and i'm not a programmer (but will sooner or later try to teach myself a little) - but where can people help?

    ReplyDelete
    Replies
    1. > but even better the program paperpile
      That is really a nice approach they went with, I'll keep an eye on it when I start change the ui to something more "useful".

      > What would you compare Conquirere (hard to keep in mind that name) to - Jabref? Zotero? Endnote? XYZ?

      The names isn't the best to write/say but I couldn't find a better one so I'll stick with it, at least it does not belong to any other program out there.

      Currently I don't like to compare it to any other solution available. Mostly because Its unfinished and more a "proof-of-concept/techpreview" rather than a finished product.

      Once I'm done with the basics you can compare it to all other paper-management solutions out there. Even if someone decides to stick with Zotero/KBibTeX/JabRef/XYZ the ground work of this program can still be used to push all available information into Nepomuk. So at the end it might act like the akonadi-nepomuk-feeders where you modify your data with your preferred program but can still use Nepomuk for the fulltext search and other connections.

      So for my case. If I don't get it "finished" in the next 2-3 month I will rely on Texmaker/KBibTeX/Zotero for my master thesis and get back to it when I'm finished ;)

      I will add ORCID like any other identifier too. Must have missed it when I've added a bunch of other identifiers the last time.

      Beside mendelay, this program might get integrated into anything I can find as soon as I don't have to worry about the basic structure/functions.

      Help is welcome but without programming knowledge not very useful at this time. I could need some artists for the icons though.

      In a while I will appreciate lots of beta testers that brake the program as good as possible to get all the bugs out ;) But I#ll blog about this when the time is right

      Delete
  13. Quite a nice project! Thanks for doing this.

    Feature request: When reading articles in Okular I quite often want to look at the referenced articles. Would it be possible to improve Okular in such a way that a referenced article in some pdf would link to the nepomuk storage of Conquirere?

    ReplyDelete
    Replies
    1. In theory yes. Any program that would add support for nepomuk could benefit from the additional information gathered by the user/automated import.

      This means either:
      * The article you are reading is already indexed by nepomuk/conquirere and okular just need to ask nepomuk for the details.

      * The file was just downloaded and not processed currently, via text processing such references could be searched in the nepomuk storage and see what comes up.

      * Where the magic stops is when you try to find a referenced article not already known to nepomuk.

      Delete
    2. I think that it is possible to embed some of the bibliographical information into pdf (with LaTeX statements as \author{} or such) - do indexing daemons gather these informations?

      Delete
    3. Yes pdf metadata can contain a lot of useful data. As far as I know none of these data is fetched by the current streamanalyzer. Hopefully this will change someday.

      My personal dream is to gather as much information as possible without user interaction and put it into nepomuk someday. But it is a long way until all of this will be reached. Nonetheless "we" have to start somewhere ;)

      Delete
  14. This looks great and it would brilliantly simplify my workflow and organisation of everything!

    ...providing two things (yeah, I'm an oddball):

    - it would work together with Calibre (the eBook management application), so I would not need to tag everything twice;
    - it would have support for RDF, so that info from ODF (e.g. produced by Calligra) and other documents that include a lot of RDF info, would be imported also from there;
    - it would integrate into browser, also taking care of WWW links with date of visit and a historic fall-back autogenerated Internet Archive link;
    - it would be possible to set bookmarks to certain pages and or paragraphes, so it's easier to dump into the bibliography of the end document where exactly the reader should search for in the cited source;
    - it would export the (selected) bibliography to a standard or custom ConTeXt bibliography file (another TeX macrolanguage, which is a lot more flexible then LaTeX).

    I haven't checked it out yet, so I assume at least some of the points above are already implemented, but if all of them were, it would be a killer app for me!

    But so far it looks like a very interesting project and I'll keep my eye on it and see when will it make things easier for me, personally :]

    ReplyDelete
    Replies
    1. Thanks for the reply.
      Sadly I can tell you, currently noen of your wishes are supported at the moment. But it is a very young project and I work as fast as possible to add all those wishes :)

      Delete
  15. as a researcher and kde user, this is a dream come true. i look forward to using it. imagine if a mature conquirere gets more academicians to use kde!

    ReplyDelete
  16. Hi joerg. I can't say how much your work is appreciated. conquirere is shaping up nicely to be THE academic tool.
    I have two suggestions that I think will help a better integration in the future:
    1) Please, add nao:maintainedBy to statements created by conquirere in nepomuk. This way, it will be easier to cleanup any data created by conquirere without interfering with data created by, say, bangarang.
    2) Please create keywords as pimo:Topic instead of tag. With other apps using nepomuk, having as many tags as article keywords becomes unmanageable. And really, keywords are topics, not tags.

    Please, accept these as good willed suggestions and a great thank you for all this work.

    ReplyDelete
    Replies
    1. Hi Luis,

      > Please, add nao:maintainedBy [..]
      Is this necessary even after a switch to nepomuks new DMS system? as far as I've understand the system such information is saved within the produced graph data. So Someone could easily remove only those properties created by one application from one resource without touching what other applications added.

      If I'm wrong, I'll gladly change it.

      > Please create keywords as pimo:Topic [...]
      Wouldn't this just shift the "problem" just a bit. As pimo:Topic is a subclass of pimo:Tag?
      I can see the "unmanageable" problem once I try to tag an email as soon as digiKam and Conquirere produced ~1000+ tags, but isn't the open approach in this to tag each and everything easily and find the common denominator via this method?

      If I start to create those tags as pimo:Topic (while being a supertopic on its own) I'll end up with the same problem. Creating a supertopic (lets say "publication") and group all of them under this hierarchy, I'll lose the possibility to tag something with the same tag that would belong to different "supertopics".

      More clarification about your choice would be nice.

      >Please, accept these as good willed suggestions [...]
      I do. In fact I'm really glad when someone comments the nepomuk part. At the current stage I'm more concerned about the integrity of the nepomuk database than actual bugs and crashes ;)

      Delete
  17. I guess I should have done my homework a bit better. After looking into the DMS copy in kdepim it seems that nao:maintainedBy is indeed added automatically by nepomuk.
    As for the pimo:Tag vs pimo:Topic I regard the issue as a semantic one. I would pimo:Tag a paper as "To read" and "Meeting" but give it a pimo:Topic of "geomagnetic field" and "satellite data". In my mind, tags should be simpler concepts than topics and be completely transversal to all types of data in my system. The example of digikam is another interesting one. Pictures should not be pimo:Tag'ed with a name but nao:relatedTo a pimo:Person in an exact parallel to the way conquirere treats the author field of bibtex.
    I hope that I explained myself well enough and that the idea is useful.

    ReplyDelete
    Replies
    1. > nao:maintainedBy is indeed added automatically by nepomuk
      Great so this problem will go away when I start to use the DMS system later on.

      > As for the pimo:Tag vs pimo:Topic
      Your explanation sound plausible.
      I will change it and create the usual bibtex keywords as pimo:Topic (while each topic is a pimi:supertopic on its own) and will allow to add the usual tags to each publication too.

      Delete
  18. Conquirere looks marvellous, but it's been almost 3 months since last post, I was wondering if this is still being developed, which I hope.

    Thanks a lot!

    ReplyDelete
    Replies
    1. Yes I'm still working on it. Just have to finish some assinements first.
      But Conquirere is not dead.

      Delete
    2. That's great news!

      Delete
  19. data such as journals, books, proceedings papers, articles and many more to Nepomuk. Combine this data with the documents on your harddisk or some online storage and helps you to organize your data to quickly find some parts of your research again. www.acnc.com

    ReplyDelete
  20. Is Conquiere still under development? Would be awosome to see some progress here.

    ReplyDelete