Sunday, August 5, 2012

Conquirere and Nepomuk MetaData Extractor Update

After a long time some new updates on Conquirere and the Nepomuk MetaData extractor.

Since KDE 4.9 now uses nepomuk-core and this the Nepomuk2 namespace some changes had to be done to make everything work again. Andreas Cord-Landwehr was so nice to do the dirty work and ported Conquirere to the new Namespace.

While he worked on the port I've decided to add some minor helpers that will deal with large datasets in the program. The new Conquirere now adds a splash screen to show whats going on during the startup and for large datasets it is possible to use a cache to speed up the loading time of all entries.

What's still left is the very long time necessary to load all data into the model if the cache wasn't created.
This will be the main ToDo for the next versions.

After I have started to work on the Nepomuk parts again I wanted to fix the last few missing corners in the Nepomuk MetaData fetcher too.

Now there is a working fetcher that adds:

  • a gui to fetch folders and files with a Dolphin Service menu
  • an automatic fetcher in the background the files without user interaction
  • Plugin for Konqueror to import data from supported websites directly to Nepomuk
  • NPAPI Plugin and Extension for Chrome to import meta data from websites
  • Everything can be added easily into any other program

What websites are supported and how they are parsed is all done in python scripts (and can be done in Ruby and Javascript too)

Currently only IMDB.com, theTvdb.com and academic.research.microsoft.com are supported.

Because everyone loves screen shots, here is one showing the Chrome integration:

You can get the the source from https://projects.kde.org

What will the future bring?

  • support for music files
  • maybe support to fetch person data
  • more websites to fetch data from

13 comments:

  1. Hi Jörg, I have been having some problems lately compiling Conquirere. If I am correct, Conquirere depends on Nepomuk2/Thing, but this was recently removed by Vishesh from Nepmuk, so compilation fails. Am I missing something?

    The other issue I have been finding is that the cache works for some types of resources (like documents) but not for others, so every time I launch Conquirere I have to wait for a very long time for the items to load, which makes it pretty much impossible to use.

    Finally, I know you fixed the issue of the window becoming really wide and unable to be resized when switching resources and views, but the last few times that I have used Conquirere the problem was still there.

    Thanks so much for all your hard work!!

    ReplyDelete
    Replies
    1. Hi,
      seems I had several old files still on my hdd when I've checked Andreas port.
      Should work now.

      The cache works with all types on my end. Still this was just a temporary solution and didn't fix the real problem.
      I will work on the Model creation for my next step so loading a bug datasets won't take hours anymore. I have to agree in its current state Conquirere is unusable for more than a few hundred entries :/

      Oh boy still the resize issue :/ this is going to haunt me.
      I'll look into it again, but the model population will come first

      Delete
    2. As I noted in my bugs.kde.org bug report on the resize issue, it only seems to happen when the user has configured Konqueror to use Webkit, instead of KHTML.

      Delete
  2. Thank you very much for your work. This is great.

    ReplyDelete
  3. This is looking truly awesome.

    I had my first play last night, and it's got some rough edges but overall really cool. Would be good to see some parts merged downstream.

    Blog more. With more screenshots!

    ReplyDelete
  4. Hi, I tried to import a BibTex-file, it seemed to have worked, there seem to be some entries in Nepomuk, however, the Gui does not show me any entries, is that normal? Is there any trick?

    ReplyDelete
    Replies
    1. Seems I have some trouble with the resource watcher at the moment.

      Looks like a restart is the only way currently to show new resources. *wupps*
      I'll fix that later on.

      Delete
  5. Hello, you seem to have done what I was gonna set out to do: plugin-ify Trueg's tvshow metadata fetcher. I'm modifying it to suit my purposes, and think it might be useful if merged upstream too. Are you accepting patches? Can I submit you some on reviewboard or something?

    ReplyDelete
    Replies
    1. Feel free to send me patches per mail.

      Everything that makes this framework better is very welcome.

      Delete
    2. Hey, sent a patch your way via reviewboard, which should drop you a mail too I think. My use-case of interest is anime shows, so it's mainly work in that direction. Do let me know what you think!

      Delete
  6. Hi, I've been following this project from the initial ideas on forum.kde.org/science with much expectation. As already mentioned by others, I also think it can be a "killer application" for scientific bibliography and contribute to increase the acceptance of KDE in science. Unfortunately, from an outsider perspective, it seems that the project has lost focus (as music and videos are hardly related to scientific bibliography except if the research is on these fields) and it is not clear when a functional package (that could be installed by a normal user without requiring a large set of source code) will be available...
    Having said that, I do appreciate all the effort put in this project and I confess my incapacity to contribute to his code. I just wanted to have something functional (although limited) for scientific use as soon as possible that I could show to my fellow colleagues and use in my projects, instead of a "multi-purpose" tool in a far future...

    ReplyDelete
    Replies
    1. I always aimed for a mostly "stable" and "functional" (as in basics are working) release for the time the frameworks release is done and was always hoping to get it done sooner (as I need it for my own Master thesis shortly).

      The Extractor part, while it started as part of Conquirere, is just a very small part of it now. All the additional features in there are easy to implement and great at the moment, because I don't have much time currently. So I didn't lost my focus I just adapt my spare time coding with stuff I can finish in a short amount of time.

      But I have a few good ideas how to get rid of the slow startup speed and increase the overall "usability" of Conquirere and you can expect a new release mid to late October.

      Delete
  7. Thank you for. Look forward for a stable release. BTW when I said I can not contribute it's not exactly true... I can't help much in terms of KDE coding but I can help with translation (Portuguese, French and Tagalog) and probably documentation. So please feel free to tell me whenever I can be useful.

    ReplyDelete