Why I don`t care for your F1 score – Evaluating Knowledge Extraction Tools for Industry Use

Industry Talk @ 15th European Semantic Web Conference 2018

June 7th, 2018, Heraklion, Greece

 

What is this about

With knowledge graphs gaining increasing popularity and importance within organizations, the need for effective and scalable knowledge extraction tools that can help accelerate the construction of such graphs is also getting more pressing. For organizations that have no such tools of their own, reusing existing ones (academic or commercial) looks like an attractive option. Yet, it is often the case that these tools do a poor job in communicating the features and abilities that an industry practitioner really cares about when he/she contemplates their usage. In this talk, I describe five key questions that we use when scrutinizing knowledge extraction tools for industry use, along with the kind of answers we expect to get:

  • Q1: What  extraction tasks and semantics does the tool exactlysupport?
  • Q2: What performance range can we expect from the system and under what conditions?
  • Q3: What are the ingredients and recipe(s) needed to use the tool?
  • Q4: How can we troubleshoot/optimize the tool’s performance?
  • Q5: How do we operationalize and maintain the tool?

 

Presentation Slides

Slides of the presentation can be downloaded here