DEFT2013

accueil – présentation – FAQ – actes

FAQ

CONSTITUTION DES CORPUS

• Comment les recettes qui sont mises à notre disposition pour l'apprentissage ont -elles été extraites du site marmiton.org (tirage aléatoire, respect d'une chronologie et/ou de proportions prédéterminées entre classes) ? Peut-on savoir dans quelle mesure les données de test seront homogènes avec celles de l'apprenitssage Cette question vaut pour les tâches T1 et T2
  
  Concernant la constitution du corpus, nous avons extrait toutes les recettes du site Marmiton, puis constitué les corpus en veillant à respecter la proportion de document de chaque classe entre apprentissage et test (une classe davantage représentée dans l'apprentissage le sera également dans le test). L'équilibre entre apprentissage et test s'arrête là, nous n'avons pas vérifié la taille des fichiers.

A PROPOS DE L'APPARIEMENT TITRE/RECETTE

• Les recettes seront données sans leur titre, mais avec en regard une liste de titres contenant le bon (combien de titres par liste ?). Il faut donc apparier chaque recette avec chaque titre de la liste fournie et effectuer un ranking ?
  
  Oui. Voir ci-dessous pour le nombre de titres.
 
• - Les titres de chaque liste font il partie des recettes du corpus de test ou peuvent ils être différents ? Autrement dit, est ce que c'est l'ensemble des titres du corpus de test qui est utilisé pour générer les listes ?
  
  Les listes de titres contiennent tous les titres du corpus 2013, aussi bien ceux du corpus d'apprentissage, ceux du corpus de test, mais aussi des documents qui ne seront pas utilisés.
 
• - Peut-on avoir qq exemples ? Pour les autres tâches on peut se fixer les idées sur les exemples mais pas sur cette tâche
  
  Les titres sont tels qu'ils apparaissent dans les méta-données des recettes. Les listes de titres ont été créées sur la base de ces titres.
 
• - Les « ressources annexes » sont elles destinées à choisir des titres proches ou plutôt à éviter trop de proximité...
  
  Il n'y a pas de ressources fournies par les organisateurs. Vous avez le droit de vous constituer vos propres ressources à l'exclusion de celles issues du site marmiton.org

A PROPOS DES INGRÉDIENTS

• Would it be possible to know more about the annotation process. It seems that there are some cases where the annotator added an ingredient from the ingredients list without this being present in the body. for instance in recette_95792 "sel" is mentioned only in the ingredients section.
  
  It appears that the terminology list has been generate automatically: for instance in recipe 10516 the "gold" list mention "couteau" rather than "mouscade" and it does not mention "champignon" which is the main ingredient of the recipe. The question is: will the evaluation be carried on a manually validated list or just on the output of the algorithm which generated the terminology list?
  
  The annotation in the gold standard is directly based on meta-information found on the Marmiton Web site, for which we have no description. We found out that it is not correct in all instances: some ingredients present in the recipe or <ingredients> element may happen to be absent from the gold standard, and some ingredients listed in the gold standard may be errors. Fortunately these represent a small proportion of the ingredients. This means that there is a ceiling on the top score that a system may obtain.
  This reflects the fact that this task is exploratory: it does not aim at obtaining perfect results but at reaching full automation.
 
• Can you confirm that no ingredient absent from the list in the terminology file will be present in the evaluation set?
  
  We have added a new terminology file (see the download page) which contains a full list of gold standard normalized ingredient names. All ingredient names which must be predicted for the test set belong to this list. Therefore, no ingredient absent from this new list will be present in the test set.
 
• Are implicit ingredients to be detected? In 96143 we find "Fariner, saler et poivrer l'espadon" which implies "sel" "farine" et "poivre", which are however not present in the gold standard: could you confirm us that the behaviour is constant?
  
  Unfortunately we cannot guarantee that (see above why).
 
• Can you confirm that the <ingredients> elements will be absent in the evaluation set? The question stems from an example such as 96176 96176|appr|2013/recette_96176.xml|Plat principal-Très facile|6|ail lardons-fumes oignon poivre pomme-de-terre sel the text of the recipe says " et les faire revenir à sec avec les lardons": "lardon-fumes" is present only in the ingredient part.
  
  Je voudrais surtout savoir quelle seront les parties du fichier d'une recette qui nous seront fournis pour la phase de test : uniquement ce qui est dans la balise "préparation", ou bien aussi la partie "ingrédients" ? (j'imagine que ce sera le même corpus pour les 4 tâches, le reste 'titre', "type", "niveau" et "cout" seront donc supprimés).
  
  It is true that some of the ingredients are only present in the <ingredients> element of some recipes, and will therefore be impossible to find from the body of the recipe. This task reveals the actual issues that a system for automatically creating a list of ingredients from the description of a recipe would have to cope with. So this kind of issue is part of the specification of the task, for which a perfect answer is not reachable in some cases.
  We expect the proportion of such cases in the test set should not be much different from that in the training set.

A PROPOS DE L'ÉVALUATION

• Is it foreseen a metrics for evaluating partial matches of single ingredients? For instance in 96143 the gold annotation mention huile-d-olive, but in the recipe we find "Le faire dorer des 2 côtés dans l'huile". A system which retrieve just "huile" is it equivalent to one which does not retrieve anything are there is a "reward" for guessing the more generic term, given the fact that the more specific is not mentioned?
  
  The main metric, which will be used in the official ranking of the systems in the challenge, will use exact match: so in this example, including 'huile' in the list of ingredients returned by the system will be considered as noise.
  A secondary metric is planned which will take into account kind-of relations. With this secondary metric, 'huile' will obtain a partial match to the gold answer 'huile-d-olive' (it will be partly a true positive and partly noise, and therefore obtain only a part of the full score). We do not plan to have this extended metric as main metric because we are not certain we can to give it a sound behaviour in cases where a system selects very generic ingredients which partially match multiple gold ingredients.
 
• Would it be possible to know more on the evaluation metrics? Ideally, as in other evaluations the best thing would be to have the evaluation software itself. In case this is not possible, which is the formula for evaluating partial matches, i.e. ingredient A B C in the gold and A B or A C E in system's answer.
  
  La métrique Mean Average Precision (MAP) ou de Mean Reciprocal Rank (MRR) est elle choisie ?
  
  The metric will be the Mean Average Precision (MAP): the average precision over all relevant documents, non-interpolated. It is the average of the precision value obtained after each relevant document is retrieved. (When a relevant document is not retrieved at all, its precision is assumed to be 0.) We shall use the standard trec_eval program to compute it:
  
  trec_eval -mmap gold_standard_qrels system_ranked_list
  
  The system must return a ranked list of ingredients. The MAP rewards systems which rank all correct ingredients close to the top of their list. Conversely, adding incorrect ingredients at the bottom of the list (after the correct ingredients) does not penalize a system.
  
  For instance, given a recipe with 6 ingredients in the gold standard, a systems which finds these 6 ingredients but adds one incorrect ingredient will obtain a score ranging from 0.7345 to 1 depending on whether the incorrect ingredient is located from the first to the eighth position.
  Conversely, with the same gold standard, a system which ranks 5 out of the 6 ingredients on top of the list and does not provide the missing sixth ingredient at all obtains a MAP of 0.8333. Finding the sixth ingredient and ranking it after ten noisy ingredients increases the score to 0.8958. As mentioned above, adding noisy ingredients at the bottom of the list does not change the score.
  
  A system which ranks the 6 gold ingredients after ten noisy ingredients obtains a score of 0.2471.
  Finally, a system which only ranks 3 of the 6 gold ingredients at the top and none of the missing 3 gold ingredients obtains a MAP of 0.5, because missing gold ingredients are considered to have a precision of zero (it is as though they were ranked at an infinitely far position).
  
  However, trec_eval ignores queries (=recipes) for which no documents (ingredients) are provided. This behaviour would not penalize a system which does not propose any ingredient for a given test recipe. To avoid this, evaluation will be performed after checking that the system answer contains at least one ingredient for every test recipe. If this is not the case, the evaluation program will add a dummy ingredient for each missing recipe (which will cause this recipe to receive an average precision of zero). The script used to perform this normalization is t4-normalize-system-results.pl. Run perl t4-normalize-system-results.pl to print its usage information."
  
  ./t4-normalize-system-results.pl terminologie_2013_appr.qrels system.results >system.complete.results
  trec_eval -q -mmap terminologie_2013_appr.qrels system.complete.results
 
• Niveau de difficulté de réalisation d'une recette il est dit que la mesure d'évaluation sera calculée à partir de la distance entre la réponse du système et la bonne réponse. De quelle distance s'agit-il ? Peut-on avoir la formulation précise du critère qui sera employé
  
  Le script d'évaluation utilisé sera inspiré du script suivant : eval_task_1.sh

A PROPOS DU FORMAT DE SORTIE

• Est-ce que les résultats devront être fournis dans le même format ? en incluant la recette en entrée ? ou uniquement la balise correspondant à la tâche. Pour la tâche 4, est-ce que la liste des ingrédients normalisés devront être présentés sous la même forme que sur le site web.
  
  Pour les tâches 1, 2 et 3, le format attendu est un fichier tabulaire par tâche avec deux colonnes : le numéro de la recette, une tabulation, la réponse (niveau de difficulté sur la tâche 1, type de plat sur la tâche 2, titre de la recette sur la tâche 3).
  Pour la tâche 4, le format de sortie est le format demandé par le programme trec_eval. Voir par exemple Notes on Trec Eval, dont un extrait du texte est adapté ci-dessous.
  
  The results file has the format: recipe_id, iter, ingredient, rank, sim, run_id delimited by spaces.
  1. Recipe id is the recipe identifier (e.g. 94727 or 73863).
  2. The iter constant, 0, is required but ignored by trec_eval.
  3. The ingredient is the normalized name of the ingredient, like echalote or huile-d-olive (as written in the reference list of normalized ingredients).
  4. The Similarity (sim) is a float value.
  5. Rank is an integer from 0 to 1000, which is required but ignored by the program.
  6. Runid is a string which gets printed out with the output.
  An example of a results file:remplacer

    94727 0 ail 1 42.38 limsi_run1
    94727 0 basilic 1 40.01 limsi_run1
    94727 0 truite 1 36.42 limsi_run1
    94727 0 echalote 1 35.2 limsi_run1
    94727 0 huile-d-olive 1 32.28 limsi_run1
    94727 0 sel 1 31.23 limsi_run1
    94727 0 pesto 1 30.81 limsi_run1
    94727 0 citron 1 28.73 limsi_run1
  

  Input is assumed to be sorted numerically by qid. Sim is assumed to be higher for the docs to be retrieved first.
  Given the gold standard, in the results file above, all ingredients are correct except sel and pesto. Besides, this answer lacks several ingredients (which will get a precision of zero): bouillon, parmesan, pignon, poivre, tomate.
  Its MAP, computed with trec_eval, is 0.5227.
  
  See files exemple-faq.qrels and exemple-faq.results.
  The gold standard file, in the format needed by trec_eval (qrels file), is available in the download area as terminologie_2013_appr.qrels.