Image Annotation Experiment

Many thanks for doing our experiment!

Please read through the instructions below carefully before starting. If you have any questions or observations about the study, please get in contact with us -- we'd be delighted to hear from you.

Prize Draw

By participating in this experiment, you will automatically be entered into our PRIZE DRAW. We will randomly choose three participants who will each receive an Amazon gift certificate for Ł25. So, please make sure to leave your email address in the appropriate box after the experimental task.

Instructions

In this experiment you will be presented with a news image, an article associated with the image, and a set of keywords describing the image. Your task is to judge how well each of the keywords describe the content of the image given the accompanying article. Some keywords will seem appropriate to you, but others will not. You will make your judgement by choosing a rating from 1 (the keywords are not appropriate) to 7 (the words are appropriate). All keywords were generated automatically by a computer program.

For example, if you were presented with the following image, keywords, and document:

airport arms carrying plane refuel Scotland book

The British government could face claims it violated international humanitarian laws by allowing US arms flights to Israel to use UK airports.

The Islamic Human Rights Commission (IHRC) is seeking permission to contest government bodies over what it says are crimes against the Geneva Convention. A number of US planes said to be carrying bombs to Israel refueled in the UK during the Lebanon conflict. The IHRC said it received complaints from Britons with families in Lebanon.

The commission is accusing the government of "grave and serious violations" of international humanitarian law. It is seeking permission to bring its case against the Civil Aviation Authority, the Foreign and Commonwealth Office and Defence Secretary Des Brown in the High Court. The IHRC said it is bringing the case after receiving "many complaints ... from British citizens whose family members are in Lebanon and facing grave danger as well as acts of terror". The US aircraft believed to have refueled in the UK are said to have been carrying supplies including "bunker buster bombs".

You would probably give the keywords plane and airport a high rating (e.g., 6 or 7) since they are relevant both for the image and the document. Indeed, the image shows a plane in an airport and the article discusses US planes landing in UK airports. Keywords arms, carrying, and refuel should also receive a relatively high rating (e.g., 4-6). Even though they are not explicitly depicted in the image, they are related to the accompanying article which discusses how the British government allowed US planes to refuel in the UK while carrying bombs. Keywords Scotland and book would probably receive a low rating (e.g., 1 or 2), since they are neither shown in the image nor mentioned in the article.

Interface

You will be presented with the image, the description keywords, and the document. Once you look at the picture and you read both the document and the keywords, please make your judgement for each keyword by selecting a number between 1 and 7. Each number will be represented by a button, all you have to do is click the button corresponding to your judgement.

There are no 'correct' answers, so whatever numbers seem appropriate to you are a valid response. While you are deciding a number for a set of keywords, try to ask the following questions:

Does the keyword describe information present in the image and the document?
Does the keyword represent the main topic of the document?
Does the keyword depict an object present in the picture?

Use high numbers if the answer to the above questions is 'yes', low numbers if it is 'no', and intermediate numbers for keywords that represent peripheral aspects of the image and document. Try to make up your mind quickly, base your judgments on your first impressions. The experiment will last less than 10 minutes.

Your personal details

Before the actual experiment begins, you'll see a form asking for details about yourself (this is the first thing you will see once you've pressed the start button below). We'd be grateful if you'd give a valid email address so that we can contact you if we have any questions about your answers, and so that we can mail you with information about the purpose of the experiment once it is completed.

Please be careful to fill in the Personal Details questionnaire correctly, as otherwise we will have to discard your responses. We ask you to supply the following information:

your name and email address;
your age and sex;
whether you are right or left handed (based on the hand you prefer to use for writing);
the academic subject you study or have studied (or your current occupation in case you haven't attended university);
In the field marked `Region' we'd like you to give us an indication of the region you grew up in, so that we have an idea of the type of English you speak (we'd like this information in case there are differences between dialects!).

Finally

Taking part in this experiment is entirely voluntary! Obviously we'd be grateful if you stayed the course, but of course you are at liberty to break off at any point during the experiment.