Sentence Compression Experiment

Many thanks for doing our experiment!

Please read through the instructions below carefully before starting. If you have any questions or observations about the study, please get in touch with us -- we'd be delighted to hear from you.

PRIZE DRAW

By participating in this experiment, you will automatically be entered into our PRIZE DRAW. We will randomly choose three participants who will each receive an Amazon gift certificate for Ł25. So, please make sure to leave your email address in the appropriate box after the experimental task.

Instructions

In this experiment you will be asked to judge how well a given sentence compresses the meaning of another sentence. You will see a series of sentences together with their compressed versions. Some sentence compressions will seem perfectly OK to you, but others will not. All compressed versions were generated automatically by a computer program.

Your task is to judge how good a compressed sentence is according to two criteria: (a) grammaticality, and (b) importance. The grammaticality judgement is based on whether the sentence is understandable. The importance judgement relates to how well the compression preserves the most important information of the original and whether it is adequately compressed. Both judgements are rated on scales from 1 (poor) to 5 (good).

A compression with a low grammaticality score is one that is almost impossible to understand. Compressions should receive low importance scores if they miss out important information from the original sentence. Or do not remove any superfluous information from the original sentence even though it is evident that it can be omitted without drastic information loss.

A good compression is one that is readily comprehensible and retains the most important information from the original sentence. Good sentence compressions should receive high grammaticality and importance scores.

For example, if you were asked to rate the following compression:

Nonetheless, FBI director Louis Freeh has today ordered a change - this is being reported by the New York Times - ordering new restrictions on the sharing of confidential information with the White House.

Nonetheless, FBI director ordered change new restrictions sharing confidential information with White House.

This sentence would probably receive a low grammaticality score (for example, 1 or 2) as it is difficult to understand. However it should receive a high score for importance (for example, 4 or 5) as it is possible to get the gist of the original. Now, consider the following compression of the same sentence:

FBI director Louis Freeh has today ordered a change - this is being reported by the New York Times.

you would give the compression a higher grammaticality score (for example, 4 or 5) but a low importance score (for example, 1 or 2). The compression preserves the least important information (the fact that the New York Times is reporting). On the other hand, if you were given the following compression:

FBI director Louis Freeh ordered new restrictions on sharing confidential information with the White House.

you would probably give it a high number for both grammaticality and importance (for example 4 or 5). Here, the compression is meaningful (grammatical), it produces a short version of the original sentence while retaining important pieces of information (i.e., the changes that have been ordered).

Interface

You will be presented with the original sentence and its corresponding compression. The latter will be always in blue colour. Once you read both sentences, please make your grammaticality and importance judgement. Simply select a number between 1 and 5 by clicking the appropriate button.

There are no 'correct' answers, so whatever numbers seem appropriate to you are a valid response. While you are deciding a number for a compression, try to ask the following questions:

Does the compressed sentence preserve the most important bits of information from the original sentence?
Is the compressed sentence easy to understand?
Has the compressed sentence removed information you deem not to be very important to the original sentence?
Does the compressed sentence seem fluent?

Use high numbers if the answer to the above questions is 'yes', low numbers if it is 'no', and intermediate numbers for sentences that are understandable, yet not entirely accurate or natural compressions of the original sentence. Try to make up your mind quickly, base your judgments on your first impressions.

The experiment will take approximately 20 minutes.

Your Personal Details

Before the actual experiment begins, you'll see a form asking for details about yourself (this is the first thing you will see once you've pressed the start button below). We'd be grateful if you'd give a valid email address so that we can contact you if we have any questions about your answers, and so that we can mail you with information about the purpose of the experiment once it is completed.

Please be careful to fill in the Personal Details questionnaire correctly, as otherwise we will have to discard your responses. We ask you to supply the following information:

your name and email address;
your age and sex;
whether you are right or left handed (based on the hand you prefer to use for writing);
the academic subject you study or have studied (or your current occupation in case you haven't attended university);
In the field marked `Region' we'd like you to give us an indication of the region you grew up in, so that we have an idea of the type of English you speak (we'd like this information in case there are differences between dialects!).

The personal data you give us is used only for scientific purposes. We will not give any of this information to anyone else, and nor will we report any information in any way that can be identified with you.

Finally

Taking part in this experiment is entirely voluntary! Obviously we'd be grateful if you stayed the course, but of course you are at liberty to break off at any point during the experiment.