IMAGACT

A system to learn the reference to actions in Italian, English, Spanish and Chinese

In order to communicate, it's necessary to refer to actions. Speakers do not have a problem finding the right verb for a specific action in their own language. However, in a foreign language, they often have difficulty choosing the appropriate verb.
The reason is that one verb can refer, in its own meaning, to many different actions and we cannot be certain that the same set of alternatives is allowed in another language.

Like second language learners, automatic translation systems also suffer from this problem, especially when the translation of a simple sentence is required. For instance, asking the Google system to translate into Italian the English sentence "John pushes the garbage into a can" we presently get "John spinge l'immondizia nel secchio" instead of "John comprime l'immondizia nel secchio".
Working in the other direction, if we ask to get in English "Mario prende il gatto per la coda" this sentence is translated as "Mario takes the cat by the tail" instead of "Mario catches the cat by the tail", which is the actual interpretation; similarly the Italian sentence "Mario gira gli zucchini nella padella" is returned in English as "John turns the zucchini in the pan" instead of "John stirs the zucchini in the pan".

The reason for these mistranslations is that the set of possible interpretations of general action verbs, such as prendere, spingere, girare, comprimere in Italian and take, catch, push e turn in English and their cross-linguistic correspondences, are not mastered by the system and, indeed, are not explicitly settled neither in dictionaries nor in any current language ontology.

The problem is a significant one because reference to action is very frequent in ordinary spoken communication and most high-frequency verbs can each refer to many different action types.
The IMAGACT project, funded by the PAR/FAS program of the Tuscan region in Italy, has now delivered a language infrastructure covering the set of actions most frequently referred to in everyday language. Using English and Italian spoken corpora, we have identified 1010 distinct action concepts and visually represented them with prototypical scenes either animated in 3D or filmed.

The cross-linguistic correspondences of those actions with the verbs that can refer to them in English, Italian, Spanish and Chinese have been established in a data base that can be freely accessed on line through an user friendly user interface.

This is crucial for users who want learn the lexicon of another language, and makes IMAGACT specifically good for the early phases of the learning process. The action concepts can be easily identified by speakers of any language, since they are clearly represented in an ontology of animated and filmed scenes.

The user can figure out the range of actions identified by a verb of his own language and can come to know what is a right verb or verbs for each action in the other languages. He can explicitly compare the range of possible actions identified by two verbs of his languages or by two verbs of different languages. The user can also search for an action independently of any language in the IMAGACT gallery of scenes.