<< back


Monsieur Jourdain: And this, the way I speak. What name would be applied to...?

Philosophy Master: The way you speak?

Monsieur Jourdain: Yes.

Philosophy Master: Prose

Monsieur Jourdain: It's prose. Well, what do you know about that! …These forty years now, I've been speaking in prose without knowing it.


(Molière, The Bourgeois Gentleman)


Abstract:
As the digital becomes part of the underlying structure of human existence and activity (just as electricity was integrated in the infrastructure), human interaction will be less and less direct. Mediated through various interfaces, human interaction via all kinds of networks becomes increasingly an expression of the semiotic condition of the human being in the post-industrial age. Professionals dedicated to human-computer interaction (HCI, a more suitable acronym than CHI) and semioticians must realize that they would benefit mutually if they would collaborate better than they have until now. Therefore, this paper submits working hypotheses to HCI practitioners and to semioticians dedicated to aspects of human interaction.

A position paper on human-computer interaction, beginning with a quote from literature is unusual. Instead of starting with bandwidth and chip spec considerations, and instead of joining the lamentation over I/O limitations, I submit that the problems of HCI are not primarily technological, but human. This is why the acronym HCI is better suited than CHI, and will be used in this text. As an aside, I suggest the HCI be adopted because this formulation establishes the cognitive framework: Humans initiate interaction. Through interactions, computation returns information and knowledge to the human being. This is why all those involved in HCI need to understand what goes on in it. More precisely, we do not know enough about forms of human interaction mediated by entities such as written commands, images, sounds, etc. that qualify as signs. All those involved in human-computer interaction "speak" semiotics, whether they are aware of it or not. In paraphrasing Paul Watzslawick's famous axiom – One cannot not communicate – I submit (again) to the HCI community that one cannot not interact. And because interaction is based on signs, one cannot not "semiotize"; that is, one cannot avoid semiotics. In other words, unlike prose, as a label for our common use of language, semiotics affects interactions the way our awareness of a process affects our understanding or that process and our ability to control it, that is, to optimize, augment, or fine tune it. Indeed, we express ourselves through various signs; we interpret them, and thus become part of the infinite sign process. Our practical experiences (work, learning, various transaction, etc.) are more and more semiotic in nature, i.e., more and more mediated by signs, visual or of other types. The computer is a semiotic engine. After all, it is not electrons that users are interested in, but the information processed and the new interpretations made possible. This is why the computer accelerated the semiotization of human life, including the semiotization of the interaction between humans and machines as well as among individuals involved in distributed activities.

What the quote from Molière should afford the reader is a very simple understanding. Yes, we all speak prose. And even those who are not aware of this can be good at it. The meta-level (definition of prose) and practical experience (speaking to each other) are not co-dependent. If we take Monsieur Jourdain to be an HCI engineer, we face a relatively similar situation: The meta-level (definition of HCI) and the practical experience (computation) are, however, co-dependent. In order to make effective computation possible, the CHI professional has to provide means of interaction that do not stand in the way ("Learn the interface"), that do not become a cognitive "overhead", but rather a cognitive support. Computational experiences should not be dependent upon the meta-level of HCI awareness. The case I am making through the formulation "One cannot NOT interact" is actually one in favor of computational ubiquity [1]. As is the case with every broad statement, it means close to nothing unless it is substantiated. Here are the arguments for my axiom:

Interactions can be direct: raising one's hand in greeting, pounding on a table, pulling a lever, cracking an egg, for example. Or they can be mediated: showing a picture in reference to a physical phenomenon, touching a soft-button on a copy machine in order to execute a desired operation, or clicking on an icon in order to connect to a Web address. A simulation is a mediated interaction through which we settle in the representation of what is simulated under the assumption that some coherence is preserved between the real and the represented. This is the underlying cognitive assumption of semiotics.

Our interaction with computers is mediated by various I/O means, methods, and procedures. There is no way to avoid this condition of semiotic mediation. In the analog world, direct interaction is still sometimes possible. In the digital realm, a language (of two digits, 0 and 1) and a grammar (Boolean logic) stands between the user and the hardware.



The intersection area – where and how the mediation takes place, i.e., how interactions occur – is of interest from the perspectives of many disciplines: psychology, cognitive science, technology, education, communication theory, design, economics (how effective is the interaction; what costs does it entail?), culture (to what extent does it fit the cultural context), among others. Each of these disciplines can affect our understanding of HCI and can, within the framework of methodological investigation, support the implementation and evaluation of user interfaces and other aspects of interaction. Some of these perspectives are more important than others; some are easier to consider when proceeding with the task of making HCI possible and effective. The only characteristic impossible to eliminate is the semiotic characteristic, because regardless of the nature of the interaction considered, it always takes place through the intermediary of signs (more precisely, sequences of 0 and 1, according to rules of encoding and decoding expressed in the adopted format, e.g., of 32-bit machines, 64-bit machines, etc.). This intermediary appears to the user in forms close to our culture: words, actions (e.g., clicking on a mouse, that is, sending an interrupt signal), images, sounds. All these form the object of semiotics. They constitute an underlying element similar to language (but not reducible to it), in the absence of which no interaction is possible.

Monsieur Jourdain had spoken prose all his life without having the slightest inkling of it. People around him understood what he had to say. The same thing holds true of HCI in respect to semiotics, but only in respect to describing how a certain sign works. A minority of those involved in the field is aware of semiotics above and beyond an elementary understanding of how a word or an image is used in order to facilitate an action. Without exception, HCI professionals want to gain access to the secret of designing effective means and methods of interaction. Monsieur Jourdain confessed, "I am in love with a lady of great rank and quality and wish to ask your help in writing her a note." Not unlike him, HCI professionals are dedicated to (and sometimes in love with) the computer and would like all the help they can get in order to make the interaction better and more intuitive. They want to support a variety of interactions corresponding to the variety of users. Eventually they try to reach that evasive goal (but a very assuring slogan!) of user-centered interaction, also known (in Norman's vocabulary [2]) as human-centered design.

Short of understanding how signs are constituted and how interpretations emerge in a pragmatic context (i.e., what is called sign process); moreover, short of understanding what the relation between the machine, the program, and the user is, the HCI community will at best continue to provide solutions that in fact reduce the effectiveness of the computation it is asked to support instead of enhancing it. Actually, the HCI community is aware of that fact that only a small fraction of what is possible through computation is realized. And again, this is not because of technology (which is ahead of us), but rather because of HCI that is not yet well conceived. The user does not need to know the meta-language (semiotics). However, the HCI professional cannot avoid learning it. In view of all this, I argue for the following practical goals:


1.The necessity of a semiotic foundation.

As long as the complexity of computation was very low (due to the fact that computers are in our day still in a very early stage of development) and the interaction limited, the HCI community has been able to get away with even rudimentary semiotic notions or with an intuitive procedure. The following assessment refers to both semioticians and HCI professionals: All kinds of dilettantes and peddlers of ideas picked up during a conference coffee break have been successful in promoting metaphors, visual conventions, and I/O devices whose packaging was better than their real performance. Let's face it, after the desktop metaphor, which was conceived on a semiotic foundation, almost nothing of similar impact has emerged. As the complexity of computation increases, richer interactions are possible. Especially as networking poses new challenges irreducible to the artistry of self-declared information architects and screen designers, interactions take new forms. Henceforth, the necessity of a semiotic foundation will make it impossible for HCI professionals to avoid studying the various aspects of interaction that networking brings about. Let me point to the relation between the represented and the representation, the dynamics of representation, visual codes, interpretation, multimedia, to name only a few of such aspects. (For more details, see Nadin, 1988 and the descriptions under Course of Study at the Website www.code.uni-wuppertal.de.)