Situated Interaction - INSTRUMENTS AND SUBSTRATES

Notes of: 

FUNDAMENTALS OF SITUATED INTERACTION - 16 SEPTEMBER 2016 MICHEL BEAUDOUIN-LAFON UNIVERSITÉ PARIS-SUD & INSTITUT UNIVERSITAIRE DE FRANCE

Professor: Wendy E. Mackay 

https://www.lri.fr/~mbl/Teach/SituatedComputing/2016/#l2

Lecture 1

Introduction

INVENTION OF THE TOOL

• Humans are the only species that creates tools to shape their environment;
• Traces of tools have been found as far back as 3.3 million years;

Most of our interactions with the real world are mediated by tools and instruments.

Tools to shape our environment, but not always easy to learn.

 

A bit of psychology

AFFORDANCES

• We directly perceive the capabilities for action of an object ;
• … the affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill…” James Gibson

PERCEPTUAL LEARNING

• Learning to recognize affordances；
• “We perceive to learn, as well as learn to perceive” Eleanor Gibson；

SIGNIFIERS

• Affordances as redefined by Don Norman；
• To be perceived, an affordance must be visible；

THE POWER OF TOOLS

• We internalize the tool as a physical extension of our body;

TECHNICAL REASONING

• We simulate in our head the physical mechanism to solve a problem;
• We appropriate the objects at hands;

APPROPRIATION

• A pen or a ruler? 
• A mug or a compass?

WHEN YOU HAVE A HAMMER…

• We create tools because we overestimate their capabilities;

What about digital tools?

GRAPHICAL INTERACTION

• SKETCHPAD IVAN SUTHERLAND, 1963

 

COMPUTER AS TOOL

• “Computers are like a bicycle for our minds” Steve Jobs

FROM PHYSICAL TOOLS TO DIGITAL TOOLS

 

Instrumental interaction

INTERACTION IS MEDIATED BY A TOOL

A DESCRIPTIVE MODEL

• From direct manipulation To tangible interaction
• But not universal:
  • Voice-based interaction?
  • Gesture-based interaction?

CPN2000

REIFICATION

• Transform a command into an object that can be directly manipulated;
  • Example : alignment
  • STICKYLINES M. Ciolfi, N. Maudet, W. Mackay, M. Beaudouin-Lafon

POLYMORPHISM

• The same tool can be used in different contexts；
  • Example : color selector；
• Free the tools from the applications where they are trapped!

REUSE

• Output reuse (objects);
  • Example : copy-paste
• Input reuse (commands)
  • Example : redo, macros

UBICOMP INSTRUMENTS

• Instruments spanning multiple interaction surfaces；
• Multi surface interaction；
• VIGO (CHI’09)；

Information substrates

INSTRUMENTAL INTERFACES

• To create and edit content
• BUT limited:  How to use the pen from the “Paper” app to write on a photo in the “iPhoto” app?

INFORMATION SUBSTRATES

• Data does not exist in a vacuum;
• Substrates provide context for interpreting data and constraints for presenting and interacting with it;
  • Examples: musical score, spreadsheet, page layout, graph…

PAPER SUBSTRATES

• Support the music composition process by combining and interpreting notations in various ways

INSTRUMENTS & SUBSTRATES

• Instruments can manipulate substrates;
• Instruments probe the substrate for specific properties or protocols to decide if they can operate;
• Instruments are themselves substrates;
• Instruments can be embedded in substrates;
• WEBSTRATES C. Klokmose, J. Eagan, S. Baader, W. Mackay , M. Beaudouin-Lafon

LAYERING SUBSTRATES

• A substrate can represent data in another substrate;
• Instruments can modify the different substrates in the stack;
  • Example: A table substrate - edit a value A graph substrate - set its type A histogram - set its color An image - paint on it

 

Conclusion

INTERACTION FROM FIRST PRINCIPLES

Reinventing interaction by separating tools from applications, and replacing applications with shareable and appropriable information substrates

 

Lecture 2

Human-Computer Partnerships or

Co-Adaptive Instruments

Computer hardware has changed dramatically over the past 40 years …

Computer capabilities are exploding but human capabilities are not ...

Can computers augment human capabilities?

 

Key Research Challenge

How can we improve interactive systems, given today’s ever-increasingly complex computational environment?

 

Situated Interaction

Focus on interaction

we cannot effectively model user behavior without taking context into account

 

Human behavior is planned, but action is situated: plans are a resource for action, not the action itself

 

Differentiate between measuring data and understanding the complexities of context

 

Methodology

Generative theory

• understand co-adaptive interaction
• principles for creating co-adaptive instruments

 

Participatory design with creative professionals

• develop novel prototypes
• real-time interaction
• personal language creation
• long-term reusable patterns of interaction

 

Empirical studies

• controlled laboratory studies
• extended field studies

 

 

ExSitu

focuses on: reinventing interactive software to support creative activities … based on two key ideas

 

We have multiple relationships with computers

• Computer as a tool I accomplish the task myself
• Computer as a servant It accomplishes the task for me
• Computer as a medium It lets me communicate with other people

 

Competing views of the future

Artificial Intelligence or Human-Computer Interaction 

 

GUIs are a vindication … and a challenge

 

Designed for executive secretaries to process documents in a completely different technology environment

Dates back to the 1970s to: copy hand-written notes check for mistakes format on letterhead

Problem: Brilliant then, out-moded today

 

 

Human-Computer Interaction research fought hard to make interfaces easier to use Today, novices easily accomplish simple tasks

Yet … advanced research in interaction techniques is rarely adopted in commercial systems

Today, experts use ineffecient techniques and are constantly forced to change their behavior 

 

Desktops, the web and apps …

• Require constant relearning:
  • each new version introduces arbitrary changes
  • each system requires slightly different interaction
• Require high visual attention
• Do not scale
• Depend on specific devices 

Smartphones are easy … but not powerful

 

We need to reassess human-computer interaction

• Early assumptions about graphical user interfaces no longer hold
• Everyone, not just experts manages increasing quantities of data faces information overload constantly relearns the details of interaction
• Redefine what we mean by “computer literacy”

 

Design Trade-offs

• Goal: Simple things should be simple … complex things should be possible
• Fundamental challenge: Balance trade-off between: power of expression simplicity of execution
• Design challenge: Shift the curve 

 

 

Strategy

Combine two key concepts

• Instrumental interaction (Michel Beaudouin-Lafon)
• Co-adaptive phenomena (Wendy Mackay)

 

 

Instrumental interaction  

Two levels of interaction: interaction with the instrument mediation with the system, Michel Beaudouin Lafon (2000)

 

 

 

 Magnetic guidelines

 

 

What makes an object an instrument?

Relationship between the user and the object

 

Human-Computer Relationships

• Between people and physical tools:
  • follow well-known physical principles users can learn them
  • users can appropriate them

 

 

• Between people and computer tools
  • follow arbitrary constantly changing rules
  • users must learn, and relearn, and relearn them users break them when they try to appropriate them

 

Focus on interaction, not interfaces

How can we let users control interaction

• in a flexible, reusable way,
• developing expertise without constantly relearning skills?

 

Solution: Co-adaptive Instruments

• Separate interaction from data and functionality
• Interaction becomes a first-class object

 

Co-adaptive phenomena

• Inspired by co-evolution in biology
  • Organisms create their environment
  • even as they adapt to it

在适应环境的过程中创造环境

 

• Anaerobic bacteria change the atmosphere making it possible for aerobic bacteria to emerge
• Users change spreadsheets from an addition tool to a tool for exploring ‘what if’ scenarios(?)

 

Key phenomenon: Co-adaptation 

• Users adapt to a new system they learn to use it
• Users adapt the new system to their own needs they appropriate and change it

 

Co-adaptive instruments

• Creative activities require both especially when integrating physical and digital information
• Create digital tools that are as intuitive, and learnable, as physical tools

Co-adaptive Instruments

• Worthwhile spending time and energy learning them
• Complex tools become accessible
  • can learn cognitive and sensori-motor skills
  • can adapt to new situations
• Move beyond graphical user interfaces to expert instruments
• To do this: Extract widgets from applications to create personal instruments 

 

Reciprocal Co-adaptation

• People adapt their behavior to technology… they learn it
• People adapt the technology for their own purposes … they appropriate it

 

• Computers adapt their behavior to people … machine learning
• Computers adapt human behavior … training

 

Human-Computer Partnerships

People have rich cognitive and sensory motor capabilities

increasingly,

so do computers

Why is the interface so limited?

 

• Physical tools
  • follow the laws of physics
  • we learn them
  • we appropriate them

 

• Computer tools
  • follow the whims of programmers
  • we learn, and relearn and relearn
  • and then we break them!

 

Our vision

• Software tools should be incrementally learnable
• People should choose and control their own tools
• Software tools should be easy to appropriate