Callahan's Pie Menu Versus Linear Menu Experiment - October 1987

Date: Tue, 14 Oct 86 23:28:14 EDT
From: mark@markssun.cs.umd.edu (Mark Weiser)
To: callahan@mimsy.umd.edu, don@mimsy.umd.edu
Subject: pie menus experiments

Jack, I am glad you decided to do a pi menu experiment. I did not read your experimental protocol carefully, but I think it is basically good. I will read it again tomorrow. Experiments are necessary to establish the efficacy of pi menus. If you get your data in time we, you, Don, and I, will have a super pi menu paper.
-mark

Date: Wed, 5 Nov 86 17:13:14 EST
From: Jack Callahan <callahan@tove.umd.edu>
To: don@tove.umd.edu
Cc: mark@tove.umd.edu
Subject: progress?

How are things going? I have subjects ready to run as of this Friday/next Monday. Shneiderman wants to stick with fixed menu lengths. He says that variable lengths adds to experiment complexity by adding dimensions on the current design. I agree. We have settled on 8 items.

Here are my 5 sample pie-menus:

North,South,East,West,NorthWest,SouthEast,NorthEast,SouthWest
Open,Close,Up,Down,Black,White,Left,Right
Full,Empty,Dark,Light,In,Out,Active,InActive
Young,Old,Fat,Thin,Tall,Short,Top,Bottom
More,Less,Larger,Smaller,Quick,Slow,Low,High

Here are my 5 sample linear menus:

1,2,3,4,5,6,7,8
One,Two,Three,Four,Five,Six,Seven,Eight
First,Second,Third,Fourth,Fifth,Sixth,Seventh,Eighth
A,B,C,D,E,F,G,H
Ape,Bear,Cat,Dragon,Elephant,Fish,Giraffe,Horse

and finally, five miscellaneous menus that fall into neither category:

Print,Move,Copy,Delete,Undo,Again,Find,Property
Clear,Next,Skip,Center,Bold,Case,Italic,Font
Apples,Spinach,Oranges,Pears,Beets,Tomato,Peach,Cabbage
Deliver,Another,Destroy,Reset,Kill,Abort,Start,Change
Create,Stop,Expand,Define,Shift,Lock,Search,Quit

These are enough for the pilot study. In the proposal, I stated there would be 5-10 tasks (menus) per cell. We have met the minimum, let's see what the pilot results say.

-- jack

Date: Mon, 17 Nov 86 10:02:04 EST
From: callahan@tumtum.cs.umd.edu (Jack Callahan)
To: don@mimsy.umd.edu
Subject: experiment subjects

I am going to post the sign-up sheets in psychology later this week and have people in for 1/2 hour sessions starting Monday (Nov 24). Later this week, we need to run pilot subjects for the draft report data. It's OK if things don't work perfectly - as long as we can run a competent experiment.

I got the NeWS-makers charter message.

-- jack

Can we run the pilot subjects this Wed or Friday? I'm busy Thurs with classes. When the real people come, it'll be Mon, Wed, and Fri for the Thanksgiving week and the first two weeks of Dec. When is the last day of class?

You won't need to be= there for the sessions unless you want to since this is my job.

Date: Wed, 19 Nov 86 15:59:52 EST
From: Jack Callahan <callahan@tove.umd.edu>
To: don@tove.umd.edu, mark@tove.umd.edu
Subject: the experiment begins!

I just posted the signups in psychology today and hordes (literally) of freshman psych types gobbled it up. I didn't see any actually sign-up, but there were about 20 of them standing around when I stapled it to the bulletin board. They officially start Monday morning at 9 am and run until 5 pm MWF until the last day of classes.

This does not mean that I am busy MWF until school ends. This was to allow the maximum people time to sign-up at their convenience. If we overbook, I will call people and tell them we are full and take down the sign-up. We only need about 25 people which includes estimates for data points we may have to throw out.

-- jack

Date: Thu, 20 Nov 86 22:53:28 EST
From: Jack Callahan <callahan@tove.umd.edu>
To: don@tove.umd.edu
Cc: mark@tove.umd.edu, callahan@tove.umd.edu
Subject: more menus

The following is the list of menus with the goals of each trial in parens. Each menu will be seen twice - once in pie format and once in linear format. Place the menu items appropriately where necessary (like up, down belong at the top and bottom of a pie menu and placed together in a linear menu). Opposites should be placed together in a linear menu and linear tasks should be placed in clockwise order in a pie menu.

Goal doesn't matter on demo and practice menus.

Here are the demo menus:

Using linear: Rook, Knight, Bishop, Queen, King, Bishop, Knight, Rook
Using pie: Hearts, Spades, Clubs, Diamonds, Ace, King, Jack, Queen

Here are the practice menus:

Using linear: Home, First Base, Second Base, Third Base, Left Field, Right Field, Foul, Foul
Using pie: Today, Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday

Here are the pie tasks:

North,South,East,West,NorthWest,SouthEast,NorthEast,SouthWest (North, SouthWest)
Open,Close,Up,Down,Black,White,Left,Right (Right, Down)
Full,Empty,Dark,Light,In,Out,Active,InActive (Empty, Active)
Young,Old,Fat,Thin,Tall,Short,Top,Bottom (Tall, Bottom)
More,Less,Larger,Smaller,Quick,Slow,Low,High (High, Low)

Here are the linear tasks

1,2,3,4,5,6,7,8 (4,8)
One,Two,Three,Four,Five,Six,Seven,Eight (Five, Seven)
First,Second,Third,Fourth,Fifth,Sixth,Seventh,Eighth (Sixth, Second)
A,B,C,D,E,F,G,H (F,B)
Ape,Bear,Cat,Dragon,Elephant,Fish,Giraffe,Horse (Giraffe,Dragon)

and finally, five miscellaneous menus that fall into neither category:

Print,Move,Copy,Delete,Undo,Again,Find,Property (Again,Move)
Clear,Next,Skip,Center,Bold,Case,Italic,Font (Font,Center)
Apples,Spinach,Oranges,Pears,Beets,Tomato,Peach,Cabbage (Spinach,Peach)
Deliver,Another,Destroy,Reset,Kill,Abort,Start,Change (Deliver,Change)
Create,Stop,Expand,Define,Shift,Lock,Search,Quit (Define,Quit)

This is the data we need from each trial:

SubjectNumber, MenuType, TaskType, TimeToItem

MenuType = linear, pie
TaskType = pie, linear, mixed
TimeToItem = time from menu up to item selected. If incorrect item selected, repeat menu and accumulate time

-- jack

Date: Mon, 1 Dec 86 08:29:28 EST
From: Jack Callahan <callahan@tove.umd.edu>
To: mark@tove.umd.edu, don@tove.umd.edu
Subject: results

Here is the first of two messages in a correspondence with Linda Weldon I should have cc'ed you two...

Sender: Jack Callahan:Computer Science:UofMaryland
Date: 30 Nov 86 19:49:42 EST (Sunday)
Subject: Re: help....
From: callahan
To: weldon%tove.umd.edu, ben%mimsy
cc: callahan

Tell you what - I'll wait until Thursday and prime you with some stuff over E-mail. Here goes:

If you remember my study, it is the pie vs pull down menus. The pie menus are the circularly formatted menus as opposed to a linear top-bottom design. The experiment is a 2x3:

                        pie             linear          other
                        tasks           tasks           tasks
using pie menus
using linear menus

A sample pie task would be like compass points where a linear task would be the numbers from 1-8. All menus have the same number of items (8). Each participant does 5 menus per cell. The menus repeat in each column so that they see the same menu twice using each menu type. Thus, there are a total of 15 separate menus and the whole experiment brings them thru 30 menus. They see each cell in random order, but see each menu in each cell in the same order. Did you follow that?

I ran the ANOVA and got the following. The dependent variable is SEEKTIME, the time they spent in microseconds to find the item once presented with the menu.

        for 300 observations (I've only run 10 subjects thus far)

        SOURCE          F VALUE         PR > F

        MENUTYPE        11.08           0.0010
        TASKTYPE        0.26            0.7682

                        MEANS

        MENUTYPE        N               SEEKTIME

        PIE             150             2.3485
        LINEAR          150             2.9968

        TASKTYPE        N               SEEKTIME

        PIE TASKS       100             2.5730
        LINEAR TASKS    100             2.7148
        OTHER           100             2.7302

I have yet to include the position of the menu item in the analysis nor the questionnaire data. Funny, but it seems they are faster using pie menus but overwhelmingly the subjective questionnaire says they prefer linear menus. (???)

-- jack

p.s. I cc'ed Dr. Shneiderman because I know he's interested in this discussion. I had asked Linda for stat help in the message previous to this one.

Date: Sun, 30 Nov 86 22:31:20 EST
From: Linda J. Weldon <weldon@tove.umd.edu>
Sender: weldon@tove.umd.edu
To: callahan@tove.umd.edu
Subject: fast pie

it certainly looks as though you found something! Namely, pie is faster than linear regardless of task type (and that task type doesn't make a difference.)

the subjective questionnaire results are interesting -- it is sometimes the case that people prefer things that they are not as good at. What are some possible explanations? familiarity with linear menus, maybe. . .

be seeing you,
linda

Date: Mon, 8 Dec 86 21:57:46 EST
From: Jack Callahan <callahan@tove.umd.edu>
To: don@tove.umd.edu, mark@tove.umd.edu
Subject: pie menus con't

I'm working on the draft paper for Ben's class tommorrow. So far, pies are doing well. Reduced errors and faster item seek times are consistent even with more subjects. Task type has finally show statistical significance as suspected. If I remember, I'll leave copies for both of you where you can find them (don - on your desk, mark - mailbox).

-- jack

Date: Mon, 23 Nov 87 23:57:33 EST
From: Jack Callahan <callahan@brillig.umd.edu>
To: don@brillig.umd.edu
Subject: pies accepted at SIGCHI 88

-- jack

Date: Thu, 3 Nov 88 14:09:24 EST
From: Jack Callahan <callahan@brillig.umd.edu>
To: don@brillig.umd.edu, weiser.pa@xerox.com
Subject: Re: ACM pie paper

The only problem with the original analysis was that I ran a BETWEEN subjects coorelation and should have run a WITHIN subjects analysis. This was changed for the SIGCHI paper. The change did not effect the performance results, but failed to show the significance for error rate reduction. Remember, the paper had two hypothesis: (to quote)

that pie menus decrease the seek time and error rates for menu items AND that pie menus are especially useful in menu applications suited for a circular format, diametrically opposed items sets, directions, or even linear sets of items and conversely, linear menus are useful for sets of linear items.

The implication of the second hypothesis is that performance and error rates are decreased because of these particular properties. I never did a per property analysis, but grouped them as "pie" like tasks. Similarly, sequential sets of items I grouped as "linear" tasks.

Kent Norman (UMd psych dept) suggested to me that I analyze the data along this third dimension to possibly precipitate the error rate reduction results out for particular item set organizations. The problem is that we start to run dry of data points. This is still possible since I have the raw data (still on tove - not on tape YET), but I haven't had the time. We may consider this for the CACM paper. I'll have to speak with the psych people again. Kent suggests that future experiments (if any) would carry this direction further. Don now works for Ben and I work for Jim and I haven't heard of any plans.

-- jack