Apple's Response:

It was a disappointment when Apple was briefed a couple of weeks ago on a new computer usage study sponsored by Microsoft. It's basically a flawed imitation of the Arthur D. Little productivity study that was sponsored by Apple a couple of years ago.

This study has some serious technical and methodological flaws and in our professional opinion was not implemented well. We wanted to share our concerns about the study.

Background on the study

About two years ago, Apple sponsored a study in which Arthur D. Little tested the productivity of Windows 3 and Macintosh users performing 24 common tasks, in ten categories, on their computers. The study was designed to document something that virtually all industry analysts knew was true but couldn't quantify: That people using Macs could do work faster and better.

Microsoft's study appears to partially duplicate seven of the ten categories, and adds three new ones (we say "partially" because Microsoft changed many of the individual tests, and because we can't get access to all the details of the testing; they're a secret). The study also tests OS/2 in addition to Mac and Windows 95. Microsoft claims its tests show that Windows 95 users are 19% faster than Mac users, and somewhat more accurate in their work.

That's the headline I expect you to hear when the study is released. What they won't tell you is that the methodology of the tests is severely flawed and appears to contain some serious biases. You should insist on reading their full report very carefully.

Concerns about the study

We feel that the methodology of the tests was severely flawed and there appear to have been some serious biases. Here are some examples:

Conclusions

For the record, Apple believes that the basic usability of Windows 95 does come closer to a Mac than Windows 3.1 did. We think there are still some major usability advantages in the Mac, though -- things like built-in scripting and a much easier to use file system. With the next version of the Mac OS, we expect to widen that lead. But the most important point is that the industry is moving beyond basic ease of use to other issues, such as the ease of working with multimedia and the Internet. Those are the areas in which today's Macs really shine. And they're not even addressed by this study.

Mac Platform Marketing
Apple Computer, Inc.

IBM's Response

Question: Outperformed doing what?

This study was conducted, by IDC, at the request of Microsoft and was fully funded and directed by Microsoft. IBM has attempted to review the study, in more detail, but has been denied access to the actual test. All descriptions of tasks were written by Microsoft. IDC did read, over the phone, descriptions of the tasks users were asked to perform. Surprisingly, there were no multitasking questions, no measuring of one's ability to access the Internet through popular service providers, update a spreadsheet, or fax and file documents. The study apparently was based on an Apple study done three years ago. Not very representative of today's environment and not designed to take advantage of true 32 bit technology. Had the study really examined how many steps it takes to accomplish specific tasks using Windows applications versus doing the same tasks using OS/2 Warp applications, the study might have shown that OS/2 Warp's drag and drop interface saves the users many steps when faxing, reaching a favorite web page, or launching an application from the toolbar.

Question: How was the study conducted?

Microsoft indicates that the study measured speed and accuracy on a variety of tasks The firm who executed the tests, Andre Associates, did group evaluation with one timer to several participants. It is not clear to us that the evaluators knew what the pass/fail criteria was. There were no intervention protocols which means that if a person was on the wrong track they could stay there until they failed. One failure on a sub task could cause the entire task family to be counted as a failure. A precarious position for an OS/2 user since they did not have the same starting point as a Windows user, yet they were required to be a user of Microsoft applications. There was no customization of the desktop to allow for migration from the workplace shell. Thus, IBM has no way of knowing whether or not the systems were set up to allow exploitation of productivity enhancing features like the desktop toolbar to allow single-click application launching and task switching, Fast Boot for quick loading of Windows applications, or even launching of Windows applications directly from the Workplace Shell desktop. Since the study dictates the use of mostly Microsoft applications, Microsoft controlled how the applications were installed and presented to the user on each platform. Unless Microsoft went back to the machines to ensure that the Windows applications had been properly migrated to the OS/2 desktop and that they had been setup properly on the desktop toolbar - a set of operations that takes only seconds to do on OS/2 and is well documented - then it is likely that many users simply used Program Manager to launch the Office applications rather than exploiting the user interface advantages of OS/2 Warp.

Of interesting note also, Microsoft monitored the testing by having staff members on hand to answer questions. We have no way of knowing what "helpful hints" they may have given the OS/2 testers.

Question: Who were the test subjects?

The study indicated that IDC had difficulty finding OS/2 users with networking experience. Microsoft provided a list of OS/2 users to IDC. Microsoft's directive was that they only wanted to evaluate OS/2 users with Netware experience. IDC screened their lists and eliminated all OS/2 users with LAN Server experience. This eliminated a very substantial portion of the OS/2 user base from which to draw since many OS/2 installations rely on IBM LAN Server as their server of choice along with its intuitive drag and drop based user interface consistent with the rest of OS/2. Limiting the tests to OS/2 users who primarily rely on Windows applications and Netware servers, while NOT limiting the selection of Windows or Apple users to those with Netware but also including Windows users with Windows NT as a server or Apple users with AppleTalk, puts OS/2 at a marked disadvantage. Users of OS/2 Warp Connect peer functions would have been excluded while users of peer functions between Windows 95 and Windows NT as a server would not have been. Selection of the Windows 95 users from the Preview Program user base virtually guaranteed Microsoft a fairly advanced user base - those wishing to participate in a beta program as an early adopter - giving them an advantage neither IBM nor Apple were provided in the tests.

It seems clear that Microsoft carefully crafted this study to portray Windows in the best light.

We believe, as Apple has stated, that the study is flawed and unfair. The tests performed have been biased towards a Microsoft user and therefore would result in an improved performance for their operating system.

This study simply lacks credibility.

Walter W. Casey Director of Marketing Personal Software Products Division International Business Machines Corporation

Microsoft's Press Release

REDMOND, Wash. - In a test aimed at measuring the productivity of personal computer users in terms of speed and accuracy on a variety of common tasks, International Data Corp. (IDC) found that users of the Microsoft(R) Windows(R) 95 operating system outperformed Macintosh(R) and OS/2(R) users. Highlights of the study include the following:

"The IDC test indicates that users of Windows 95 completed common tasks in 19 percent less time than Mac users, and with greater overall accuracy," said David Card, author of the report and director of PC software research at IDC.

In July, IDC teamed with consulting firm Andre' Associates of Oakland, Calif., to execute a series of tests and focus groups for Microsoft that measure user productivity under different operating systems. The tests compared 54 Windows 95 Preview Program participants, 55 Mac operating system users and 52 OS/2 users with comparable skills on identical business-computing tasks and on comparably configured systems. To IDC's knowledge, this was the largest such study ever undertaken.

The tests comprised 10 families of tasks covering local and networked file management and printing, document and application management, checking system resources, creating an alias or shortcut, and customizing the desktop. The test also required the users to attach an external CD-ROM drive.

The study's intent was to evaluate the productivity of users of the operating system itself, rather than to measure application usage. The minimal interaction with applications was controlled through identical applications in each environment. Likewise, this test was designed to measure a mix of everyday tasks, such as manipulating files, as well as things business users do less frequently. IDC weighted all task groups equally.

My Comments

This is, in my opinion, typical of the Microsoft marketing machine. In nearly every review of Windows 95 it was acknowledged that, while better than Windows 3.11, Windows 95 was still not as good as the Macintosh. They are now trying to obfuscate that fact with carefully doctored studies.

Look at this claim in particular:

In a key measure of productivity, users of Windows 95 who completed eight or more of the 10 task groups successfully did so in 22 percent less time than Mac users and in 51 percent less time than OS/2 users. Eighty-five percent of that group of users of Windows 95 completed the test in less than one hour, compared to 47 percent of the Mac users.

Note that they are careful to distinguish the group of "users of Windows 95 who completed eight or more of the 10 task groups successfully" while they do not do so for the Macintosh or OS/2 users. This implies that they counted the entire groups of these users, including the ones who did not accomplish the tasks successfully. They did this again with "that group of users of Windows 95" versus simply "Mac users".

Only by comparing their successes to our failures can they come out ahead. It's pretty sad when they have to resort to this sort of thing.

Remember, Mark Twain said it best: "There are three types of lies. Lies, Damn Lies, and Statistics." I think this study fits all three of these categories.