Quick Links

• Office UI Bible: Customer Experience Improvement Program (aka SQM) - Part 1
• Office UI Bible: Customer Experience Improvement Program (aka SQM) - Part 2
• Office UI Bible: Customer Experience Improvement Program (aka SQM) - Part 3
• Office UI Bible: Overview of UI design process for pre-existing features

Office 2007 is the first Office ever in Microsoft’s history designed with statistics. That means, real world data from lots of real world users was used to design it. In contrast, previous versions were designed using educated guesses of Microsoft employees, partners and selected users. The amount of data collected is staggering: “About 1.3 billion sessions since we shipped Office 2003 (each session contains all the data points over a certain fixed time period.)” and “over 352 million command bar clicks in Word over the last 90 days.”

The data collection mechanism

The data collection mechanism is internally called “Service Quality Monitoring”, or just SQM. It was introduced in Office 2003, and presents itself to the user as “Customer Experience Improvement Program” (CEIP), or you might also see it under the heading of “Help Make Office Better”. SQM collected some basic data about your computer (e.g. screen resolution or CPU speed), and everything you do in an Office program that wouldn’t violate your privacy. That means, none of the actual characters you typed in Word were recorded, or any of the names of your Outlook mail folders, but it recorded all keyboard shortcuts you used, how your UI was customized, how many mail folders you had in Outlook, which buttons you clicked how frequently, etc. An excerpt of what the data revealed can be found in this Office UI Bible post.

The impact of the data

What did Microsoft do with the data? It turns out, a lot. The data combined with human judgment was the basis for the placement of all commands on the Ribbon. The Home tab in all programs is a great example of the statistics at work. The commands on the Home tab represent the 80% most used commands of that particular application. Some commands there are of course not as frequently used, but rather belong together with a frequently used command. For example, strikethrough was probably not in that 80% list, but we would all expect to find it near the frequently used command bold.

The statistics approach though doesn’t mean that frequently used features were strictly placed prominently on the Ribbon, while less frequently used features were always buried. Rather, features that people at Microsoft judged to be valuable to users, but underused according to the data, were made more accessible in the Ribbon. A revealing example of this is that some people discovered during the private beta phase of Office 2007, that Word now had drawing tools. Revealing about this is that Word has had those drawing tools since Word 97.

The SQM data was also used to decide whether to keep a feature in Office 2007 or cut it. Most of the time, the Office team had to make such a decision when a feature was judged not to be good enough for 2007. That means, in order for a feature to remain in 2007, development resources would have to be invested into it. One of the features that ended up on the chopping block was “Insert from Scanner or Camera”. It wasn’t good enough for 2007 and it wasn’t frequently enough used to justify spending resources on it. Instead, the Office team decided that users would be better off using the camera and scanner support provided by Windows, or shipped with a camera or scanner.

So what?

Basing design decisions on statistics led to a rather well though out user interface. For example, I had to customize PowerPoint 2003 heavily in order to be able to work effectively and productively with it. In contrast, I can use PowerPoint 2007 (after Microsoft incorporated the feedback from the SQM data collected during the beta) pretty well without having to customize it at all.

But, and this is a huge but, the new Ribbon UI shortchanges power users: Power users are the ones who just want to customize the Ribbon UI, but this isn’t exactly easy to do. The new charting UI works great for beginner to intermediate users, and will most likely lead to more users being able to create good looking charts. However, power users now have to dig in more places with more steps to get there than in previous versions. That means, the detailed chart settings are now at least one more click away than in previous versions, and their organization really leaves a lot to desire. These are just two examples of unfortunately too many, but the conclusion is clear: the Ribbon UI was not designed for power users.

What happened?

How can this be though, if statistics were used to design Office 2007? A mathematician once told me that she was asked by her boss to statistically analyze a large data collection. Her reply to her boss was: “what would you like me to conclude?” Or, in other words, she wanted to know which statistics to pick and how so that the desired result would come out of the data. Statistics therefore can lie, and it is easy to paint a wrong picture with them.

As a case in point, I read yesterday about the licensing changes in Vista on slashdot and the later post by Paul Thurrott titled “Windows Vista Licensing Changes - Everything you’ve read is wrong“. Paul spent the entire post outlining how the argument made on the Internet was wrong, especially because it only affected such a small percentage of users:

“What’s more amazing is that the number of people who actually try to do this is incredibly small. Since you can’t transfer a copy of Windows that comes with a new PC anyway, less than 10 percent of all Windows licenses are transferable at all. And of those, only a tiny percentage of users have ever tried to even transfer a Windows license once. The only people that really need to do this regularly are hardcore PC enthusiasts who change their machine configurations regularly. In short, this new restriction isn’t all that new and it won’t affect any mainstream users.”

Nothing wrong with this you might say? If you consider that there are over 400 million installed copies of Windows XP, then this means that ~40 million bought a transferable license (a retail license). “A tiny percentage of those” is then still a sizeable number of people. For example, just 1% of those represents 400,000 users.

Is this what happened with Office 2007? From what I can tell, probably not. It looks like Microsoft used the available data responsibly and has posted relevant data frequently in the Office UI Bible.

Was data collected from you?

What seems to be more the case though is that power users were underrepresented in the data. There is no statistical data to prove this, and Microsoft has tried to demonstrate that power users were included sufficiently in the data:

“Furthermore, because Office 2003 is still relatively new, it is deployed disproportionately among power users–the very people who most likely to customize. In all of our analyses, we try to be aware that the raw statistical data skews slightly towards early-adopting power users. This is probably much less true today than it was more than two years ago when we first compiled customization statistics.

(One metric that causes us to believe that the Office 2003 data slightly skews towards power users even today: Every month, the average screen resolution of people using Office 2003 decreases as more of the core installed base adopts it.)”

All I can bring forth against that is anecdotal evidence. I have not found a single power user yet, who did not switch off the SQM feature. Why? No one likes a phone home feature that sends personal data back to Microsoft. Especially when we have to believe Microsoft that our privacy isn’t being violated. Believing Microsoft that around the same time when things like product activation and related privacy concerns where still on everyone’s mind is difficult. For example, I personally advised everyone to switch SQM off, and even went as far as making sure that every Office I used had the feature switched off (in fact, switching SQM off was often the very first thing I did after opening an Office copy for the first time). It is also likely that many companies didn’t allow SQM at all, or blocked SQM traffic at the firewall. Last but not least, it is also possible that some power users tested Office 2003, but never upgraded their production environment to it, as it didn’t contain any improvements they judged valuable enough to upgrade.

Where you included in the data? Open Word 2003, and pick “Customer Feedback Options” from the Help menu. If you have selected Yes, as shown in the following screenshot, then you were among the lucky ones that had their usage of Office 2003 considered for the design of Office 2007. If you selected No, then you simply didn’t have a voice in the redesign of the Office UI.

Conclusion

Whether the data included a sufficient number of power users or not is a question we will never be able to answer. All I can say is that we have this new UI now as it is, and we have to make the best out of it. The most important conclusion of this post though is:

Always switch on the Customer Experience Improvement Program