Assessing Your Team’s UX Skills

by Jared M. Spool

“I didn’t realize it required so many different skills,” the newly-appointed user experience (UX) team manager told us. “I mean, it seemed so straight forward when we came up with the idea, but once we got into it, we kept realizing all the things we didn’t know how to do.”

Unfortunately, this isn’t the first time we’d heard this from a manager. In fact, we hear it quite often. Managers embark on a project, say a redesign of a critical internal application, only to realize their team, proficient in a few skills, doesn’t have the breadth of skills necessary to develop a quality experience. The result is an app that works, but frustrates the users because it doesn’t quite meet their needs.

Migrating Away from the Specialists Approach

Traditionally, when an organization set out to build a UX team, they did so by recruiting and hiring individuals trained in the various specialties. The individuals, having studied and practiced in areas, such as information architecture, visual design, and usability research, were hired to spend their days applying these skills to solve the organization’s design challenges.

However, most organizations couldn’t afford such a team. Either they didn’t have the budget to have someone dedicated to one area or they didn’t have the workload demand to justify it. Since the specialists weren’t really trained beyond their specialty, they couldn’t easily be assigned other work. (And if they were, they often resented the vast amounts of non-specialty work.)

Organizations that couldn’t afford a team would then “wing it”, training the personnel at hand to solve the problems on an as-needed-basis. These individuals, not having the experience and solid work time to dive deep into a specialty became generalists who could do many things on a satisfactory basis, but few things as well as the specialists.

Yet, over time, the specialties have become better at explaining what they do. Through conferences, a vast amount of well-written books, certificate programs, and online resources, the wealth of knowledge formerly only available to trained specialists is now generally available. Add that to the growing sophistication of today’s projects, and today’s best teams have generalists with experience and a level of knowledge on par with the top specialists.

Assessing the Team’s UX Skills

Part of the role of today’s UX manager is to ensure the UX team has all the necessary skills, at the best level possible. To do this, they need an easy way to assess the current skills of the team. Once they’ve got their assessment in hand, it’s fairly straightforward to design a coaching and training curriculum to fill in the gaps.

To help managers with the assessment, we’ve created a simple 130-point scoring process. They can just walk through the various skill areas, rating their team on a 0-5 scale, based on the teams current level of competency. When done, they’ll not only have an understanding of the team’s overall effectiveness, but a clear idea of where to improve.

The assessment is easy to complete. For each of the skill areas listed below, a manager rates their team using the following scoring criteria:

5—Give your team a five when, for the given skill areas, every member of the team has in-depth knowledge of the area, knows how to perform advance functions, and readily applies the necessary skills to every project that requires them (which, in most cases, will be practically every project). Teams that score fives regularly send members to conferences and have reading groups and discussion forums for reviewing the latest advances.
4—Give your team a four when just one individual on the team has the above-mentioned in-depth knowledge and everyone else on the team has a basic knowledge and experience with the skill area. Teams that score fours are often collections of specialists, with each specialist training the other team members on the basics, so they can function adequately when the specialist is tied up on other work.
3—Give your team a three when everyone on the team has just a basic understanding of the skill area. They can perform rudimentary tasks, but struggle when they run into scenarios demanding more creative solutions.
2—Give your team a two when there is an individual in the overall organization with the requisite skills, but not part of your team. This shows the organization sees the value in the skill area, but your team hasn’t made the case for having those skills locally. It’s good to have the skills within the organization, but these skills are so critical, someone who reports outside the team can only have limited impact.
1—Give your team a one when you regularly hire outside contractors with the requisite skills. While hiring contractors is a fine short-term solution, their experience and knowledge goes home when the contract is up, thus forcing the team to continually seek more outside help. Our research has shown that few of the most successful teams hire contractors, and, when they do, it’s with the intention of learning from the contractor’s work and bringing the skills into the team.
0—Give your team a zero when there aren’t any immediate resources available to the team for this skill area. Teams with zeroes regularly go without the areas at all or find themselves struggling when the skills become critical to the project.

The Core UX Skills

Now that we know how to rate the team, what do we rate them on? For the last few years, we’ve been studying the most effective teams, isolating those skills that make the most difference.

In our research, we’ve identified eight Core UX Skills. Teams directly apply these skills during the planning, design, and development phases to ensure the best possible user experience. The Core UX Skills are:

Information Architecture—Almost every design today involves organizing information, whether it’s an online policies-and-procedures library, product information, or user-generated videos. Information architecture helps us organize that content in a way that makes it easy for users to hone in to the specific content they’re seeking. Skills include understanding methods for organizing information, such as taxonomies, folksonomies, facets, and ontologies; techniques for deriving user hierarchies, such as card sorting; and creating design deliverables, such as site maps.
User Research—As we create designs, we need to ensure they meet the needs of the user. User research helps us collect information about who our users are, what they are trying to accomplish, what frustrates them, and what will delight them. Skills include identifying user population; techniques for evaluating design ideas, such as usability testing; and passing that information on to rest of the team members, so they can be making informed decisions.
Visual Design—One hallmark of good design is having a strong visual appearance. This is more than just aesthetic goodness, stretching into ensuring the priority of information is communicated visually — the most important information jumps off the screen while more subtle details are visible, yet not demanding unwarranted attention. Visual design skills include page layout, form design, color selection, and icon design. (While not directly “visual”, we consider designing for accessibility to fall into this skillset, as it focuses on much the same issues.)
Information Design—Presenting complex information for easy interpretation is key for a successful user interface. Knowing when to use specific table or graph types and using novel approaches for exploring detailed data sets, whether it’s pricing information, product comparison tables, or trend charts, makes solid information design a core component of the design process. Skills include knowing when to apply the variety of chart and table formats, such as pie charts, hi-low diagrams, and cluster treemaps; how to create interactive data explorers, such as star fields and drill-down pivot tables; and working with combining multiple data sources, such as data-mining techniques.
Interaction Design—Modern applications have moved past filling out a one-page form and pressing the submit button. Instead, they are now complex interactions, combining business requirements with a easy-to-follow user flow. Interaction design skills include knowing when to utilize different application structures, such as hub-and-spoke designs versus interview flows; which design elements are best for certain types of information, such as when to use radio buttons versus drop-down menus; and creating design deliverables such as wireframes and design priority descriptions.
Fast Iteration Management—Today’s best organizations are constantly learning from their designs. Instead of projects taking months or years, they now go from concept to implementation in weeks. Fast iteration management helps us learn to break designs into small, bite-sized implementations and to collect data from each deployment to inform the decisions in the next iteration. Skills include schedule planning, change management, and usage-data collection, to help the team move quickly.
Copywriting—Nobody likes using a design whose on-screen text reads like a 1950’s Army instruction manual. The best user experiences have copy that excites and compels, making the user feel comfortable and secure about the design. Copywriting skills include identifying the style of voice and tone that matches the organization’s brand, creating persuasive copy that motivates users to explore the design, and clearly stating benefit statements, to help the user understand the value of using new capabilities and functions.
Editing—What’s not in a design is as important as what’s included. Editing is not just about correcting bad grammar, but about creating a cohesive experience that doesn’t have extraneous distractions. Skills include using techniques such as alignment maps to match the users’ needs to the available functionality.

Because these skills are critical to the user experience, we double the rating we give them when we’re combining the scores to determine a team’s overall rating. For example, if a team warrants a 4 in Information Design, we’d count it as 8 in the final assessment. If a team scored 5 points in each of the Core UX Skills, they’d get a total of 80 points.

The Enterprise UX Skills

Along with the Core UX Skills, teams now need skills to enhance their value across the enterprise. User experience extends beyond the on-screen interactions to all touch points with the user. Special skills are needed to ensure the team interacts with the rest of the organization in a productive manner. While the teams don’t need to know how to do the jobs of others in the organizations, they need to know how those other roles will influence the design. These Enterprise UX skills include:

Development Methods—Organizations are using a wide variety of development methods these days. Team members need to understand how to integrate their work with development approaches, such as Agile techniques.
Design-To-Development Documentation—Communicating the design and its rationale effectively is critical to successful projects. Developing personas, design pattern libraries, and use cases are a regular practice to ensure what is imagined becomes reality.
Web Analytics—Gleaning important information from the mounds of data collected by today’s web servers can dramatically enhance the design process. Team members need to know how to integrate the available analytics to inform their design process, by seeing what designs are working for the users and where design iterations fall short.
Ethnography—Techniques like ethnography, contextual inquiry, and field research can help teams gain tremendous insights into the users’ environment and goals, leading to radical improvements to the experience. Understanding how to facilitate ethnographic projects and how to report the results, using tools such as contextual modeling and personas, is important for today’s teams.
Social Networks—Computer applications are no longer just a person interacting with a computer. Many are now computer-mediated experiences of people interacting with other people. Teams need to understand the different models for social interaction, from ratings and recommendation systems to full-blown social network capabilities, to know when these techniques can enhance the interaction and to avoid places where the social components take away from the core functionality.
Marketing—Previously relegated to an isolated function of the enterprise, marketing skills have become a core component in the user experience. Teams need to successfully communicate the design’s value to users and need to ensure it blends seamlessly into the rest of the experience with the organization and the brand.
Technology—It is no longer acceptable for designers to propose interactions that can’t be implemented because team member don’t understand how the technology works. From front-end technology, such as CSS, Ajax, and Flash, to back-end components, such as server technology and legacy servers, designers need to be keenly aware of what is possible and where they will bump into constraints.
ROI—A successful UX team has the skills to explain the business value of their work. Whether it’s a specific enhancement or a complete rethinking of the way things are done, team members need to concisely describe the benefits and risks associated with new design.
Business Knowledge—Today’s business environment is full of complexity and hard-to-navigate constraints. Designers need to be fully aware of how the business works, how it makes money, and what the internal constraints are, so they can ensure the design services the business as well as the users.
Domain Knowledge—The industries we service are themselves very complex, whether they be about financial services, travel, or a university. Team members need to be fully versed in the domain, so they can understand the terminology, processes, and objectives of the people using their designs.

Of course, you may find there are other skills critically necessary to succeed in your organization or environment. You can add them into your assessment formula alongside the skills listed above.

Arriving at the Team’s Rating

The formula we use for calculating a team’s rating is (Core UX Skills x 2) + Enterprise UX Skills. With 8 Core UX skills and 10 Enterprise UX Skills, the top score is (40 x 2) + 50 or 130.

How well did your team do? Here’s some guidance we give the teams we assess:

104 to 130: The team has all the right skills and is well on the way to mastering them. The next step would be expanding the knowledge by spreading it to other design agents in the organization.
78 to 103: The team has basic coverage of the core and enterprise skills, but needs to start enhancing the capabilities of the individual members. These teams need to focus on exploring outside educational opportunities, such as summits and conferences dedicated to these particular skillsets.
36 to 77: Many of the skills are found in the organization, but not all of them are found in the team itself. The team manager should actively search out training opportunities for team members, whether it’s tapping into the experts found elsewhere in the organization or outside seminars and workshops.
0 to 35: The team has few internal skills and is probably too reliant on outside contractors for many key skills. Working side-by-side with the contractors can help bring the skills in-house for future projects.

With the assessment in hand, team managers can now look for ways to enhance their team’s capability and improve the team’s productivity and results.

Published here on December 10, 2007.