peer review

The privacy review was a complex process involving many different steps. The one in which most engineers and non-technical users got stuck was evidence submission. This resulted in long back-and-forths with the technical auditors, who were flooded with queries from lost users. The privacy landscape also changed rapidly, and the auditors were overwhelmed were requests for more information before they were even trained on any new steps. All this held product teams back and led to product launches getting delayed.

To enable these users to find the help they need while also freeing up technical auditors a little, we proposed a peer review solution that users could opt into or skip. This was an added step that allowed users to elect peer reviewers: someone else from their team, their manager, or even from another team that had completed evidence submission successfully. These peer reviewers would then be notified and they would go over the evidence submitted before it went to the auditors.

The product designer and I mapped out the workflow as well as the alerts and notifications. This helped us visualize the process before getting to the design part. I also added the final content to this map.

The content explained what the peer review process entailed and what the potential benefits are. Some products had a mandatory peer review requirement.

We wanted to build the process out in a way that also encouraged users to give us feedback. I worked with the data scientist to figure out what kind of "data hooks" I could include in the content. 

We decided to measure 

• Adoption 

  • • Which teams were jumping on the opportunity to adopt this solution? 

    • Whom were they electing as their peer reviewer(s)? 

    • Which teams were skipping this step? What reason were these teams citing for not adopting peer review?

• Success rate

  • • What percent of users got their evidence approved in the first try after peer review?

    • Did peer review lower the workload and speed up the process for tech auditors?

    • Did users who opt in have more confidence in peer review? Did (or would) they opt in again?

• User experience

  • • Did users have all the necessary information about the peer review process?

    • How easy was it for users to elect peer reviewers?

    • How easy was it for the peer reviewers to review the evidence?

We added the peer review step to the end of the evidence submission process. (This screen does not have the final content.)