Welcome to the writeup of Experiment One of our open-analytics experiments on AskThem, conducted with support from Google Civic Innovation. Background on this project.
The focus of this test was AskThem’s homepage above-the-fold copy, the first impression that many new visitors would have when the site launched. Many AskThem visitors first encounter an AskThem page from sharing a question over a social networking service; but in this case, the focus is the big homepage copy, and how it describes AskThem’s core branding & value proposition.
User-generated questions on AskThem circulate like online petitions towards a signature threshold, at which point they’re delivered to the recipient for an official response. Some elected officials have signed-up to be verified responders on AskThem, answering a couple of popular public questions every month. In this way, AskThem works like a version of the White House’s “We The People” petition platform, for every level of government: federal, state, county and city.
Of course, when we began development of AskThem in 2012, there were hugely well-established online petition services (Change.org, MoveOn Petitions, Causes, SumOfUs, Avaaz, etc.). And the White House’s “We the People” platform provided a positive example for AskThem – over 10 million action takers and several very-popular case studies (Death Star, cell phone unlocking, Bieber, Obamacare, etc.), pioneering the expectation of an official response to widespread questions that reach a goal.
So the question to test was – given our visitors’ familiarity with online petitions, and possibly “We The People” specifically, should AskThem bill itself as a petition site on its homepage? After all, questions can be used to petition elected officials, and it’s not an inconsistent message with our site’s value propositions, basic functionality or non-profit mission. But at the same time, the core value proposition of AskThem is significantly different – ask a question to your reps, and push for a public answer, then continue the conversation, with handy access to official government data to ask civically-minded questions. AskThem’s UI & UX didn’t seek to replicate commercial petition sites, but rather to connect visitors to questions from people near them and to enable them to connect with all of their elected officials by entering a street address.
The experiment tested engagement in the form of clicks on the page and clicks specifically on the zip-code locator button and the orange “Ask your own Question” button. It ran slightly after AskThem’s public launch on Feb. 10th, for four days from Feb. 13th-16th.
Variant A was the original copy, “Ask questions and get answers / every elected official and any verified Twitter account”:
Variant B was test copy, “Ask questions and start petitions / to every elected official and any verified Twitter account”.
After four days, though Optimizely didn’t fully declare a winning variation, the results were strong enough to justify a decision for the original (later experiments did run long enough to have Optimizely declare a winner – users can pause or end an experiment earlier than the software determines a winner):
On overall page engagement, for example, the “start petitions” variant started out performing better, but then trailed off, resulting in -2.8% improvement.
Clicks on the address / zip code locator were down similarly, -3.8%, though overall trend was closer. (This click gets visitors into the site to see questions from people near them and shows more faces of elected officials at every level of government, some of the site’s visual assets.)
Clicks on the “ask your own question” button fared worst, down 6.7%, with only a 33% chance to beat the baseline:
So in the days immediately after our launch, we determined that the “start petitions” language was depressing engagement, and we paused the experiment copy on our homepage, sticking with the “Ask questions and get answers” baseline to emphasize our value proposition of online conversation. After all, the petition circulation & signature-threshold models have become widespread, and as in “We the People” have helped to reliably indicate public opinion for issues to be addressed, but our non-profit site is first and foremost a Q&A platform, and by focusing on that loop – ask questions, deliver, post a response – we’ll create the best public-benefit product.
More-extensive user-surveys could help illuminate this empirically – but we can briefly hypothesize, do visitors to AskThem (perhaps referred from open-government allies & civic-tech mailing lists) not prefer to take the action of starting a petition to their city council member or mayor or governor, instead preferring a perhaps-softer action of asking a public question for a response? Seems valid, and more studies could determine if sound.
Additional thoughts on the (well-understood, but still) limitations of the pure petition model can be found in our non-profit advisors’ work: Micah Sifry’s forthcoming book, “The Big Disconnect: Why Technology Hasn’t Changed Politics (Yet)“; Anthea Watson Strong’s writing on her personal site and micropublishing stream; and Prof. Dave Karpf’s research & blog posts, “Shouting Loudly: building a healthy info ecosystem“, which is often critical of the ubiquity of e-petitions by advocacy groups; and Susannah Vila’s blog, including her recent report with Engine Room Advocacy, Â “User Engagement Strategies for Open Data”. Shouts to Nathan Woodhull of ControlShift Labs, for providing p2p organizing tools as a non-profit org., and our other six non-profit advisors.
From these revealing & crucial analytics experiments, my open-source development team and I now have empirical evidence on how we can enhance our non-profit platform. From here, I have other good revisions to connect visitors to their local elected officials – but we need charitable funding support to continue operating and move towards sustainability. Please support our non-profit work. Questions & comments welcome, david at ppolitics.org.