Last weekend, I had the pleasure of attending the Wikimedia CEE conference where I was scheduled to speak about the Paralympics. I had spent a fair amount of time before the start of the conference talking to people about metrics, so five minutes before the start of the presentation, I informed the conference organizer I was going to do a presentation about metrics. Original session notes from the presentation are available here. I’ll discuss the same themes a bit here.
What is the value of a single edit to Wikipedia? What is the value of a single file upload to Commons? This is the metric we are often asked to use in the movement to assess the strength of our programming. Give me a dollar amount. How much is too much to spend dollar wise for a single edit? I asked the audience this, and no one knew. We know that 100 edits is good. We know editor retention as defined by making multiple single edits over a period of time is desirable. But what does that mean?
The answer to the question of what is the value of a single edit is nothing. In and of itself, edit counts are not a valuable form of assessing the effectiveness of programming. In order to effectively your programming, you need a basket of metrics to understand exactly what is going on. The basket of metrics need to be understood against the backdrop of your clearly defined objectives as they relate to your programming. Inside the movement, there is a tendency to use edit counts because they are an easy measure to do. Really easy data to get. Proper data analysis takes work. It takes contextualization. It actually starts before you even do your programming by understanding your goals and objectives.
I didn’t go into this much, but let’s do a radical rethink here. We don’t sit there after the fact and measure. We start out by writing objectives. (Objectives and goals are different. Objectives are clearly definable. Goals are broad general things. A goal is to learn to edit Wikipedia. An objective is to have a contributor make 50 total edits, where they add 100 KBs of content, four references and 3 pictures.) What are your objectives? Are they reasonable? How do your objectives compare to similar projects? What were the outcomes for similar projects? What can you change to improve your outcomes related to shared objectives?
Back to metrics. The example I used at the conference was asking the WMF to give Wikimedia Serbia US$500,000 to sent printed copies of Serbian and Macedonian Wikipedia into space. For this blog post, let’s go simpler.
Lots of Wikimedia work is local, and there are a lot of volunteers doing things that are not always in a chapter context. Much of my own work currently falls into this category, and I really like self-assessment. The European Wheelchair Basketball Championships are coming up next year in the Czech Republic. I’m in Spain. It is feasible for me to go. 8 days of wheelchair basketball with media accreditation. €40 a night for 9 nights is €360. €180 for airfare. Figure another €200 for food and transport. That’s €740 which is kind of eek. As a volunteer, I might be willing to spend that because ZOMG! WHEELCHAIR BASKETBALL IS AWESOME! As some one who likes to understand what they are doing, I need to define my objectives before going. I can tell you the budget does align with some what closely with the IPC Alpine World Championships, would be much less than the IPC NorAm Cup and the London Paralympics. It is probably about twice the cost of the Rollers and Gliders World Championships. These costs are important to know because they provide a budget template, an outcomes template, and can lay the ground work for objectives. They can assist me with goals.
Goals are broadly defined goals. A lot of these are simple, and can easily align with the strategic priorities. I want to improve content about a group under represented content and participation wise. I want to increase visibility of Wikimedia in these groups and more broadly speaking. I want to strengthen partnerships with strategic partners. I want to increase collaboration between sister projects and languages on the same project. A lot of this can be done cheaply, on the internet and at little cost to myself, which is why defining objectives is important. (I can write articles, look for pictures on Flickr, send an e-mail and comment on a water cooler.)
Objectives here would include creating and improving content about wheelchair basketball players in advance of the Rio Paralympic Games, take pictures of wheelchair basketball players to illustrate articles in multiple language Wikipedias, and publish news stories about the championships on multiple languages. I want to increase the visibility of wheelchair basketball content on Wikimedia projects by reaching a non-Wikipedia audience. I want to strengthen my relationship with the Spanish Paralympic Committee, and develop a relationship with the International Wheelchair Basketball Federation. I want to encourage people with disabilities to edit Wikipedia. I want to increase participation of the Wikimedia community in editing content about wheelchair basketball.
These all align with previous successful work. They are all aligned with past work. (In this case, my own. It doesn’t need to be. If some one else is doing something similar, ask them for their own metrics, what their outcomes are, what their goals are, what their objectives were and if they thought they aligned. This sort of talk in the community needs to happen more. Get relevant information from the source. These relationships can and do come in handy. Related objective to asking people: Improving your own performance when it comes to program delivery. Setting up your contact network to facilitate a move to another country, or getting a job in another chapter by creating your successes using what you learned. If you’re doing WLM in a country that hasn’t done so before, it only makes sense to model that work. Why re-invent the wheel? Discuss. Ask. Forget about what others think of your work. Think about your own definitions of success, or as this example suggests, spending €740 out of pocket for a benefit that makes you happy.)
So we have our objectives. These need to be more measurable. About $350 spent resulted in 8 Wikinews articles in one language and about 250 pictures uploaded to Commons. This was for a three day event. London had about 60 articles written in about 10 days at a cost of AU$14000. Insane, two people. Not a reasonable outcome. This is me comparing outcomes. With one reporter, I’d probably shoot for around 20 articles. This aligns most closely with the IPC Alpine World Championships. Why did I look at this objective first? Because it is the one that is the most important for me. English Wikinews articles frequently outperform comparable English Wikipedia articles, especially for articles about non-English speaking subjects. This goes to audience reach. I want to connect with that audience. Output here for English Wikinews articles? It actually builds in a fair amount of reach because all Wikinews content is linked on Facebook and on Twitter. Twitter links are often retweeted by one to three different accounts. Facebook material is often shared by one or more people from the official Wikinews account page. Facebook has a reach of about 30,000 people. Twitter has a potential reach of about 100,000 people. Thus, I know I need to be tracking all links on Facebook and Twitter for all 20 of the stories I want to write. How many of these were retweeted/reposted? Who did the retweeting? Was it IWBF? If so, this assists in building a strategic partnership and building awareness, which I have established as objectives. Who did that content resharing is just as important as how many people. When I am spend €740, I am not looking at total edits. (Which for Wikinews can be two edits or forty edits. Talk about useless metric when the outcome that matters is published articles.) I am looking at creating Wikinews content, engaging with people I want to partner with, building awareness with the general public and influencers in the space. These are all easy enough to measure, but they take time. The data is public. It is a basket of data to give context for success. If my content isn’t tweeted or Facebooked, it becomes a problem. I will not be as successful. It isn’t just content itself. (If on the other hand I had data to suggest Facebook and Twitter were not useful, I would pass on this and on objectives related to them. It is well worth the time to look at the effectiveness of your Wikimedia project’s effectiveness of driving traffic to content. Some accounts are better drivers of traffic than others. Some accounts are not about generating traffic, but about saving staff time by sharing messages in the same place so as to avoid having to answer the same time consuming e-mails on a topic. There, the ROI for social media is volunteer and staff time saving. This is where metrics get messy. What exactly are you measuring and why?)
Pictures matter. They tell a story and convey ideas. I want to take pictures of wheelchair basketball players beyond what I have for the Wikinews articles. I know from having done the Australian Paralympic material that sometimes uploading a less than great picture of a person results in a better image being donated by some one connected to the person. I also know articles with pictures appear to get more page views than those without pictures. (I have the data around some place.) Often articles with pictures appear to be of higher quality. Thus, taking pictures builds into multiple measurable objectives. First, it creates an incentive for people with disabilities or an interest in disabilities to upload similar content on Commons. It also incentives them to edit Commons. It incentives them to edit Wikipedia. Sometimes, it incentives them to edit Wikipedia in different languages. I could probably build in an objective at this point to create 8 DYKs about wheelchair basketball players as a way of personally measuring the quality of content improvement being done, increasing awareness of the content type, and the flow on effect of getting more contributors from Wikimedia projects to edit this content. Here, I can measure the number of editors to the Wikipedia content, and it makes sense because the context is improving content. I think I can safely define changing an image as generally improving an article, typo fixing as improving an article. I’d like more but realistically, previous cases say no. Still, it can be measured. Another thing that can be measured is how many times those pictures are used. What’s the value of 100 pictures on Commons if none of them are getting used? Using them on articles adds value and that should be recorded. Also, I’d want to be looking again at page views because back to that objective about reach, which includes Wikimedia page views.
So if looking at the metrics to assess personal value based on my objectives, I am generally speaking going to look at the volume of content of content created, traffic volume, content linking and content inclusion, the social media audience, and editor participation. These begin to give a true measure of impact. They are based on realistic objectives. They do not assume one ultimate goal. They have the general underlying knowledge that a single metric is not feasible in terms of understanding what is going on. If I measured based on edit count alone, I probably would not spend the €740 or only spend it with the knowledge that I’m a true sport fan going for the love of the Game. A well rounded approach, with clearly defined objectives that you know how to measure and collect, balanced against a historical overview can tell you the value of your programming, be it personal or be it chapter run.