Determining the relative quality of one Wikipedia project to another: One approach with English, Spanish, Catalan, Galician, Aragonese and Euskera Wikipedias

This is not as polished or as finished as I would have liked.  Apologies. Life got in the way.  I think the core findings and methodologies are still interesting and worth sharing despite the lack of polishing.The raw data is here: Spanish female politicians, and may be useful in terms of understanding how the results differ when you include null values as zero, rather than leaving them out of the conclusion.

Recently, on the research list, there has been a discussion regarding understanding the relative quality of articles on one language Wikipedia project to another Wikipedia project.


Exactly how to go about doing that is a rather subjective task, as quality could be potentially defined as, well, subjective. I’m going to try to do this within a very limited context.


The reason for these limits is because the metrics for easily measuring quality largely depend on the specific field of inquiry.  Quality sport articles will have different features than quality articles about plants which will in turn have different features than quality articles about military battles.


At the same time, I am not a programmer.  I cannot easily do programming things that would allow me to do bulk analysis of articles.  I need a very small sample to be able to feasibly work with.


When dealing with different languages, there is also an issue of best sourced material.  Often, people use English sources because those are easily available.  In the case of translation of articles, in many cases, people appear to just use those sources or find a best local fit.   To get a better idea of the actual quality, it appears to me that the subject matter to assess quality should largely be outside the English speaking domain, for the purpose of best understanding source usage.  Quality, thus, cannot become purely based on the quality of the translation and the local translators willingness to use other sources.


After some thought, I have decided to use the articles for “Female MEPs for Spain.”  (Women representing Spain who are or have been Members of the European Parliament.) This is a small list and finite list, which means I can manually get a large amount of data for comparison purposes.  All the articles are about the same topic, and because they are all about women, there are unlikely to be issues related to systemic bias in content creation. These articles are likely to exist in English, Spanish and possibly other languages for Spain.   Most of the sources should be from Spain because the topic is Spain, so a better feel for local quality can be understood in the context of language.


For this analysis, the decision was made to not examine other languages outside the languages used in Spain.  This is because the sample for other languages is very small, and given the already small sample, it not likely to have a lot of useful information. In English, there are twenty total articles.


In Spanish, there are 14 articles.  In Catalan, there are 20 articles.  In Galician, there are 7 articles.  In Euskera, there are 3 articles.  In Occidental, there are 0 articles.  In Extramaduran, there are 0 articles.  In Aragonese, there is 1 total articles.  In the context of the level of coverage, the best languages are English and Catalan because they have 20 total articles.  The next best Wikipedia, in terms of level of coverage is Spanish, followed by Galician, Euskera and Aragonese.  Other languages in Spain are not represented.


This lends itself to a philosophical question: With the languages having uneven sample sizes, should the analysis for overall article quality be based on the actual articles and ignore the non-existent articles, or should it treat the non-existent articles as having values of zero?  In this case, I will use both measures to determine quality.  The emphasis will be placed more on the existing articles because it allows for actual comparisons between articles.


There are a lot of criteria for determining the quality of an article about a female Spanish politician.  By using many criteria, examining them together, you can begin to get a comprehensive idea as to the relative quality while realizing that each article’s quality may differ.


Generally, this analysis defines article quality on Wikipedia about a politician as having four components.   First is appearance, and the presence of things not necessarily connected to the article text.  This arguably is the least important criteria. It includes having key external links, easy ways to get simple information without having to read the text, and having a picture.  The second is the content of the article itself in terms of length and other general features related to overall perception of article quality that do not relate to the topic.  This would be the second least important criteria, because they are independent of the actual textual content in some ways. The third criteria would be sourcing.  This matters a great deal as it defines the foundation of knowledge.  The fourth criteria is comprehensiveness of the article as a “political biography” by having some of the features that define a good political biography.  These criteria should be weighted to favor the more important article criteria over the least important ones.

Article “Appearance” Criteria

As this criteria is the least important one, points were weighted to make this criteria count less than others, with a maximum total 4.85 points available.


The first criteria I am going to use is: Does the article have a picture?  I believe this is a criteria for quality because many people want to know what a politician looks like.  Related to this, Does the article use a high quality picture of the politician? If the article has 1 picture, it gets 1 point. If the article has a picture but only because it was derived/cropped from another picture, it gets half a point.   If the article has 2 or more pictures, it gets 2 points.[1]  Because none of the articles have images labeled being high quality, there is no value in assigning further value to pictures.


The second appearance related criteria I am going to use: External links found either in an infobox or on the article to the politician’s official page, and any official social media presence they may have.  The reason for including this criteria is because the personal belief of the importance of going to an officially sanctioned source as part of knowledge formation around a subject.  Half a point will be given for a link to an official site, and half a point will be given to a link to an official social media presence.  The most available points for an article will be 1.


The third appearance is related criteria is: Presence of an infobox and a footer.  Infoboxes provide a lot of quality information in an easy to consume manner.  Related to this, the presence of a footer that provides related conceptual topics, such as the person who preceded or proceeded the woman in her position or other members of the same political party.  The presence of an infobox has been assigned a value of 0.6 and the presence of a footer has been assigned the value of 0.25.


The final criterion is the presence of a warning box on the article that says there is a problem with the article that almost certainly relates to content.   It is a potentially strong visual cue to readers that the article is not of high quality. All articles have 1 point. If there is a warning on the page itself, the article loses a point.


Using only appearance criteria, the maximum value points an article could have is 4.85.   There are only two articles which have full points, both are found on Spanish Wikipedia: Pilar del Castillo and Rosa Díez.   The lowest theoretical possible points is zero.  No articles have zero.  The lowest quality article in terms of total points is Inés Ayala on Spanish Wikipedia.


Article “Text Quality” Criteria

The first text quality issue is article section.  The presence of article sections suggests the article has organization and real structure.  Each article gets 1 point for each unique article text related section.  Headers for external links, see alsos, and references are not counted.


The second criteria used for text quality was article length using words.  The method for determining this was to determine the length of the text of the article, minus references, external links, infobox text, footer text, table text, image descriptions and lists.  The articles were then sorted based on length.  The longest 20% of articles were given 4 points.  The next longest were given 3 points.  The middle fifth of articles were given 2 points. The next shortest were given 1 point.  The shortest 5% were given 0 points.  This was done to give longer articles comparatively more value if they were the longest, and less value if they were shorter without passing any relative judgment on the quality of the volume of the text.


The third criterion again uses article size.  This time it divides the number of words by 250 to derive a number.  The number 250 was largely to offset the large values given to the outliers by bringing the number down to be more in line with relative weighting used with other criteria.   This gives value to the actual length of the article, as opposed to relative length.


Using readability as a criterion was considered.  For English, Flesch-Kincaid was used.  For Spanish, Fernandez-Huerta Scale was used.  The problem was that in trying to use one or both for the languages they were not intended for led to results that seem implausible.  Both scales had problems with Euskera, claiming the articles were written at an extremely high level.   The sole Aragonese article had similar problems.  While Catalan and Galician appear to be somewhat in line with Spanish articles, the inability to use two of the three languages and the requirement to use another system for English means this readability is not feasible as a criterion.


Because Wikipedia articles generally do not have a predictable maximum ceiling for article word length or number of sections, there technically is no ceiling for the maximum number of points available in this category.


Article “Sourcing” Criteria

The first sourcing related criterion is total number of sources.  An article gets 0.3 points for each source found in the reference section of the article.   The number 0.3 was largely to offset the large values given to the outliers by bringing the number down to be more in line with relative weighting used with other criteria.


The second sourcing criterion is the language of the sources.  Linguistic diversity amongst Spain’s languages should assist in offsetting potential POV problems and assist in providing best coverage for politicians from areas where Spanish is not the sole regional language.  The use of other language sources also potentially provides a more global perspective on the politician’s influence. For every different, language an article has outside the language of the project, it will be rewarded half a point.  (In some cases, the original source may be broken.  In this case, the website language will be used and value given based on that.)  Few points are being rewarded because of a desire not to provide too many additional points to articles just by virtue of the article having sources.


The third sourcing criterion is diversity of sources.  Ideally, the article should draw from different types of sources in order to provide a comprehensive, factual and neutral presentation of the person’s political life.  This is in line both with Wikipedia’s 5 pillars and with the requirements of a good political biography.  The different types of sources include newspapers  (television, radio, magazines), books, academic and trade journals, academic and education websites, social media, government (and parliament) sites, conference/commercial/social organization (not political or governmental) sites, party/political websites, and official sites.  These were all weighted as one point for having a reference in these categories.


Article “Political Biography” Criteria

The criteria for a good political biography tends to involve, broadly, getting a better idea of how politics and government works, while reading about the subject of the biography.  Some of the criteria for good political biographies may be slightly problematic in a pure Wikipedia sense in that the source material just may not be available to address the points adequately.  Still, in at least some cases, there should be adequate material about two or three of the politicians involved that are represented in most languages to begin to get a adequate picture and allow these outliers to pull up the average for the remaining article subjects. Because of the relative importance of these criteria against all other criteria, each has 12 available points, where if the biography partially meets the requirement, some points may be given.  This provides a maximum total points of 72 points, which accounts for roughly half the available points the maximum article has available.


The assessment of these criteria is purely subjective.  To a certain extent, the criteria also universally require greater depth so as to contextualize events that take place in the life of a politician.  The shorter the article, the less information about a specific topic, the less points will be subjectively given. For instance, if the thought process is only provided for one particular incident and the explanation is short, then one point will be given.  If the explanation is longer or there are two incidents where short thought processes are explained, then two points may be given.


The first criteria is, does the article present information about how the person governed.  This includes basic information about what the person did while in power.  To get at least one point, there needed to be at least one or two facts about what legislation the politician was involved with or voted for.  Holding the office alone was not defined in this case as getting any sense of government.


The second criteria is, does the article present information on the thoughts of the leader in terms of how they governed.  The article needs to explain some of the politician’s thought process behind political decision making.  The article cannot present events absent any context as to the politician’s reasons for their actions.


The third criteria is, does the article provide insight into how the person impacted political structures and policies in Spain, their specific region or internationally.  Context needs to be provided as to the impact of these policies so the reader understands the short and long term consequences of the politicians actions. To a certain extent, if the article mentioned how the politician performed relative to their party during an election, at least one point was awarded.  One point may also have been awarded had some background information been provided about their political party works in a national, non-office holding context.


The fourth criteria is, does the article show how being in power impacted the individual politician.  This can be biological or personal.  The person had a heart attack, their hair went gray, etc.  Their involvement in politics ruined their relationships, or put them in a position where they met a future spouse, or kept them in the closet.  The individual went to jail, or was continually followed by journalists who allowed them no privacy.    In cases of the politician going to prison for corruption or being found guilty of corruption, zero points were awarded here unless details were provided on how this impacted them personally.


The fifth criterion is the biography does not separate the person in terms of having a purely private life, and having a purely public life.  The two should be explained as they relate to each other, especially as the person’s primary notability will be for being a politician.  Details about a politicians life should not be present just for the sake of having them there, but be contextualized against their political life.  At university, did they display an interest in politics? Did a labor dispute put them into a place where they became politically active inside a job?  What events led them to becoming a politician?  How did their previous life experiences prepare them for being a politician?   To a certain extent, the article having a paragraph with facts about their education and other details about their life earned one point.  Only after there were more of those details and they connected more directly in the text to their political activities was a biography more than 1 point.


The sixth criterion is the relevance of the biography to Spanish and other Europeans who may have been impacted by the political events the politician has been involved in.  Readers need to be able to understand the politician’s impact on their own lives.


With 72 available points for each article, the most points earned by any article was 16.  It was the Spanish language article about Rosa Díez.  On the other side, 13 articles were assessed as having 0 points.   Over half of these articles were English, accounting for 8 of the 13 total articles.  Catalan had 3 articles assessed as 0 points.  Spanish and Euskera each had 1 article.  Galicaian and Aragonese had 0 points.



Overall, the article quality across all languages was relatively poor.  Not a single article would be objectively defined as a good political biography.  In terms of Wikipedia, none of the sample articles met local Wikipedia standards for being a good article.  Most were extremely short, averaging 288 words across all languages.   Most were poorly sourced.  While having an average of 2.3 sources per article, the median and mode of zero give a better idea as to the actual volume of the sourcing.  Most articles lacked pictures, or had a picture that was cropped from another picture and of poor overall quality.   Most articles did not give the reader a clear idea of the policies the politician supported, nor the impact of legislation a politician supported had on the lives of the electorate.  Almost all articles failed to explain the wider political impact, or lack of impact, the politician had on Spain.  The articles, across all languages, were not very useful.


Overall when measured against the assessed criteria, Spanish Wikipedia had the highest quality of articles.   With the highest single article point total of 60, articles on Spanish Wikipedia averaged 18.75 points.  This is significantly higher than the next highest assessed language project, Galician Wikipedia which had an average point total of 11.27.


Rank Language Score
1 Spanish 18.76
2 Galician 11.28
3 Catalan 8.48
4 Euskera 8.38
5 English 7.81
6 Aragonese 4.00


Rounding things out, Catalan Wikipedia was third, Euskera was fourth, English was fifth and Aragonese was last.  Even when the absence of articles is factored in with null values for these articles, Spanish Wikipedia still ranks as having the best article quality. Catalan finishes second, English Wikipedia third, and Galician fourth.  The absence of 13 of the 20 available articles hurts Galician Wikipedia a lot.


Why is the quality of Spanish Wikipedia so relatively high?  Why is Galician the second best language project?   Spanish ranked first in 8 of the 11 criteria.  Galician Wikipedia finished in the top 1 or 2 for five of the criteria.  In some of these categories, they had almost a full point above the lower performing language projects.  This was particularly important in the category of political biography, where Spanish Wikipedia articles averaged 4.4 points and Galician Wikipedia averaged 3.2 points per article.  In contrast, English Wikipedia averaged 1.5 points and Catalan Wikipedia averaged 1.35 points. Galician Wikipedia also picked up almost a full point on both English and Catalan Wikipedia when it came to average number of sections per article, getting 1.7 points to English Wikipedia’s 1.05 points and Catalan Wikipedia’s 0.9 points.  Galician Wikipedia also picked up half a point on both Catalan and English Wikipedia when it came to sourcing, averaging 0.7 points per article, which was still measurably less than Spanish Wikipedia which averaged 2.2 points per article. As each source was worth 0.3 points, this gave Galician Wikipedia more opportunities to get points for quality when it came to language diversity and source diversity.[2]


The political quality of the article correlates strongly with the article length, total sources in the article, and the number of sections an article has.  It would be more surprising if these qualities did not correlate well to each other, because articles need length, sources and organization as part of being able to successfully meet the political biography quality criteria.



[1]  Originally, the intention was to give articles with a recognised quality picture used on it  1 point. In this case, high quality would have defined as the picture being recognised either on Commons or a local Wikipedia project where the image is being used as a good picture, or a quality picture.  Unfortunately, none of the pictures used in the article met this criteria and so this was not used.

[2] In reality, that did not happen because only one article on Galician Wikipedia had any sources, and it had 17 of them.  The number and diversity of sources was limited in the article, and consequently, both English and Catalan Wikipedia outperformed Galician Wikipedia on source language and source type diversity.


Actual Sochi Paralympic budget

Armenian team at the Opening Ceremonies.

Armenian team at the Opening Ceremonies.

Before going to Sochi, I tried to budget and discussed this more in depth than people probably cared to know.  Budgeting is very important when you’re doing citizen journalism and you want to possibly get money to support your efforts.

Transportation involved two trips on the Russian metro at 40 rubles each, airport express train at 640 rubles, and a round trip train ticket from Moscow to Sochi at AU$125.  I got zapped with 116 RUBs for the train twice for sheets. Plane tickets were bought using frequent flier miles.  Retail price is showing me US$331. Do some converting: €0.78 + €12.59 + €81.12  + €4.564 + €238.06 = €337.11.  Not bad. About €100 if you subtract the plane ticket part out.

Hotel expense was €33 a night for five nights.  That equals €165.  Food was… That’s a bit harder to calculate.  I took with me €200 that I converted to rubles with no commission at €1 to 40 RUB.  la la la la.  Let’s go with €160 on for food and postage, with about €25 of that at the airport on the last day, including a breakfast that was 760 RuB / €14.87 from Burger King that included lots of stuff I did not want including a disgusting breakfast roll thing with a tomato in it.  Sbarros for lunch was much cheaper at 220 RUB / €4.328 which included two slices of pizza and a very large drink. Two bottles of Pepsi each ran 70 RUB / €1.377.

IMG_5223I screwed up and converted USD to RUB and did not convert it before I left Russia.  Ooops.  Add US$75.

All told, assuming actual cost of airline tickets, going to Sochi cost me €551.05 / US$766.18.  That isn’t that much.  Going to the London Paralympics, the cost was around AUD$7,500.  Costs were lower because I did not fly to Sochi, because I did not attend the whole games, because I missed meals, because I bought fewer souvenirs.   (It was AUD$15,000 for two people. This included everything from airfare to food to internet.)

What did this get me? Page views for all 2014 Winter Paralympics articles from 1 March to 14 March 2014 on English Wikinews total 14,685 views.  To be fair, I produced only 10 articles while in Sochi.

In London, myself and my fellow report produced around 50 to 60 total articles. That’s a huge volume.  My reporting partners in Sochi were Ukrainians, who were primarily writing in Ukrainian and doing their own work.  It wasn’t so much a partnership of working together to support each other’s English Wikinews reporting.  The page views for London original reporting around the Paralympic period total 78,943 views.  That’s about 5 times as many views.  The costs for London were 17 times higher: €9734.25 / €551.05 = 17.  I think reporting wise, I got my money’s worth here.

I think, when I do a better metric analysis, some of the breakdowns will be interesting.  Where this reporting project fell down was background research and background writing for English Wikipedia… but I think the Ukrainian project will demonstrate why that matters and how useful that particular aspect can be.  I know that they had zero articles about the Paralympics before 1 March 2014 on Ukrainian Wikipedia.  They now have 53 pages with 23,803 total views from 1 March to 14 March, the fifth most visited Wikipedia for articles about the 2014 Winter Paralympics found in that category.  But that’s another analysis to look at Return of Investment for another time.


Thoughts before leaving Sochi Paralympics: Reporting issues and Paralympic news

Look! I'm media! So cool!

Look! I’m media! So cool!

I leave the Sochi Paralympics today. As a reporter, I met my publishing goals that I set before I left. I need to remember that: 1 article a day is a lot on many levels, and anything else was just bonus. Also, I’m not getting paid to be here, I’m a citizen journalist writing articles for a Wikimedia project. I do not have to answer to an editor “back home” or justify the expense of going to Sochi.

But at the same time, I would have liked to have done more. One of the inherent problems with not being able to dedicate yourself to the craft of journalism is there is a lack of contacts, sometimes a lack of knowledge, and a lack of practice. Reporting relies on contacts. Reporting also relies on boldness. Be bold. Ask the questions. Go where things are. I do not have the contacts, and I can get only 2/3rds of the way there on boldness. (I might have gotten more had I done more research.)

A selfie from the cross country on Sunday

A selfie from the cross country on Sunday

This is why at times I am frustrated by my reporting here: For a few things I wrote, I feel like 70% of what I wrote came from information sheets journalist are given, 25% from pictures I took on a journalist (not photography) accreditation. The color feels hard to come through when my knowledge of things like curling fails me. Plus, I feel like I should write about all four curling sheets, all four matches… not just one. One well placed rock in one end that does not appear like it had a result on the outcome. How do you write that well?

In the afternoon match of wheelchair curling, the United States threw some really bad rocks, weren’t playing aggressively to win initially against the Russians. They went to an 8th end down by three rocks. My understanding is after the 7th end, if you do not think you can win, no 8th end required. The United States wheelchair curlers played the final end much more aggressively. They were assisted in the 8th end by some bad Russian stones which shot the gap between US stones and left the scoring area. This could only be banked on so much, and the Russians did not always misplace rocks. Despite much better playing and having three stones in the scoring area with 2 rocks to go, Russia managed to place a stone the USA could not get out. The United States ended up short, scoring only two of the three points they needed in the 8th end. Unlike the game Canada was playing against Norway on the sheet next to them, there was no forcing an extra end.

Having done so much writing for Wikipedia at times, I struggle with how to write neutrally. I second guess and end up neuturing things. That felt like the case here.

When I get back somewhere (Spain or the USA), I will write more on some of

The United States enters during the Opening Ceremony

The United States enters during the Opening Ceremony

the issues involved with Sochi, better planning thoughts, how to be more successful at this sort of thing, and the metrics at the end of the day.

Let’s move on to other news agencies coverage of the Paralympics. In London during the Paralympics, the Associated Press was not to be found. I read the news often, and I rarely if ever any coverage of the Paralympics. I did not see any reporters on the ground. Here in Sochi, it is a different case. There are USA journalists, several of whom are based up in the mountains. There is media coverage. There is television coverage. (Though I have been told NBC has few people here, with most of the NBC people using OBS feeds and as a consequence being based in the USA.)

I’m not entirely certain I am happy with other United States coverage that I have seen. It appears to take three forms: 1) Ukraine. 2) Oscar Pistorious. 3) Crashes.

I feel tremendous sympathy for the Ukraine. That situation has to a degree impacted my ability to do my reporting here. It is horrible what is going on in the Ukraine, and I cannot imagine the additional stress on the Ukrainian athletes and officials. But some of the reporting appears to hugely political. Craig Spence and Philip Craven were right about things being about sport. So when journalists appear to use the Paralympics as a throwaway line to make a political point about Putin while ignoring the broader issues of inclusion and the elite sport going on, it gets annoying. This is especially annoying when the Paralympics are not contextualized and most USAians have no idea what the Paralympics are about. Many of these sportspeople here get little news coverage outside the Paralympic period, so taking away their moment in the sun by making their participation part of some political disalogue gets annoying. Did I mention annoying? Perhaps I would find this news coverage about the Ukraine aspect less annoying if for every mention of the Ukraine in a Paralympic context, there was a sports articles about the performance of sportspeople… you know, like the Olympics.

The other thing that is annoying is the Oscar Pistorious thing. I don’t know whether I should be blaming the IPC here or the media. Everything is fundamentally political on some level in decision making. The IPC embracing of Vladmir Putin was political. The IPC embracing Oscar Pistorious and chosing to highlight his accomplishments on the big screen was a political decision of sorts. The IPC is chosing to align itself with a man who killed a woman (murdered is to be determined, killed is not disputed), who allegedly cheated on another woman and who liked to play with guns. The IPC chose to align with some one who, after winning silver in the London Paralympics, protested that the person who won gold had an unfair competitive advantage because of his blade. This was a guy who went to the Court of Arbitration for Sport to say his own blades did not give a competitive advantage. Hypocrisy, thy name is…

Anyway, back to Oscar and the media. The Associated Press was asking sportspeople about that. You’re a Ukrainian Paralympian. Your race ends. You have won a medal. A United States journalist asks you about… OSCAR PISTRIOUS! You are an American Paralympic medalist and your race ends and a United States journalist asks you about OSCAR PISTORIOUS! ZOMG. So awesome. Where was this during the Olympics? During the Olympics, I did not see a single Associated Press article where, following a person winning their medal in Sochi, the winner was asked, “Are you following the Oscar Pistrious situation? How does his actions impact your Olympic experience? How do you feel as an Olympian being connected to Oscar?” This feels like lazy journalism. Bold to be sure, but lazy. Maybe I have been too inside the movement, but I think the key to Paralympic success and disability access being improve is to focus on the sports. These are elite athletes. They train really hard. They give up a lot to be where they are. When their moment in the sun comes, when they become the best in their field, they get asked about Oscar. Their own accomplishments are ignored.

The last thing I have seen involves crashes and hospitalizations. That make news. I have a harder time faulting this one, though I would like to see more emphasis on success. Great Bitain can do that. Why can’t the USA media?

The reality of reporting from the Olympic and Paralympic Games

I have seen a fair amount of criticism lobbed at the media for their failure to cover a number of Russian related topics during the Olympic Games.  The LGBT topic should have gotten more attention.  Orphan children should have gotten more attention.  The deplorable situation regarding the environmental impact and the unhappiness of the locals in Sochi over the Games should have gotten more attention.

All of these are likely true, but ignore certain reporting constraints that take place during a major event like this.  I write for Wikinews, a Wikimedia Foundation project. I feel lucky to be able to attend the Paralympics because I am not part of a major newspaper, am not a broadcast rights holder, write for a volunteer driven project.  In order to get media accreditation, my news organization has to do a lot of work to demonstrate that we cover winter sports.  This involves going to world championships or other events, interviewing people, and otherwise demonstrating that we know something about the Paralympics.  We’re basically on some level showing we can promote the Paralympics.  For the Olympics, this process is particularly competitive and readership plays a huge part.  Freelancers are just not able to get media accreditation.  You have to be part of a media organization.  Most media organizations going are going to report on The Games.  Full stop.

My visa to get to Russia for the Games arrived February 14.  I could have probably left immediately and tried to do some reporting on those issues for Wikinews.  Several problems though: 1) Lack of funding.  I could not afford to do that if I wanted to.  If you look on Kickstarter and other funding projects for journalism, there isn’t much money being given to that. 2) I do not speak Russian.  This is a huge hurdle.  It makes it very difficult to do reporting. 3) I do not have the contact base to do reporting from Russia on these issues.  Contacts matter in reporting.  It means people talk to you, know you, trust you.  They are key to getting things done. 4) I wanted media accreditation to primarily cover sports and provide attention to disability sport.  I did not come to Sochi primarily to cover these other topics, which would mean spending time and energy away from The Games.

I have not been inside other news organizations, so I can only guess what is going on but I would think their issues would be similar.  It costs money to do that sort of reporting.  Newsrooms across the globe have been shedding global staff, and shuttering their international correspondences desks.  Instead, they have opted for freelancers and stringers.  The economics of this reporting then are not great.  (In my case, I am a volunteer so the economics are what can I afford? And the sister project to Wikinews is Wikipedia, which allows no original reporting.  They just repeat the main stream media.) Many of the people going to the Paralympics and Olympics are sport reporters. They are here to report on sports.  That is what they know how to do.  That is what they are paid to do.  Would you go to a neurosurgeon for knee reconstruction?  Probably not.  Larger news organizations with more media accreditations may be able to afford to do that.  They may also be able to be better placed to hire stringers who can report on these issues because visas are not an issue if a person is a local reporter already inside Russia.  In London, a lot of these questions got asked at press conferences because the local media asked them.  The local media here is unlikely to be asking those questions. But by and large, it just isn’t very feasible here.

There are ways to report on these issues from inside the Games in Sochi.  They just take work, and editorial people back at home base approving of this.  I’ll try to do my bit, but please do not be hugely disappointed if I cannot deliver a lot on Putin, the Ukraine and lastly the Paralympics.  Not what I am here for though I am really going to make the effort to try.

My concerns about the Wikimedia Foundation’s proposed changes on the Terms of Use

There is a conversation going on on meta about changes to the Wikimedia Foundation terms of use.  One of my comments in that discussion is pasted below.

I too have concerns, not just for professors but for the religious, for members of the armed forces and for anyone who receives an income. As the proposal stands, it creates a climate where contributors are actively encouraged to violate Assume Good Faith, and ascribing motivations to edits and encouraging contributors to make accusations of bad faith, COI editing. This creates a highly toxic editing climate, which we can see from the Chelsea Manning case that went to ArbCom and the related controversy. Advocates of certain positions had people seek out personal details on their lives. The type of information sought out would not be allowed in a Wikipedia article about the individual, but could be used to discredit the user. Accusations of a financial incentive to promote a certain position took place during the Chelsea Manning case with the implication that members of the military were acting as spokespeople to promote a government position. (Similar accusations were also made about people on the opposing side of the issue, with related COI complaints.) Others face similar accusations on a regular basis while editing English Wikipedia, with the text of the edit not being examined for its alignment with local policies regarding acceptable content on Wikipedia. Instead, accusations of COI editing are used to undermine the body of their work. This proposal should include Terms of use/Harassment and outing amendment which details about what non-personally disclosed details about a user may be shared by other contributors on pages hosted by the Wikimedia Foundation, methods for dealing with disclosure of non-public information (such as employment details) on Wikimedia projects that would actively discourage people from refusal to interact with just the text, and details about how the WMF will support its user base who comply with this policy while making edits that comply with pillars on Wikipedia and BLP but are targeted for harassment by Wikipedia users.

Wikipedia’s selective gendering of national team article names

In an earlier post, I talked about how some articles are over represented when it comes to article’s about men’s versus women’s national teams.  I know from my own experiences that many articles about men’s national teams are gendered while women’s teams are not.

400 gendered men, 1800 ungendered

The graph above clearly shows this pattern. The majority of women’s article titles are gendered.  The majority of men’s national team articles are not.  Beyond totals, one way of understanding English Wikipedia’s systemic bias against women in sport is to look at what happens when there is a pair of articles about a country’s national team for a particular sport, with one article about the women’s team and one article about the men’s team. Softball, water polo, pitch and putt, lacrosse, inline hockey, floorball, goalball and wheelchair basketball national teams are pretty much all have titles indicating gender, even in situations where there is no match pairs.  Tennis articles consistently do not gender both genders.  Some countries defy traditional gender categorizing, including the United States and Canada, which are both more likely to gender indicate male teams for sports when almost every other country does not.

Before going into this analysis deeper, the group “Male – gendered, female – ungendered” never appears.  There were zero matched pairs where a male article contained the word men and the female article did not contain women. There is no systemic bias in terms of article titles that favour women.

This was done using the same list of national team articles.  It mostly relied on pulling article names from the categories for national teams on English Wikipedia. All articles on the resulting list were tagged as either being gendered because they contained the word “men” or “women” in the article title, or being “not gendered” because they did not contain  word “men” or “women” in the article title. Matched pairs of men’s and women’s teams were sought by country. As there is a much larger number of articles about men’s teams and some sports are more female oriented, the number of articles covered was not going to be equal. 836 matching pairs were found for 223 different countries and 28 different sports. The pairs were then labeled “Male – gendered, female – gendered”, “Male – gendered, female – not gendered”, or “Male – not gendered, female – not gendered”.

There were 517 instances of “Male – gendered, female – not gendered”, 200 instances of “Male – gendered, female – gendered”, and 119 instances of “Male – not gendered, female – not gendered”. (All 119 instances of neither gendered are tennis.)  Total, 61% of Wikipedia’s national team articles involve selective gendering favouring men. 29% of the time, selective gendering is not done.

As mentioned previously, this pattern changes from country to country and sport to sport.  In the case of Great Britain, Canada and United States, over 60% of the time, both teams are gender identified.  If we eliminate tennis’s neither gendering, Turkmenistan, Great Britain, Canada, Tiawan, Botswana, Puerto Rico, United States, Philippines, Czech Republic, Egypt, Mexico, and Venezuela all have more than 60% of their genderized pairs both having genderized titles.  The following countries have their matching pairs both being genderized between 50 and 60% of the time: Finland, Serbia, Australia, Croatia, Kazakhstan, Slovakia, Chile, Colombia, Ecuador, Indonesia, and Peru.

Let me be clear: This looks yay! on some level, but it still sucks.  The number should be 100% of matching pairs either both including gender or neither article in a matched pair including gender.  Anything else is selective gendering of a national team article to the neutrality detriment of women.

Beyond country naming patterns, there is the sport naming patterns.  Nine sports genderized both men’s and women’s national team articles 100% of the time.  Those sports are Volleyball, Softball, Squash, Goalball, Lacrosse, Ice hockey, Water polo, Wheelchair basketball, and Australian rules.  This is absolutely fantastic, because the total articles involved are higher per sport than for most countries.  (Inline hockey is also high at 89% and floorball at 83%.  No other sports are above 31%.)  It also suggests the problem with systemic bias against women when it comes to articles titles is probably not entirely dependent on the nation but on the sport and its proponents involved in that sport’s Wikiproject.

But we also have the other side: Most sports have problems in that the selectively genderize women’s national team articles while choosing not to genderize men’s national teams articles.  The following sports have between 0% and 2% of their articles in that group: Handball, Soccer, Cricket, Rugby sevens, International rules, Baseball, Beach handball, Rugby league, Rugby union, Kabaddi, American football, and Bandy.  The remaining sports are field hockey at 30%, basketball at 20% and futsal at 14%.

And this is a problem because the sports that are violating Wikipedia’s neutrality policy by selectively genderizing one team over another to the benefit of promoting the men’s game through article title include 515 articles (total men not gendered, women gendered for sports where this represents 70% or more of the articles) compared to 175 for the first cohort (total men gendered, women gendered where this represents 83% or more of the article count).

None of the sports on the list of genderizing women but not genderizing men particularly surprise me.  These are sports where professionalism is dominated by men, some with high degrees of perceived violence or associated with male norms of masculinity.   By asking Wikipedia to enforce neutrality and stop selectively genderizing some articles, there is an implicit challenge to that male masculinity and male dominance in sport.

And that pattern appears unlikely to change, with Wikipedia selectively gendering those sports because, as I have been told before as a female editor, Wikipedia need not be be neutral and factual but should reflect the cultural norms in which it is written.

In the mean time, I ask that when you read a national team article, you look at the title and critically ask yourself about the gender found in the title.

The role of English Wikipedia’s top content creators in perpetuating gender bias

Clarification: Account and content creator are used interchangeably as some content creators are bots who are operated by human editors. Thus, the same content creator may appear more than once. The total number of human content creators is likely to be less than 5,000. article creators when compared to the rest of the community, 8 January 2014.svg/0,

Top article creators on Wikipedia when compared to the rest of the community, 8 January 2014. A graph by Ktr101.

According to Ktr101, the top 5,000 article creators on English Wikipedia have created 60% of all articles on the project.  The top 1,000 article creators account for 42% of all Wikipedia articles alone.

Wikipedia has a well known gendergap when it comes to articles about women.

Ktr101 made the connection between the two issues in their piece for the Signpost, saying:

” With the already low numbers of females on the site, this means that there will be more coverage of male-oriented topics. If an article is not covered immediately, there is a good chance that it will be created in the coming years. Unfortunately, this means that whatever female-oriented topics are out there will probably get further neglected, as there is less of a chance that someone will even know that the subject exists, never mind it being notable enough for an article (when in doubt, go for it). The amount of these super page creators only exacerbates the problem, as it means that the users who are mass-creating pages are probably not doing neglected topics, and this tilts our coverage disproportionately towards male-oriented topics.”

This does bring up the question: How bad is the genderap in terms of article creation by Wikipedia’s top content creators?  Are “super users” exacerbating the problem by overwhelming creating new articles at males and not creating large numbers of articles about women?

The easy answer to that question is to get the percentage breakdown by gender for of all of English Wikipedia’s top 5,000 editors.  This is easier said then done for a number of reasons.  The first is the ability to easily label articles as male, female and neutral.  Some of this will be inherently subjective.  Some of it might actually require content analysis, because an article about say “Netball in Jamaica” could have been primarily written by some one interested in the men’s game despite the sport being historically female.  In that case, the article could be turned on its head and simple female coding for female could be wrong.  If just doing it from a list, it requires a lot of knowledge about names and verifying gender facts.  Lindsay is one of those unisex names that can be male or female.  If a person is writing mostly about Australians, the name is probably going to be male.  If a person is writing about USAians, then it will probably be female.  Again, cultural knowledge or authentication by viewing the article is needed.  Then there is the purely subjective stuff: Should “Sex and the City” be female, should “Futurama” be male or should “West Wing” be gender neutral?  Should the Abbott Ministry be male because Tony Abbott is male and most of the ministry is male (and some policies are seen as anti-female) or should it be gender neutral because women are on it and a ministry is not inherently a sexed concept? Such coding is inherently problematic and makes potential replication very difficult, especially since we are not looking at a few articles but thousands of unique articles.  Any research realistically may not be replicable.

Despite that, the question is still worth answering and worth considering.  I wanted to do this, but given the time constraints because of some of the coding issues mentioned above, I was only able to examine the contributions of 20 of the top 5,000 contributors.  This sample size represents only 0.4% of all people on that list.  To give an idea as to the top 5,000 article creators on the list, the mode number of articles created was 101, the median was 108 and the average was 4,009.  Across all 5,000 article creators, this is not quite a match.  For the 5,000 the mode was 107, median was 205 and the average was 536.  The quartiles for the sampled population are 101, 108, 1315.5, and 40016.  For all 5,000 contributors, they are 135, 205, 400.5 and 94756.  In all, 80,196 articles were included in this sample.  Not an exact representative sample but for my purposes of trying to begin to understand patterns and hoping to encourage others to continue this research, it is good enough.

For my purposes, women’s articles are defined as biographies about women, articles about groups of women, things heavily featuring women, articles about fictional women, or articles that almost entirely discuss only women. Example: Hillary Clinton, Canberra Capitals, The Good Wife, Lisa Simpson, African American women in politics. The same applied for articles about men.  Neutral gender articles were articles that did not fit into these categories.

Using this criteria, 1412 articles were identified as female, 4595 were identified as male, and 74189 were identified as gender neutral.  On the face of it, woot woot.  Ignoring the gender-neutral articles, 23.5% of all articles were about women.  This certainly beats the estimated contributor genderap.  Except the data suggests this is factually no true in terms of “super users” creating articles about women. Of those in the sample, 5 people did not write an article that was gendered either way.  Four people wrote zero articles about women but did write articles about men.  That puts it at 45% of the sampled contributors not writing about women (and men), and of the people writing a gendered article, 26% of them not writing about women.   This is where a bigger sample size would probably come in handy, but it is still a bit depressing.

When looking at gendered article writers only for only their gendered content, only one contributor was at 50% of their articles being about women.  The next closest created 38% of their articles about women.  The third was at 25%.  The fourth most popular was 19% and the fifth was 13%.  That rounds out the top 25% of creators of content about women.  The remaining 75% (including our non-gendered writers) average 2.6% of their content about women.  The remaining 75% writing about gendered topics write 4.7% of their content about women.

English Wikipedia’s “super users” are not contributing much female content.  This is problematic on multiple levels.  The first is ROI.  A lot of money is currently being spent on encouraging new contributors to come to the project and write articles about women.  There are editathons and training sessions and wikistormings.  All of these cost in terms of volunteer hours and money.   Research shows that edit-a-thons are not actually very cost productive in terms of generating new content and developing a new cohort of users.  A lot of times, articles developed at these events get deleted or nominated for deletion within seconds of going live.  The return on investment is very high to create a cohort of new users to fix the gap.

That isn’t to say that women should not be recruited and should not be encouraged to add articles about women to Wikipedia.  They absolutely should. On some level, the more this editing is normalized, the better.

It just is not a cost and time effective solution to fixing the representation gap for women on Wikipedia.  The best option is to encourage the top 5,000 editors to create articles about women and to incentivize this group. The sheer volume of articles they have created indicates they have a good understanding of what makes a person or topic notable for the purposes of being eligible for an article.  They do not need to learn the interface because they probably mastered it on their way to creating these articles.  They have accumulated reputation that for a number of them makes their articles much less likely to be deleted.  The group is clearly passionate about Wikipedia, enough to create a large number of articles.  The costs to get them to switch over to creating content about women is probably much lower.

The second problem, once ROI is out of the way, is one Ktr101 alludes to: If top content creators continue with their current contribution patterns, the under representation of women is likely to get worse, not better.  If one assumes a new article creation rate of only 0.1% (including non-gendered) or 8.9% (excluding non-gendered) articles are about women, it means that the remaining non-“super users” who have only created 40% of the existing articles need to fill the gap. And existing research on Wikipedia editor recruitment and retention suggests this is just not a feasible solution.  Despite all the efforts to recruit and retain editors, it just isn’t happening.  More and more articles are being created by “super users” and there is no growth pattern that suggests this option of relying on new users is not feasible.

The third issue is relying almost exclusively on new contributors to create new content women as a way of offsetting the gender imbalance does nothing to address perception problems related to Wikipedia being male and cliquey. Using business jargon, Wikimedia Foundation provides a service: free knowledge for public consumption.  The service has stakeholders, a key group of which are the elite content creators.  The “super users” in this elite content creating group provide 60% of Wikipedia’s content.  They provide most of the material for public consumption for another one of Wikimedia’s key stakeholders which are colloquially known as readers.  In this area, the two groups of key Wikimedia stakeholders are actually acting counter to the goal of the Foundation because one group is actively not providing information that another wants.  Worse yet, because of behaviors by one group (or at least the perception of their behaviors), it hurts the ability of the Wikimedia Foundation to grow readers and to grow another stakeholder group, regular and new-contributors.  One of the ways to offset this gender imbalance that creates this perception problem and lack of information problem is to change not reader desires but the behavior of the super users who are perceived as “being” Wikipedia.  And after these super users create the articles about women, highlight them and talk up their work.

English Wikipedia’s top content creators play a role in perpetuating gender bias on the project, and steps should be taken to do more research on the project and to understand the implications of what this means in a broader gender gap perspective.