Inside WordPress.org Theme Review Team: Money, Abuse and Inconsistent Leadership
This post is about the Theme Review Team (TRT) at WordPress.org, which I am (still) a small part of. I would like to address some weird things that have been going on lately in this team and what it looks like from a distance.
I might be burning some bridges with this post, but if someone needs to take “one for the team”, so be it.
The Basics of WordPress.org Theme Repository
With 4,860 of free themes (at the time of writing this), the official WordPress.org Theme Repository (short: “repository”) is the first thing that any WordPress adopter sees and is probably one of the main reasons why WordPress grew to what it is today.
You can sort the themes in the repository by 3 main filters: Featured, Popular, Latest.
Latest is pretty straightforward: the newest themes are at the top, the oldest themes are at the bottom. I believe that this page is not very useful to end-users, as a theme’s usefulness doesn’t depend on how new it is, 1 week old or 3 months old.
Featured is the starting page for the Themes Repository, this is what users see first.
Contrary to popular belief the themes are not screened or moderated in any way. The list of themes is completely random and displays 15 themes that have been updated at least once in the last 2 years.
Look at the random screenshot below and tell me if you think this is what a new WordPress user wants to see. Is this selection representative for WordPress and the community?
It is surprising to find out that the FIRST list of themes that ALL users of WordPress see is a random list of 15 themes, without any consideration to quality, popularity or purpose.
This list of themes is often bugged and will often display 0 themes or the most recent theme instead.
And last but not least is the Popular page. This is what I will be talking about in this post.
Currently the ranking algorithm is very simple and obvious. It mainly uses 2 data fields that are known for each theme:
- The date when the theme went live in the repository.
- The number of (estimated) Active Installs of the theme.
Once a theme is older than 2 weeks – it is automatically eligible to appear on the Popular list. The number of Active Installs is divided by the number of days/weeks since it went live.
How Are Active Installs Estimated?
An Active Install is counted for a website only if the theme is Activated on that website. It is not enough to just have it in the /wp-content/themes/ folder.
Whatever your theme is called, your WordPress website will regularly make calls to WordPress.org to check if an update for your theme name is available, even if you are using a custom theme downloaded from someplace else. This call is made only if your website gets visitors (even from search engine bots).
So if you have a WordPress website with the theme “Twenty Ten” activated, but you had 0 visitors yesterday – your website was not counted.
According to personal observations of this algorithm, if a theme manages to reach 1,000+ active installs in its first 2-3 weeks, it is guaranteed a spot somewhere in the Top 30-50 of the Popular page.
Here are some examples (as of 17/07/2017):
- Hestia theme went live on 19th of May 2017 and currently has 10,000+ active installs.
It is 5th on the Popular list. 10,000 / 9 weeks = 1,111 / week.
- Total theme went live on 4th of October 2016 and currently has 50,000+ active installs.
It is 8th on the Popular list. 50,000 / 41 weeks = 1,219 / week.
- OceanWP theme went live on 12th of June 2017 and currently has 5,000+ active installs.
It is 10th on the Popular list. 5,000 / 5 weeks = 1,000 / week.
- Shapely theme went live on 12th of February 2017 and currently has 20,000+ active installs.
It is 11th on the Popular list. 20,000 / 22 weeks = 909 / week.
- Twenty Thirteen theme went live on 1st of August 2013 and currently has 200,000+ active installs.
It is 12th on the Popular list. 200,000 / 206 weeks = 970 / week.
- Astra theme went live on 30th of May 2017 and currently has 3,000+ active installs.
It is 26th on the Popular list. 3,000 / 7 weeks = 428 / week.
According to these figures we can determine some raw estimates for the success of a theme on the Popular list. A Top 30 spot is easily guaranteed if your theme will be adopted by at least 400 websites / week in the first 2-8 weeks since going live.
Seems like a reasonable algorithm, right?
If your theme is able to be adopted by ~50 websites every day – it is a good indicator that the theme is useful. Now if you manage to convince ~100 websites every day to start using your theme – you will easily get into Top 15-20 of the Popular list.
The Concept of Name Policing
One of the main weaknesses of this ranking algorithm is that the “Active Installs” counter cares only about the theme’s name (slug).
If you have 3 completely different themes all called “Example”, then a theme with this name submitted to .org repository will count the active installs of all 3 themes. I think you know where this is going.
If a theme exists in .org repository and it shares the name with themes outside of the repository – it will gather active installs for all websites that are using themes with the same name. All the active installs will be counted and this hypothetical theme will rank higher, as technically it is more popular.
In order to patch this weakness, theme reviewers are advised to check for theme name collisions. Before a theme is approved in the .org repository, a reviewer should check if that theme name is already in use someplace else. So theme reviewers became the “Name Police”, a role that many reviewers don’t agree with and is being brought up every couple of months.
Theme Name Collision
Publicly the argument for name policing doesn’t refer to the “ranking weakness”.
The issue with themes sharing a common name can be a little more serious. For example at WPZOOM we have a travel theme called “Discovery”, released in early 2012.
In 2014 a theme with the same name was submitted to the .org repository. After a couple of updates the .org theme version got higher than the version we had for our own Discovery theme.
When our Discovery theme customers entered their Dashboard they saw a convenient “An update is available for your theme” notice. Of course most of them opted to automatically update, and our theme was completely replaced with the unrelated .org Discovery theme.
We started getting angry customer emails on a daily basis: “What kind of update is this?”, “I don’t like the new version”, “Where’s my theme?” etc.
It took us some time and resources to explain to our customers what happened and how WordPress theme updates work. In the end we had to patch things up in order to prevent this from happening again across all WPZOOM themes.
This “update issue” is the main official argument for name policing. Nowadays when new themes are uploaded to the .org repository, an automated check is done to see if this name is used outside of the repository.
This is what this automated check looks like:
Popular Themes Page – The Magnet for Abuse and Controversy
Unfortunately this ranking algorithm made it easy to abuse the system, which I will describe below. Due to the fact that admins have been habitually inconsistent in their decisions about theme name collisions, I decided to publicly speak out (and I was already punished for it).
Let’s start with a timeline of the events.
One of the first controversies was reported on 26th of October 2016 in the following WPTavern article: Controversy Surrounding WordPress.org “Popular” Themes Exposes Weaknesses in the Algorithm.
In short, here’s what happened in October 2016.
“Total” Theme Name Collision Breaks the Ice
~27 hours later the theme is suspended by @grapplerulrich (admin) because it was discovered that Total is one of the best-selling themes on ThemeForest, the largest marketplace for premium WordPress themes.
On the day of the discovery, Total theme had ~30,000+ active installs, mere hours after being accepted in the repository.
~6 hours later the author of the .org Total theme (@hasthemes) contested the theme’s suspension saying that the theme’s name corresponds to its purpose and that the collision with the ThemeForest theme is merely an accident.
This was enough to persuade the admins to reinstate the theme just 2 minutes and 30 seconds later.
At least 3 key reviewers were involved in this ticket and they decided that a name collision with 30,000+ live websites is not worth the time.
“Vertex” Theme Name Collision
The reviewer @sakinshrestha noticed a potential name collision with the popular Vertex theme by ElegantThemes, probably the most popular WordPress theme shop on the planet. He did the right thing and pinged @Otto42 (admin) to ask if it is OK to set the theme live or request a name change instead.
The theme’s author immediately offered to change the theme’s name so as not to cause any issues.
@Otto42 (admin) reacted with a simple recommendation to change the theme’s name. Not a request. Not a requirement. Just a recommendation.
An hour later the theme is set Live with the message “I got approval from other Key Reviewer.“. So another admin confirmed that the name collision argument is worthless.
On the next day the theme’s page already displayed 10,000+ active installs.
What Happened to Total and Vertex themes?
Both Total and Vertex themes jumped to the top of the Popular list.
Total theme jumped to #2 and stayed in Top 5 for at least the following 6 months.
Vertex theme jumped to #4. To confirm this, here’s a video made by Matt Medeiros (of Matt Report) about Vertex theme jumping to #4 spot.
Even today, ~9 months later, Total theme is on #8 and Vertex is on #66.
In these 9 months Total theme jumped from 30,000+ active installs to 50,000+ active installs and managed to stay in Top 10 for 9 months in a row. How many of these come from the theme on ThemeForest is unknown, most likely at least ~75% of them. Total theme on ThemeForest has 25,195 Sales at the moment of writing this.
Vertex fell down because it failed to increase its Active Installs, it stayed at the same 10,000+ active installs as on the day of being set live. How many of these come from ElegantThemes? Most likely at least 90% of them.
What Is the Benefit of Being on the Popular List?
Of course we cannot know how much free traffic was passed down to these 2 themes, but being in Top 5 for 6 months in a row should bring at least ~250-500 daily visitors to your website. One of my themes was in Top 50 for about 4 weeks and I was getting ~50-80 daily visitors just from it.
250 x 30 days = 7,500 free visitors a month. Considering that some of us are paying up to $1 per click in CPC campaigns, I wouldn’t mind this amount of free traffic.
What we can do though is look at Alexa ranking for the developers of these 2 themes.
As you can see the traffic for Total theme’s author immediately went up for 3 months in a row and practically stayed at the same level since.
Even though the graph for Vertex theme’s author is not as impressive, it is to be noted that jumping from rank 60k to 40k in Alexa is much more impressive than jumping from rank 600k to 400k.
How Much Money is a Popular Spot Worth?
It is well known that Zerif Lite, a theme that reached the top of the popular list (on its own merit), has been steadily bringing in over $50,000 of monthly revenue to the company behind it, ThemeIsle.
These figures were made public by ThemeIsle in their Transparency Reports.
Zerif Lite was suspended from the .org theme repository on October 6th 2016 for breaking multiple guidelines and in the following months they have been reporting a huge drop in their revenue.
Here’s what the ThemeIsle’s founder, Ionut Neagu, had to say in February 2017 in a WPTavern post:
“What was interesting is that revenue continued to decrease for all those months and we are now at around $45k/month instead of $120k.”
So it is evident that a spot in the Popular list can bring huge financial gains. Considering their drop from $120K/month to $45K/month, we are talking about half a million dollars a year at least (in this particular case).
Of course not all themes can generate that, but the potential is there. If you choose to monetize it or not is outside the scope of this discussion.
Another Popular Way to Game/Abuse the Ranking Algorithm
Since October 2016 brought a lot of discussion around theme ranking and name collisions, reviewers have been on the lookout to prevent any future cases like with Total or Vertex.
Unfortunately some theme developers discovered an easier loophole to abuse the Popular ranking algorithm. Here’s how it works.
- You create a WordPress theme and give it any name, the name itself is not relevant.
- Release the theme on your own website and promote it outside of the .org repository. Depending on how big and popular you are, you can promote the theme for a couple of months and gather as many active users as possible.
- After you feel that you’ve hit a critical mass of theme users – you upload it to the .org theme repository.
- Even if you get a “name collision” warning – it is disregarded, because the name collision is with your own theme – so it is not an issue.
- Once your theme is reviewed and goes live (another 2-4 months later), you will hopefully start with at least 1,000 – 2,000 active installs on day 1, guaranteeing you a spot in Top 30 for at least a couple of months.
This loophole is consistently abused by the bigger theme shops that have a large user base. Many of them easily start with 2,000 – 5,000+ active installs on the first day. You can go and check right now the Popular page. You will see at least 10 themes like that in Top 50.
Other theme developers use a different tactic. They upload a certain theme and a few weeks later request that the ticket be closed for one reason or another, without approving the theme. However their theme name is already reserved and no one else will be able to upload a theme with the same name.
Some months later they re-upload the theme with this name, after promoting it for a couple of months. Once the theme goes live – they have a higher active installs counter.
To prevent this loophole there is a rule in the guidelines: “You are not allowed to reserve theme names”, as in upload tickets just to keep a theme name. Multiple theme developers have been openly doing just that with zero consequences.
Inconsistent Leadership & Random Decisions
After the events of October 2016 not many name collisions were reported.
Things returned to normal, more or less.
Fast forward to Spring 2017.
In March 2017 a theme called Juliet is uploaded and goes live on 13th of May. Besides the fact that the theme developer reserved their theme name (using a tactic described above), there are 2 themes with the same name on ThemeForest.
The theme went live with 1,000+ active installs on release day without any discussions about name collision.
In April 2017 a theme called Astra is uploaded by an author with no other themes in the repository or anywhere else. It immediately starts showing 2,000+ active installs, even though the author wasn’t able to provide any proof that this counter corresponds to their theme.
Googling “Astra” returns multiple results for multiple themes called Astra, some of them still being actively updated.
This is how the Astra name collision issue was resolved on May 25th with just 5 words:
No other details were provided, no explanation as to why this was the decision.
The theme went live on May 30th 2017 with 2,000+ active installs on release day without any complaints from any of the admins (team leaders).
On May 25th 2017 a theme called Consulting is uploaded by one of the key theme reviewers as a form of protest against the recent lax attitude towards name collisions, and specifically because of Astra theme being accepted. This theme name is uploaded knowingly that there is a very popular theme on ThemeForest with the same name.
A great argument is brought up (again): if someone is allowed to upload themes with a known name collision, why can’t everyone do that?
After 6 days of waiting for admins to post a single comment about the ticket (3 of them being pinged directly), their silence is taken as lack of complaints.
@acosmin sets the theme live on May 31st 2017 with an Active Installs counter of 10,000+. After two weeks Consulting got into Top 5 of the Popular List where it would stay for at least 3-4 months.
It has been six weeks since then and none of the admins commented in the Consulting ticket. Not a single opinion was expressed.
Demotivation & Double Standards
On June 25th 2017 a theme called North is uploaded by @acosmin. The active installs counter was showing 7,000+ active installs for this theme name. There are 3 distinct themes on ThemeForest all called “North”.
After a whole week of waiting for admin confirmation that the theme is OK, on July 2nd 2017, @poena decided to close the ticket motivating that a theme with 2,000+ can be easily approved, while one with 7,000+ cannot. Nothing like this is anywhere in the guidelines.
On May 8th 2017 I have released a Food Blog theme called “Rosemary”. As I had another theme in the .org review queue, I wasn’t able to upload it to the repository. Finally I was clear to upload Rosemary to the repository on June 6th 2017, only 6 days after Consulting theme went live.
When I uploaded the theme I have discovered that this generic theme name shows 5,000+ active installs.
And here’s how the ticket went down:
My ticket was closed on July 14th 2017, the main argument being theme name collision.
After my ticket was closed I re-opened the tickets for Total and Consulting (myself being a moderator), linking to my closed ticket and asking that the themes be suspended as to avoid further collisions.
Admins reacted very quickly by suspending my moderator status and by keeping the themes online. Consulting was suspended for a couple of hours, but the theme was republished without any public explanation from anyone.
Inconsistent Leadership & Evasive Attitude Towards Theme Authors and Reviewers
None of the admins did anything to prevent these themes (Total, Vertex, Juliet, Astra, Consulting) from going live.
In all public communication in the #teamreview channel on Slack as well as in Trac tickets, the admins avoided to come with a public, clear statement regarding theme name collisions or take responsibility for any past mistakes.
It seems like decisions were coming from personal opinions, preferences and mood. It was (and still is) unclear as to who deserves the praise or the blame for all these decisions.
Are these unanimous team decisions?
Are these personal opinions of whichever admin gets involved in a ticket?
The Aftermath: Selective Punishments and Reactions
There’s really not much to say here. All above-mentioned themes are doing fine, except North and Rosemary, both of which were closed.
@acosmin has been demoted from moderator to reviewer status on July 9th or July 10th.
@acosmin, the man who reviewed 247 themes in the last 10 months, but couldn’t even get a normal, timely reply from an admin for his own ticket.
I have been demoted from moderator to reviewer status on July 14th. The official motive is that I have re-opened the trac tickets for the 2 troublesome themes (Total and Consulting).
Here’s the official explanation for my demotion (sent to me in private). I’ve shortened it a little bit. I’m not going to mention who sent it to me, because that’s not really important, as long as it was a team decision. No need to single out the messenger.
“This is just a note that you’ve been demoted to “reviewer” from “moderator” with the TRT.
This decision was not taken lightly. You reopened tickets that you shouldn’t have been reopening. There has to be some sort of consequence for that.
On a related note, we all want to solve the active installs issue. I don’t know how we’ll solve it. But, I do know that having our team moderators knowingly and openly challenging the team leads is not the way to get to where we need to be. We have to find a way to fix things together.”
What Other Theme Reviewers Think?
It is important to note that there are many disagreements among team leads and some moderators.
Here are some quotes from my recent conversations with fellow reviewers.
“I do not think name collisions should be allowed, however as usual the consistency of treatment is abhorrent…”
“Part of the problem is that the foundation, rules, requirements promote inconsistencies.
Also there is the Otto factor. There are TRT rules/requirements, then there are the Otto rules and requirements.”
“I also am really pissed about demoting @acosmin.
Let me ask you. How can the TRT leads demotivate one of the top reviewers? You would think he deserved better treatment than that.”
“I think if names are being policed, then this should apply to everyone, without exceptions.”
“TRT is lacking real leadership. There should be clear rules, no exceptions and punishment if authors intentionally break rules.”
“That’s really unbelievable. The behavior is a no-go… They just closed your ticket without answering the question. I don’t know what’s going on here, but this isn’t right. They just seem to make rules based on their personal opinion, which may be biased. It’s scandalous…”
What to Expect in The Future?
I don’t expect much to change, besides the fact certain team members will be treated worse and with more arrogance.
Unless you are in good standing with whoever is at the top of the chain, your voice is not that important.
Sponsoring WordPress events can also help you with better treatment.
Right when I was writing this post an announcement went up on the Make.WordPress.org blog.
Here is the part that is relevant to this post:
There should be only one team lead who makes decisions.
Being a team lead is a lot of responsibility and the regular conflicts and long discussions are draining. The conclusion that we came to that the only way it was going to work with having a single team lead is that if they are supported by the theme mods. The theme lead should be able to delegate responsibilities and decision making. If a decision is needed from the team lead then it should be escalated through the theme mods.
So basically, if until now there were 4-5 people that would drop by with an opinion, now we are going to get only one decision maker.
Is this good news for the future of TRT?
Is this good news for the quality and consistency of the theme repository?
Having our team moderators knowingly and openly challenging the team leads is not the way to get to where we need to be.
A Crazy Theory For the Conspiracy Theorists Out There
Changes in the .org theme directory are done at a snail’s pace for no apparent reason. There are simple meta tickets that remain open for years on end.
I have joined TRT a little more than a year ago and there already were talks about new theme tags.
Seems pretty straightforward, considering that a couple of meetings were dedicated to this very topic.
The default Twenty themes are competing on the Popular tab just like all other themes. Currently 6 out of top 10 Popular themes are the default Twenty themes. Should they be considered for the Popular page?
And Here’s The Crazy Theory
TAKE THIS WITH A GRAIN OF SALT.
Jetpack is rapidly introducing more and more paid features to their paying customers. With the recent announcement of Jetpack becoming sort of a theme distribution platform for curated themes, I won’t be surprised if the official .org theme directory will be left as outdated as possible on purpose, so that there is a “better place” to find themes.
So if .org users start getting annoyed at the outdated official theme repository, with all the drawbacks and outdated filters, they might start considering a different channel. And who is there to capture all those users? Automattic with Jetpack of course :)
Why would Automattic be interested in .org providing free, high quality and modern themes in a competitive way, when for only $300 / year you can get professional themes via Jetpack?
But that’s not a rabbit hole that I want to go down into right now…
Even though this post might have sounded very critical towards certain individuals, the individuals themselves are not the subject of this post, it is their actions.
I have nothing against any of the developers of the above-mentioned themes, as everyone is doing their own thing. If the system bends under their weight – it is not their fault.
This just isn’t right.