« Guild Churn by Server Type | Main | City Migration »
April 07, 2006
The Dreaded Level 60 Scraper Cap
When Eric and Nic first implemented the WoW census scraper early last year, one potential problem they foresaw was the 49 character limit to the return from the /who command. Because the scraper polled each race, class, and level combination separately, this limitation meant that there would be a growing problem with characters who are not scraped because there are more than 49 characters in that category during peak hours (i.e., level 60 Night-Elf Rogues).
As the WoW servers have matured, this issue has become more and more potentially problematic in terms of data quality and analytic validity. And this is an issue that many comments here at the blog have touched upon. To this end, we sought to estimate the percentage of characters that were not being scraped due to this cap. This would give us an idea of how bad the problem was.
To do this, we grabbed one Saturday from every month from a high population server from our logs (from July 2005 to January 2006). Since this was a high population server, we took this to be our worst case scenario. We then parsed the number of level 60s logged per race per class combination (during each snapshot) - thus 20 combinations for Horde and 20 combinations for Alliance.
One problem was that Blizzard started to have authentication problems after November of 2005, and this interefered with our census scrapers logging on during peak hours. To avoid analyzing data after this period, we chose the regular November 2005 data for this analysis.
Next, we scrolled through these parsed logs to find the snapshot with the most number of overloads on that November Saturday. We then proceeded to estimate the number of overloads in that snapshot (the worst case scenario). To estimate the number of characters that we were missing from that overload, we looked at the observed number of characters in a non-overloaded race/class combination and then referring to the overall ratio of race/class combinations (on WarcraftRealms census), we inferred how many characters there should be in the overloaded combinations. We did this separately for the Alliance and the Horde and then calculated the % missing due to overload.
For the Horde, at worst peak time in November, we missed 3% of level 60s. For the Alliance, at worst peak time in November, we missed 13% of level 60s.
So overall, even in the worst case scenario on a high population server in November 2005, we were only missing 13% of level 60s. For most of the day, no race/class combinations were overloaded.
Posted at April 7, 2006 12:26 PM
Trackback Pings
TrackBack URL for this entry:
http://blogs.parc.com/cgi-bin/mt-tb.cgi/77
Comments
You mentioned WarcraftRealms and I regularly use its Census+ mod, and I know it doesn't have a problem with overloaded /whos (at least it didn't seem like it when I left Verbose on to not see the spam in IF, heh):
/who 60 NE Rogue overloads it starts doing:
/who 60 NE Rogue Names A through F (for example)
/who 60 NE Rogue Names F through M, etc.
With that said, what's the reason you don't further sub-divide by name if a Level/Race/Class combo overloads?
Posted by: Isaac Cajina at April 9, 2006 10:58 AM
You could always add the zone to the scraper. Rare that more than 49 of a race, level, and class will be in the same zone at the same time.
With the notable exception, of course, of Hunters in Dire Maul.
Posted by: Sar Deliac at April 14, 2006 09:17 AM
Parsing by Guild Name could also help.
Posted by: Squiggle at July 3, 2006 08:03 AM
This problem can easily be fixed by including Zone in the /who queries. Fewer than 49 results will be retuned and there will be no need to guess. You will have more combinations to cycle through and lose some time resolution, but oh well. Real use of any raid content will take at least half an hour.
I have never seen 49 of the same race, class and level in the same zone.
Posted by: Ronoran at August 14, 2006 01:51 PM