WikiApiary talk:Operations/2013/March

API Redirects
Had a problem in segment 8 with Wikimedia Labs. The site was issuing a redirect to Wikitech and messing up the API request in the process, resulting in a return value that was causing User:Bumble Bee to throw an exception. I've deactivated Wikimedia Labs since Wikitech already existed. I've also made a note that User:Bumble Bee should not follow redirects. I'll add some code so that the bot issues a warning if it is given a redirect, and skips the site. Thingles (talk) 15:10, 1 March 2013 (UTC)

Audit Bee activated
User:Audit Bee was activated this morning and is auditing 10 websites at the beginning of each hour. We'll see how this works, deal with any bugs, and depending on how it goes may increase the rate of audit. Thingles (talk) 15:51, 2 March 2013 (UTC)


 * Early runs look fine, audit bee seems to be doing a good job. I'm increasing the audits to 10 sites every 30 minutes. Thingles (talk) 19:24, 2 March 2013 (UTC)


 * Things seem to be going okay and I'm not super patient so I've updated it to run every 15 minutes. That seems like a fine state to work through the thousands of sites in the backlog. Bumble Bee instances don't get overly backed up and it staggers sites on different intervals for updating. Thingles (talk) 02:38, 3 March 2013 (UTC)

Some sites activated by audit bee wrongly
I just fixed a bug in audit bee diff that was causing it to activate sites that it failed to audit. This is now resolved. Expect some errors to show up in Bot log for sites that were activated. You can tell that this was the case because audit bee activated and validated them, but did not set them as audited. Thingles (talk) 17:12, 3 March 2013 (UTC)

More than 1M property values set
Today WikiAPIary exeeded the 1,000,000 property values mark and is now one of the 10 biggest known Semantic MediaWiki installations. --&#91;&#91;kgh&#93;&#93; (talk) 23:25, 7 March 2013 (UTC)



New status: Defunct
There is a new flag for websites, Property:Is defunct. This flag is used to indicate that a site should not be checked by User:Audit Bee (or frankly any robots). The Web site is probably no longer available. It may also be used to indicate that the API is defunct, where a working wiki is not allowing API access use defunct to get rid of the errors in Bot log from attempting to connect to it. Thingles (talk) 20:42, 9 March 2013 (UTC)

First step to new graphs
I took the first step to getting the new graphs online. My plan is that websites will have three graphs displayed, and the user can control the defaults for those three slots as well as change them and create a new window to leave a graph up on the screen. To do this, I've wired Widget:Website graphs to use the Javascript code that is in the WikiApiary code repo. I also added the PHP, HTML and JS stuff into  in that repo (previously this only had the Python robots in it). If you're reading this and feeling like making the PHP and JS better, your help would be greatly appreciated.

A nice improvement from this is that the graphs on the Web site page resize with your browser. Finally. :-) Thingles (talk) 20:45, 9 March 2013 (UTC)

Initial audit complete!
All sites have been audited (or marked defunct)!

Completed audit 0 sites 0 succeeded 0 failed

Audit Bee will continue to run every 30 minutes to conduct new audits (Concept:Websites never audited) as well as refresh expired audits (Concept:Websites expired audit). I modified the script though so that it will make no entries in Bot log if there are no audits performed (previously it reported it did 0 audits, as shown above). Thingles (talk) 13:02, 10 March 2013 (UTC)

Tried Semantic Drilldown
You may notice I did some activity in the Filter namespace trying to get Extension:Semantic Drilldown working. I added two filters (Filter:Defunct and Filter:Bot segment) and even set values for them, but it seems that Drilldown still queries the database and every property to build the user interface? I'm not sure, it just times out now. I'm actually not even sure that Drilldown works with SQLStore3 and the newest Extension:Semantic MediaWiki so it may all be a moot point. It would be a nice way, particularly for operators, to slice and dice the wikis. I'm going to leave it be for now. Thingles (talk) 14:01, 10 March 2013 (UTC)

DST Rippling through
This is interesting. Now that DST has flipped, we are seeing timeoffset values all getting updated. example Thingles (talk) 03:19, 11 March 2013 (UTC)

Bumble Bee Managing Error State Completely
User:Bumble Bee is now properly updating Property:In error when he detects a website is having issues. He will also increment that error status on subsequent failures, and finally clear the error when the site is responding again. You will see messages in Special:RecentChanges like:


 * recording error (example)
 * incrementing error count to 5 (example)
 * clearing error (example)

The intended behavior is that when Bumble Bee fails to get a response the first time, he will set Property:In error to true, he will also set Property:Has error count to 1, Property:Has error message will be set to the error message and Property:Has error date will be set to the current time in UTC.

On each repeated attempt to talk to that site that generates an error Bumble Bee will update Property:Has error count and Property:Has error message. He will not update Property:Has error date. That date is intended to be when the error event started.

When (or if) Bumble Bee does successfully reach the site on another attempt, it will set Property:In error to false, but will leave the other fields as they are so that operators can see that information as they wish. But the flag for being in error state is false. If this cycle repeats, the fields will be reset for the new error.

Or at least that is how it is supposed to behave. If you see otherwise, please let me know. The code to do this is in this commit.

This is a pretty huge milestone. Now User:Audit Bee is doing all auditing and activating, and User:Bumble Bee is indicating errors and clearing them. The only remaining step is for User:Audit Bee to deactivate a site that has been in error for too long (2 weeks?).

Thingles (talk) 02:05, 12 March 2013 (UTC)


 * After the excitement of a nasty bug that put Bumble Bee in a total tailspin it's cool to see that the sites in error count has dropped from 209 to 104 from the clearing of error status when contact is re-established! Thingles (talk) 04:34, 12 March 2013 (UTC)


 * 84 now. Thingles (talk) 05:30, 12 March 2013 (UTC)


 * Well, this is just fantastic, it means that we can feed the bees almost unlimitedly and increase the honey production at will! The human work will still be manageable.
 * I think it's time to link this from the extension pages on mediawiki.org; if the 504 errors below are temporary and we don't risk killing the site, I'll do it shortly. --Nemo 06:46, 12 March 2013 (UTC)

Skin collection disabled
I've temporarily disabled skin collection due to the randomized order causing unnecessary revisions issue. I'll bring it back online when I've implemented sorting to insure that we wont get thousands of silly edits. Thingles (talk) 03:43, 12 March 2013 (UTC)

Fixed major bug
Wow. If you were just trying to use the site in the last hour it was terrible. I introduced a bug in my recent bot work that caused Bumble Bee to no longer update the timestamp that it last checked a Web site for stats. After a while it was trying to update nearly all websites every time it ran! Bad! This commit fixes the problem. Whew! Will take a few runs to normalize but will return to good behavior shortly. Thingles (talk) 04:16, 12 March 2013 (UTC)
 * I got an error 504 on a page, are things still normalising? --Nemo 06:39, 12 March 2013 (UTC)
 * It took a couple of hours for things to normalize. It's possible that was still while you were looking at it. Here is what happened to CPU during that period. Note graph time is US/Central. Thingles (talk) 14:57, 12 March 2013 (UTC)