User talk

Difference between revisions of "Thingles"

From WikiApiary, monitoring the MediaWiki universe

(Not semantic?: new section)
Line 151: Line 151:
== Not semantic? ==
== Not semantic? ==
This wiki is semantic: [[OpenEI#tab=Usage]]
This wiki is semantic: [[OpenEI#tab=Usage]] --[[User:Jeroen De Dauw|Jeroen De Dauw]] ([[User talk:Jeroen De Dauw|talk]]) 19:14, 21 March 2013 (UTC)

Revision as of 14:14, 21 March 2013

Archives: ... through January 2013.

Re: Extension version grid

I agree, this extension grid is very pretty and can come of great use for the extensions which at least use versioning! All this "infrastructure" seems a bit wasted by the extension data not being pulled from the 4700 not validated wikis... I'd be so curious to see it ten times more full! --Nemo 09:06, 1 February 2013 (UTC)

Look forward to connecting

Hi Jamie, listened to a podcast, ready for more. Would very much like to connect for a few minutes via voice. Best, MarkDilley (talk) (of WikiIndex)

Hey there Mark! Would be fun to connect. Not sure where you are, timezones and all. You can find me on Skype under jthingelstad. You can also find me on Freenode in </code>#mediawiki</code> as thingles pretty frequently. I'll shoot you my email as well via the wiki. Thingles (talk) 20:52, 3 February 2013 (UTC)

MediaWiki versions

Heiya Jamie, I just managed to remove the MediaWiki from the version picked up by Bumble Bee. However this is only a quarter of the bet and MediaWiki does not use "-" to label special versions, alphas, betas or rcs. I guess some Regex is needed to get things going. What we need is 1.21 which collects 1.21wmf, 1.21alpha and 1.21rc (all of these may be regarded as non-production and fit into one count) and 1.21.0, etc. Hmm, this is something for Master T, I guess :). Can you figure out a Template:Get simple version number for this? Cheers --[[kgh]] (talk) 18:41, 7 February 2013 (UTC)

This is exactly what I'm working on in User:Thingles/Scratch4. I'm referring to those labels as "flags". As you can see, it mostly works. My biggest issue is that for some reason "2.2" is parsing into the first and fourth variable, rather than the 1st and 2nd. I plan on this regex method replacing Template:Get simple version number. I'll put this further up my list. The "flag" versions are bothersome and skew things a lot. Thingles (talk) 19:38, 7 February 2013 (UTC)
Hey, that's cool. I knew it was somewhere in the pipeline. :) These "flags" are indeed a pain in the neck. :( I was asking because we have that many different versions in the meantime that the statistics page gets pretty crowded. There we should just display the branches like 1.20.x etc. and show the details on individual (sub-)pages. I believe the statistics page will gain from this a lot (less is more). The really interested people may proceed to the exact breakdowns. Cheers --[[kgh]] (talk) 19:47, 7 February 2013 (UTC)

Skins on Wikia

Heiya Jamie, the API implementation for skins seems to cause problems for skins on Wikia or Wikia does some strange things. Bumble Bee constantly rearranges the order of the skins according to what it picked up. This creates a new revision every day though the selection of skins stays the same. So far I have not seen this for skin data retrievals on other wikis. Cheers --[[kgh]] (talk) 12:09, 13 February 2013 (UTC) PS I have just seen this on another non-Wikia wiki there it happens. Probably the API then. I wonder why this does not happen in every case. --[[kgh]] (talk) 16:40, 14 February 2013 (UTC)

Thanks for highlighting this. I think I'll have to manually sort skins before I put them into the wiki. I'm not going to deal with this for a while. It's causing unneeded edits but otherwise no harm. Thingles (talk) 02:58, 26 February 2013 (UTC)
It is just that all these revisions enlarge the database with not additional benefit at all. Indeed, not a top important issue. --[[kgh]] (talk) 01:01, 27 February 2013 (UTC)
This seems to happen as soon as code updates for existing skins or extensions get moved onto the server. Since Wikia is probably doing it every day ... The sorting will be the best way to solve this. Cheers --[[kgh]] (talk) 22:48, 9 March 2013 (UTC)


Heiya Jamie, I think it will be useful to set the "Extension:", "Farm:" and "Skin:" namespace to true. Cheers --[[kgh]] (talk) 21:22, 13 February 2013 (UTC)

Brilliant! Done! Thingles (talk) 16:22, 14 February 2013 (UTC)
Perhaps not brillant, but helpful. :) Thank you. --[[kgh]] (talk) 16:38, 14 February 2013 (UTC)

Re: Updates!

Great! I hope to see all the Pavlo's sites updated with data by the bot soon: they're such a tiny part of the wiki world, we need to move on. ;-) --Nemo 09:18, 25 February 2013 (UTC)

Concepts within concepts

Heiya Jamie, you might like bug 44467. :) --[[kgh]] (talk) 00:09, 26 February 2013 (UTC)

 :-) Was bummed when that didn't work when I tried it. I added a vote on that bug! Thanks! Thingles (talk) 02:53, 26 February 2013 (UTC)

Collection frequency for wikis < 1.11.0

I guess a daily run (1440) is more than sufficient for these wikis since the collectable general information for them is stable. Probably a week (10080) would do the job too. What do you think? --[[kgh]] (talk) 00:36, 26 February 2013 (UTC) PS What about adding version specific categories to the website pages, e.g. Category:1.9.0 for all wikis of version 1.9.0. This would make text replacing much easier. Doing an aggregated Category:1.9.x in a second step will be nice too. Probably to much voodoo to add the aggregated category right away instead of version specific ones. --[[kgh]] (talk) 00:36, 26 February 2013 (UTC)

Something feels wrong about adding categories for semantic data. :-) But, I might be being dumb. It would be nice for certain queries for sure, and definitely Replace Text. However, User:Audit Bee is now doing most of what we would do version dependent replace text on. You feel strongly about this? Thingles (talk) 19:44, 2 March 2013 (UTC)
I actually described a dirty voodoo workaround since ReplaceText only allows to narrow down results by namespace or category. However I strongly feel positive about reducing the number of visits to wikis with versions lower than MW 1.11.0. If there would be a Bee (e.g. Dirty work Bee ;) which is checking for the wikis in question and changes the update frequency this would definitely be the better solution. --[[kgh]] (talk) 22:45, 2 March 2013 (UTC)

Vote -> boom

I clicked the vote button for Wikidata and stuff went boom, see the history there :) --Jeroen De Dauw (talk) 00:45, 2 March 2013 (UTC)

Thanks for letting me know Jeroen! This is nasty behavior in Semantic Forms when the |restricted=operators flag is being used, seemingly in conjunction with a |default=No, and probably only when intersecting with an autoedit call. Phew! You are not in the operator group (no offense. ;-P) but the autoedit seems to still allow you to do the edit (note your vote change did occur), but the other fields that were set to restricted reverted back to their defaults. Bummer. For now, I've removed the restricted parameters. They've caused a handful of challenges and I don't think they are really necessary (at least for now). Thanks for checking in on WikiApiary! I just enabled another bot to audit and activate sites automatically so the data set here is going to get way more robust in the coming days! (PS, I've filed bug 45644 on this issue.) Thingles (talk) 19:42, 2 March 2013 (UTC)

System change

Heiya Jamie,

I was very bold an bought myself an new computer. This resulted into a new OS and the transition turned out to be a very painful process. More than expected. Basically this is the reason why I was not very active here lately.

Cheers --[[kgh]] (talk) 18:58, 6 March 2013 (UTC)

Very cool! What did you get yourself? What new OS? BTW, you picked a good time to be less active. Audit Bee is slamming the site and load on my server is 4x higher than normal. Audit Bee should be done by Thursday or so, and then things will return to normal. The very first audits with page creations have a higher load. See WikiApiary page graph. :-) Thingles (talk) 20:11, 6 March 2013 (UTC)
Hmm, very cool...? Well, this is a matter of taste. I switched from NT 5.1 to NT 6.2, but since I was using the Win 95 GUI for 18 years now, even on NT 5.1, this is kinda big change for me. Still I will probably install a Squeeze soon since I bought a 2 TB harddrive and now have plenty of space. As an old and quaint habit I still compress most of my files.
Yeah, Audit and Bumble Bee are very hard working bees right now and it is indeed getting a bit frantic on your server. :) I am surprised that there are that few problems apart from the JSON decoding. --[[kgh]] (talk) 22:39, 6 March 2013 (UTC)

Website form broken?

Just tried adding a wiki using the Website form. On submission, I just got a blank page, and no actual save operation occured. --Jeroen De Dauw (talk) 01:41, 7 March 2013 (UTC)

Damn! Sorry about that Jeroen. I pulled SyntaxHighlight GeSHi from master today and it looks like there was a change that made it dependent on 1.21.x. I see my error log had
PHP message: PHP Notice:  Use of undefined constant CONTENT_MODEL_JAVASCRIPT - assumed 'CONTENT_MODEL_JAVASCRIPT'
in /srv/www/mediawiki/public_html/w/extensions/SyntaxHighlight_GeSHi/SyntaxHighlight_GeSHi.php on line 76
…over and over. It looks like it was also causing edits via the API to fail which means auditing wasn't working either. Thanks for letting me know. I've switched that extension to it's REL1_20 branch now which hopefully will keep things happier. Thingles (talk) 02:18, 7 March 2013 (UTC)
Yeah, looks like this commit broke me. Thingles (talk) 02:35, 7 March 2013 (UTC)
Ugh... looks like my bots triggered this error 39,627 times today. :-( Thingles (talk) 02:20, 7 March 2013 (UTC)
Odd, it looks like API autoedit's worked when this problem was present. Thingles (talk) 02:27, 7 March 2013 (UTC)

The bees really have done their job well

It is great to see that these two workaholic bees have done such a good job. :) Which lot is next? ;) I have posted here about the great changes to the display you did today. There are also a couple of suggestions. Cheers --[[kgh]] (talk) 22:52, 9 March 2013 (UTC)

The first audit of every site is done! The next step for automation is to get the error fields being used properly. Right now none of the bots set it, so soon I will have then incrementing those fields and then figure out when Audit Bee should mark them as defunct. Need to add a function to to do this. The other item that is high on my priority list is to start creating the daily and weekly summary data in the database and start allowing those to be charted. I've got an idea for a very cool set of graphs using the daily and weekly data that just won't work with the raw data as it is collected now. I also need to add an audit bee process that prunes things like the Bot log which will now grow forever. That's pretty easy though. The graphs need a ton of help, but they need it all in Javascript which isn't a terribly strong point for me. Might see if I can get someone to help out on that. Oh, and the data PHP scripts too. :-) Thingles (talk) 13:19, 10 March 2013 (UTC)
True, automated error handling should really be the next step now that all the wikis are on board and active. This alone will at least reduce the error log and reduce the work operators will have to do manually. It is so sad that I do not know JavaScript since I would have loved to help here. My JavaScript is basically as good as your Swahili. :( --[[kgh]] (talk) 22:58, 10 March 2013 (UTC)
First step in this direction finished tonight. See this diff and here is the commit with the functionality added. Will watch this for a while. Next step will be to clear the error flag (right now it won't reset). Then would be having Audit Bee start deactivating sites that are over a certain error count. Thingles (talk) 03:16, 11 March 2013 (UTC)
Almost forgot a cool thing I'm adding to Bumble Bee too. I figured out that I could sort of cache the response for things like skins by storing a SHA-256 hash of the result the remote wiki gives me. So, I'm going to add this to Bumble Bee so that he will stop barraging WikiApiary with nochange edits for things like extensions and general info. Instead he'll do a SHA-256 on the API result from the target wiki and will only attempt a save to WikiApiary when the hash doesn't match. Nifty! I'm also changing the refresh rate for those data sets. Right now they all check every 24 hours. I'm going to have general update every 3 days, extensions daily and skins every week. I'm also going to add a random offset to it to make sure that the segments don't general hot spots. Thingles (talk) 14:45, 10 March 2013 (UTC)
Once again a goodie. :) New revisions for no cause are really not a good thing to have. The hash approach is as nice as it is nifty I guess. :) Support from me for the change in update frequency though you could even change the skin updates to a fortnight. This would elegantly solve the < 1.11 "issue". --[[kgh]] (talk) 22:58, 10 March 2013 (UTC)

Swap jobqueue with edits

ATM the "job queue" graph spans the whole page while edits only half of it. Why not swap them? Seems like edits is more interesting to most people. in fact, I guess a lot of people do not even know what the job queue is all about :) --Jeroen De Dauw (talk) 16:48, 13 March 2013 (UTC)

Yeah, definitely. You are seeing a bit of work in progress there. The plan is to just have those three larger charts, and have a control to allow the user to switch what data they wish to view in those. Included in that is the link to popup those charts into a separate window (instant dashboard!). You are right... I doubt many care or even know about their job queues. Thingles (talk) 02:34, 14 March 2013 (UTC)
Just swapped edits in. See WikiApiary. :-) Thingles (talk) 02:42, 14 March 2013 (UTC)
I care! While it shouldn't be too visible, it's very useful. If you made a graph also for each farm, including all Wikimedia projects, I'd be so happy! You'd fix [1]. --Nemo 16:33, 14 March 2013 (UTC)


Could you please whitelist Is a good site, honest. Alex Mashin (talk) 06:46, 16 March 2013 (UTC)

Sorry about that. I was using the lists from meta for Extension:SpamBlacklist and it's definitely a bit aggressive. is activated and will start collecting very soon. Thingles (talk) 13:21, 16 March 2013 (UTC)


Are the bookmarks at collected manually or automatically? (impressive job, by the way).
Alex Mashin (talk) 20:20, 16 March 2013 (UTC)

Thanks Alex! They are entered manually (although semi-automated using a bookmarklet). I previously used, and before that I had thought that Semantic MediaWiki could support a nice link wiki so I made one and then wrote an importer to pull all my previous bookmarks over. It even has a bot that updates URL redirects and checks the health of links. I have a bot that will replicate that wiki to others (see my friend User:Garrickvanburen's one at Thingles (talk) 22:42, 16 March 2013 (UTC)

Installed version/latest version in Extensions section

Hi Jamie,

when I look at Special:Version of wikis I often wonder what is the latest version of the installed extensions. I think it would be nice to have that on WikiApiary, meaning on the Extension section/tab of each wiki's page not only display the installed version of each extension but also what would be the latest version of each extension. --Eric (talk) 12:59, 18 March 2013 (UTC)

Thanks for the suggestion Eric. This is definitely in my sites. It may first happen as an email for site administrators. If you have a Web site on WikiApiary that you would want that for edit it with form and add a notification for yourself. Also, you can click through to the Extension pages themselves and it will often have the version info there. But it would be much better to put it right in the Web site page. I'm just wary that it will invoke a query for every extension that the site has. Probably fine... 🐝 thingles (talk) 02:11, 19 March 2013 (UTC)


Is it practical to get Bumble Bee to automatically upload and add logos when they're missing? At least if they're in a usual place. Adding logos takes significantly more time than the rest of the add wiki process, even when they're easily collected up like I had with the NIWA wikis.

Also, about "Allow wikis to use WikiApiary to do weekly backups. (Likely a paid service.)", if space becomes not much of an issue would you consider doing an annual XML text backup for free for non-huge wikis, but not retrievable without a one off charge (used to fund more backups)? Many wiki admins don't take care to backup, and as a result wikis are lost. And have a list of wikis which have gone down that you've got a backup off available. That'd be awesome, practically ensuring that no wiki is ever entirely lost.

And did you see my suggestion on WikiApiary:MediaWiki_Extension? Is that likely to be practical long term?--Ete (talk) 17:10, 20 March 2013 (UTC)

Not semantic?

This wiki is semantic: OpenEI#tab=Usage --Jeroen De Dauw (talk) 19:14, 21 March 2013 (UTC)