User talk:Thingles

Archives: ... through January 2013.

Re: Extension version grid
I agree, this extension grid is very pretty and can come of great use for the extensions which at least use versioning! All this "infrastructure" seems a bit wasted by the extension data not being pulled from the 4700 not validated wikis... I'd be so curious to see it ten times more full! --Nemo 09:06, 1 February 2013 (UTC)

Look forward to connecting
Hi Jamie, listened to a podcast, ready for more. Would very much like to connect for a few minutes via voice. Best, MarkDilley (talk) (of WikiIndex)


 * Hey there Mark! Would be fun to connect. Not sure where you are, timezones and all. You can find me on Skype under . You can also find me on Freenode in #mediawiki as   pretty frequently. I'll shoot you my email as well via the wiki. Thingles (talk) 20:52, 3 February 2013 (UTC)

MediaWiki versions
Heiya Jamie, I just managed to remove the MediaWiki from the version picked up by Bumble Bee. However this is only a quarter of the bet and MediaWiki does not use "-" to label special versions, alphas, betas or rcs. I guess some Regex is needed to get things going. What we need is 1.21 which collects 1.21wmf, 1.21alpha and 1.21rc (all of these may be regarded as non-production and fit into one count) and 1.21.0, etc. Hmm, this is something for Master T, I guess :). Can you figure out a Template:Get simple version number for this? Cheers --&#91;&#91;kgh&#93;&#93; (talk) 18:41, 7 February 2013 (UTC)


 * This is exactly what I'm working on in User:Thingles/Scratch4. I'm referring to those labels as "flags". As you can see, it mostly works. My biggest issue is that for some reason "2.2" is parsing into the first and fourth variable, rather than the 1st and 2nd. I plan on this regex method replacing Template:Get simple version number. I'll put this further up my list. The "flag" versions are bothersome and skew things a lot. Thingles (talk) 19:38, 7 February 2013 (UTC)


 * Hey, that's cool. I knew it was somewhere in the pipeline. :) These "flags" are indeed a pain in the neck. :( I was asking because we have that many different versions in the meantime that the statistics page gets pretty crowded. There we should just display the branches like 1.20.x etc. and show the details on individual (sub-)pages. I believe the statistics page will gain from this a lot (less is more). The really interested people may proceed to the exact breakdowns. Cheers --&#91;&#91;kgh&#93;&#93; (talk) 19:47, 7 February 2013 (UTC)

Skins on Wikia
Heiya Jamie, the API implementation for skins seems to cause problems for skins on Wikia or Wikia does some strange things. Bumble Bee constantly rearranges the order of the skins according to what it picked up. This creates a new revision every day though the selection of skins stays the same. So far I have not seen this for skin data retrievals on other wikis. Cheers --&#91;&#91;kgh&#93;&#93; (talk) 12:09, 13 February 2013 (UTC) PS I have just seen this on another non-Wikia wiki there it happens. Probably the API then. I wonder why this does not happen in every case. --&#91;&#91;kgh&#93;&#93; (talk) 16:40, 14 February 2013 (UTC)


 * Thanks for highlighting this. I think I'll have to manually sort skins before I put them into the wiki. I'm not going to deal with this for a while. It's causing unneeded edits but otherwise no harm. Thingles (talk) 02:58, 26 February 2013 (UTC)


 * It is just that all these revisions enlarge the database with not additional benefit at all. Indeed, not a top important issue. --&#91;&#91;kgh&#93;&#93; (talk) 01:01, 27 February 2013 (UTC)


 * This seems to happen as soon as code updates for existing skins or extensions get moved onto the server. Since Wikia is probably doing it every day ... The sorting will be the best way to solve this. Cheers --&#91;&#91;kgh&#93;&#93; (talk) 22:48, 9 March 2013 (UTC)

$wgNamespacesToBeSearchedDefault
Heiya Jamie, I think it will be useful to set the "Extension:", "Farm:" and "Skin:" namespace to true. Cheers --&#91;&#91;kgh&#93;&#93; (talk) 21:22, 13 February 2013 (UTC)


 * Brilliant! Done! Thingles (talk) 16:22, 14 February 2013 (UTC)


 * Perhaps not brillant, but helpful. :) Thank you. --&#91;&#91;kgh&#93;&#93; (talk) 16:38, 14 February 2013 (UTC)

Re: Updates!
Great! I hope to see all the Pavlo's sites updated with data by the bot soon: they're such a tiny part of the wiki world, we need to move on. ;-) --Nemo 09:18, 25 February 2013 (UTC)

Concepts within concepts
Heiya Jamie, you might like bug 44467. :) --&#91;&#91;kgh&#93;&#93; (talk) 00:09, 26 February 2013 (UTC)


 * :-) Was bummed when that didn't work when I tried it. I added a vote on that bug! Thanks! Thingles (talk) 02:53, 26 February 2013 (UTC)

Collection frequency for wikis < 1.11.0
I guess a daily run (1440) is more than sufficient for these wikis since the collectable general information for them is stable. Probably a week (10080) would do the job too. What do you think? --&#91;&#91;kgh&#93;&#93; (talk) 00:36, 26 February 2013 (UTC) PS What about adding version specific categories to the website pages, e.g. Category:1.9.0 for all wikis of version 1.9.0. This would make text replacing much easier. Doing an aggregated Category:1.9.x in a second step will be nice too. Probably to much voodoo to add the aggregated category right away instead of version specific ones. --&#91;&#91;kgh&#93;&#93; (talk) 00:36, 26 February 2013 (UTC)


 * Something feels wrong about adding categories for semantic data. :-) But, I might be being dumb. It would be nice for certain queries for sure, and definitely Replace Text. However, User:Audit Bee is now doing most of what we would do version dependent replace text on. You feel strongly about this? Thingles (talk) 19:44, 2 March 2013 (UTC)


 * I actually described a dirty voodoo workaround since ReplaceText only allows to narrow down results by namespace or category. However I strongly feel positive about reducing the number of visits to wikis with versions lower than MW 1.11.0. If there would be a Bee (e.g. Dirty work Bee ;) which is checking for the wikis in question and changes the update frequency this would definitely be the better solution. --&#91;&#91;kgh&#93;&#93; (talk) 22:45, 2 March 2013 (UTC)

Vote -> boom
I clicked the vote button for Wikidata and stuff went boom, see the history there :) --Jeroen De Dauw (talk) 00:45, 2 March 2013 (UTC)


 * Thanks for letting me know Jeroen! This is nasty behavior in Semantic Forms when the  flag is being used, seemingly in conjunction with a , and probably only when intersecting with an   call. Phew! You are not in the operator group (no offense. ;-P) but the autoedit seems to still allow you to do the edit (note your vote change did occur), but the other fields that were set to restricted reverted back to their defaults. Bummer. For now, I've removed the restricted parameters. They've caused a handful of challenges and I don't think they are really necessary (at least for now). Thanks for checking in on WikiApiary! I just enabled another bot to audit and activate sites automatically so the data set here is going to get way more robust in the coming days! (PS, I've filed bug 45644 on this issue.) Thingles (talk) 19:42, 2 March 2013 (UTC)

System change
Heiya Jamie,

I was very bold an bought myself an new computer. This resulted into a new OS and the transition turned out to be a very painful process. More than expected. Basically this is the reason why I was not very active here lately.

Cheers --&#91;&#91;kgh&#93;&#93; (talk) 18:58, 6 March 2013 (UTC)


 * Very cool! What did you get yourself? What new OS? BTW, you picked a good time to be less active. Audit Bee is slamming the site and load on my server is 4x higher than normal. Audit Bee should be done by Thursday or so, and then things will return to normal. The very first audits with page creations have a higher load. See WikiApiary page graph. :-) Thingles (talk) 20:11, 6 March 2013 (UTC)


 * Hmm, very cool...? Well, this is a matter of taste. I switched from NT 5.1 to NT 6.2, but since I was using the Win 95 GUI for 18 years now, even on NT 5.1, this is kinda big change for me. Still I will probably install a Squeeze soon since I bought a 2 TB harddrive and now have plenty of space. As an old and quaint habit I still compress most of my files.
 * Yeah, Audit and Bumble Bee are very hard working bees right now and it is indeed getting a bit frantic on your server. :) I am surprised that there are that few problems apart from the JSON decoding. --&#91;&#91;kgh&#93;&#93; (talk) 22:39, 6 March 2013 (UTC)

Website form broken?
Just tried adding a wiki using the Website form. On submission, I just got a blank page, and no actual save operation occured. --Jeroen De Dauw (talk) 01:41, 7 March 2013 (UTC)


 * Damn! Sorry about that Jeroen. I pulled SyntaxHighlight GeSHi from master today and it looks like there was a change that made it dependent on 1.21.x. I see my error log had

PHP message: PHP Notice: Use of undefined constant CONTENT_MODEL_JAVASCRIPT - assumed 'CONTENT_MODEL_JAVASCRIPT' in /srv/www/mediawiki/public_html/w/extensions/SyntaxHighlight_GeSHi/SyntaxHighlight_GeSHi.php on line 76


 * …over and over. It looks like it was also causing edits via the API to fail which means auditing wasn't working either. Thanks for letting me know. I've switched that extension to it's  branch now which hopefully will keep things happier. Thingles (talk) 02:18, 7 March 2013 (UTC)


 * Yeah, looks like this commit broke me. Thingles (talk) 02:35, 7 March 2013 (UTC)


 * Ugh... looks like my bots triggered this error 39,627 times today. :-( Thingles (talk) 02:20, 7 March 2013 (UTC)


 * Odd, it looks like API autoedit's worked when this problem was present. Thingles (talk) 02:27, 7 March 2013 (UTC)

The bees really have done their job well
It is great to see that these two workaholic bees have done such a good job. :) Which lot is next? ;) I have posted here about the great changes to the display you did today. There are also a couple of suggestions. Cheers --&#91;&#91;kgh&#93;&#93; (talk) 22:52, 9 March 2013 (UTC)


 * The first audit of every site is done! The next step for automation is to get the error fields being used properly. Right now none of the bots set it, so soon I will have then incrementing those fields and then figure out when Audit Bee should mark them as defunct. Need to add a function to apiary.py to do this. The other item that is high on my priority list is to start creating the daily and weekly summary data in the database and start allowing those to be charted. I've got an idea for a very cool set of graphs using the daily and weekly data that just won't work with the raw data as it is collected now. I also need to add an audit bee process that prunes things like the Bot log which will now grow forever. That's pretty easy though. The graphs need a ton of help, but they need it all in Javascript which isn't a terribly strong point for me. Might see if I can get someone to help out on that. Oh, and the data PHP scripts too. :-) Thingles (talk) 13:19, 10 March 2013 (UTC)


 * True, automated error handling should really be the next step now that all the wikis are on board and active. This alone will at least reduce the error log and reduce the work operators will have to do manually. It is so sad that I do not know JavaScript since I would have loved to help here. My JavaScript is basically as good as your Swahili. :( --&#91;&#91;kgh&#93;&#93; (talk) 22:58, 10 March 2013 (UTC)


 * First step in this direction finished tonight. See this diff and here is the commit with the functionality added. Will watch this for a while. Next step will be to clear the error flag (right now it won't reset). Then would be having Audit Bee start deactivating sites that are over a certain error count. Thingles (talk) 03:16, 11 March 2013 (UTC)


 * Almost forgot a cool thing I'm adding to Bumble Bee too. I figured out that I could sort of cache the response for things like skins by storing a SHA-256 hash of the result the remote wiki gives me. So, I'm going to add this to Bumble Bee so that he will stop barraging WikiApiary with  edits for things like extensions and general info. Instead he'll do a SHA-256 on the API result from the target wiki and will only attempt a save to WikiApiary when the hash doesn't match.  Nifty! I'm also changing the refresh rate for those data sets. Right now they all check every 24 hours. I'm going to have general update every 3 days, extensions daily and skins every week. I'm also going to add a random offset to it to make sure that the segments don't general hot spots. Thingles (talk) 14:45, 10 March 2013 (UTC)


 * Once again a goodie. :) New revisions for no cause are really not a good thing to have. The hash approach is as nice as it is nifty I guess. :) Support from me for the change in update frequency though you could even change the skin updates to a fortnight. This would elegantly solve the < 1.11 "issue". --&#91;&#91;kgh&#93;&#93; (talk) 22:58, 10 March 2013 (UTC)

Swap jobqueue with edits
ATM the "job queue" graph spans the whole page while edits only half of it. Why not swap them? Seems like edits is more interesting to most people. in fact, I guess a lot of people do not even know what the job queue is all about :) --Jeroen De Dauw (talk) 16:48, 13 March 2013 (UTC)


 * Yeah, definitely. You are seeing a bit of work in progress there. The plan is to just have those three larger charts, and have a control to allow the user to switch what data they wish to view in those. Included in that is the link to popup those charts into a separate window (instant dashboard!). You are right... I doubt many care or even know about their job queues. Thingles (talk) 02:34, 14 March 2013 (UTC)


 * Just swapped edits in. See WikiApiary. :-) Thingles (talk) 02:42, 14 March 2013 (UTC)


 * I care! While it shouldn't be too visible, it's very useful. If you made a graph also for each farm, including all Wikimedia projects, I'd be so happy! You'd fix . --Nemo 16:33, 14 March 2013 (UTC)

Whitelist
Could you please whitelist traditio-ru.org? Is a good site, honest. Alex Mashin (talk) 06:46, 16 March 2013 (UTC)

Alex Mashin (talk) 20:04, 16 March 2013 (UTC)
 * Sorry about that. I was using the lists from meta for Extension:SpamBlacklist and it's definitely a bit aggressive. Traditio-ru.org is activated and will start collecting very soon. Thingles (talk) 13:21, 16 March 2013 (UTC)
 * Thank you.

Bookmarks
Are the bookmarks at http://links.thingelstad.com/wiki/Main_Page collected manually or automatically? (impressive job, by the way). Alex Mashin (talk) 20:20, 16 March 2013 (UTC)


 * Thanks Alex! They are entered manually (although semi-automated using a bookmarklet). I previously used Pinboard.in, and before that Del.icio.us. I had thought that Semantic MediaWiki could support a nice link wiki so I made one and then wrote an importer to pull all my previous bookmarks over. It even has a bot that updates URL redirects and checks the health of links. I have a bot that will replicate that wiki to others (see my friend User:Garrickvanburen's one at http://links.garrickvanburen.com/). Thingles (talk) 22:42, 16 March 2013 (UTC)

Installed version/latest version in Extensions section
Hi Jamie,

when I look at Special:Version of wikis I often wonder what is the latest version of the installed extensions. I think it would be nice to have that on WikiApiary, meaning on the Extension section/tab of each wiki's page not only display the installed version of each extension but also what would be the latest version of each extension. --Eric (talk) 12:59, 18 March 2013 (UTC)


 * Thanks for the suggestion Eric. This is definitely in my sites. It may first happen as an email for site administrators. If you have a Web site on WikiApiary that you would want that for edit it with form and add a notification for yourself. Also, you can click through to the Extension pages themselves and it will often have the version info there. But it would be much better to put it right in the Web site page. I'm just wary that it will invoke a query for every extension that the site has. Probably fine... 🐝 thingles (talk) 02:11, 19 March 2013 (UTC)

Questions/suggestions
Is it practical to get Bumble Bee to automatically upload and add logos when they're missing? At least if they're in a usual place. Adding logos takes significantly more time than the rest of the add wiki process, even when they're easily collected up like I had with the NIWA wikis.

Also, about "Allow wikis to use WikiApiary to do weekly backups. (Likely a paid service.)", if space becomes not much of an issue would you consider doing an annual XML text backup for free for non-huge wikis, but not retrievable without a one off charge (used to fund more backups)? Many wiki admins don't take care to backup, and as a result wikis are lost. And have a list of wikis which have gone down that you've got a backup off available. That'd be awesome, practically ensuring that no wiki is ever entirely lost.

And did you see my suggestion on MediaWiki_Extension? Is that likely to be practical long term?--Ete (talk) 17:10, 20 March 2013 (UTC)

Not semantic?
This wiki is semantic: OpenEI --Jeroen De Dauw (talk) 19:14, 21 March 2013 (UTC)


 * Was removed by User:Kghbln along with collect skin data, I imagine something about that wiki makes the SMW API inaccessible? I'll try turning it on, and note errors in the edit reason if it's still not working. (hope it's cool to reply to things like this on your talk page thingles)--Ete (talk) 19:38, 21 March 2013 (UTC)


 * Totally cool. I would tend to put a thread like this on Talk:OpenEI, but doesn't matter a ton. 🐝 thingles (talk) 20:20, 21 March 2013 (UTC)


 * When OpenEI was first added and Extensions pulled it was Feb. 15, 2013 and it was using Semantic MediaWiki 1.5.1.1 at that time (rev 25258). SMW 1.6 or greater is needed for  and hence the sentiment of User:Kghbln's comment "that's sad". On or right before March 11 it looks like they upgraded to Semantic MediaWiki 1.8 (rev 59051). So, now we can collect Semantic data! Yeah! Now, furthering the story. User:Audit Bee visited this site on March 8 to do an audit. On March 8, they were still running SMW 1.5.1.1 so everything was left as is. If nobody would have noticed this, User:Audit Bee would have enabled semantic data collection on it's next audit around June 9, 2013. 🐝 thingles (talk) 20:20, 21 March 2013 (UTC)


 * That's exactly what happened. Great observation. Beginning of February they also had big trouble with availability. This was the reason why I acted that late on them. Cheers --&#91;&#91;kgh&#93;&#93; (talk) 22:10, 21 March 2013 (UTC) Currently I am pretty busy. This is why you do not see me around that often these days. Will be back though.

Wikia is dreaming about this :)
See here - or they have traffic from outer space. :D --&#91;&#91;kgh&#93;&#93; (talk) 19:24, 27 March 2013 (UTC)


 * Ha! Indeed! 🐝 thingles (talk) 22:26, 28 March 2013 (UTC)

API help
First, great site and use of SMW! Love being able to see stats, something lacking out of the box on MW installs without really searching for them.

Second, I noticed you have a stat on edits per week. How are you grabbing that? We are trying to come up with a way API edits per month, but I'm failing to grasp the what and how without hitting the revision table directly with a direct query. Appreciate a pointer in the right direction. Thanks --- Hutchy68 15:31, 5 April 2013 (UTC)


 * Hey there! You can see the calculation of that index in Template:Website indexes. In short, I have the active user count for now and from 7 days prior ($$activeusers\prime$$), as well as the edit count from now and 7 days prior ($$edits\prime$$). And then I do:


 * $$(edits - edit\prime) \div ( (activeusers + activeusers\prime) \div 2 )$$


 * I can do this because I store the user and edit values over time from the API, so I can use the differential. I'm not sure how you would do this if you were calculating it by querying the MediaWiki database directly. Hope that helps. All of the site indexes are ones I made up, with the exception of Depth. So I would be happy to hear of any suggestions for improvement. 🐝 thingles (talk) 20:29, 9 April 2013 (UTC)

Hi thanks for creating an interesting site
Hi, just stopped by to say thanks for creating an interesting wiki. Love the concept of the site; anything that gives more publicity to all the wikis out there should be celebrated.

Sadly the data about my site, yellpedia.com, is wildly off due to a flaw in the design of how the magic word works (see my note on the talk page for yellpedia.com). Not sure anything can be done about this, or if it should be done, since my issue is most likely unique. Regardless I wish you well, and hope this site thrives. Chris Tharp (talk) 04:00, 9 April 2013 (UTC)


 * Thanks for the complement Chris! I responded at Talk:Yellpedia. 🐝 thingles (talk) 20:23, 9 April 2013 (UTC)

Vision?
Hi Jamie! You're doing very good job - the job WMF have to do, in fact. What vision so you have for Wikiapiary? I'm particularly interested in extension and skin management, do you plan to add anything for them? Yury Katkov (talk) 18:53, 14 April 2013 (UTC)


 * First off — thank you for the donation Yury! That is generous of you and totally awesome! I would be happy to enable more frequent data collection for a few wikis that you care about. By default User:Bumble Bee only checks websites every 4 hours but I'd be happy to enable 15 minute collection for some if you would like. Let me know.


 * Regarding vision I probably have more of that than I have time. If you dig back on talk pages you'll see that initially I wasn't sure about bulk adding a ton of sites but that turned out to be awesome. I would like to add as many wikis as possible to WikiApiary but it's not my top priority right now. If others wanted to write spiders to add them I would happily help with the code and run them. I'm focusing more on improving the site itself, continuing to improve the bots, and hopefully soon getting User:Notify Bee working so I can start sending emails to people when they have old software or interesting things happen on their wikis. I've also considered adding some more streams of data. The stats that I pull along with extensions and skins are good. I think I could do some really interesting anti-spam work if I pulled in the list of new users on each wiki.


 * In terms of vision, I've been following closely the conversation on the mailing lists about better extension management. I think WikiApiary is in a great position to help provide data for that. I don't believe anyone else knows what version of what extensions are being used on what versions of MediaWiki. WikiApiary does. And I think the recent addition of related extensions provides some more utility. I would be very interested in WikiApiary being the API/data repository for the questions that a more robust extension manager would need to answer. I've been hesitant to jump into this quickly though, just being cautious and don't want to be presumptive.


 * In other areas of vision I would ultimately love to also develop a WikiApiary extension that people could install to tie even more information from WikiApiary into their sites. Not sure what all it would do, but I'm very interested in detected spam registrations and remotely nuking the accounts from WikiApiary.


 * I should be clear that I do see a commercial opportunity here too. I fully plan to offer backup services and other help for admins and I see there being a free, basic and premium offering. The main variations for those services will be: how long is data kept for the wiki, how deep is the backup that WikiApiary keeps and how frequently will your site be checked.


 * I guess I jumped into backups without mentioning that I am starting to work on backing up ALL wikis (smaller than say 10,000 pages for now). I plan to only backup the current version of pages (no history) and only the text. Thing of this as a time capsule archive. This would be a free version of backup, but if your site were to die and you need it there would be a one-time fee to get it.


 * Anyway, that is a start. I'm all ears on vision and where WikiApiary could go though. I'd love to know what you think about the items above and what other thoughts you have for WikiApiary! 🐝 thingles (talk) 02:15, 17 April 2013 (UTC)