Showing posts with label General. Show all posts
Showing posts with label General. Show all posts
Tuesday, August 10, 2010
Thursday, June 17, 2010
Update
Been a while since the last post. Still tagging handles, still removing warts from my code. The latest code is almost where I want it. I could do a bit more factoring, removing some repetition but it would be more for aesthetic purposes. The code does what I want with satisfactory performance. Test coverage as is usually the case could be much better.
On a more relevant note, April and May 2010 are now in the database so I thought I'd update the big picture that I first sketched here.
As of the end of May 2010:
8,082 articles over the 73 months of HA.org's existence.
430,912 comments. (Haven't filtered out the spam yet.)
and last but not least:
Out of 15,728 unique handles!
Top 10 # of comments by handle:
10 GBS @ 5,317
9 rhp6033 @ 5,478
8 ArtFart @ 6,004
7 Steve @ 6,167
6 Marvin Stamn @ 6,774
5 Daddy Love @ 8,580
4 YLB @ 9,323
3 Mr. Cynical @ 9,827
2 Puddybud @ 11,369
and number one?
Our beloved Roger Rabbit at 58,310.
So Steve moves ahead of Art, gaining on Stamn. Puddybud barely budges due to begging everybody else to come to me to justify his drivel. (Useless, he'll ALWAYS be #2.) All the usual disclaimers on that top 10 list still apply.
An updated brackets report follows:
One handle has joined the 1000+ comments club. Right now I don't know who you are but congrats!
Update to "Update": The newest member of the 1000+ comments club is "John" which is a handle that's been used by many, many people, hero and troll alike, over the years. Again, congratulations "John"!
On a more relevant note, April and May 2010 are now in the database so I thought I'd update the big picture that I first sketched here.
As of the end of May 2010:
8,082 articles over the 73 months of HA.org's existence.
430,912 comments. (Haven't filtered out the spam yet.)
and last but not least:
Out of 15,728 unique handles!
Top 10 # of comments by handle:
10 GBS @ 5,317
9 rhp6033 @ 5,478
8 ArtFart @ 6,004
7 Steve @ 6,167
6 Marvin Stamn @ 6,774
5 Daddy Love @ 8,580
4 YLB @ 9,323
3 Mr. Cynical @ 9,827
2 Puddybud @ 11,369
and number one?
Our beloved Roger Rabbit at 58,310.
So Steve moves ahead of Art, gaining on Stamn. Puddybud barely budges due to begging everybody else to come to me to justify his drivel. (Useless, he'll ALWAYS be #2.) All the usual disclaimers on that top 10 list still apply.
An updated brackets report follows:
One handle has joined the 1000+ comments club. Right now I don't know who you are but congrats!
Update to "Update": The newest member of the 1000+ comments club is "John" which is a handle that's been used by many, many people, hero and troll alike, over the years. Again, congratulations "John"!
Friday, April 30, 2010
Entering the Twitterverse
Tagging handles in earnest indeed. In the meantime, I may tweet a few eureka moments about code or trolls.
While I was at it, I added an rss feed.
Ciao..
While I was at it, I added an rss feed.
Ciao..
Monday, April 26, 2010
Still Distracted but Turning a Big Corner
In the last post I was all jazzed about an opportunity to make the data collection from my "select sources" run much faster.
Well, I'm pretty much at the other end of it and I'm quite happy about it. I did a major rework of the code. It's much better than before and my only regret is that I wish I'd thought of this speedup sooner.
The old code base accomplished the collecting (almost 8,000 articles and 415,000 comments) but it did the job kind of slowly and that was even with some good speedups I had spotted early. But if I had had what I've got now, it would have gone much faster. Also I would have totally minimized data collection from HA itself. Oh well. Better late than never.
In the last post I mentioned that I used code that would automate a web browser to navigate the web and collect HA articles from select sources. I also mentioned that by bringing up one browser instance and feeding requests to it, you will maximize the data collection performance.
The obvious speedup of course is to distribute requests across more browser instances - make it scale. But now we're talking about forking off objects from classes that encapsulate the browser-driving code. And once you create sub-processes you need a means to communicate requests to and retrieve results from those forked instances.
The tried and true method to accomplish this communication is to use a message queuing daemon. In a previous Ruby project I'd used a pretty slick one with a handy Ruby support library. No problem mon - it's just a matter of finding the right places to insert this code. It was a lot of work but it made me take a hard look at the entire code base and throw away a lot of stuff that just wouldn't be needed anymore. And as I separated out code into Ruby modules by function, I was forced to come to a better understanding of instance variables. In my Repo class I totally overused class methods and was passing way too many parameters from method to method. Why? Way back in the beginning I thought it would be easier to test if I did it that way. Totally wrong. Thanks to this rework, the code in this project and my other projects will be so much better.
One of my select sources in the past really made me tear my hair out. I had only collected 4 or so articles from it because of all the trouble it gave me but now that's all changed. I've gotten much more than a handful from that source for March 2010 alone and I expect it to only get better. And there's yet another select source out there that I didn't even bother to try because it was so weird and different from the others. I'm feeling pretty confident now. I will give it a shot but I have to do a couple things first.
I'm going to start tagging handles in earnest. This is really critical to separating the trolls from the heroes and then being able to make big picture analyses of troll behavior over the months and the election cycles.
Then I've got to scratch an itch about current month activity. So far I'm caught up to the end of March 2010. Normally I wouldn't add a month to my collection until 10 days into the next month. Why? Goldy doesn't shutdown comments to an article until 8 days or so after the article is published. So if I'm curious about some interesting activity in the current month, I have to develop a way to create a snapshot database with an eye on merging various snapshots into the real thing to save time when a month finally closes.
Well, I'm pretty much at the other end of it and I'm quite happy about it. I did a major rework of the code. It's much better than before and my only regret is that I wish I'd thought of this speedup sooner.
The old code base accomplished the collecting (almost 8,000 articles and 415,000 comments) but it did the job kind of slowly and that was even with some good speedups I had spotted early. But if I had had what I've got now, it would have gone much faster. Also I would have totally minimized data collection from HA itself. Oh well. Better late than never.
In the last post I mentioned that I used code that would automate a web browser to navigate the web and collect HA articles from select sources. I also mentioned that by bringing up one browser instance and feeding requests to it, you will maximize the data collection performance.
The obvious speedup of course is to distribute requests across more browser instances - make it scale. But now we're talking about forking off objects from classes that encapsulate the browser-driving code. And once you create sub-processes you need a means to communicate requests to and retrieve results from those forked instances.
The tried and true method to accomplish this communication is to use a message queuing daemon. In a previous Ruby project I'd used a pretty slick one with a handy Ruby support library. No problem mon - it's just a matter of finding the right places to insert this code. It was a lot of work but it made me take a hard look at the entire code base and throw away a lot of stuff that just wouldn't be needed anymore. And as I separated out code into Ruby modules by function, I was forced to come to a better understanding of instance variables. In my Repo class I totally overused class methods and was passing way too many parameters from method to method. Why? Way back in the beginning I thought it would be easier to test if I did it that way. Totally wrong. Thanks to this rework, the code in this project and my other projects will be so much better.
One of my select sources in the past really made me tear my hair out. I had only collected 4 or so articles from it because of all the trouble it gave me but now that's all changed. I've gotten much more than a handful from that source for March 2010 alone and I expect it to only get better. And there's yet another select source out there that I didn't even bother to try because it was so weird and different from the others. I'm feeling pretty confident now. I will give it a shot but I have to do a couple things first.
I'm going to start tagging handles in earnest. This is really critical to separating the trolls from the heroes and then being able to make big picture analyses of troll behavior over the months and the election cycles.
Then I've got to scratch an itch about current month activity. So far I'm caught up to the end of March 2010. Normally I wouldn't add a month to my collection until 10 days into the next month. Why? Goldy doesn't shutdown comments to an article until 8 days or so after the article is published. So if I'm curious about some interesting activity in the current month, I have to develop a way to create a snapshot database with an eye on merging various snapshots into the real thing to save time when a month finally closes.
Wednesday, April 7, 2010
Opportunities, Distractions
One goal of this project was to minimize data collection as much as possible directly from HA.org itself. I would have been a lot farther along much sooner if I didn't care about this. Why is this important?
I not only want to slice and dice troll behavior but I also want to learn all these awesome cool ruby-based software libraries. They've been a total joy to learn about and work with.
The data collection problem and the composition of its solution into classes has been pretty simple so far. There's a Session class whose job is to collect one month of HA.org's existence. The most important instance method of this class is add which at its heart is a for loop which starts at the first article of the month and ends at the last. Each article of course contains the comments which is the gold I'm after. But it's within the loop that the fun really begins.
I said I wanted to minimize data collection from the well, HA.org. To do this I constructed a least cost route of sources from which to draw articles with HA.org being the highest cost, the last resort. So I have a Repo class (short for repository) that's blessed from a Session instance. I feed it an article number and a source and it tries to get the article. If it comes up short then the Session instance tries the next source in the route.
So far pretty simple. Not rocket science. Fun? Only for a geek, right? Well it gets even better. Some of my sources require calling a simple http get which is available from lots of ruby http libraries. This is how I get stuff, again as a last resort, from HA.org. Butt simple and too boring. But a few sources, I call them select sources, require something a little more sophisticated. They require driving a web browser to go and get the article. Without getting into too much detail:
Making this go was just too much freaking fun for words.
So, after fist pumping and watching my code drive a web browser to navigate the web and get the stuff I wanted from it I quickly realized I'd done something a little dumb:
My code was opening and closing the web browser for each request to the web.. Ugh.. It's a lot faster Sherlock to just leave one browser instance up and feed all the many requests you want to it and then close it when the session is concluded. Another hour or two of searching for the right spot in the code to insert the needed changes and I had a big increase in performance.
Which is all a long winded introduction to the point of this post - there's yet ANOTHER opportunity (and yep, distraction from my initial goal) to make this puppy run even faster. Next post. I'm gonna code.
I not only want to slice and dice troll behavior but I also want to learn all these awesome cool ruby-based software libraries. They've been a total joy to learn about and work with.
The data collection problem and the composition of its solution into classes has been pretty simple so far. There's a Session class whose job is to collect one month of HA.org's existence. The most important instance method of this class is add which at its heart is a for loop which starts at the first article of the month and ends at the last. Each article of course contains the comments which is the gold I'm after. But it's within the loop that the fun really begins.
I said I wanted to minimize data collection from the well, HA.org. To do this I constructed a least cost route of sources from which to draw articles with HA.org being the highest cost, the last resort. So I have a Repo class (short for repository) that's blessed from a Session instance. I feed it an article number and a source and it tries to get the article. If it comes up short then the Session instance tries the next source in the route.
So far pretty simple. Not rocket science. Fun? Only for a geek, right? Well it gets even better. Some of my sources require calling a simple http get which is available from lots of ruby http libraries. This is how I get stuff, again as a last resort, from HA.org. Butt simple and too boring. But a few sources, I call them select sources, require something a little more sophisticated. They require driving a web browser to go and get the article. Without getting into too much detail:
Making this go was just too much freaking fun for words.
So, after fist pumping and watching my code drive a web browser to navigate the web and get the stuff I wanted from it I quickly realized I'd done something a little dumb:
My code was opening and closing the web browser for each request to the web.. Ugh.. It's a lot faster Sherlock to just leave one browser instance up and feed all the many requests you want to it and then close it when the session is concluded. Another hour or two of searching for the right spot in the code to insert the needed changes and I had a big increase in performance.
Which is all a long winded introduction to the point of this post - there's yet ANOTHER opportunity (and yep, distraction from my initial goal) to make this puppy run even faster. Next post. I'm gonna code.
Tuesday, April 6, 2010
QA winds down. HA's most commented posts.
To kind of, sort of check on the quality of the data collection process, I whipped up a report of the most commented articles of each month of HA's existence. Then here and there I went to the well to see if what I have and what HA has matches up..
I've found tiny variances in, strangely enough, both directions.. I'll definitely have to look more thoroughly at those months where I seem to have MORE than what Goldy has. But so far the variances seem to be made mostly of spam.
Once I had that report working, I tweaked it to select one article at random from each month. Again, things seem to match up just fine. Even better than the most commented articles sample.
So confidence is high.
Anyhow the most commented posts merit further mention. Here's a pretty picture:
The first column is the year/month, the second is the article number and the third is the total number of comments for the thread. The most commented HA blog post, in March of 2005, was a knock-down, drag-out thread about Terry Schiavo - a major body-blow for the extreme right wing. Between trying to weaken Social Security, using Terry Schiavo's vegetative body as a prop, letting New Orleans drown, corruption/sex scandals, torture, dollar black hole wars of choice and the final, ugly meltdown of the economy - let us never forget what it means to have the right wing in control of this country.
I've found tiny variances in, strangely enough, both directions.. I'll definitely have to look more thoroughly at those months where I seem to have MORE than what Goldy has. But so far the variances seem to be made mostly of spam.
Once I had that report working, I tweaked it to select one article at random from each month. Again, things seem to match up just fine. Even better than the most commented articles sample.
So confidence is high.
Anyhow the most commented posts merit further mention. Here's a pretty picture:
The first column is the year/month, the second is the article number and the third is the total number of comments for the thread. The most commented HA blog post, in March of 2005, was a knock-down, drag-out thread about Terry Schiavo - a major body-blow for the extreme right wing. Between trying to weaken Social Security, using Terry Schiavo's vegetative body as a prop, letting New Orleans drown, corruption/sex scandals, torture, dollar black hole wars of choice and the final, ugly meltdown of the economy - let us never forget what it means to have the right wing in control of this country.
Saturday, April 3, 2010
Tagging Handles
Trolls and Heroes adopt many aliases over the course of their on-line existence. To separate them from one another and make searching and analysis easier I'm going to stamp each comment with a tag. No not all umpteen thousands of them - just associate the 15,078 handles (and growing) with a more coherent tag e.g. puddybud for all the versions of puddybud's handle and that will in turn stamp his legacy of over 26 thousand comments.
The tags will live in a separate file, a hash whose key is the tag and whose value is a category:
h - hero
t - troll
ul - a lefty who hurts the cause by getting too much like a righty.
rr - a right winger whose tone is reasonable most of the time. (Very rare at HA.org.)
s - spammer
In other news I'm torn a bit about using Sinatra over Rails for the QA portion of this project. Rails 3 is worth a look because it's new and much more flexible. I'll probably use both for portions of the remainder of this project.
After all most of the motivation and benefit of this project has been from learning and applying all these wonderful ruby-based software development
Late update: Do you believe in synchronicity? I just found this out minutes before I published this post. The author of the command/tasking framework that I use to develop my reports on this project is heavily motivated by the concept of machine tagging which is used by Flickr and other projects. For now I'm sticking with the simple tagging implementation I've sketched here but I may have to revisit this more thoroughly later.
The tags will live in a separate file, a hash whose key is the tag and whose value is a category:
h - hero
t - troll
ul - a lefty who hurts the cause by getting too much like a righty.
rr - a right winger whose tone is reasonable most of the time. (Very rare at HA.org.)
s - spammer
In other news I'm torn a bit about using Sinatra over Rails for the QA portion of this project. Rails 3 is worth a look because it's new and much more flexible. I'll probably use both for portions of the remainder of this project.
After all most of the motivation and benefit of this project has been from learning and applying all these wonderful ruby-based software development
Late update: Do you believe in synchronicity? I just found this out minutes before I published this post. The author of the command/tasking framework that I use to develop my reports on this project is heavily motivated by the concept of machine tagging which is used by Flickr and other projects. For now I'm sticking with the simple tagging implementation I've sketched here but I may have to revisit this more thoroughly later.
Tuesday, March 30, 2010
An Idea for a Rails QA App forms...
I've collected all the data up to the end of February 2010 and had some fun running a few reports. Now what?
There's the QA process. Do I really have all the data or do I just think I do? How do I exclude what I don't want, i.e. spam?
How do I separate the HA.org heroes from the trolls from the spammers? That's the most important question because this project is about troll behavior and having fun dissecting it.
The work of collection was a month by month process. The HA.org frontpage contains a comprehensive list of the archives by month and the number of articles for each month. So I'm thinking right now that a good QA process' work will have to be divided the same way.
So first I see a selection toolbar at the top of the browser viewing area with a year/month and backwards/forwards arrows. At the bottom center right of the screen a radio button group with hero, troll, spammer selections and a submit button. The rest of the screen will contain a sampling of comments from the handle owner.
So at a glance, I can come to a decision, is this a hero, a troll or a spammer? Once the decision is made the comments and the handle can be marked and then excluded from further year/month examinations.
And of course there's all the handles whose status I already know up front. No need to look at those. Roger Rabbit's stuff alone accounts for 11 percent!
So that's good enough for pass 1.
Pass 2 is then first a matter of submitting some spammer's spam for further examination to Akismet. Why? A hero or a troll's handle may have been hijacked by a spammer for a time. Submitting the comments to Akismet can easily mark what is spam and what is not.
Next is the hardest part.. A handle like Jane Doe may at one time be a progressive from Tacoma and at another time a troll from Bumfuck, WA. That's going to take a good amount of work but of course if we're smart, the work could be minimized.
15,078 handles.. Don't remind me of the comments. Someone's got to do it.
There's the QA process. Do I really have all the data or do I just think I do? How do I exclude what I don't want, i.e. spam?
How do I separate the HA.org heroes from the trolls from the spammers? That's the most important question because this project is about troll behavior and having fun dissecting it.
The work of collection was a month by month process. The HA.org frontpage contains a comprehensive list of the archives by month and the number of articles for each month. So I'm thinking right now that a good QA process' work will have to be divided the same way.
So first I see a selection toolbar at the top of the browser viewing area with a year/month and backwards/forwards arrows. At the bottom center right of the screen a radio button group with hero, troll, spammer selections and a submit button. The rest of the screen will contain a sampling of comments from the handle owner.
So at a glance, I can come to a decision, is this a hero, a troll or a spammer? Once the decision is made the comments and the handle can be marked and then excluded from further year/month examinations.
And of course there's all the handles whose status I already know up front. No need to look at those. Roger Rabbit's stuff alone accounts for 11 percent!
So that's good enough for pass 1.
Pass 2 is then first a matter of submitting some spammer's spam for further examination to Akismet. Why? A hero or a troll's handle may have been hijacked by a spammer for a time. Submitting the comments to Akismet can easily mark what is spam and what is not.
Next is the hardest part.. A handle like Jane Doe may at one time be a progressive from Tacoma and at another time a troll from Bumfuck, WA. That's going to take a good amount of work but of course if we're smart, the work could be minimized.
15,078 handles.. Don't remind me of the comments. Someone's got to do it.
Monday, March 29, 2010
It is accomplished...
As of the end of February 2010.
7,680 articles over the 70 months HA.org had been to that point in existence.
414,965 comments. (Haven't filtered out the spam yet.)
Whew! A mother lode of right wing inanity and foolishness. With some intelligent and fun liberal-leaning comment thrown in..
and last but not least:
Out of 15,078 unique handle names!
Top 10 # of comments by handle:
10 GBS @ 4,952
9 rhp6033 @ 5,016
8 Steve @ 5,582
7 ArtFart @ 5,839
6 Marvin Stamm @ 6,774
5 Daddy Love @ 8,246
4 YLB @ 8,678
3 Mr. Cynical @ 9,314
2 Puddybud @ 11,368
and number one?
Our beloved Roger Rabbit at 56,751.
Commenter's various aliases are not yet factored in. Steve might be several different "Steves". (And the HNMT is a particularly nasty case.) It's going to require quite a bit of study. And I first have to do some qa on the collection process as a whole. So this post will be updated as the picture gets clearer..
But so far, so good! Fun, fun, fun on the runup to November!
Special note to that most moronic of trolls (#2? It fits!):
$ du -h .ha
201M .ha/data
...
Thursday, July 26, 2007
Troll Dossiers
I'm going to compile dossiers on each of the trolls. There'll be links on the sidebar to each dossier to make it easy for you to contribute items on each troll that I may have overlooked. You do this in the comments sections of the post.
The dossiers will have the following sections.
Quick facts - the quick and dirty back story on the troll
Aliases - This is important. Trolls often post with many aliases. The more of these I know, the better I can data mine. You can really help out here with aliases that I'm not familiar with.
Misc - fun facts the come to the surface about our wingnut friends.
Links - to a greatest hits lists of troll comments.
These files will be a quick, easy way to collect info on each troll for both entertainment and to aid data collection and analysis.
My next post will be an example of a dossier.
Update: include your ha.org handle with your information and I'll hat tip you in the dossier.
Update II: I've added a status section to the dossiers. Active, Retired, Semi-Retired, Occasional, etc.
The dossiers will have the following sections.
Quick facts - the quick and dirty back story on the troll
Aliases - This is important. Trolls often post with many aliases. The more of these I know, the better I can data mine. You can really help out here with aliases that I'm not familiar with.
Misc - fun facts the come to the surface about our wingnut friends.
Links - to a greatest hits lists of troll comments.
These files will be a quick, easy way to collect info on each troll for both entertainment and to aid data collection and analysis.
My next post will be an example of a dossier.
Update: include your ha.org handle with your information and I'll hat tip you in the dossier.
Update II: I've added a status section to the dossiers. Active, Retired, Semi-Retired, Occasional, etc.
Wednesday, July 25, 2007
I'm going to expose your asses!
Greetings fellow HA readers, progressives and yes, you trolls. Who amongst you loves to have fun with trolls? Raise your hand! All of you, good!
Well, why am I doing this? Maybe out of some respect for Goldy's new commenting policy. To turn off my caps lock key. But also to brush up on my perl and ruby skills and learn the google API. Why you say?
The HA.org comment threads are a priceless treasure of right-wing lunacy just ripe for some data mining and analysis. As I promised Puddybud one fine day, "I swear, I'm going to spider this site and expose your ass!".
So that's what I'm going to do. I'm going to try not to spider HA.org directly. That'd run up Goldy's bandwidth bill. My goal is to use the google cache only.
I'm also going to be looking at the other website. You know the one run by some kept fella on Greenlake? But only to add some spice to the fun.
What is spidering? Books have been written on it. It's not that hard to do.
So folks, look forward to facts and figures and greatest hits lists on your favorite trolls. Of course I'll be welcoming your input and insights on the subject.
We're going to have fun with this!
Well, why am I doing this? Maybe out of some respect for Goldy's new commenting policy. To turn off my caps lock key. But also to brush up on my perl and ruby skills and learn the google API. Why you say?
The HA.org comment threads are a priceless treasure of right-wing lunacy just ripe for some data mining and analysis. As I promised Puddybud one fine day, "I swear, I'm going to spider this site and expose your ass!".
So that's what I'm going to do. I'm going to try not to spider HA.org directly. That'd run up Goldy's bandwidth bill. My goal is to use the google cache only.
I'm also going to be looking at the other website. You know the one run by some kept fella on Greenlake? But only to add some spice to the fun.
What is spidering? Books have been written on it. It's not that hard to do.
So folks, look forward to facts and figures and greatest hits lists on your favorite trolls. Of course I'll be welcoming your input and insights on the subject.
We're going to have fun with this!
Subscribe to:
Posts (Atom)
Posts
