Archive for the ‘Editorials’ Category

Going Dark

Wednesday, October 14th, 2009

Nearly 12 months on from the start of my blog (the first post went up at the end of November 2008), I’m finding less and less time to do the quality kind of writing I have set out to do. I feel that, as I change, so too does my website need to change.

As much as this site is currently a programming blog, it’s also my electronic representation. The website shares my name, and is representative of me. There’s more to me than just programming. For a while I’ve tried blogging or writing about non-programming related topics on another domain, however I don’t feel this is fair to myself.

I guess the best way to describe myself is an aspiring polymath. I have an interest in a broad range of topics, notably programming/computer science, mathematics and art. This website doesn’t really represent that. I can’t maintain multiple websites to separate various areas of my life – I’m a single person with multiple facets, and I should have a single website, with multiple facets.

Additionally, I find myself with less and less time to devote to this kind of writing, as it slowly makes it’s way down to the bottom of my priority list. Currently I’m trying to rekindle my math education by going back to basics, learn Punjabi so I can impress the fiance’s parents, do several programming projects at once and get back into doing art regularly, and writing is a very far last.

So, all that lead up to say that I’m going dark, and will be overhauling the site sometime in the future. I’m going to change the format of the site so that it more accurately represents me as a whole, is better organised and has a nice format that allows me to post quick, smaller posts that can be filtered out when subscribing to RSS feed. I may or may not stick with WordPress – I think I’ll probably start hacking away at Enki and do some funky stuff with wildcard subdomains or something.

All older posts will remain, as a few of my posts are rather popular (by more than 50% compared to the rest of the site) and I suspect they’re really helping people out.

Now Appearing In Austin, Texas

Monday, April 27th, 2009

This blog has gone dark over the last two or so weeks, but for a very good reason. On the 21st of April, I flew from Perth, Western Australia to Austin, Texas, where I will be staying for the next 6 months. For those not acquainted with the trip, it’s half a world away at over 13,000km, for a total flying time of 29 hours not including layovers (for my trip, anyway).

I’ve been spending the last week settling down here in Austin, and the week before that preparing for the trip. I’m still suffering from some jet lag, and catching up on the last 8 months with my girlfriend, who I hadn’t seen since August 2008. I haven’t had the time nor the concentration required to write a blog post, and I haven’t touched my Google Reader account, which I can see has over 1,600 unread articles.

However, I do have some interesting posts coming up. I’m planning a response to an AI article that was linked to me in a comment, a more in depth look at the importance of searching by solving the 8-Queens problems in Python using some different search techniques, and a look at creating some graceful degradation control adapters for ASP.NET server controls that don’t work without Javascript.

So, don’t unsubscribe from my feed in your readers, the blog isn’t abandoned, life has just taken priority over it. When things settle down, I’ll start writing articles again.

Where Do We Draw The Line On Complexity?

Thursday, April 16th, 2009

I’ve got a few posts planned about AI and HCI, however these require a lot of time to research and write code for, time that I don’t have given I’m flying to the US on the 21st of April. So I apologise to people who saw my last AI article on Hacker News and Reddit and subscribed to my feed expecting more of the same. There will be more, I promise. I just need more time. For now, just some musings.

I’ve been thinking a lot lately about solving problems. I read a number of programming aggregators and startup blogs, and the term “solve a problem” keeps nagging at me. To be successful, at programming and business, you need to solve a problem. I’d love to eventually run a development company, but I don’t know what problem to solve. So, I consider the problem my current employer is solving.

I currently work for a web development company that is doing fairly well. We have a suite of products that we sell to our clients, and sell services related to those products. They’re doing well, so obviously, they must be solving a problem. So, from “client” point of view, the problem we solve is that of letting people who don’t know anything about websites build and run complex websites.

We have a CMS product that allows non web geeks manage websites. This includes writing new pages and editing content on existing pages, publishing news items, distributing these to an email list, managing uploaded files and inserting modules of dynamic content to their hearts desire. The modules we offer include contact forms, random image rotators, resume upload controls, auto-generating bread crumbs and menus, and so on. These modules aim to reduce the complexity of managing dynamic sites, so that ordinary people can do it.

On top of that, we do custom applications built on top of that CMS site. Clients have a site they run themselves with the CMS, but some also have dedicated applications with a separate interface. These are separate because they’re custom jobs. They provide functionality that is not available in the CMS (yet), or functionality that would be too complex to maintain using the CMS. So again, we’re reducing the complexity of running a custom application.

So, on first looks, we build websites and applications that non-geeks can use. But looking further, we manage complexity. We reduce the complexity for the client, so that the client can do difficult things with ease, and don’t have to know how it’s done, only that they can do it.

On the other hand, good programming is about managing complexity. That is, trying to keep your code as free from complexity as you can get away with while still doing the job. Complex code is buggy and unmaintainable.

Complexity exists, and it is immutable. Once you’ve simplified a task to the point that the only steps involved are those that accomplish that task, and that task cannot be accomplished without those steps, you have your complexity, and it isn’t going anywhere. The task cannot be made more simple, because to do that we’d have to leave out steps, which would not accomplish the task.

What we do from then on is delegate complexity, shift it around and make it other people’s problem. Take for example, the task of creating and publishing a blog post. Our software has tagging, categories, post titles and post content. Creating a post involves writing a post, giving it a title, adding tags, selecting categories and publishing it.

Breaking it down, we need to create a body, create a title, create all categories, create all tags, link the tags to the post, link the post to the categories and publish it. Each link has a display name and a URL slug, and each category has a display name and a URL slug. URL slugs cannot contain certain characters, however their display names can.

Now, we can delegate this complexity to the user, which will result in simple code. The user will be responsible for creating tags and categories, and giving them URL slugs that fit our rules, while the application will just take the data and store it, and retrieve it when asked. The user will be responsible for assigning tags to the post and the categories, and the application just stores it. This is obviously an unacceptable level of complexity for the user.

We could also delegate all the complexity to the application, making the user experience simple. The user just types in the tags they want, picks the categories, and the application will link the tags if they exist, create them if they don’t, automatically strip unwanted characters from URL slugs and hide a lot of the complexity. In more complicated cases, this will result in very complex code that will be bug prone.

So the question is, where do we draw the line? What balance of code complexity and user interface complexity is successful? I guess you could say “make it 50/50″ and think that’s good enough, but I don’t agree. I think the balance is very contextual, and often leans in favour of code complexity. In simple examples like above, taking on all the complexity of the task in the code is a trivial thing and easily done. For nuclear reactor control software, things might not be straight forward.

This is the key to a successful software product/business, I think. Finding the right balance between complexity. If you put too much complexity on the user, user’s won’t be able to use it, and your software won’t sell, and the whole thing will bomb. If you put too much complexity on the software, it will be buggy and difficult to maintain, and will eat up development time and money, and possibly even end up costing more than it earns, and then it will bomb. That’s not even including the more personal aspects of programmer job satisfaction and sanity.

Success is finding the sweet spot when balancing complexity.

Correcting 5 Misconceptions About AI

Wednesday, April 8th, 2009

For the longest time, I have been interested in artificial intelligence. The idea of computers that could make decisions and think independently fascinated me. Of course, I knew that AI wasn’t quite that advanced, but the idea still captivated me. However, the barrier to learning AI was too high. It was too complex and too academic. Just looking at the mathematical notation involved made my head spin.

I eventually took the plunge and convinced my family to buy me Artificial Intelligence: A Modern Approach for Christmas, and 12 months later, I actually started reading it. I’m still reading it, and will be the first to admit I don’t understand everything that is discussed, however there are a few things that I’m realising about AI.

  1. It’s All About Searching
    Searching is a constant in AI techniques. Problems are defined and solution spaces are created. Problems can be represented in a number of ways (graphs, trees, logic knowledge bases), however in the end, it always comes back to searching.

    This is important because searching is easy. Searching is something that a lot of programmers already know, even if it’s only at a most basic, brute force level. We do searching all the time. We loop through arrays searching for values, we use regular expressions to match string patterns, we retrieve records from databases. We search.

    At it’s simplest, you can search using brute force searching, iterating over all combinations and permutations of solutions looking for one that satisfies the problem, but beyond that, you can involve tricky heuristics to make optimal decisions about how to search. You can have local searches, which will pick a solution space and search it for local maximums, such as hill climbing, and searches that will find global minimums, such as simulated annealing. But it’s still search. It’s still something you can do.

  2. It’s More Common Than You Think
    AI is a hell of a lot more prevalent than most people realise. I didn’t think AI had much commercial application before I started learning about it, but luckily that didn’t dim my interest. For those who are interested, but holding back because they don’t see how it would benefit them, here’s some good news. AI techniques are used everywhere.

    That international 4 city flight you booked used AI techniques. A constraint satisfaction problem solver took all the constraints about needing to be at this city by that time, flying on this airline for that much, and creating a plan for you. When Amazon recommends products you might be interested in, it calculates this with Bayesian networks and classifiers, a method of probabilistically linking a set of variables. Circuit design, product manufacturing, supply chain optimisation, all of these things use techniques that AI use. They’re not the sole domain of AI, but learning AI will cause you to learn these too.

  3. You Can Use It Today
    Whatever you’re working on, you can probably use AI techniques in it. Even some of the more exotic sounding techniques like neural networks can be of use to you. Self healing databases? Hell yeah. Even Bayesian classifiers to catalog and categorise products, heuristic searches to mine data in databases, hierarchical task planners to plan that holiday or manage that Gantt chart.

    Can you use it anywhere? No. Your simple CRUD app probably won’t benefit from a wizz-bang heuristic search. But if you’re doing anything that involves large amounts of data, interacting with people, predicting trends and recognising patterns, you can use it.

  4. It’s In Demand
    You may have heard of the Netflix Prize. Guess what? That’s AI. Google is the biggest search engine around, and index billions upon billions of pages on the internet, and can get you relevant results to a question in a matter of seconds. That’s one hell of a big knowledge base, and one smart search algorithm. Amazon sell products all over the world, and aggressively upsell and cross sell. I get emails about related products I might like based on my wishlist and purchase history, and they’re actually pretty accurate.

    Also, computer games. Enough said.

    AI skills are in demand. Not huge demand, but probably more that you would have guessed. These skills are hugely profitable in the right hands, and big companies want to extract every single little morsel of useful information about your browsing, shopping, eating, travelling, viewing and reading habits in order to market to you more effectively. Now that sounds a little creepy to me, but if that doesn’t bug you, more power.

  5. There’s a Lot of Information Out There
    AI isn’t some weirdo niche science topic. There’s actually quite a lot of information out there, once you start going down the rabbit hole. “AI: A Modern Approach” cites hundreds of papers and books. There’s thousands of websites out there on the subject. There are many academic papers that are made available for free online. There are communities, like AIGameDev, dedicated to spreading that delicious knowledge.

    I think the hardest part about finding information is getting the terminology. It’s pretty dense when you first get into it, especially when talking about acyclic directed graphs, and your idea of a graph was like mine was about a year ago, namely a few bars on a 2D Cartesian axis. But once you’ve got a foot in the door of the lingo, it can become pretty accessible, and information starts becoming more bountiful. That foot in the door can be either a good, basic website, or in my case, a university level textbook designed to introduce people to AI.

AI is a big field, full of fascinating and interesting concepts and techniques, and it’s a young field that’s still full of potential. It’s not as complex or confusing as film and television would have you think. That’s not to say it’s a walk in the park, as I stated earlier, I’m probably running a 70% rate of understanding what I’m reading, but I’m managing. And if I can manage, so can you. So if you’re interested, there’s no better time to start than now.

A CMS in Ruby on Rails, and Why I Stopped

Monday, April 6th, 2009

For the last couple of months I have been writing a series of posts describing a personal Ruby on Rails based CMS. I have been writing tutorial style posts outlining what I was doing, and why I was doing it. Don’t bother trying to look for those posts, because I’ve archived them.

This decision was pretty easy after I slowed down and reviewed the situation objectively for a few minutes. That clarity, coupled with a few truths that hit home while reading the source of a few other Rails applications pretty much sealed the deal.

The next post in the series was going to be an overview of another Ruby CMS project, BrowserCMS. I saw a video from RailsConf about this project, and recognised a lot of their goals as coinciding with mine. So I was going to poke through their code, see how they’ve done things, compare it to how I did the same things, or was planning to do them.

While I was reading through their source, it occurred to me just how much I don’t get Rails. I have a bit of a poke around with scaffolding and get familiar enough with the generators and I decide that I’m ready to tell people how do create a CMS in Ruby, because I’ve nearly finished creating a CMS in ASP.NET.

It’s clear to me, now, how silly that was.

Rails isn’t just a framework for Ruby, it’s a whole change in paradigm. Intellectually, I know Rails is “opinionated software”, however I don’t think I understood what that really means. I tried to make my code as flexible and configurable as possible, and I was struggling with all but the most basic CRUD tasks.

So, first I realised I wasn’t ready to tell anyone how to do anything in Rails.

But I have a series of about 5 posts doing just that. What do I do with those? Do I continue on, plugging away at learning while trying to instruct? I decided the answer to that depended on what I had achieved with my posts. What was the value of them?

Then I realised that I had covered very little other than simple CRUD. Sure, I had some unusual object relations, like trees and self referential comments, and I implemented some business logic like compiling pages into templates, however the total sum of the non-CRUD related content could have fit into 1 post. So I spent 4 posts blathering away at how to achieve the same task as running script/scaffold.

What’s worse is that I realised I was back in the rut of creating a CMS. How on earth did I do that again, after stating loudly and proudly that I don’t even like programming websites.? Well, I told myself, you are planning on using this on your own sites, your many, many online business ideas. Sometimes you have to do the same old thing to earn money. Which is true, if I had done anything of worth, but I hadn’t, I was a long winded scaffold script. I got carried away in the joy of learning a new language/framework (which isn’t a bad thing), and fell into the familiar territory of doing what I always do (which is a bad thing).

So, now I realise that I’m not only trying to teach people something I don’t understand, I’m trying to teach them how to do something I don’t even like doing.

That’s stupid.

I’ve archived the posts. I may revisit, as I still have grand ideas of what a CMS should be, but for now I’m shelving the whole thing. Luckily I have such a small readership (read: none) at this stage, nobody will be affected. For that I’m grateful.

And yet, through all this stupidity, I have learnt a few things of value.

I’ve learnt that Ruby on Rails is more complex than I thought, and will hit the books again to pick up some more advanced techniques.

I’ve learnt that I’m scared to push myself in programming. This revelation is a pointy one. I love programming, it’s my job, it’s my hobby and it’s my passion. The fact that I’m scared to push myself to innovate, hiding behind the excuse that I’m not smart enough or I’m not creative enough is double edged. On one hand, it’s sad that I’m not as ambitious as I thought. On the other, it’s great that I know now, so that I can get stuck into remedying that.

I’ve also learnt a lot about blogging. Those posts of mine weren’t really providing much value. I don’t have many readers because I’m not saying anything new, and I’m not showing anybody cool things they can’t find in a hundred other blogs. I also didn’t have much of a personal voice, and was writing them like I would a textbook, which is missing the point of a blog. It’s supposed to be informal, and I’m supposed to show my personality. So, I’ve found my voice, while realising that I’m not providing value. Another pointy idea, but ultimately good, because that too is something I can work on.

So, inevitably, I have to question the value of this post. What could this post provide to someone who stumbles across this website? Well, a few things.

Firstly, self awareness is a great thing to possess as a programmer. If you know where your weaknesses are, you can train in them, and get better. But that’s not enough, you also need to know what you’re afraid of, so you can recognise that you don’t even know that it’s a weakness. That fear hides the existence of the things you’re not good of, and that’s a major roadblock to improving.

And lastly, if you don’t have anything to add to the conversation, don’t say anything. The whole internet is a conversation, and I’ve just been babbling to myself in a corner the whole time. If you want to contribute, add your own thoughts, your own interpretation on subjects that are well known. Go into new levels of detail on old ideas and technologies. Introduce new ideas or technology, or modify existing ones. Contribute, don’t just talk for the sake of talking.

Visual Studio 2008 Deletes LINQ DBML Designer File

Wednesday, April 1st, 2009

Do you spend hours wrangling with your “Interactive” Development Environment, trying to prevent it from deleting your boiler plate .designer file for your LINQ DBML? Do you get baffled watching Visual Studio delete said file from your software version control system, and then struggle to get the two back into sync?

Then you might be encountering this bug right here. The reason for this post is because this bug seems very hard to find information on in Google. So hopefully I can raise some awareness, and at least have it pop up on a Google search.

Typical symptoms are the designer file for the DBML being deleted when the DBML is updated. If you’re also using source control, Visual Studio will attempt to delete it from your source control at the same time. Renaming the DBML file will cause it to regenerate, however it can cause major headaches when combined with source control, removing it, adding it, pending delete actions hidden in a project you thought you had unchecked out.

The solution is the suitably moronic action of moving all your using statements inside your namespace declaration. A co-worker found this solution in this MSDN forum thread. Once you move your usings, Visual Studio happily stops deleting your designer file, and you can finally get back to work programming instead of fighting your tools.

In Defense of the Jerk

Sunday, March 29th, 2009

Not just any jerk, but the jerk who knows his stuff. The technically strong, socially weak programmer who does not play well with others.

During a recent employee review, I was told that I need to work on a few areas, namely the way I communicate with my co-workers, and my tendencies to shoot down ideas without supplying alternatives. I was also told that my technical proficiency was more than sufficient.

The technically capable but socially awkward programmer is a cliche in every day society, and the rogue programmer is a cliche in development teams. Seth Godin’s post, and another one that cropped up in Hacker News reminded me of this, and made me think. I realised that, to some extend, I am that jerk.

The point of this isn’t to toot my own horn, rather to try to offer an explanation on behalf of all well-meaning jerks in software development teams. Now, I’m not talking about those jerk jerks, who code maze-like code to secure themselves a job, those who are needlessly rude and abrasive, and those that ignore the other members of the team to the detriment of the whole project.

I’m talking about the jerks who are otherwise good employees. We’re not trying to sabotage the project, to exert our will over the project, to make anyone feel bad or insufficient.

We just care about the code.

At least, I do. I care about what I’m doing. I care about the project. I want my code to be the best code I’ve ever written, I want this project to be the most successful project the company has. I own my code, and take pride in it.

When somebody suggests something that I know won’t work, that will negatively affect the quality of the code, I freak out, somewhat. The first thing on my mind is to shoot down this idea, because I don’t want it reducing the quality of the project.

If my manager is planning on putting another programmer on the project with me, and I don’t believe they’re quite up to standard, I’m going to try to limit the damage they do to the application. I’m going to try and push them to work in a relatively isolated corner of the application. Unfortunately, I know this is a bad team work based attitude, however I can’t help it.

Now, I’m not defending jerks. Being a jerk is a bad thing, I know this. I agree with what my boss said, and agree that I need to work on these areas. However, instantly firing the jerk, if they’re a well meaning jerk, is not a good idea, because you’ll lose someone who is passionate about what they do, and constantly striving to improve them.

Instead, just try to work things through, and soften the edges of the jerk.

Apps Hungarian is Just Good Naming

Saturday, February 7th, 2009

Hungarian flag

Application Hungarian is just good variable naming convention, but worse, because it’s needlessly concise, and hard to read at a glance. It also comes from an environment that is growing more and more obsolete. More on that in a bit.

Firstly, clarification is needed as to the definition of Application Hungarian notation, as opposed to Systems Hungarian. All of this is relatively useless, however, if you are in the same situation I was in a few years back and don’t know what Hungarian notation even is.

Hungarian notation is a variable and function naming convention where you add either prefixes or suffixes to names. What these prefixes/suffixes mean depends on what type of Hungarian notation you’re using. Systems Hungarian is the method of adding information about the type of the value of a variable, or a return from a function. For example, a string that represents a name would be called “strName” using Systems Hungarian. Applications Hungarian is where you add information about the semantic meaning of the value of a variable.If this is a little foreign, that’s because most people think Systems Hungarian when they think about Hungarian notation.

In Making Wrong Code Look Wrong, which is what inspired this post, Joel Spoelsky outlined exactly what Applications Hungarian notation is, and why it’s so useful. He outlines that in Applications Hungarian, you add information about the semantic value of a variable/method return. To illustrate, he demonstrates that you can use Applications Hungarian to differentiate safe and unsafe strings in a web application environment. He uses the prefix ‘s’ to define a “safe” string, that is safe against XSS attacks (ie: A html encoded string). He uses the ‘us’ prefix to define an “unsafe” string, that is a string read directly from user input that may contain potentially dangerous code.

Spolsky also demonstrates using the notation to make wrong code long wrong. That is, by combining prefixes on methods, based on their return types, you can use the names as a sort of litmus test. So if the return prefix of a function specifies it returns a ‘us’ or unsafe string, and you assign it to a ‘s’ or supposedly safe function, this is clearly a logic bug, and looks wrong.

I think this is great, and it’s something I’ve never thought of, but my question is, how is it different from properly descriptive variable naming, aside from it’s somewhat obscure brevity?

The virtues of clear, descriptive names in programming have been sung again and again. If you make your variables and method names descriptive enough, your code becomes somewhat self documenting. You capture the intention and meaning in your code when you use variables like “customerFirstName” and “formatAddress”. You look at those names and you can immediately see the intention behind the code.

I see a huge overlap between the best practice of descriptive variable naming and Spolsky’s intention behind his use of Apps Hungarian, so much so that I don’t see a difference between the two, with relation to their purported goals. Apps Hungarian is more terse, and more obscure, and has somewhat of a learning curve if you’re not familiar with the nomenclature of the system. This, to me, seems like a needless barrier to understanding.

Am I missing something? Why is Apps Hungarian better than regular, descriptive and intelligent variable naming? I can’t see any benefit to using Apps Hungarian explicitly, and to my understanding, Apps Hungarian is just a more terse method of good variable naming.

Pixel Perfection is Impossible

Wednesday, February 4th, 2009

This post is an elaboration of a comment that I left on a post about CSS frameworks. I agree wholeheartedly with Joren’s position on CSS frameworks after having used 960gs for the layout of this site (at the time of posting this).. As I mentioned in my reply to Joren’s post, I used the framework to try to cut down the stress I feel when I try to make things line up properly and that most certainly didn’t work as intended.

Frustration

Specifically, I have a little widget that shows my most recent tweet on the front page, and I’m using a cute Twitter bird vector from DarkMotion.com as the background for this element. If you are viewing this post in IE, Chrome on a small resolution, and probably Opera and Safari, the bird is probably sitting obnoxiously below the recent post bar, getting in the way of content.

This little bird has given me so much grief in the design of this site, that I’ve thrown my hands up in despair and given up. The reason it’s plagued me so is the way 960gs automatically floats everything for it’s grid system. If you want to work outside the grid system, even a little bit, the framework makes you pay with blood.

I love the bird, I love showing my most recent tweet. I will not remove it. However, I can’t claim to be a serious developer, whilst still allowing obvious display bugs on the site, and I can’t bring myself to use a third-party theme to represent my personal blog. It just seems wrong.

In Joren’s post, there is an excellent comment defending frameworks, which makes mention that Joren’s blog has a 3 pixel difference in margins of the content, between the header and the footer. While I acknowledge that this wasn’t the author’s main point, it brings up something that I’ve never understood in web design, and the mentality of many web designers, both new to design and coming from print design.

A web page is a fluid medium. Web designers advocate abandoning the pixel for the em, citing greater flexibility and fluidity. So, at some level, they recognise this. However, people still insist on making designs that rely on pixel perfect precision positioning across all major browser vendors, which is where the need for CSS frameworks and reset stylesheets come from.

Web browsers are products competing in a market, and they have different ways of rendering CSS, and they have their own unique quirks. We know this, as well. Beyond glaring CSS interpretation bugs and discrepancies in box models, they have unique margins and padding on elements, and sometimes these only differ by a few pixels.

Why is this a problem?

Why do these people feel the need to contort CSS, torture it into burying these differences, into displaying  the same across each browser. There are far too many permutations of resolution and browser, not to mention whether or not someone has their browser full screened (I don’t) to try to enforce this pixel perfect design. 

Embrace the differences. Embrace the fact that your margins may not look identical. Recognise that at the design stage, and design your sites to compensate for this. Don’t split images into different elements if they’re intended to fit together seamlessly. Find another way. Don’t stress if the margin between your content and your header is 3 pixels bigger on IE than it is in Firefox. As long as the general communication goals of your design work across all browsers, and it doesn’t look like a train wreck, be happy.

I know all of this is easier said than done. I’m a server side programmer, not a CSS guru or a designer. However, I think it’s a worthy to keep in mind while designing. Don’t fight the diversity, embrace it.

Using Twitter to Gather Information

Saturday, January 31st, 2009

Birds

I’ve mentioned briefly the virtues I see in Twitter when I discussed how to implement a project in 3 hours. Now I want to expand on how to use Twitter to gather information. I mentioned in that the key to utilising Twitter to gather information was to follow the right kinds of people. However, you’ll find that when you start following more than 100 people, things get a little hectic.

Every programmer is more effective with a solid selection of tools, and Twitter is no different. TweetDeck is the best Twitter desktop client I’ve used, and it’s an AIR app, so it runs on all major operating systems. Most Twitter clients have a single stream that Tweets come in through, and everything gets lumped together. TweetDeck’s strongest feature, for the purposes of leveraging Twitter to gather information, is the option of splitting your stream into groups.

TweetDeck allows you to put people you follow into groups, and have their tweets appear in a different window. The notification alert will also show this segregation. In addition to splitting your followers, you can specify windows that will feed you the results of searches, using the Twitter search operators outlined here.

The search operators and TweetDeck are the engine of our information gathering machine. Using the search operators you can follow the public stream and cherry pick Tweets that contain information you want. You can look for specific phrases, specific hashtags, tweets after a certain date and tweets with links.

For example, I use use a search for “artificial intelligence” with links to find people who tweet interesting links concerning artificial intelligence, which is a subject I’m highly interested in. I can run several of these filters in different windows at the same time, and grok the whole public stream. No longer am I restricted to listening only to those that I follow. Of course, that’s not to say I no longer follow people. If someone pops up multiple times in my filters, I’ll follow them to be sure not to miss a single tweet from them.

This method, however, isn’t flawless. If you look too narrowly, you won’t find any tweets. Also, when you finally do find tweets, even by trying to pick out key phrases/keywords and links, there will be a significant signal to noise ratio. My search for AI topics often lead to students linking university class websites and time tables, which is something I’m obviously not interested in. Often people who pop up in searches all the time turn out to have a high noise to signal ratio.

Having said that, while there are drawbacks, the ability to monitor the public stream as well as the people you follow, and sift out tweets that have a higher likelihood of being relevant is a powerful thing, and will often connect you to bits of information you would have never found otherwise. That’s the power of Twitter, and that’s why I love Twitter.