Monday 24 September 2012

Beta

In the world of stuff, there's things that you know, and things that you don't know.  It can sometimes feel like all of us are simply questing to try and know more and more stuff.

But there's another way of looking at the world of stuff.  We can also split stuff into the things that other people know, and also things that other people don't know.  It can sometimes be confusing thinking about what other people know.  Fortunately we live in a social world so we should all be good at it.

In any case, Mathematicians, like me, sometimes like to use Venn Diagrams to describe these kinds of relations:


We can then build fun subcategories, like "Secrets", which contains all the stuff that we know, but that noone else does.

Or "The unknown" - that's all the stuff that noone knows about.  Yet.

But there's another interesting subcategory that I wanted to talk about today.  I'm sure there's a great German word for this category, but I don't think it has a name in English. It's that category of stuff that other people know (about us), but that we don't know about ourselves.  It's all the stuff like:

  • What does the back of my head look like?
  • What was everyone laughing about just before I walked in to the room?
  • Why do I always struggle to open doors?
And the all-consuming:
  • Does this pair of jeans make my anorak look out of proportion?
Even in principle, this stuff is unknowable to us. Why? Because even if we were to ask a trusted friend, there's a complex web of social contracts which ensures the information they relay to us be 100% believable, yet have no basis whatsoever in reality.

Beta Testing


When we're making video games, there's a number of tools we can use to improve the quality of the game prior to launch.  One of the most useful is Beta Testing.  Of course, we do all manner of testing when making video games, but Beta Testing is the one which can really make or break a game.

In a true Beta-Test, all of the intended features are actually present in the game in a playable state.  It might even be one of the first builds where this is true.  The game is then played by people who are outside of the development team.

In a (true) Beta-Test, it's the development team looking for feedback about all the stuff that other people know.


So if you're ever asked to Beta Test some software, what does that look like?


Well first things first, remember the social contract?  That's not what a Beta Test is about.  The dev team isn't looking for praise or apologies, they want solid, genuine feedback about the features in the game. Be that an emotional response ("it felt great when..."), or something actionable ("I couldn't reach the exit because...")

Naturally they're looking for confirmation about the stuff they (think they) already know, so be sure to include some high level comments about the stuff that seems obvious to you, the stuff that "...everybody knows...."

More valuable to the dev team is going to be feedback about stuff they don't know.  Try and tell them about the bugs that only you encountered.  Try and share anecdotes that are relevant, but that will be unknown to the dev team.  Try and give them access to the areas they can't otherwise see.


The Beta Blog Post


So all of this is stuff that I know, and after reading though, now it's stuff that you know too.  I happen to think this is important stuff that everyone should know, but I may be in the minority on that one.

However, there is a chance that this blog post contains serious defects.  For example, I know that this post still has lots of room for improvements.

For that reason, I'm officially declaring this blog posting to be a "Beta Blog Post".  What that means is, I think it contains most of the good ideas, but there might be stuff about this topic that I don't know (yet) or haven't covered adequately.

Why not help me out and use the comments section below to give me some feedback, maybe an emotional response, or even something actionable?   ;)

1 comment:

  1. Hey Chris, I find this piece of post very interesting. This is also a problem that we also face in software engineering or computer science during usability testing. We call it the The Observer-expectancy effect , where the participants does not act naturally or gives inaccuate responses when they consciously know they are being watched. The current solution that I can think of (on the spot) is that to take a large enough sample of people and hopefully the inaccurate results will show up as being outliers, but with that there is still a possibility that the majority of people may still give you inaccurate data.

    ReplyDelete