Scott Johns
6/14/2002
Click here to see the Classroom!

In last week's article I went over the kinds of decisions that go into forming your testing gauntlet. This week the focus is on the real meat and potatoes: the play decks. Unlike the test decks, which are built with the idea of being as generic (and representative) as possible, the play decks are those builds which the team is actually considering running. For this article, I'm going to focus in on some of the common mistakes I see teams make at this stage, as well as giving some goals that I aim for when my own testing gets to this point of the process. Even in a multi-part series like this one, the subject matter we're delving into here is still way too big to cover in any kind of comprehensive fashion. With that in mind, consider this a look at some of the more important issues regarding this stage of your process, at least as far as I'm concerned.


The Process
Playtesting is a process. Much like early (and most) math, each step of learning gives you the tools needed to master the next step, and so on. So far I've covered the Alpha Stage and Test Decks, and even in that short list I'm sure you can already grasp the incremental nature of this method. In fact, one of the easiest ways to illustrate this is by just working backwards from our goal - The point here is to build the best deck for the environment - to do that, we need to choose the deck that beats the field most reliably - to know which deck does that we have to be able to test potential decks against the projected field, refining these decks until one of them stands out - to do that, we need to have a projected field to play against in the first place - and to know that, we have to have a solid comprehension of the defining factors of the environment itself.

So the point here (I have one, honest!) is that the better work you've done on the previous steps, the better you're going to do here. The bad news is, the worse you've handled the previous steps, well, you get the picture.

Presumably by the time you've reached this stage, you have a set of test decks that you feel gives you a decent feel for what you expect to face. Now that you have that gauntlet of decks up and running, you're ready to build your own decks to throw at them. Assuming you've got a good enough gauntlet in place, it's often just a question of taking the time to test everything out as thoroughly as possible until something rises to the top. While you're doing that, here are a couple things to keep in mind:


Playing the Metagame
The more focused a metagame becomes, the more clear its defining characteristics tend to become, which in turn tends to open the door for "metagame decks" (decks that may be weaker in raw power, but which matchup well against the expected field). An easy example of this was last season's Pro Tour Tokyo. The format was Invasion block constructed, and there turned out to be very few viable decks, at least in the early stages. Not only were there just a few expected decks (GR beats, UBR heavy control, BR beatdown), but there also turned out to be a single defining element to the expected metagame:

Red was everywhere, including almost all of the true creature removal.

As I mentioned in the Alpha Stage, one of the fundamental goals to this whole process is finding those factors that really define the environment. The more accurately you can identify these, the better equipped you are to exploit them. To find answers, you first need the questions. In this case, the format as we expected to see it turned out to be almost entirely based around Red removal. Nearly every deck we were testing, and all of the decks we expected to be popular, shared that in common. Once that realization was made, our team got the breakthrough we were looking for. By taking a Blue/White aggrocontrol design which had been moderately successful in testing, and then adding four Crimson Acolytes to the main deck, suddenly we had a powerhouse on our hands, at least in terms of performance. With the exception of Fact or Fiction (and sometimes Meddling Mage), the deck actually felt fairly under-powered in terms of raw card power, but because we had a good match for the expected decks, we now had a real contender on our hands. A couple weeks later Zvi won the whole thing with "The Solution" - a deck specifically crafted as a foil to the format's defining characteristics.

Note by the way that as IBC later evolved with more cards and more experience in the pool, building pure metagame decks became more and more difficult. The broader the environment's expected field becomes, the tougher it tends to be to build a pure metagame deck.


"Pet Decks"
One of the important factors to good playtesting is having agreement between your team's members as to what your goals are. Having a realistic concept of how much time people are expected to be able to spend working on decks and matchups is important, because the amount of time you have will often be the single most important factor in determining how many decks you can include in your gauntlet, how much time you can spend testing the various matchups, refining the decks, testing them again, working on them some more, and so on.

Because few of us actually have the kind of time (or even desire) to actually test an environment with extreme thoroughness, sacrifices will need to be made along the way. I bring this up under the above heading because I've seen the subject of pet decks often cause trouble on playtesting teams.

An example of this is a team with six actively testing members, but one of which always wants to test exotic creations which almost surely won't be played by anybody on the team once the event comes around. And because time is limited, the feeling can come to be that a full third of the team is in many ways wasting their time, since the player in question is also taking up the time of whomever he is testing against.

The answer to this issue is two-fold. What are the team's goals, and how likely is this player to come up with useful tech? First, as I mentioned above, teams that share common goals tend to be the most successful, at least in my experience. If half the team is doing everything they can to qualify, there's a lot more potential for issues when the other half of the team is mostly in it for the social aspect. Whether or not the group has a real problem with the guy who's in it to play crazy decks with very little chance of ever being used will depend very much on the group's reasons for playing together in the first place. If the real goal is just getting together and having fun (and there's certainly nothing wrong with that), this guy is probably going to be a great teammate. If however the team is actually trying their best to qualify, a real problem may be in the works.

Since the situation is just fine if the rest of the team isn't gunning to dominate the event, I'm going to turn to the opposite scenario. In this case, the first question needs to be one of honest evaluation - how much time can the team realistically afford to allow this member (and whoever he's testing against) to spend on the decks in question. And along those lines, just how likely are these decks to ever get played? If the guy just spends his time "testing" decks that clearly are never going to be a contender, his process is clearly not fitting the team's goals, and he may be better off with a more casual-minded group.

But there are also plenty of times where it's worth it to have this guy in your group. With enough talent, these are often the guys who manage to come up with decks that nobody else will see coming, and if they are competitive, you've gained a significant advantage. Alternatively, even if the decks themselves don't end up cutting it, specific cards or strategies may present themselves which might otherwise have gone overlooked.

To reduce it down to math, the real question becomes how much time can your team afford to give this guy to work on his mad experiments, and how likely are they to produce something of value to the team worth the time invested. To use a more personal example, I've been working on teams with Alan Comer since the 1997-98 season, and I can promise you that most decks he comes up with will never see the light of day. But, Alan is also a really good deck builder and a very good player. Because the teams I've been on have often tended to be on the larger side (8+ players), it's always been worth it to let Alan go nuts. The reason is that plenty of great things will come from him in return for the time invested, and the size and commitment of the group means that there's enough other people testing that we can afford to have somebody like this.

One important point here needs to be made though: under no circumstances should any significant testing time be spent with players testing their pet decks against each other. The whole point to having a gauntlet is so that you know which decks needed to be tested against. If two decks are being tested against each other and neither of them is in the gauntlet (and correctly so), all you're really doing is testing a matchup that isn't likely to even occur. And while some value can come of this in terms of coming to a better understanding of the decks in question, the point is that nearly any testing will have some value. But when time is limited and your goal is performance, you need to be more efficient with your time than this.


Shifts
Which leads to another common occurrence at this stage. Often teams will get into this part of the testing process and right away a certain deck (or decks) will emerge as more successful against the gauntlet than the rest. And suddenly the gauntlet decks start really gunning for this deck in particular. The power deck in question isn't in the gauntlet itself, but the gauntlet is shifting to compensate for it.

The real question here is, should the deck in question be moved into the gauntlet? Any time one of your play decks starts really standing out, the question that needs to be answered by the team is, "how likely are we to face this deck?" The more strongly you answer yes, the more important it is to move a copy of this deck into the gauntlet, because now you expect this deck to be part of the metagame, and thus something you should be testing against. As with anything else involving the gauntlet, there's a considerable amount of speculation and prediction involved, and so care needs to taken to evaluate the situation as honestly as possible. One recent team I was on would, nearly without fail, come up with a powerful deck that handled the gauntlet well, but which (to me at least) seemed extremely unlikely to show up in the hands of others in any significant numbers. But, since the deck was doing so well, the other decks would all shift to smash this new deck in question, going so far as running maindeck hate cards that would never have otherwise shown up. The result was that these decks suddenly seemed a lot worse than they really were, since now the gauntlet was gunning for them too specifically. Knowing when a deck belongs in the gauntlet or not is often one of the most difficult questions you will face as a team.

Knowing when a deck belongs in the gauntlet or not is often one of the most difficult questions you will face as a team.

The best advice I can give you is to take the long view and, after every event, take a good honest look back on your testing and see where your predictions went wrong (and right!). If a play deck is doing great against the gauntlet and you don't realistically expect others to show up with it in any real numbers, the worst thing you can do is bias your gauntlet against this potential tourney winner, possibly convincing yourself that the deck isn't as good as it seemed. Another good way to avoid this mistake is by not biasing your gauntlet decks against specific play decks too early in the process. Make sure you've got a good feel for this stage of the testing before you start doing anything like running maindeck hate to beat your #1 deck.


Record Keeping
Some things in testing are pretty cut and dry, but this is one of those grayer areas as far as I'm concerned. My own personal opinion is that data is a good thing, as long as you collect it with as much impartiality as possible. While I don't generally start keeping any real logs of my testing until I've begun to get a real feel for which decks I think might be contenders, once I've reached that point I tend to log nearly all of my matches. Because typically most of my online testing is with Zvi, I'm going to use the process we tend to use as illustration.

First off, we alternate which deck goes first each game, and if we're actually at the point where we're keeping records, we'll generally stick to an even number of games to keep this fair. The things I typically record are: which deck went first, which deck won, were there any mulligans (and how many?), and a quick explanation of what the most important factors seemed to be. The farther into testing we are, the more detailed that last step tends to be, both because we tend to have a better understanding of the decks (and thus, hopefully, what is actually happening) and also because at that stage of the testing being able to communicate the important facts to the rest of the team becomes much more important.

Another thing we often do is switch the decks around if one player seems to be gaining any significant advantage (or vice-versa). Some decks will perform much better for certain members of the team, and having a good understanding of this is crucial when it comes time for each player to choose their own deck. Just because a deck is great for your friend doesn't by any means decide that you have to play it as well. With proper testing, you'll have a better idea of which decks work for you.

However, this also illustrates the fact that data can be dangerous. If Zvi can win with a deck 80% that I can only take to 50%, combining our results into one number and reporting it back to the team can be misleading. Because of that, Zvi and I both tend to be fairly suspicious of data from others. It's not that I don't trust other people, it's just that statistics are a whole hell of a lot more complex than the simple numbers they look like, and as such I think they should always be taken with a very healthy grain of salt.

And with that in mind, I should point out another, somewhat less common, problem I've run into in the past: the guy whose deck always seems to beat everything 90%+. I'm not going to name names on this one simply out of respect, but suffice it to say that some people (including one very well known player I worked with in the past), intentional or not, will report statistics that just don't jibe at all with your own understanding of things. For whatever reason, whatever deck they like at the time reportedly smashes everything, even though nobody else can duplicate the results. However, the more you're able to test the matchups yourself as well, the less impact this is likely to have on you. Also, my experience has been that this kind of problem tends to be chronic. Those who do it tend to do it most if not all of the time, so you'll usually know pretty early on if and when you need to be on guard against this kind of thing.


Bringing it all Home
Like many of the other steps in a strong playtesting process, this stage is cyclical. You and your teammates will come up with an initial gauntlet, you'll build your first set of test decks, and then you'll bash them into each other a bunch. Once you've done this enough times to get some idea of what's going on, you'll adjust one or both sides and do it all over again. Remember that the real goal here is to get the best gauntlet possible, and then find the deck(s) that can handle it. While there are plenty of ways to maximize your efficiency, there is no substitute for knowledge and the actual experience of practicing. The more time you spend well here, the better your results will turn out, at least on average. In next week's article I'll get into the final piece of the puzzle to the process - the one thing that I'm still convinced more people screw up than any other step.

Until then,
- Scott Johns