70 réponses à ce sujet

[Allan Schumacher]

I am just pulling this out into its own thread just in case others are curious about the answer, without taking the other thread too far off topic.  Plus it feeds into my narcissistic tendencies when people actually care about what a QA guy is really like: 

Avejajed wrote...

I'm just really fascinated by the
technical aspect of making games. For instance- I always thought QA came
rather far down the line (meaning- after there was already a fair bit
of content, or nearly done) , but really you guys are with the game from
the very beginning? Everyone's sort of working all at once? What's the
difference between QA and other types of testers?

I apologize if I'm asking stupid questions.

Not a stupid question at all.  I think most people see a movie like Grandma's Boy or something and assume ALL QA is like that.

Your assumptions aren't that far off.  Even at BioWare, the QA staff ramps up as development approaches release because, as you say, there's more content to test and we typically make large games which can use a good amount of eyes in a variety of different perspectives.

There are advantages to having dedicated QA though, and I say this mostly to validate to any important people that may read this that they should continue to employee me!


I'm a software programmer by education, so I actually know how to program and have a bit deeper understanding of how software development happens programmatically compared to say, just an average, really passionate video game player (note that a QA that is "just" a passionate video game player could very well be much better at critiquing story, setting, etc. etc. than I would ever be

As a result, I typically join projects earlier in development, and for DA3 I was definitely embedded into the digital acting team early into "preproduction" when a lot of things are still vague concepts and general ideas and not anywhere close to being "in game."  I am able to look at potential features from a different perspective and help the programmers and designers come up with edge cases that may have been overlooked.  So early on I actually read a fair bit of documentation in terms of what our ideas are for how to implement some of the core features.  Frankly this is useful for me just so I have an idea on what and how to test features once they come down the pipeline.

Without boring you with the actual day to day, a big advantage of this is that typically, the earlier in development you can find a bug, the cheaper it is to fix it.  Really, the first level of QA is typically from whoever implements the code/feature.  A programmer will add a feature and run his/her own tests on it to make sure it is working properly.  I am basically the liaison between production managers and the programmers in that nothing is "truly" done until I sign off on the listed feature, as I'm that second set of eyes to make sure things are working both as the programmer intended, as well as what the customer is expecting.  For example, a programmer may think "Yup, all done" and it doesn't have any bugs in it, but for example a writer might look at it and go "Ohhh, this wasn't really what I was thinking" or "This feature is not at all what I thought I was asking for!"  So tasks get set up with a level of "acceptance criteria" that the Cinematics Lead feels the feature needs to be able to do.  This helps the programmer know exactly what is being requested and helps me know what the key areas are to test.

If I find any issues with a feature, bugs get filed and they are triaged.  Serious issues that show the system isn't really working at all, or issues that may prevent other people from working, are escalated as "blocker" issues that have to get fixed ASAP.  Other might be minor usability things with easy work around or only rare edge cases that we usually allocate for "bug weeks" when the team focuses on fixing bugs rather than creating new features.


On top of testing new features as they come in, I create and design "regression tests" either through automation (scripts and programs that test things automatically) or through a relatively structured test level to help set up test situations.  I created a "smoke test" level (testing if things are on fire) that I run daily to make sure core Digital Acting systems still work.  During this I'll make sure that the conversation wheel is still working, plots still work, stage systems are still working, Voice Over still plays etc.  Not very in depth (or it'd take all day), but a quick pass for critical features to make sure that people are able to work.  If John Epler can't set a stage mark while creating a scene, he comes to me and says "WTF Schumacher you bum!" (after trying to persuade me to play DayZ...), so I try to help make sure that doesn't happen.  Sometimes a presumably unrelated change someone had made on, say, the exploration team, has inadvertently caused our conversation cameras to go all wonky and now we need to get it working!


We do have "content QA" right now as well, and we're slowly starting to get more and more as more content goes Live.  Even early though, we'll get people taking passes over the story and checking for consistency and making sure the narrative makes sense, and so forth.  And all this is just for the conversation/cinematics type stuff.  There's other "tech QA" types that are embedded into combat, audio, level design, and so forth.


Towards the end of the project, when less and less new features are coming into the game, we spend less time regressing the systems, and I start to become more of a general playthrough tester.  Usually they find a way to still leverage my programming skills (such as focusing on trapping already reported issues for programmers to investigate directly in code... basically saving the programmer the time of reproducing a complicated bug with a debugger attached to it), but sometimes it's just "Hey, another person to play through the game!?  Okay, we haven't had a playthrough with Choices X, Y, and Z made, with the following party members.  Can you do that for us?" and I go yessir and play through the game more traditionally.


So that's a not so abridged response to your question


EDIT:  Poking fun at John aside, he and the rest of the team have actually been remarkably helpful in terms of helping set up test content as well, as well as going over their workflows and whatnot and helping me learn new systems.  I think we're a pretty solid team if I do say so myself

Modifié par Allan Schumacher, 03 octobre 2012 - 07:08 .

[LPPrince]

Thank God for paragraph breaks. haha

Good read.

[Allan Schumacher]

Some of my professional feedback has actually been to improve my conciseness :innocent:

Though the way I see it posting here is only partly professional so I have no problems rambling on to you guys.

[Sable Rhapsody]

Thanks for the peek into the development process. It's always really cool to see how you guys go about making these games, and it provides a perspective most of us don't have.

And we ramble too.  Just look at the character threads 

Modifié par Sable Rhapsody, 03 octobre 2012 - 07:16 .

[Dominus]

A fair amount of that may be over my head, but it's a healthy dose of QA/Game-Related info. I was wondering what I should peruse at 3 O'clock in the morning, and found it.

If John Epler can't set a stage mark while creating a scene, he comes to me and says "WTF Schumacher you bum!"

I could see that play out.

[Meltemph]

PSH! Sounds like a cake job... >_>

But umm, I'll just stick with being on the road most of the year...

Seriously though, nice information on the early parts of keeping the game clean(or cleaner). Good insight on the proccess for sure, even if it is posted quite late at night. Make me feel like I'm not the only insomniac.

Modifié par Meltemph, 03 octobre 2012 - 07:20 .

[LPPrince]

Allan Schumacher wrote...

Some of my professional feedback has actually been to improve my conciseness :innocent:

Though the way I see it posting here is only partly professional so I have no problems rambling on to you guys.

Its all good. Some of us appreciate going into long elaborate detail. I'm one of them. 

[Direwolf0294]

Good stuff. I always enjoy learning how the behind the scenes stuff that the customer never gets to see is done.

[Ninja Stan]

Avejajed wrote...
Are "bug weeks" done intermittently through the development process, or are these weeks that are tackled after the game is "complete"? Meaning, do you guys stop creating content say, once a month, to work out bugs, or is that usually something you do when the game is nearing completion? I understand that you probably come across bugs on a day to day basis- but when you're actually stopping to look specifically for them.

"Bug weeks" are dedicated time for the dev team to deal solely with bugs, either to catch up on a backlog of bugs already filed or to bash on the game as it currently stands in order to get as many bugs filed and fixed as possible. These can happen on a regular basis or when certain milestones are reached.

That's just the dedicated time assigned to bugs. The rest of the time, bugs are still found, filed, and fixed, but usually while a whole bunch of other stuff is going on, like creating content.

Is it difficult to switch from engine to engine? I'm ignorant of how engines like Frostbite work- is it something that you would need to be familiar with to do what you do? Is it something you guys can easily deal with- switching from the DA2 engine to the Frostbite engine, based on education? Do they teach you that in school? Does everyone have some kind of "frostbite 101" course or are the basics generally the same no matter what engine you use?

Game engines form the foundation of a game, like a giant set of Lego, with which you can build anything you can imagine. Some things will be easier to build than others. If you have a castle set, for example, it would come with horses, so building an environment that incorporates horses would be easy. That set might make it difficult to make an alien spaceship, but it's still possible with all the pieces in the set. If, on the other hand, you had a set of Space Lego, but still wanted horses, you could still do it, but you'd have to make the horses manually, which requires more work.

When switching from one engine to another, or switching from one giant set of Lego to another, you can still do much of the same things, just in a different way. You may be upgrading to a set, for example, that has wheels, which allows you to build all manner of ground vehicles really easily. Maybe your engine (or Lego set) now has the ability to make vehicles easier to incorporate into the game, like the Lego Power Functions Motor Set.

Each engine will have its strengths and weaknesses.

Are "content QA"'s specifically chosen for certain things- say, if you work on combat as a QA will that be all you do through the entire process? Or do you switch it up so you have fresh eyes. If I was a QA would I work combat one week and then level design the next, or always combat.

QA changes tasks as required, both to get fresh eyes on a given system and because systems are at different stages of development and require different levels of attention. A system that's still being built will have different testing requirements than a system that is implemented and needs a shakedown, and that has different testing requirements than a system that is complete and needs polish.

And do you find that it's easier or harder to have testers that are fans of the game- or testers that are not personally tied to the game in any way? I would imagine that it would be more subjective for content testers to..say..not be snatched up from the BSN, if that makes sense. What exactly do QA people have degrees in? Is it usually some sort of programming?

Many QA have their education in the computing sciences, though Content QA can be more liberal arts-oriented. I did Content QA for much of my time at BioWare, and I have a theatre and performance background. Regardless, anyone interested in QA should have excellent oral and written communication skills, at the very least.

[Cutlasskiwi]

Very interesting, thanks for taking the time to write and post this Allan. It's always interesting to get a peek behind the scenes.

[Kidd]

Thanks for a very informative post, Allan! Was a great read. I love posts like these the most, not only cause I'm a huge fan of BioWare, but also because it's nice to be prepared for what life just might be in the industry should I ever manage to enter it =)

EDIT: And thanks to Stan as well =)

Allan Schumacher wrote...

people actually care about what a QA guy is really like: 

*pats Allan*

Allan Schumacher wrote...

There are advantages to having dedicated QA though, and I say this mostly to validate to any important people that may read this that they should continue to employee me!

I promise I'll do my part in sending mean cupcakes should your employment status ever end without your consent =) Deal?

Allan Schumacher wrote...

Some of my professional feedback has actually been to improve my conciseness [smilie]http://social.bioware.com/images/forum/emoticons/angel.png[/smilie]

That's my future performance review riiiight there. I'm feeling you. Walls of text just bring the point forward with no risk of misunderstandings, no?! ;D

Modifié par KiddDaBeauty, 03 octobre 2012 - 08:22 .

[Icinix]

Avejajed - I think you're taking this Inquisitor business rather serious BA DOM TISH!

I kid - I kid.

This thread is a really good read - I'm studying games and entertainment design and this is excellent stuff.

[Foolsfolly]

Stanley Woo isn't at BioWare anymore? That's, apparently, exactly what this thread has taught me.

[Battlebloodmage]

Although people often say that DA3 is Bioware's last chance at redemption after what some perceived as "screw-up" for DA2 and/or ME3. I always am under the assumption that DA and ME are handled by 2 different teams with different staffs, it's not fair to really attack Bioware as a whole since both series are developed by different groups and have different schedule and development. I could understand not being a fan of one series, but it's a bit ridiculous to hate on a whole company and their sister games just because they are not satisfied with one particular series. I'm kinda wondering about the staff situation. If there is a shortage in staffs, do they take people from one team and assign it to another to catch up with the schedule?

[Allan Schumacher]

Stan has great responses but I'll just top some of them off with my (more limited experience compared to Stan) perspectives too

Avejajed wrote...
Are "bug weeks" done intermittently through the development process, or are these weeks that are tackled after the game is "complete"? Meaning, do you guys stop creating content say, once a month, to work out bugs, or is that usually something you do when the game is nearing completion? I understand that you probably come across bugs on a day to day basis- but when you're actually stopping to look specifically for them.

Stan summed this up pretty well.  Bug weeks are where the focus is on fixing (and to some extent, finding) bugs.  I personally find it to be a bit of a challenge to see if I can find more bugs than get fixed on my team.  It's tough though as there's 9 programmers and one of me!

In order to keep things stable though, you get to a point where you need to stop/slow adding new things and make sure the existing stuff is working properly.  Building new stuff on a shaky foundation leads to badness, compounded.  You get systems that end up working around a bug elsewhere, and when that original bug is fixed, the system that worked around is now broken, and if any other systems depended on THAT system, well, I think you get the idea.

Is it difficult to switch from engine to engine? I'm ignorant of how engines like Frostbite work- is it something that you would need to be familiar with to do what you do? Is it something you guys can easily deal with- switching from the DA2 engine to the Frostbite engine, based on education? Do they teach you that in school? Does everyone have some kind of "frostbite 101" course or are the basics generally the same no matter what engine you use?

Stan has much more experience in this regard than I do.  Frostbite is my first engine transition (DA2 was just an extension of DAO's engine), and I don't think I found it much more challenging than I expected.  It's going to be able to do a lot of the same things that a different engine can do, it's just sometimes the workflow isn't the same so you need to learn that.

For the super techie/geek answer, I find it similar to learning a new programming language.  The programming principles I know still generally apply, but I need to learn the syntax and the way it does things in order to do the things I did in a different language.

Are "content QA"'s specifically chosen for certain things- say, if you work on combat as a QA will that be all you do through the entire process? Or do you switch it up so you have fresh eyes. If I was a QA would I work combat one week and then level design the next, or always combat.

I think I'm a bit unique, in that I've actually been a part of the Digital Acting/Tools team since DA2 (on DAO, I was a fresh, wet behind the ears grunt on a contract and mostly executed testplans specifically for the DAO designer toolset, which isn't that far off in what I've done).

I consider myself quite specialized, in that I have basically been working with the same team for over 2 years.  Which means I have some advantages with Frostbite, because I'm very familiar with how we did things in DAO/DA2 so I can quickly go "Oh, we'll need this in order to be able to do everything we could do in the old toolset."  But we still try to mix up tasks.  I have found there's been a bit more mobility among the other Tech QA groups, with really only one other guy that has been static in his placement.  Though I have found BioWare to be pretty accommodating, in that if I had asked to try out a different part of the project they'd haev been okay with that and tried to make it work.

And do you find that it's easier or harder to have testers that are fans of the game- or testers that are not personally tied to the game in any way? I would imagine that it would be more subjective for content testers to..say..not be snatched up from the BSN, if that makes sense. What exactly do QA people have degrees in? Is it usually some sort of programming?

I think almost all of the Tech QA team has some degree of technical proficiency.  Most have some form of programming education, either a CompSci degree or something equivalent from a technical school.  I think content QA is a bit more flexible.  The "content" guy for Digital Acting has an English degree, which I presume helps him in going over what the writers write! 

As for the types of fans?  In most cases I'd say people are, if not necessarily fans of BioWare specifically, certainly fans of gaming.  If you hate gaming QA won't be for you (it might still work as a programmer or something crazy technical).  But I think it's good to have a healthy mix.  I have a lot of experience with BioWare's games, which means I have some assumptions with the control schemes and whatnot.  There's someone that has a lot of PnP RPG experience, but less video gaming experience, and it can be useful because she'll not have as many implicit assumptions as I have so something that I feel is fairly intuitive may not be so intuitive for her.  That's useful feedback for sure.

[Allan Schumacher]

Avejajed wrote...
So, who makes sure everything doesn't get too chaotic? It seems like it could be a bit overwhelming with so many people doing so many different things. Who keeps everyone on task? I know each department has a "lead" or "manager"- and I'm sure each of those heads have meetings with each other- is that usually done weekly? Daily?

I have a "stand up" meeting with the Digital Acting guys as well as the Tech QA team every day.  Quick synopsis of what's going on which takes about 15 minutes each.  The Tech one happens after the other meeting, just so I can pass on any critical issues that might be happening (Cameras are broken, don't file bugs, fix by end of day expected type of stuff)

Taking DA2 for an example- were the "acts" broken up? So everyone at the studio, from writers to QA area ll working on "ACT I" at the same time, then the whole team moves on? Or on any given day would one person be working on Act I and someone at the desk next to them would be working on Act III?

This, sadly (and I'm working on improving my visibility here), is where I'm less knowledged.  Since I am working with system developers, I'm a bit separated with the order the content was made and added into game.  In many ways it's a lot like a movie I think, though.  Everyone working in the exact same area leads to "too many cooks in the kitchen" and "parallelizing work" can allow work to be done a bit faster, I find.

I'm just trying to work it out in my head how in the world you guys keep things running smoothly- working on content then having to stop and go back to something you've already moved on from to fix it- then trying to get your head back into what you were doing before. It seems to me that it would be really difficult. Probably just experience?

This is in part why it's so valuable to find bugs early.  If I can find a bug shortly after a programmer worked on a system (or better yet, WHILE he's working on it), then he doesn't have much of a context switch to fix it.  If he were to look at it a year later, he may have to go back to figure out what the heck he was doing there the first time around!

[Allan Schumacher]

If there is a shortage in staffs, do they take people from one team and assign it to another to catch up with the schedule?

There is sometimes internal mobility yes. Though it's never affected me personally. Though if a programmer, for example, is particularly good with memory management, and there's some memory issues on the other project the chances of going out and lending a hand is pretty decent.

Context switching between projects can incur a cost though, especially if done too frequently. There's overhead in terms of that person having to get reacquainted with the systems, codebase, reporting styles, etc. Personally I have only been a part of the Dragon Age team, and I'd say most of the people on the team that's probably pretty true. Most of the time I'd say any new faces I see are typically outright new hires, rather than people up from Mass Effect. Not all though.

[Allan Schumacher]

So when your looking for bugs is it all text based~programming code..or is it actual gameplay? Do you have to go back and recheck an area after another layer of..say..detailing is added?

I don't usually look at code directly, no. When I do, it's usually because I've found a bug and I have an idea what might be causing it, or just to straight up help a programmer using it.

I do a lot of work in the toolset directly. A new feature will go in and I'll make sure that it's working as expected, as well as make sure that it's not crashing the toolset. For example, someone may add a field to a conversation, so when we open up the conversation editor, on the side there's a drop down box setup that indicates what state the conversation is in (e.g. First Draft, For Review, Ready for VO, Locked Down).

For a change like that, I might do some simple cursory checks to make sure the rest of the editor is still working, but it's low risk for something like that. After a while you start to get an idea for what types of changes need more thorough testing beyond just the feature based on how risky they are.

For instance, if we needed to change the way the conversations were saved to improve streaming, that's a high risk change because data integrity is crucial. So I'll not only make sure that things can save in general, but I'll make changes to pretty much ALL aspects of the conversation and make sure that they are not only being properly saved, but that those changes are being properly recognized when I load it up in the game too, and will try to test all aspects of the conversations in game as well.


99% of the the QA guys don't spend much time in code though. If they're testing levels they're usually testing them in game (same with combat). The audio guys will make sure that audio tools work the way we expect in the editors, as well as in game, and so forth.


EDIT: Sleep now overcomes me.  Will check for follow ups tomorrow.

Modifié par Allan Schumacher, 03 octobre 2012 - 09:11 .

[Ninja Stan]

Avejajed wrote...

Thanks, NinjaStan. Correct me if I'm wrong but you used to work for Bioware, correct? In any case thanks to both you and Allan for responding and answering with detail.

Yes, I left the company back in April atfer 11 wonderful years as part of BioWare's awesome QA team.

So, who makes sure everything doesn't get too chaotic? It seems like it could be a bit overwhelming with so many people doing so many different things. Who keeps everyone on task? I know each department has a "lead" or "manager"- and I'm sure each of those heads have meetings with each other- is that usually done weekly? Daily?

Usually, each individual task group has a couple of heads, one more dedicated to managing the team day to day and one more dedicated to maintaining the schedule. Between these heads, the task group sticks to its scheduled tasks and milestones or have a really darned good reason why they didn't or couldn't. In turn, these heads get their timelines and headcount from project management, who operate on a more macro scale and guide the entire project. Project management, in turn, both runs the project schedule and liaises with their counterparts on the publisher side of things, keeping them in the loop and bringing up any major issues that might affect the project.

Meetings with and between various heads can happen daily, weekly, or monthly, depending on the group and what information needs to be passed along or gathered.

Taking DA2 for an example- were the "acts" broken up? So everyone at the studio, from writers to QA area ll working on "ACT I" at the same time, then the whole team moves on? Or on any given day would one person be working on Act I and someone at the desk next to them would be working on Act III?

The game isn't worked on "chronologically." The Design team, for example, lays out the basic groundwork for the plot of the entire game and details the various major characters that will play a part in the story. Art will be designing character concepts and colour palettes based on Design's outlines. All the while, the programmers are making the engine work and dealing with feature requests from the rest of the team. Once the foundation is laid, then individuals set to work on different parts of the game. The Design team parcels out the writing and scripting assignments, Art will assign work to individual artists and animators, and programming assigns work to individuals as well. And those task groups I mentioned before are formed.

So yes, it is possible, depending on seating arrangements, for one person to be working on Act I and the person next to them to be working on Act III.

I'm just trying to work it out in my head how in the world you guys keep things running smoothly- working on content then having to stop and go back to something you've already moved on from to fix it- then trying to get your head back into what you were doing before. It seems to me that it would be really difficult. Probably just experience?

You misunderstand a little bit. You don't "move on" from anything. At any given time, something you've already worked on might break due to something someone else has done, or new content placed into the game, or new features added, or, you know, Tuesday happens and the build machine decides to take up smoking and skip school that day. But because it's something you've already done and you know how you did it, troubleshooting and fixing the problem can be much easier than starting from scratch (depending on th pissue, of course). And these are professional developers, this is what they do!

[The dead fish]

Cutlasskiwi wrote...

Very interesting, thanks for taking the time to write and post this Allan. It's always interesting to get a peek behind the scenes.

Agreed. Thanks Allan and Stan.