14 réponses à ce sujet

[Allan Schumacher]

I am just pulling this out into its own thread just in case others are curious about the answer, without taking the other thread too far off topic.  Plus it feeds into my narcissistic tendencies when people actually care about what a QA guy is really like: 

Avejajed wrote...

I'm just really fascinated by the
technical aspect of making games. For instance- I always thought QA came
rather far down the line (meaning- after there was already a fair bit
of content, or nearly done) , but really you guys are with the game from
the very beginning? Everyone's sort of working all at once? What's the
difference between QA and other types of testers?

I apologize if I'm asking stupid questions.

Not a stupid question at all.  I think most people see a movie like Grandma's Boy or something and assume ALL QA is like that.

Your assumptions aren't that far off.  Even at BioWare, the QA staff ramps up as development approaches release because, as you say, there's more content to test and we typically make large games which can use a good amount of eyes in a variety of different perspectives.

There are advantages to having dedicated QA though, and I say this mostly to validate to any important people that may read this that they should continue to employee me!


I'm a software programmer by education, so I actually know how to program and have a bit deeper understanding of how software development happens programmatically compared to say, just an average, really passionate video game player (note that a QA that is "just" a passionate video game player could very well be much better at critiquing story, setting, etc. etc. than I would ever be

As a result, I typically join projects earlier in development, and for DA3 I was definitely embedded into the digital acting team early into "preproduction" when a lot of things are still vague concepts and general ideas and not anywhere close to being "in game."  I am able to look at potential features from a different perspective and help the programmers and designers come up with edge cases that may have been overlooked.  So early on I actually read a fair bit of documentation in terms of what our ideas are for how to implement some of the core features.  Frankly this is useful for me just so I have an idea on what and how to test features once they come down the pipeline.

Without boring you with the actual day to day, a big advantage of this is that typically, the earlier in development you can find a bug, the cheaper it is to fix it.  Really, the first level of QA is typically from whoever implements the code/feature.  A programmer will add a feature and run his/her own tests on it to make sure it is working properly.  I am basically the liaison between production managers and the programmers in that nothing is "truly" done until I sign off on the listed feature, as I'm that second set of eyes to make sure things are working both as the programmer intended, as well as what the customer is expecting.  For example, a programmer may think "Yup, all done" and it doesn't have any bugs in it, but for example a writer might look at it and go "Ohhh, this wasn't really what I was thinking" or "This feature is not at all what I thought I was asking for!"  So tasks get set up with a level of "acceptance criteria" that the Cinematics Lead feels the feature needs to be able to do.  This helps the programmer know exactly what is being requested and helps me know what the key areas are to test.

If I find any issues with a feature, bugs get filed and they are triaged.  Serious issues that show the system isn't really working at all, or issues that may prevent other people from working, are escalated as "blocker" issues that have to get fixed ASAP.  Other might be minor usability things with easy work around or only rare edge cases that we usually allocate for "bug weeks" when the team focuses on fixing bugs rather than creating new features.


On top of testing new features as they come in, I create and design "regression tests" either through automation (scripts and programs that test things automatically) or through a relatively structured test level to help set up test situations.  I created a "smoke test" level (testing if things are on fire) that I run daily to make sure core Digital Acting systems still work.  During this I'll make sure that the conversation wheel is still working, plots still work, stage systems are still working, Voice Over still plays etc.  Not very in depth (or it'd take all day), but a quick pass for critical features to make sure that people are able to work.  If John Epler can't set a stage mark while creating a scene, he comes to me and says "WTF Schumacher you bum!" (after trying to persuade me to play DayZ...), so I try to help make sure that doesn't happen.  Sometimes a presumably unrelated change someone had made on, say, the exploration team, has inadvertently caused our conversation cameras to go all wonky and now we need to get it working!


We do have "content QA" right now as well, and we're slowly starting to get more and more as more content goes Live.  Even early though, we'll get people taking passes over the story and checking for consistency and making sure the narrative makes sense, and so forth.  And all this is just for the conversation/cinematics type stuff.  There's other "tech QA" types that are embedded into combat, audio, level design, and so forth.


Towards the end of the project, when less and less new features are coming into the game, we spend less time regressing the systems, and I start to become more of a general playthrough tester.  Usually they find a way to still leverage my programming skills (such as focusing on trapping already reported issues for programmers to investigate directly in code... basically saving the programmer the time of reproducing a complicated bug with a debugger attached to it), but sometimes it's just "Hey, another person to play through the game!?  Okay, we haven't had a playthrough with Choices X, Y, and Z made, with the following party members.  Can you do that for us?" and I go yessir and play through the game more traditionally.


So that's a not so abridged response to your question


EDIT:  Poking fun at John aside, he and the rest of the team have actually been remarkably helpful in terms of helping set up test content as well, as well as going over their workflows and whatnot and helping me learn new systems.  I think we're a pretty solid team if I do say so myself

Modifié par Allan Schumacher, 03 octobre 2012 - 07:08 .

[Allan Schumacher]

Some of my professional feedback has actually been to improve my conciseness :innocent:

Though the way I see it posting here is only partly professional so I have no problems rambling on to you guys.

[Allan Schumacher]

Stan has great responses but I'll just top some of them off with my (more limited experience compared to Stan) perspectives too

Avejajed wrote...
Are "bug weeks" done intermittently through the development process, or are these weeks that are tackled after the game is "complete"? Meaning, do you guys stop creating content say, once a month, to work out bugs, or is that usually something you do when the game is nearing completion? I understand that you probably come across bugs on a day to day basis- but when you're actually stopping to look specifically for them.

Stan summed this up pretty well.  Bug weeks are where the focus is on fixing (and to some extent, finding) bugs.  I personally find it to be a bit of a challenge to see if I can find more bugs than get fixed on my team.  It's tough though as there's 9 programmers and one of me!

In order to keep things stable though, you get to a point where you need to stop/slow adding new things and make sure the existing stuff is working properly.  Building new stuff on a shaky foundation leads to badness, compounded.  You get systems that end up working around a bug elsewhere, and when that original bug is fixed, the system that worked around is now broken, and if any other systems depended on THAT system, well, I think you get the idea.

Is it difficult to switch from engine to engine? I'm ignorant of how engines like Frostbite work- is it something that you would need to be familiar with to do what you do? Is it something you guys can easily deal with- switching from the DA2 engine to the Frostbite engine, based on education? Do they teach you that in school? Does everyone have some kind of "frostbite 101" course or are the basics generally the same no matter what engine you use?

Stan has much more experience in this regard than I do.  Frostbite is my first engine transition (DA2 was just an extension of DAO's engine), and I don't think I found it much more challenging than I expected.  It's going to be able to do a lot of the same things that a different engine can do, it's just sometimes the workflow isn't the same so you need to learn that.

For the super techie/geek answer, I find it similar to learning a new programming language.  The programming principles I know still generally apply, but I need to learn the syntax and the way it does things in order to do the things I did in a different language.

Are "content QA"'s specifically chosen for certain things- say, if you work on combat as a QA will that be all you do through the entire process? Or do you switch it up so you have fresh eyes. If I was a QA would I work combat one week and then level design the next, or always combat.

I think I'm a bit unique, in that I've actually been a part of the Digital Acting/Tools team since DA2 (on DAO, I was a fresh, wet behind the ears grunt on a contract and mostly executed testplans specifically for the DAO designer toolset, which isn't that far off in what I've done).

I consider myself quite specialized, in that I have basically been working with the same team for over 2 years.  Which means I have some advantages with Frostbite, because I'm very familiar with how we did things in DAO/DA2 so I can quickly go "Oh, we'll need this in order to be able to do everything we could do in the old toolset."  But we still try to mix up tasks.  I have found there's been a bit more mobility among the other Tech QA groups, with really only one other guy that has been static in his placement.  Though I have found BioWare to be pretty accommodating, in that if I had asked to try out a different part of the project they'd haev been okay with that and tried to make it work.

And do you find that it's easier or harder to have testers that are fans of the game- or testers that are not personally tied to the game in any way? I would imagine that it would be more subjective for content testers to..say..not be snatched up from the BSN, if that makes sense. What exactly do QA people have degrees in? Is it usually some sort of programming?

I think almost all of the Tech QA team has some degree of technical proficiency.  Most have some form of programming education, either a CompSci degree or something equivalent from a technical school.  I think content QA is a bit more flexible.  The "content" guy for Digital Acting has an English degree, which I presume helps him in going over what the writers write! 

As for the types of fans?  In most cases I'd say people are, if not necessarily fans of BioWare specifically, certainly fans of gaming.  If you hate gaming QA won't be for you (it might still work as a programmer or something crazy technical).  But I think it's good to have a healthy mix.  I have a lot of experience with BioWare's games, which means I have some assumptions with the control schemes and whatnot.  There's someone that has a lot of PnP RPG experience, but less video gaming experience, and it can be useful because she'll not have as many implicit assumptions as I have so something that I feel is fairly intuitive may not be so intuitive for her.  That's useful feedback for sure.

[Allan Schumacher]

Avejajed wrote...
So, who makes sure everything doesn't get too chaotic? It seems like it could be a bit overwhelming with so many people doing so many different things. Who keeps everyone on task? I know each department has a "lead" or "manager"- and I'm sure each of those heads have meetings with each other- is that usually done weekly? Daily?

I have a "stand up" meeting with the Digital Acting guys as well as the Tech QA team every day.  Quick synopsis of what's going on which takes about 15 minutes each.  The Tech one happens after the other meeting, just so I can pass on any critical issues that might be happening (Cameras are broken, don't file bugs, fix by end of day expected type of stuff)

Taking DA2 for an example- were the "acts" broken up? So everyone at the studio, from writers to QA area ll working on "ACT I" at the same time, then the whole team moves on? Or on any given day would one person be working on Act I and someone at the desk next to them would be working on Act III?

This, sadly (and I'm working on improving my visibility here), is where I'm less knowledged.  Since I am working with system developers, I'm a bit separated with the order the content was made and added into game.  In many ways it's a lot like a movie I think, though.  Everyone working in the exact same area leads to "too many cooks in the kitchen" and "parallelizing work" can allow work to be done a bit faster, I find.

I'm just trying to work it out in my head how in the world you guys keep things running smoothly- working on content then having to stop and go back to something you've already moved on from to fix it- then trying to get your head back into what you were doing before. It seems to me that it would be really difficult. Probably just experience?

This is in part why it's so valuable to find bugs early.  If I can find a bug shortly after a programmer worked on a system (or better yet, WHILE he's working on it), then he doesn't have much of a context switch to fix it.  If he were to look at it a year later, he may have to go back to figure out what the heck he was doing there the first time around!

[Allan Schumacher]

If there is a shortage in staffs, do they take people from one team and assign it to another to catch up with the schedule?

There is sometimes internal mobility yes. Though it's never affected me personally. Though if a programmer, for example, is particularly good with memory management, and there's some memory issues on the other project the chances of going out and lending a hand is pretty decent.

Context switching between projects can incur a cost though, especially if done too frequently. There's overhead in terms of that person having to get reacquainted with the systems, codebase, reporting styles, etc. Personally I have only been a part of the Dragon Age team, and I'd say most of the people on the team that's probably pretty true. Most of the time I'd say any new faces I see are typically outright new hires, rather than people up from Mass Effect. Not all though.

[Allan Schumacher]

So when your looking for bugs is it all text based~programming code..or is it actual gameplay? Do you have to go back and recheck an area after another layer of..say..detailing is added?

I don't usually look at code directly, no. When I do, it's usually because I've found a bug and I have an idea what might be causing it, or just to straight up help a programmer using it.

I do a lot of work in the toolset directly. A new feature will go in and I'll make sure that it's working as expected, as well as make sure that it's not crashing the toolset. For example, someone may add a field to a conversation, so when we open up the conversation editor, on the side there's a drop down box setup that indicates what state the conversation is in (e.g. First Draft, For Review, Ready for VO, Locked Down).

For a change like that, I might do some simple cursory checks to make sure the rest of the editor is still working, but it's low risk for something like that. After a while you start to get an idea for what types of changes need more thorough testing beyond just the feature based on how risky they are.

For instance, if we needed to change the way the conversations were saved to improve streaming, that's a high risk change because data integrity is crucial. So I'll not only make sure that things can save in general, but I'll make changes to pretty much ALL aspects of the conversation and make sure that they are not only being properly saved, but that those changes are being properly recognized when I load it up in the game too, and will try to test all aspects of the conversations in game as well.


99% of the the QA guys don't spend much time in code though. If they're testing levels they're usually testing them in game (same with combat). The audio guys will make sure that audio tools work the way we expect in the editors, as well as in game, and so forth.


EDIT: Sleep now overcomes me.  Will check for follow ups tomorrow.

Modifié par Allan Schumacher, 03 octobre 2012 - 09:11 .

[Allan Schumacher]

That being said, I had previously asked this question of Allen, but I'll ask again: the new Unreal engine is being designed to be not just a graphic and physics powerhouse, but also to use significantly less direct programming in order to implement features. The team's stated goal was to be able to have non-programmer developers be able to develop lots of real, active content and be able to test it as soon as they design it, without having to send it to a programmer to make into a reality, relying on programmers to only do the last 10%, of tying the code to the overall end product.

I don't know if I entirely understand the question, because what you describe has been my impression of the way engines and games have been for quite some time now. Do you have any type of example?

[Allan Schumacher]

Epic has been touting this for a while now in relation to the Unreal 4 Engine, that it is designed to have everything able to be created with very little actual coding experience required. So an artist can adjust the lighting mechanics to best reflect their vision of a dungeon, or a writer can create multiple variables and outcomes to a situation that can be tested immediately afterwards to see if the experience is what they anticipated, etc., etc. The design goal for the new engine is to stem the tide of resource bloat with video game development, resulting in less content needing to be designed, created by a programmer, tested by a QA analyst and then returned back to the designer to see if the finished product is what they had in mind.

IMO what Epic is describing is nothing new. The quote you're mentioning is just continued improvements to the tools to allow faster turnaround for designers. It's still a significant programming investment to get things set up like that. But the goal of "faster iteration" is always paramount. If you always have to spend 30 minutes building assets to make a small change, you lose a lot of time, and in some ways Frostbite does indeed have large improvements to verifying this sort of stuff compared to Eclipse.

In other ways Eclipse is still superior, in that Eclipse had better conversation previewing and whatnot because stuff like that didn't really exist in Frostbite. But it's the type of thing that my team is actively working on so that people like Mr. Epler can better preview their changes with as little time wasted as possible. On the plus side, these are problems that were already solved with Eclipse, so it's just a matter of seeing what it would take to work with Frostbite.

But with the aspect of live changes (making changes to the level/scripts/whatever while the game is running), yes Frostbite does allow for stuff like that. It's also super easy to fire off relevant events and whatnot without having to design specific console commands or anything like that. I can set up a conversation, quickly fire off some events in the editor to set various plot states, and then start the conversation up in game and see if it's behaving the way I expect based on the plot state.


Tools have long been developed with the idea of an ease of use in mind, because the reality is that there's a lot of creative story teller types that just don't know how to program. I actually worked on a project while at University, called ScriptEase, which worked with the NWN Aurora Toolset and was designed explicitly to allow people that don't know how to program to create scripts. We ran experiments with high school English courses where they would create stories in Aurora. It was pretty amazing actually. Programmers don't really create much content unless they kind of have a dual role. Epler isn't a programmer and never looks at code, but he's the one that creates the in game content. None of the programmers on the digital acting team are ever directly responsible for any of the explicit content you see in a game. They don't frame the shots for cinematics, they don't place the triggers that start up a conversation and move the guys around. The programmers create the tools for the designers to be able to do all this stuff on their own.

1. Is bioware working on an agile type of development process?(that is if you are in the position to answer)

Yes. We do employ agile development. Our scrum work is usually split into 2 or 3 weeks of work with short term deliverables.

2. Problem with software today is how not a lot of firms are able to release the software on time and on budget. This causes qa to have the mentality of not trying to catch all the bugs but rather to let in as few bugs as possible(I prefer this method as you could have a ship but you need it to sail at some time). When it comes to making games is it a similar mentality?

I still try to "catch them all" in that I try to find as many as I can. The reality is that not everything can be caught though. Just for reference sake, if 100,000 people pick up the game day one and play it for one hour, they just spent about 34 years worth of time into the game assuming that I am working 8 hours a day for 365 days a year. Or, if we have 34 people QA guys going crazy, a year's worth of time. And there's no way that all that time was all spent on JUST the first hour of the game! So yes, towards the end of the project the focus definitely shifts to game breaking type of issues in terms of what gets fixed simple due to "zot management."

[Allan Schumacher]

What do the digital acting guys do? This may sound stupid so forgive me, but I always sort of imagined that after the "coding" is done by programmers, that the process was a little like animating movies in a way. Where on a computer screen there are guys who take an arm and move it to where it needs to go next frame by frame. But I'm thinking that's not what happens at all?

So the Digital Acting team is made up of a bunch of programmers and Cinematic Designers and similar type of support staff like animators and stuff like that.

Frostbite has a toolset to put content in the game, and the work the programmers typically do is create new features/editors in this toolset as well as the underlining game code to make sure the systems work in game.

So to use your example regarding animating characters in a scene, programmers would work on the tools to allow animations (which will get created in something like 3dsmax or Maya or something) to be used in the Frostbite toolset, and properly recognized and animated in the game. The animations themselves are created by an animator that's familiar with whatever animation tools are made, and then a Cinematic Designer will take said animation work and place it in the scene and manipulate the cameras and whatnot to make it look good in game. The programmers only really worked on the underlying infrastructure to let the content guys work their magic with as little programmer involvement as possible.

[Allan Schumacher]

Haha, the context of a question like that can be tricky, though your answers aren't particularly poor haha.

I often find myself with a pretty diverse gaming experience, so when I ask people what their favourite games are, they'll usually mention one (or several) that I have played. So I'll ask them questions about those games. What they like, what they would improve. Get them talking about games and looking at them from a critical perspective.

Although QA need not really a "mathy" type of position at all.


In terms of a valuable job quality, the idea of having a good eye for detail is great, specifically with respect to how to reproduce issues. Even if you struggle to describe precisely what you're seeing, if you can make it easy for someone to reproduce the issue you're seeing, that's probably 90% of the battle. A lot of time can be wasted, and bugs missed and not fixed, if other people aren't able to understand what they need to do to reproduce the bugs.


Having a desire to want to release something that is kickass, and having pride in being associated with said product really helps stay motivated in keeping to these details throughout the project.

[Allan Schumacher]

It'd likely still need some level of technical proficiency if you were looking at leveraging it into creating/improving physics in the engine (i.e. we do have Physics Programmers I believe...). Although I know a fair number of Physicists at the local University that are actually pretty proficient programmers haha.

One of the Tech QA guys happened to have a postgraduate degree in Physics even, although he recently left to head back East for family reasons. As a result his ability to test Physics was pretty specific (he also knew how to program as well though haha. Must be fairly common in the curriculum?)

[Allan Schumacher]

This may not be able to answered here, but how do the writers fit in
with all of this? Do they have programming degrees, or English and
performing arts degrees? I don't think there is exactly a career point
for writers to enter video games (at least, not even remotely easily).
And do they get to see the process (i.e. maybe picking over a facial
expression, or a particular staged bit? I'm sorry, I'm not sure if I
make any sense)

Hmmmm, IMO the idea is to create tools in such a way that minimize the "general technical proficiency" so that writers (and heck, even designers) are able to make cool stuff without needing programming degrees or to be experts with many of the technical details.

My impression based on our tools for Eclipse and now Frostbite is that there is no need for programming/technical degrees in order to be a capable writer.

For the details of writing and picking expressions and whatnot Gaider or another writer would be much better equiped than I. I know that scenes are commented with context for what's happening, but I can only assume it's the writers doing it. I don't know if it actually is.

A new thread, or perhaps posting in this thread might solicit a more direct response from writing

Modifié par Allan Schumacher, 04 octobre 2012 - 08:13 .

[Allan Schumacher]

And don't let the absence of the preferred type of degree stop you. My degree is nontraditional for my job. It only took them 3-4 years to figure out that I'm not a statistician and had spent the majority of those 3-4 years writing software. They changed my job title and I continued to write code.

There's a universal aspect of having a degree that I really like: someone was able to start something that isn't very easy, and then also finish it. This last point is really telling and I think is a great character trait.

Not that people that don't have degrees may not have better types of commitment to getting tough things done, but rather I can look at an applicant and have a bit more definitive proof that at least one time they have demonstrated this, regardless of what the degree is actually in


In general I agree with your advice though. There's a lot shared between general software development. We have some people in QA that used to QA tax software and they do just fine here

[Allan Schumacher]

I like my Avatar! It's traditionalist in that sense!

[Allan Schumacher]

I do fancy myself to be a GUID.