Aller au contenu

Photo

Overview of Dragon Age launch, and where we go from here


7 réponses à ce sujet

#1
Fernando Melo

Fernando Melo
  • BioWare Employees
  • 639 messages
First and foremost, I have to congratulate and say thank you! to all of you BioWare and Dragon Age fans for making the launch of Dragon Age: Origins exceed all of our expectations. It has been incredible to see the fan reaction to the game, the overwhelming game activity levels online and on the BioWare Social site - a special thank you to those that PM'd myself and other team members about how much you are enjoying it.  

Looking at the various data and positive feedback we’ve received so far it has been an exceptional launch for both BioWare and Dragon Age: Origins. However, it was also clear that there were a few bumps and teething pains along the way that has had an impact to a small, but important portion of our community. 

Despite our best planning for the launch of a single player game of this magnitude we have to admit we clearly underestimated the desire for players to want to play connected, their pleasure at sharing their stories on the BioWare Social Site, and the overall level of interest in the launch DLC (downloadable content).  

Ultimately all these factors worked together to make for a rocky start for some of you – for those that did experience or are still experiencing any issues, our sincere apologies.

However, I wanted to share with you some of the things we are following, to reassure you that we are continuing to work on improving the launch of Dragon Age: Origins even further, and more importantly where we are at in terms of addressing these. 

Social site and game connectivity


As most fans now know, BioWare has launched a new Social site for all of our fans to discuss their games on the forums, to share their in-game achievements, to upload screenshots and much more. If you are reading this outside the social site and have not visited yet, you can register on the Social site here: http://social.biowar...m/user_home.php

When Dragon Age: Origins launched, we expected things to get busy on the Social site as fans redeemed DLC content, used the faces they created in the Character Creator and more. We anticipated a big increase of traffic and having passed a slew of load testing we were confident that we were ready.

Unfortunately, no matter how well you prepare, sometimes things don’t work out. The process that updates the social website failed overnight on Monday 2nd and Thursday 5th due to volume of traffic.  Social has a master server on which all content, posts, etc, are uploaded to. From there, for load balancing reasons, that content & the page updates are actually copied over to a number of ‘slave’ servers – these are what you and I actually connect to and view our social pages on.

With the process to update these display servers dying, it meant that over time each of these display servers began to fall further and further behind what was actually being uploaded to the master server – in terms of content, posts, etc. As such, when each of us was being directed to one of the display servers to view one of the web pages we were occasionally bouncing between older/newer content.  And even something simple like refreshing the browser or make a forum post would place you on whichever next available slave server there was, which could again be anywhere from minutes to hours behind ‘reality’ - leading to it appearing that your post etc never happened.  Ultimately each display server would end up receiving no updates, making it appear that the social site had zero updates for hours at a time - even users online would time out on the display servers, eventually looking like no one was on the site. 

While no data was lost – everything still arrived at the master server – it created a very confusing site experience for everyone. It took a few hours working non-stop with the great folks at EA IT to resolve this issue, and once resolved it was a slow process to then get all of the missing data to each slave server in order to get caught up over time to ‘reality’.   Before a more robust solution was put in place (on Friday 6th), we had a similar failure again on the 5th due to the start of European traffic / game launch.

As of noon on Friday November 6 this was fully resolved. Steps have already been put into place to not have this occur again and the Social site is now up and running smoothly.

A similar major contributor to general early game issues for some players was also due to sheer server load. Our initial estimates were created working with EA IT staff who have managed launches for numerous successful multi-player titles including FIFA, Madden, and the Battlefield series, as well as Spore –our expected closest relative in terms of online usage. We put together an initial estimate for server capacity that seemed appropriate based on those figures, in conjunction with a barrage of load capacity testing on DA:O itself in the months leading up to launch. 

Much closer to launch, we revised our figures coming out of the data we saw from the stand alone character creator app launch – just about doubling capacity yet again from our initial figures. For good measure, we then set in place a contingency plan – our plan B – to allow for further rapid deployment of additional capacity to yet again nearly double that amount, should things really get going.

As it turns out, before the end of the first day (Nov. 3rd - US & Asia launch) we had not only exceeded our initial capacity, but our plan B as well! For a single player game with online features, this was unprecendented and the levels we were seeing was completely unexpected.  Had this been an MMO, such as TOR, the initial setup and type of setup would have been very different indeed and we'd be laughing even with traffic in the orders of magnitude higher - but for a single player game no would predicted it would exceed this type of setup.

Since then the IT and online teams here at BioWare and EA worldwide have been steadily ramping up additional capacity – for peak usage on Wed Nov. 4th, for example, we did not have any significant server issues, and at the time of writing this we are now maintaining a steady buffer of capacity over usage. It appears we have managed to get ahead of the launch curve. 

Still, in expectation of the European and rest of world launch, and our first weekend where we anticipate our greatest traffic load yet the IT teams are literally building additional servers at our request to keep capacity ahead of load.

Having gotten over the initial wave, we’ve been monitoring our forums and working in collaboration with EA’s customer support group, to track and resolve the most common remaining issues that players have been reporting to us.  

DLC (Downloadable content):

Some players have reported that they have been unable to see their DLC (on PC). DLC is delivered through our custom content updater service, so this needs to be up and running. The normal game installer should correctly set this up on any version of windows. 

However, it appears that some digital download service installations were not correctly setting up the service – as such, those games were effectively ‘blind’ to the DLC on the servers. In addition, despite extensive testing on all supported version of Windows, even with the non-digital version of the game there is a small sub-set of configurations where it appears that the service is still not installing correctly.

There are some manual work-arounds in the BioWare and Stardock & D2D forums. In addition, we are working on an update to address the remaining configurations.

If you still are not seeing any DLC, we apologize – please bear with us as we get our updates out for these unforeseen cases. In the meantime, you should confirm that you have the DLC attached to your account (through code redemption or purchase), by looking at the registered game promotions page on the social site (http://social.biowar...ntitlements.php). As long as the content is there you are good, you will not lose that content. We ask for your patience while we find a way to deliver that content dependently to your specific configuration as soon as possible.

Digital Download versions:


A segment of players, predominantly with digital game versions from a variety of e-stores, had encountered an issue where they had first registered their game key on the social site, and subsequently were unable to authenticate their game with their respective digital game service.

There was an issue resolved here on social which should now prevent this problem moving forward. If you are still experiencing this problem please contact EA support as they have a fix for this already.

With a segment of digital consumers there have been some cases where the online authentication system used was also impacted by server load, or experienced a different issue due to a specific system setup not seen before – this could manifest itself as a ‘could not connect to EA Core’ or ‘WC.dll’ error. We are following this issue closely with the team that handles this technology and hope to have a solution shortly.

This system also uses Adobe Air was its web interface. As such, there have been some individual cases where the player uninstalled Adobe Air and was subsequently unable to authenticate the digital version of the game. At this time, the best solution is to keep Adobe Air installed (or to re-install).

A small segment of players also reported download issues with EADM (when purchasing through EA Store). This is being investigated as well. For now, the best solution appears to retry the download.

On reflection:


Releasing any PC title these days is always a challenge. Releasing a PC title on a brand new game engine with this many online moving parts, and as a new OS is launched, makes it that much more interesting.  Still, when we look at the actual total number of players who are happily playing DA:O without any issues, it is a real testament to the level of quality assurance for a title of this scale and across the three platforms shipped on.

Despite the amount of testing conducted at BioWare, EA worldwide, and compatibility testing at various labs including the major hardware manufacturers, nothing ever quite prepares you for a piece of technology hitting the wider gaming audience and the mind boggling permutations of hardware, software, drivers, firewalls and custom configurations out there.  

We really appreciate everyone’s patience as we work to iron out these last wrinkles that are impacting some of you. 

Know that we are continuing to monitor your feedback here on the forums, as well as throughout EA’s support network, including FAQ usage, email and phone calls – there is actually a significant number of BioWare and EA staff lurking through the various forums on social site, our previous forums and EA support sites with the sole purpose of capturing this feedback. Those issues are collated and prioritized on an on-going basis and already being planned against a roll out game feature updates and fixes, which we'll continue to do as needed. 

The relative number of staff posts doesn’t do justice to the effort that actually exists behind the scenes - please understand that our priority remains in working on any issues. We have a dedicated post launch team, which includes not only the DLC creation team but also individuals devoted to supporting the DA toolset, and releasing new features and updates to social site and the game – and this team will be supporting Dragon Age: Origins exclusively for quite a while.

As I wrap this up, I can see more and more of our European players (and beyond) going online for the first time and enjoying the game and DLC – we’re looking forward to hearing more from you, and to deliver even more content in the future to expand your DA:O experience, in addition to continuing to address these issues as they come up. We hope you will continue enjoying DA:O as much as we loved making it.

(edited for spacing)

Modifié par Fernando Melo, 07 novembre 2009 - 03:14 .


#2
Fernando Melo

Fernando Melo
  • BioWare Employees
  • 639 messages
Thanks again for your understanding and overall positive reception of the game itself.



A quick note that if you are still experiencing issues, please make sure you raise those either to EA support or on the tech support forums here on social if you haven't already done so - there is a lot of eyes on that forum in particular, and also many of the issues already have workarounds in those threads either officially or community generated.



These solutions continue to grow by the minute thanks to the great contributions from all of you - I also just have say, my sincere personal thanks to those of you taking your own time out to help others out on the community by the way! It is greatly appreciated.



As a general update as to how we're tackling things - all the issues raised so far are falling into 4 broad categories.

1) Issues around redeeming codes / and social site:

A lot of these were down to traffic load, or not clear enough instructions for code redemption. We believe this group of issues are largely sorted out now. There are still some folks who are reporting missing their actual DLC item codes (mostly from some digital retailers), and our last info is that those emails have now gone out (or are going out as of Friday). Please contact your digital store if you still did not get these.



2) Issues around Downloading dlc / updater service not installed correctly/running

We're continuing to work on these and should have an update coming soon. In the meantime, there are a lot of threads and FAQ items on help.dragonage.com now with possible workarounds, depending on your actual flavour of the issue. Please make sure you try those out.



3) Hardware related issues

This is the more traditional kind of PC game issues around a launch. It can include crashes, performance, OS related, and general hardware permutation issues.



We're prioritizing game blockers from these (e.g. game won't start), which thankfully are extremely low. But expect to continue to deliver updates for other non-game blocking issues such as instability over time, or performance on specific hardware cases, as well as work with our external hardware partners to get updated drivers out as appropriate. That's sort of par for the course with PC given the variety of hardware etc out there.



4) General game bugs

This too is something we suspect will come up over time given the sheer volume of content and ways to do things in DA. So far, it has been exceptionally low as well - a current example being people losing their DLC items after dwarf noble origin. (i saw a workaround in the help.dragonage.com faq btw)



Thanks,

F.

#3
Fernando Melo

Fernando Melo
  • BioWare Employees
  • 639 messages
Thanks again for the continued feedback. Apologies for the hiatus, along with several Bio folks I was struck down with the flu last week - slowly starting to catch up once again. Fortunately, behind the scenes the devs here have been continuously monitoring your feedback, filing these as bugs internally, and working on a lot of these issues - it is worthwhile doing a quick update.



Somebody raised a point about the 1.01a patch as a fail. Yes and no. It was tested - nothing leaves the studio without that - but anytime you put out a fix there is always the potential that something else trips on edge cases, despite very hard attempts through our own internal QA as well as our external partners and EA to cover all the bases. For 1.01a specifically, we definitely need to take our share on the chin as it was something we were trying to fast track - something we're not keen to repeat.



In the case of 1.01a for what it is worth we know that it actually did help more folks than those that had reported issues, for example. That's not any consolation for those affected by it though, and since then we've certainly put a lot more process in place for how we'll be doing patches.



By now you've probably seen the 1.01b update. This was actually reacted to very quickly after the first reports came back from 1.01a, and we also began working directly with some of you - affected players of the 1.01a - on a case by case basis, while doing our full test passes in parallel. Only once that passed through the test process and the 'live' cases was that actually rolled out at large.



In the meantime, we put in place a process of monitoring the forums & EA support, and began working on the launch issues in earnest. We are now very close to releasing a 1.02 patch - we're hoping for next week, but obviously all dependent on how the rest of the test cycle goes.



While 1.01 (both a & B) were primarily issues you reported from the character creator app (pre-game launch), as well as things we caught ourselves after the PC game was locked but before it launched; 1.02 will actually target many of the issues you've been reporting since game launch, in addition to general performance improvements and other fixes made on the consoles (after PC was locked) that the PC will benefit from.



Given the timing of the full test cycle for patches, 1.02 has basically been 'closed' for what fixes it will provide for a bit now, and things will only be removed or touched at this point if they fail the test cycle, nothing will be added.



In the meantime we have already opened the 1.03 patch list internally, and this contains remaining issues that we are tracking from your forum posts and EA support, and were unable to address in time for 1.02 - either due to timing of when those issues were raised, or they are still being investigated, or a solution is still in progress - or issues that fall out of the 1.02 test cycle and are not severe enough to hold up the 1.02 patch.



No timing yet for 1.03 - let's get 1.02 out of the way first. But overall, the process will largely continue like that for as long as it is needed. You get the idea - rather than wait say 3 months and put out a mega patch, we're keep prioritizing the issues and keep rolling these out to unblocking as many folks as possible as often as possible.



Similarly, we also have the process going for our first console patches (title updates). The nature of additional test cycles at Microsoft and Sony, and hardware limits on what and how many of these we can do however means we'll see these much less frequently than on PC. But given the nature of the consistent hardware per platform, and online delivery systems that have already gone through their teething pains, we shouldn't need as many either.



Separate from all of that, from the DLC team, today we announced our next DLC pack - Return to Ostagar. Watch for more info on that very soon.



That's about all the news that's fit to print for the moment. :)

#4
Fernando Melo

Fernando Melo
  • BioWare Employees
  • 639 messages
Thanks again for the great feedback folks.



Another week - and some pretty awesome progress on 1.02 patch. Some of the proposed issues were a bit touch and go this week, but the QA & tech team really pulled it off - we're getting very close now. We should have specific info to share with you next week, such as a proposed release date along with the actual list of issues being addressed.



Console update work has also started and yes, at the top of what we would like to address for the 360 is the achievement unlocks for those of you that did those early in our launch week.



Beyond that we're doing the finishing touches on Return to Ostagar and should also have more info to share with you on what you can expect in our upcoming DLC, plus talk about the release date.



Nice short update this week. Hope our US players had a great turkey day yesterday, and wish everyone a good darkspawn slaying weekend!

#5
Fernando Melo

Fernando Melo
  • BioWare Employees
  • 639 messages
Keep the great feedback coming folks - and if you do have any issues please make sure you check out the platform specific tech support forum, there are a lot of fixes or workarounds already available until we can get that into a formal patch.



Speaking of which I'm very happy to say that patch 1.02 is all but wrapped up! We hope to get that out to you early next week, possibly Monday, barring any last minute hick-ups.



As always, we'll make that available here on social (as well as other sites) and include the full fix notes. 1.02 addresses a lot of issues including Balancing and general Gameplay (such as the Daggers Dex issue), Plot related issues (such as folks losing their DLC items after Dwarf Noble), as well as bringing to PC some of the fixes found & made during console development after PC was locked.



As mentioned before, as part of the process there were also some issues that were pulled from 1.02 for a variety of reasons and moved to 1.03 so as not to hold up this patch. That work is now underway as well - please keep letting us know on the tech forum of anything else you are encountering.



For consoles, work is underway for the first title update (patch) which we hope amongst other things, to address the 360 achievements bug for those early DLC players on that platform.



Have a great weekend folks.

#6
Fernando Melo

Fernando Melo
  • BioWare Employees
  • 639 messages
Thanks again for the suggestions.



This week we saw the release of patch 1.02 for PC, and from speaking with customer support as well as monitoring our forums it has been a relatively smooth launch - there are some edge cases here and there which we're already looking at, but for the most part it was a very successful patch. As always, if you do have any issues please make sure to post in the tech support forums.



Continuing on from my previous posts, we are still working on 1.03 for PC, as well as the first console patch (console update #1). These will largely address very separate issues and are being worked on in parallel.



I can't confirm yet what will end up as final fixes in each yet, but I did mention that we hope to address amongst other things the missing achievements from 360 for those folks.



Why this? Well, it was one of the first issues found so work began on it fairly early on. We also discovered some unrelated items in investigating around that issue that made us want to dig in a bit more on the 360 - so it wasn't a simple cosmetic/player gamerscore win, if successful this fix will actually have a longer term benefit for the main game as well and that's why it's fairly high on the priority.



Also, that was just one of the issues I used as an example - of course we're looking at the other high priority items console folks are reporting, using the process I mentioned before - we have a lot of folks lurking through our tech support forums + talking to EA support, they enter bugs on what they find, we triage these regularly, these get allocated to folks here for fixing, based on timing it gets slotted into an update, or moved to next available update slot. Likewise, fixes may fall out or be moved on purpose from one update to a later one during the testing phases for a variety of reasons.



What else is coming, you ask? Well, we are *really* close to having some more official news on Return to Ostagar - I wouldn't be surprised if we see something on this before next week is out. This delay hasn't been intentional, but we're waiting to get solid dates for console release in order to be able to communicate that back to you correctly.



In other parts - expect to see some very cool new items on your 360 next week! (sorry, no, it's not the title update yet, nor Return to Ostagar) As well as a new announcement early in week that will make some BioWare fans happy.



Have a great weekend everyone.

#7
Fernando Melo

Fernando Melo
  • BioWare Employees
  • 639 messages
Hi everyone,



My apologies for missing the last updates. I've only now managed to sneak away for some time on the computer :) so we'll break usual friday update form this time.



Since the last update, if you are still wondering:

- The cool items I referred to were indeed the 360 Avatar items! The red blood dragon t-shirt is pretty sweet.

- The announce that some fans would like was the Mac version.



Since then, we've just put out an official update for Return to Ostagar DLC and released the video for it in the add-ons page. The release date for PC & 360 is set for Jan 5. PS3 will come out later. I did a update on the RtO thread if you are curious about more on this.



RtO for consoles will come with title update #1 (patch). More on what is included in the TU closer to the date.



We've also brought in Victor W to join our ranks - one of Victor's first tasks will be to help clean up for forum organization a little. So we will see an official BW posts area, where things such as this update will be located - and separate threads for the actual discussion points (good suggestions btw)!



Work on the PC patch continues - but with the holidays this has not really progressed very much since the last update - look for more news in the new year. We're targeting fixes for the DLC downloader as well as what memory leak/slow downs we have been able to reproduce so far, and continue working directly with some members of the community to get more details on others. We'll put out the official list of fixes closer to release.



Happy holidays! (must sneak back to family once again)


#8
Fernando Melo

Fernando Melo
  • BioWare Employees
  • 639 messages
Closing this... i'll start a new thread for each update so it is easier to follow.