Skocz do zawartości

Zdjęcie

Extracting face from MOR and importing into toolset morph editor


  • Zaloguj się, aby dodać odpowiedź
18 odpowiedzi w tym temacie

#1
DarthParametric

DarthParametric
  • Members
  • 1409 postów
The other day in reply to this post, I stated that it wasn't possible to extract the face data from a MOR and bring it back into the toolset morph editor. It's something I have tried in the past but with no success. However, after musing on it for a while, I decided to revisit the issue. I played around with it for a bit, until I hit on a method that, while extremely messy, appears to work. Here's an example of how to go about it.

We'll start with our test subject, a human male noble that is an outstanding example of manliness named, appropriately enough, Bruce:

DAO_MOR_Extract_01_TH.jpg

The first step is to extract the MOR from one of his save files. We'll use NewByPower's DA Face Replacer for this. I won't cover the specifics of how to use it, check the readme or various forum posts for that. The short version is you choose the save file as the source, a new face as the destination, then export. And voilà, one newly extracted MOR.

If you open the MOR in the GFF editor (either the toolset's or the standalone one), and look under the MORPH_NODES section, you'll find one labelled FaceM1 (node 2 in this example, but the actual number may vary for different sex/race combos). What we are looking for here is the MESH_STREAM_VERTEXDATA sub-node. As the name suggests, this is the vertex data for the face mesh, i.e. the vertex co-ordinates. The problem is, there is no apparent way in the GFF editor to easily extract this data, and copying and pasting it one value at a time would be ridiculously impractical (there are 9,128 individual values for a human male face mesh).

DAO_MOR_Extract_02_TH.jpg

So Bioware's GFF editor won't do the job. Next we turn to pyGFF. Open the MOR with editor.exe and you'll see much the same thing as before. Again, there's no apparent way to simply copy and paste the data we need. But there is a workaround. Here's where things start to get messy. First, find the MESH_STREAM_VERTEXDATA sub-node of FaceM1. Once you have located it, you can manually select every other node one at a time and hit delete. This is necessary to prevent pyGFF erroring out in the next step. You should be left with something like this:

DAO_MOR_Extract_03_TH.jpg

Now you can go to File->Save As and change the file type drop down to YAML (make sure to change the file extension in the filename box as well, as pyGFF doesn't seem to do it automatically). Once that is done, you can close pyGFF (without saving the MOR). Open the new YAML file with a text editor. As ever, I recommend Notepad++ for its superior functionality (and wide array of additional plugins). At the top of the file you'll see a bunch of information about the various nodes. All we want however is the vertex data. Select everything else and delete it. Make sure to scroll down to the bottom and get the superfluous lines there as well. If you have a human male head, you should have 9,128 lines, each with a single numerical value. Other race/sex combos will have slightly different numbers. You'll see that each number is preceded by some additional garbage that we don't want; two spaces, a dash, then another space. You can simply do a find and replace (Search->Replace in N++) and choose to replace that string with nothing to get rid of it. Save the resultant cleaned up data as a plain TXT file.

So, we've overcome the first major hurdle. We have the raw morph vertex data extracted from our MOR. The next step is to reconstitute it into something we can work with. We need to start by converting this data into the game model format, a MSH/MMH/PHY set. In order to do that, we'll take advantage of a method I described previously in the taller elves thread. Rather that repeat myself, I'll just direct you to the first post on page 2 where I describe how to extract a base head model and generate XML files for it. Choose the head of the appropriate sex/race for your morph, and then follow the steps detailed there until you reach the part where you need to paste in new vertex data.

That process was fairly straightforward. However, there's a problem in this instance with our vertex data. In our human male noble example, our data is 9,128 lines, one value per line. As some of you may have noticed, when looking at the MOR in the GFF editor there was a node named MESH_STREAM_VERTEXCOUNT with a value of 2,282, which matches the number in the XML file. That is the actual number of vertices in our face mesh. The amount of data we have is exactly 4 times that number, which is no coincidence. Vertices (and other related values, like normals, etc.) are specified as a float4, which basically means a single vertex is defined by an matrix of 4 values. The first three values are positional data in 3d space, while the 4th value is always 1.0 (for vertices at least. Anything that is a vector needs a 0 value. It's all about matrix transformations, light years beyond the scope of this post). So, the problem is that we have a dataset that is in a single column but needs to be transposed into 4 columns, with each row derived from 4 consecutive values. If this was a handful of data, we could just do it manually, but as with extracting the data in the first place, with over 9,000 lines of data, doing it manually is completely impractical.

There are a few different ways to approach the problem. The best way would be for someone to write a custom program to do it (well, perform the whole process really, and if anyone wants to volunteer, feel free). Trying to do it in Excel would be an obvious route, although I couldn't get it to work with vanilla Excel (there are some addons/plugins however that appear to do it, generally commercial though). An IT friend helped me out by suggesting Powershell, a commandline set of tools bundled with Windows (as of 7), and providing a code snippet to do the job. Go to the Start menu->Run, type in powershell and hit enter. A console window will pop up. Now paste in the following (change the values to the path and filenames you are using):
 
$x = Import-Csv "E:\\Temp\\Vertex_Data_RAW.txt" -Header Data -delimiter ' '
for($i=0; $i -lt $x.Count; $i++){$out += $x[$i].Data; if(($i + 1) % 4 -eq 0){$out += "`n"}else{$out += ' ' } }
$out | Out-File "E:\\Temp\\Vertex_Data_OUT.txt"
And hit enter. There will be a slight pause at the second line while it processes all the data, after which the last last line will pop up. Hit enter once again to finish and get to a new prompt. You can now close the Powershell window. If you look in the output directory, you should see the output file you specified. As an example, if we feed in a test file with just the first 16 lines of data, like this:
 
3.71073838323e-10
0.0610870011151
1.52558004856
1.0
9.1121182777e-06
0.153705343604
1.70198023319
1.0
-1.92087463802e-06
0.137376964092
1.66151988506
1.0
-7.21627293387e-06
0.133858963847
1.64632999897
1.0
The resultant output is:
 
3.71073838323e-10 0.0610870011151 1.52558004856 1.0
9.1121182777e-06 0.153705343604 1.70198023319 1.0
-1.92087463802e-06 0.137376964092 1.66151988506 1.0
-7.21627293387e-06 0.133858963847 1.64632999897 1.0
Rearranged as 4 rows of 4 columns, exactly the format we need. So, now we have the vertex data properly formatted, we can copy and paste it into the mesh XML file, as described in the taller elves thread linked to earlier. Once that is done, resume with the procedure described in that thread, processing the XML files and importing them into Max/GMax. So, now we have our extracted morph as a mesh in Max/GMax, like so:

DAO_MOR_Extract_04_TH.jpg

Note that I have hidden the rig and collision objects in that pic, for the sake of clarity.

The final stage is to get the face mesh back into the toolset's morph editor. To facilitate that, we need to export our face mesh from Max/GMax with the same filename as the base head mesh of the appropriate sex/race (i.e. the one you used to generate the donor XML above). In this example I'm using a human male head, so the appropriate filename hm_uhm_basa_0 for the MMH/PHY and hm_uhm_bas_0 for the MSH. Eshme's exporter gives the option to have a different name for the MSH, so just tick that box in the export window. Export the models from Max/GMax and put them in My Documents\BioWare\Dragon Age\packages\core\override. You can create a new sub-directory for them to keep things tidy if you want.

Now it's time to open the toolset. Open the morph editor (File->New->Morph) and choose the appropriate sex/race. Fingers crossed, instead of the original base head you should see your morph, like so (you may wish to employ this for dramatic effect):

DAO_MOR_Extract_05_TH.gif

Success! You can now tweak the parameters of your face. However, before getting too excited, be aware that you will likely have a much more limited range of customisation than you would with a typical morph. A lot of sliders will be greyed out and unusable. I'm guessing this is because the new base head vertices don't match their expected positional values. I'm not sure, but it's better than the likely alternative (i.e. a toolset crash). Anyway, play around as you see fit. Here's what I did to my head:

DAO_MOR_Extract_06_TH.jpg

Save an MRH and Post to Local as you normally would in order to create a new MOR. Now you can use DA Face Replacer once again to transplant the new morph back into your save file. Make sure once you have finished that you remove the temporary base head models from the Override folder. Here's my modified morph in-game:

DAO_MOR_Extract_07_TH.jpg

I've uploaded the source files from this example here for anyone that is interested. Additionally, here are the base head models for every player sex/race.
  • DarthGizka lubi to

#2
Berserk1479

Berserk1479
  • Members
  • 13 postów

Thank you once again for directing me to this very useful thread DarthParametric! I was looking for a way to make msh/mmh files starting from a mor file and this procedure worked flawlessly for me. :)

 

As I mentioned in the other thread, I'm wondering if the process can also be implemented with the DA2 mor files. I assume that extracting the vertex data with pyGFF should not be a problem: the sub-nodes in the DA2 mor files don't seem to have "proper" names, but I can tell the one corresponding to FaceM1 from the others. What I'm not sure about is whether (once I have the vertex data in a useful format) I can use the "taller elves" procedure to make the msh/mmh files.

 

In particular, I have not found a tool to convert the base DA2 face files (e.g. mt_bas_hm.msh/mmh) to the correponding msh.xml and mmh.xml files: Eshme's import/export script will not work with DA2 files and tapzn's command line tools for DA2 apparently do not convert the msh/mmh files to xml files that I can use. Am I missing anything here or is there any other tool/approach I should use to create the xml files?

 

Assuming the hurdle of getting the xml files can be overcome, then I'm under the impression that the DAO toolset GraphicProcessors should be able to convert the xml files to a set of msh/mmh/phy files. That would be enough for me, since I do not necessarily ned the resulting files to work in DA2 and it would be enough if they can be imported with e.g. Eshme's import/export script or NewByPower's blender tools.



#3
DarthParametric

DarthParametric
  • Members
  • 1409 postów
As I said in the other thread, you need to explain what it is you are trying to do in order for me to offer the best advice. If you are dealing with Origins, there generally should be little need to convert morphs to meshes when you can just deal directly with the MORs themselves. If you are trying to get DA2 assets into Origins, then that makes more sense. In that instance then yes, you can follow the same basic procedure outlined above to extract DA2 morph data and convert it into a mesh for use in Origins. Here's a brief demonstration with Varric.

To start with we'll get the basic assets. I'm not going to cover the extraction process. I will assume you know how to use pyGFF and the hash list to extract DA2 assets. Varric's morph is dm_varric.mor (found in \packages\core\data\face.erf). If you open the MOR in pyGFF, you'll see under the MORPH_PARTS section the base head model (mt_bas_dm) and the hair model (dm_har_varric). Other morphs will have additional models like beards. The base head model (mt_bas_dm.mmh/msh/phy) and the hair model (dm_har_varric.mmh/msh/phy) are found in \packages\core\data\artfp.erf and once extracted in art\characters\playercharacter\dwarfmale\.

With the MOR open in pyGFF, as per the original post you can extract the vertex data. The only difference is this time you'll also want to extract the vertex data for the eyes and eyelashes at least, if not the scalp as well. You can extract them all at once, just make sure to delete any excess nodes the same as in the original post before exporting as a YAML. If you do export them in a single file, I'd advise breaking them up afterwards into a file per mesh before running through Powershell. It may be easier to export them individually from pyGFF in the first place to prevent errors. Whichever way you go, once you have your vertex data files ready, process them with Powershell as per the original post.

Now to get our donor XML files. To do this we need to get the base head model (mt_bas_dm in this case) into Max. There is a MaxScript DA2 importer floating around, but it doesn't seem to import meshes correctly. It adds additional vertices, at least in multi-mesh models. I'm not sure what the underlying cause is, but it makes it useless for this purpose. There's also a script for Blender, but I never could get that to work properly. However, there is an alternative. As was the case for Origins, tazpn created one of the first mesh tools for DA2, available on the legacy site. This tool converts DA2 meshes to FBX, which can then be imported into Max. The conversion is imperfect, at least as far as the rig and skin goes, but is sufficient for our purposes. Two things to note though. The first is that it flips UVs, so you'll need to address that during the import or manually flip them afterwards. The second is that the meshes are rotated 180°.

Once you have the mesh imported, unhide the rig and delete it, leaving just the meshes. Create a GOB from the DA section of the Helpers tab added by Eshme's tools. Link the meshes to the GOB, then add a Mesh parameter to each using the ModelManager. If you haven't already, add a Unwrap UVW modifer to each mesh. Create some new DA materials and apply them to the meshes. You can simply create a single empty DA material and apply it to all the meshes, as it is just to prevent the exporter from complaining, however adding the proper textures is a good way to confirm that the UVs are aligned properly. When you are done, you should end up with something like this:

DA2_Morph_2_Mesh_01_TH.jpg

Now you can export this model to generate the XML files. Then follow the same procedure in the first post, copy and paste the vertex data, although this time for each mesh. Compile your XMLs and then import back into Max via Eshme's tool. If everything went according to plan, you should now have something like this:

DA2_Morph_2_Mesh_02_TH.jpg

And here he is with his hair mesh:

DA2_Morph_2_Mesh_03_TH.jpg

Now it is ready to be skinned to the Origins rig and re-exported for use in the game. Because of the way Origins works, normally you would want to export eyes, hair and beards as separate models, although this will require the extension/addition of GDAs for them to show up in the toolset's creature editor if you are making playable races.
  • Berserk1479, DahliaLynn, luna1124 i 1 inna osoba lubią to

#4
Berserk1479

Berserk1479
  • Members
  • 13 postów

As I said in the other thread, you need to explain what it is you are trying to do in order for me to offer the best advice.

 

Your posts are always providing me with the information I've been looking for and I keep forgetting to explain what I'm trying to do here! ;) 

 

Actually, my objective is two-fold:

Objective #1 is to use the vertex data in the Awakening MOR files to generate msh/mmh files for the face, eyes and lashes of a number of characters that I can work with in blender (I'm using NewByPower's dragonblender tools to import them). Thanks to your first post in this thread, I've already managed to achieve this objective!

 

Objective #2 is similar to the above, except that I want to use the vertex data in the DA2 MOR files to generate msh/mmh files for those characters for which there is no msh/mmh files already in the DA2 files (e.g. Gamlen, Elthina, etc...). The plan is to use the DA2 importer for blender and/or NewByPower's tools to import in blender. I think the additional info in your post above should allow me to achieve also this objective...

 

Now to get our donor XML files. To do this we need to get the base head model (mt_bas_dm in this case) into Max. There is a MaxScript DA2 importer floating around, but it doesn't seem to import meshes correctly. It adds additional vertices, at least in multi-mesh models. I'm not sure what the underlying cause is, but it makes it useless for this purpose. There's also a script for Blender, but I never could get that to work properly. However, there is an alternative. As was the case for Origins, tazpn created one of the first mesh tools for DA2, available on the legacy site. This tool converts DA2 meshes to FBX, which can then be imported into Max. The conversion is imperfect, at least as far as the rig and skin goes, but is sufficient for our purposes. Two things to note though. The first is that it flips UVs, so you'll need to address that during the import or manually flip them afterwards. The second is that the meshes are rotated 180°.

 

Using pyGFF to get the necessary files is not a problem, but I might have a little problem getting the donor xml files: I'm running Eshme's tools in GMax (no Max available here...) and apparently GMax cannot import FBX files. I seem to remember that tazpn's tools can also convert to the 3DS format which can be imported in GMax: I'll give this 3DS format approach a try to see whether it works and look whether there's a FBX import script available for GMax.

 

Thank you once again for all your help!



#5
DarthParametric

DarthParametric
  • Members
  • 1409 postów

Actually, my objective is two-fold

 

I'm still unsure on what your actual end-goal is. Are you trying to export stuff for either game, or just playing with meshes in Blender?

 

apparently GMax cannot import FBX files. I seem to remember that tazpn's tools can also convert to the 3DS format which can be imported in GMax


Ah. Yes, tazpn's tools can export 3DS, OBJ, and DAE in addition to FBX. I'd recommend OBJ over 3DS. There are a few GMax OBJ import scripts around. The one I had the most success with was Renx OBJ Tools. The readme for tazpn's tools should explain how to export in different formats.

 

If you are proficient with Blender, you may want to try out this DA2 import script. I never could get it to work myself, but perhaps you will have more luck. You'd still have to get the meshes into GMax in order to generate the XMLs with Eshme's script though, so I'm not sure how much that gains you. It might be a better option in some instances though, as tazpn's tool is a little flaky.



#6
Berserk1479

Berserk1479
  • Members
  • 13 postów

Apologies for the unclear explanation about my end-goal, but your second guess is correct: I want to import the meshes (with rigging) in Blender for playing around with them.

 

As I suspected, my "weak point" here is getting the XMLs for the base DA2 files: unfortunately, my first attempts were not met with success...

 

For the record, it appears that conversion of the DA2 meshes to 3DS with tazpn's tools doesn't work properly: I used the "mt_bas_hm" mesh as my test subject, but the 3DS I obtained with tazpn's tools was a real mess. On the other hand, conversion to OBJ with tazpn's tools was successful.

 

Since I still had not read your reply about importing OBJs in GMax at the time, I followed a different route to get the 3DS file, i.e. I imported the "mt_bas_hm" mesh in Blender with the very same DA2 import script you have mentioned above and then exported to 3DS. This time the mesh was preserved in the 3DS file and I imported it into GMax.

 

Now, this is were I get into trouble, since (much to my embarrassment) I don't see how to create a GOB and follow with the rest of the process. The tools' manual wasn't helpful, unless I'm missing something... My guess is that I overestimated my knowledge of Gmax and/or practice with Eshme's tools (so far I merely used both to import/export DAO meshes to generate the XMLs). I don't think that importing an OBJ rather than a 3DS in Gmax is going to be any different, but I'll try that too.

 

Is there anything further you can suggest to help me? If not, them it sounds like I really have to understand how to add a GOB with the tools under GMax or figure out whether there may be a problem when running the tools under Gmax that prevents me from doing what I need.

 

I'm starting to wonder whether there's any volunteer who is willing to convert the DA2 base head files of the various sex/races to XMLs and put them somewhere... Of course, unless that's a taboo... :rolleyes:



#7
DarthParametric

DarthParametric
  • Members
  • 1409 postów
Creating a GOB is pretty easy. I'm sure it is detailed in the manual somewhere, but here's a pictorial guide:
 
GMax_Create_GOB_TH.jpg
 
Alternatively, you could always just import an Origins model, or hit the Create Preset button, and delete everything but the GOB (probably useful for checking scale and alignment anyway).
 
As far as the XMLs go, try this - http://dpmods.wheb.o...2_Base_Heads.7z

#8
Berserk1479

Berserk1479
  • Members
  • 13 postów

Thank you for the input on adding GOBs and for the XMLs! You're a real lifesaver! :)

 

And I confirm I was successfull with the process! See below a quick render of Athenril:

 

1qj3gz.jpg


  • HBC Dresden, theskymoves, DarthGizka i 1 inna osoba lubią to

#9
xanuen

xanuen
  • Members
  • 71 postów

better could be to if someone create a program which from mor is export to dragon age toolset as file mrh



#10
Berserk1479

Berserk1479
  • Members
  • 13 postów

Hi DarthParametric!

 

Hope you're still keeping an eye on this thread and that you will not mind helping me out some more with the step of getting the donor XML files for the DA2 base head models.

 

What I'd like to do is getting XML files for the DA2 base heads where the "standard" DA2 bones are kept through the process, such that when I load e.g. in blender the MMH/MSH files generated from these XML files I'll have (working) DA2 bones in the model.

 

The XML files you have graciously provided do not "include" the DA2 bones, since according to the process you've explained in the earlier post you've deleted the rig coming with the imported FBX of the base head models. Since I don't have 3DS Max, I asked a friend to generate the XML files from the FBX for me on the basis of the instructions in your post, but skipping the step of deleting the rig. However, the resulting XML files provide MMH/MSH files (see pic below) with no vertex groups for the mesh parts and the armature is... well, it's not where it's supposed to be (guess that the GOB simply should not be where it is...).

 

I am unsure what steps and/or settings were overlooked before doing the export with Eshme's exporter. Assuming what I want to do is something that can be achieved, can you please provide me with some directions I can pass to my friend to solve the issue(s)?

 

Thanks in advance! :)

 

2in1gn.png



#11
DarthParametric

DarthParametric
  • Members
  • 1409 postów

The problem is tazpn's tool doesn't generate any vertex weight data. Neither does the Maxscript importer from Xentax. The only option is to manually rig the meshes yourself, as I mentioned in an earlier post. Unless you can get the Blender import script to work or know someone with Lightwave that can use tmp's plugin.



#12
Berserk1479

Berserk1479
  • Members
  • 13 postów

^_^ Oh, I thought that converting to FBX (rather than 3DS or OBJ) with tazpn's tool would carry over the vertex weight data, but now I understand where the problem lies!

 

So far, the only alternative I've figured out is using the bone weight copy script in Blender in order to copy the bone weight data from one of the DA2 head models (imported using the very same Blender import script from Xentax that you have mentioned) to the head models generated through the XML files you have provided earlier. Unfortunately, the copy is not perfect and some bones (mostly the jaw bone) cause unsightly mesh deformations when rotated. I'm still trying to figure out how to fix this (my Blender skills are a bit limited), but this is somewhat OT.

 

Do you believe that the Blender import script from Xentax could somehow help me in another way? Please let me know if you have some other suggestion I could implement using the Blender import script! :)

 



#13
DarthParametric

DarthParametric
  • Members
  • 1409 postów

Well assuming that you can use the Blender script to successfully import the base head model correctly, one option could be to then export that in a format that can be imported by Max/GMax, then use Eshme's script to generate a set of XMLs and swap out the vert data from the morph as per usual. What formats can Blender export in that maintain skin weights? DAE perhaps?



#14
Berserk1479

Berserk1479
  • Members
  • 13 postów

Thanks for all your input, you're always so helpful!

Using a different formast is a great idea... why didn't I think about it?!? :) Uhmmm, with GMax I can import only 3DS and OBJ, but these formats are no good here. I'll give DAE a try, I see there are two different DAE exporters in Blender "Collada 1.3.1" and "Collada 1.4". Maybe one of these will do!

Assuming that I can find a format which maintains the bone weight data and that can be imported in Max and seeing that you are the most experienced person on this matter, would you be available to make the XML files for me? If so, please let me know and I'll contact you with a PM to provide a link to the files once I convert them.



#15
DarthParametric

DarthParametric
  • Members
  • 1409 postów

I thought there was a DAE plugin for GMax, but it seems not. There are some MD3 import/export plugins floating around from the Quake days, although a lot of the links are dead. You could try this, which pretty much has all the GMax plugins there are to be had it seems - http://neverwinterva...ript-collection  The ones of relevance would be MD3, SMD, MS3D, some or all of which Blender should support natively, or have plugins for.

 

Otherwise, yeah just PM me a link with the files and I'll take a look when I get the chance.



#16
Berserk1479

Berserk1479
  • Members
  • 13 postów

Thank you DarthParametric for all the assistance!

 

For anyone interested: XML files of the DA2 base head models with intact vertex weights and bones may be generated with Lightwave using tmp7704's script (http://social.biowar...m/project/2246/).



#17
Faervel

Faervel
  • Members
  • 4 postów

Hi!  Thanks for this thread, but I'm having a few problems that I'm hoping you'll be able to help with.

 

I want to edit one of my Wardens's faces in the toolset, and I've already extracted the mor. file.  I followed your instructions in the pyGFF editor, but when I tried to open the YAML file from there in Notepad++, it ended up looking like this (sorry for the small-ish image):

 

tumblr_nolfloDWpv1s3e936o1_540.png

 

In addition to the funky text, there's only 430 lines.  Any idea what could be going on there?  I'm completely new at this kind of thing.



#18
DarthParametric

DarthParametric
  • Members
  • 1409 postów

You presumably didn't change the file type drop-down to YAML. Looks like it is still a binary GFF.



#19
Faervel

Faervel
  • Members
  • 4 postów

Ahh, that was it!  I just added ".yaml" to the ending before.  Thanks!