Replay parsing lib [S2R2]

[theli]

S2R2 is a Python module to parse replay files
It is NOT complete, NOT accurate, may do something incorrect, may be based on incorrect assumptions, etc

What this library does is just parsing entity snapshots ... and thats it
Some data like chats is just skipped for now since main concern was to parse entities
parsing speed is pretty slow ...
Interface is undocumented :eek: and pretty lowlevel ... you just have NextFrame() method to advance to next server frame and you have EntityPool property which is the Pool of all world entites and their states for current frame :-/

i doubt someone will find this usefull but i had great time reverse engineering replay format :p
and you already can make some stupid things like this heat map of TREES deaths (matchid: 6314909):
http://img215.imageshack.us/img215/9825/treesc.png

or this player positions/movement map for madman (played by YoDa, matchid: 7066386)
http://a.imageshack.us/img29/6922/madmanout2.png

(i may have messed something with those maps though)
this library (and maybe some example usage) will be/is worked on untill i completely lost interest in this


Requirements:
Python 2.6
bitarray python module http://pypi.python.org/pypi/bitarray/
example heatmap scripts require python module heatmap http://www.jjguy.com/heatmap/
disclaimer:

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.

oh,and about legality of this: S2Ikkyo said one is 'free to figure on your own' about replaydata format ( http://forums.heroesofnewerth.com/showthread.php?p=1140614 )

download : http://dl.dropbox.com/u/4443078/HoN/s2r2/s2r2.7z

this is abandoned, left for research purposes only. you better learn how to get plain text game logs as that should contain most of information you want

[Quzzap]

I must say; awesome job, this can become a very good analyzing tool, in whatever way :)

[Silv3ri]

Well done, that's impressive! The tree script took 15 minutes to complete (on a 11mb file with 51m40s of game play) but it seems to work nicely:
http://dl.dropbox.com/u/7150186/Images/S2R2-heatmap-M4957526.png

Could you also share the map background? Couldn't find it in any of your files.

[theli]

Well done, that's impressive! The tree script took 15 minutes to complete (on a 11mb file with 51m40s of game play) but it seems to work nicely:
http://dl.dropbox.com/u/7150186/Images/S2R2-heatmap-M4957526.png

Could you also share the map background? Couldn't find it in any of your files.

yup, as i said its pretty slow ... it needs to process more than 62000 frames for your replay :(


minimap:
textures.s2z\00000000\maps\caldavar\minimap.dds

[HueHang]

awesome :o
thanks for your great work :)

[Sauron`]

Nice work man!

[BASH]

Great research, looking forward to more!

[powettino]

I would have a tecnical question if someone can answer to me..

Inside the struct used for the library, in the "Hero" struct, there are a lot of variable that define the entity hero. One in particolary is in my interest, 'm_yStatus' using to define the status of the hero... what's the difference from the value 0,1,2,3 of this field ?
In your example is used this control:

if x.EntityPool[id]['m_yStatus'] != 0

why?

tnx :)

[oxman]

Your work is awesome a documentation about the replaydata structure will be very nice :)

[Phe`V]

Please
write detailed instructions

[theli]

Your work is awesome a documentation about the replaydata structure will be very nice :)


Please
write detailed instructions

sorry, but i don't have time, desire or motivation to do any more work on that

but, just out of curiosity - why do you need that anyway? most crucial data can be parsed out just from game logs only

[devINVISIBLE]

unpack_enabled is some pretty gnarly rerversing - nice! Are you still alive and kicking theli?

[theli]

nat that alive and active as i used to be, also, this particular one project is kinda abandoned since i can't think of a really good use case for this which is worth the efforts

[devINVISIBLE]

I tried sending you a private message, but was wondering if you'd care to chat over some other messaging service. I'm in the process of porting (and extending) your library to C# and had some questions of some of your reversing. As an avid (albeit newbie) reverse engineer, I've been enjoying comparing your work and to the disassembly and debugger. Let me know if you have any time or interest in sharing some of your experience. :]

[Namreeeb]

I do this in C#, but I do it as the game is running rather than parsing the replays by hooking the snapshot parsing functions. Is that ultimately what you're trying to do, devINVISIBLE?

[Mazungu]

sorry, but i don't have time, desire or motivation to do any more work on that

but, just out of curiosity - why do you need that anyway? most crucial data can be parsed out just from game logs only


Id like to mod this code a bit just to parse out the chat logs of the replay files.

[theli]

chat logs are more accessable from game logs then from replays

[Mazungu]

chat logs are more accessable from game logs then from replays

But since the S2 XML log parser is busted indefinitely our only choice is to parse them out of the replays them self right? I don't know of another way to access the game logs.

[Rejanu]

I have something I wrote for parsing replay files. What exactly do you want to get out of the replay?

[theli]

But since the S2 XML log parser is busted indefinitely our only choice is to parse them out of the replays them self right? I don't know of another way to access the game logs.

you can use http://hondiff.appspot.com/gamelog/matchid to easily download zipped text game logs

[Mazungu]

you can use http://hondiff.appspot.com/gamelog/matchid to easily download zipped text game logs

Wow thats a lot easer then the hex searching i was doing. Thanks Theli! Is this an official site or something you rigged up?

[theli]

Wow thats a lot easer then the hex searching i was doing. Thanks Theli! Is this an official site or something you rigged up?

hondiff is my site but all what it does is just redirects you to correct location on S2 replay servers.