The Witcher 3 Tech Analysis

Posted on by
1

witcher3_2015_05_19_19_52_05_657I wrote an article about The Witcher 3 for PC Gamer. I focused primarily on some tips for achieving the smoothest possible gameplay, but what I’m really surprised by is the CPU results I obtained.

I might have chosen a bad location for that benchmark, and perhaps I’ll repeat it somewhere else, but I really expected an open world DX11 RPG to be more CPU-heavy. Certainly not playable with two 2.2 GHz cores at 30 FPS and 4 at 60 FPS. Very well done, and another indicator to me that seems to hint towards the gains with future low-level graphics APIs being less pronounced than some expect, outside of very specific use cases.

Of course, that level of control will at the very least be a huge boon to VR rendering, which inherently requires a high framerate, low-latency and extremely consistent performance.

GeDoSaTo FPS capping, modding controversies

Posted on by
3

FPS Capping

I recently started implementing FPS capping in GeDoSaTo. For those not familiar with the term, basically you want to achieve a consistent framerate in a given game, and in order to do that you cap the framerate at a given maximum.

In practice, this is usually done by simply inserting waiting periods after you are done with a given frame. E.g. if you want to cap to 30 FPS, you would wait until you reach a frametime of 33.3 ms.

There are multiple external tools which already do this (e.g. NVidia drivers have an option, RTSS can do it, some games have a built-in cap, …) so up to now I wasn’t interested in doing the same in GeDoSaTo — though I was sometimes frustrated by the lack of fine-grained control in external tools. However, I recently had an idea on how to improve upon the commonly employed method for doing this, in order to perhaps slightly improve input lag in a capped scenario.

fps_cappingThe image above shows 3 use cases: uncapped framerate, a traditional 30 FPS limit implementation (“Capped”) and my new method (“Predictive”). As you can see, the traditional method simply inserts a waiting period after each frame. However, this means that you are potentially losing a few milliseconds of input latency for no good reason.

Instead, with predictive FPS capping, GeDoSaTo keeps track of the frame times for previous frames, and takes a set fraction (configurable) of that as a predictive waiting period before each frame. The result (for games which do synchronous input sampling on the rendering thread or at least on another thread synchronized with it) is that the input after waiting is used for the new frame, reducing input lag by some fraction of the frame time.

The only dangerous aspect of this method is if there is a sudden spike in frame rendering times. For example, in F2 in the picture you can see that the frame almost doesn’t get done in time. If it were to cross the threshold, predictive FPS capping would result in a framedrop which would not happen in a traditional FPS limiting scheme. For this reason, the ratio of how much waiting time should be moved to the start of the frame is configurable — you can use a low value like 0.25 for a game with very erratic frametimes and something like 0.9 for one with extremely consistent performance.

Here are the new configuration options (which can be configured as per-game user profiles, like always)

The busy waiting or sleeping option is there for completeness, but outside of running on a laptop battery I don’t really see why you would not use busy waiting. In my tests it’s a lot more exact.

Note that you can use floating point numbers as the frame limit, I find it often useful to go with e.g. 30.5 FPS if I want a solid 30 in order to overcome unmeasured overheads.

Modding Controversies

Since my last blog post there have been two pretty large controversies about modding. The first was regarding paid mods for Skyrim. I was asked to provide a comment for PCGamesN, which was used in this article. My full comment, which is provided on the second page of the article, sums up my feelings on the issue. I have to admit that I was most disappointed by the level of vitriol some (purported) members of the community stooped to.
The other issue was once again the old friction between modding and multiplayer. Both in DS2 and GTAV, people who claimed to be using only mods which do not affect gameplay in an unfair way were relegated to the cheater pool in online multiplayer. This is a very delicate issue. It is extremely hard or even impossible for a game to assess whether a in-memory modification is benevolent or not, so I can certainly understand “no online modding” policies in principle. in the end, that is just another instance which shows why developer-supported modding is the way to go: in such a setup, it’s easy to control which modding functionality is available in what game mode.

GeDoSaTo minor update, Pillars of Eternity, DX11

Posted on by
4

I just pushed a minor update to GeDoSaTo (which you can download here, as always), mostly to integrate the pull requests and new profiles people have contributed over the past 3 months or so.

With the release of Scholar of the First Sin, I also get a lot of renewed interest and questions about DX11 support in GeDoSaTo. The answer to that is still that I’d love to do it (and I also already laid some of the groundwork for it last year), but I simply don’t have the time right now between some increased load at my “real job” and still trying to get some gaming in. I do of course welcome any contributions, as always.

Speaking of gaming, Pillars of Eternity was released recently, and it’s really really good. You should play it.

 

Graveyard of Forgotten Projects (Part 2)

Posted on by
Reply

This is a direct continuation of the first part. Let this partial quote from there serve as an introduction:

This series (hopefully) of blog posts will gather and shortly describe some previously unreleased and/or forgotten projects I’ve worked on over the past decade and a half, starting from 2000. This is by no means a complete listing, I’m still skipping over the vast majority of unfinished ideas and half-baked projects, but I’m trying to include everything which worked to an extent or has some interesting features.

The primary purpose of this is to serve as an archive for myself, because, as you will notice when I go through the list, I’ve already lost a lot of somewhat interesting stuff and would rather not have that happen again. Perhaps one thing or another might also be useful to someone else, but I obviously won’t troubleshoot code I wrote more than a decade ago and haven’t touched since then! And, just to make that clear, it’s also obviously not indicative of my current skills.

I stopped the previous post around 2003-2004, a timeframe of abortive attempts at creating a 3D multiplayer physics-driven space ball game.

Rudloid

I wasn’t sure where to slot this in, as I started this project in ~2002, but did a lot of work on it in 2005. I chose to go with something closer to the latter than the former. This is an Arkanoid clone written in Ruby. It used the RUDL SDL wrapper (another amazing example of my naming sense really), and initially did software rendering. Later on I switched to using OpenGL for the rendering for performance reasons, which is why the final 2005 iteration was called RudloidGL.

What is interesting about this project is that it’s actually written rather well, in some ways at least. For example, I implemented features like collision detection, sound effects and sprite animation as Ruby mixins, and they are re-used in every object where it makes sense. So the balls, paddle, blocks etc. use the same basic collision code with distinct response events, and animated blocks, paddles and balls use the exact same sprite animation code just with slightly different parameters.

Of course, there are also some things were I was maybe going a bit too far with trying to be smart. For example, I used the “evil” library (yes, that’s actually its name) to implement different ball/paddle/etc. types and changing between them (when picking up a powerup pill) by actually changing the class type. It works, and in a way it’s elegant, but it’s also a bit wild.

Talking about powerups, that’s actually another thing where the code for this game is a lot better already than most of what I discussed in the first post in this series. For example, I seem to have had attained a decent understanding of the “once and only once” principle at this point, and the value of setting up tools in the code base so that it can be effectively used. For example, this is the actual, entire definition code for all 12 powerup pills in the game:

While eval’ing some random strings would make my hair stand on end now, at least the heart for this code is in the right place, and I still appreciate how concise it is.

The game also features a level editor, which nicely reuses most of the actual game rendering and related code. It also allows for undo/redo and copy/paste operations.

On the whole, this isn’t too bad really. What was bad, no, terrible are the sound effects. I never spent much time on those and it really, really shows. Check it out for yourself in this Shadowplay video I took of the game (after spending literally 2 hours finding all the dependencies in compatible versions and getting them to work together).

I will not upload the .zip for this, since I don’t think I am actually allowed to distribute some of the music and sound effects I used. If anyone cares a lot, post a comment and I’ll try to create a working package without those.

Wasser

The imaginatively-named “Wasser” (German for Water) is a very small program I hacked up in early 2005 in order to generate daily data fitting a decent-looking curve from monthly data about the electricity production from hydroelectric power plants. Until I made this, that was apparently a process carried out manually in Excel (I shit you not).

wasser2There are two interesting things about this one:

  • It uses both Tk for the UI window and SDL (via RUDL) for the visualization window. That’s insane, but I guess I just used what I knew, and it works.
  • I came up with the method of how to distribute data across the days of each month on my own, without really knowing how one would actually solve such a thing correctly. Nonetheless, what I do in the end isn’t too different from what an explicit iterative solver would look like.

You can get it here, though I have no idea why you’d want to.

Ragex

Released (actually released!) Christmas 2005, Ragex is a tiny generator for XHTML. I don’t need to write much about it here, because the web page is still online, I’m just including it for completeness’ sake. The best part about this one was when I got an email from someone actually using it, in 2008 or so, when I had almost forgotten creating it.

Crystalise

After my 3D experiments in 2004, and other game projects that never quite reached completion, I really wanted to actually finish a game. That’s why I came up with a very simple action/puzzle principle, and Crystalise was born (also check out the help file for a visual explanation of its gameplay).

Here’s a bullet point summary:

  • It’s made for PSP, the hottest gaming device of 2005 ;)
  • The vast majority of the game is written in Lua, with some of the heavy lifting (rendering, collision detection) in C. Pretty modern in that regard!
  • As far as I can tell, I completely lost the entire C source code for this project, as well as all the original (non-distribution) assets. A valuable lesson about backups, and even more so the value of releasing the source. The lua code at least is included in the distribution.
  • It really is a complete game, including stuff like a main menu, difficulty levels and high score tracking.
  • The levels (including boss and special levels) are procedurally generated, probably one reason why I actually managed to finish this game.
  • If you hit a certain level (which probably no one except me ever did) the game increases the PSP clock speed to 300 MHz in order to keep up solid 60 FPS:
  • I believe I invented the now commonly used paradigm for writing text with a analog gamepad in this game. That is, selecting a direction with the analog stick and then selecting a character from that direction using a button:

This is one of those projects where I’m really sad I lost (part of) the source. At least in this case most of the Lua is still available. And some of it is pretty neat, check e.g. circle.lua which (despite the name) keeps track of and handles updates to the current state of the level.

LuXr

This is a C#-based program launching (and other stuff) tool that I’m still quite proud of. I made and released it in early 2006, and the web page is still online. I just downloaded and ran the 54 kB package, and it still works well on my current Windows 7 installation.The major reason I created this was because I was fed up with the Windows (pre-Vista) way of launching programs. Of course, Microsoft seemingly agreed, and the keyboard / string search driven way of using the start menu in Vista onwards pretty much made LuXr obsolete.

Still, there are a few things to like about this one:

  • The aesthetics for the startup effect were influenced by the OS in Serial Experiments Lain. Which was much more awesome of course.
  • It features a plugin system which dynamically loads plugins written in C#, which can have their own config dialogue pages (they are adressed by adding a prefix to what you type, e.g. “g something” to google “something” with the google plugin, or “e” to use the eval plugin:
  • I still prefer how this resolves substring matches comapred to the Windows Start menu search. For example, if I write “vis 13″ in the start menu on my current PC, it finds no matches. LuXr finds “Visual Studio 2013″.

What’s not to like is that I believe I lost the source for this one as well (except for the plugins, which are part of the distribution package). It seems that with the years progressing I got better about actually releasing stuff from time to time, but worse about keeping backups.

 

That’s it for today, next time around we’ll get to actual 3D graphics programming. WHOOOHOO!

Hyperresolution Neptunia 0.2 alpha release

Posted on by
14

I did some small code cleanups, and more crucially, compiled both .dlls in release mode instead of debug mode. I forgot about that for the initial release, so it had a higher CPU performance impact then expected. Do note that there will still be some CPU overhead from the interception compared to running the vanilla game — this is unavoidable.

Here’s the download, enjoy! And here’s the entirely optional donation link.

Is anyone interested in the source for this one? I don’t mind releasing it, but I made it by hacking an existing glintercept plugin so there’s tons of superfluous stuff in there.

 

Hyperresolution Neptunia 0.1 alpha release

Posted on by
6

Today I looked into the rendering resolution issue for Hyperdimension Neptunia Re;Birth 1, which was recently released on Steam.

What’s interesting about this release is that it uses OpenGL, which is really rare in Windows games these days. It’s the first Japanese port I’ve seen which uses it. Therefore I was interested in having a look, as I’d never done any interception work on an OpenGL game before.

The good news is that the free software situation is much better for OpenGL than it is for DirectX (and particularly, than it was for DIrectX before I started DSFix and later GeDoSaTo :P), with multiple options available for handling the mechanics of interception/logging etc. After some googling I chose glintercept to base this work on, and I haven’t regretted this choice at all. The codebase seems a bit heavy at a first glance, but you only really need to deal with the plugin interface, which seems solid from what I’ve tried.

As for the game itself, after acclimating myself to OpenGL it was basically the same process as always: find buffer creation calls, intercept, try, find stuff that breaks, and iterate until done. In this particular case HUD and minmap rendering were the easiest to fix of any game I’ve worked on so far, and it took less than an hour for those after I got basic rendering at higher resolutions working.

Anyway, here is the download.

And if you want to contribute to my continuing struggle for more pixels, you can do so here (this link includes a reference that lets me know you are donating because of Neptunia in particular).

 

 

DSFix 2.4

Posted on by
26

This might in fact be the final version of DSfix. Why? Because Nwks (the original author of the 60 FPS unlocker!) came back and did something which I had considered ideal for a long time now: make it scan for the addresses it needs to patch instead of hard-coding them.

What does this mean for you as a DSfix user? Hopefully, that when/if the game is updated in the future, DSfix, including the FPS unlocking, will still simply work. The only reason this would fail is if the binary code right around the places which need to be patched were to change, which seems unlikely for the smaller patches which can be expected at this point.

Also, since it seems like some hosts are still taking down DSfix for no discernible reason, here are 3 separate links to it:

Have fun!

Edit:
Just to reiterate: this version will not work with the GFWL version of the game, only with the beta Steamworks version (which will replace it at some point).

Graveyard of Forgotten Projects (Part 1)

Posted on by
3

I’ve wanted to do this for a very long time, at least since early 2014, and one of my resolutions for 2015 was to finally get started. This series (hopefully) of blog posts will gather and shortly describe some previously unreleased and forgotten projects I’ve worked on over the past decade and a half, starting from 2000. These projects will mostly be games (or game ideas, rather) but also include a few applications. This is by no means a complete listing, I’m still skipping over the vast majority of unfinished ideas and half-baked projects, but I’m trying to include everything which worked to an extent or has some interesting features.

The primary purpose of this is to serve as an archive for myself, because, as you will notice when I go through the list, I’ve already lost a lot of somewhat interesting stuff and would rather not have that happen again.

Perhaps one thing or another might also be useful to someone else, but I obviously won’t troubleshoot code I wrote more than a decade ago and haven’t touched since then! And, just to make that clear, it’s also obviously not indicative of my current skills. That said, let’s start! I’ll try to do this chronologically.

Fastimg

fastimg_mainThis early 2000 (with some 2001 updates) – so, when I was in the middle of high school – project was a Delphi-based GUI tool for quickly generating pure HTML image galleries, automatically resizing images and generating thumbnails. Shout outs to Delphi for being by far the most convenient tool for generating GUI applications back in those days!

Thanks to the wonders of Windows binary compatibility and static linking, the executable I found which was compiled on 2001-11-05 simply launches and works on my current Windows 7 system. Seriously, if nothing else Microsoft deserves major kudos for making that possible, it must be absolutely maddening to work on.

fastimg_prefsYou can see the preferences dialog above. Anyway, the tool has obviously been made completely obsolete by technical progress, but it’s fun to see that it still works. Here’s a zip archive containing the sources (and holy crap, are they ever terrifying!) and binary. I don’t recommend looking at it ;)

Another Delphi application I remember writing around that time which seems to be completely lost now was an advanced Windows Registry search/replace/editing tool (which was also much faster than regedit). A bit like Advanced Regedit, but not as fully featured.

Tetris Attack Puzzle Mode & Hitori Solvers

Around 2001 or 2002 or so I wrote a few Ruby scripts to solve puzzles. The first one solves Tetris Attack puzzle mode levels, outputting the optimal turn sequence. It’s command line only, so no screenshots to show off here.

hitoriThe other one helps solving Hitori puzzles, and even visualizes them, as you can see above. Here’s an archive with both scripts, though note that they require a very outdated Ruby version to get working – and are terrible in terms of code quality of course.

Urlaubtris

I wrote this Tetris clone around the end of August 2002 while it was raining for a few days when I was on vacation with my family in Italy (“Urlaub” is German for “vacation”, as you can see with e.g. GeDoSaTo my naming skills are still just as amazing as they were back then).

It uses Ruby 1.8 and RUDL (a Ruby SDL layer), and as such is almost impossible to get running on modern systems. I managed to create a packaged binary which should include the right versions of the interpreter and all the libraries, but I’m very unsure that it will actually work anywhere.

There are a few neat things about this one, considering I spent all of 2 or 3 days over 12 years ago writing it:

  • It supports an arbitrary number of cooperative players — or, well, as many as you can manage to find keyboard keys for :P – and arbitrarily sized levels.
  • It saves/loads settings as YAML files. Considering YAML history, that was some pretty damn early adoption of a technology which is still around and which I still like.
  • The code isn’t completely terrible even if I look at it now (which makes sense since it’s very small and written in a very short amount of time, makes it hard to accumulate cruft).
  • It features the Gameboy Tetris theme performed a capella by my sister as background music ;)

Here’s the archive in case you are interested. As I said above, you probably won’t be able to get it to work, and I’m obviously not going to support it.

Eball3D

This was my first 3D game project which got further than just displaying a model on screen. It was also my first larger-scale C++ (as opposed to C) project, and it’s as terrible code-wise as you might expect from that description. I started on it in 2002 and worked on it for a few weeks, and also came back to it for a few days in 2003 — at least that’s what it looks like from the file timestamps.

The game uses the Ogre3D engine, ODE for physics and OPCODE for collision detection. All in some very specific and utterly outdated version, and in a Visual Studio .NET (that is, 7) project and as such, I didn’t even try to compile it. I also made some “art” for it, using Wings3D (still a very neat, free polygon/subdivision modelling tool IMHO) and Rhinoceros, which was available at the university I had just started at.

The idea was to create a game which plays a bit like Uniball, but completely in 3D. That is, there are two teams of players, a ball, and two goals, and they can shoot the ball either to pass them between each other or at the goal, while being affected by recoil and slowdown while carrying it. I was going to go for an “energy” ball in this version, as it would have made a bit more sense in 3D when quickly “picking it up” or taking it off another player.

As I said earlier, it’s nearly impossible to get to compile (never mind running!) on modern systems, and I don’t have any screenshots anymore, so sadly some model shots must suffice:

To the left you see a ship: the semi-round thing in the front carried the energy ball if it had one, and I was going to change the blue colored stripes on the back depending on team assignments – of course I now realize that this wouldn’t have been nearly visible enough. And that the model is utter crap ;). To the right is a level. The game would take place inside the ellipsoid, with the ball bouncing off both its outer walls and the obstacles within, allowing for some skillful ricochet play. Note that a ship was small enough for two of them to easily fit through each of the holes in the central obstacle.

I got as far as loading everything, getting physics/collision for the level, ball and ships working, as well as making a single craft playable and capable of picking up and shooting the ball, before losing interest. I remember that when I wanted to get back into the project later on (in early 2003 I think) I tried upgrading it to newer versions of all the libraries used, hit some roadblocks and dropped it again.

UBC

I picked up the Eball3D idea once more in late 2003 and 2004, but realized that full 3D gameplay would simply be too complex to be fun (or even playable really). Especially since much of the enjoyment in Uniball came from pulling off precise team moves and passes.

As such, the new plan was to go with an essentially 2D level structure, but with some elevation, jumps and perhaps multiple stacked planes. However, I got distracted with developing more and more intricate revisions of the level editor with C# and never even got to a state where you could do anything in-game (unlike Eball3D).

On the left you can see a screenshot of the second unfinished revision of the UBC level editor, and on the right the initial model for the player crafts. The energy ball would have floated in the central space if carried — as you can see, the shape is much simpler (and probably more elegant) than in Eball3D, built for “2.5D” gameplay,  and would also have allowed for much faster and more exact collision detection. So I guess I learned something at least.

-

That’s it for today, as I already spent around 5 hours crawling through backups and backups of backups as well as trying to get completely outdated software working to an extent that will allow me to at least take screenshots. Next time I’ll have some more modern projects to show, which might be interesting from more than just a historical perspective.

GeDoSaTo 0.16 Christmas Release

Posted on by
6

As a holiday special so to speak, today I’m releasing GeDoSaTo 0.16.

The TL;DR version of the updates is this:

  • Fixed a number of memory leaks which caused instability in pretty much all games. In some (most?) it was unnoticeable, but in Final Fantasy 13 and 13-2 crashes could occur as early as after 15 minutes of playtime.
    All of that should be fixed now.
  • As a result of that fix, if you are looking to find a PSHash or VSHash for a game (and only then!) you should enable the new trackShaders option.
  • I’ve also improved keybinding modifier support and borderless fullscreen support.

You can get the latest version with the installer provided here. And, as always, you can donate to support GeDoSaTo development here.

Happy holidays everyone!

 

The long version for those who are interested:

I’ve been chasing the sporadic crashing issues on and off since the release of FF13. For the longest while, I assumed that they had to be related to screenshot taking, as that was when they first and most often occured. I tried many things related to that (pre-allocating buffers, using the library in a way that causes no additional memory allocations, …) but it turns out that this was, more or less, useless, as I was looking in the entirely wrong place.

When I finally realized that, despite the issue only appearing in the FFs, it had to be a general memory leak in all of GeDoSaTo, I manually checked every single instance of new in the code for the corresponding delete . This was less work than it sounds since I try to write modern C++ where you very rarely call those directly. Over the course of this, I found a few issues, specifically  operator delete  being called instead of operator delete [] . However, since those were arrays of POD types, while bad practice and undefined behavior it wasn’t the issue I was looking for either.

Afterwards, I made a few abortive attempts to use existing leak checking tools, but like many tools designed to make programmer’s lives easier they are really hard or impossible to apply in the — very unusual, admittedly — interception use case of GeDoSaTo. It would certainly be a good idea to implement a stand-alone executable testbench at some point, but these things, they take time.

Yesterday, I finally made the breakthrough. My new approach was to fill the memory almost completely at start-up so as to more quickly reproduce the issue. Then, I got a repeated crash during loading in FF13-2. This reminded me of earlier studies of FF log files, and that it loads far more separate shaders than any other engine I’ve looked at, and some of them 3 times each. And it continues to do so heavily throughout the game, which is somewhat unusual. This lead me to looking at the shader management code, and there I finally found the culprit.

And, as is almost always the case, looking back after finding the bug it seems painfully obvious: to enable things like PSHash HuD rendering identification or injection of postprocessing or AO in the case of specific plugins, I keep a map of all shader pointers and their names (which are derived from their code so as to be reproducible across runs). However, I never remove entries from that map. As it is impossible to know when an entry is save to remove without fully instrumenting shaders (which seems like overkill for this purpose) I’ve solved the issue by adding the new  trackShaders  option. If it is not set (the default), shaders will still be identified if they were set as PSHashes or VSHashes, but only those will be tracked. The only reason to enable the flag is to look for a new hash in a new game.