Skirmish Update

Being sick and travel does a number on productivity for side projects. Alas this means that comparatively little has been completed in the last month.

The Caravan level's introduction, dialogue, and scripting are written and working (although it needs polish of course). The main content of the scenario is next, and that was a daunting prospect. This is the first scenario where we need a somewhat intelligent enemy. Intelligent in that it can use skills at the right times, find new paths when paths are blocked, and react to the player. The original concept was to build this within the scripting system for the scenario.

But we knew in the future that we were going to want to build procedural scenarios. Procedural scenarios can't have custom scripts to control enemies. Thus was born the need for a enemy control system; an AI one tier higher than our current AIs.

The current AIs in the system behave very primitively. It manages movement and pathing, deciding on an action to execute, handling getting to a relative range of their target, deciding on a target (through aggro/threat mechanics), etc. These operate at an individual level; each unit runs these control structures independently. One of those controls is the formation follower behavior, which keeps them trying to work together and in formation. There's plenty of complexity there that had to get handled, but it doesn't have any concept of future planning or any kind of analysis.

A while back (maybe 2 years ago) I worked on a system to control enemy formations, consisting of a set of directives which were selected through a cost function and periodically sampled. It really, really doesn't work. It kind of worked, but then it didn't. If you're wondering why the current scenarios don't have any enemy formations, this is why. Those of you who have played know there is one scenario with enemy commanders, and that they are barely barely barely functional.

So the need for an AI that runs on top of the behaviors is obvious. That's the current project. I'm building an AI Planner that looks at the current state, derives its options from available actions, derives its goals from the objectives (or script), and tries to come up with a sequence of actions to accomplish it. Then, searches over the space of sequences to find the best one.

The low level planner took a couple weeks to get working at a level I'm happy with. I'm now working on integrating the planner into the actual game. As of right now, the planner runs and the plan can be executed, but all it knows about is naive movement. It can move to its goal but basically it just knows how to dumbly right-click and hope for the best.

But it's a start. I built an AI obstacle course. When it can find its way through I'll make a vid.

.... AI is hard!

Earth and Air Elemental Rigs/Skins

Hey hey. I went back and built a new Air Elemental model, then realized I never posted the Earth ele!

Take a look :)

The last one's kinda... big :)

Keep in mind these are differently scaled versions of the elemental skin. Just like with the player's squad members, each unit has a variety of sizing attributes. The various types of elemental have the same parameterization. So yeah, that last one's kinda funny :)

Oh oh, and the Stone and Air bits animate; I didn't record a GIF for you though. Next time.

Making parameterized ClickOnce installers work

Alright. I spent entirely too long working on this one purpose yesterday. Judging by the state of internet questions on the topic, others have also spent too long working on this one purpose. Since I did end the night with a functioning system, I thought today I would write how I did it.

Two days ago I live streamed development for a short while, and wanted to let people on the stream test the game with me, and give feedback. I've had click-once based installers for years, but I found myself missing a certain feature. I wanted to be able to authenticate users, and be able to selectively enable clients. I wanted to be also able to give out custom URLs to people to download the game from, in a way that modified their installation so that I know which URL the game was installed from. Each installation URL should still self-update normally, however, and I want to be able to get rid of a given URL and authentication easily at a later date.

So this is a story about how to parameterize a click-once installation.

Attempt 1: Use query parameters on the installation URL to provide unique data to each installation URL.

My first idea was to write a script on the build server that generated a unique URL for the installer using a unique identifier placed in the query-string. So instead of installing from World/World.exe, the link would be World/World.exe?id=yourcustomidentifier.

This attempt ultimately failed, but in finding that out I solved other related problems, so here were the steps and their problems and solutions:

1. Building the unique links turns out to be easy: 
1. PowerShell scripts can generate Guids by invoking the .NET class libraries as in $authGuid = [guid]::NewGuid().ToString().
2. Likewise in MSBuild use the following in a PropertyGroup: $([System.Guid]::NewGuid())
2. Getting the query parameter to be used by ClickOnce is less easy:
1. Like many aspects of ClickOnce, it has become obvious that it was not meant to be used by build scripts. Passing the query parameters into the installer requires setting a flag in the application manifest; but there is no means to set this flag using the command-line Manifest Generator (mage). Since we don't want a person involved for every run, UI solutions will not work either. This leaves us with editing the manifest xml ourselves.
1. Specifically, we need to add trustURLParameters="true" to the  xml tag in the root of the manifest.
2. In PowerShell I did this hacky monstrosity:
1. (gc <product>.application) -replace '<deployment ','<deployment trustURLParameters="true"' | Out-File -Encoding "UTF8" <product>.application; exit;
2. Basically, a string replacement, but take care:
1. Problem 1: My build script was not powershell, but rather a batch file! Running powershell from the batch requires careful escaping, and looks like powershell -Command "{above command with escaped quotes}"
2. Problem 2: Why the 'exit'? Because powershell was stopping script execution and that seems to fix it.
3. Problem 3: Don't forget the -Encoding! My manifest got corrupted and unusable without it. I think it had to do with BOM (Byte order marks).
4. Problem 4: Be VERY careful to do this operation BEFORE you do any signing of the application file, obviously signing has to come last.
3. Really it's just amazingly painful using mage.exe.
2. And now the kicker; This doesn't work!. Why not? Because the links I provide are not to the application, but rather to a bootstrapper Setup.exe that ensure prerequisites are installed, such as the .NET Framework. The bootstrapper does not forward query parameters, which makes sense, it's an EXE, very separate from the URL itself.

Attempt 2: Find some other way to parameterize that is readable during the run of the installed application

1. So query parameters are out. What else can be use? The next approach I tried was to encode the identifier into the filename itself. So now the download link would look like World/World-identifierhere.exe.
1. This fails for similar reasons as above; This is the filename of the bootstrapper; not the application manifest. ClickOnce provides access to the launch URL in C#, but this does the application URL, not the bootstrapper EXE URL.
2. Next attempt: The bootstrapper contains in it the URL of the application; maybe we can add a query-string there and the application will receive it?
1. So now we have to make a custom Setup.exe for every user that embeds their identifier; this is surprisingly not that difficult! The Bootstrapper is built using MSBuild, and customizing the URL to include a new Guid is not hard
2. But a new problem arises: When I attach a query to the URL, the bootstrapper no longer works automatically: It forwards the user to a webpage to download the application manifest, rather than just installing it! I don't know why! Google didn't help! 
3. I thought here that I could just make a custom application manifest for every client, but then realized that to maintain automatic updates I would have to forever-after update EVERY single client's installation for every single patch. NO thank you.

Attempt 3 (Success!): Build the user's ID into the username of the URL of the application manifest, when constructing the Bootstrapper EXE

Every attempt at getting data into the application from the bootstrapper failed, EXCEPT adding a username to the URL, which was passed through and available in the C# ApplicationDeployment.CurrentDeployment.ApplicationURI. 

Caveat implementor: This URI is only available in the first run, from the bootstrapper! You need to save that identifier somewhere in your application or you'll lose it forever!

So to recap my hopefully final solution:

1. Build the application manifest and installation the same as usual.
2. Build a unique Bootstrapper EXE for every client authentication token you want to have.
1. The bootstrapper's ApplicationUrl needs to include the authentication token inside the username section of the URL.
2. Rename the bootstrapper's exe to include the authentication token, to provide a unique URL to give to the user to install.
3. When building the bootstrapper, register your authentication with whatever database you have, to be able to track it later.
3. In the client application, read from ApplicationDeployment.CurrentDeployment.ApplicationURI.UserInfo, and save the results somewhere permanent.
1. Use the authentication token saved to authenticate and identify.
4. Later, just delete the custom Setup.EXE when you want to cut off access, and remove the authentication token from your database.

It may not be the prettiest solution, but it works, for now.

Good luck.

Update + LiveStreaming

What's new?

• I'm testing out live streaming at www.livecoding.tv/ventare. It's strangely entertaining, though I admit it isn't terribly good for actually getting work done. On the plus side the few guys that chatted with me last night did help me solve a few bugs that were as of yet undiscovered. I don't know how often I'll do this, but I may find some time tonight if not afterwards.
• I've built a couple new visual effect types, based on the materials system and mesh duplication and offsets. Am using it for some slightly more subtle weapon glows.
• Caravan, the second tutorial scenario, is in progress; the story bit and tutorial bits are written (but need polish), and the actual scenario part has begun, but only just.
• I'm working on authenticated identification of installs, so existing installations will break at some point. I'll be able to supply custom links so that I can deprecate links and authentication codes later. 
• I built some meshes in Blender! Specifically I built a skin for the main human mesh, the elemental default mesh, and specific multimaterial meshes for flame and stone elementals. Eirikr's weapons are getting UV coordinates as well, when I find time.
• I did a pass over sound effects a few weeks ago; they are better but not thoroughly better.
• I implemented a means for the server to instruct the client to do a spline-based camera flyby, useful for tutorial and cinematic sections.

Just some pics

The last couple months have been full of important updates to the game with absolutely nothing visible to show for it. It's somewhat depressing :p

So here's a couple pics from something actually visible.

New Flame Elemental Skin, version 1
Angle 2
Angle 3

That's it for now. I'll do a thorough update when my head's less hurty.

Caravan Sculpting

Quick update here!

The tutorial revamp with new tech is complete. I did one sound pass (many more needed), added music queues, explosions from space, changed the ending section logic to work with the new terrain mana skills, and added some other fluff in other sections to make it a better experience.

Caravan work is beginning in earnest! I built some new editor tools to alleviate some of the pain; namely, a flood fill terrain painter, a growing sphere-based sculpting tool, and a visualizer for a layer column so we can see how deep the materials really go, and what is down there.

Here are some new Caravan pics :)

http://redtoast.net/Images/Blog/CaravanEditor1/StartView.png
http://redtoast.net/Images/Blog/CaravanEditor1/DownRiver.png

http://redtoast.net/Images/Blog/CaravanEditor1/RiverTunnel.png

http://redtoast.net/Images/Blog/CaravanEditor1/RiverFord.png
http://redtoast.net/Images/Blog/CaravanEditor1/RockCanyon.png
http://redtoast.net/Images/Blog/CaravanEditor1/MudGate.png

http://redtoast.net/Images/Blog/CaravanEditor1/Overhead.png

That's it for now :)

2 new tracks

Did some new piano recording with the new microphone. It's not the most inspired work but it's alright:

w_tut_first_main.mp3
w_com2.mp3

Let me know if you think the recording quality is any improved. I personally think it's a lacking a bit in the bass department, so I'll adjust the mic setup next time to try and capture more of the low ranges.

Both recordings have some note misses that are obvious to me and a bit painful; not the best session I've ever had, but people tell me not to worry about that sort of thing and to record anyway :) :p

When Animations Fail

Pew Pew Airplanes

On Programming Languages and Complexity

It's been awhile since I ranted a bit on programming methods. I've had some thoughts on languages recently that I want to write down to help solidify. Bear with me or skip this if languages aren't your thing.

In college I was taught in Java, with a bit of C, and even less of other languages like Python, ML, Prolog. I and Tristan learned C++ to write our first game. Later I took more software engineering courses and learned some new patterns for complicated projects. I wrote a pile of libraries in C++ and got a pretty good handle on it. At Bunkspeed I continued with C++ but we pretty quickly replaced that with C#, which I worked with from version 1.1 until I left. World has been almost entirely C# since its inception in 2007. Now for work I'm working with C++ again, now for approaching 2 years.

So I have a pretty good comparison now of C++ and C#, both used for larger codebases. Bunkspeed's main product was pushing 600k, my current work is around 1M, and the World project is around 150k. I admit I do have better experience in C#; I try to keep that in mind.

I firmly believe that what you can do in one you can do in the other. The limitations of both can be circumvented to achieve the same ends.

 C# has the reputation of being slow, and the problem of being single platform and having a harder time interacting with 3rd party libraries (which are usually native). The slowness problem I think is a red herring; optimizing is usually done at the algorithmic level, which is indifferent to language. The GC and memory management can be controlled if you desperately need to, but honestly you do not. .NET Native coming out soon should alleviate the rest of whatever performance issues exist.  The single-platform issue is improving with Mono and recent C# changes announced by Microsoft. The third party library issue is more complicated and I don't know of a good solution. P/Invoke is not a good solution, nor is C++/CLI.

C++ has the reputation of being fast and powerful, but complex, low-level and imprecise. The low-level issue can be resolved by writing code within C++ to add high-level features. C++ code MUST manage memory, and so large C++ codebases MUST write something to track pointers and allocations. In the end these systems approximate garbage collection, and they work. By 'imprecise', I mean that there are often unanswerable questions, the answers for which are not apparent immediately in the code. The size of pointers and integers, the nature of strings, how 'out' parameters are passed, naming conventions, includes ordering, namespace management, smart-pointer management, etc. All of these things have to be handled in some way by the programmer. And they can be with great grace and power, but they must be handled.

Herein lies my point. C#'s failings are fixed by external structure. C++'s failings are fixed by internal structure.

The upper limit of manageable complexity in C++ is lower, because some of that manageable complexity is ubiquitous and cannot be ignored. When writing in a large C++ codebase, you cannot forget about the internal structures that handle C++'s limitations. Every last line needs additional consideration as you ponder string encodings and which of 6 ways to declare your pointers (smart, shared, dumb, reference, constness, etc...).

The upper limit of manageable complexity is higher in C# than in C++, because when the structure becomes more complicated, the code does not. The complexity is managed in a parallel way, a way that does not impact the programmer when trying to read and understand a lone function.

Maybe with time I'll be able to background the necessary considerations in C++ and be able to acheive similar productivity. Or maybe I'm just spoiled by the cleanliness of C# lambdas, extensions, and continuations. The only thing to do is to try and see if I can get there. Tsuyoku Naritai indeed.

Mana Manipulation (and Videos!)

At the tail of the last post, I said that the incendence system and environmental magic were next. Guess what! It was next. It was complicated, but it is working and it is good :)

Let's break this down and make some new video clips.

So. Environmental Magic. This is a system designed to let the player change the terrain. We already had (somewhat stable) terrain changes, but the nature of those changes is quite a bit different from the new system. A few of the old-style changes included the bridge builder (featured at the end of the tutorial, and on Outpost), and a couple hacky commander skills that no one but I used. These will be gone soon. A second set of old-style changes are the elemental impacts system, which will be staying. It compliments the new system nicely, and will be used for non-mana manipulation skills. Any explosions or active flames and whatnot will still use the old impacts system, which works well.

The new system is built specifically to give the skills to the player. To that end it has to be limited and controlled enough to prevent the player from abusing the skills to absurd extent. Honestly letting the player arbitrarily change the terrain is a terrible idea if you want to control the gameplay. The fact we're trying this at all is a testament to pushing boundaries that oughtn't be attempted. The payoff of course, is that no one else has really succeeded in giving this style of control, so if we do succeed to any extent with it it's a mark in our favor.

So anyway, the limiters. The new system is pretty much entirely based on the concept of conservation of energy. The energy in this case is not potential, nor kinetic, nor chemical, it's just Mana. Every type of material has a mana density; how much mana they contain per unit volume. Mana, by the way, comes in 4 forms, as per classical elements. Mana is conserved. From this concept we defined 5 basic operations, which are pieced together to build the actual skills.

First is extract; this takes a chunk of material and converts it into a less mana-dense form, while preserving overall mana. The net effect is that new material volume is created, but it is less mana-dense.
Second is draw; this is very similar to extract, except instead of creating new volume, the excess mana is absorbed into the caster, stored and usable later. The net effect is that material is converted to weaker types in the local area.
Third is infuse; this is the opposite of draw; mana is pushed into materials, converting them to more mana-dense forms without change in volume.
Fourth is compress; which is the analogue of extract to draw for infuse. Compress reduces overall volume when converting to more mana-dense forms.
Last is shift; which just moves a chunk of mana from one location to another.

Each has some mana cost; small but important.

The nature of these is that mana is conserved or lost. This enforces that things do not get too crazy on the map; There's a limit to what can be generated; implied by the original amount of mana present. If we build a map to play on that consists entirely of sand (a very low-mana material), then that by itself completely removes any possibility of manipulation. There simply isn't enough mana to do anything with. The density also limits any volume changes. The total mana of a bridge's material has to be put into the bridge to construct it, and that mana (plus expenses) has to come from somewhere.

After that we added more limiters. Extractions and Infusions and Shifts have an overall power requirement, per material. Granite is hard to work with; extra skill is required to work with it. Likewise at the bottom of the earth-spectrum, sand is complicated. This limits the player in where he can manipulate the terrain, but in a graphically obvious way that can be played around.

Then after that we have to manage reach; how far the operations can work through the ground. I invited the concept of mana-pressure; how much pressure the manipulator can exert on the terrain. Pressure travels out through the materials using a conductivity calculation that depends on their mana density. Earth pressure reaches further through earth-aligned materials, for example. This conductivity is fairly unbalanced; so reach will fall along aligned conduits in the ground, potentially extending total range in the right circumstances. You could even generate explicit lines of high conductivity material to control how the mana flow is directed.

How about some visuals?

Here's the first version of the Mana Manipulation skill tree.

And here are short videos for the various new skills in action:

Mana Draw (Pulling mana from the grass and dirt, turning it into lesser rocks. The first outward pulse represents the mana pressure, that is, the effective range of the draw force)
Mana Infuse (Pushing mana into dry mud and mud, turning into mud and dirt, then running out of mana)
Mana Extract (Extracting Gravel from Shale, and Sand from Gravel, to produce a column of material at distance)
Mana Wall Construction (Drawing a line in the ground, then infuse/extract/shift to move it into a wall shape)
Mana Bridge Construction (1) (Extract/Shifting a bridge out of the ground)
Mana Bridge Construction (2) (Building a second bridge from less dense materials. You can see that sand is getting used at the end of the bridge, since the source area is running out of mana-dense materials. This bridge could not get much longer, and when physics is written, the sand would fall out of it)
Mana Shift-Smoothing (Using Shift to smooth out the surrounding area, making it traversible)

Hope you enjoyed!

Next time I'll talk about Incitement and Incendence. It is currently WIP, maybe halfway through implementation.

Over the holidays

Time was limited of course, with moving, buying a house, selling a house, and holidays, but nonetheless a small update is in order:

• Revamped (rewritten entirely) Chat system
• We once again have an OpenFire XMPP server running on the uni server. Accounts are created by the server to match universe accounts, and some chat channels are running.
• The chat UI was rebuilt and is substantially more functional! It runs in the bottom center of the game now, has autocomplete for commands, and supports whispers, say (in game speech), channels (global exists right now, later others), and custom commands (who, reloadui, etc). It's somewhat nicely styled to allow scrolling but not get too much in the way of the camera. As always polish will improve over time as annoyances show up. 
• Whispers in chat communicate via direct message in XMPP, without the game server being a part of it at all. So you can, in fact, use any other XMPP client to join the global channel or be online for whispers (and whisper back).
• Server stability
• Did a bit of work on the server to prevent some weird hangs, we'll see if it works, but so far so good.
• Dynamic reloading
• The client can now reload the ui, reload assets, etc.
• It can also be run in a mode where it monitors the client database file, and if it changes, reloads it dynamically and reloads the zone. It makes updating visual effects fairly seamless. I can keep the visualizer open while a game is playing, save from the visualizer, and the new effects appear in 4 or 5 seconds.
• Incremental database building
• The old build from google docs for the majority of the game data would destroy everything it built previously, and build it all back up every time. I moved to an incremental approach based on the last editted timestamp of the document, so it is much much faster.
• We also have a polling system for updates :)
• It also can update live while the zone runs, swapping in changes on the fly :)
• So basically you can run a local zone and a debug client set up to automatically pull changes from the database, compile them and establish them in game in 10 seconds. This should make the editting process way way way faster. The old reload/test cycle was 3 or 4 minutes at best, the new is 10 seconds. BEHOLD progress.

That's that. Next on the docket is the incendence system and the elemental skills update, for which the above were written. And then after that finally back to scenarios :)