It's gone

First useful MERPG game engine release

TL;DR

Memapper/merpg-engine (I really should rethink branding of this thing) version 0.1.3, the first one without glaringly obvious bugs AND in which you can make games you can deploy to users, has been released. Pick it (and the accompanying emacs mode if you wish to script your games) up from the github.

A long long time ago

Back in the dark age of 2010 I thought "Hmm, Pokemon Mystery Dungeon is an awesome game. I want to make a clone of its mechanics". I began hacking the engine with C++/SDL, because big boys had told me that you're not anyone unless you're hacking games in C++. Well, they forgot to mention that hacking C++ in Windows (which was my sole OS at the time) is a pain in the arse compared to Linux ("whaddaya mean you want the *.lib files (or whatever Windows called object files) available EVERY time you compile?" - Visual C++ 2010). Alongside I began to write the map editor using Best Software Engineering Practices™ in Java, because at the time I had my first course in Java starting in a month.

A shorter while ago

In about half a year I grew tired of VC++ 2010 ("what the fuck is a multithreaded apartment and why does my code break with 'MERPG STOPPED WORKING - LOOKING FOR SOLUTIONS' if I don't want to live in one?" - Me, 2010) and instead of doing the right thing and migrating to a sane build environment (Emacs, gcc, Linux with modern repositories (ie. not CentOS nor debian), scons or make) I decided to scrap that codebase and either write the whole game in Java or not write it at all. That was a passable plan for a while, but then I got employed and saw what a mess software written in languages without metaprogramming, introspection and macros will become. Always.

I learned to Clojure and rebooted the project. I did that for half a year and found out painfully that using a not useless language is not magical fairy dust you can sprinkle over a project and save it from the certain doom

After half a year of meditating and doing everything but this project, I had an idea: game in which I've cloned the mechanics of another game ISN'T A VIABLE DESIGN DOCUMENT. After that realization, I scrapped the first clojure codebase that had gone bonkers and rebooted the project, with the aim of creating an editor in which you can edit maps and put sprites in a GUI and script the live instance like you'd script Emacs. I like to think that this idea began already a year before with this blogpost, but that post's ideas are mostly either evolved or still unimplemented (because that'd make this engine too specialized lazy developer without enough time).

Present

That particular reboot has finally brought some results. I published memapper 0.1.2 in github a few hours ago, and an hour later I did a bugfix which made the mouse api usable. You can import scripts, sprites (both animated and static) and everything gets saved in the project file. There's an embedded nrepl-server which extends the protocol by adding a few nrepl ops that cider-merpg (the emacs mode that knows how to find and save script files inside the live editor) requires. Thus if you need to run the editor from the source, use 'lein run' and connect manually with M-x cider-connect (or C-c M-c). When you need to run the final game, mark a map as initial map, save the image, copy the editor's jar next to the image and run 'java -jar ./editor.jar --image ./project.memap' in the shell.

I realize that this way of deployment leaves Windows users in a difficult position, with cmd.exe being a piece of shit and Oracle's jre being too dumb and stupid to actually install the relevant binaries in tha %PATH%. I'm sorry about that, and I'm searching for a better way to deploy, but the desktop deployment is the JVM's achilles heel mostly because of Windows. If anyone knows how to generate a jar-archive that runs on double click on explorer.exe AND an installer that for sure installs the jre-dependency in an unbroken state, let me know!

Download MEMAPPER-0.1.3 from github

Thoughts on scripting a game engine

Scripting is hard

Letting users script your system is a hard problem. How do you do it. Do you just expose eval of your favourite scripting environment with the selected apis to your system's machinery? Do you hack up an eval with an all-access pass to your machinery?

I'm using the terms I think I learned from Steve Yegge. The machinery, the engine refer both to the immutable parts of the software system set in stone by the original compiler. Think of the parts in Emacs or a web browser written in C. Scripting layer refers to the parts written in, using those examples, Lisp or Javascript.

I haven't researched Emacs' codebase at all, so I'm basing my understanding on reading blogs and inferring on its behaviour. AFAIK Emacs is implemented with the absolute minimal base system written in C, and as much as possible scripted on top of that in Lisp. Thus almost every parameter is trivially open to mutation by the user. There seems to be a way to extend Emacs with shared objects, but you don't want to do that unless you have to, because unless doing native, system-dependent things Lisp and C are equivalent in power and the tooling is a lot better in the lisp side. In fact, with Lisp you can mold Emacs to a email app or browser or anything which doesn't resemble the original text editor at all. I know Amazon used to have an email app written inside Emacs. I have also seen a few POS-systems running inside unix shells which might've been more useable to both the users and the developers if done inside Emacs.

But, but, but. Emacs has complete access to the shell and the filesystem. That's... that's... a security hole isn't it?

Might be. It doesn't really matter though, unless you're using Emacs as a http server and routing input from the sockets to eval, which might be a bit stupid. If you're paranoid of the third party Emacs extensions you're installing, it might be useful to grep for '(shell-command' and other IPC functions in the extension's sources. But that's why we have MELPA these days. Scripts user runs inside Emacs are assumed to be trusted.

Let's have a look on the browsers. There's a shitload of C, C++ and other compiled things in their machinery. There are the websites browser is a runtime for. A website can be a static, declarative document which makes interpreting it easy(ish). A website can also include procedural scripts. Those scripts, written in JS, are by definition not trusted. There's a sandbox in the browser the scripts are run in so that a website can't just ‘rm -rf --no-preserve-root /‘. Scripts are not allowed (or supposed? I'm not sure if the apis exist to do this) to mess with the browser's window chrome, only with the viewport browser assigns them.

The difference is the level of trust. Emacs scripts are assumed to be run by a user who knows what they're doing, and thus if a third-party script succesfully ‘rm -rf‘s everything, it's user's fault for running Emacs as root. Javascript is assumed to be run either by a babbling bumbling band of baboons who haven't dedicated their lives on computers or silently, completely unbeknownst to the user. However both leave the eval function open. Thus, if you have a way to ask textual input from the user, implementing a runtime REPL is easy.

  while(true) {
    alert(eval(prompt("Input: ", "")));
  }

;; don't do infinite loops in single-threaded, completely synchronous Emacs
(while true
  (message (prin1-to-string (eval (read)))))

Maybe it isn't that hard after all

How do Emacs and browsers map to the ways-to-script I started this text with? Browsers follow the first principle. Everything is forbidden until someone with enough authority (be it the user or someone in the w3c) authorizes it. Emacs does the second. It wishes to be able to script everything runtime, but because Unix won and our machines aren't lisp all the way down, that's not plausible.

Secret in these two environments is that they are easy and fun to script. I think Microsoft's tools like Office and Visual Studio are also runtime scriptable with a Visual Basic variant, but opening their scripting environment has never caused me the feeling to poke the system with a stick and see how it behaves. Last I checked (which I admit might have been in the XP era, me being somewhere around 10-16 years old) the VBA environment was a lousily documented, heavy gui app, with APIs that make, IDK, Java's ZIP-file API beautiful.

What are browsers then? Well, when I got into the web programming, the introspection in browsers was lousy and the DOM API has never won any beauty contests either. However, I thin these days they have introspection (in a form of the console and the dom viewer) and discoverability/self documentation (in a form of console auto-complete at least), they have always (in the context of one who learned to spell 'internet' well past the first browser wars) been freely available, and the feedback loop is much faster than with the compiled languages. The first argument was mostly a filler to achieve the magical count of arguments, but the latter two are important. They invite poking, and even though the user didn't fully grok how DOM works, they can hack functional stuff up fast and iteratively make it better afterwards.

Even though DOM isn't beautiful, it's necessary to have. In the browsers you don't have access to the machinery without it. If the scripting layer implemented only the base language, stdlib (dates etc.) and a runtime eval, you'd have a glorified calculator. You need a way to read stuff from the user and to print to them. If scripting had completely access to the environment's windowing system, that'd be cool, but then they'd be able to shell to rm too. Unnecessarily called rm isn't cool. Instead the browser decides to expose the selected parts of the windowing toolkit inside its own window to the scripts. The environment's windowing machinery is hidden under a common, system independent abstraction known as the dom.

I don't know much about how Emacs fills the needs like the one filled by the DOM in browsers. I haven't had the need to do much more with Emacs Lisp than re-assigning a few keystrokes, setting up modes and generating repetitive Java- and C#-shit. Yet. What I know is that Emacs does both the introspection and self-documentation a lot better than the browsers. In every mode you can write a small lisp snippet and press Ctrl+x Ctrl+e (or wherever you've bound eval-last-sexp). This evals the snippet in the context of the current buffer (similarly to how browser console evals javascript in the context of the currently open website). The lisp symbols carry their own documentation with them. Unlike in Java where documentation has to be separatedly compiled from the metadata in the source, you can ask an Emacs Lisp function for its documentation.

Let's assume you're writing an Emacs extension where you need to run a query-replace in the current buffer. As user you'd call it by pressing Alt+Shift+5 (M-%), but how do you do it as a programmer? The help system is under Ctrl+h. You need to know the group in which you are trying to get help. Now you need to know what happens when pressing Alt+Shift+5 (M-%). Let's ask help on keybindings: Ctrl+h k (for keybindings) Alt+Shift+5 (or C-h k M-%). This tells us all we need, API looks like this: (query-replace FROM-STRING TO-STRING &optional DELIMITED START END BACKWARD). The info-page has a lot more interesting stuff too, M-x count-words tells me there's about 328 words documented.

The example is a bit foolish, but that's because I've not done anything deep enough on Emacs to have any real-world examples. It still demoes the help system that's always near your fingertips, and unlike Visual Studio's unIntellisense, stays hidden when necessary and doesn't disappear halfway through the reading. To get more information on the help categories press C-h ?

(Somewhat stupid or ironic that just after I tell how the help system doesn't hide without user's consent, I find the one cursor in which I can put neither mark nor point in)

Scripting MERPG

So, how can these observations be used when designing my engine's scripting system? You might've not heard (ha :D) but the engine is as deeply written in Lisp as possible. This provides an excellent base to build a scripting api on top of. To be clear, this particular Lisp is Clojure and the machinery is written in Java. The only part I've had to do in Java that's not in either JVM's base class library or Clojure's runtime is the map renderer. There are a few parts in the render process written in Clojure which are the low hanging fruits if I ever have to optimize things. So, aside from the finer rendering details, everything else is done in the Clojure layer. The important stuff is even documented, so the users can connect to the engine's nrepl-server (nrepl-server basically exposes in-process eval to sockets) and either run (doc 'merpg.important.symbol) in the repl or use Cider's C-c C-d C-d - keybinding that opens the dedicated *help* buffer.

But exposing eval and rudimentary introspection tools provided by nrepl to the end-user isn't enough. Well, if your app's importance is similar to browsers', then you might get away with it, but those having to work with your half-assed system will curse you with their dying breaths. It's better to write a lot of documentation, and provide a lot of hooks for events you expect user's to have a need to script on (like, for example, DOM's onfoobar - event api). We also need a few really well thought out abstractions. Not like DOM's, which has useful abstractions but a horrible API, but like in Emacs, which's buffer abstraction is beautiful and the language permits making beautiful API's around it.

I like to think highly on my choices on how the game assets and -state is stored in The Registry, how registry is optimized on insertion, deletion and easy lookups when you know what you're looking for, and how it's transformed in the background to structures more optimized for rendering and other stuff. Time will tell though whether this design works or if it leads to worse performance than Minecraft's. Similarly, I like to think that the registry is an easy abstraction for the end-user to comprehend (I mean, what could be simpler than key-val - tables?), but only time will tell. When this editor has a "Build Executable" - button, I plan on implementing a couple of simpler games on the engine and improving the it based on those experiences.

Anyway, the relevant abstractions are the registry and every object type you see on the editor's domtree. Maps, tilesets, layers, animated and static sprites, tiles (which aren't visible on the domtree), and after I've designed this through, scripts.

The simplest way to add reactions to events would be to add watches on registry based either on the concrete ID (think of the following scenario

;; we're inside some other event
(let [sprite (animated-sprite! (re/peek-registry :selected-map) "./my beautiful spritesheet.png" 10)]
  (re/add-watch-on-key sprite
                       ;; This is called before committing the new-sprite-obj to the registry so you can check what's changed.
                       ;; new-sprite is an atom so that events can manipulate the object before it ends up in the registry without
                       ;; causing endless loops
                       (fn [new-sprite]
                         (let [{:keys [x y] :as new-sprite-obj} @new-sprite
                               {old-x :x
                                old-y :y} (re/peek-registry sprite)]
                           (if (or (not= x old-x)
                                   (not= y old-y))
                             (println "The sprite has moved!"))))))

this reacts to the movement of the dynamically loaded animation. animated-sprite! (like all the others that load assets from disk) puts the loaded asset automagically to the registry and returns the key with which you could fetch it from the registry. With re/add-watch-on-key you add the event that's called every time someone does a re/update-registry with the key you're interested in. The event gets called with the value that's en route to the registry. Surprisingly the value is an atom though. This way events can modify it without firing any events.

The user should also be able to set watches on the :types of objects. Call an event every time a sprite has moved.

There should be a simple way to poll the status of both the keyboard and the mouse. Every new environment I've moved to after I stopped doing coolbasic has disapointed me with the complexity of reading those. Events are cool and follow the best practices (or whatever is the buzzword for following whatever they were teaching as the gospel decade and a half ago in the java certification universities) for reading the IO state when you're doing a CRUD app, but with polling you can do a lot more complex actions with really simple code.

I'll provide filterable and map-able reagi streams for keydown, keyup, mousedown, mouseup and mouse coordinates. I'm not 100% certain these are as simple as I think. In case they prove to be too complex in the demo phase, I'll fart up a Windows VM and check if my recollection of the Coolbasic's IO API is just pure nostalgy or if it really was simpler.

Of course I have to expose :onload and :onclose - events, which are run on the startup and closing of the final game.

Script - assets and the editor

So, how would an user actually write and eval these scripts? The obvious way would be to let them write the scripts in their favourite text editor, keep the files with the .memap - project file and somehow import those files into the editor. I think last I checked Unity did something like this, but Unity's project directories are a nightmare. Let's not do that. Users having to carry multiple files in a project directory provides a lot more possibilities for the system to break than having a single file that contains the whole project image.

Thus I have to embed the scripts as assets inside the image. Which means having users using their favourite text editor becomes a bit more difficult. In theory you can open files-in-a-zip with Emacs' dired, but that's not exactly simple, Emacs is hardly everyone's favourite text editor (for reasons I've never understood :P) and the more complex editors are complex enough to miss the extremely simple use case of being able to save to a file handle pointing to inside a zip file.

I could implement an Emacs-like editor inside the map editor. But I sure as hell am not going to do that. I'd have the lisp machine model ready, but to make it useful even to those fond of Emacs I'd have the fart up an elisp->clojure - compiler (not impossible), rewrite Emacs' abstractions and do a fuckload of testing to find the corner cases. I'm not ready to clone Emacs, better people have tried and not-yet-succeeded.

The third option: make a server. If we already have an nrepl-server running for poking the live instance with a stick, it's no problem to make another server you could query the text-file assets from. I'm not sure if I could just extend nrepl-server to do this or if I have to invent a completely new protocol and dedicate another socket for this. If we assume we can trust everyone connecting to this file-server-socket, making it support find-file and save-buffer isn't hard. If the user has been able to connect with cider (or any other nrepl client) before reading the script asset from the server, we get the eval- and introspection capabilities for free.

Making a dedicated server or extending nrepl means I have to hold my nose and dive into the wonderful world of elisp. I plan on making a minor-mode which overloads find-file and save-buffer to work with urls pointing to the running server. I have almost no clue on how this would work technically, I just know that Emacs does async socket IPC and I have the Emacs' online help and the whole internet near my fingertips. The UX would be such that the user first connects to the running nrepl. Then in a buffer with cider running they'd use C-x C-f and input an url like "localhost:33500/your.games.ns.core". Format is "server:port/ns-name.here". If I can make this server by extending nrepl, I probably could make the format such that host:port/ isn't compulsory and if left out, Emacs would just use the default nrepl connection. When pressing C-x C-s in a buffer that's been loaded from a server, it could remember the path in a buffer-local variable, send it there and the server would either save the new source or ask the client to merge if there's been new material from another client after the last read on this client.

Format of the asset

The assets in the game server's registry is simple. Relevant properties are :id with which you refer to the asset in-engine, :name (which is completely irrelevant for development, and is used only in the editor's domtree as a prettier string than the id) and :parent-id. :parent-id belongs to the set of the loaded map-ids. It matters also in a way that when :selected-map changes in the registry, those scripts with the new :map-id as their :parent-id will be run. :order specifies the order in which the files are loaded. The most import property is :src. It's a textual representation of the source, what will be sent to the editor requesting it with C-x C-f and what will be overridden when (C-x C-s)ing. There might be more properties if I implement the support for concurrent editing.

Watches are installed when loading map's scripts. Scripts are autoloaded based on their :parents. There's no automatic cleanup of the old map's scripts, because generating a complement of an indefinite impure function is somewhat difficult. If you require cleanup, keep a hold of your watches' ids and install a watch on :selected-map that drops them.

There are also the game's :onload and :onclose which are run on process startup and process shutdown. To those you bind dedicated script assets in the game editor.

Zonetiles

This became a bit longer post than I anticipated. Bear with me, this should be the last title.

Zonetiles are an old concept based on the idea that a certain code will be run when there's a sprite entering a certain tile. My favourite use case for zonetiles are the doors to houses or dungeons or whatever. In the past they've been implemented as a hashmap of [tile-x tile-y] => lambda. That doesn't cut it currently, though, because entering lambdas sucks without all the base work I've specified on this text. Besides, simple coordinate-lambda - mapping is... a bit too simple in way.

Instead I plan on making a zonetile api in which the user filters the set of current map's tiles they wish to set the zone in with an indefinite predicate. Then they filter the sprites the zone matters to with another predicate. Then every whateverth millisecond, the engine shall search all the colliding tile-sprite pairs and call a zonetile lambda with their ids.

Conclusion

There's a lot to do. First I implement the script assets in the merpg editor. Then I'll research how to implement the file server. Afterwards is time to hack Emacs. When that's done, the editor side of this project is done. I think running the game shall require an optimized mode where it doesn't, for example, rerender the whole map every frame. Only when it has changed. After that mode is done, I need a way to dump the project image to disk as an executable jar file. Then... it's... playtime?

A slight file format changed cause by implemented sprites and animations

I implemented sprites, both animated and static, in the memapper codebase. That required some changes on the file format. This document applies from tag sprites-implemented onwards until otherwise mentioned (or src/merpg/IO/out.clj gets undocumented changes).

Surprisingly you can't view a png image and say whether it's certainly a tileset, a sprite or a spritesheet. Thus I had to change the tilesets' filenames to follow format "TILESET - :kwid - Tileset's name.png". Then I implemented saving of the sprites. They (both sprites and spritesheets, which are to be split to a list of frames on load) follow the filename format "SPRITE - :kwid.png".

Sprite metadata is put into the file called sprite-registry. It's a Clojure map, where keys map to the sprite filenames. On this registry is saved everything about the loaded sprites that can be serialized to s-expressions. In other words, everything but the concrete image data. Expected keys of the sprites and spritesheets are found in the /src/merpg/mutable/sprites.clj.

Next, I think, I'm supposed to design how the hell users are supposed to script this engine. Wish me luck, there's some hacking around nrepl and emacs to be expected

MEMAPPER 0.1.1

I pushed the new memapper to github. It's been tested to run on Arch Linux, almost-latest OS X, and it should work on Windows, but I don't have any with a relatively recent jre to test on. Reports and comments are appreciated, but the editor is so barebones/useless that I'm not going to be disappointed by the lack of them. Here's the changelog copypasted from github:

Changelog:

• A completely new state model which makes stuff accidentally both simpler and faster by using Reagi library
• GUI is a lot cleaner due to new state model making it easy to put the whole dom in a treeview.

New MEMAPPER-specification

Because after 7 years I still have no functioning syntax highlighter on this blog, I'll be using github's gists instead of <pre> - blocks when necessary. I've been toying on migrating this blob of documents to a more programming-friendly environment, but that's obviously not done yet.

I'm really close on releasing a new version of memapper. This new version was a bit delayed by Java's Swing's JTree being a horrible pain in the ass to work with and by me having to do actual work for my degree last semester. But after a bit more than a year the state model is redone, the mess of listboxes is replaced with a treeview (which makes the relationships of parents and children explicit), and the app saves and loads projects correctly again. On this text I shall document the new format somewhat informally.

The new state model is based on a single hashmap on which all the components are saved. There is a single canonical place (merpg.mutable.registry/registry) where the state is saved. This format is optimized for quickly forming the dom tree. There are "views" that transform this registry to other formats that are more optimized for other things by using Reagi-library in background threads. Thus the rendering can be done by looping through a three-vectors deep data structure and getting the relevant data by just dereffing indexes (O(1)) instead of continuously hitting the central registry and filttering tiles based on their :map-x and :map-y (O(MFG)).

In the *.memap file (which is a zip-file) saved to disk there's a file called 'registry'. This is just a snapshot of whatever data there is in the registry, created by (pr snapshot-of-registry-without-tilesets).

Here is an example of what the registry could contain. The first map's keys are obviously the IDs you look stuff in the engine up with. Values are usually maps themselves too. All values have the following properties:

• :id

  The same as the key you get this object with. This exists because values need to know their keyes even though the keyset has been dropped from the registry

• :parent-id

  Who's this object's parent? To which map does this layer belong? Etc. Etc. Objects that want to be visible in the domtreeview's top-level need to have :root as their :parent-id.

• :type

  Is this a :layer, :tile, :tileset, :tool or what?

Further keys are either self-explanatory in their context (as if :D), or have been documented before.

Additionally there are also .png - files in the .memap - zip directory. Those contain the tilesets which obviously can't be serialized to s-exprs. As in the last specification, dimensions have to be divisable by 50px. The names of the .png files are important. The format of the filename is the following: ":clojure-kw-to-us-as-id - Tileset's name.png". Those filenames starting with : will cause grey hair to those hacking these files on Windows, but I believe zip format's filename requirements equal those usually seen in linux's filesystems, only disallowed characters are / and \0, so I don't care.

The string in between the first dash and the file extension is used as the tileset's name in the domtreeview.

That's it, I guess. During weekend I try to push a new release to github, after which I'll begin adding game objects and writing down how I've thought the scripting to work.

I'm alive - and I've been hacking ClojureScript

TL;DR: I've been making a thing. It's not finished, but I might hold an interest long enough to finish it. Test instance of this thing is running on Heroku. Sources are in the prehistoric IM-project github repo of mine. Try it if you will, don't if you wont. There's only friend-requests, friendship and the basic messaging implemented.

With that done, hi. It's been a while. The timesinks after the last blogpost last august have been: I accidentally found myself in a romantic relationship, graduated a year early (well, I've done all the required coursework. It still remains to be seen if the bureacracy shall grant me the degree) from the polytechnic and related to that, I've been doing little projects here and there which either have never been finished enough to write a blogpost of or have required secrecy. After writing my thesis I relocated to Tampere. Then I began hacking.

I tried to apply to a Clojure job in this city. By doing that I also found out I'm probably not going to get employed if I can answer the question "Have you any experience on ClojureScript?" only with "None whatsoever, but I can try if you give me the specs". Well, after reviving the prehistoric IM-project design and hacking a server in Clojure and a client in ClojureScript in a couple of weeks, I can now answer "Yes, I know ClojureScript" and show what I've done with it.

The client is implemented with Reagent, the cljs-wrapper for the React - javascript framework. Had I been doing this in Javascript, I'd feel somewhat dirty right now, but with Reagent building the stateful UI was just building the dom as a clojure tree. With re-frame MVC what-ever managing the state of the app was even easier than in a traditional desktop app. Even though the introduction was a pain to read, I quite like what it does and will search if there's anything like it for the desktop apps using Seesaw.

On the server side it's a pretty basic ring/compojure rest api, using PostgreSQL and designed to be run on Heroku as easily as possible.

How shall I develop it further?

Issue-tracker provides a rather nice roadmap and a snapshot of the current status. It's easyish to break the ui with large display image (now that I think of it, it provides a nice xss-ish attack vector too), the user can't change their own details after registration and there are a few rough edges in the UI.

After admin-ui is done and the image sizes are limited, I'll probably research how websockets work and try to make real-time poll-less notifications with them.