Anger's Alternative

ok guys, been a while i know but... the game engine is officially past the minimum requirements where it can support actual games.

below is sample code for the test scene, and the result. nothing fancy to look at but it supports the basic elements of a game, and the engine architecture is such that aside the bound calls to the engine (gs namespace) the entirety of the engine API is open-source in Lua itself (the dofile line - works like an 'include' command in other languages but with dynamic compilation). in short this means that if you don't like my wrapper API (aka GEO), you're free to write one of your own, or even, if you dare, just use the engine bindings directly (not advisable though).

pics or it didn't happen:

Threadus resurectus!

I see you've been keeping busy

Loki wrote:Threadus resurectus!

I see you've been keeping busy

indeed. and still am

i've been waiting for this stage for a long time - hard part's over now, the proof of concept exists, and now i'm in 'build mode' refining the API as i'm putting together a moonbreakers-like test game.

of my original 7 steps (almost two years ago, how time flies...) -
1 - meh, when other people become involved i'll deal with it.
2 - done. opengl works without any problems this way, assuming other libraries will behave.
3 - model rendering, and display lists on the way, textures pending resource manager module, otherwise done. sfx - leaving to later.
4 - GEO API above covers this, net code is another thing that can wait a while.
5 - basically where i'm up to.
6 & 7 - visible at the end of the tunnel.

furthermore... i'm still open to what i suggested back then...

myself from 2 years ago wrote:what i am looking for once the client is in a stable state and ready for game dev:
anyone with game design / balancing experience.
anyone who likes working with scripting languages.
modelers / graphics designers / audio engineers / etc.
more testers.

I can commit to some testing, but no longer have the time available for project management.

ok, that will do... either this weekend or more likely next, ill have something semi-working to test / play around with. mostly i'm curious what platforms this will work on because compilers.

Good to hear from you again. I don't have time to directly contribute to your project. You could use your game engine to develop a learning/testing environment for the AI. Sooner or later advanced learning environments are going to be needed for any advanced AI development.

never worked with AI learning or testing directly this way, but i imagine these are more proof of concept tasks to test & research with? i just don't expect a computation-heavy AI to be implemented well within a game engine i suppose...

The AI is going to be the player, not an npc. So the heavy computations are outside the game engine.
Yes it's proof of concept tasks to test/demonstrate the learning capabilities of the agent. Just imagine the AI is like a newborn and has to learn everything. When it fails it gets punished by the game and when it wins it gets rewarded. It's reinforcement learning, as if you would train an animal.
Here's an example: https://www.youtube.com/watch?v=EfGD2qveGdQ

makes sense... i'm thinking a TCP interface or pipes should suffice for the AI to interact with the engine / environment. at least from my end, it is easiest to assume that either a player or an AI of some sort can connect through some common interface and take command of an object / entity / structure (and receive feedback for it the same way)

regarding the protocol of comms once a channel medium is agreed on, not terribly important how it is done, from my end anyway.

Abstract, when your nascent AIs grow up and take over the world, I just want you to know that I think your work is amazing!

Thanks Loki.
In case you want to hear a guy who's been working on building human level AI for the last 30 years:
https://www.youtube.com/watch?v=dcZrP3RbWhk
https://www.youtube.com/watch?v=L6LMZ3MyFCM

the-anger sounds like you know what you're talking about. I might invite you to work with us during September. I might also invite some developmental psychologists to help designing the learning environment.

awesome
see how things look in september then i guess...

at the moment i've taken up rewriting the API somewhat - most of it was a rough draft so i havent had the chance to abstract out all the core behavior into one library. on the plus side, after this rewrite the API modules' code shrinks about 80% so... worth it i think... well... when the FSM code becomes hard to deal with, there is already a problem, and that's what happened with the state module

ok, that's better... state module now fits in one screen like it should - the core Lua API is now 400 lines of SDK code for the engine like it's meant to be, and the state module is now barely 70 (about 30 taking out the whitespace) lines and impossible to misread.

now to do the same with the render module, hopefully within a few hours.

point of the 3-day rewrite was largely the SDK / API - now it takes mere minutes to extend the SDK / API with a new module, instead of hours. really good news is it doesn't take long at all to export more functionality from c++ as well; i hate the acronym so much but appropriately the engine has entered a RAD phase.

quick update on where things are at.

first, have a gander at this - my own rendering of a viper model. just to show it can be done.




the engine is maturing, slowly, though it's at a point where it's becoming quite easy to add on modules like sound and network support... but it's become impossible to avoid the issue of multithreading anymore when all of these things need to compete for processing time, and one process / thread just isn't going to cut it.

i've recently (2 weeks ago give or take) commenced a final rewrite, after which Lua will have the capacity to fork / branch into multiple threads all it wants. Yes there are tons of ways to do this already out there, LuaJIT or Lanes or what have you - I need specific types of parallelism that don't warrant such a heavy-handed approach, so writing my own.

side note - don't attempt to learn multithreading quickly. it will drive you insane, take it from me. not because it's hard but, because there's little in the way of standardized, time-tested approaches - mostly cowboy solutions everywhere.

and lastly, google code is shutting down (because google) so im hosting this on gitlab under a new name: volt-oven*. will be made sharable when there's something worth sharing.

*Volt - sentimental, name of a game engine design that inspired this one; OVEN - engine architecture philosophy, and pun: turns your computer into an oven from the strain of running it.

quick update on things.

engine rewrite is off to a glorious start (for once): Lua running in its own separate process faster than i thought it'd take, and i'd already known of a means to transparently run Lua functions on a separate Lua stack... in short, the engine is now capable of running multiple Lua-controlled processes, which themselves will be able to make more if they need to; the rewrite has accomplished half of what it set out to do.

currently i'm in the middle of designing how 2 or more Lua threads interact with one another. specifically how they will do this from within Lua, and pass data back and forth (each Lua stack tracks memory usage separately, making things difficult). basically, threads will have RPC-style calls with minimal need for low level control.

and still to go with the rewrite, a revamped resource API geared towards memory speed and parallel access / processing.

at the rate i'm going, that'll take about 2-3 more weeks.

from there the graphics api will need to be redone to work on precompiled geometry only, and at some point during that, a revisit of the input api. all in all a week or two, from how things look now.

so.... about 2 months from now should have the engine at a point where it's rendering textured models in a multithreaded environment. from there.... additional modules (sound, network, aux-input) and it will be fairly quick to put together the actual game (in terms of controls and graphics, sans shaders, at least).

myself wrote:will be made sharable when there's something worth sharing.

https://gitlab.com/TheAnger/volt-oven

i've removed the binary / exe since it changing every test build was eating up more room than expected, will add it back in when the engine core is stable enough that it doesn't need to change on a daily basis. instead i've added an output.txt file to show the program trace / debug messages being shown right now (which is about all it will do for the time being, until gfx is redone) - doesn't say much except show that Lua is indeed running in multiple threads.

otherwise... i guess feel free to browse the source code if you're curious how any of it works / will work, i've nothing to hide there (yet). if you have a gitlab account (or make one), feel free to add suggestions / issues etc concerning the engine.

quick update.

Lua inter-thread comms is about half-way done, it involves 3 methods -

1. global lock - global critical section, basically. implemented already, didn't take long.

2. thread message queues

basic idea is that each thread has a queue of values it can read in, which other threads can add to from the other end. adding and reading operations require locks on the queue, so in essence a thread can continue uninterrupted while other threads are sending it values, and vice versa when it needs to read any queued values, other threads don't have to wait until the read happens to continue with their tasks.

Lua threads can already export Lua values to other threads' message queues.
threads can't read their own queues yet, and intermediate value memory management needs a solid once-over to make sure it isn't leaky, though that shouldn't take more than a few days.

3. superglobals (Lua global = thread global; superglobal = global to all threads)

essentially mimicking an Lua table, the fields of which are 'superglobal' / ENV variables.
implementation depends on the same mechanism that allows message queues to hold Lua values.


resource API and superglobals will be merged in some form - they are more or less the same thing, and it's up to the Lua code how to use it / expose it to a game.

that much i should get done by May - the remaining modifications will take a few more days, and a week possibly to design the Lua bindings to create / start / stop threads, send them code / messages and interact with superglobals etc.

all goes well June will be spent on rendering and input rewrites - most of it is going to be cut-and-paste with refactoring but i'm still giving it a month to squeeze in some improvements / additions.

July - will have to see how things look. probably be taking a short break. otherwise will be rewriting the Lua-side interface (formerly GEO) to be as lightweight as possible and store all app-global objects as resources / superglobals.

August - this is looking like the month where the core of the game itself will start to manifest. by this point there should be full model rendering support (sans shaders and multitexturing unless time permits it) as well as easy-to-use game logic constructs for Lua.

September and onwards - see how things go, around then i expect to start adding features to the engine on demand / as needed. also around then i'll be looking at formally kicking off the game project separately from the engine.

very quick update.

this was anticipated, but im still annoyed of a week-long delay due to a conceptual problem with iterating over Lua tables stored in C++.

details here if anyone cares for it. https://gitlab.com/TheAnger/volt-oven/issues/3

estimating by next week i'll have multithreaded Lua at a prototype stage. from there it's a matter of deriving a superglobal class based on LuaTable, writing an Lua interface for all these things (not hard) and documenting it all - still on track for end of April.

short update - gotten about a month / 5 weeks behind due to life things going on.

i've been slowly refactoring the Lua<->C++ read/write functions, finishing that off tonight/tomorrow night. the efficiency of these functions isn't great (nor was it going to be), but compared to IO operations on regular tables, its roughly 2-3x slower; not bad as long as each message contains at least 4 values, where it starts to become worthwhile. BUT - these are not going to be messages passed at random, they're inter-process signaling tools that are not supposed to handle large data traffic, which is what superglobals are for, a sort of "enumerable host memory space" available to all Lua processes created.

conceptually i've got superglobals down in theory, requires a reference counter to hold the data and a special userdata type to hold references to it in Lua and in the message queues.

superglobals can be passed by reference in effect between Lua stacks, so sending references to superglobals is about as efficient as sending individual values.

all goes well and time permitting, this weekend or next i should have caught up with the single-threaded incarnation and can make some proper progress again.

well, the c<->lua data transfer ops are working somewhat. need to clean up the code and revise a few things here and there. few hours-worth of effort.

bit of a change in priorities; moving into getting superglobals working next, because with that done the graphics can be re-written to work with data buffers only. the single-threaded incarnation allowed raw geometry commands, which while adequate do not offer that great performance aside display lists (which i hope to avoid using without clear bounds).

then, game objects will become proper superglobal-type objects from the beginning, which should massively simplify the API from both C and Lua ends.

OK!!!!

been a little while, bad habit of mine to be a hermit. anyway.

https://gitlab.com/TheAnger/zero-volt

I've started over, the original code was too.... intertwined to put it one way. things were being put together faster than they had working code for, so to stop that bullcrap before it got out of hand i've archived the old and started fresh with abstract objects first and better namespacing. but you can see that for yourself.

the new architecture

ive decided it is no longer a priority or need for Lua to be aware it's in a multithreaded environment (as long as the external data interface it uses is aware, which it will be).

instead there will be some number of worker threads running in Lua, processing (effectively but not actual) co-routines for every game object. roughly: every object has one or more logical 'states' it can be in, like operational modes, that determine how it behaves at a high level. this behavior determination is done once per game cycle, per object.

in practice these objects will sit in a pool of processing tasks, that the Lua worker threads pick up and process independently. for this to work of course, the game object data cannot reside in Lua, and will sit behind a thread-aware reference-counting interface.

outside of this going on, the graphics, sound and any other subsystems (eg physics / network) will run parallel. there will be more fine-tuned restrictions on what can run when, but i'm not decided on that yet and tbh it's way too early to be sure yet.

progress so far

picking up from where i left off... the non-local type is now the de-facto standard for Lua / C++ data harmony. internally to the engine, the reference type that Lua and C++ will deal with is ZV::Variant. this has to work before anything else is worth being built, excepting multithreading constructs and their primitives (ZV::Sync, ZV::Primitive).

if i can keep up the pace, and this time around it's not stalling due to technical issues being found, it's looking like sep/oct there will be a working prototype of the engine with graphics and sound.

alternate control methods for the game - TBD. anything Lua can do already for one, i'm not in any hurry to implement network code or file pipe or gamepad support etc... when it's worthwhile, and no earlier than a working engine anyway, then i'll get around to this.

network play - per above, not necessary until much later.

ETA for "much later" - sketchy. at least, november, perhaps by march next year i'd want to be at a point where even if i dont feel like it, it wouldn't burden the engine layout anymore for sidetracking additions.

been a few months since last update. some gloriously productive ones, for once.

above recent link still valid. theres a test window you can look at, and press Esc to test the input (yes it works, was hard ). you get (most of) the source too, if you can improve on it i'll gladly take on the help. odds are ill keep the engine open source but if history is any lesson i'm not ruling out otherwise just yet; too early to tell but even so, there's other ways, like protecting the game content which - unlike the engine - is most certainly proprietary.

progress?

much.

the reason there's even a test window at all is thanks to SFML - it's tricky to get working because i hate microsoft compilers and insist on being different. results? worth it. highly recommend for pet projects involving multimedia programming from basics through to, well, this at least lol.

it set me back a few months deciding on this and implementing it, along with set backs in overthinking the parallelism. breaks were had as this was a frustrating august - october.

since then i've let the ideas settle and picked up where i left off, throwing out a lot of what i've done for now and starting with SFML as a base. over the last couple of weeks i've started adding back in the useful chunks of code tried and tested before.

at this stage it's very close to being a proper game engine. im estimating primitive capacity for a game around next weekend, and around 3-6 weeks from now there's a high chance i'll have started on a game.

presently im finalizing the game object code, and have already concocted a thread / process model for the engine. the game api has been prototyped in a javascript experiment (pm me for link if you really want to see, i dont think its anything impressive, just a proof of concept in its unfinished state) so im very confident it can be built up on once it's finished.

i couldn't get to this part without finishing the lua <-> c interface and a variant type. both have more or less been finished (though not fully tested) to a working point. any gaps can be quickly filled.

very roughly speaking, the engine has 3 phases that it go through, with operations in each phase run in parallel:

1. physics & animation / core object updates

2. graphics, sound, scripting (read-only, writes-queued)

3. object property updates (queued writes), engine IO (keyboard, network, gamepad, etc)

scripting is message-based, but can't say more than that yet. within each phase there may be serialization yet, or more phases, but atm that's what it looks like.

there's definitely no more than 2 weeks of effort (at the pace im going at lately) before at the lowest level it can be used to create games...

in fact it's so close im announcing an invitation for testers. feel free to test what you can see there, but as i say, for another month probably it will be nothing of interest to 99% of people imo.

my ability to stick to a schedule is a legendary failure so feel free to check on it any time. even by that standard, i can taste how close it is now...

hopefully more good news soon...

- anger

and good news of a different sort.

whether by laziness, good fortune the engine design is compatible, or sheer chance i happened to remember about bullet, that i'm now using that physics library for the engine.

there are a lot of things happening with the engine design lately i didn't foresee; i did not expect old lua-based code to be of use for this iteration of the engine, i certainly did not expect i would need almost zero effort to integrate something as impressive as bullet into the engine.... but there you go lol...

going to do that next, because im coming up to finalizing the engine pipeline - ignoring physics would not be wise, best deal with that now.

in saying that, game object class just isn't complete without knowing the physics implementation to be used later, and the game api shouldn't pre-empt what the engine can do (unlike 2 rewrites ago).

after bullet is done, shouldn't take long for a rudimentary implementation, i'll be starting on test demos making use of physics, graphics, sound and input. in the process i'll decide on how the Lua-side of the API will work, and in turn finalize the game engine structure & pipeline.

i really wish i could finish the engine without doing it in this roundabout way, but from experience i work best upwards and towards something, not from a high level, ordered plan but in order of pragmatic need and definiteness; it has approached that point where i need to start using the engine to determine how it should work, hence the call for testers earlier though i didnt think id need to this early...

edit - just to point out how much of a game changer bullet is, it basically adds stability to the engine design with all the unknowns that a specific implementation brings. as a result, im happy to say, the hardest part left of the engine to design, is the network code. everything else is not really critical and can be implemented per-game as needed, and if something happens to be utilitarian or computationally demanding, then it gets upgraded to engine module status. it's just that at the moment, there are no engine modules! i'm writing them right now, well, the core that will allow modules to work at all to be precise. and with that, it means prototyping the API, and I wont be able to do that until i try using it from the ideal conception point of what i want a game script to look like...

alright... so far so good. bullet builds on my machine, and the engine compiles with it being used. things have accelerated (pun).

in doing so ive archived the code i had and am changing the build order for what i hope is the last time. atm the engine has been reset to a clean slate, and i'll build the engine components as-needed from the code i've collected and written so far. this should take a few days to get done.

more updates soon...

alright, final major rewrite going off to a good start.

bullet is working beautifully, with the crappy test demo it has (sphere on a static flat plane, numerical sim only). but... key point is that it built, compiles, links and now confirmed that it functions as it should too (was a bit worried as it defaults to float, and Lua not having an integer format relies on the larger-than-int precision of a double for such things, so i built it with double btScalar to avoid unnecessary conversions internally where possible)

i've revised and slimmed down variants. rather than them having a subclass to store refcounted data, it will now do the refcounting itself for known variant types that would probably be better off being copied explicitly - specifically str, queue and map subtypes.

i've abandoned trying to give Lua any pointer data to game objects. it can only be abused and an id (typedef size_t id) is plenty to refer to game engine objects by context (eg, in a texture bind call, its a texture id, in an object search call, its an object id, etc). I'm still leaving the option for pointers (userdata will be converted to light pointers, memory ownership will not cross Lua/c++ at this time as it makes a bottleneck dependency) it to be transferred in an out of Lua stacks.

in terms of code produced, ive made a quick version of shared_ptr (because gcc and c++11 on windows is head banging grade stubbornness to get working, not even boost likes my system / setup) - zptr. using this as a base im writing a "selective" shared_ptr variant which does the role of both a variable scalar container and a pointer refcounter container. you should get a good idea of how this is achieved from what ive finished just last night - files /src/var/base.* .

when zv::var is finished i'll thoroughly test it including subtypes. if all goes well (i'll know tonight probably, day job and all) it's going to be a head-first dive into creating a Lua helper class. i started on this before this rewrite, but only barely. there was a lot of potential to create a complete Lua interface (that is, turn the c-style calls and a stack instance into a lua stack class with inlined class methods). that is clearly overkill.

however there was an ambiguity that this rewrite resolves - does zv::var or zv::lua classes own the code to read and write to/from eachother? or should it be split? i informally went with "lua depends on var" and in this rewrite it's going to be an explicit dependency.


so.... finishing off variants, testing them. bringing back the lua class i had after that, and building the var <-> lua functions into the lua wrapper class itself. thanks to var's copy, move and swap semantics being an assurance now, it will keep the code almost entirely related to lua itself against the lua wrapper class, otherwise lua or var had to partially pre-empt one another and that meant global operators (ew) with code in yet a third place. no thanks. started doing that, not a good idea, doing it right this time.


i assume the lua import export code ive made 'mostly works' - ive rewritten the damn thing 4 times in 2 weeks, only to stick with a minor variation on what i had before (some loops collapsed, rewrite to reference passing vs ptr, etc). if you've seen the code for it before, you'd understand why im so nervous using it - the import / export i made sure would work with lua even with limited stack size. eg, the export needs 5-7 stack slots at its peak and this includes walking through nested tables (no recursion checks though, im hoping i or other game writers using the engine would be more careful than that).

in any case i'll be testing this tonight / tomorrow, so at some point saturday, the main engine loop and game objects will be started on (assuming few delays / issues up to that point). i have a good idea of the engine pipeline now that all 7 critical engine subsystems are (more or less) able to be built now to well defined specs - scripting, networking, graphics, input, physics, sound, resourcing / object management.

i'll probably do graphics first, so objects can be at least seen with a mesh-compatible structure (texturing too; i have the code to load wrl and tga files done for Lua already so this is not a problem to prioritize). next, they need to be scripted so the graphics can be defined via script, which will invoke building the main engine cycle / loop to handle objects. in turn scripting would be started on, enough to get basics going, followed by linking game objects to bullet objects, exposing some properties (position, orientation) and using them to transform the object for rendering.

it's possible to finish this to a very basic degree - room full of spheres - this weekend or early next week.

from that point extras - like sounds - will be mostly re-using how the above code will work.

networking requires a special touch and will be all but last. it is a topic in and of itself one could spend years on.


so. rambling aside that's the short term roadmap for the engine at this time. and you can see, at this pace it's likely a game could be seriously done using the engine in about 3 weeks time, give or take.