Other tasks

[pez]

Are there other tasks that haven't made it to the forums yet? The config dialog is almost working so after that what should I try next? I don't know if any of the current tasks are within my abilities

[hughperkins]

Config dialog is very cool to have!

First I'll list all the stuff I'm thinking of putting into 0.6.1, like once I've trekked through Laos and stuff and come back out in civilisation (few weeks).

- fly button. This is almost done, just need to test/debug. In theory you hit "fly" button and it turns gravity off/on for your avatar object

- add phantom. New property of OBJECTs, new entry in database for objects. In odephysicsengine.cpp, quick check of if( !p_Object->bPhantom ) before adding an object into physics engine.

- skybox, whole sim. New contextmenu option "Sim admin..." -> "SetSkybox". Sends message to mvserver, via mvclient, "<simupdate><skybox texture="..."/>", mvserver sends it to dbinterface which stores it in simconfig table in database, (table structure: create table simattributes ( attributename varchar(32); attributevalue varchar(128); ) ). mvserver sends update to all clients, which will update the global skybox texture value in a global variable, and henceforth display this texture as the global skybox

- skybox, terrain. New context menu option "Set skybox...". Sends message to mvserver via mvclient "<objectupdate type="..." ireference="..."><skybox texture="..."/></objectupdate>
". Mvserver sends to dbinterface which stores it in updated objects table. Mvserver sends message out to all clients. Then the tricky part: upgrade physics engine dll public interface to allow for detection of intersection of your avatar with all skybox prims. If avatar intersects a skybox-prim, show the corresponding skybox, otherwise show global skybox.

For an example of a skybox, download nevrax Snowballs, and hit the function keys till the terrain and skybox appear in wireframe (f9 or f10 I think?), you'll see the skybox as a huge sphere which doesnt move relative to you.

- propagate collisions to scripts

This is a chunk of work. Let me know if you want detail on this. The physics engine detects all collisions, we just need to propagate them to the scriptingengine. It's not really complicated or mind-numbing, just a lot of work.

- object inventory

Chunk of work. Let me know if you want more details on this. I have them written down in a notebook (as with all of this), but it'll take me a while to type in.

Basically, this will be a new contextmenu entry to view the contents of an object. When you select it, it copies the object contents to a temporary local directory (copying from cache things like textures and so on, and using reference/link files for items to which you dont have access), and opens it in Explorer. When you change stuff it saves it to the server. When you close the window it saves any remaining changes to the server and deletes the temporary directory.

- file association, .sim

Runs simloader.py which will create a new database with the same name as the file, call restoredb.bat on the file then load mvserver, with a new option --db, followed by metaverseclient

The idea is that when you click a .sim, which is a database snapshot file, it loads it into a new database, loads metaverseserver and loads the client so you find yourself in the sim.

- file association, .mvl

File that links to a sim/server already running.

File structure:
<simlink serverip="..." serverport="..."/>

File association is to simloader.py which detects the file association and starts metaverseclient with appropriate options to connect to referenced server

- .bvh file loading, rigid mode

.bvh files are just texture files with two sections. The first section gives you the names of the objects to move, with their offsets from each other and linkages. The second part gives you each keyframe with a time and a list of rotations for each object/prim.

The way this will work in OSMP is that you create a linked object (flat, no hierarchy) with the objects having the same name as in the bvh file (hence the addition of prim/objectnames recently). You can run a bvh file from a script, and it will just move the objects around appropriately. It's reasonably trivial to do - much easier than it sounds.

Note that this is a first step in .bvh file loading, since we are just moving the objects, not warping a mesh. What this means is that arms, legs, fingers, heads and so on will work fine, but you wont be able to make someone smile or somethign like that.

One really cool thing about OSMP .bvh file loading is that you can use it on your avatar but you can also use it to make animated NPCs.

- authentication server

The idea of an authentication server is to:

- centralize authentication so that you dont have to leave your password on multiple servers
- centralize authentication so that hyperlinking becomes transparent (no need to reauthenticate each time)
- and it can provide a list of connected MetaverseServers so it can become a switch-board for multiple sims.

Note that I dont envisage having a single universal authentication server, though it's not excluded, but rather small communities of sims linked by a single auth server.

The way it works si that the auth server starts up, and stays up, then the various metaverseserver sim servers connect to the auth server.

The auth server holds in memory the list of connected sim servers, and uses this to provide a list of Metaverse Servers to clients when they connect.

When a client connects to a MetaverseServer, the server will check with its configured auth server to validate the client.

The MetaverseServers have the address of their auth. server in config.xml.

Correlation between clients will be initially by IP address, for simplicity and because I dont anticipate hacking at this stage. Spoofing is therefore possible. When spoof-attacks start to become a pain, we will introduce a connectionkey, which is a random key generated when you connect to the auth. server. You could obtain this key by sniffing, but that's not really likely for now, so basically if you know the auth key, the Metaverse Server assumes you're the same person as the person who authenticated to the auth. server lately.

The connectionkey will be jsut another attribute in the messages; but like I say we wont implement it just yet because it adds no value yet.

The authentication server itself... well, we could use a Kerberos server at some point; for now I'd be tempted to just use a hacked MetaverseServer.cpp as the auth server.

- Keyboard event

Allows scripts to capture keyboard events from clients. Any keyboard event, not just up,down,left,right.

Again, I've got the details in my notebook; I'll post them on request.

- Send table through synchro RPC

This is fairly doable, teaches a lot about Lua, and is a very useful unique selling point for OSMP.

Right now, you can make synchronous RPC calls to other scripts, but you are limited to passing a single string as the message parameter.

This upgrade will make it possible to pass *any* single value as the message parameter, including nested tables.

The code that handles synchronous RPC is in two modules: scriptingengineprotlua2.cpp and LuaScriptingStandardRPC.cpp.

The modification requires knowledge of using the Lua Stack to pass arguments to and from Lua scripts. You can see examples of this in the current Lua modules (LuaPhysicsAPI.cpp, LuaScriptingStandardRPC.cpp and so on (these names probably arent the exact module names but they're close enough))

- customizable timers for Lua scripts

- Make it possible to configure timer duration in Lua scripts
- Make it possible to add multiple timers for one script

Notes available on request

- WAN Compression

- Compress XML into binary format and decompress at other end.

Basically, load the xml into some standard structures, and pump the binary structure down the wire.

Some examples of how to do this:

Message structure will be:

struct objectrefresh
{
char cBinaryTag = '!'; // (The "!" indicates binary, "<" woudl be xml)
enummessagetype type = MESSAGETYPE_OBJECTREFRESH; // dont think you can initialize this here, but putting it here for clarity

int iReference;
enumobjecttype type;
wanpos pos;
wanrot rot;
wanscale scale;
wancolor color0;
char sTextureReference[33];
};

enum enummessagetype
{
MESSAGETYPE_OBJECTREFRESH,
MESSAGETYPE_OBJECTMOVE,
MESSAGETYPE_OBJECTUPDATE,
};

These three message-types (in the enum above) take up the bulk of our bandwidth, so at this point we'll just binarize these messages.

ObjectMove struct will consist of a simple header containign the iReference of the object, the number of MoveStructures, and the move duration. MoveStructures will look like:

struct MoveStruct
{
enummovetype movetype; // whether this is pos, rot, color, or scale
single x,y,z,s;
};

The wanpos type in the objectrefresh is almost identical to a POS, except that its a simple struct looking like:

struct wanpos
{
single x,y,z;
};

rather than the huge POS class object we use within the client and server itself for internal processing.

(And same for wanrot, wancolor and wanscale)

- add debug module

We really need to replace all our couts with a debug command that still uses streams but is switchable easily on and off, eg by changing a value in config.xml

There are a few libraries around to do this. libcwb could be interesting.

- add Lua script error handlers

It would be pretty cool to get a stack dump when a script crashes for whatever reason, and pass this up to the user as a Say.

There is some documentation in the Lua Reference Manual on this.


I think that covers most things...

What is easiest? Partly it depends on what interests you most. Also, dont hesitate to add your own features (eg texture animation).

The Lua scripting items could be a good place to start because most of the code is in two modules: the luascriptingengineprot2 Lua scripting engine module, and the module containing the Lua function (eg LuaPhysics.cpp or LuaStandardRPC.cpp).

I think that being able to pass tables through synchronous RPC would be very cool, and being able to pass nested ones would be awesome.


What do you think? What thinks would you most like to see in OSMP at this time?

Hugh

[hughperkins]

Hmmm, I guess that was kindof a long post.

How would you feel about working on sending tables through synchronous RPC in Luascript? You'll want to check the Lua Reference Manual (available at http://www.lua.org). The module impacted is LuaStandardRPC.cpp.

Hugh

[pez]

Wow thats a huge list ! :)

I'll try to tackle the following items :

- Send table through synchro RPC
- Keyboard event for scripts, capture any key
- Make it possible to configure timer duration in Lua scripts
- Make it possible to add multiple timers for one script


Should take me a good couple weeks

[eggy]

Could we just use gzip as the "binary translation layer"?
We talked about it before and XML compresses really well. It's not like we actually need it since we don't have bandwidth problems (yet).

[hughperkins]

Ok, good choices.

> How hard (just wondering, no plans to really try this) how hard would it be to write a Perl Scripting library?

You mean so you can use Perl scripts in objects?

Ermmm.... it really depends on the Perl runtime more than the language itself.

Basically, the scripting engine must be capable of running in a fairly secure sandbox. Java and Lua are both good for this: Java is built from the ground upwards to be sandbox-compatible, Lua was built deliberately for creating secure game scripting engines.

There are a few strategies for running a particular scripting language in a secure sandbox (and I'm sure this list is far from complete, you might be able to think of others):

- rigorously process and filter incoming scripts then feed it to native runtime. An example of this is if you wanted to use C as your scripting language, then you'd strip out all commands beginning with #include, and all pointers. Actually, you'd probably have a list of permitted commands, and strip out all other commands.

- use a runtime that can run in sandbox mode. This is what we do with Lua which runs natively in a secure sandbox. By default the scripts cant do anything, not even print to the screen; you can choose exactly which commands are available to scripts

- write your own interpreter/compiler/runtime from scratch. This is a big job. I really dont recommend this as a rule, unless you're passionate about writing one. It does have several advantages, like very fine control over how it runs, but it's just a lot of work

Looking at the details of sandboxing, there are three things you need to guard against:
- hogging all the CPU
- hogging all the memory
- doing illegal stuff like formatting your harddrive

The first two are not too critical, you can simply launch each script in its own thread or process and kill it if it goes over a certain limit.

The last one is the critical bit.

If you can handle the sandbox part, the rest should look after itself.

Note that the scripting engine can be written in any language you like. You could have a Perl engine written in Java, a Java engine written in Perl, pretty much any combo you like.

Hugh

[hughperkins]

Eggy, gzip would compress the xml a lot, but converting the xml into binary by hand will give a better compression because the converter understands xml keywords.

Basically, a binary converter can take a keyword like "objectrefresh" and code it as a single byte value. gzip would have to run compression on the raw text "objectrefresh" which is both relatively long to do and will probably result in about 3-4 characters, which is 300-400% larger.

Note that it's quite possible that we will gzip the binary. We'd need to check compression ratios and CPU usage before making a decision on that.

Hugh

[pez]

Ha I tried to remove my question about a perl scripting lib because I figured I wouldn't understand the answer but you replied too fast []

[hughperkins]

The following doc tells you how to interface perl with C:

http://www.perldoc.com/perl5.6/pod/perlembed.html

(but actually you could write the entire engine in Perl).

Havent found anything yet on Perl sandboxing, I'll keep an eye out for that.

Hugh

[hughperkins]

Oh, by the way, I was having a look at writing a Python engine before, and it looks fairly doable. There are a few ways of doing this. Python actually comes with a Restricted Execution Framework, which is a module that allows you to run Python scripts in a sandbox.

I know Python isnt Perl, but just thought Id throw this in.

Hugh

[filo]

The python sandbox isn't secure ATM.
_________________

[hughperkins]

Well, the documentation says it isnt secure but doesnt go into details. If you dig around, it says that it cant guarantee that a script wont use up all available memory, CPU or crash the process.

Whilst none of these things are brilliant, we can deal with them, they dont worry me. You can control memory and CPU by just monitoring the process and killing it if it starts running away. The process crashing is not really bad either, you can just restart it.

Dont get me wrong, it'd be better if you could prevent all these things at the source, but they're dealable with, and it's a lot easier than writing your own engine from scratch.

Hugh

[hughperkins]

Pez,

The following Perl module might be useful for creating a Perl scripting engine:

http://search.cpan.org/~micb/Safe-b2/Safe.pm

Hugh

[pez]

thx Hugh. It's a dream of mine to be able to script in perl and use perl Hash'es in a 3d world. But for now I'll try to stick with small tasks [:I]

[eggy]

Yeah, and a feature voting script to help us prioritize stuff