Notice: Any messages purporting to come from this site telling you that your password has expired, or that you need to verify your details, confirm your email, resolve issues, making threats, or asking for money, are
spam. We do not email users with any such messages. If you have lost your password you can obtain a new one by using the
password reset link.
Due to spam on this forum, all posts now need moderator approval.
Entire forum
➜ SMAUG
➜ Running the server
➜ automatic shutdown?
It is now over 60 days since the last post. This thread is closed.
Refresh page
Pages: 1
2
| Posted by
| David Haley
USA (3,881 posts) Bio
|
| Date
| Reply #15 on Wed 02 Jul 2003 10:26 AM (UTC) |
| Message
| Thanks for the pointer Nick. That basically shows me what I wanted to see.
Eep. My MUD uses up 59 mb of memory, and the next one on the list is 29mb. I tried rebooting the MUD thinking it might be a memory leak (wouldn't surprise me considering the sloppiness of certain critical parts of the code), but as soon as it restarted it was back up to 59mb.
In any case I was just wondering if this could be simply because of a whole pile of rooms in the game. We have something like 25,000 or even 30,000 rooms... I guess I'd have to go through and look at the data structures and see what's going on there. I know one problem is that there are many places which use longs or ints, when they could be using unsigned ints or even shorts... but anyways... Just for comparison's sake, are there are other MUD admins out there who have memory usages that high, and what are your room counts and such?
Thanks... |
David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone
http://david.the-haleys.org | | Top |
|
| Posted by
| Nick Gammon
Australia (23,173 posts) Bio
Forum Administrator |
| Date
| Reply #16 on Wed 02 Jul 2003 09:17 PM (UTC) |
| Message
| I wouldn't get too excited changing longs to unsigned ints.
All of the following data types will use 4 bytes: int, unsigned int, long, unsigned long.
Even changing to short, which would nominally save 2 bytes, might not because the compiler might allocate them on a 4-byte boundary, plus you then have the problem of the things they are storing (eg. exp, gold) wrapping around and becoming negative when they reach 32767.
Have you tried typing "memory hash"? This is why I get doing that on stock SMAUG ...
System Memory [arguments - hash, check, showhigh]
Affects: 479 Areas: 23
ExtDes: 935 Exits: 3839
Helps: 1117 Resets: 2458
IdxMobs: 500 Mobiles: 861
IdxObjs: 679 Objs: 1339(1335)
Rooms: 1702 VRooms: 0
Shops: 32 RepShps: 1
CurOq's: 0 CurCq's: 0
Players: 1 Maxplrs: 2
MaxEver: 2 Topsn: 266(500)
MaxEver was recorded on: Sat Jan 4 14:53:58 2003
Potion Val: 0 Scribe/Brew: 0/0
Pill Val: 0 Global loot: 2
Hash statistics:
Hash strings allocated: 11296 Total links : 30442
String bytes allocated: 1549244 Bytes saved : 392971
Unique (wasted) links : 7679 Hi-Link count: 7605
This seems to suggest that the strings take over 1 Mb on stock SMAUG. Plus you could multiply the count of various things by their structure sizes (eg. 861 mobs by whatever a mob takes).
Another thing you could try is finding the size of your area directory (eg, "du -sh ." ). The size of the raw area files would be a guide to how much memory they would take when loaded in. In my case I got 2 Mb which tallies reasonably well with the 1.5 Mb shown above.
|
- Nick Gammon
www.gammon.com.au, www.mushclient.com | | Top |
|
| Posted by
| David Haley
USA (3,881 posts) Bio
|
| Date
| Reply #17 on Thu 03 Jul 2003 09:05 AM (UTC) |
| Message
|
This is what I get when I type memory hash. Note that my MUD was derived from SMAUG 1.0, and evolved nearly completely independently since. I don't know if the coders before me regularly updated the stock code parts with code released from the SMAUG team.
memory hash
Affects 991 Areas 72
ExtDes 2154 Exits 12172
Helps 933 Resets 4432
IdxMobs 1514 Mobs 2022
IdxObjs 2169 Objs 30913 (6558)
Rooms 5341 VRooms 0
Shops 132 RepShps 9
CurOq's 0 CurCq's 0
Players 4 Maxplrs 16
MaxEver 98 Topsn 339 (350)
Stables 5
MaxEver time recorded at: Thu Nov 16 13:01:13 2000
Hash statistics:
Hash strings allocated: 30586 Total links : 102783
String bytes allocated: 4734642 Bytes saved : 1380066
Unique (wasted) links : 19472 Hi-Link count: 33776
My area directory is 80 mb is size... so that probably means that there is 80 mb in memory somewhere, yes... I wonder if there's a way to do more intelligent memory management, where it doesn't load what it doesn't need. Hmm. Perhaps some kind of database model could be useful for such a thing... Seems also that I vastly miscalculated the number of rooms we have... I am kind of surprised though, hmm, maybe I was thinking of total number of room, object and mob prototypes. Or we have a lot of holes in our vnum usage...
Anyways, I have to admit that I don't understand why int and long are the same in storage size. I know why int and unsigned int are the same (because in signed, one bit is used to indicate + or -...) but I thought the whole point of int and long was to have different sizes?
I believe that on Linux, which is where I'm developing, char = 1 byte, short = 2 bytes, int = 4 bytes, and long = 8 bytes. Even assuming that I got confused somewhere along the line, which is probably, and int AND long are the same in size... how can I tell the compiler to allocate TWO bytes for a short and not four? I can deal with the overflow myself, and if I say short it's because I was a short, darnit :) It just seems silly to even bother with data types of different sizes if the compiler allocates the largest size in all cases. |
David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone
http://david.the-haleys.org | | Top |
|
| Posted by
| Nick Gammon
Australia (23,173 posts) Bio
Forum Administrator |
| Date
| Reply #18 on Fri 04 Jul 2003 12:17 AM (UTC) |
| Message
| My understanding is that a short is always 2 bytes, however it is the int you need to worry about. int used to mean "integer" meaning the largest available number for the compiler in question. However these days I believe int is generally 4 bytes even though longer types have become available. This is born out by my test program, below.
You don't need therefore to worry about telling the compiler a short it 2 bytes, however you might rejig ints as shorts where required. However I doubt the overall savings will be that great.
Your other options are:
- Pay for more memory - if you owned the PC this would easily be the simplest option as memory is cheap, however if you are paying for a service this might not be the case.
- Use a database - however this would be complex to implement and your service provider may well charge more to allow you to do that.
- Compress your strings. This is probably best option if you can't get more memory. The zlib library is freely available and pretty quick. I have done a test program to demonstrate, see below. Compression is generally better for longer strings because it looks for redundancies, however zlib lets you supply your own dictionary which would probably suit you very well. You could compile an (in-memory) dictionary of commonly occuring words (eg. "mob,room,you,are,walking,fighting,the,sword") and supply it to both the compression and decompression routines, thus catching a lot of common words.
My test of this seems to indicate that for a reasonable size room description you can save around 40% of description space (237 / 402 = 59% of the original).
You may be able to improve on my figures by having a longer dictionary. The example program illustrates compressing and then decompressing a string using zlib, and also displays the data sizes. Note that the structure is larger than the sum of its parts, this is because of the compiler aligning things on word boundaries.
#include <iostream>
using namespace std;
#undef _DEBUG
#include <zlib.h>
#ifdef WIN32
#define longlong __int64
#else
#define longlong long long
#endif
struct mystruct
{
char a;
short b;
long c;
longlong d;
int e;
};
int main (void)
{
cout << "Size of char = " << sizeof (char) << endl;
cout << "Size of short = " << sizeof (short) << endl;
cout << "Size of long = " << sizeof (long) << endl;
cout << "Size of long long = " << sizeof (longlong) << endl;
cout << "Size of int = " << sizeof (int) << endl;
cout << "Size of mystruct = " << sizeof (mystruct) << endl;
cout << endl;
char * s =
"You are standing within the expanse of the famous Darkhaven Square. "
"A stone statue of occupies the square's center, surrounded by gardens "
"of shrubbery which enhance the air of serenity and peace here in the center "
"of the city. The main road lead away in the cardinal directions, while to "
"the northeast and northwest are forested paths. The spires of a cathedral "
"can be seen rising to the northwest.";
char * dictionary =
"mob room you are walking fighting the sword north south east west up down "
"can occupies here not walk fight kill run sleep score city inventory nearby";
unsigned char compressed [5000];
unsigned char uncompressed [5000];
z_stream c_stream; // compression stream
memset (&c_stream, 0, sizeof c_stream);
deflateInit(&c_stream, Z_BEST_COMPRESSION);
deflateSetDictionary (&c_stream, (const Bytef *) dictionary, strlen (dictionary));
// source
c_stream.next_in = (unsigned char *) s;
c_stream.avail_in = strlen (s) + 1;
cout << "Uncompressed size = " << c_stream.avail_in << " bytes." << endl;
// destination
c_stream.next_out = compressed;
c_stream.avail_out = sizeof compressed;
// compress it
deflate(&c_stream, Z_FINISH);
deflateEnd(&c_stream);
cout << "Compressed size = " << c_stream.total_out << " bytes." << endl;
// now decompress it
z_stream d_stream; // decompression stream
memset (&d_stream, 0, sizeof d_stream);
inflateInit(&d_stream);
// source
d_stream.next_in = compressed;
d_stream.avail_in = c_stream.total_out; // bytes from compression
// destination
d_stream.next_out = uncompressed;
d_stream.avail_out = sizeof uncompressed;
while (true)
{
int err = inflate(&d_stream, Z_NO_FLUSH);
if (err == Z_STREAM_END)
break;
if (err == Z_NEED_DICT)
inflateSetDictionary (&d_stream, (const Bytef *) dictionary, strlen (dictionary));
}
// finish decompression
inflate(&d_stream, Z_FINISH);
inflateEnd(&d_stream);
// display results
cout << uncompressed << endl;
return 0;
} // end of main
Output
Size of char = 1
Size of short = 2
Size of long = 4
Size of long long = 8
Size of int = 4
Size of mystruct = 20
Uncompressed size = 402 bytes.
Compressed size = 237 bytes.
You are standing within the expanse of the famous Darkhaven Square. A stone statue of occupies the square's center, surrounded by gardens of shrubbery which enhance the air of serenity and peace here in the center of the city. The main road lead away in the cardinal directions, while to the northeast and northwest are forested paths. The spires of a cathedral can be seen rising to the northwest.
This example excludes error checking, and of course you would want to make sure the compression buffer (5000 bytes here) is large enough to hold the compressed/decompressed data, or alternatively use the zlib "streaming" technique for handling large streams of data.
|
- Nick Gammon
www.gammon.com.au, www.mushclient.com | | Top |
|
| Posted by
| David Haley
USA (3,881 posts) Bio
|
| Date
| Reply #19 on Mon 07 Jul 2003 10:19 AM (UTC) Amended on Mon 07 Jul 2003 10:20 AM (UTC) by David Haley
|
| Message
| Thanks for the info, Nick. I'm going to see how the string compression could be easily applied to my MUD... I'm probably going to have to rework the string hashing which is kind of poor (slow.) I'd do it now, but I'm in the middle of completely reworking the network code... I'll post a separate topic at some point about that. In brief I'm moving away from the typical SMAUG "flush buffer" which just keeps looping until the buffer is emptied (which can make the MUD "hang" for everyone else while one person is hogging the output channel), and instead I'm not using a sleep pattern, and just looping more and running update_handler when it's time.
That's kind of a lousy explanation, but I'll give more details later when I have more time :)
On a side note, I found some pretty cool software that might be useful... SleepyCat (makers of BerkeleyDB) also have something called BerkeleyDBXML, which seems to be a database that stores/retrieves files in XML format, and comes with XML parsers and all that. It seems to be really great and its potential is enormous. I have to do some thinking about how to best use it in my MUD, and if I find anything productive I'll be sure to let you know :) The main advantage is that you don't need a separate DB process for this to work. It uses its own file routines in a plain file format.
I'll also let you know if changing around the data sizes helps anything. I'm not convinced it'll make a big difference, but if I can save a few megabytes of memory, why not... :) |
David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone
http://david.the-haleys.org | | Top |
|
| Posted by
| Nick Gammon
Australia (23,173 posts) Bio
Forum Administrator |
| Date
| Reply #20 on Wed 09 Jul 2003 08:47 PM (UTC) |
| Message
|
Quote:
... BerkeleyDBXML, which seems to be a database that stores/retrieves files in XML format, and comes with XML parsers and all that. It seems to be really great and its potential is enormous. I have to do some thinking about how to best use it in my MUD ...
I hadn't heard of this particular program, however I am actively looking at doing something along similar lines. I wrote an XML parser for MUSHclient, and am now reworking that to be a bit more general, and use STL rather than the MFC libraries, so it will compile on other platforms (eg. Linux).
It seems to me that using XML for area files (etc.) would be a lot more flexible, you could add or remove flags and things without breaking existing areas. Also ideas like groups of things (eg groups of rooms) and ownership (eg. objects with other objects inside) could be easily expressed. |
- Nick Gammon
www.gammon.com.au, www.mushclient.com | | Top |
|
| Posted by
| David Haley
USA (3,881 posts) Bio
|
| Date
| Reply #21 on Wed 09 Jul 2003 11:58 PM (UTC) |
| Message
| Exactly. In fact, I was going to use something called FlexML first. It's pretty neat: you write an XML document definition and syntax rules for what to do when you find a given attribute/whatever, and then it uses Flex to generate a parser for you. I don't know if you're familiar with the YACC parser generators... it's what I use for the scripting language I wrote for my MUD. Basically, you give it a language definition, semantic attributes and whatnot, and it generates a lexer and a parser for you. It's incredibly, incredibly cool :) Although, I would recommend that amateur programemrs get some background in compilers before attempting to use it... abstract syntax trees and the like can be awfully confusing if you don't know what's going on.
Here's the site for a Windows version of the lexer/parser generator: http://www.bumblebeesoftware.com/ .
Anyways, I was going to use this XML parser generator, when I stumbled across this BerkeleyDBXML stuff. It seems that the major advantage it has is the database functionality. Being able to say "Give me all swords that have more than 10 damage" and having it give you a result set seems incredibly cool. And yes, you're entirely right about ownership and the like... it's an excellent, and much more readable, way of presenting what object is in what container.
STL is also very cool. I've been following the posts you've been making about it. I've been using it for lists and the like for a long time, but have never gone into a lot of detail beyond just pushing things onto the list and removing them. I was iterating the "old fashioned way" by declaring an iterator, then using a standard for loop from begin() to end() doing itor++. The for_each code you showed helps to make such code much more readable... and probably faster... The string utilities are neat as well.
Is that STL code free to use in my MUD? Who do I give credit to? Some of the stuff, the string utilities especially, looks like it could be extremely useful to me, but I don't want to steal code or anything.
Personally, just to throw in my own grain of salt, I have never used MFC and probably never will. I know that it's very useful for Windows GUI applications - but I don't write those and so I have no need whatsoever for it. So there. :) |
David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone
http://david.the-haleys.org | | Top |
|
| Posted by
| Nick Gammon
Australia (23,173 posts) Bio
Forum Administrator |
| Date
| Reply #22 on Thu 10 Jul 2003 02:55 AM (UTC) |
| Message
|
Quote:
Is that STL code free to use in my MUD?
Yes, I published it on this site to be of general help to anyone who is interested in STL. If you wanted you could credit me in the source somewhere, or in one of the MUD credits.
Quote:
Being able to say "Give me all swords that have more than 10 damage" and having it give you a result set seems incredibly cool.
I agree, however what I am attempting to do with STL is achieve the same thing, but perhaps faster. I have yet to work out the details, but something like a find_if or copy_if would effectively do the same thing.
(It would actually be remove_copy_if, which I had an example of).
Say, you wanted such a list, you might make a function object that tested for it - hopefully in a general way where you specify what you want to test for - and then iterate it over the list. Something like this, although I am just making it up, the syntax may not be 100% ...
list<object> templist;
remove_copy_if (objects.begin (), objects.end (),
back_inserter (templist),
not1 (
test_for_object ("sword", 10) ()
)
);
Now templist is your "result set" which you can iterate through to list them on the screen, or whatever.
In fact, as I understand things like output iterators better, you could probably make an output iterator (which you can send things to), which is the tcp/ip connection for the player, so you could directly output to the player in a single line of code. eg.
remove_copy_if (objects.begin (), objects.end (),
player_iterator (someplayer),
not1 (
test_for_object ("sword", 10) ()
)
);
As for MFC, I have been using it for a while now, and MS pushes it somewhat with a lot better help files than STL, however it is, of course, MS-specific, whereas STL which does the same thing in a lot of ways, is more generic.
|
- Nick Gammon
www.gammon.com.au, www.mushclient.com | | Top |
|
| Posted by
| David Haley
USA (3,881 posts) Bio
|
| Date
| Reply #23 on Fri 11 Jul 2003 04:22 AM (UTC) |
| Message
| Well, the problem with using iterators and a remove_if type of routine is that I need to either make a very customizable search function, or foresee every search that someone would need to make. Maybe someone wants to find all swords that are long swords... or all long swords that do exactly 5 damage, and are colored "blue".
In a database environment, that would be pretty easy. Users could probably even enter the query string directly (security issues, yes, I'd have to think about those.) But hardcoding the searches seems like it would be a lot of work.
Also, the robust filing system is the main advantage I see in saving things in XML format. It makes it so much easier to read, and possibly even read by an area editing program, that doesn't need to be able to decipher numbers - it can just read the XML attributes.
Out of curiosity, are you getting all this information on STL from a book? If so, which one? It sounds like something I'd want to buy. You're right, the STL documentation is pretty sparse, and most of the time rather cryptic.
How fast is it to iterate through a list of, say, two thousand objects? I know STL is fast, but still, if you loop through all that often enough, it has to have some kind of noticeable slow-down effect on the MUD.
Thanks for the code. I'll be putting you into help credits as soon as I put the functions into my source code. |
David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone
http://david.the-haleys.org | | Top |
|
| Posted by
| Nick Gammon
Australia (23,173 posts) Bio
Forum Administrator |
| Date
| Reply #24 on Fri 11 Jul 2003 08:57 AM (UTC) |
| Message
| I love questions like this because it appeals to my desire to time things and know how fast they are. :)
I modified my test XML parser slightly to time iterating through all the elements in a test XML file using the NT high-resolution timer on a fairly slow PC (166 MHz).
This is a nested iteration (like XML things are nested) and also includes iterating through the attributes per element.
In fact, this is the code:
void ShowAttribute (const pair<const string, const CAttribute const *>& attribute)
{
iCounter++;
} // end of ShowAttribute
void ShowElement (const CXMLelement const * element)
{
iCounter++;
// show all attributes for this element
for_each (element->m_AttributeMap.begin (),
element->m_AttributeMap.end (),
ShowAttribute);
// now recurse to show children
for_each (element->m_ChildrenList.begin (),
element->m_ChildrenList.end (),
ShowElement);
} // end of ShowNode
... later on ...
{
timer t ("iterate through all elements");
for_each (tree.m_xmlRoot.m_ChildrenList.begin (),
tree.m_xmlRoot.m_ChildrenList.end (),
ShowElement);
cout << iCounter << " elements" << endl;
}
My results were:
689 elements
Time taken: iterate through all elements = 0.002567 seconds.
This is less than 2000, but you can see that you would do 2000 in, say, .008 seconds. Empirical evidence is that you would do better (of course) on a more modern PC (say 500 MHz) and under Linux.
As for generalised tests, short of letting the players enter SQL, you would have to solve a similar problem. Take their request and generate SQL, or in the case of STL you could build up a "test list" that the function object could use for a general compare function.
eg. each iteration could walk a short list (test X for Y, and then test A for B, and so on), and return true or false.
|
- Nick Gammon
www.gammon.com.au, www.mushclient.com | | Top |
|
| Posted by
| Nick Gammon
Australia (23,173 posts) Bio
Forum Administrator |
| Date
| Reply #25 on Fri 11 Jul 2003 09:30 AM (UTC) |
| Message
| The best book so far to learn STL from, with heaps of examples and discussion is:
The C++ Standard Library - a tutorial and reference
Nicolai M. Josuttis
ISBN 0-201-37926-0
799 pages.
Also interesting but a bit heavier going is:
STL Tutorial and Reference Guide
David R. Musser
Atul Saini
ISBN 0-201-63398-1
400 pages.
Also try:
Effective STL
Scott Meyers
ISBN 0-201-74962-9
260 pages.
|
- Nick Gammon
www.gammon.com.au, www.mushclient.com | | Top |
|
The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).
To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.
84,958 views.
This is page 2, subject is 2 pages long:
1
2
It is now over 60 days since the last post. This thread is closed.
Refresh page
top