Spent last 2 weeks fighting with the perforce eclipse plug-in (war is not over, I even submitted an enhancement request to support .p4ignore). I wish I knew how poor perforce works with eclipse, I would have fought again it. As a source control it works fine as long as you use their GUI tool, but subversion it is not.
I have used a lot of source control software in my life: CVS, subversion, ClearCase, SourceSafe, Mercurial, Git, perforce and few others for short time.
My favorite has to be subversion. In 7 years of using subversion for my personal development, I have never had any problems. It works from command line, with windows explorer, with eclipse, with visual studio... it just works as you expect it, no fuss, no config woes. Just point it to a URL and you are done.
I want to preface this by saying that eclipse is an excellent IDE, the eclipse plug-in which is not associated with eclipse at all, and is written by perforce team.
Some things with perforce that I find unacceptable for commercial software:
- No ignore file (this is the first source control software that I found that doesn't have ignore), there are times you need to mark a local file so that it is not added to source control (mostly config files that have been customized for the local environment). Eclipse plug-in allows you to create .p4ignore but their GUI tool does not support it.
- Eclipse integration is shameful. There seems to be non-transactional nature to the plug-in, so when a problem is encountered you never know what state the file is in and even worse some problems fail silently. Login issue is the main cause, after the login expires eclipse plug-in starts silently failing unless you restart it; so now I close my eclipse before I go home and open it again when I get into work (annoying).
- Weird states created by eclipse plug-in, if the file is in read-write state due to an error, you are now in an odd state when you can't merge with latest, can't checkout and can only revert to the latest (while backing up your file) then using BeyondCompare (which is an awesome tool) to merge your changes in.
- Use of RO flag for management of check-ins, it feels like its 1995 and SourceSafe again, this flag can get clobbered by some editors or eclipse plug-in when checkout fails.
Overall I would recommend against using eclipse and perforce together if you are used to the seamless integration with CVS or subversion.
Sunday, May 24, 2009
Wednesday, March 11, 2009
Version 1.7.0.0 ready for prime time!
Lots of evening and weekends spent on getting more stuff in. Most of the changes are internal but I did add a nice visual admin site viewer using jQuery plugins.
After battling windows service code I finally got it working well enough that I feel comfortable using it instead of the console mode. It did take a while to figure out all the little kinks that arise when writing services and having to deal with Microsoft's security system (it's like trying to play Spelunker, read the Gameplay section).
I also decided to organize the entire website into the _website folder which makes getting up and running trivial (not that it was hard before, just now everything is under one folder and you can switch website roots easily which not having to copy config or resources). Something that people that run multiple instances have requested.
After battling windows service code I finally got it working well enough that I feel comfortable using it instead of the console mode. It did take a while to figure out all the little kinks that arise when writing services and having to deal with Microsoft's security system (it's like trying to play Spelunker, read the Gameplay section).
I also decided to organize the entire website into the _website folder which makes getting up and running trivial (not that it was hard before, just now everything is under one folder and you can switch website roots easily which not having to copy config or resources). Something that people that run multiple instances have requested.
Monday, December 08, 2008
Onward and forward...
Progress
So I added a simple memory tracking class to the server executable (debug mode only and 32-bit at the time, it uses asm directive which is not supported in 64-bit mode so I have to come up with creative ways of accessing the 64-bit registers for the stack traces). Anyhow, it wound up being extremely useful in reporting memory leaks (since I am still working on saving up enough to buy BoundsChecker).
With OpenSSL
First one I found was in how I was cleaning up the SSL library (while most are global lifetime allocation and not really a big issue, they make the reports ugly with 600+ unallocated objects). I found various API calls to explicitly free up memory on shutdown and brought it down to only 2 leaks:
From ASocketLibrary_SSL.cpp:
ASocketLibrary_SSL::ASocketLibrary_SSL()
{
SSL_library_init();
ERR_load_crypto_strings();
SSL_load_error_strings();
}
ASocketLibrary_SSL::~ASocketLibrary_SSL()
{
ERR_remove_state(0);
ENGINE_cleanup();
CONF_modules_unload(1);
ERR_free_strings();
EVP_cleanup();
CRYPTO_cleanup_all_ex_data();
}
With GdLib
This one was actually quite serious in applications that generate dynamic images. While this is a good example where C++ destructor would have been perfect, the gdlib API is in C and thus you have o find appropriate functions to call:
From AGdCanvas.cpp:
AGdCanvas::AGdCanvas(int sx, int sy)
{
m_GdImagePtr = gdImageCreateTrueColor(sx, sy);
}
AGdCanvas::~AGdCanvas()
{
gdImageDestroy(m_GdImagePtr);
}
I was using gdFree() which unfortunately did not release the sub-objects and thus leaks a bit of memory.
Current State
At this point there are no known leaks and I ran the server under a moderate 20 client load for an hour with no variance in memory used (~30MB in full debug build, release build memory footprint is ~14MB). I disabled all caching for the test to make sure that everything was being allocated and deallocated correctly.
Miscellaneous Notes
Next version looks like it may be done this month, if I can find enough weekend time to do more testing and validation. I may rewrite the threaded queue that handles persistent sockets that are pending data of the next request, currently it uses the synchronization model from std::list and I replaced it with my home-grown ABasePtrQueue which is faster and has built-in ability to synchronize modification if given an ASynchronization object when created.
Finally
I've been spending time reading my new Lua Reference book and planning some more sample code with it. I really like Lua as a scripting language and at this point its speed and ease of embedding is tops in my list.
Overall, I was able to get majority of the sample calls to execute in AMD for that quad-core processor and dual-core processor and motherboard; made client and server with those parts and they prodly wear the "Powered by AMD" stickers!).
So I added a simple memory tracking class to the server executable (debug mode only and 32-bit at the time, it uses asm directive which is not supported in 64-bit mode so I have to come up with creative ways of accessing the 64-bit registers for the stack traces). Anyhow, it wound up being extremely useful in reporting memory leaks (since I am still working on saving up enough to buy BoundsChecker).
With OpenSSL
First one I found was in how I was cleaning up the SSL library (while most are global lifetime allocation and not really a big issue, they make the reports ugly with 600+ unallocated objects). I found various API calls to explicitly free up memory on shutdown and brought it down to only 2 leaks:
From ASocketLibrary_SSL.cpp:
ASocketLibrary_SSL::ASocketLibrary_SSL()
{
SSL_library_init();
ERR_load_crypto_strings();
SSL_load_error_strings();
}
ASocketLibrary_SSL::~ASocketLibrary_SSL()
{
ERR_remove_state(0);
ENGINE_cleanup();
CONF_modules_unload(1);
ERR_free_strings();
EVP_cleanup();
CRYPTO_cleanup_all_ex_data();
}
With GdLib
This one was actually quite serious in applications that generate dynamic images. While this is a good example where C++ destructor would have been perfect, the gdlib API is in C and thus you have o find appropriate functions to call:
From AGdCanvas.cpp:
AGdCanvas::AGdCanvas(int sx, int sy)
{
m_GdImagePtr = gdImageCreateTrueColor(sx, sy);
}
AGdCanvas::~AGdCanvas()
{
gdImageDestroy(m_GdImagePtr);
}
I was using gdFree() which unfortunately did not release the sub-objects and thus leaks a bit of memory.
Current State
At this point there are no known leaks and I ran the server under a moderate 20 client load for an hour with no variance in memory used (~30MB in full debug build, release build memory footprint is ~14MB). I disabled all caching for the test to make sure that everything was being allocated and deallocated correctly.
Miscellaneous Notes
Next version looks like it may be done this month, if I can find enough weekend time to do more testing and validation. I may rewrite the threaded queue that handles persistent sockets that are pending data of the next request, currently it uses the synchronization model from std::list and I replaced it with my home-grown ABasePtrQueue which is faster and has built-in ability to synchronize modification if given an ASynchronization object when created.
Finally
I've been spending time reading my new Lua Reference book and planning some more sample code with it. I really like Lua as a scripting language and at this point its speed and ease of embedding is tops in my list.
Overall, I was able to get majority of the sample calls to execute in AMD for that quad-core processor and dual-core processor and motherboard; made client and server with those parts and they prodly wear the "Powered by AMD" stickers!).
Wednesday, November 26, 2008
AOS Rhino 1.6.0.0 is coming soon
I have been busy extending the I18N and L10N abilities and reorganizing the web content locations to make more sense (along with a slew of bug fixes and enhancements).
Now I just have to do load testing and flush out any anomalies and release it soon. This should be a very stable release and very usable with multi-language content.
Stay tuned...
Now I just have to do load testing and flush out any anomalies and release it soon. This should be a very stable release and very usable with multi-language content.
Stay tuned...
Friday, September 26, 2008
The big and silent release
On September 24th 2008, the first official release of AOS Rhino 1.5.0.1 hit SourceForge.net and is available for download. I still want to do heavier load testing and write more samples, but might as well throw it out into the wild and see what happens.
I have been load testing, debugging, cleaning up and documenting for a few months and it's as good and complete as it can be for the first official release.
There is still a lot that I want to do, but all that is additions going forward and I promise to not change the base API for any 1.5.x.x version, 1.6.x.x is fair game!
I have been load testing, debugging, cleaning up and documenting for a few months and it's as good and complete as it can be for the first official release.
There is still a lot that I want to do, but all that is additions going forward and I promise to not change the base API for any 1.5.x.x version, 1.6.x.x is fair game!
Wednesday, June 04, 2008
64 bit in a day!
I finally sat down and decided to port my code to 64 bit. I created a new configuration for x64, added Win64 to the predefines and fired off the build. And then it dawned on me, I have 3rd party libraries linking in a few projects (python, lua, gdlib, zlib, xerces/xalan, mysql, sqlite3, odbc and openssl). So I had to make quick decisions of whether to continue or stay with 32bit for a bit longer.
lua: using the actual source to build a DLL, no problem.
sqlite3: using the actual source to build a DLL, no problem.
zlib: 64 bit library I found and linked, no problem.
mysql: had to install the 64 bit version to get include/lib, no problem.
odbc: already native, just had to make a few variable adjustments in the code.
openssl: took a bit of wrestling with the code (assembly file had a syntax error and makefile was including an incorrect static library), in the end I build the 64 bit libs and all worked out.
gdlib: this was a bit tricky, since I could not find 64 bit libs, I had to build it myself inside my code (as I did with lua and sqlite3) but it then needed libpng and libjpeg, which had to be included. After some more wrestling with the config I managed to build it successfully.
And now for the casualties:
python was on the chopping block for a while, I really like lua as an embedded language and python for scripting in the shell. So I made a decision to not have python support from within the app server.
xalan/xerces has been a bit problematic from the beginning and multi-threaded tests occasionally failed which never happened with MSXML. I could not find 64 bit libs and the build system was depended on Configure, so I decided to not include it and rely on MSXML which is way faster and more stable on Windows. When I port to linux I will probably go with libxml2 and libxsl.
After all that the server itself built without a hitch and I was able to start and test it. I surprised myself that there were no portability issues with 64bits compile.
Made me so confident that I installed VMWare player with Ubuntu on it and created a makefile that built 1 of my source files... with about 300 to go; hey it's a start. I suspect the toughest part is going to be porting the synchronization, socket and file based classes; I have been on projects where I had to port code from windows to unix, so it shouldn't be too tough. The 3rd party stuff i have so far is pretty portable, so it should not present a big problem.
My biggest dilemma is what do I call my app server?!
Do I need an animal/plant/insect/object mascot?
All the good names that represent fast and lightweight have been taken by products that are often neither...
lua: using the actual source to build a DLL, no problem.
sqlite3: using the actual source to build a DLL, no problem.
zlib: 64 bit library I found and linked, no problem.
mysql: had to install the 64 bit version to get include/lib, no problem.
odbc: already native, just had to make a few variable adjustments in the code.
openssl: took a bit of wrestling with the code (assembly file had a syntax error and makefile was including an incorrect static library), in the end I build the 64 bit libs and all worked out.
gdlib: this was a bit tricky, since I could not find 64 bit libs, I had to build it myself inside my code (as I did with lua and sqlite3) but it then needed libpng and libjpeg, which had to be included. After some more wrestling with the config I managed to build it successfully.
And now for the casualties:
python was on the chopping block for a while, I really like lua as an embedded language and python for scripting in the shell. So I made a decision to not have python support from within the app server.
xalan/xerces has been a bit problematic from the beginning and multi-threaded tests occasionally failed which never happened with MSXML. I could not find 64 bit libs and the build system was depended on Configure, so I decided to not include it and rely on MSXML which is way faster and more stable on Windows. When I port to linux I will probably go with libxml2 and libxsl.
After all that the server itself built without a hitch and I was able to start and test it. I surprised myself that there were no portability issues with 64bits compile.
Made me so confident that I installed VMWare player with Ubuntu on it and created a makefile that built 1 of my source files... with about 300 to go; hey it's a start. I suspect the toughest part is going to be porting the synchronization, socket and file based classes; I have been on projects where I had to port code from windows to unix, so it shouldn't be too tough. The 3rd party stuff i have so far is pretty portable, so it should not present a big problem.
My biggest dilemma is what do I call my app server?!
Do I need an animal/plant/insect/object mascot?
All the good names that represent fast and lightweight have been taken by products that are often neither...
I have been busy for the last few months profiling, linting and optimizing the code. After numerous runs I have convinced myself and (used lint to prove) that there were no memory leaks in the code.
But I was still not convinced, so a friend suggested I run it through his DevPartner BoundsChecker to verify. After a bit of mucking around and restoring old MSDev 2005 vcproj files (apparently BoundsChecker doesn't work with MSDev 2008 at the moment or at least his version didn't). We ran it through a kitchen-sink style test and asked server to exit (I made sure you can do that via admin site). It did so and no memory leaks reported, and they all rejoiced and ate the minstrels. I tend to be very picky with code and was pretty sure that there were no leaks but had to be sure, it's a server afterall and a leak would adversely affect uptime.
Speaking of uptime, the server has been up for over a month (it gets bounced not because there are problems but rather I keep rebuilding it with newer features). I do get a decent number of hits mostly from zombies and bots looking for the latest PHP exploit (I am not kidding, 10-30 attacks on PHP per day, everyday, on my server which obviously doesn't even run PHP). It also helps that many of the HTTP requests are malformed or very long (like that Exchange bufferoverflow one I get every so often) and it tests lots of error handling. And all those port scans. Just more fodder for testing.
But I was still not convinced, so a friend suggested I run it through his DevPartner BoundsChecker to verify. After a bit of mucking around and restoring old MSDev 2005 vcproj files (apparently BoundsChecker doesn't work with MSDev 2008 at the moment or at least his version didn't). We ran it through a kitchen-sink style test and asked server to exit (I made sure you can do that via admin site). It did so and no memory leaks reported, and they all rejoiced and ate the minstrels. I tend to be very picky with code and was pretty sure that there were no leaks but had to be sure, it's a server afterall and a leak would adversely affect uptime.
Speaking of uptime, the server has been up for over a month (it gets bounced not because there are problems but rather I keep rebuilding it with newer features). I do get a decent number of hits mostly from zombies and bots looking for the latest PHP exploit (I am not kidding, 10-30 attacks on PHP per day, everyday, on my server which obviously doesn't even run PHP). It also helps that many of the HTTP requests are malformed or very long (like that Exchange bufferoverflow one I get every so often) and it tests lots of error handling. And all those port scans. Just more fodder for testing.
Tuesday, October 16, 2007
Lua Lua, Everywhere!
Lua is now embedded in AOS, it was a lot easier than I thought. The code is ANSI C so I embedded it into a DLL with ability to dynamically load Lua libraries. I added it to the templates which now behave similar to JSP pages with Java embedded in them. For example:
header.html
<>Page for %[LUA]{{{
print(
aos.emit("/root/REQUEST_HEADER/ACookies/username")
);
}}}[LUA]%< /sometag >
Since you can combine a set of templates to make a full page, you can have a common header/footer/etc pages that are shared by many other pages. And here aos.emit() extracts from the XML model the cookie named username and
using alibrary.print() function writes it to the output stream.
It's very easy once you get used to Lua (and in spirit of C almost everything is defined via libraries); so getting used to it is actually very easy. Try it out: http://www.lua.org/ and if you are not completely satisfied, I'll refund your ... umm... it's free!
header.html
<>Page for %[LUA]{{{
print(
aos.emit("/root/REQUEST_HEADER/ACookies/username")
);
}}}[LUA]%< /sometag >
Since you can combine a set of templates to make a full page, you can have a common header/footer/etc pages that are shared by many other pages. And here aos.emit() extracts from the XML model the cookie named username and
using alibrary.print() function writes it to the output stream.
It's very easy once you get used to Lua (and in spirit of C almost everything is defined via libraries); so getting used to it is actually very easy. Try it out: http://www.lua.org/ and if you are not completely satisfied, I'll refund your ... umm... it's free!
Subscribe to:
Posts (Atom)
