Monday, January 26, 2009

Programming Techniques for Memory Management (Part 2) by Roger Donnay

This section written by Roger Donnay is the second part of CA-Clipper 5.x Memory Management by Roger Donnay & Jud Cole.

The Affect of Databases on Memory Usage

Many programmers have taken an affection to "data-driven" programming to speed up the development of custom applications. Since CA-Clipper evolved as an X-Base type language, the common approach to data-driven programming is to create an "engine" or "kernel" .EXE program that retrieves "custom" information about the application from a set of database files. The more sophisticated the system, the more data files are required for configuring the application. It is important to understand the impact of databases on memory usage when designing applications that use many databases.

When a database is opened with the USE command, all the field names are placed into the public symbol table. This allows database field names to be used in expressions in the same manner as memvars. Because this symbol table is PUBLIC, the field names will remain in the symbol table, even after the database is closed. CA-Clipper employs no mechanism to remove symbols from the symbol table, only to add them. As each database is opened the symbols are added, thereby reducing the remaining available root memory ( MEMORY(0) ). It is conceivable that an application could run out of memory if many databases are opened and closed. Fortunately, symbols of the same name are "reused" in the symbol table, so if you open and close the same database many times, the symbol memory space will not be reduced. Keeping this in mind, it is a good practice to use databases with fields of the same name.

To improve speed operation, some programmers will open data-dictionary databases at the start-up of the program, load the custom program configuration into arrays, then close the databases. This action will reduce the amount of memory needed for data structures and file buffers and lower the number of DOS handles required, but the symbols will still be added to the symbol table even though they may never be accessed by the program. A data-driven application in which the databases are used only during the start-up of the application could be re-designed to convert the database information to a text file to generate a "runtime" version of the application that will load the array(s) from a text file rather than databases, thereby eliminating the symbol-table problem.

Declare FIELDS in your Compiled Codes

To prevent the need to add Field Names to the symbol table at runtime, it is always a good idea to declare fields to the Compiler by using the FIELD statement in your source code. This will insure that the symbol table memory is allocated at the time the application is linked into an executable program rather than during the running of the program.

Managing Code Segment Size ( C/ASM)

If you are writing code in C or Assembly, then the linker's overlay manager must bring an entire code segment into memory at one time. If you have many large code segments, then the size of the overlay pool may need to be increased. Normally, the requirements of the application dictate the size of the code segments, but it should be noted that a little extra time in optimizing the size of your code will usually pay off if you have a large library of routines.

CA-Clipper-compiled P-Code is loaded in fixed-size pages so this is not a requirement when programming in CA-Clipper.

Reducing RDD Memory

Try to use as few database drivers in your application as is necessary. For example, if you choose to use the DBFCDX driver for compatability with with FoxPro files, then make sure that your application does not link in the DBFNTX (default) driver unless you must have .DBT or .NTX compatability. This is accomplished by modifying the source code in the RDDSYS.PRG that is included with CA-Clipper.

Using the "Garbage Collector"

A variety of memory problems can be caused by memory "fragmentation". Fragmentation can be caused when symbols are added to the symbol table or other "locked" memory is allocated before other "unlocked" memory segments can be released. Database languages require that the symbol table be non-contiguous and not pre-allocated but instead must be allowed to "grow" during the application. This "late binding" is the essence and power behind data-driven engines (like many Clipper applications), unfortunately this kind of flexibility can cause symbols and work-area structures to be added haphazardly thruout the conventional memory space. MEMORY(0) can report that there is plenty of memory available but after the program runs for while it exists in many small segments rather that the single large segment that was available at the start of the application.

When a large segment of memory is required by the application and is not available, Clipper has to try to move around segments that are not "locked" so it can get sufficient memory for the routine that's being called. This is "garbage collection". Unfortunately, symbol memory and other memory allocation are "locked" and cannot be moved. Applications should take into consideration reducing the amount of fixed heap, stack, and VMM memory that gets allocated during the running of the program, and also to insure that garbage collection is forced on a methodical basis to prevent the fragmention.

CA-Clipper uses a technique called "scavenging" in its memory garbage collector. The garbage collector is automatically invoked during writes to the screen and "wait" states while waiting for keyboard input. Programs that have routines which have few screen-write or input routines can fragment memory badly particularly if a lot of file opening and closing, reindexing, appending, etc. is going on with little operator interaction. It is recommended that you put calls to Devout("") in your code to invoke the garbage collector in the event that you experience "Conventional Memory Exhausted", "Memory Low", or "Stack Eval" errors. My own experience with this method has not always given me desirable results, however, and I have found that placing calls to MEMORY(-1) in strategic locations in my code, such as just before opening databases, or creating large arrays actually produces better results. This method is not supported by CA or many other developers so use this recommendation with caution. Another popular method of perfoming garbage collection is to use the FT_IDLE() function from the public domain NANFORUM TOOLKIT available on many BBS's including the CLIPPER forum of COMPUSERVE. Garbage collection should be forced especially before opening databases and indexes.

Managing DGROUP Memory

There is an area of memory in CA-Clipper, as in all large model languages, referred to as DGROUP. This 64k region of low (root) memory is also called the DS (Data Segment) because it is pointed to by the DS and SS registers. Data is often stored in this area because it can be accessed faster than data stored elsewhere. DGROUP is essentially a NEAR CALL area because accessing information in this area is accomplished by a one-word address rather than the more common 2-word addressing required to access other data areas.

Assembly-language programmers like to store data in the DGROUP area to improve performance of their application. Data in this area doesn't get overlayed and it is insured that it will always be accessible regardless of the state of the application. Interrupt handlers always store data in DGROUP to insure that the data will be accessible in the event of an interrupt. CA-Clipper stores it's stacks and static values in DGROUP.

DGROUP is structured like this.

direction of growth | --> <-- |

+--------------+-----------+------------+---------+----------------+ | C/ASM static | cpu stack | eval stack | "space" | Clipper static | +--------------+-----------+------------+---------+----------------+

The eval stack is where local and private variable VALUEs are stored. The CA-Clipper static area is where CA-Clipper static VALUES are stored. ITEMs generated by programs using ITEM.API also create VALUE entries in the CA-Clipper static area. Each VALUE entry uses 14 bytes.

If the eval stack grows into the CA-Clipper static area (or vice versa), you get a UE 667 (stack fault) error.

In low memory situations, the VMM will allocate "space" to the conventional memory pool. After this happens, if the eval stack grows into the allocated space, you get a UE 668 error. Likewise, if the CA-Clipper static area needs to grow, you get a UE 669 error.

When the eval stack or CA-Clipper static area expands, and later retracts, the system maintains "watermarks" to indicate the farthest expansion of them. The VMM allocates only the space between the "watermarks". So one way to control things is to make sure the "watermarks" are set to allow your program to execute normally.

There is very little that a CA-Clipper programmer can do to resolve DGROUP problems other than to avoid using third-party products and/or C/ASM code that uses SMALL MODEL rather than LARGE MODEL programming techniques and replace LOCAL variables with LOCAL arrays. CA-Clipper is a LARGE MODEL programming language and it is recommended in the CA-Clipper API that extensions to the language should also be compiled as LARGE MODEL. Unfortunately, many C/ASM programmers develop libraries designed for speed performance rather than memory performance. I refer to these products as "DGROUP Hogs". These libraries will consume so much of the C/ASM static space that there will be literally nothing left to run the application. If you find that you cannot limit the usage of third-party libraries, then and alternative solution is to use one or more of the methods described below under "Managing the EVAL Stack".

There are several methods to determine how much DGROUP a library uses. See next page:

Monitor memory with //Info

When you start your CA-Clipper application with the //INFO option, CA-Clipper reports the condition of memory at the start of the application. The reports looks like this:

DS=4F4E:0000 DS avail=30KB OS avail=255KB EMM avail=960KB

DS= is the starting address of the DGROUP segment.

DS avail=KB reports the amount of DGROUP available.

OS avail=KB reports the amount of conventional memory available.

EMM avail=KB report the amount of expanded memory allocated to the current application.

When the DS avail (DGROUP) is less than 15k at the start-up of an application, some large applications could experience stack eval errors at runtime. This number is arbitrary and your application may actually run just fine with less available DGROUP, however it is important that you monitor DS especially if your application uses third-party libraries or lots of LOCAL and STATIC memvars. It's always a good idea to make note of the amount of DS avail in your current application before and after you add new functions or libraries to the application. If you suddenly notice a dramatic decrease in DS avail after adding new code, then you should take note that you may be using a function from a library that uses excessive DGROUP.

Create a .MAP file

Rtlink and Blinker support a MAP= command that creates a map file with information about segment usage. This map file will show you which routines use DGROUP and exactly how much they consume.

Managing the EVAL Stack

If your application uses up a lot of DGROUP memory, then you may be required to re-visit your code and either reduce the hit on the EVAL stack or expand the size of the EVAL stack. In my opinion, it is quite easy to resolve these problems with minor changes to source code.


Every time a new function is called, all LOCAL variables are "pushed" onto the EVAL stack. They are "popped" off the stack when returning to the calling program. If a procedure or function is called recursively, then it can be quite easy to blow up the eval stack with a 667 Stack Eval Error. For example, I had this problem in one of my larger applications due to the fact that programs were deeply nested via calls to a common set of menu functions. I found that my main menu function could blow up the stack if called recursively as few as 5 times. Each time this function was called, 85 LOCAL variables were pushed on to the eval stack. I completely eliminated the problem with about 2 hours of programming and debugging by replacing the 85 separate LOCAL variables with 1 LOCAL array. A LOCAL array uses the same amount of stack space as any other LOCAL variable, therefore I reduced the amount of values pushed on the stack from 85 to 1. After doing this, I could not blow up the stack even if I called the menu system recursively more than 30 times. This was accomplished quite easily with very minor code changes. Here is an example of the BEFORE and AFTER code.



LOCAL cColor, cMenuScrn, nInkey, cTempStr, nTempNum, nItemLen,; cOldColor, nItems, nMsgLen, nHotStart, cBoxColor, ; cMenuColor, cHotColor, cSelColor, cBuffer, nRow, nCol, ; lMouseHit, nSaveRow, nSaveCol, lMessage, cGrayColor, ; nMouseStat, nStRow, nStCol, nEnRow, nEnCol, lMouVisible, ; nPass, aSubMenu, aSubBlock, nElement, aMenuItems, ; aHotKeys, aMenuBlocks, aSubItems, aBlockItems, cType, ; nStart, cTitle, lBar, lReturnVal, lShadow, aMouseKeys, ....



LOCAL aMenu := Array(85)

#define cColor aMenu[1] ; #define cMenuScrn aMenu[2]

#define nInkey aMenu[3] ; #define cTempStr aMenu[4]

#define nTempNum aMenu[5] ; #define nItemLen aMenu[6]

#define cOldColor aMenu[7] ; #define nItems aMenu[8]

#define nMsgLen aMenu[9] ; #define nItems aMenu[10]

#define nHotStart aMenu[11] ; #define cBoxColor aMenu[12]

#define cMenuColor aMenu[13] ; #define cHotColor aMenu[14]

#define cSelColor aMenu[15] ; #define cBuffer aMenu[16]

#define nRow aMenu[17] ; #define nCol aMenu[18]

#define lMouseHit aMenu[19] ; #define nSaveRow aMenu[20]

#define nSaveCol aMenu[21] ; #define lMessage aMenu[22]

#define cGrayColor aMenu[23] ; #define nMouseStat aMenu[24]

#define nStRow aMenu[25] ; #define nStCol aMenu[26]

#define nEnRow aMenu[27] ; #define nEnCol aMenu[28]

#define lMouVisible aMenu[29] ; #define nPass aMenu[30]

#define aSubMenu aMenu[31] ; #define aSubBlock aMenu[32]

#define nElement aMenu[33] ; #define aMenuItems aMenu[34]

#define aHotKeys aMenu[35] ; #define aMenuBlocks aMenu[36]

#define aSubItems aMenu[37] ; #define aBlockItems aMenu[38]

#define cType aMenu[39] ; #

No comments:

Post a Comment

Welcome to Clipper... Clipper... Clipper

In 1997, then using Delphi 3, I had already created 32-bits Windows applications for HRIS, ERP and CRM. In 2007, using Ruby on Rails, an AJAX powered CRM site running on Apache & MySQL was created and I am now using Visual Studio .Net 2008 to create web-based projects and Delphi 7 for Win32 applications using SQL2005 & DBFCDX.

So, why then am I reviving the Original Clipper... Clipper... Clipper via a Blog as CA-Clipper is a programming language for the DOS world ? Believe it or not, there are still some clients using my mission-critical CA-Clipper applications for DOS installed in the late 80's and up to the mid 90's. This is testimony to CA-Clipper's robustness as a language :-)

With the widespread introduction of Windows 7 64-bits as the standard O/S for new Windows based PCs & Notebooks, CA-Clipper EXE simply will not work and it has become imperative for Clipper programmers to migrate immediately to Harbour to build 32/64 bits EXEs

Since 28th January 2009, this blog has been read by 134,389 (10/3/11 - 39,277) unique visitors (of which 45,151 (10/3/11 - 13,929) are returning visitors) from 103 countries and 1,574 cities & towns in Europe (37; 764 cities), North America (3; 373 cities) , Central America & Caribeans (6; 13 cities), South America(10; 226 cities), Africa & Middle-East (12; 44 cities) , Asia-Pacific (21; 175 cities). So, obviously Clipper is Alive & Well : -)

TIA & Enjoy ! (10th October 2012, 11:05; 13th November 2015)

Original Welcome Page for Clipper... Clipper... Clipper

This is the original Welcome Page for Clipper... Clipper... Clipper, which I am republishing for historical and sentimental reasons. The only changes that I have made was to fix all the broken links. BTW, the counter from is still working :-)

Welcome to Chee Chong Hwa's Malaysian WWW web site which is dedicated to Clipperheads throughout the world.

This site started out as a teeny-weeny section of Who the heck is Chee Chong Hwa ? and has graduated into a full blown web site of more than 140 pages (actually hundreds of A4 size pages) ! This is due to its growing popularity and tremendous encouragements from visiting Clipperheads from 100 countries worldwide, from North America, Central America, Caribbean, South America, Europe, Middle-East, Africa and Asia-Pacific. Thanx Clipperheads, you all made this happen !

What is Clipper ?

You may ask, what is this Clipper stuff ? Could Clipper be something to do with sailing as it is the name of a very fast sailing American ship in the 19th century ?

Well, Clipper or to be precise, CA-Clipper is the premier PC-Software development tool for DOS. It was first developed by Nantucket Corporation initially as a compiler for dBase3+ programs. Since then, CA-Clipper has evolved away from its x-base roots with the introduction of lexical scoping & pre-defined objects like TBrowse. As at today, the most stable version ofClipper is 5.2e while the latest version, 5.3a was introduced on 21 May 1996.

As at 11th November, 1996, an unofficial 5.3a fixes file was made available by Jo French. See the About CA-Clipper 5.3a section for more details. BTW, Jo French uploaded the revised 5.3a fixes file on 20th November, 1996.

Latest News

The latest news is that CA has finally released the long-awaited 5.3b patch on 21 May, 1997.

For 5.3b users, you must a take a look at Jo French's comments on unfixed bugs in 5.3b.

BTW, have you used Click ? If you're a serious Clipperprogrammer and need an excellent code formatter, Click is a natural choice. How to get it ? Simple, access Phil Barnett's site via my Cool Clipper Sites.

32-bits Clipper for Windows ?

Have you tried Xbase ++ ? Well, I have and compared to Delphi (my current Windows programming tool of choice), I'm still sticking to Delphi.

Anyway, you should visit the Alaska Home Page. Give it a chance and then draw your own conclusions !.

The Harbour Project

Is this the future of Xbase ? Take a look at at the Harbour Project

You are Visitor # ...

According to, you are visitor since 3 June 1996.

If you like or dislike what you see on this website, please drop me a line by clicking the email button at the bottom of this page or better still, by filling out the form in my guest book. If you are not sure what to write,click here to take a look at what other Clipperheads have to say.