Programming Techniques for Memory Management (Part 2) by Roger Donnay

This section written by Roger Donnay is the second part of CA-Clipper 5.x Memory Management by Roger Donnay & Jud Cole.

---

The Affect of Databases on Memory Usage

Many programmers have taken an affection to "data-driven" programming to speed up the development of custom applications. Since CA-Clipper evolved as an X-Base type language, the common approach to data-driven programming is to create an "engine" or "kernel" .EXE program that retrieves "custom" information about the application from a set of database files. The more sophisticated the system, the more data files are required for configuring the application. It is important to understand the impact of databases on memory usage when designing applications that use many databases.

When a database is opened with the USE command, all the field names are placed into the public symbol table. This allows database field names to be used in expressions in the same manner as memvars. Because this symbol table is PUBLIC, the field names will remain in the symbol table, even after the database is closed. CA-Clipper employs no mechanism to remove symbols from the symbol table, only to add them. As each database is opened the symbols are added, thereby reducing the remaining available root memory ( MEMORY(0) ). It is conceivable that an application could run out of memory if many databases are opened and closed. Fortunately, symbols of the same name are "reused" in the symbol table, so if you open and close the same database many times, the symbol memory space will not be reduced. Keeping this in mind, it is a good practice to use databases with fields of the same name.

To improve speed operation, some programmers will open data-dictionary databases at the start-up of the program, load the custom program configuration into arrays, then close the databases. This action will reduce the amount of memory needed for data structures and file buffers and lower the number of DOS handles required, but the symbols will still be added to the symbol table even though they may never be accessed by the program. A data-driven application in which the databases are used only during the start-up of the application could be re-designed to convert the database information to a text file to generate a "runtime" version of the application that will load the array(s) from a text file rather than databases, thereby eliminating the symbol-table problem.

Declare FIELDS in your Compiled Codes

To prevent the need to add Field Names to the symbol table at runtime, it is always a good idea to declare fields to the Compiler by using the FIELD statement in your source code. This will insure that the symbol table memory is allocated at the time the application is linked into an executable program rather than during the running of the program.

Managing Code Segment Size ( C/ASM)

If you are writing code in C or Assembly, then the linker's overlay manager must bring an entire code segment into memory at one time. If you have many large code segments, then the size of the overlay pool may need to be increased. Normally, the requirements of the application dictate the size of the code segments, but it should be noted that a little extra time in optimizing the size of your code will usually pay off if you have a large library of routines.

CA-Clipper-compiled P-Code is loaded in fixed-size pages so this is not a requirement when programming in CA-Clipper.

Reducing RDD Memory

Try to use as few database drivers in your application as is necessary. For example, if you choose to use the DBFCDX driver for compatability with with FoxPro files, then make sure that your application does not link in the DBFNTX (default) driver unless you must have .DBT or .NTX compatability. This is accomplished by modifying the source code in the RDDSYS.PRG that is included with CA-Clipper.

Using the "Garbage Collector"

A variety of memory problems can be caused by memory "fragmentation". Fragmentation can be caused when symbols are added to the symbol table or other "locked" memory is allocated before other "unlocked" memory segments can be released. Database languages require that the symbol table be non-contiguous and not pre-allocated but instead must be allowed to "grow" during the application. This "late binding" is the essence and power behind data-driven engines (like many Clipper applications), unfortunately this kind of flexibility can cause symbols and work-area structures to be added haphazardly thruout the conventional memory space. MEMORY(0) can report that there is plenty of memory available but after the program runs for while it exists in many small segments rather that the single large segment that was available at the start of the application.

When a large segment of memory is required by the application and is not available, Clipper has to try to move around segments that are not "locked" so it can get sufficient memory for the routine that's being called. This is "garbage collection". Unfortunately, symbol memory and other memory allocation are "locked" and cannot be moved. Applications should take into consideration reducing the amount of fixed heap, stack, and VMM memory that gets allocated during the running of the program, and also to insure that garbage collection is forced on a methodical basis to prevent the fragmention.

CA-Clipper uses a technique called "scavenging" in its memory garbage collector. The garbage collector is automatically invoked during writes to the screen and "wait" states while waiting for keyboard input. Programs that have routines which have few screen-write or input routines can fragment memory badly particularly if a lot of file opening and closing, reindexing, appending, etc. is going on with little operator interaction. It is recommended that you put calls to Devout("") in your code to invoke the garbage collector in the event that you experience "Conventional Memory Exhausted", "Memory Low", or "Stack Eval" errors. My own experience with this method has not always given me desirable results, however, and I have found that placing calls to MEMORY(-1) in strategic locations in my code, such as just before opening databases, or creating large arrays actually produces better results. This method is not supported by CA or many other developers so use this recommendation with caution. Another popular method of perfoming garbage collection is to use the FT_IDLE() function from the public domain NANFORUM TOOLKIT available on many BBS's including the CLIPPER forum of COMPUSERVE. Garbage collection should be forced especially before opening databases and indexes.

Managing DGROUP Memory

There is an area of memory in CA-Clipper, as in all large model languages, referred to as DGROUP. This 64k region of low (root) memory is also called the DS (Data Segment) because it is pointed to by the DS and SS registers. Data is often stored in this area because it can be accessed faster than data stored elsewhere. DGROUP is essentially a NEAR CALL area because accessing information in this area is accomplished by a one-word address rather than the more common 2-word addressing required to access other data areas.

Assembly-language programmers like to store data in the DGROUP area to improve performance of their application. Data in this area doesn't get overlayed and it is insured that it will always be accessible regardless of the state of the application. Interrupt handlers always store data in DGROUP to insure that the data will be accessible in the event of an interrupt. CA-Clipper stores it's stacks and static values in DGROUP.

DGROUP is structured like this.

direction of growth | --> <-- |

+--------------+-----------+------------+---------+----------------+ | C/ASM static | cpu stack | eval stack | "space" | Clipper static | +--------------+-----------+------------+---------+----------------+

The eval stack is where local and private variable VALUEs are stored. The CA-Clipper static area is where CA-Clipper static VALUES are stored. ITEMs generated by programs using ITEM.API also create VALUE entries in the CA-Clipper static area. Each VALUE entry uses 14 bytes.

If the eval stack grows into the CA-Clipper static area (or vice versa), you get a UE 667 (stack fault) error.

In low memory situations, the VMM will allocate "space" to the conventional memory pool. After this happens, if the eval stack grows into the allocated space, you get a UE 668 error. Likewise, if the CA-Clipper static area needs to grow, you get a UE 669 error.

When the eval stack or CA-Clipper static area expands, and later retracts, the system maintains "watermarks" to indicate the farthest expansion of them. The VMM allocates only the space between the "watermarks". So one way to control things is to make sure the "watermarks" are set to allow your program to execute normally.

There is very little that a CA-Clipper programmer can do to resolve DGROUP problems other than to avoid using third-party products and/or C/ASM code that uses SMALL MODEL rather than LARGE MODEL programming techniques and replace LOCAL variables with LOCAL arrays. CA-Clipper is a LARGE MODEL programming language and it is recommended in the CA-Clipper API that extensions to the language should also be compiled as LARGE MODEL. Unfortunately, many C/ASM programmers develop libraries designed for speed performance rather than memory performance. I refer to these products as "DGROUP Hogs". These libraries will consume so much of the C/ASM static space that there will be literally nothing left to run the application. If you find that you cannot limit the usage of third-party libraries, then and alternative solution is to use one or more of the methods described below under "Managing the EVAL Stack".

There are several methods to determine how much DGROUP a library uses. See next page:

Monitor memory with //Info

When you start your CA-Clipper application with the //INFO option, CA-Clipper reports the condition of memory at the start of the application. The reports looks like this:

DS=4F4E:0000 DS avail=30KB OS avail=255KB EMM avail=960KB

DS= is the starting address of the DGROUP segment.

DS avail=KB reports the amount of DGROUP available.

OS avail=KB reports the amount of conventional memory available.

EMM avail=KB report the amount of expanded memory allocated to the current application.

When the DS avail (DGROUP) is less than 15k at the start-up of an application, some large applications could experience stack eval errors at runtime. This number is arbitrary and your application may actually run just fine with less available DGROUP, however it is important that you monitor DS especially if your application uses third-party libraries or lots of LOCAL and STATIC memvars. It's always a good idea to make note of the amount of DS avail in your current application before and after you add new functions or libraries to the application. If you suddenly notice a dramatic decrease in DS avail after adding new code, then you should take note that you may be using a function from a library that uses excessive DGROUP.

Create a .MAP file

Rtlink and Blinker support a MAP= command that creates a map file with information about segment usage. This map file will show you which routines use DGROUP and exactly how much they consume.

Managing the EVAL Stack

If your application uses up a lot of DGROUP memory, then you may be required to re-visit your code and either reduce the hit on the EVAL stack or expand the size of the EVAL stack. In my opinion, it is quite easy to resolve these problems with minor changes to source code.

Use LOCAL ARRAYS instead of LOCAL VARIABLES

Every time a new function is called, all LOCAL variables are "pushed" onto the EVAL stack. They are "popped" off the stack when returning to the calling program. If a procedure or function is called recursively, then it can be quite easy to blow up the eval stack with a 667 Stack Eval Error. For example, I had this problem in one of my larger applications due to the fact that programs were deeply nested via calls to a common set of menu functions. I found that my main menu function could blow up the stack if called recursively as few as 5 times. Each time this function was called, 85 LOCAL variables were pushed on to the eval stack. I completely eliminated the problem with about 2 hours of programming and debugging by replacing the 85 separate LOCAL variables with 1 LOCAL array. A LOCAL array uses the same amount of stack space as any other LOCAL variable, therefore I reduced the amount of values pushed on the stack from 85 to 1. After doing this, I could not blow up the stack even if I called the menu system recursively more than 30 times. This was accomplished quite easily with very minor code changes. Here is an example of the BEFORE and AFTER code.

BEFORE:

FUNCTION DC_MenuMain( )

LOCAL cColor, cMenuScrn, nInkey, cTempStr, nTempNum, nItemLen,; cOldColor, nItems, nMsgLen, nHotStart, cBoxColor, ; cMenuColor, cHotColor, cSelColor, cBuffer, nRow, nCol, ; lMouseHit, nSaveRow, nSaveCol, lMessage, cGrayColor, ; nMouseStat, nStRow, nStCol, nEnRow, nEnCol, lMouVisible, ; nPass, aSubMenu, aSubBlock, nElement, aMenuItems, ; aHotKeys, aMenuBlocks, aSubItems, aBlockItems, cType, ; nStart, cTitle, lBar, lReturnVal, lShadow, aMouseKeys, ....

AFTER:

FUNCTION DC_MenuMain( )

LOCAL aMenu := Array(85)

#define cColor aMenu[1] ; #define cMenuScrn aMenu[2]

#define nInkey aMenu[3] ; #define cTempStr aMenu[4]

#define nTempNum aMenu[5] ; #define nItemLen aMenu[6]

#define cOldColor aMenu[7] ; #define nItems aMenu[8]

#define nMsgLen aMenu[9] ; #define nItems aMenu[10]

#define nHotStart aMenu[11] ; #define cBoxColor aMenu[12]

#define cMenuColor aMenu[13] ; #define cHotColor aMenu[14]

#define cSelColor aMenu[15] ; #define cBuffer aMenu[16]

#define nRow aMenu[17] ; #define nCol aMenu[18]

#define lMouseHit aMenu[19] ; #define nSaveRow aMenu[20]

#define nSaveCol aMenu[21] ; #define lMessage aMenu[22]

#define cGrayColor aMenu[23] ; #define nMouseStat aMenu[24]

#define nStRow aMenu[25] ; #define nStCol aMenu[26]

#define nEnRow aMenu[27] ; #define nEnCol aMenu[28]

#define lMouVisible aMenu[29] ; #define nPass aMenu[30]

#define aSubMenu aMenu[31] ; #define aSubBlock aMenu[32]

#define nElement aMenu[33] ; #define aMenuItems aMenu[34]

#define aHotKeys aMenu[35] ; #define aMenuBlocks aMenu[36]

#define aSubItems aMenu[37] ; #define aBlockItems aMenu[38]

#define cType aMenu[39] ; #