Programming Techniques for Memory Management (Part 2) by Roger Donnay

This section written by Roger Donnay is the second part of CA-Clipper 5.x Memory Management by Roger Donnay & Jud Cole.

---

The Affect of Databases on Memory Usage

Many programmers have taken an affection to "data-driven" programming to speed up the development of custom applications. Since CA-Clipper evolved as an X-Base type language, the common approach to data-driven programming is to create an "engine" or "kernel" .EXE program that retrieves "custom" information about the application from a set of database files. The more sophisticated the system, the more data files are required for configuring the application. It is important to understand the impact of databases on memory usage when designing applications that use many databases.

When a database is opened with the USE command, all the field names are placed into the public symbol table. This allows database field names to be used in expressions in the same manner as memvars. Because this symbol table is PUBLIC, the field names will remain in the symbol table, even after the database is closed. CA-Clipper employs no mechanism to remove symbols from the symbol table, only to add them. As each database is opened the symbols are added, thereby reducing the remaining available root memory ( MEMORY(0) ). It is conceivable that an application could run out of memory if many databases are opened and closed. Fortunately, symbols of the same name are "reused" in the symbol table, so if you open and close the same database many times, the symbol memory space will not be reduced. Keeping this in mind, it is a good practice to use databases with fields of the same name.

To improve speed operation, some programmers will open data-dictionary databases at the start-up of the program, load the custom program configuration into arrays, then close the databases. This action will reduce the amount of memory needed for data structures and file buffers and lower the number of DOS handles required, but the symbols will still be added to the symbol table even though they may never be accessed by the program. A data-driven application in which the databases are used only during the start-up of the application could be re-designed to convert the database information to a text file to generate a "runtime" version of the application that will load the array(s) from a text file rather than databases, thereby eliminating the symbol-table problem.

Declare FIELDS in your Compiled Codes

To prevent the need to add Field Names to the symbol table at runtime, it is always a good idea to declare fields to the Compiler by using the FIELD statement in your source code. This will insure that the symbol table memory is allocated at the time the application is linked into an executable program rather than during the running of the program.

Managing Code Segment Size ( C/ASM)

If you are writing code in C or Assembly, then the linker's overlay manager must bring an entire code segment into memory at one time. If you have many large code segments, then the size of the overlay pool may need to be increased. Normally, the requirements of the application dictate the size of the code segments, but it should be noted that a little extra time in optimizing the size of your code will usually pay off if you have a large library of routines.

CA-Clipper-compiled P-Code is loaded in fixed-size pages so this is not a requirement when programming in CA-Clipper.

Reducing RDD Memory

Try to use as few database drivers in your application as is necessary. For example, if you choose to use the DBFCDX driver for compatability with with FoxPro files, then make sure that your application does not link in the DBFNTX (default) driver unless you must have .DBT or .NTX compatability. This is accomplished by modifying the source code in the RDDSYS.PRG that is included with CA-Clipper.

Using the "Garbage Collector"

A variety of memory problems can be caused by memory "fragmentation". Fragmentation can be caused when symbols are added to the symbol table or other "locked" memory is allocated before other "unlocked" memory segments can be released. Database languages require that the symbol table be non-contiguous and not pre-allocated but instead must be allowed to "grow" during the application. This "late binding" is the essence and power behind data-driven engines (like many Clipper applications), unfortunately this kind of flexibility can cause symbols and work-area structures to be added haphazardly thruout the conventional memory space. MEMORY(0) can report that there is plenty of memory available but after the program runs for while it exists in many small segments rather that the single large segment that was available at the start of the application.

When a large segment of memory is required by the application and is not available, Clipper has to try to move around segments that are not "locked" so it can get sufficient memory for the routine that's being called. This is "garbage collection". Unfortunately, symbol memory and other memory allocation are "locked" and cannot be moved. Applications should take into consideration reducing the amount of fixed heap, stack, and VMM memory that gets allocated during the running of the program, and also to insure that garbage collection is forced on a methodical basis to prevent the fragmention.

CA-Clipper uses a technique called "scavenging" in its memory garbage collector. The garbage collector is automatically invoked during writes to the screen and "wait" states while waiting for keyboard input. Programs that have routines which have few screen-write or input routines can fragment memory badly particularly if a lot of file opening and closing, reindexing, appending, etc. is going on with little operator interaction. It is recommended that you put calls to Devout("") in your code to invoke the garbage collector in the event that you experience "Conventional Memory Exhausted", "Memory Low", or "Stack Eval" errors. My own experience with this method has not always given me desirable results, however, and I have found that placing calls to MEMORY(-1) in strategic locations in my code, such as just before opening databases, or creating large arrays actually produces better results. This method is not supported by CA or many other developers so use this recommendation with caution. Another popular method of perfoming garbage collection is to use the FT_IDLE() function from the public domain NANFORUM TOOLKIT available on many BBS's including the CLIPPER forum of COMPUSERVE. Garbage collection should be forced especially before opening databases and indexes.

Managing DGROUP Memory

There is an area of memory in CA-Clipper, as in all large model languages, referred to as DGROUP. This 64k region of low (root) memory is also called the DS (Data Segment) because it is pointed to by the DS and SS registers. Data is often stored in this area because it can be accessed faster than data stored elsewhere. DGROUP is essentially a NEAR CALL area because accessing information in this area is accomplished by a one-word address rather than the more common 2-word addressing required to access other data areas.

Assembly-language programmers like to store data in the DGROUP area to improve performance of their application. Data in this area doesn't get overlayed and it is insured that it will always be accessible regardless of the state of the application. Interrupt handlers always store data in DGROUP to insure that the data will be accessible in the event of an interrupt. CA-Clipper stores it's stacks and static values in DGROUP.

DGROUP is structured like this.

direction of growth | --> <-- |

+--------------+-----------+------------+---------+----------------+ | C/ASM static | cpu stack | eval stack | "space" | Clipper static | +--------------+-----------+------------+---------+----------------+

The eval stack is where local and private variable VALUEs are stored. The CA-Clipper static area is where CA-Clipper static VALUES are stored. ITEMs generated by programs using ITEM.API also create VALUE entries in the CA-Clipper static area. Each VALUE entry uses 14 bytes.

If the eval stack grows into the CA-Clipper static area (or vice versa), you get a UE 667 (stack fault) error.

In low memory situations, the VMM will allocate "space" to the conventional memory pool. After this happens, if the eval stack grows into the allocated space, you get a UE 668 error. Likewise, if the CA-Clipper static area needs to grow, you get a UE 669 error.

When the eval stack or CA-Clipper static area expands, and later retracts, the system maintains "watermarks" to indicate the farthest expansion of them. The VMM allocates only the space between the "watermarks". So one way to control things is to make sure the "watermarks" are set to allow your program to execute normally.

There is very little that a CA-Clipper programmer can do to resolve DGROUP problems other than to avoid using third-party products and/or C/ASM code that uses SMALL MODEL rather than LARGE MODEL programming techniques and replace LOCAL variables with LOCAL arrays. CA-Clipper is a LARGE MODEL programming language and it is recommended in the CA-Clipper API that extensions to the language should also be compiled as LARGE MODEL. Unfortunately, many C/ASM programmers develop libraries designed for speed performance rather than memory performance. I refer to these products as "DGROUP Hogs". These libraries will consume so much of the C/ASM static space that there will be literally nothing left to run the application. If you find that you cannot limit the usage of third-party libraries, then and alternative solution is to use one or more of the methods described below under "Managing the EVAL Stack".

There are several methods to determine how much DGROUP a library uses. See next page:

Monitor memory with //Info

When you start your CA-Clipper application with the //INFO option, CA-Clipper reports the condition of memory at the start of the application. The reports looks like this:

DS=4F4E:0000 DS avail=30KB OS avail=255KB EMM avail=960KB

DS= is the starting address of the DGROUP segment.

DS avail=KB reports the amount of DGROUP available.

OS avail=KB reports the amount of conventional memory available.

EMM avail=KB report the amount of expanded memory allocated to the current application.

When the DS avail (DGROUP) is less than 15k at the start-up of an application, some large applications could experience stack eval errors at runtime. This number is arbitrary and your application may actually run just fine with less available DGROUP, however it is important that you monitor DS especially if your application uses third-party libraries or lots of LOCAL and STATIC memvars. It's always a good idea to make note of the amount of DS avail in your current application before and after you add new functions or libraries to the application. If you suddenly notice a dramatic decrease in DS avail after adding new code, then you should take note that you may be using a function from a library that uses excessive DGROUP.

Create a .MAP file

Rtlink and Blinker support a MAP= command that creates a map file with information about segment usage. This map file will show you which routines use DGROUP and exactly how much they consume.

Managing the EVAL Stack

If your application uses up a lot of DGROUP memory, then you may be required to re-visit your code and either reduce the hit on the EVAL stack or expand the size of the EVAL stack. In my opinion, it is quite easy to resolve these problems with minor changes to source code.

Use LOCAL ARRAYS instead of LOCAL VARIABLES

Every time a new function is called, all LOCAL variables are "pushed" onto the EVAL stack. They are "popped" off the stack when returning to the calling program. If a procedure or function is called recursively, then it can be quite easy to blow up the eval stack with a 667 Stack Eval Error. For example, I had this problem in one of my larger applications due to the fact that programs were deeply nested via calls to a common set of menu functions. I found that my main menu function could blow up the stack if called recursively as few as 5 times. Each time this function was called, 85 LOCAL variables were pushed on to the eval stack. I completely eliminated the problem with about 2 hours of programming and debugging by replacing the 85 separate LOCAL variables with 1 LOCAL array. A LOCAL array uses the same amount of stack space as any other LOCAL variable, therefore I reduced the amount of values pushed on the stack from 85 to 1. After doing this, I could not blow up the stack even if I called the menu system recursively more than 30 times. This was accomplished quite easily with very minor code changes. Here is an example of the BEFORE and AFTER code.

BEFORE:

FUNCTION DC_MenuMain( )

LOCAL cColor, cMenuScrn, nInkey, cTempStr, nTempNum, nItemLen,; cOldColor, nItems, nMsgLen, nHotStart, cBoxColor, ; cMenuColor, cHotColor, cSelColor, cBuffer, nRow, nCol, ; lMouseHit, nSaveRow, nSaveCol, lMessage, cGrayColor, ; nMouseStat, nStRow, nStCol, nEnRow, nEnCol, lMouVisible, ; nPass, aSubMenu, aSubBlock, nElement, aMenuItems, ; aHotKeys, aMenuBlocks, aSubItems, aBlockItems, cType, ; nStart, cTitle, lBar, lReturnVal, lShadow, aMouseKeys, ....

AFTER:

FUNCTION DC_MenuMain( )

LOCAL aMenu := Array(85)

#define cColor aMenu[1] ; #define cMenuScrn aMenu[2]

#define nInkey aMenu[3] ; #define cTempStr aMenu[4]

#define nTempNum aMenu[5] ; #define nItemLen aMenu[6]

#define cOldColor aMenu[7] ; #define nItems aMenu[8]

#define nMsgLen aMenu[9] ; #define nItems aMenu[10]

#define nHotStart aMenu[11] ; #define cBoxColor aMenu[12]

#define cMenuColor aMenu[13] ; #define cHotColor aMenu[14]

#define cSelColor aMenu[15] ; #define cBuffer aMenu[16]

#define nRow aMenu[17] ; #define nCol aMenu[18]

#define lMouseHit aMenu[19] ; #define nSaveRow aMenu[20]

#define nSaveCol aMenu[21] ; #define lMessage aMenu[22]

#define cGrayColor aMenu[23] ; #define nMouseStat aMenu[24]

#define nStRow aMenu[25] ; #define nStCol aMenu[26]

#define nEnRow aMenu[27] ; #define nEnCol aMenu[28]

#define lMouVisible aMenu[29] ; #define nPass aMenu[30]

#define aSubMenu aMenu[31] ; #define aSubBlock aMenu[32]

#define nElement aMenu[33] ; #define aMenuItems aMenu[34]

#define aHotKeys aMenu[35] ; #define aMenuBlocks aMenu[36]

#define aSubItems aMenu[37] ; #define aBlockItems aMenu[38]

#define cType aMenu[39] ; #

Programming Techniques for Memory Management (Part 1) by Roger Donnay

This section written by Roger Donnay is the second part of CA-Clipper 5.x Memory Management by Roger Donnay & Jud Cole.

---

How you write your CA-Clipper code will greatly determine how the application uses memory. Once you understand and become grounded in the basics you will discover that memory management will become second nature to your programming style. You should never write code or use any third-party library without a basic understanding of the impact on memory usage, otherwise you can paint yourself into a memory-deficient corner that can not be easily undone.

Virtual Memory

CA-Clipper includes an automatic memory manager referred to as the Virtual Memory Manager or VMM. There is very little the CA-Clipper programmer needs to know about the VMM system other than it requires EMS memory or available disk space for the creation of swap files. After CA-Clipper uses up the available EMS for VMM, it will start creating swap files. In a network situation with no local hard drive or ram drive, this can slow down the application, therefore it is recommended that you set up your environment to allocate from 500k to 1 meg of EMS for average Clipper applications. You may monitor CA-Clipper's usage of EMS during your application by reporting the remaining EMS with the MEMORY(4) function. If the number reported falls below 100, then it is recommended that you increase available EMS by several hundred K.

If no EMS is available, then Virtual memory can be allocated from XMS memory by using a Third-party product named ClipXMS. See the section titled DOS Extenders for more information on this product.

Conventional Memory

The CA-Clipper free-pool (also sometimes referred to as the fixed-heap) is always allocated from conventional memory. I have found, from experience, that applications will perform poorly or run the risk of running out of memory completely if the amount of free-pool deteriorates to less than 50k. You can count on the fact that your application will have less free-pool available after running for awhile than it has at start-up time. The CA-Clipper MEMORY(0) function can be used to monitor the amount of memory remaining in the fixed heap. Opening databases has the most serious long-term affect on the use of conventional memory. See the section titled "The Affects of Databases on Memory Usage" for more details. To give the CA-Clipper application more conventional memory, you must start at DOS with ample conventional memory. After you load device drivers, network drivers, mouse drivers, TSR's, etc. your environment just may not have sufficient conventional memory remaining to run your Clipper application. The new generation of expanded memory managers are designed to not only create EMS/XMS memory but they also will load drivers and TSR's into the upper memory blocks (UMB) area of memory above 640k so they will not use up valuable conventional memory. I have used QEMM and 386MAX in the past to accomplish this task but have recently found that, on some systems, the MEMMAKER.EXE utility and the EMM386 driver supplied with DOS 6.x are adequate in creating a better memory environment.

SYMBOL Management

For a dynamic-overlay manager to work effectively, it is important that it load overlays at the smallest possible code-segment level, i.e., the procedure/function rather than the entire object. This requires managing the symbol table separately from the code and data, therefore, these linkers place the symbol table into the root memory area and eachfunction into a separate overlay segment.

Symbols cannot be overlayed or swapped to VMM, therefore it is important that you program in a manner consistent with producing the smallest number of symbols in your CA-Clipper-compiled objects. Here are some tips for reducing the symbol table size in your applications.

Use Constants instead of Memvars

All PRIVATE, PUBLIC and STATIC CA-Clipper memory variables are treated as "symbols". Refrain from using a memory variable if a constant is sufficient. For example, an unnecessary symbol can be eliminated by changing the code:

nEscapeKey := 27

DO WHILE INKEY() # nEscapeKey

* CA-Clipper code

ENDDO

to:

DO WHILE INKEY() # 27

* CA-Clipper code

ENDDO

or:

#define K_ESC 27

DO WHILE INKEY() # K_ESC

* Ca-Clipper code

ENDDO

Use Arrays Instead of Memvars

Every different CA-Clipper PRIVATE, PUBLIC, or STATIC memvar name creates a "symbol", whereas an array name creates only ONE symbol. The following example shows how to save considerable memory in a CA-Clipper application by reducing the symbol count with an array.

This code produces 5 symbols:

PRIVATE cName := customer->name

PRIVATE cAddress := customer->address

PRIVATE cCity := customer->city

PRIVATE cState := customer->state

PRIVATE cZip = customer->zip

@ 1,1 SAY 'Name ' GET cName

@ 2,1 SAY 'Address' GET cAddress

@ 3,1 SAY 'City ' GET cCity

@ 4,1 SAY 'State ' GET cState

@ 5,1 SAY 'Zip ' GET cZip

READ

This code produces 1 symbol:

PRIVATE aGets[5]

aGets[1] := customer->name

aGets[2] := customer->address

aGets[3] := customer->city

aGets[4] := customer->state

aGets[5] := customer->zip

@ 1,1 SAY 'Name ' GET aGets[1]

@ 2,1 SAY 'Address' GET aGets[2]

@ 3,1 SAY 'City ' GET aGets[3]

@ 4,1 SAY 'State ' GET aGets[4]

@ 5,1 SAY 'Zip ' GET aGets[5]

READ

Some programmers choose memvars over arrays because it makes their source code more readable. Source code that refers to hundreds of array elements can look very cryptic and can be hard to maintain. To reconcile this problem, use the CA-Clipper pre-processor to create a "symbolic" reference to each array element like in the following example.

STATIC aGets[5]

#define cNAME aGets[1]

#define cADDRESS aGets[2]

#define cCITY aGets[3]

#define cSTATE aGets[4]

#define cZIP aGets[5]

cNAME := customer->name

cADDRESS := customer->address

cCITY := customer->city

cSTATE := customer->state

cZIP := customer->zip

@ 1,1 SAY 'Name ' GET cNAME

@ 2,1 SAY 'Address' GET cADDRESS

@ 3,1 SAY 'City ' GET cCITY

@ 4,1 SAY 'State ' GET cSTATE

@ 5,1 SAY 'Zip ' GET cZIP

READ

Use the Same Name Memvars whenever possible

Again, every "different" PUBLIC, PRIVATE or STATIC CA-Clipper memvar in a module creates a symbol. If an object contains several procedures, use the same name for memvars even though they may not perform the same or similar functions. For example, procedure A and procedure B both need 5 memvars. If procedure A declares its memvars with 5 unique names and procedure B declares its memvars with 5 unique names, then 10 symbols are used in the linked application. To eliminate 5 symbols, make sure that procedure B assigns the same name to the memvars as procedure A. This is not possible of course, if the memvars need to be PUBLIC to both procedures and perform different functions, only if they are PRIVATE.

Use Complex Expressions instead of Memvars

The following three lines of code represents the method that most CA-Clipper programmers choose to accomplish most programming tasks. It makes sense to code this way for readability and debugging, but if you are writing a very large application, the complex expression technique can save some memory. The following three lines of code will read the disk file READ.ME into a memvar named cReadFile, save the changed code into a file named cEditFile, then write the changed code back to the disk file READ.ME.

cReadFile := MEMOREAD('READ.ME')

cEditFile := MEMOEDIT( cReadFile )

MEMOWRIT('READ.ME', cEditFile )

These three lines of code can be replaced by one complex expression which uses no symbols at all.

MEMOWRIT("READ.ME",MEMOEDIT(MEMOREAD("READ.ME")))

An additional advantage to coding this way is that less free-pool or VMM memory is used because the process of temporarily storing the text in cReadFile and cEditFile is completely eliminated. If you find that creating complex expressions such as this are unreadable and hard to maintain then use the CA-Clipper pre-processor to accomplish the same task as follows:

#define cReadFile MEMOREAD('READ.ME')

#define cEditFile MEMOEDIT(cReadFile)

MEMOWRIT('READ.ME',cEditFile)

The above code is nearly as readable as the original three lines of code but will not create any variables at compile time. Try compiling this code with your CA-Clipper compiler and use the /P switch to write the pre-processed code to a .PPO file, then look at the .PPO file to see what is actually compiled.

Use LOCALS Instead of PRIVATES

Sometimes it is just not possible or practical to write code without using symbols, so if you find yourself in this situation, CA-Clipper provides the feature of "LOCALIZING" symbols to the code segment which is currently being executed rather than placing the symbol in the main symbol table. LOCAL symbols are effectively "overlayed" because they are treated as part of the code segment rather than given a place in the main symbol table.

Not only does this save valuable memory but it also improves speed performance because the public symbol table does not need to be searched each time a LOCAL symbol is referenced in your code. Of course, if the symbol you are referencing is needed for the entire application or is used in a macro, then it must be declared as PRIVATE or PUBLIC. Symbols which are not declared at all are automatically assumed to be PRIVATE, so make sure you use the LOCAL declaration for all symbols in your code which you do not want to end up in the main symbol table. In the previous code example, the 5 PRIVATE memvars consume 110 bytes of memory, the single PRIVATE array consumes 22 bytes of memory, whereas 5 LOCAL declarations would consume 0 bytes of memory.

Use STATIC functions instead of PUBLIC functions

Every PUBLIC function and procedure name will occupy 22 bytes of the symbol table. Don't make functions and/or procedures PUBLIC unless they need to be called from anywhere within your application. STATIC procedures and functions are called only from within the source code module in which they are declared thereby eliminating the need to place the function name into the public symbol table.

Use PSUEDO functions or CONSTANTS instead of PUBLIC functions

A Psuedo-function is a function that does not really exist in the compiled code but exists only in your source code. Many programmers will use a large number of functions with different names even when each function may do something very similar like a conversion or a table lookup. I have seen a lot of code that looks like this:

SetColor( blueonwhite() + "," + redongreen() + "," + black() )

FUNCTION blueonwhite

RETURN 'B/W'

FUNCTION redongreen

RETURN 'R/G'

FUNCTION black

RETURN 'N/N'

Programmers will use this technique to make their code easy to read and maintain without understanding how this can bloat the size of the application. Probably the best technique to replace the above code would be to #define constants for each as follows:

SetColor( BLUEONWHITE+ "," + REDONGREEN + "," + BLACK )

#define BLUEONWHITE "B/W"

#define REDONGREEN "R/G"

#define BLACK "N/N"

Your existing source code may have "many" references to public functions and you don't want to risk changing all the function names to constants, or maybe the functions return something a little more complex than can be handled by a simple constant. In this case, you can #translate the functions into psuedo-functions as follows:

BEFORE:

FUNCTION BLOBget ( nPointer, nStart, nCount )

RETURN dbInfo( BLOB_GET, { nStart, nCount } )

FUNCTION BLOBput ( nPointer, xBlob )

RETURN dbInfo( BLOB_PUT, { nPointer, xBlob } )

FUNCTION BLOBExport( nPointer, cTargetFile, lMode )

RETURN dbInfo( BLOB_EXPORT, { nPointer, cTargetFile, lMode } )

AFTER:

#xTranslate BLOBget( , , ) => ;

dbInfo( BLOBGET, { , , } )

#xTranslate BLOBput( , ) => ;

dbInfo( BLOBPUT, { , } )

#xTranslate BLOBExport( , , ) => ;

dbInfo( BLOBEXPORT, { , , } )

In the BEFORE example above, three PUBLIC functions were created simply to provide a better way to call the same dbInfo() function, whereas in the AFTER example, no PUBLIC functions were needed to accomplish the exact same task. Not only did we save 3 symbol table entries but we also eliminated 3 LOCAL variables that get pushed onto the EVAL stack. Of course it must be remembered that since psuedo-functions don't actually exist they cannot be used in macros or index keys.

Don't use STATICs or PRIVATEs to pass parameters

Many CA-Clipper programmers will assign a variable as PRIVATE or STATIC so it can be accessed and changed within multiple procedures in the same source code module. Variables should be STATIC only if their value needs to be maintained throughout the program, not for the convenience of eliminating the need to pass parameters. By using the pass-by-reference symbol "@" you can change the value of LOCAL variables anywhere within your calling program as shown by the following example.

LOCAL cName, cAddress

MyFunction( @cName, @cAddress )

Return( { cName, cAddress } )

STATIC FUNCTION MyFunction ( cName, cAddress )

cName := "CA-Clipper"

cAddress := "New York, NY"

Return( nil)

The "Myth" of PUBLICs verses STATICs

There is no memory advantage to using a STATIC variable in lieu of a PUBLIC variable. Both types of variables use up permanent space in conventional memory. A STATIC variable will use space in DGROUP, whereas a PUBLIC variable will use space in the Symbol Table. In fact, there are situations in which a good case can be made for using a PUBLIC array instead of a STATIC array. Take the example of a system-wide color system in which the colors are defined in an array. If the array is STATIC, then extracting the information from the array requires a call to a PUBLIC function which exists in the same source code module as the array definition. This would actually add more symbols to the symbol table and take more processor time than if the color array were defined as PUBLIC thereby allowing direct access to the array without the need for the public function. A STATIC array may be a wiser choice in cases where the application is a library which is incorporated into another CA-Clipper application, thereby eliminating any possibility of symbol conflicts with the application.

DGScan-The DGROUP Usage Scanner For Clipper by Ian Day & Dave Pearson

By Ian Day and Dave Pearson

Background On DGROUP

(Portions of this section are taken, with the kind permission of Dark Black Software, from the MrDebug Norton Guide)

DGROUP is a 64K chunk of memory that, for a Clipper program, can be broken down into five distinct sections, each serving a specific purpose. DGROUP can be easily described as a 64K block of memory that is used by Clipper as a common area of memory that Cliper and third party products can rely on to find various items of information. There are five main parts to DGROUP:

1. The MemVar table (Dynamic)

This is used to store Clipper STATIC variables and ITEMs (declared through the ITEM API) with their contents if the contents and variable/ITEM definition can be held within 14 bytes, otherwise the variable/ITEM definition is held along with a pointer to a Virtual Memory segment where the variable/ITEM contents are held.

2. DS Available (Dynamic)

This is the amount of free memory available within the DGROUP. This amount can be seen from //INFO as DS AVAIL or from the Memory/Info window. This will be reduced during the execution of the program as the Eval Stack and the Memvar table may both 'grow' into this area.

3. Eval Stack (Dynamic)

This is where Clipper LOCAL variables are held (if the variable definition and contents take up 14 bytes or less), otherwise a pointer is stored to a Virtual memory segment that contains the variable contents.

4. CPU Stack (Fixed)

The CPU stack is used to store the function return addresses each time a new function is called, as well as local 'C' variables.

The size of the CPU stack is set when you link your program. It depends upon the linker that you are using and linker specific commands that you have in the link script that might increase or reduce the stack

5. Fixed C and ASM Data (Fixed)

This area is used by other third party libraries and other languages that do not use their own data segments and use the default data segment (otherwise known to us Clipperites as DGROUP) to store fixed pieces of text and global variables used by other languages

For example, inside Clipper, the un-recoverable error messages are stored within this section of the default data segment (DGROUP).

As you can see from the above, the DGROUP area is pretty important to the smooth running of your Clipper application, and a lack of DGROUP can cause your software to fall over with a number of internal errors.

In an effort to make sure this does not happen you should try to reduce the impact your code has on DGROUP. Many 3rd party libraries use large ammounts of DGROUP (in my experience they can bump up the fixed C and ASM data usage by quite a bit. If you need to use a number of 3rd party libraries this can become quite a problem.

However, given that your chances of changing the usage of 3rd party libraries are pretty slim, the only chance you have of making a difference is by improving your own code to reduce it's DGROUP usage.

What Can Be Done?

If you write nothing but pure Clipper code, you still have the ability to reduce your DGROUP usage. The main area you can address is your use of STATIC variables. STATIC variables are, without a doubt, a good thing, but, they come with a price.

Unlike the other variable types, each STATIC variable requires a fixed (14 byte) entry in the the MemVar table. On the surface this may not seem like a big deal. A STATIC only needs 14 bytes in DGROUP, what does it matter? To see how this matters, lets take the GETSYS.PRG code you can find in the SOURCE\SYS directory of your copy of Clipper as an example.

Close to the top of the source you will find the following code:


//
// State variables for active READ
//
STATIC sbFormat
STATIC slUpdated := .F.
STATIC slKillRead
STATIC slBumpTop
STATIC slBumpBot
STATIC snLastExitState
STATIC snLastPos
STATIC soActiveGet
STATIC scReadProcName
STATIC snReadProcLine


As you can see, there are 10 STATIC variables, each using 14 bytes in the MemVar Table. With a simple little trick we can reduce that 140 bytes of usage down to 14 bytes. Watch:


// Hold all the statics in one static.

Static _aSysStuff := { NIL, .F., NIL, NIL, NIL, NIL,; NIL, NIL, NIL, NIL }

// Translated static variable names.

#xtranslate sbFormat => _aSysStuff\[ 01 \] #xtranslate slUpdated => _aSysStuff\[ 02 \] #xtranslate slKillRead => _aSysStuff\[ 03 \] #xtranslate slBumpTop => _aSysStuff\[ 04 \] #xtranslate slBumpBot => _aSysStuff\[ 05 \] #xtranslate snLastExitState => _aSysStuff\[ 06 \] #xtranslate snLastPos => _aSysStuff\[ 07 \] #xtranslate soActiveGet => _aSysStuff\[ 08 \] #xtranslate scReadProcName => _aSysStuff\[ 09 \] #xtranslate snReadProcLine => _aSysStuff\[ 10 \]

With that simple little trick you will have reduced the DGROUP usage from 140 bytes to just 14 bytes.

If you have lots of code of your own that has a large number of file wide STATICs in a single file you can make quite a difference.

If you write your own C functions, then it's a wise move to take a look at the compiler options at your disposal, as some of these can drastically reduce the impact on DGROUP.

For example, take the following piece of code:


#include

CLIPPER YesNo( void ) // Return yes or no string { if ( _parl(1) ) // .T. passed to function? { _retc( "Yes" ); // Yes, so return "Yes" } else { _retc( "No" ); // No, so return "No" } }

Simple enough, but already we've just used 7 (or 8) bytes of DGROUP without thinking! Why? you ask. Answer: It's the strings. By default, C compilers will put constants (which means strings, doubles, floats and some arrays) into DGROUP. And when I say '7 (or 8)', it's because some compilers will align data on a WORD boundary as default, and you can usually expect the DGROUP usage to be a tad more than you thought.

You can change this by looking for a compiler option to reverse this default. For example, with the Microsoft C compilers, there is the /Gt switch and for Borland C compilers there is the -Ff switch. Given a number of 1, it will automatically put any constants larger than 1 length into a FAR segment, and out of DGROUP! This means, that no matter how much data is used by your C functions for strings and things, it won't make a difference to DGROUP.

With assembler code, we use the same principles. Don't use _DATA, _BSS, CONST or DGROUP for data storage if you can possibly help it.

So, if you have a module like this:


_DATA SEGMENT WORD PUBLIC 'DATA'
cYes DB 'Yes', 0
cNo DB 'No', 0
_DATA ENDS

DGROUP GROUP _DATA

YESNO_TEXT SEGMENT WORD PUBLIC 'CODE'

ASSUME cs:YESNO_TEXT, ds:DGROUP

YESNO PROC FAR PUSH bp MOV bp, sp

MOV ax, 1 PUSH ax CALL __parl ADD sp, 2

OR ax, ax JZ ZeroSoItsFalse

MOV ax, OFFSET cYes JMP SHORT ReturnString

ZeroSoItsFalse: MOV ax, OFFSET cNo

ReturnString: PUSH ax PUSH ds CALL __retc ADD sp, 4

POP bp RETF YESNO ENDP

YESNO_TEXT ENDS END

You've again used 7 (or 8) bytes of DGROUP without thinking, so in this case you would have to alter the segments and anything that used the data in them so that DGROUP can once again be saved:


YESNO_DATA SEGMENT WORD PUBLIC 'FAR_DATA'
cYes DB 'Yes', 0
cNo DB 'No', 0
YESNO_DATA ENDS

YESNO_TEXT SEGMENT WORD PUBLIC 'CODE' ASSUME cs:YESNO_TEXT, ds:NOTHING

YESNO PROC FAR PUSH bp MOV bp, sp

MOV ax, 1 PUSH ax CALL __parl ADD sp, 2

OR ax, ax JZ ZeroSoItsFalse

MOV ax, OFFSET cYes JMP SHORT ReturnString

ZeroSoItsFalse: MOV ax, OFFSET cNo

ReturnString: PUSH ax PUSH YESNO_DATA CALL __retc ADD sp, 4

POP bp RETF YESNO ENDP

YESNO_TEXT ENDS END

As you can see, if your application is tight on DGROUP, with the source to hand and a little bit of work you can free up some of that usage. Lets just hope you have the source code for the worst offending code in your application.

Using DGSCAN To Find DGROUP Usage

Back when I first became aware of this "problem" I started to work on my own library code and managed to reduce some of the DGROUP usage. However, working through all that code was getting to be pretty boring. That's when I decided to write DGSCAN. What I wanted was a tool that could scan OBJ and LIB files and find anything that might be taking up precious DGROUP space.

If you want to get and try out DGSCAN then pop over to Hagbard's World and grab a copy. Got it? Good.

Ok, lets take a simple example of using DGSCAN. Look at the following code:

Static xVar1
Static xVar2
Static xVar3
Static xVar4

Function Main()

? "I'm eating up DGROUP"

Return( NIL )

Compile it into an OBJ and then run DGSCAN over it:

F:\DGTEST>dgscan foo.obj

DgScan v3.00 - DGROUP Usage Scanner By Dave Pearson

File-------- Module--------------------------- Segment----------- Bytes----- FOO.OBJ FOO STATICS$ 56 FOO.OBJ *** Total *** 56

As you can see, DGSCAN has detected 56 bytes of possible DGROUP usage. This is because we have 4 STATIC variables, each with a 14 byte impact. If you applied the "array trick" we spoke about earlier you would see the usage reduced to 14 bytes.

To generate a report for all your OBJ files just do:

F:\DGTEST>dgscan *.obj


It's as simple as that, and the same goes for LIB files.

I won't say much more about DGSCAN, everything you need to know is covered by the file DGSCAN.TXT found in the DGSCAN archive. However, if you do have problems with the utility please feel free to mail me and I'll try to help you out.

What is dgscan ?

[3rd Sept 1996]

Since Oc'95, I have noticed occasional postings on comp.lang.clipper pertaining to 66x runtime internal errors related to over-usage of the limited DGroup memory used by CA-Clipper. So what's the solution. In the course of communicating with Dave Pearson, one of my nominated personalities on comp.lang.clipper, I noticed that his e-mail signature referred to something that he called dgscan. Curious as ever, I hassled him for more information and here's his reply :

What is dgscan - DGROUP scanner for Clipper ?

Well, as you probably know the DGROUP area of a Clipper program is pretty important to it's health. Abuse of the above can lead to plenty of problems, mostly ending up with one of the 66x internal errors.

Clipper code and C/ASM code can eat away at DGROUP quite easily, so I wrote DGScan to scan OBJ and LIB files and track down possible dgroup usage. With this utility I've managed to shave off quite a bit of DGROUP usage in my own LIBS.

To see exactly what it does download it from my pages and have a play. You might be suprised at how much possible DGROUP usage you can find and remove. I tend to use a number of tricks in my own code. For example, if you have a number of static variables in a PRG, reduce them to a single array and #xtranslate the variable names, for example, where you had:

Static lVar := .F.

Static nVar := 0

Static cVar := ""

Static dVar := ctod( "" )

you could replace this with:

Static aStatics := { .F., 0, "", ctod( "" )

#xtranslate lVar => aStatics\[ 1 \]

#xtranslate nVar => aStatics\[ 2 \]

#xtranslate cVar => aStatics\[ 3 \]

#xtranslate dVar => aStatics\[ 4 \]

The first example code has a possible impact of 14 * 4 bytes, while the second has only a 14 byte possible impact.

The other usage to look out for is things like static text in C and ASM code. For example, with Borland C++ I tend to add "-Ff=0" as one of the switches to the command line. This ensures that any static text is placed in (I think) far data rather than in DGROUP. This reduces the overhead on DGROUP.

Want to know more about dgscan Version 3.00 by Dave Pearson ?

What it is

DGScan is a simple little command line utility that will scan your Clipper 5.x oriented OBJ and LIB files and produce a report of their usage of Clipper's DGROUP.

DGScan will work with and has been tested on C, ASM and Clipper OBJ and LIB files.

Please also note that DGScan is free software, protected by the GPL. Please read the file COPYING for full details.

What it isn't

DGScan can't currently tell you who is using what DGROUP in your application. All it can do is detect possible DGROUP impact in any OBJ or LIB file. At some point in the future I aim to either expand DGScan or develop a new utility that will scan your application and tell you exactly who is using what parts of DGROUP.

For the moment DGScan is a tool to help you work out what could be using DGROUP. When scanning 3rd party library files keep in mind that you probably don't make use of all the functions so you probably don't link in all the modules. On the other hand it's not easy to know what modules the library file uses internaly.

How to use it

The syntax is as follows:

DGScan [] | @ []

where is the name of a single OBJ or LIB file or a wildcard file spec for a number of OBJ or LIB files. Example usage:

DGScan main.obj // Scan a single OBJ file.

DGScan *.obj // Scan all OBJ files.

DGScan clipper.lib // Scan a library file.

DGScan *.lib // Scan all library files.

etc.

As an alternative you can use a "script" file. This file should contain the names of files that you want to scan, each file on it's own line. The name of the file should start on the first column, any lines that start with either a space or a ';' are considered to be comments. Blank lines are allowed.

Example usage of a script file is:

DGScan @CCode.Lst

The switches are as follows:

/S - Ignore Clipper static variables. If used this switch tells DGScan not to include Clipper static variables in the scan.

/T - Totals only. If used this switch tells DGScan to only display the total usage.

/Q - Quiet. If used this switch stops DGScan from showing it's "banner".

All of the above switches can also be placed inside an environment variable called DGScan. So, if you want DGScan to ignore Clipper static variables and only display totals by default you would:

Set DGScan=/s/t

Format of output

DGScan writes a simple report to the screen, this can be redirected to a file using standard DOS redirection. The format of the report is:

File-------- Module------------------------ Segment-------- Bytes-----

where: is the name of the OBJ or LIB file.

is the name of the module in the OBJ or LIB file.

is the name of the segment that us using DGROUP.

is the number of bytes of DGROUP in use.

An example report is:

File-------- Module------------------------ Segment-------- Bytes-----

FLOPNAME.OBJ flopname.c _DATA 4

FLOPNAME.OBJ *** Total *** 4

If more than one file is scanned then a grand total will also be displayed.

What to look for

The first and most obvious thing to look for is the segment name STATICS$ for Clipper compiled code. If the byte figure is high you may want to consider reducing the number of statics in a file. The number of bytes is calculated based on the fact that a single Clipper static variable will have a 14 byte impact on DGROUP.

If you have, for example, ten file wide static variables in a PRG file you may consider storing the ten items in an array held in a single static. With such an example the DGROUP impact would be reduced from 140 bytes to just 14 bytes.

With C code you will probably know yourself what the "problem" areas are. Obvious things to watch out for are hard coded strings in a C function, for example:

_retc( "" );

will have a one byte impact on DGROUP (the segment will probably show up as _DATA), you could get rid of this with a simple:

char *p = { 0 };

_retc( p );

Other things to look out for are static variables (the segment will probably be _BSS), but I would assume that you have placed them there for a reason.

To a large extent when it comes to C and ASM code you are kind of on your own because the segment names etc. may differ from compiler to compiler or you mave have given them your own names.

ErrorLevels

Just in case you ever wanted to have DGROUP detection in your Compile batch file/make file DGScan will return an ErrorLevel of 1 if any DGROUP impact is found in a scan and 0 if none is found.

I doubt that this will be of any use but its there if you need it.

OS/2 Support

Used to exist, but no longer. Since I added more memory to my machine OS/2 has refused to play ball. Dos, Windows, Win95 and Linux are all happy, only OS/2 won't play. :-(

Perhaps Merlin will be better. :-)

If you still want to run under OS/2, you have the source.

The History

Release Date Reason

1.00.0 23/11/94 First version, release to DBSL for testing. Had no library file support and had trouble with unknown record IDs.

1.00.1 24/11/94 Added support for library files by ignoring unknown record IDs. Changed the memory model to large. Without this the utility would die when reading large library files. Added support for Clipper Static variables. Re-wrote the output so that it is now a proper report of sorts.

1.00.2 25/11/94 Added code to ignore _SYMP segment entries. Because of this DGScan will now show you DGROUP usage that you can affect in your own code (you can do nothing about the usage of the _SYMP segment). Changed the way we calculate the impact of C and ASM code. We can now detect static variables in C and ASM files whereas before they were not included in the calculation.

1.01.0 02/12/94 Added a verbose mode switch to the command line that causes all segments to be displayed in the report.

1.02.0 05/12/94 Fixed a stupid bug that caused DGScan to ignore DGROUP usage from time to time. This was a side effect of the Verbose switch of all things! Thanks for finding that one Ian.

1.03.0 06/12/94 Fixed a stupid memory allocation bug that caused strange results on Ian Day's machine. Also added errorlevel support. If DGROUP usage is found in a scan then an errorlevel of 1 is returned, otherwise an errorlevel of 0 is returned, Also added support for "Script" files.

1.04.0 08/12/94 Fixed a strange bug found by Ian Day. Also removed the verbose switch because of incorrect user expectations. There was little need for it in the first place.

1.05.0 13/12/94 Far better library scanning. This should have correctly fixed a couple of the previous bugs found by Ian and should also make the scanning of some library files a bit faster.

1.06.0 15/12/94 Long module and segment names will no-longer screw up the format of the report.

1.07.0 23/12/94 Added the /S and /T command line switches. If the filename on the command line has no extension DGScan will now look for .OBJ then .LIB.

1.08.0 28/12/94 Added the /Q switch. Added support for switches in an environment variable called DGScan. Fixed another silly bug that would cause DGScan to miss some DGROUP usage. Included BLIDGRP.

2.00.0 01/03/95 Changed into Dos and OS/2 versions. Also changed the name of the .EXE 'cos I got fed up with typing such a long name. ;-)

2.00.1 22/03/95 Got rid of DGSCAN2.EXE. DGSCAN.EXE is now a bound executable. Also dropped support for BLIDGRP, it was a pointless idea in the first place.

3.00.0 18/07/96 Finally got round to sorting things out so that I can release the source code. DgScan is now placed under the GPL.

Feedback

If you have any comments or problems with DGScan you may mail me at davep@hagbard.demon.co.uk. Also, you can allways find the latest version of this utility at:

http://www.acemake.com/hagbard

Any feedback would be great so just fire away.....

Many Thanx

Many thanx must go to Ian Day who showed me how to calculate the number of Clipper static variables from information found in a Clipper OBJ file and also managed to find that one extra library that would break the scanner each time I was happy that it was working fine. Thanx also to Darren Lancaster who let me pester Ian when he had better things to be doing with his time and Brian Dukes and Stu Beesley for testing this utility.

Thanx must also go to Kip Davidson and Dave Cortesi, two guys I've never met or spoken to but who wrote a utility called READOBJ, the code for this utility helped me understand the format of OBJ files.

DavePearson

By the way, I am impressed with Dave's creation ! Are you ? Click here to download this excellent tool.