Linking Techniques for Memory Management (Part 2) by Roger Donnay

This section written by Roger Donnay is the third part of CA-Clipper 5.x Memory Management by Roger Donnay & Jud Cole.

Overlaying Library Modules
The MODULE command provides the important capability of overlaying the larger modules of the CA-Clipper libraries or any third-party library. With previous versions of these linkers, the overlaying of libraries was restricted to ALL modules in a library or NO modules in a library. This is ok for libraries which were designed for overlaying, but in most cases it is simply not practical to overlay an entire library due to speed performance problems. There are many modules in the CA-Clipper libraries which can be overlayed without crashing the program, yet runtime performance can be disastrous. Determining the proper "set" of modules and the optimum "overlay pool size" is simply a matter of trial and error. For example, overlaying the MACRO module in one application may have very little affect on speed performance because of the design of the program and it's probably negligible.

The MODULE command is very useful in that it allows your linker to place "modules" from libraries into overlays. You can organize your projects by placing all your C/ASM objects in libraries with a library manager such as Microsoft's LIB.EXE then by using the SECTION MODULE command in your link file you can place any module into any overlay area.

Rtlink:

LIB mylib

BEGINAREA

SECTION MODULE myfileA,myfileB

SECTION MODULE myfileC

SECTION MODULE myfileD

ENDAREA

The MODULE command in RTLINK requires that the exists in one of the declared libraries and that the name is "unique" to that module otherwise the command will be ignored and the module will be linked into the root memory area. This is not a problem with the CA-Clipper libraries because the modules are given unique names.

BLINKER supports the MODULE command differently and, in my opinion, better than RTLINK. Blinker allows you to define a specific module from a specific library to be linked into the application, thus you don't need to worry about the module being unique to one library. For example, if you have two libraries, both containing an ERRORSYS module, you can decide which ERRORSYS you want linked into the program.

Blinker:

LIB mylib

BEGINAREA

LIB grump

MODULE errorsys FROM grump

MODULE myfileA,myfileB

MODULE myfileC

MODULE myfileD

ENDAREA

Determining the Module Names in a Library

Every ".OBJect" file placed into a library is also assigned a "module" name. This module name is stored in the "COMENT" record of the object in the library. The "module" name of an object is assigned by the compiler of the object and is usually given the name of the original source code. The CA-Clipper compiler assigns the source code file name to the module name however and many other compilers will assign the name of the SOURCE file as the module name including drive letters and directories. For example, let's say we want to place the Funcky FINDATTR() function into an overlay. First, we must figure out what "module" name was assigned to the FINDATTR() function. There are library manager utility programs which can help you figure this out, but I'm going to show you how to do it simply with your linker. During VERBOSE linking the "module" name of each object being linked into the program is displayed on the screen as follows:

FUNCKY50.LIB(C_MAXCHO) <- CA-Clipper code FUNCKY50.LIB(C_PUTKEY) <- CA-Clipper code FUNCKY15.LIB(findattr.C) <- "C" code FUNCKY15.LIB(finddate.C) <- "C" code FUNCKY15.LIB(findfirs.C) <- "C" code FUNCKY15.LIB(chrfound.ASM) <- "Assembly" code FUNCKY15.LIB(strcente.ASM) <- "Assembly" code You must make sure that you insert the command VERBOSE into your link script file to insure that this information is displayed. Next, when you perform the link, make sure route the display output to a file so you can have the information in text file format: RTLINK @MYPROG > MYPROG.TXT

MYPROG.TXT will contain the complete list of modules linked into your program. Now, we must take this information and use it to develop an overlay strategy. The name of each module linked into the program is shown (in parenthesis) and must be referred to "exactly" as it is spelled. For example, to place the function FINDATTR() into an overlay, we must first determine the "module" name which would contain the FINDATTR() function. Fortunately, the FUNCKY15 library uses the same prefix name as the actual function, so in perusing the list of linked objects in MYPROG.TXT we find a module named "findattr.c". To place this module in an overlay section, simply use the command:

SECTION MODULE findattr.c

BLINKER's MODULE command is more useful than RTLINK's because it searches both the THEADR record and the COMENT record for the name of each module. This means that you don't need to reference the MODULE name as precisely with Blinker as with RTLINK. For example, if the module name in a library is FINDATTR.C, BLINKER will overlay it by with the following command: MODULE FINDATTR, because it will determine that what you are trying to overlay is FINDATTR.OBJ.

Overlaying the CA-Clipper Libraries

Overlaying the CA-Clipper libraries is accomplished quite differently with each linker, therefore I have included examples for each linker.

Overlaying Code with RTLINK

RTLINK uses Static reloadable overlays with and the MODULE command to overlay much of the CA-Clipper libraries and thereby reduce memory usage in your CA-Clipper-5.x applications. The below link script example creates an overlay area for overlaying the larger modules in the CA-Clipper libraries which are not likely to call each other recursively, therefore you will probably notice very little difference in speed performance yet you will get up to 50K more memory overhead depending on how much of the CA-Clipper libraries your application uses.

You may get a "warning" message during link time if your application does not call one of the modules referenced in an overlay area. In the event this happens, simply remove that module from the link file. For example, if you are not using MEMOEDIT() in your application remove the line:

SECTION MODULE D:\CC\EDIT\MEMOEDIT.C

FI

LIB

OUTPUT

RELOAD FAR 200

# Create overlay area 1

BEGINAREA

#1

SECTION MODULE D:\CC\EDIT\MEMOEDIT.C

MODULE D:\CC\MEMO\MEMOTRAN.C

MODULE D:\CC\MEMO\MEMOREAD.C

MODULE D:\CC\MEMO\MEMOWRIT.C

MODULE D:\CC\MEMO\MEMOLINE.C

MODULE D:\CC\MEMO\MLCOUNT.C

MODULE D:\CC\MEMO\MLPOS.C

#2

SECTION MODULE D:\CC\STR\IS.C

MODULE D:\CC\MEMO\HARDCR.C

MODULE D:\CC\STR\PADL.C

MODULE D:\CC\STR\PADC.C

MODULE D:\CC\STR\PADR.C

MODULE D:\CC\ARRAY\ASCAN.C

MODULE D:\CC\ARRAY\ASORT.C

MODULE D:\CC\ARRAY\DIRECTRY.C

MODULE D:\CC\BASE\AEVAL.C

MODULE D:\CC\BASE\ACOPY.C

MODULE D:\CC\BASE\ADEL.C

MODULE D:\CC\BASE\AINS.C

MODULE D:\CC\BASE\ATAIL.C

MODULE D:\CC\MISCX\COPYFILE.C

MODULE D:\CC\MISCX\TYPEFILE.C

MODULE D:\CC\TERM\SCROLL.C

MODULE D:\CC\MISCX\GETE.C

MODULE D:\CC\MISCX\DISKSPAC.C

#3

SECTION MODULE D:\CC\SDF\SDF1.C

MODULE D:\CC\SDF\SDF0.C

MODULE D:\CC\SDF\SDFDYN.ASM

MODULE D:\CC\SDF\DLM1.C

MODULE D:\CC\SDF\DLM0.C

MODULE D:\CC\SDF\DELIMDYN.ASM

MODULE D:\CC\EDIT\ACHOICE.C

MODULE D:\CC\DBCMD\DBCREATE.C

MODULE D:\CC\DBCMD\DBJUNCT.C

MODULE D:\CC\DBCMD\DBSTRUCT.C

#4

SECTION MODULE D:\CC\DBF\NET.C

MODULE D:\CC\TERM\ACCEPT.C

MODULE D:\CC\SUPPORT\OSDATE.ASM

#5 SECTION MODULE D:\CC\BASE\FGET.C

MODULE D:\CC\TERM\OLDBOX.C

MODULE D:\CC\TERM\OLDCLEAR.C

MODULE D:\CC\INITEXIT\RUN.C

MODULE D:\CC\BASE\SEND.C

MODULE D:\CC\DBCMD\JOINLIST.C

MODULE D:\CC\NUSASCII\SORTBLOC.C

MODULE D:\CC\DBF\SORTOF.C

ENDAREA

# Create overlay area 2

BEGINAREA

#6

SECTION MODULE D:\CC\EDIT\TBROWSE0.C

#7

SECTION MODULE D:\CC\EDIT\TBROWSE1.C

#8

SECTION MODULE D:\CC\MISCX\EXAMPLEA.ASM

MODULE D:\CC\DBF\DBF0.C

MODULE D:\CC\DBF\DTX0.C

MODULE D:\CC\INITEXIT\INITEXIT.C

MODULE D:\CC\SUPPORT\TXOPEN.C

MODULE D:\CC\TERM\BOX.C

MODULE D:\CC\STR\STRTRAN.C

ENDAREA

When placing modules into overlays, you must use extreme care to not overlay functions which are called by "interrupts" or may be called "recursively". Code which is called by interrupts is called by a direct vector rather than an overlay vector. This can cause computer lockup if the code segment is not currently in memory. Code which is called recursively would be something like special string handling routines that might be called in a do..while loop. If this function were placed in an overlay, you may experience considerable slowing and/or disk access.

Many third-party library developers now distribute their product in two libraries: a "resident" library and an "overlayable" library. Most of the work has already been done for you in determining which modules can be overlayed. If the module is part of the "resident" library, don't make any attempt to try to overlay it or you will experience runtime problems. Also, don't make an attempt to overlay "CA-Clipper code" because this code is automatically overlayed into "dynamic-pages".

A file named MEM52.LNK is provided on the diskette. This is a standard link-script for Rtlink that can save approximately 50k of conventional memory by just calling it from your existing link-script.

Overlaying Code with BLINKER

Overlaying the CA-Clipper libraries with Blinker is simpler than Rtlink because only one overlay area needs to be defined for all overlayable modules, files and libraries. Like Rtlink, Blinker will automatically overlay any code in any library that is compiled with the CA-Clipper compiler.

BLINKER OVERLAY OPSIZE 50

BLINKER PROCEDURE DEPTH 50

FI

LIB

OUTPUT

BEGINAREA

FILE

ALLOCATE

# Modules from CLIPPER.LIB

MOD accept,acopy,adel,aeval,ains,appinit,atail,box,cmem,date

MOD dbcmd2,dbcmd3,dbcmd4,dbcmd5,dbcreate,dbf0,dbfdyn,dbjunct

MOD dbstruct,dtx0,dtx1,dtxdyn,dynina,errsys0,errsys1,fget

MOD getenv,gets0,gets1,gets2,gx,joinlist,lupdate,memory

MOD mrelease,msave,net,oldbox,oldclear,philes,run,saverest

MOD sdf0,sdf1,seq,setcurs,sortbloc,startsym,tb,txopen,vall

MOD vblock,vdb,vdbg,version,vmacro,vnone,vops,vpict,vterm

MOD workarea,xmacro

ALLOCATE extend

ENDAREA

LIB clipper, terminal ,dbfntx

There are a standard set of link-scripts included with Blinker that make it very simple to overlay the Clipper 5.2 libraries. They are as follows:

CL520MIN.LNK - Used to overlay the smallest portion of the Clipper libraries for maximum speed performance but minimum memory performance.

CL520MID.LNK - Used to overlay a moderate portion of the Clipper libraries for moderate speed performance and moderate memory performance..

CL520MAX.LNK - Used to overlay the maximum allowable portion of the Clipper libraries for minimum speed performance by maximum memory performance.

These are sub link script files that can be simply included in your master link script file like so:

BLINKER OVERLAY OPSIZE 50

BLINKER PROCEDURE DEPTH 50

FI

LIB

OUTPUT

@CL520MAX.LNK

Blinker has become the standard for linking Clipper applications, so much of the third-party community also will include "ready-to-go" link-scripts with their libraries to make it easy to add their library to your application. For example, if you are already using a Blinker link script and decide to add the Six Driver (RDD) to your application, you would simply add one line of code to your link-script:

Linking Techniques for Memory Management (Part 1) by Roger Donnay

This section written by Roger Donnay is the third part of CA-Clipper 5.x Memory Management by Roger Donnay & Jud Cole. If you like to have the whole book in .WRI format, click here to visit Roger's Site

The Role of Linkers in developing large applications has taken a new turn in the past few years. In addition to providing segment-linking, they are now also expected to provide the best "memory image" possible. This means that the linker must be sophisticated in it's management of code segments during the running of an application. The physical size of a CA-Clipper application is no longer representative of it's memory image, because the linker "overlays" the code segments to use the smallest possible amount of memory. Some type of code (such as CA-Clipper P-CODE) is easy to overlay while other types (ASM) may be more difficult. Applications that are almost all CA-Clipper-compiled code will usually produce the best memory image regardless of the physical size of the .EXE.

The subjects of this book are the four most commonly-used linkers available on the market today for linking CA-Clipper 5.2 and 5.3 applications. They are:

RTLINK - A "Real Mode" CA-Clipper-5.2 specific linker developed by Pocketsoft, Inc. This is the linker that is bundled with CA-Clipper 5.2.

BLINKER - A "Real Mode" Third-Party linker published by Blink, Inc. This is the linker that is bundled with CA-Clipper 5.3. A "Dual-Mode" version of Blinker is also available from Blink, Inc.

EXOSPACE - A "Protected Mode" linker developed by CA and sold as an add-on to Clipper 5.2. This linker is bundled with Clipper 5.3.

CAUSEWAY - A third-party "Protected Mode" linker developed by Michael Devore for use with Clipper 5.2 and 5.3.

The CA-Clipper compiler has the responsiblity of converting source code to "machine-readable" code. This .OBJect code can not be "executed" by the machine however until it is "linked" to other .OBJect code segments in the CA-Clipper libraries. This is the responsibility of the linker. Linking or "fixing up" references made by your code to external variables and functions is necessary to insure that your compiled program will address or call other programs, including functions in the CA-Clipper libraries and DOS functions.

A linker's basic job is to combine objects which have been compiled separately. Objects contain symbols of three types:

a. PUBLIC symbols - those symbols which are callable from other objects

b. STATIC symbols - those symbols which are callable only from within the same object

c. UNDEFINED symbols - symbols which exist in other objects and are referenced or used by this object ((also referred to as EXTERNAL symbols).

DYNAMIC Overlaying of CA-Clipper Code

Very large CA-Clipper applications can consist of more code than can be loaded into the available amount of DOS memory and may not be able to execute due to insufficient memory. CA-Clipper 5.x has a "Dynamic Overlay Manager" actually built into the CA-Clipper libraries that helps prevent this problem by loading the code into memory only as it is needed by the application that is running. The linker helps prepare the application to use this capability of CA-Clipper 5.2 by splitting the CA-Clipper-compiled application code into fixed-size "pages" that share a pre-allocated area of memory. These pages of code are then written to a seperate .OVL file or to the end of the .EXE file as determined by your linker options. When a procedure or function is called, the "dynamic-overlay manager" first checks to see if it is in memory. If it is, then it jumps to the called function. If it is not in memory, it loads the function from disk into any "free" location available in the overlay pool. If no free space can be found in the overlay pool, the overlay manager discards pages that have not been called recently to free space to load the called procedure.

The larger the overlay pool, the less often the overlay manager needs to load from disk and the faster your program will run. The size of the dynamic overlay pool needed for your application is automatically determined by CA-Clipper.

The great thing about CA-Clipper 5.x's dynamic overlaying system is that you, the programmer, don't need to do anything to create overlays. It is all done automatically by the linker. Only CA-Clipper-compiled modules, not C/ASM modules, are overlayed "dynamically", although since much of the CA-Clipper libraries has been compiled in CA-Clipper, even large portions of the CA-Clipper libraries will be dynamically overlayed.

Application libraries and third-party libraries written in CA-Clipper will also be automatically overlayed by RTLINK and BLINKER. In many cases it is more beneficial to write functions in CA-Clipper (using LOCAL memvars) than in C or ASM because it will not increase the executable memory model. Even CA-Clipper code that has been linked into to Pre-link libraries will be dynamically overlayed.

INTERNAL/EXTERNAL Overlays

INTERNAL overlays are dynamic pages which are written to the end of the .EXEcutable program file. During the running of the application, the dynamic overlay manager reads the disk and loads dynamic pages from this file. This is the default overlay method of RTLINK and BLINKER and it is often advantageous to use internal overlays to reduce the number of file handles being opened by your application. In some cases however, you may find that using multiple overlay files will increase the speed performance of your application because extracting code from several small files which are all opened at the same time is often faster that extracting code from one large file.

EXTERNAL overlays are sets of dynamic pages which are written to a seperately designated file more commonly referred to as an overlay file. The positional command /DYNAMIC[:] or the freeformat command DYNAMIC INTO will tell RTLINK to write the dynamic pages to a file named . You may wish to create a seperate overlay file when your application .EXE is too large to fit on a distribution disk.

Examples:

Rtlink> DYNAMIC INTO myprog.ovl Blinker> SECTION INTO myprog.ovl

Rtlink> DYNAMIC INTO myprogs.ovl MYPROG1, MYPROG2 DYNAMIC INTO myfuncs.ovl MYFUNC1, MYFUNC2 Blinker> SECTION INTO myprogs.ovl MYPROG1, MYPROG2 SECTION INTO myfuncs.ovl MYFUNC1, MYFUNC2

DYNAMIC Overlaying of C / ASM Code

BLINKER also provides the ability to dynamically overlay code written in C or Assembly. This is the most common code found in many third-party libraries. RTLINK also provides for overlaying of C/ASM but it is more complex and requires understanding more about the code being overlayed. See STATIC OVERLAYS and RELOADABLE OVERLAYS for more information on overlaying C/ASM code with RTLINK.

In the previous section, we discussed that application code compiled with the CA-Clipper compiler is dynamically overlayed using the overlay manager built inside the CA-Clipper libraries. This CA-Clipper-specific internal overlay manager is incapable of overlaying C/ASM code, so it is the responsibility of the linker to provide this capability. Each linker has it's own methods for overlaying this kind of code. BLINKER uses an overlay pool that is allocated at the start of the application. The size of this pool is determined at link-time. The larger the overlay pool, the faster the application will run because overlays will not be reloaded into memory from disk so often.

The dynamic-overlay capabilities of BLINKER require that the objects being overlayed are "well-behaved". After much experimentation with our own libraries and many third-party libraries, it is now more clear what constitutes well-behaved modules:

a. Well-behaved code uses the CA-Clipper EXTEND API registers or the stack for parameter passing and does not use undocumented features of CA-Clipper.

b. All CA-Clipper-compiled .OBJects are dynamically-overlayable.

Modules which CANNOT be dynamically-overlaid are:

a. Routines which handle interrupts.

b. Modules in which the DATA area is changed during runtime rather than allocating memory in the root area.

Dynamic overlaying of third-party libraries can be an aggravating learning experience if you don't get support from the third party vendor in trying to accomplish this task. Most of the third-party vendors now break their libraries into separate overlayable and non-overlayable .LIB files to make the task much easier.

Once you have determined which .OBJ files and/or .LIB files are acceptable for overlaying, the task of telling the linker to do this is quite simple. A BLINKER link-script will look similar to an RTLINK script except there is always only one (1) overlay area because all code uses the same overlay "pool". Example of a BLINKER script for overlaying C/ASM:

# Allocate 50k to the overlay pool

BLINKER OVERLAY OPSIZE 50

BEGINAREA

# CA-Clipper-compiled application code

FILE myprog1.obj, myprog2.obj

# C/ASM files

FILE ccode.obj, asmcode.obj

# C/ASM libraries

LIB ccode.lib, asmcode.lib

ENDAREA

Using EXPANDED or UMB RAM for OVERLAYS

BLINKER provides a several very useful feature for increasing the amount of available memory for CA-Clipper applications, i.e., the use of Expanded (EMS) or Upper Memory Blocks (UMB).

EMS Memory

Many computers have EMS memory. EMS memory managers such as QEMM, 386MAX, and DOS's own EMM386 create a memory area above conventional memory referred to as the "page frame". This 64k block of memory is mapped to an area of ram that can execute 8086 code. Blinker takes advantage of this feature by allocating the 64k EMS page frame rather than conventional DOS memory for their runtime overlay pool. This frees an additional 64k of memory for use by CA-Clipper because the overlays are loaded into the page frame rather than conventional ram. If no EMS is present when the application starts, then the overlay pool will be established in normal (conventional) ram. To enable this feature, use the command:

BLINKER OVERLAY PAGEFRAME ON

UMB Memory

Expanded memory managers will also create an area between the 640k and 1 meg memory address called UMB (upper memory blocks). This is the area where you would normally load your memory-resident programs such as DOS, network-drivers, mouse-drivers, etc. If your enviroment gives you about 30k - 60k of contiguous memory available in the UMB area, Blinker will load the overlay pool into the UMB area rather than conventional area, thus freeing more conventional memory for your CA-Clipper application. To enable this feature, use the Blinker command:

BLINKER OVERLAY UMB ON

Note: In today's modern environments, this UMB area is usually consumed by so many TSR's that the possiblity of finding a contiguous block larger than 30k is virtually impossible, so it recommended that you do not rely on the availability of this memory.

STATIC Overlays

STATIC overlays are supported by RTLINK only. They provide a method of overlaying C and ASM code which cannot be overlayed by the dynamic-overlay manager. C and ASM code is not as easily relocatable as CA-Clipper compiled PCODE (pseudo-code) therefore it cannot be overlayed "dynamically". A function which is dynamically loaded and unloaded from a pool is usually found in different places in memory at different times during the running of the program. When a C or assembly routine is called by an application linked with RTLINK it must be loaded in memory at the same address every time. This is accomplished by designating overlay AREAS where many different modules will occupy the same memory space, but only one-at-a-time. Designing Static overlays requires a more thorough understanding of the structure of your program. Static overlay segments usually look like this in your link file:

# area 1

BEGINAREA

SECTION FILE A

SECTION FILE B

SECTION FILE C

ENDAREA

# area 2

BEGINAREA

SECTION FILE D

SECTION FILE E

SECTION FILE F

ENDAREA

You must be sure that you properly place your objects in the overlay segments to prevent computer "lock-up" or "crashing" at runtime. In the above example, any procedure or function in FILE A may call any procedure or function in FILE D, E or F because they are in different areas, however if a function in FILE A calls a procedure or function in FILE B, then your program will crash as soon as the program returns to FILE A because FILE A was removed from memory to load FILE B.

Overlay management with "static overlays" is the most time-consuming because it requires extensive analysis of the structure of the program to insure that the modules are placed properly in the overlay areas.

RELOADABLE Overlays

The following overlaying technique should be applied at your own risk because it is an "unsupported" and "undocumented" feature of the RTLINK linker supplied with CA-Clipper-5.2. I feel however, that it is important to cover this subject because without an understanding of how to use this feature it may be impossible to develop certain types of CA-Clipper applications without resorting to a third-party linker.

I started RELOADING modules from the CA-Clipper libraries years ago using RTLINK and the Summer 87 version of Clipper, so when I received my copy of CA-Clipper-5.2, the first thing I attempted to do was to overlay some of the larger C/ASM modules in the CA-Clipper libraries using the RELOAD command in my linker script file. The CA-Clipper development team had the foresight to insure that their C and ASM compiler assigned a UNIQUE name to each CA-Clipper C/ASM object in the CA-Clipper libraries, thus allowing, you, the CA-Clipper programmer to use both the RELOAD command and MODULE command in your .LNK script files.

So what are "Reloadable Overlays"? These are a form of STATIC overlay in which segments of code are placed into overlay sections that occupy the same memory space at runtime, however the "calling module" is automatically "reloaded" into memory when returning from the "called module".

Reloadable overlay segments usually look like this in your link file:

RELOAD FAR 200

# area 1

BEGINAREA

SECTION FILE A

SECTION FILE B

SECTION FILE C

ENDAREA

# area 2

BEGINAREA

SECTION FILE D

SECTION FILE E

SECTION FILE F

ENDAREA

Not all CA-Clipper applications are going to need to use these reloadable static overlays, but if your application relies heavily on third-party libraries written in C/ASM, then it is recommended that you learn this technique.

The main advantage of "reloadable overlays" over conventional "static overlays" is that you are never in danger of "lockup" in the event that a procedure or function in FILE A calls a function or procedure in FILE B because FILE A will be reloaded into memory on return from FILE B. Make sure when you use the RELOAD command that you always use it as follows:

RELOAD FAR

where the is the amount of memory in hex bytes to use for saving addresses. My experience is that most applications will run just fine with a stack size of 200. If your application does recursive nesting, however, you may need to increase the stack size. Recursive nesting is described as follows: Procedure A calls Procedure B which calls Procedure A which calls Procedure B, etc, etc, etc. Each time a procedure is called, its return address is "pushed" onto the stack, and each time you return to the calling procedure the address is "popped" from the stack. In the above example, if this recursive condition were allowed to continue for a large number of iterations, the stack would be overrun and the program may crash.

Although your application will not crash using RELOAD, it may slow down a bit if overlay segments are not structured properly, because each time an overlay is reloaded the application must go to disk. In the above example if a function in FILE A repetitively calls a function in FILE B then your application will be very "disk intensive" and will run slowly. If a function in FILE A repetitively calls a function in FILE D there will be no slowing at all because both modules will remain in memory.

Overlay management of C/ASM code with "reloadable overlays" is less time-consuming and more reliable than simple conventional "static overlays" and provides an additional advantage of allowing more modules to be overlayed, thereby saving additional memory usage.

Which RDD to use ? - DBFCDX versus DBFNTX

CA-Clipper 5.2e/5.3 comes with the following replaceable database drivers or RDDs

DBFNTX - Native Clipper

DBFCDX - Foxpro 2 compatible

DBFMDX - DBase IV compatible

DBFNDX - dBase III plus compatible

DBFPX - Paradox compatible

You should note that the DBFCDX driver that comes with 5.2e is the Successware SIX driver while 5.3/5.3a comes with the Lodestone COMIX driver.

The most popular RDDs as can be seen from posts on comp.lang.clipper are DBFCDX & DBFNTX. As for me, I use both drivers as appropriate. Therefore I shall limit this section to the DBFCDX & DBFNTX drivers.

The following information is extracted from NGs that came with CA-Clipper 5.2e

• RDD Architecture
• DBFNTX - CA-Clipper Native RDD
• DBFCDX - Foxpro 2.0 Compatible RDD

CA-Clipper's RDD Architecture

CA-Clipper supports a driver architecture that allows CA-Clipper applications to use Replaceable Database Drivers (RDDs). The RDD system makes CA-Clipper applications data-format independent. Such applications can, therefore, access the data formats of other database systems, including the dBASE IV (.mdx), FoxPro (.cdx), and Paradox (.db) formats on a variety of equipment. This driver architecture can even support database drivers that are not file-based, although all of the drivers supplied with CA-Clipper 5.2 are file-based.

The concept of replaceable drivers is not new to this version of CA-Clipper. In previous versions, the use of the default database driver (DBFNTX.LIB) was hidden by the fact that it was automatically linked into your application. In fact, this is still the case. The DBFNTX driver has been replaceable since it was first introduced in version 5.0. Before this version, the DBFNTX driver was the only RDD supplied as part of the system.

With the introduction of the new RDDs, CA-Clipper provides many new and enhanced commands and functions that access and manipulate databases. These language elements can enable your applications to access data regardless of the RDD under which it is ordered. There are also commands and functions that give you specific information about the RDDs in use.

DBFNTX - CA-Clipper Native RDD

DBFNTX is the default RDD for CA-Clipper. This new database driver replaces the DBFNTX database driver supplied with earlier versions of CA-Clipper and adds a number of new indexing features. With DBFNTX, you can:

Create conditional indexes by specifying a FOR condition

Create indexes using a record scope or WHILE condition, allowing you to INDEX based on the order of another index

Create both ascending and descending order indexes

Specify an expression that is evaluated periodically during indexing in order to display an index progress indicator

As an update of the default database driver, DBFNTX is linked into and used automatically by your application unless you compile using the /R option.

New Features

The replaceable driver lets you create and maintain (.ntx) files using features above and beyond those supplied with the previous DBFNTX driver. The new indexing features are supplied in the form of several syntactical additions to the INDEX and REINDEX commands. Specifically you can:

Specify full record scoping and conditional filtering using the standard ALL, FOR, WHILE, NEXT, REST, and RECORD clauses

Create an index while another controlling index is still active

Monitor indexing as each record (or a specified record number interval) is processed using the EVAL and EVERY clauses

Eliminate separate coding for descending order keys using the DESCENDING clause

Compatibility

Index files (.ntx) created with the original DBFNTX driver are compatible with DBFNTX and can be used in new applications without reindexing. Index files (.ntx) created with this version of DBFNTX will also work with previous CA-Clipper applications provided that you use no FOR, WHILE, , or DESCENDING clauses.

Important! Indexes produced with DBFNTX using FOR or DESCENDING are incompatible with earlier version (.ntx) files. If you attempt to access them with the original DBFNTX database driver or programs compiled with versions earlier than CA-Clipper 5.2, you will get an unrecoverable runtime error. In CA-Clipper, this generates an "index corrupted" error message, causing the application to terminate.

New Locking Scheme

The DBFNTX database driver implements a new locking scheme to resolve several problems identified in previous versions of CA-Clipper and to prevent potential problems that might arise when running CA-Clipper applications in a network environment. This section discusses these changes and their implications, including compatibility issues.

Lock Time-outs

Problem: Index locking in previous versions of CA-Clipper was handled automatically by the database driver, and had no time-out provision. This created the potential for problems in network environments if a workstation died while holding a lock. If this situation occurred all other workstations waiting for an index lock would appear to freeze while waiting to obtain their lock. This could also happen if a user placed a CA-Clipper application in the background on a multitasking system without sufficient processing time allocated to it. Eventually, most network operation systems would clear a connection that had no activity for a specified period of time. This would free the lock and everything would resume as normal, but frustrated users may have rebooted their machines possibly causing file corruption.

Solution: In CA-Clipper 5.2 the NTX driver will generate a recoverable runtime error if it fails to lock the index after a predetermined number of retries. The default error handler for this system simply returns (.T.) to retry the operation. This emulates the behavior of previous CA-Clipper versions.

Error Handling

Time out handling: The handling of this error is problematic because the lock is issued from various internal index routines. Therefore the only safe recoveries are to retry or quit. Choosing to default from the error or issuing a break will more than likely leave the index in a corrupted state. If either of the options is employed, the application should immediately recreate the index. The preferred way to handle a time out such as this is to alert the user of the situation so they don't think their machine has hung, and then have the network administrator determine what workstation is causing the problem. When the problem workstation is cleared, the users that have timed out can select retry and continue processing.

NTXERR.PRG: The file NTXERR.PRG contains the source code for the default error handler INIT procedure. This error handler can be modified to allow user-defined error handling for index lock time-outs. Care should be exercised when modifying the error handler as detailed above.

Compatibility: The lock time-out capability when used in conjunction with the default error handler is totally compatibility with previous versions of CA-Clipper. No changes are made to the NTX file structure and no action is required by the developer to activate the time-out functionality.

New Lock Offset

Problem: Index locking, which is transparent to the developer, uses a single-byte semaphore locking system. This semaphore was placed at a virtual offset (beyond the physical end of file) in the index file. In previous versions of CA-Clipper, this offset was located at one billion (1,000,000,000) which was adequate at the time. But many systems today are capable of producing indexes that are large enough to cause the actual data present at the lock offset to become physically locked. This leads to problems when trying to read or write to the data at that offset.

Solution: The solution is to move the offset where locking occurs to a location at a greater offset. We have chosen FFFFFFFF hex, which is the largest offset possible under the DOS operating system. The problem with this solution is that new applications using the index will be locking this new byte while old applications using the same index will lock the old position. Clearly this would cause both applications to fail because each could have a lock on the file at the same time.

To avoid this, the signature of the index (in the index header) is modified to prevent pre-CA-Clipper 5.2 applications from being able to open the index. CA-Clipper 5.2 applications can detect the correct offset to use by the flag in the header and will automatically use the correct one. In Figure 7-1 below, each bit represents a flag:

BIT 7 6 5 4 3 2 1 0 FLAG R R R O P I I C

R Reserved I Index type - both bits set (NTX) C Index created with a Condition, condition in header T Created as a Temporary index O New Offset for exclusive (semaphore) lock

Figure 7-1: Bit Field for the Signature Byte of a CA-Clipper 5.2 NTX File

Activation

If CA-Clipper 5.2 automatically modified the signature in the header when it created indexes, programs with automatic reindexing routines would be creating indexes that appeared corrupt to pre-CA-Clipper 5.2 applications. This has an obvious problem with backward compatibility. Therefore, in order to create indexes with the new signature, the developer must link in the module NTXLOCK2.OBJ with the full knowledge that this will create indexes that older applications will not be able to access.

Header Changes

The signature byte of a .NTX file is 6 for an unenhanced NTX index. The inclusion of the NTXLOCK2.OBJ will cause the signature to become 26 hex. (6 hex ORed with 20 hex). See Figure 7-1 for an illustration of all the possible values for the signature byte.

Error Handling

CA-Clipper 5.2 applications will automatically recognize the signature byte of the header, and depending on the signature value, will use the correct index lock location. Applications built with previous versions of CA-Clipper, however, do not have the capability to detect the optional new information in the signature byte. Therefore, when an order application tries to open a file that has been created with the NTXLOCK2.OBJ linked in it will produce a Corruption Detected error.

Compatibility

The new locking location, if used, is not backward compatible with applications compiled with previous versions of CA-Clipper.

Indexes created by applications built with a previous version of CA-Clipper can be used by CA-Clipper 5.2 using the new location and will not be modified unless the index is recreated in application.

Since older applications have no knowledge of the new index locking scheme nor of the significance of the header signature, these applications will assume the index is corrupt and will produce an Index Corrupted error.

Conditional Indexing

Conditional indexes are a feature of the DBFNTX driver. This section discusses this feature of the DBFNTX driver in some detail, giving you specific information about the implementation of conditional indexes. Compatibility issues are also discussed.

Conditional indexes are produced by using a FOR condition in the index creation process. These indexes are made fully maintainable by storing the FOR condition in the index header. This condition is subsequently retrieved and compiled each time the index in opened. During updates, items are added to the index only if they meet the criteria of the condition.

Since older applications do not have the ability to recognize and use the condition stored in the header, they must be prevented from opening the index since they corrupt the index. This is accomplished by modifying the signature of the index (in the index header) preventing pre-CA-Clipper 5.2 applications from being able to open the index. CA-Clipper 5.2 applications can detect the flag in the header and will automatically use the stored FOR condition correctly.

Temporary Indexes

Temporary indexes are produced by using any scoping clause other than the FOR condition in the index creation process. These indexes are not automatically maintainable because the condition is not stored for later use. These indexes can be made maintainable if the condition can be expressed as a FOR condition and is added using the FOR clause. But the main use of temporary indexes is for fast creation of indexes for read- only browses or reports that operate on a subset of the database.

Since older applications would not operate properly with indexes that do not contain all the keys in a given database, they must be prevented from using them. This is accomplished by modifying the index signature to prevent pre-CA-Clipper 5.2 applications from being able to open the index.

EOF()

If an index is created with a FOR condition and an attempt is made to update the index with a key that does not match the condition, the update is suppressed and the index is placed at EOF(). This is consistent with the current behavior for indexes created with the unique flag when an update is attempted with a non-unique key.

Also if a navigational action is attempted (SKIP) and the current record is not found in the index, the index will place the record pointer at EOF(). This is true for both conditional and temporary indexes.

Installing DBFNTX Driver Files

DBFNTX is supplied as the file DBFNTX.LIB.

The CA-Clipper installation program installs this driver as the default in the \CLIPPER5\LIB subdirectory on the drive that you specify, so you need not install the driver manually.

Important! Before installing CA-Clipper, you may want to rename the DBFNTX.LIB that currently resides in your \CLIPPER5\LIB directory to DBFNTX.001. The new version, when installed, will overwrite DBFNTX.LIB. If you do not rename or otherwise protect the old version of DBFNTX.LIB, you will lose it.

Linking the DBFNTX Database Driver

Since DBFNTX is the default database driver for CA-Clipper, there are no special instructions for linking. Unless you specify the /R option when you compile, the new driver will be linked into each program automatically if you specify a USE command or DBUSEAREA() function without an explicit request for another database driver. The driver is also linked if you specify an INDEX or REINDEX command with any of the new features.

Using the DBFNTX Database Driver

In applications written for the new DBFNTX driver, you can use the INDEX and REINDEX commands exactly as you have used them in the past. The index files (.ntx) you create and maintain in this way are completely compatible with those created using previous versions of the driver.

Changes to existing code are necessary only if you use the new indexing features. The (.ntx) files you create using the new features will have a slightly different header file and cannot be used by programs linked with a previous version of the driver.

Using (.ntx) and (.ndx) Files Concurrently

You can use (.ntx) and (.ndx) files concurrently in a CA-Clipper program like this:

// (.ntx) file using default DBFNTX driver

USE File1 INDEX File1 NEW

// (.ndx) files using DBFNDX driver

USE File2 VIA "DBFNDX" INDEX File2 NEW

Note, however, that you cannot use (.ntx) and (.ndx) files in the same work area. For example, the following does not work:

USE File1 VIA "DBFNDX" INDEX File1.ntx, File2.ndx

Compatibility with dBASE III PLUS

The default DBFNTX driver makes CA-Clipper programs behave differently than traditional dBASE programs. Some of these differences are discussed below.

Supported Data Types

The DBFNTX database driver supports the following dBASE III PLUS- compatible data types for key expressions:

Character

Numeric

Date

Logical

Supported Key Expressions

When you create (.ntx) files using the DBFNTX driver, you can use all CA-Clipper or user-defined functions compatible with dBASE III PLUS as well as other functions accepted by the extended CA-Clipper functionality.

Error Handling

The indexing behavior of DBFNTX and DBFNDX in a CA-Clipper application is identical unless otherwise noted. With the default DBFNTX driver, you can handle most errors using BEGIN SEQUENCE...END SEQUENCE as illustrated in the next section.

FIND vs SEEK

In CA-Clipper, you can use the FIND command only to locate keys in indexes where the index key expression is character data type. This differs from dBASE III PLUS where FIND supports character and numeric key values.

Note: In CA-Clipper programs, always use the SEEK command or the DBSEEK() function to search an index for a key value.

The DBFNTX driver lets you recover from data type errors raised during a FIND or SEEK. However, since Error:canDefault, Error:canRetry or Error:canSubstitute are set to false (.F.), you should use BEGIN SEQUENCE...END to handle such SEEK or FIND data type errors. Within the error block for the current operation, issue a BREAK() using the error object that the DBFNTX database driver generates, like this:

bOld := ERRORBLOCK({|oError| BREAK(oError)}) . . . BEGIN SEQUENCE SEEK xVar RECOVER USING oError // Recovery code END . . .

ERRORBLOCK(bOld)

Sharing Data on a Network

The DBFNTX driver provides file and record locking schemes that are different from dBASE III PLUS schemes. This means that if the same database and index files are open in CA-Clipper and in dBASE III PLUS, CA-Clipper program locks are not visible to dBASE III PLUS and vice versa.

Warning! Database integrity is not guaranteed and index corruption will occur if CA-Clipper and dBASE III PLUS programs attempt to write to a database or index file at the same time. Therefore, concurrent use of the same database (.dbf) and index (.ndx) files by dBASE III PLUS and CA-Clipper programs is strongly discouraged and not supported by Computer Associates.

DBFCDX - Foxpro 2.0 Compatible RDD

DBFCDX is the FoxPro 2 compatible RDD for CA-Clipper. As such, it connects to the low-level database management subsystem in the CA-Clipper architecture. When you use the DBFCDX RDD, you add a number of new features including:

FoxPro 2 file format compatibility

Compact indexes

Compound indexes

Conditional indexes

Memo files smaller than DBFNTX format

Overview of the DBFCDX RDD

The DBFCDX driver lets you create and maintain (.cdx) and (.idx) files with features different from those supplied with the original DBFNTX driver and is compatible with files created under FoxPro 2. The new features are supplied in the form of several syntactical additions to the INDEX and REINDEX commands. Specifically, you can:

Create indexes smaller than those created with the DBFNTX driver. The key data is stored in a compressed format that substantially reduces the size of the index file.

Create a compound index file that contains multiple indexes (TAGs), making it possible to open several indexes under one file handle. A single (.cdx) file may contain up to 99 index keys.

Create conditional indexes (FOR / WHILE / REST / NEXT).

Create files with FoxPro 2 file format compatibility.

Compact Indexes

Like FoxPro 2, The DBFCDX driver creates compact indexes. This means that the key data is stored in a compressed format, resulting in a substantial size reduction in the index file. Compact indexes store only the actual data for the index keys. Trailing blanks and duplicate bytes between keys are stored in one or two bytes. This allows considerable space savings in indexes with much empty space and similar keys. Since the amount of compression is dependent on many variables, including the number of unique keys in an index, the exact amount of compression is impossible to predetermine.

Compound Indexes

A compound index is an index file that contains multiple indexes (called tags). Compound indexes (.cdx)'s make several indexes available to your application while only using one file handle. Therefore, you can overcome the CA-Clipper index file limit of 15. A compound index can have as many as 99 tags, but the practical limit is around 50. Once you open a compound index, all the tags in the file are automatically updated as the records are changed.

Once you open a compound index, all the tags contained in the file are automatically updated as the records are changed. A tag in a compound index is essentially identical to an individual index (.idx) and supports all the same features. The first tag (in order of creation) in the compound index is, by default, the controlling index.

Conditional Indexes

The DBFCDX driver can create indexes with a built-in FOR clause. These are conditional indexes in which the condition can be any expression, including a user-defined function. As the database is updated, only records that match the index condition are added to the index, and records that satisfied the condition before, but don't any longer, are automatically removed.

Expanded control over conditional indexing is supported with the revised INDEX and REINDEX command options as in the new DBFNTX driver.

Installing DBFCDX Driver Files

The DBFCDX driver is supplied as the file, DBFCDX.LIB.

The CA-Clipper installation program installs this driver in the \CLIPPER5\LIB subdirectory on the drive that you specify, so you need not install the driver manually.

Linking the DBFCDX Database Driver

To link the DBFCDX database driver into an application program, you must specify DBFCDX.LIB to the linker in addition to your application object files (.OBJ).

1. To link with .RTLink using positional syntax:

C>RTLINK ,,,DBFCDX

2. To link with .RTLink using freeformat syntax:

C>RTLINK FI LIB DBFCDX

Note: These link commands all assume the LIB, OBJ, and PLL environment variables are set to the standard locations. They also assume that the CA-Clipper programs were compiled without the /R option.

Using the DBFCDX Database Driver

To use FoxPro 2 files in a CA-Clipper program:

1. Place REQUEST DBFCDX at the beginning of your application or at the top of the first program file (.prg) that opens a database file using the DBFCDX driver.

2. Specify the VIA "DBFCDX" clause if you open the database file with the USE command.

-OR-

3. Specify "DBFCDX" for the argument if you open the database file with the DBUSEAREA() function.

-OR-

4. Use RDDSETDEFAULT( "DBFCDX" ) to set the default driver to DBFCDX.

Except in the case of REQUEST, the RDD name must be a literal character string or a variable. In all cases it is important that the driver name be spelled correctly.

The following program fragments illustrate:

REQUEST DBFCDX . . . USE Customers INDEX Name, Address NEW VIA "DBFCDX"

-OR-

REQUEST DBFCDX RDDSETDEFAULT( "DBFCDX" ) . . . USE Customers INDEX Name, Address NEW

Using (.idx) and (.ntx) Files Concurrently

You can use both (.idx) and (.ntx) files concurrently in a CA-Clipper program like this:

// (.ntx) file using default DBFNTX driver

USE File1 INDEX File1 NEW

// (.idx) files using DBFCDX driver

USE File2 VIA "DBFCDX" INDEX File2 NEW

Note, however, that you cannot use (.idx) and (.ntx) files in the same work area. For example, the following does not work:

USE File1 VIA "DBFNTX" INDEX File1.ntx, File2.idx

Using (.cdx) and (.idx) Files Concurrently

You may use (.cdx) with (.idx) files concurrently (even in the same work area); however, in most cases it is easier to use a single (.cdx) index for each database file or separate (.idx) files. When using both types of index at the same time, attempting to select an Order based on its Order Number can be confusing and will become difficult to maintain.

File Maintenance under DBFCDX

When an existing tag in a compound index (.cdx) is rebuilt using INDEX ON...TAG... the space used by the original tag is not automatically reclaimed. Instead, the new tag is added to the end of the file, increasing file size.

You can use the REINDEX command to "pack" the index file. REINDEX rebuilds each tag, eliminating any unused space in the file.

If you rebuild your indexes on a regular basis, you should either delete your (.cdx) files before rebuilding the tags or use the REINDEX command to rebuild them instead.

DBFCDX and Memo Files

The DBFCDX driver uses FoxPro compatible memo (.fpt) files to store data for memo fields. These memo files have a default block size of 64 bytes rather than the 512 byte default for (.dbt) files.

DBFCDX memo files can store any type of data. While (.dbt) files use an end of file marker (ASCII 26) at the end of a memo entry, (.fpt) files store the length of the entry. This not only eliminates the problems normally encountered with storing binary data in a memo field but also speeds up memo field access since the data need not be scanned to determine the length.

Tips For Using DBFCDX

1. Make sure index extensions aren't hard-coded in your application. The default extension for DBFCDX indexes is (.idx), not (.ntx). You can still use (.ntx) as the extension as long as you specify the extension when you create your indexes. The best way to determine index extensions in an application is to call ORDBAGEXT().

For example, if you currently use the following code to determine the existence of an index file:

IF .NOT. FILE("index.ntx") INDEX ON field TO index ENDIF

Change the code to include the INDEXEXT() function, as follows:

IF .NOT. FILE("index"+ORDBAGEXT()) INDEX ON field TO index ENDIF

2. If your application uses memo fields, you should convert your (.dbt) files to (.fpt) files.

There are some good reasons for using (.fpt) files. Most important is the smaller block size (64 bytes). CA-Clipper's (.dbt) files use a fixed block size of 512 bytes which means that every time you store even 1 byte in a memo field CA-Clipper uses 512 bytes to store it. If the data in a memo field grows to 513 bytes, then two blocks are required.

When creating (.fpt) files, the block size is set at 64 bytes to optimize it for your needs. A simple conversion from (.dbt) files to (.fpt) files will generally shrink your memo files by approximately 30%.

3. Add DBFCDX.LIB as a library to your link command or link script.

Codeblocks - by Tom Leylan - 30 June 1996

This article on code blocks is publish here with the permission of Tom Leylan (he has moved to New York) on 30 June 1996.

It is rather old (I wrote it for The Aquarium prior to the 5.0 release) but somewhat applicative today. It may be hard to find but look fora copy of my book. It shows how to write Clipper applications and demonstrates a method which emphasizes writing the least amount of code to obtain the most flexible, reusable solutions.

They're mysterious they're fun they're often misunderstood they're code blocks

by Tom Leylan

Clipper 5.0 is almost here and with it comes a zillion or so new things to learn. Among them something called a "code block", but before we can discuss what code blocks do we need to agree on what code blocks are.

What Code Blocks Are

Code blocks are a new datatype. This is important, they are not procedures, they are not functions, they are not objects, they are a new datatype and they act like data. They happen to contain compiled Clipper code. Nantucket describes them as "assignable unnamed functions".

Permit me a momentary digression.

We accept the fact that characters are a datatype and we likely agree that a bunch of characters in a row (called a character string) is a datatype, perhaps as Clipper programmers we don't think of the two as particularly different.

There is no doubt that we consider numbers a datatype and in the Clipper / dBase world we are led to believe that dates and logical variables are unique datatypes also.

One might suspect that defining datatypes is something best left to Computer Science types or that there is some listing of acceptible datatypes in a old binder somewhere and that everyone agrees that this is the sum total of them. Perhaps some people think that datatypes are somehow tightly connected to the design of computers and that they need to be particularly discreet little chunks of stuff easily represented on the screen.

None of this could be further from the truth. Characters, numbers, dates and logicals were chosen because they represented things that we all find useful. Datatypes could easily be rendered for telephone numbers and zipcodes though one could argue that they are just a patterned collection of numbers... but then so are dates.

Inventing a datatype requires only the definition of acceptible values and one or more operations that can be carried out on that datatype.

We can "add" numbers and we can "add" strings but the results are considerably different. We can "divide" two numbers but we cannot divide two strings. One the other hand we can convert a string to upper case but that operation is meaningless when applied to numbers.

Code blocks are essentially data, the data is executable code, but because they act like data it doesn't mean that you can automatically add or subtract them or convert them to upper case. Code blocks have their own set of operations appropriate to code blocks.

Operations On Code Blocks

Variables can be initialized to specific datatypes and can be assigned values of that datatype.

sVar := "this is a string" // string

nVar := 5 // numeric

dVar := date() // date

tVar := .T. // logical

aVar := { 1, 2, 3, 4 } // array

It turns out that we can initialize and assign code blocks also.

bVar := {|| Qout("A Hello World Code Block")}

Notice that a code block is contained within curly braces ({}) the same characters used to contain arrays. To distinguish them from arrays, code blocks require the inclusion of a set of two vertical bars (||). These bars also serve to delimit the formal parameters but more on that later.

We can "add" numbers, "concatenate" strings, "increment" dates, "invert" logicals, "sort" arrays and we can "execute" code blocks.

Using the proper terminology we can "evaluate" code blocks and the way we do it is through the EVAL() function or through a number of other functions that accept a code block as a parameter.

Notice that each datatype has some operation that is unique. This is quite probably the reason that we find any particular datatype worthwhile. If, for instance I could add the strings "123" + "456" and obtain an answer of "579", the need for a numeric datatype would be lessened.

Code blocks just like all other datatypes can be passed as arguments to and can be returned from functions. In their ASCII representation they can saved to a file on the disk or stored in a .DBF field.

Evaluating Code Blocks

Let's EVAL() our first code block example.

cls

bVar := {|| Qout("A Hello World Code Block")}

eval(bVar)

return

Enter, compile, link and run this example and with any luck at all the phrase "A Hello World Code Block" appears on the screen. Never one to trust in luck... if the message did not appear on your screen check everything (you are using Clipper 5.0 right ?) and try again.

You can simply EVAL() the code block directly by the way.

cls

eval( {|| Qout("A Hello World Code Block")} )

return

Not to be overly confusing but someone has undoubtedly noticed that our simple example would continue to operate identically if we coded it more traditionally as follows:

cls

Qout("A Hello World Code Block")

return

The QOUT() function by the way is the equivalent of the ? command in function form (a quick look at STD.CH will confirm this for the doubting among you).

Well If I Can Just Use a Function What's The Point ?

Hang on, not so fast... stick around, I'll show you more.

Don't get the idea that code blocks are equated with a method for printing values on the screen, they are not. While possible, (we proved that), that alone wouldn't serve much purpose.

A code block can just return a number

cls

Qout( eval( {|| 5 } ) )

return

A code block can perform math and return the answer

cls

? eval( {|| 5 * 5 } )

return

A code block can call another function

cls

@ 10, 5 say eval( {|| MyUdf() } )

return

function MyUdf

return "my udf return string"

Notice that in each case I have had to output the value returned by EVAL(). The EVAL() function works in that way just like any other function. EOF() for instance, returns the answer it doesn't automatically print it on the screen.

Like EOF(), RECNO() and all other Clipper functions you can assign the value returned by EVAL() to another variable.

cls

sVar := eval( {|| MyUdf() } )

@ 10, 5 say sVar

return

function MyUdf

return "my udf return string"

We can instead return the return value of the EVAL() function

cls

@ 10, 5 say MyUdf()

return

function MyUdf

return eval( {|| "my udf return string" } )

And as was mentioned earlier, code blocks themselves can be returned

cls

@ 10, 5 say eval( MyUdf() )

return

function MyUdf

return {|| "my udf return string" }

It Still Looks Pretty Trivial

Well then let's add some parameters.

cls

? eval( {| nVar | nVar * nVar }, 6 )

return

In this case we have designed the block to accept a block parameter named nVar which is squared and returned. We have passed along the value 6 as the second argument in the EVAL() function. nVar will be assigned the value 6 and the answer 36 will be returned and displayed.

Talk of parameters and code block variables brings up an important question. What scope do the block parameters have ? And the answer is that they have LOCAL scope which the following fragment should demonstrate.

cls

nVar := 1

? eval( {| nVar | nVar := nVar * nVar }, 6 )

? nVar

return

The answers 36 (actually 36.000000000) and 1 are displayed which indicates that the assignment of 6 to nVar within the code block didn't affect the PRIVATE nVar which was assigned outside of the code block.

On the other hand, barring any naming conflicts variable references inside of the block take on the scope of the variables as defined outside of the block.

cls

nVar := 6

? eval( {|| nVar := nVar * nVar } )

? nVar

return

In this case nVar is displayed as 36 each time because the nVar within the block is the same variable as the one outside the block.

With some slightly convoluted code we can demonstrate another related and important plus about variable scope and code blocks.

local nVar

cls

nVar := 2

bSquare := {| nNum | nNum ^ nVar }

? MyUdf(bSquare)

return

function MyUdf(bBlock)

return eval( bBlock, 10)

The answer displayed is 100 which might seem just a little bit odd because nVar was declared as LOCAL to the main routine but it was still visible in the MyUdf() function. Ordinarily that is behavior demonstrated by PRIVATE and PUBLIC scoped variables only.

It is referred to as exporting a variable and it happens because both nVar and the block referring to nVar are defined in the same routine. When the code block is passed as a parameter to another function nVar (in this example) is made visible to the function.

It Gets A Bit Trickier

Multiple parameters can be passed along.

cls

sRet := eval( {| dVar, sVar | Qout(sVar, dVar) }, date(), "today is :")

? sRet

return

Compiling, linking and running this example will yield something a little unusual. The value "NIL" is output on the line, ? sRet.

NIL is the return value of the Qout() function (it says so in the documentation). NIL, by the way, is another new datatype whose only value is NIL but that is a subject for another time. We don't have to display the return value of the EVAL() function of course but I thought that you might want to see it.

Multiple expressions can be included within a code block (but you cannot include commands (they aren't expressions)).

cls

sRet := eval( {| nVar1, nVar2 | ++nVar1, --nVar2 }, 0, 0)

? sRet

return

Two numeric zero values were passed to the code block which assigned them to the formal parameters nVar1 and nVar2. The first expression incremented nVar1, the second expression decremented nVar2, (the comma is used to separate expressions). When there are multiple expressions in a code block the "value" of the code block is the value of the last expression and in this example, sRet was assigned the value of nVar2.

And don't be overly discouraged by the "no commands" restriction, because we can include functions we can easily write a UDF that executes as many commands as we like.

cls

sRet := eval( {|| MyUdf() } )

return

function MyUdf

cls

@ 10, 5 say "This is a test"

return 0

Gets Rid Of A Few Macros

Code blocks themselves can be passed as parameters which is arguably their strongest point. The closest we've been able to come to this kind of functionality in Summer '87 is through the use of macros to express a function name.

do case

case iKey == "A"

sFunc := "addrec()"

case iKey == "D"

sFunc := "delrec()"

endcase

&sFunc.

The unnecessary use of macros is always best avoided and using blocks would be ideal here.

do case

case iKey == "A"

bFunc := {|| addrec() }

case iKey == "D"

bFunc := {|| delrec() }

endcase

eval( bFunc )

They're Used Within Clipper

It isn't a poor guess to suppose that we have access to code blocks because the developers of 5.0 needed them. They are used extensively in 5.0 and can be seen being referenced in the STD.CH file.

The following list of commands reference code blocks :

SET FORMAT TO

@ SAY/GET RANGE

MENU TO

SET KEY TO

COUNT

SUM

AVERAGE

DELETE FOR/WHILE

RECALL FOR/WHILE

REPLACE WITH FOR/WHILE

Additionally some Clipper functions have been enhanced to allow code blocks as arguments.

ASCAN()

ASORT()

SETKEY() // this one is new

VALTYPE() // this one is new also

While most of the commands have hidden the internal workings of the code block a few of them use documented "iterator" functions.

Iterator Functions

AEVAL() and DBEVAL() are iterative functions. Essentially what they do is EVAL() a code block against each element of an array (AEVAL()) or each record of a .DBF file (DBEVAL()).

To use the AEVAL() function you must pass at least two arguments, the first is the array name and the second is the block to execute.

aVar := { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 }

bDouble := {| iVal | Qout( transform( (iVal * 2), "999" )) }

aeval( aVar, bDouble )

This example displays the value of each element after it has been doubled. I tossed in the transform() function to "add some spice" and to eliminate those trailing zeros.

Optionally third and/or fourth arguments can be passed to AEVAL() that restrict the range of elements to be processed. The third argument is the index into the array representing the first element to apply the block to (if you don't pass it the default is 1).

The fourth argument is the number of elements to process, (not the last element, the number of elements). If you don't pass this along the default is all the elements in the array.

aVar := { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 }

bDouble := {| iVal | Qout( transform( (iVal * 2), "999" )) }

aeval( aVar, bDouble, 5, 3)

This example restricts the elements to be "blocked" to the 5th, 6th and 7th elements.

Lets have a "real" example shall we ?

/* CDIR.PRG

From an idea presented by Bill Christison on NANFORUM and rather heavily modified by Tom Leylan

Some work remains to make it operate "properly" but when I wrote this there was a bug in the directory() function which made my DirAdj() function necessary

compile with : Clipper cdir /M /N /W */

#define F_NAME 1

#define F_SIZE 2

#define F_DATE 3

#define F_TIME 4

#define F_ATT 5

FUNCTION main(sParm1, sParm2)

local sPath, tPause

do case

case pcount() == 0

sPath := "*.*"

tPause := .f.

case pcount() == 1

if (upper(sParm1) == "/P")

sPath := "*.*"

tPause := .t.

else

sPath := DirAdj(sParm1)

tPause := .f.

endif

case pcount() == 2

sPath := DirAdj(sParm1)

tPause := (upper(sParm2) == "/P")

endcase

Qout()

aeval( directory(sPath, "d"), {|aFile| DirSay(aFile, tPause)} )

return NIL

/* this is incomplete but should not be necessary */

FUNCTION DirAdj(sParm1)

local sPath := ltrim(rtrim(sParm1))

if right(sParm1, 1) == "\"

sPath := sParm1 + "*.*"

elseif right(sParm1, 1) == ":"

sPath := sParm1 + "\*.*"

elseif right(sParm1, 1) == "."

sPath := sParm1 + "\*.*"

endif

return sPath

/* list the files */

FUNCTION DirSay(aFiles, tPause)

static nlines := 0

if nlines >= 23 .and. tPause

QOut( "Strike a key when ready . . . " )

inkey(0)

nlines := 0

endif

nlines++

QOut( DirFile(aFiles[F_NAME]), " ", ;

DirSize(aFiles[F_SIZE], aFiles[F_ATT]), ;

DirDate(aFiles[F_DATE]), " ", ;

DirTime(aFiles[F_TIME]) )

return NIL

/* dump the dot */

FUNCTION DirFile(sFile)

local sTemp, iDot, sName, sExt

if left(sFile, 1) == "."

sName := left(sFile + space(12), 12)

else

sTemp := sFile + " ."

iDot := at(".", sTemp)

sExt := left(subs(sTemp, iDot + 1) + space(3), 3)

sName := left(subs(sTemp, 1, iDot - 1) + space(9), 9) + sExt

endif

return sName

/* could be a directory */

FUNCTION DirSize(iSize, sAtt)

return if( ("D" $ sAtt), "

", str(iSize, 8))

/* date delimiters are screwy */

FUNCTION DirDate(dDate)

local sDate := strtran(dtoc(dDate), "/", "-")

return str(val(left(sDate, 2)), 2) + right(sDate, 6)

/* convert to 12 hour format */

FUNCTION DirTime(sTime)

local iHr, sRet

iHr := val(left(sTime, 2))

if (iHr <>

sRet := str(iHr + if((iHr == 0), 12, 0), 2) + subs(sTime, 3, 3) + "a"

else

sRet := str(iHr - if((iHr == 12), 0, 12), 2) + subs(sTime, 3, 3) + "p"

endif

return sRet

Compile and link this example and you have a DOS directory program.

To use the DBEVAL() function you must have a .DBF file open in the currently selected work area and you must pass at least one argument which is the block to execute against each record.

use test

/* test.dbf must have the following structure and you should have some data in it

NAME C 30 0

CITY C 20 0

STATE C 2 0

*/

bShow := {|| Qout( NAME, CITY, STATE ) }

dbeval( bShow )

close test

Optionally there are five other arguments corresponding to

a FOR condition

a WHILE condition

a NEXT n records specifier

a RECORD n specifying a single record number

a REST logical value used to indicate whether the record pointer should be rewound prior to beginning the process or whether to process only the remaining records

I will have to avoid giving any explicit details about the iterator functions like AEVAL() and DBEVAL() in this article. They honestly deserve a separate article with more substantial examples.

And That Isn't All

Except for the example which demonstrated eliminating the macroing of functions it is likely that you cannot think of one overwhelming reason to use a code block instead of a function call. Don't let this bother you. Code blocks are not intended to replace functions they are not the "greatest thing on Earth", they are what they are they do what they do and their best use is where they are needed and not where they can be molded to fit.

We will certainly see their use enhanced in future versions of Clipper I think the ACHOICE(), DBEDIT() and MEMOEDIT() functions are likely candidates but well before then I'm certain that we will see numerous good (and probably some not so good) examples published.

Please don't rush headlong into using code blocks but also don't back away... just take them in stride like all the other things that make Clipper "Clipper".

About the author: Tom Leylan has been a guest speaker at conferences around the world including events in Amsterdam, London, Sydney, Johannesburg, Miami Beach, Orlando, Los Angeles, Palm Desert and Honolulu. He has written numerous articles and is the author of the book, Writing Applications With Clipper (c) 1994 MIS Press ISBN: 1-55851-382-5.

Tom can be reached electronically at: tleylan@nyiq.net

The CA-Clipper Memory System - by Jud Cole

This section written by Jud Cole is the first part of CA-Clipper 5.x Memory Management by Roger Donnay & Jud Cole. If you like to have the whole book in .WRI format.

Jud Cole is the President of Blink, Inc and has been programming on microcomputers since 1979 in many languages including Assembler, C, Pascal, PL/I, dBase and CA-Clipper. After working as a consultant programmer in the early 1980's he worked at IBM for three years in their PC division doing training and support on their entire range of PC products. Following this he wrote contract database applications interfacing CA-Clipper databases to external devices such as magnetic card readers, vehicle tachographs and mechanised storage systems. During 1989 and 1990 he developed BLINKER, the first dynamic overlay linker. Since then he has been enhancing and promoting BLINKER and speaking at user groups and conferences.

---

The aim of this section is to impart a greater understanding of how a CA-Clipper application works internally, with a view to writing more compact and efficient CA-Clipper applications.

This section describes how the CA-Clipper 5.x Virtual Memory Manager uses conventional memory, expanded memory and disk space to store both data and CA-Clipper code. We will examine the PUBLIC, PRIVATE, LOCAL and STATIC variable classes, storage of memory variable values of different types, and how the dynamic paging system manages CA-Clipper 5.x code at application run time.

The level of expertise of the reader is expected to be medium to high, assuming an in-depth knowledge of CA-Clipper programming, and a good knowledge of PCs, networks and programming techniques in general.

Terms and definitions Conventional memory is the memory which exists on all PCs and compatibles, and typically consists of 512kb or 640kb of memory on the mother board. Due to the architecture of the early IBM PCs, the maximum amount of conventional memory is usually limited to 640kb, although certain memory managers can provide another one or two hundred kb on some machines. In theory, the maximum conventional memory on a 8088 / 8086 processor is determined by the 1 Mb address space.

Expanded memory is memory which is also accessible on all PC compatibles. Programs which are to use expanded memory have to be explicitly written to do so. Expanded memory may be provided in the form of hardware or software emulation, and in later versions of the specification, known as EMS, is limited to 32 MB. It is managed in pages, typically 16 kb in size, which may be brought into an area of conventional memory to be accessed by a program.

The currently executing program will request a particular page of expanded memory from the expanded memory manager, and will provide an address in conventional memory at which to place the page. On return from the manager, the data contained in the requested page can be read or written to as if it were permanently resident in conventional memory.

Extended memory is memory accessible by the 80286 and later processors, and exists outside of the 1 Mb range of conventional memory. These processors can access extended memory directly when running in one of their enhanced modes. When running in an 8086 emulation mode, however, a programming interface to extended memory is necessary, and a number of these have been specified. The most widely used of these interfaces is known as the XMS specification.

Software memory managers will often manage extended memory and provide both EMS and XMS programming interfaces to it for maximum versatility.

Virtual memory is a technique which has been used for many years to enable programmers to write programs requiring more memory than is directly available on the destination machine. The technique provides a simple interface to memory for storing and retrieving code and data, whilst hiding the fact that the information may be stored on one or more alternative devices until it is needed again.

The virtual memory manager, which may be implemented in hardware, software, or a combination of the two, monitors the frequency and duration of usage of the information, and decides where to keep each piece of information for maximum overall performance of the system. Typically, the least recently used information will be saved out to slower devices, while the more recently or more often used information will be kept in fast, real memory.

CA-Clipper's virtual memory manager

CA-Clipper 5.x contains its own virtual memory manager, known as the VMM, to manage the data and CA-Clipper code belonging to the application. By default, the CA-Clipper VMM allocates all available conventional memory and up to 8 MB of expanded memory for this purpose. In addition, if all the available memory has been used up, the VMM will swap out data, but not code, to a temporary file on disk.

CA-Clipper currently makes no direct use of extended memory, so if the application will be running on a 386 PC or above, then obtaining a memory manager such as QEMM, 386MAX or the one supplied with MS DOS 6.0 will be a good investment.

Once a CA-Clipper application has loaded into memory and started executing, the application allocates the remaining real memory according to parameters set with the CLIPPER environment variable or the // command line options. The format of these is the same, and consists of the //, the letter or group of letters denoting the area, e.g. E for EMS, a ':' and a number indicating the size in Kb to be used for that area.

The parameters controlling allocation of memory are X:nnn and E:nnn. The X parameter specifies how much conventional memory to eXclude from use by CA-Clipper, and takes a value from 0 to 256 kb. The E parameter specifies how much expanded memory to allocate to the VMM, and takes a value from 0 to 8192 kb.

For example : TEST //E:1000

which would limit CA-Clipper to using 1 Mb of expanded memory.

It is worth noting in passing that the default of all available EMS up to a maximum of 8MB, or the E value if one is specified, is allocated to the VMM in one block at the start of the application. This means it is not available to any other part of the system until the application terminates and the memory is freed. In addition, the application could run out of conventional memory if there is too much EMS available to it, since a table of proportional size to the amount of EMS used is allocated in conventional memory. Depending on the amount of data manipulated by the application, a suitable maximum value may be 1000 to 2000, representing 1 - 2 Mb of expanded memory.

The other CA-Clipper parameters relevant to the VMM are the SWAPPATH:'path' and SWAPK:nnn parameters. If the application's conventional memory and EMS memory is fully utilised then the VMM will create a temporary swap file in the directory indicated by the SWAPPATH parameter, or in the current directory if no SWAPPATH is specified. This disk file will be used to store the least recently used data owned by the VMM, and will gradually increase in size until either the application has terminated and the file deleted, or the size limit set by the SWAPK parameter has been reached. The default size limit for the swap file if no SWAPK parameter is specified is 8 MB.

Virtual memory as managed by the VMM is allocated in segments, each of which may contain from 1K to 64K of data. When memory is allocated from the VMM, instead of returning a pointer to real memory it returns a form of segment number to identify the segment, in the same way as DOS returns a handle when a file is opened. Whenever the data within the segment is needed, a request is made to the VMM to return the current location of the segment in real memory where it can be read or written to.

Initially all the segments will be located in real memory, and because each segment is movable, real memory can be organised efficiently by filling up the gaps as segments are freed. Once real memory fills up, the VMM will swap out least recently used segments to EMS if it is available, or disk if not, to make room for new segments. If those segments are used at a later stage in the program, the VMM will swap out other segments to make room and bring the original segments back in, in the same way as an overlay manager manipulates code overlays.

CA-Clipper 5.x also contains a special type of memory manager designed to manage complex data values such as character strings and arrays. The CA-Clipper 5.x object memory is called the Segmented Virtual Object Store (SVOS). SVOS uses virtual memory managed by the VMM to store data values, including character strings, arrays, and dynamically created (macro-compiled) code blocks.

SVOS provides two important functions beyond the basic capabilities offered by the VMM, memory compaction and garbage collection.

Memory compaction consists of automatically compacting stored values on an ongoing basis. This eliminates fragmentation of the virtual memory and reduces swapping, since each segment can be fully utilised before requesting further segments.

Some CA-Clipper 5.x values (e.g., arrays) may be referred to by several program variables or array elements at the same time. The garbage collection routine automatically reclaims space occupied by values which are no longer accessible through any variable or array. By default, this occurs in background when CA-Clipper is in an idle state, e.g. waiting for keyboard input.

The real memory remaining to the VMM is set up as a swap area to bring swapped out pages of data into memory for use in the CA-Clipper program. When a RUN command is performed, as much of the top of the swap space as possible is cleared and returned back to DOS to be combined with the X area, and then the RUN command is issued. In this way more memory is freed up for RUN commands than would have been with Summer '87, although the exact amount will depend on the size and usage of the lower end of the swap space.

CA-Clipper's symbol table

The CA-Clipper language, and the dBase language on which CA-Clipper was originally based, is a dynamic language with a number of very powerful constructs. These allow and cause certain functions normally performed by the compiler to be postponed until run time, such as setting the type of variables and using macros to create new variables not known at compile time.

Because of this dynamic nature, at run time CA-Clipper requires more information about variables and procedures than traditonal lower level languages such as Pascal, C and Modula 2. Some of this information is available at compile and link time, such as the name of the variable, but some of it, such as its type, will only be available once the application has started executing.

For these reasons, each CA-Clipper .OBJ file is created with a symbol table of 16 bytes per symbol, and all code in the .OBJ file refers to that symbol table. At run time the symbol table entry is used to point to the control information and value or code for the symbol. The symbol table is created in its entirety in the root of the application, and can grow to upwards of 64 kb, so it can significantly affect the amount of conventional memory required by the application. This is why even 100% overlayed applications grow when code is added.

CA-Clipper 5.x introduced static and local variables to the language to encourage better and more efficient coding practices. Another important benefit is that these classes of variables do not require a symbol table entry as they cannot be accessed via macros. Changing as many PUBLIC and PRIVATE variables as possible to LOCAL or STATIC variables can therefore significantly reduce the amount of conventional memory required.

The major linkers now available remove the duplicate symbols from the symbol tables in the various .OBJ and .LIB files at link time, creating one large consolidated symbol table. This process, known as symbol table compression, can significantly reduce the run time memory requirement of the .EXE, leaving more memory for the application's data and overlays. All the duplicates are removed except the symbols belonging to procedures declared as static, since these are local to each .OBJ and will have different code associated with each occurrence of the symbol.

It is worth noting that prior to link time symbol table compression, the only way to reduce the number of duplicate symbols was to minimise the number of .OBJ files, but this is no longer necessary.

CA-Clipper code

When compiled, each CA-Clipper procedure or function in the .OBJ file has a separate unit know as a segment, which consists of a small Assembly language header and a string of tokens. The header simply consists of pointers to the CA-Clipper symbol table and the tokenised code and a call to the CLIPPER.LIB procedure __PLANKTON. The tokenised code represents calls to functions within the CA-Clipper library and parameters to those functions. At application run time when the procedure or function is called the __PLANKTON procedure processes these tokens sequentially and performs the appropriate library calls with the parameters held in the tokens. Each token is usually only one byte long, with parameters varying in length, e.g. a real number will take up 8 bytes and a character string will be stored as the length followed by the string. Tokens may also refer to symbols in the symbol table described above, rather than referring to absolute locations, so each reference to a variable will consist of a two byte symbol number.

For example, in the code :

FUNCTION T A = B + C

we would have a symbol table containing :

T A B C

and the tokenised code would consist of (in simplified terms) :

Take symbol 2 (B) Take symbol 3 (C) Add them together Store result in symbol 1 (A)

This tokenised approach has a number of advantages over true compiled code, with only a neglible cost in performance. The code produced is very compact, for example taking only three bytes for a procedure call, as opposed to five for a direct call. It is also very self contained. All external references go via the symbol table, so operations such as incremental linking are made significantly easier. This approach also makes it possible to use the dynamic paging system described below for faster overlayed applications with lower memory requirements.

The size overhead of a simple CA-Clipper compiled .EXE is actually made up of the runtime routines from the CLIPPER.LIB which are called by the processing of the tokens. The apparently large size of even a "Hello world" type program is due to the potential for macro operations, which could execute just about any CA-Clipper command from even a two line program.

Instances of variables and their values

In conventional languages the scope, size and type of a named variable is known at compile time, so the exact amount of space can be reserved for it in memory at run time. This memory will always be used to store the value, no matter how often the value is changed.

The remaining memory above the program's .EXE image is usually known as the heap and is managed by a heap manager, which will allocate blocks of memory of varying size to the program as and when requested. Space for data allocated dynamically at run time, for constructs such as linked lists or buffers, whose sizes are not known at compile or link time, will be allocated from and returned to this heap.

With CA-Clipper, determination of the type and size of all variables and the scope of public and private variables is left until run time, so a more complicated mechanism for storing the values of variables is required.

CA-Clipper 5.x offers several different storage classes for program variables, depending on how they are declared and used in the program. LOCAL and STATIC variables are stored in a dedicated area of real memory, as described below. PRIVATE and PUBLIC variables, known as MEMVAR variables, are created and destroyed dynamically while a program is running, and are stored in VM segments.

For performance reasons, these segments remain locked in real memory during most operations except memory intensive operations and RUN commands. Each MEMVAR uses 20 bytes in a VM segment, so converting PRIVATE and PUBLIC variables to LOCAL and STATIC variables can reduce memory requirements for some applications.

At run time, each instance of a variable is allocated a value, which is represented internally as a data structure called a VALUE. The contents and format of a VALUE differ depending on the type of data it represents. Simple data, such as integers, are stored directly into the VALUE. Larger items, or data of variable length such as strings or arrays, have a "reference" to the string or array stored in the VALUE, and the actual data is stored elsewhere. Internally, CA-Clipper is organized as a stack based machine which uses an area of memory called the Eval Stack to contain temporary variables such as function parameters, intermediate results of expressions and local variables. The Eval Stack is simply a contiguous group of VALUEs that are accessed as a stack, in the same way as the processor stack is used by C programs.

For example, in a CA-Clipper function call, parameters are pushed onto the Eval Stack before the function is executed. The function operates on the top-most items in the Eval Stack and produces a result. After the function completes, the parameter values are popped from the Eval Stack and replaced with the function result.

Each entry in the Eval Stack, i.e. each VALUE, occupies 14 bytes, and for complex data types such as character strings, arrays and code blocks there will be an additional memory requirement handled by the VMM where the actual value is stored.

The Eval Stack is allocated from the default data segment, defined as the start of the group DGROUP, when the program starts executing, so initialisation will fail if DGROUP is too full. This is not usually a problem with pure CA-Clipper applications, but if a number of third party libraries are linked in to the application it may possibly fill up unless they have avoided storing data in DGROUP. The number of kb remaining in DGROUP for CA-Clipper's use can be examined by executing the program, with the //INFO parameter, and the amount of conventional and expanded memory available will be displayed at the same time.

LOCAL variables are the simplest variables, and are allocated as locations within the Eval Stack to store their VALUEs. To manipulate a LOCAL variable, the system simply copies the variable's VALUE from one position in the Eval Stack to another.

Local variables are visible only within the current procedure or function, and are created automatically each time the procedure in which they were declared begins executing. When that procedure terminates through a return, all it's LOCALs are removed from the Eval Stack and any associated VMM memory freed up.

STATIC variables are similar to LOCAL variables, but have a duration of the lifetime of the application. Because of their permanence, they are allocated as fixed locations at one end of the Eval Stack, but are manipulated in the same way as LOCAL variables simply by copying their VALUEs.

This means that every STATIC variable in the system also requires 14 bytes on the Eval Stack in DGROUP, which is another reason for C and ASM programmers to avoid storing data in DGROUP.

PRIVATE and PUBLIC variables are more complex than LOCAL or STATIC variables because in addition to an associated VALUE they also have a name which may be referred to during execution of the program via a macro or its equivalent. MEMVAR variables are allocated locations for their VALUEs in dedicated VM segments and these locations are stored with their names in the symbol table.

When a MEMVAR is manipulated, the symbol table entry is used to point to the VALUE which can then be placed on the Eval Stack in the normal way. FIELD variables differ from the other storage classes because they have no memory location at all, since their values are stored in a database record buffer. To manipulate a FIELD, the system generates a request to the file's database driver, which then creates an appropriate VALUE to be manipulated on the Eval Stack.

Arrays

An array VALUE contains a reference to the array rather than an actual value, so when an array is assigned to a variable, the system simply overwrites the variable's VALUE with a new VALUE containing a reference to the array. The array itself is simply a group of VALUEs stored in virtual memory, where each element of the array is a VALUE. Any VALUE can contain another reference, so multidimensional arrays are created by having each element refer to another array rather than have an absolute value. When values are assigned to array elements, the VALUE for that element is updated. When an array is assigned to another variable, only a copy of the VALUE referring to the array is made, and the array data itself is not duplicated.

Character Values

A character string VALUE contains a reference to the character data, which is stored elsewhere in the VM. As with arrays, assigning a character value to a variable simply overwrites the variable's VALUE with a new VALUE containing a reference to the character data.

In a similar way to arrays, assigning a character value from one variable to another simply duplicates the VALUE (i.e., the reference to the data). The character data itself is not duplicated.

This reference-based memory management technique is the same for strings, arrays, and code blocks. CA-Clipper's garbage collector monitors references to objects, and when there are no longer any references to a particular piece of data, the space occupied by that data is automatically reclaimed.

Macros

During program execution, when a macro is evaluated to the name of a variable or procedure, the symbol table is searched to find the requested name. Once the name is found, CA-Clipper follows the pointer in the symbol table to the VALUE where all the general information about the symbol is actually stored. The VALUE will indicate whether the procedure or variable being referenced has been defined, and CA-Clipper checks this before continuing any further. If it is undefined and is not a variable being created, CA-Clipper immediately returns an appropriate error - "undefined function" for procedures or functions, and "variable does not exist" for variables. If the procedure or function has been defined correctly, then the VALUE will contain a pointer to the program code to execute for that procedure, and control can be transferred to the procedure.

The remaining case of creating a new variable is handled by adding a new entry to the end of the symbol table. This new entry will have the name of the variable filled in, along with a pointer to a VALUE for the symbol, and will be used from then on to refer to the variable.

Both Summer '87 and CA-Clipper 5.x provide other mechanisms to avoid the creation of these dynamically named variables in the majority of circumstances, such as using an array of elements to store the values, or using code blocks in 5.x. These alternative mechanisms should be used wherever possible, if only because macro operations are inherently very slow, as each name in the symbol table has to be checked until a match is found before execution can continue.

If the use of a macro cannot be avoided, but the name to be created will be one of a known set, then these names should be mentioned explicitly somewhere in one of the programs. The code does not ever have to be executed, but just using the names causes them to be added to the symbol table at compile time, thus avoiding the above situation.

Code blocks

Code blocks are represented internally as strings of tokenised code, in the same way as normal procedures and functions. When a code block is assigned to a variable at run time, a pointer to the tokens making up the code block is stored in the variable, along with information pertaining to the currently active procedure.

Because the code block consists of normal tokens, it will include references to the symbol table, so the equivalent symbol table must be available when the code block is actually evaluated. This is one of the reasons why it will prove difficult (but not impossible) to save code blocks in a database from one application and restore and evaluate them at a later time in the same or another application.

CA-Clipper's dynamic paging system When linked with BLINKER or .RTLink, CA-Clipper 5.x performs its own form dynamic overlaying of compiled CA-Clipper code, which results in extremely fast, memory efficient execution of the code.

During linking all CA-Clipper modules are broken down into pages of 1 kb in size. These pages are stored either in the executable file or in separate overlay files. The manipulation of overlays in these 1 kb pages removes the effect the size of compiled functions or modules has on the memory required to load the overlay. Large modules are broken into multiple pages, and small functions are grouped together in a single page.

At execution time, CA-Clipper 5.x's dynamic overlay manager loads pages based on information embedded in the .EXE by the linker. The dynamic pages are loaded into VM (Virtual Memory) segments, allowing the VMM to manage the overlay pages on a competitive basis with other uses of memory such as the application data.

The paging architecture allows the system to discard low-use sections of code even if the code is still active, and reload it only when control returns to that piece of code. Code pages which are being heavily used are maintained in memory by the VMM's LRU swapping policy.

When possible, the VMM will place dynamic overlay pages in expanded memory, reducing overlay reads. Overlay pages are never written to the VMM disk swap file, however. If a VM segment containing an overlay page is to be removed from memory altogether, it is simply discarded. If it is needed subsequently, it is re-read from the overlay file. In addition to virtual memory, the dynamic overlay manager uses a dedicated area of real memory to cache the most active dynamic overlay pages. This page mechanism is made possible by the nature of the CA-Clipper code. As explained before, it is not actually code but a series of tokens which are processed at run time. This means that the __PLANKTON procedure from CLIPPER.LIB which is processing the tokens can detect when it has reached the end of a page and request the next one to be loaded. All CA-Clipper code is therefore overlayable, so there are no restrictions on which CA-Clipper .OBJs can be placed in the overlay area. It should be noted that linkers which use the dynamic paging mechanism of CA-Clipper 5.x automatically overlay ALL CA-Clipper code unless directed otherwise.