Monday, January 26, 2009

Linking Techniques for Memory Management (Part 1) by Roger Donnay

This section written by Roger Donnay is the third part of CA-Clipper 5.x Memory Management by Roger Donnay & Jud Cole. If you like to have the whole book in .WRI format, click here to visit Roger's Site

The Role of Linkers in developing large applications has taken a new turn in the past few years. In addition to providing segment-linking, they are now also expected to provide the best "memory image" possible. This means that the linker must be sophisticated in it's management of code segments during the running of an application. The physical size of a CA-Clipper application is no longer representative of it's memory image, because the linker "overlays" the code segments to use the smallest possible amount of memory. Some type of code (such as CA-Clipper P-CODE) is easy to overlay while other types (ASM) may be more difficult. Applications that are almost all CA-Clipper-compiled code will usually produce the best memory image regardless of the physical size of the .EXE.

The subjects of this book are the four most commonly-used linkers available on the market today for linking CA-Clipper 5.2 and 5.3 applications. They are:

RTLINK - A "Real Mode" CA-Clipper-5.2 specific linker developed by Pocketsoft, Inc. This is the linker that is bundled with CA-Clipper 5.2.

BLINKER - A "Real Mode" Third-Party linker published by Blink, Inc. This is the linker that is bundled with CA-Clipper 5.3. A "Dual-Mode" version of Blinker is also available from Blink, Inc.

EXOSPACE - A "Protected Mode" linker developed by CA and sold as an add-on to Clipper 5.2. This linker is bundled with Clipper 5.3.

CAUSEWAY - A third-party "Protected Mode" linker developed by Michael Devore for use with Clipper 5.2 and 5.3.

The CA-Clipper compiler has the responsiblity of converting source code to "machine-readable" code. This .OBJect code can not be "executed" by the machine however until it is "linked" to other .OBJect code segments in the CA-Clipper libraries. This is the responsibility of the linker. Linking or "fixing up" references made by your code to external variables and functions is necessary to insure that your compiled program will address or call other programs, including functions in the CA-Clipper libraries and DOS functions.

A linker's basic job is to combine objects which have been compiled separately. Objects contain symbols of three types:

a. PUBLIC symbols - those symbols which are callable from other objects

b. STATIC symbols - those symbols which are callable only from within the same object

c. UNDEFINED symbols - symbols which exist in other objects and are referenced or used by this object ((also referred to as EXTERNAL symbols).

DYNAMIC Overlaying of CA-Clipper Code

Very large CA-Clipper applications can consist of more code than can be loaded into the available amount of DOS memory and may not be able to execute due to insufficient memory. CA-Clipper 5.x has a "Dynamic Overlay Manager" actually built into the CA-Clipper libraries that helps prevent this problem by loading the code into memory only as it is needed by the application that is running. The linker helps prepare the application to use this capability of CA-Clipper 5.2 by splitting the CA-Clipper-compiled application code into fixed-size "pages" that share a pre-allocated area of memory. These pages of code are then written to a seperate .OVL file or to the end of the .EXE file as determined by your linker options. When a procedure or function is called, the "dynamic-overlay manager" first checks to see if it is in memory. If it is, then it jumps to the called function. If it is not in memory, it loads the function from disk into any "free" location available in the overlay pool. If no free space can be found in the overlay pool, the overlay manager discards pages that have not been called recently to free space to load the called procedure.

The larger the overlay pool, the less often the overlay manager needs to load from disk and the faster your program will run. The size of the dynamic overlay pool needed for your application is automatically determined by CA-Clipper.

The great thing about CA-Clipper 5.x's dynamic overlaying system is that you, the programmer, don't need to do anything to create overlays. It is all done automatically by the linker. Only CA-Clipper-compiled modules, not C/ASM modules, are overlayed "dynamically", although since much of the CA-Clipper libraries has been compiled in CA-Clipper, even large portions of the CA-Clipper libraries will be dynamically overlayed.

Application libraries and third-party libraries written in CA-Clipper will also be automatically overlayed by RTLINK and BLINKER. In many cases it is more beneficial to write functions in CA-Clipper (using LOCAL memvars) than in C or ASM because it will not increase the executable memory model. Even CA-Clipper code that has been linked into to Pre-link libraries will be dynamically overlayed.


INTERNAL overlays are dynamic pages which are written to the end of the .EXEcutable program file. During the running of the application, the dynamic overlay manager reads the disk and loads dynamic pages from this file. This is the default overlay method of RTLINK and BLINKER and it is often advantageous to use internal overlays to reduce the number of file handles being opened by your application. In some cases however, you may find that using multiple overlay files will increase the speed performance of your application because extracting code from several small files which are all opened at the same time is often faster that extracting code from one large file.

EXTERNAL overlays are sets of dynamic pages which are written to a seperately designated file more commonly referred to as an overlay file. The positional command /DYNAMIC[:] or the freeformat command DYNAMIC INTO will tell RTLINK to write the dynamic pages to a file named . You may wish to create a seperate overlay file when your application .EXE is too large to fit on a distribution disk.


Rtlink> DYNAMIC INTO myprog.ovl Blinker> SECTION INTO myprog.ovl


DYNAMIC Overlaying of C / ASM Code

BLINKER also provides the ability to dynamically overlay code written in C or Assembly. This is the most common code found in many third-party libraries. RTLINK also provides for overlaying of C/ASM but it is more complex and requires understanding more about the code being overlayed. See STATIC OVERLAYS and RELOADABLE OVERLAYS for more information on overlaying C/ASM code with RTLINK.

In the previous section, we discussed that application code compiled with the CA-Clipper compiler is dynamically overlayed using the overlay manager built inside the CA-Clipper libraries. This CA-Clipper-specific internal overlay manager is incapable of overlaying C/ASM code, so it is the responsibility of the linker to provide this capability. Each linker has it's own methods for overlaying this kind of code. BLINKER uses an overlay pool that is allocated at the start of the application. The size of this pool is determined at link-time. The larger the overlay pool, the faster the application will run because overlays will not be reloaded into memory from disk so often.

The dynamic-overlay capabilities of BLINKER require that the objects being overlayed are "well-behaved". After much experimentation with our own libraries and many third-party libraries, it is now more clear what constitutes well-behaved modules:

a. Well-behaved code uses the CA-Clipper EXTEND API registers or the stack for parameter passing and does not use undocumented features of CA-Clipper.

b. All CA-Clipper-compiled .OBJects are dynamically-overlayable.

Modules which CANNOT be dynamically-overlaid are:

a. Routines which handle interrupts.

b. Modules in which the DATA area is changed during runtime rather than allocating memory in the root area.

Dynamic overlaying of third-party libraries can be an aggravating learning experience if you don't get support from the third party vendor in trying to accomplish this task. Most of the third-party vendors now break their libraries into separate overlayable and non-overlayable .LIB files to make the task much easier.

Once you have determined which .OBJ files and/or .LIB files are acceptable for overlaying, the task of telling the linker to do this is quite simple. A BLINKER link-script will look similar to an RTLINK script except there is always only one (1) overlay area because all code uses the same overlay "pool". Example of a BLINKER script for overlaying C/ASM:

# Allocate 50k to the overlay pool



# CA-Clipper-compiled application code

FILE myprog1.obj, myprog2.obj

# C/ASM files

FILE ccode.obj, asmcode.obj

# C/ASM libraries

LIB ccode.lib, asmcode.lib



BLINKER provides a several very useful feature for increasing the amount of available memory for CA-Clipper applications, i.e., the use of Expanded (EMS) or Upper Memory Blocks (UMB).

EMS Memory

Many computers have EMS memory. EMS memory managers such as QEMM, 386MAX, and DOS's own EMM386 create a memory area above conventional memory referred to as the "page frame". This 64k block of memory is mapped to an area of ram that can execute 8086 code. Blinker takes advantage of this feature by allocating the 64k EMS page frame rather than conventional DOS memory for their runtime overlay pool. This frees an additional 64k of memory for use by CA-Clipper because the overlays are loaded into the page frame rather than conventional ram. If no EMS is present when the application starts, then the overlay pool will be established in normal (conventional) ram. To enable this feature, use the command:


UMB Memory

Expanded memory managers will also create an area between the 640k and 1 meg memory address called UMB (upper memory blocks). This is the area where you would normally load your memory-resident programs such as DOS, network-drivers, mouse-drivers, etc. If your enviroment gives you about 30k - 60k of contiguous memory available in the UMB area, Blinker will load the overlay pool into the UMB area rather than conventional area, thus freeing more conventional memory for your CA-Clipper application. To enable this feature, use the Blinker command:


Note: In today's modern environments, this UMB area is usually consumed by so many TSR's that the possiblity of finding a contiguous block larger than 30k is virtually impossible, so it recommended that you do not rely on the availability of this memory.

STATIC Overlays

STATIC overlays are supported by RTLINK only. They provide a method of overlaying C and ASM code which cannot be overlayed by the dynamic-overlay manager. C and ASM code is not as easily relocatable as CA-Clipper compiled PCODE (pseudo-code) therefore it cannot be overlayed "dynamically". A function which is dynamically loaded and unloaded from a pool is usually found in different places in memory at different times during the running of the program. When a C or assembly routine is called by an application linked with RTLINK it must be loaded in memory at the same address every time. This is accomplished by designating overlay AREAS where many different modules will occupy the same memory space, but only one-at-a-time. Designing Static overlays requires a more thorough understanding of the structure of your program. Static overlay segments usually look like this in your link file:

# area 1






# area 2






You must be sure that you properly place your objects in the overlay segments to prevent computer "lock-up" or "crashing" at runtime. In the above example, any procedure or function in FILE A may call any procedure or function in FILE D, E or F because they are in different areas, however if a function in FILE A calls a procedure or function in FILE B, then your program will crash as soon as the program returns to FILE A because FILE A was removed from memory to load FILE B.

Overlay management with "static overlays" is the most time-consuming because it requires extensive analysis of the structure of the program to insure that the modules are placed properly in the overlay areas.


The following overlaying technique should be applied at your own risk because it is an "unsupported" and "undocumented" feature of the RTLINK linker supplied with CA-Clipper-5.2. I feel however, that it is important to cover this subject because without an understanding of how to use this feature it may be impossible to develop certain types of CA-Clipper applications without resorting to a third-party linker.

I started RELOADING modules from the CA-Clipper libraries years ago using RTLINK and the Summer 87 version of Clipper, so when I received my copy of CA-Clipper-5.2, the first thing I attempted to do was to overlay some of the larger C/ASM modules in the CA-Clipper libraries using the RELOAD command in my linker script file. The CA-Clipper development team had the foresight to insure that their C and ASM compiler assigned a UNIQUE name to each CA-Clipper C/ASM object in the CA-Clipper libraries, thus allowing, you, the CA-Clipper programmer to use both the RELOAD command and MODULE command in your .LNK script files.

So what are "Reloadable Overlays"? These are a form of STATIC overlay in which segments of code are placed into overlay sections that occupy the same memory space at runtime, however the "calling module" is automatically "reloaded" into memory when returning from the "called module".

Reloadable overlay segments usually look like this in your link file:


# area 1






# area 2






Not all CA-Clipper applications are going to need to use these reloadable static overlays, but if your application relies heavily on third-party libraries written in C/ASM, then it is recommended that you learn this technique.

The main advantage of "reloadable overlays" over conventional "static overlays" is that you are never in danger of "lockup" in the event that a procedure or function in FILE A calls a function or procedure in FILE B because FILE A will be reloaded into memory on return from FILE B. Make sure when you use the RELOAD command that you always use it as follows:


where the is the amount of memory in hex bytes to use for saving addresses. My experience is that most applications will run just fine with a stack size of 200. If your application does recursive nesting, however, you may need to increase the stack size. Recursive nesting is described as follows: Procedure A calls Procedure B which calls Procedure A which calls Procedure B, etc, etc, etc. Each time a procedure is called, its return address is "pushed" onto the stack, and each time you return to the calling procedure the address is "popped" from the stack. In the above example, if this recursive condition were allowed to continue for a large number of iterations, the stack would be overrun and the program may crash.

Although your application will not crash using RELOAD, it may slow down a bit if overlay segments are not structured properly, because each time an overlay is reloaded the application must go to disk. In the above example if a function in FILE A repetitively calls a function in FILE B then your application will be very "disk intensive" and will run slowly. If a function in FILE A repetitively calls a function in FILE D there will be no slowing at all because both modules will remain in memory.

Overlay management of C/ASM code with "reloadable overlays" is less time-consuming and more reliable than simple conventional "static overlays" and provides an additional advantage of allowing more modules to be overlayed, thereby saving additional memory usage.

No comments:

Post a Comment

Welcome to Clipper... Clipper... Clipper

In 1997, then using Delphi 3, I had already created 32-bits Windows applications for HRIS, ERP and CRM. In 2007, using Ruby on Rails, an AJAX powered CRM site running on Apache & MySQL was created and I am now using Visual Studio .Net 2008 to create web-based projects and Delphi 7 for Win32 applications using SQL2005 & DBFCDX.

So, why then am I reviving the Original Clipper... Clipper... Clipper via a Blog as CA-Clipper is a programming language for the DOS world ? Believe it or not, there are still some clients using my mission-critical CA-Clipper applications for DOS installed in the late 80's and up to the mid 90's. This is testimony to CA-Clipper's robustness as a language :-)

With the widespread introduction of Windows 7 64-bits as the standard O/S for new Windows based PCs & Notebooks, CA-Clipper EXE simply will not work and it has become imperative for Clipper programmers to migrate immediately to Harbour to build 32/64 bits EXEs

Since 28th January 2009, this blog has been read by 134,389 (10/3/11 - 39,277) unique visitors (of which 45,151 (10/3/11 - 13,929) are returning visitors) from 103 countries and 1,574 cities & towns in Europe (37; 764 cities), North America (3; 373 cities) , Central America & Caribeans (6; 13 cities), South America(10; 226 cities), Africa & Middle-East (12; 44 cities) , Asia-Pacific (21; 175 cities). So, obviously Clipper is Alive & Well : -)

TIA & Enjoy ! (10th October 2012, 11:05; 13th November 2015)

Original Welcome Page for Clipper... Clipper... Clipper

This is the original Welcome Page for Clipper... Clipper... Clipper, which I am republishing for historical and sentimental reasons. The only changes that I have made was to fix all the broken links. BTW, the counter from is still working :-)

Welcome to Chee Chong Hwa's Malaysian WWW web site which is dedicated to Clipperheads throughout the world.

This site started out as a teeny-weeny section of Who the heck is Chee Chong Hwa ? and has graduated into a full blown web site of more than 140 pages (actually hundreds of A4 size pages) ! This is due to its growing popularity and tremendous encouragements from visiting Clipperheads from 100 countries worldwide, from North America, Central America, Caribbean, South America, Europe, Middle-East, Africa and Asia-Pacific. Thanx Clipperheads, you all made this happen !

What is Clipper ?

You may ask, what is this Clipper stuff ? Could Clipper be something to do with sailing as it is the name of a very fast sailing American ship in the 19th century ?

Well, Clipper or to be precise, CA-Clipper is the premier PC-Software development tool for DOS. It was first developed by Nantucket Corporation initially as a compiler for dBase3+ programs. Since then, CA-Clipper has evolved away from its x-base roots with the introduction of lexical scoping & pre-defined objects like TBrowse. As at today, the most stable version ofClipper is 5.2e while the latest version, 5.3a was introduced on 21 May 1996.

As at 11th November, 1996, an unofficial 5.3a fixes file was made available by Jo French. See the About CA-Clipper 5.3a section for more details. BTW, Jo French uploaded the revised 5.3a fixes file on 20th November, 1996.

Latest News

The latest news is that CA has finally released the long-awaited 5.3b patch on 21 May, 1997.

For 5.3b users, you must a take a look at Jo French's comments on unfixed bugs in 5.3b.

BTW, have you used Click ? If you're a serious Clipperprogrammer and need an excellent code formatter, Click is a natural choice. How to get it ? Simple, access Phil Barnett's site via my Cool Clipper Sites.

32-bits Clipper for Windows ?

Have you tried Xbase ++ ? Well, I have and compared to Delphi (my current Windows programming tool of choice), I'm still sticking to Delphi.

Anyway, you should visit the Alaska Home Page. Give it a chance and then draw your own conclusions !.

The Harbour Project

Is this the future of Xbase ? Take a look at at the Harbour Project

You are Visitor # ...

According to, you are visitor since 3 June 1996.

If you like or dislike what you see on this website, please drop me a line by clicking the email button at the bottom of this page or better still, by filling out the form in my guest book. If you are not sure what to write,click here to take a look at what other Clipperheads have to say.