Linking Techniques for Memory Management (Part 1) by Roger Donnay

This section written by Roger Donnay is the third part of CA-Clipper 5.x Memory Management by Roger Donnay & Jud Cole. If you like to have the whole book in .WRI format, click here to visit Roger's Site

The Role of Linkers in developing large applications has taken a new turn in the past few years. In addition to providing segment-linking, they are now also expected to provide the best "memory image" possible. This means that the linker must be sophisticated in it's management of code segments during the running of an application. The physical size of a CA-Clipper application is no longer representative of it's memory image, because the linker "overlays" the code segments to use the smallest possible amount of memory. Some type of code (such as CA-Clipper P-CODE) is easy to overlay while other types (ASM) may be more difficult. Applications that are almost all CA-Clipper-compiled code will usually produce the best memory image regardless of the physical size of the .EXE.

The subjects of this book are the four most commonly-used linkers available on the market today for linking CA-Clipper 5.2 and 5.3 applications. They are:

RTLINK - A "Real Mode" CA-Clipper-5.2 specific linker developed by Pocketsoft, Inc. This is the linker that is bundled with CA-Clipper 5.2.

BLINKER - A "Real Mode" Third-Party linker published by Blink, Inc. This is the linker that is bundled with CA-Clipper 5.3. A "Dual-Mode" version of Blinker is also available from Blink, Inc.

EXOSPACE - A "Protected Mode" linker developed by CA and sold as an add-on to Clipper 5.2. This linker is bundled with Clipper 5.3.

CAUSEWAY - A third-party "Protected Mode" linker developed by Michael Devore for use with Clipper 5.2 and 5.3.

The CA-Clipper compiler has the responsiblity of converting source code to "machine-readable" code. This .OBJect code can not be "executed" by the machine however until it is "linked" to other .OBJect code segments in the CA-Clipper libraries. This is the responsibility of the linker. Linking or "fixing up" references made by your code to external variables and functions is necessary to insure that your compiled program will address or call other programs, including functions in the CA-Clipper libraries and DOS functions.

A linker's basic job is to combine objects which have been compiled separately. Objects contain symbols of three types:

a. PUBLIC symbols - those symbols which are callable from other objects

b. STATIC symbols - those symbols which are callable only from within the same object

c. UNDEFINED symbols - symbols which exist in other objects and are referenced or used by this object ((also referred to as EXTERNAL symbols).

DYNAMIC Overlaying of CA-Clipper Code

Very large CA-Clipper applications can consist of more code than can be loaded into the available amount of DOS memory and may not be able to execute due to insufficient memory. CA-Clipper 5.x has a "Dynamic Overlay Manager" actually built into the CA-Clipper libraries that helps prevent this problem by loading the code into memory only as it is needed by the application that is running. The linker helps prepare the application to use this capability of CA-Clipper 5.2 by splitting the CA-Clipper-compiled application code into fixed-size "pages" that share a pre-allocated area of memory. These pages of code are then written to a seperate .OVL file or to the end of the .EXE file as determined by your linker options. When a procedure or function is called, the "dynamic-overlay manager" first checks to see if it is in memory. If it is, then it jumps to the called function. If it is not in memory, it loads the function from disk into any "free" location available in the overlay pool. If no free space can be found in the overlay pool, the overlay manager discards pages that have not been called recently to free space to load the called procedure.

The larger the overlay pool, the less often the overlay manager needs to load from disk and the faster your program will run. The size of the dynamic overlay pool needed for your application is automatically determined by CA-Clipper.

The great thing about CA-Clipper 5.x's dynamic overlaying system is that you, the programmer, don't need to do anything to create overlays. It is all done automatically by the linker. Only CA-Clipper-compiled modules, not C/ASM modules, are overlayed "dynamically", although since much of the CA-Clipper libraries has been compiled in CA-Clipper, even large portions of the CA-Clipper libraries will be dynamically overlayed.

Application libraries and third-party libraries written in CA-Clipper will also be automatically overlayed by RTLINK and BLINKER. In many cases it is more beneficial to write functions in CA-Clipper (using LOCAL memvars) than in C or ASM because it will not increase the executable memory model. Even CA-Clipper code that has been linked into to Pre-link libraries will be dynamically overlayed.

INTERNAL/EXTERNAL Overlays

INTERNAL overlays are dynamic pages which are written to the end of the .EXEcutable program file. During the running of the application, the dynamic overlay manager reads the disk and loads dynamic pages from this file. This is the default overlay method of RTLINK and BLINKER and it is often advantageous to use internal overlays to reduce the number of file handles being opened by your application. In some cases however, you may find that using multiple overlay files will increase the speed performance of your application because extracting code from several small files which are all opened at the same time is often faster that extracting code from one large file.

EXTERNAL overlays are sets of dynamic pages which are written to a seperately designated file more commonly referred to as an overlay file. The positional command /DYNAMIC[:] or the freeformat command DYNAMIC INTO will tell RTLINK to write the dynamic pages to a file named . You may wish to create a seperate overlay file when your application .EXE is too large to fit on a distribution disk.

Examples:

Rtlink> DYNAMIC INTO myprog.ovl Blinker> SECTION INTO myprog.ovl

Rtlink> DYNAMIC INTO myprogs.ovl MYPROG1, MYPROG2 DYNAMIC INTO myfuncs.ovl MYFUNC1, MYFUNC2 Blinker> SECTION INTO myprogs.ovl MYPROG1, MYPROG2 SECTION INTO myfuncs.ovl MYFUNC1, MYFUNC2

DYNAMIC Overlaying of C / ASM Code

BLINKER also provides the ability to dynamically overlay code written in C or Assembly. This is the most common code found in many third-party libraries. RTLINK also provides for overlaying of C/ASM but it is more complex and requires understanding more about the code being overlayed. See STATIC OVERLAYS and RELOADABLE OVERLAYS for more information on overlaying C/ASM code with RTLINK.

In the previous section, we discussed that application code compiled with the CA-Clipper compiler is dynamically overlayed using the overlay manager built inside the CA-Clipper libraries. This CA-Clipper-specific internal overlay manager is incapable of overlaying C/ASM code, so it is the responsibility of the linker to provide this capability. Each linker has it's own methods for overlaying this kind of code. BLINKER uses an overlay pool that is allocated at the start of the application. The size of this pool is determined at link-time. The larger the overlay pool, the faster the application will run because overlays will not be reloaded into memory from disk so often.

The dynamic-overlay capabilities of BLINKER require that the objects being overlayed are "well-behaved". After much experimentation with our own libraries and many third-party libraries, it is now more clear what constitutes well-behaved modules:

a. Well-behaved code uses the CA-Clipper EXTEND API registers or the stack for parameter passing and does not use undocumented features of CA-Clipper.

b. All CA-Clipper-compiled .OBJects are dynamically-overlayable.

Modules which CANNOT be dynamically-overlaid are:

a. Routines which handle interrupts.

b. Modules in which the DATA area is changed during runtime rather than allocating memory in the root area.

Dynamic overlaying of third-party libraries can be an aggravating learning experience if you don't get support from the third party vendor in trying to accomplish this task. Most of the third-party vendors now break their libraries into separate overlayable and non-overlayable .LIB files to make the task much easier.

Once you have determined which .OBJ files and/or .LIB files are acceptable for overlaying, the task of telling the linker to do this is quite simple. A BLINKER link-script will look similar to an RTLINK script except there is always only one (1) overlay area because all code uses the same overlay "pool". Example of a BLINKER script for overlaying C/ASM:

# Allocate 50k to the overlay pool

BLINKER OVERLAY OPSIZE 50

BEGINAREA

# CA-Clipper-compiled application code

FILE myprog1.obj, myprog2.obj

# C/ASM files

FILE ccode.obj, asmcode.obj

# C/ASM libraries

LIB ccode.lib, asmcode.lib

ENDAREA

Using EXPANDED or UMB RAM for OVERLAYS

BLINKER provides a several very useful feature for increasing the amount of available memory for CA-Clipper applications, i.e., the use of Expanded (EMS) or Upper Memory Blocks (UMB).

EMS Memory

Many computers have EMS memory. EMS memory managers such as QEMM, 386MAX, and DOS's own EMM386 create a memory area above conventional memory referred to as the "page frame". This 64k block of memory is mapped to an area of ram that can execute 8086 code. Blinker takes advantage of this feature by allocating the 64k EMS page frame rather than conventional DOS memory for their runtime overlay pool. This frees an additional 64k of memory for use by CA-Clipper because the overlays are loaded into the page frame rather than conventional ram. If no EMS is present when the application starts, then the overlay pool will be established in normal (conventional) ram. To enable this feature, use the command:

BLINKER OVERLAY PAGEFRAME ON

UMB Memory

Expanded memory managers will also create an area between the 640k and 1 meg memory address called UMB (upper memory blocks). This is the area where you would normally load your memory-resident programs such as DOS, network-drivers, mouse-drivers, etc. If your enviroment gives you about 30k - 60k of contiguous memory available in the UMB area, Blinker will load the overlay pool into the UMB area rather than conventional area, thus freeing more conventional memory for your CA-Clipper application. To enable this feature, use the Blinker command:

BLINKER OVERLAY UMB ON

Note: In today's modern environments, this UMB area is usually consumed by so many TSR's that the possiblity of finding a contiguous block larger than 30k is virtually impossible, so it recommended that you do not rely on the availability of this memory.

STATIC Overlays

STATIC overlays are supported by RTLINK only. They provide a method of overlaying C and ASM code which cannot be overlayed by the dynamic-overlay manager. C and ASM code is not as easily relocatable as CA-Clipper compiled PCODE (pseudo-code) therefore it cannot be overlayed "dynamically". A function which is dynamically loaded and unloaded from a pool is usually found in different places in memory at different times during the running of the program. When a C or assembly routine is called by an application linked with RTLINK it must be loaded in memory at the same address every time. This is accomplished by designating overlay AREAS where many different modules will occupy the same memory space, but only one-at-a-time. Designing Static overlays requires a more thorough understanding of the structure of your program. Static overlay segments usually look like this in your link file:

# area 1

BEGINAREA

SECTION FILE A

SECTION FILE B

SECTION FILE C

ENDAREA

# area 2

BEGINAREA

SECTION FILE D

SECTION FILE E

SECTION FILE F

ENDAREA

You must be sure that you properly place your objects in the overlay segments to prevent computer "lock-up" or "crashing" at runtime. In the above example, any procedure or function in FILE A may call any procedure or function in FILE D, E or F because they are in different areas, however if a function in FILE A calls a procedure or function in FILE B, then your program will crash as soon as the program returns to FILE A because FILE A was removed from memory to load FILE B.

Overlay management with "static overlays" is the most time-consuming because it requires extensive analysis of the structure of the program to insure that the modules are placed properly in the overlay areas.

RELOADABLE Overlays

The following overlaying technique should be applied at your own risk because it is an "unsupported" and "undocumented" feature of the RTLINK linker supplied with CA-Clipper-5.2. I feel however, that it is important to cover this subject because without an understanding of how to use this feature it may be impossible to develop certain types of CA-Clipper applications without resorting to a third-party linker.

I started RELOADING modules from the CA-Clipper libraries years ago using RTLINK and the Summer 87 version of Clipper, so when I received my copy of CA-Clipper-5.2, the first thing I attempted to do was to overlay some of the larger C/ASM modules in the CA-Clipper libraries using the RELOAD command in my linker script file. The CA-Clipper development team had the foresight to insure that their C and ASM compiler assigned a UNIQUE name to each CA-Clipper C/ASM object in the CA-Clipper libraries, thus allowing, you, the CA-Clipper programmer to use both the RELOAD command and MODULE command in your .LNK script files.

So what are "Reloadable Overlays"? These are a form of STATIC overlay in which segments of code are placed into overlay sections that occupy the same memory space at runtime, however the "calling module" is automatically "reloaded" into memory when returning from the "called module".

Reloadable overlay segments usually look like this in your link file:

RELOAD FAR 200

# area 1

BEGINAREA

SECTION FILE A

SECTION FILE B

SECTION FILE C

ENDAREA

# area 2

BEGINAREA

SECTION FILE D

SECTION FILE E

SECTION FILE F

ENDAREA

Not all CA-Clipper applications are going to need to use these reloadable static overlays, but if your application relies heavily on third-party libraries written in C/ASM, then it is recommended that you learn this technique.

The main advantage of "reloadable overlays" over conventional "static overlays" is that you are never in danger of "lockup" in the event that a procedure or function in FILE A calls a function or procedure in FILE B because FILE A will be reloaded into memory on return from FILE B. Make sure when you use the RELOAD command that you always use it as follows:

RELOAD FAR

where the is the amount of memory in hex bytes to use for saving addresses. My experience is that most applications will run just fine with a stack size of 200. If your application does recursive nesting, however, you may need to increase the stack size. Recursive nesting is described as follows: Procedure A calls Procedure B which calls Procedure A which calls Procedure B, etc, etc, etc. Each time a procedure is called, its return address is "pushed" onto the stack, and each time you return to the calling procedure the address is "popped" from the stack. In the above example, if this recursive condition were allowed to continue for a large number of iterations, the stack would be overrun and the program may crash.

Although your application will not crash using RELOAD, it may slow down a bit if overlay segments are not structured properly, because each time an overlay is reloaded the application must go to disk. In the above example if a function in FILE A repetitively calls a function in FILE B then your application will be very "disk intensive" and will run slowly. If a function in FILE A repetitively calls a function in FILE D there will be no slowing at all because both modules will remain in memory.

Overlay management of C/ASM code with "reloadable overlays" is less time-consuming and more reliable than simple conventional "static overlays" and provides an additional advantage of allowing more modules to be overlayed, thereby saving additional memory usage.