Monday, January 26, 2009

DGScan-The DGROUP Usage Scanner For Clipper by Ian Day & Dave Pearson

By Ian Day and Dave Pearson

Background On DGROUP

(Portions of this section are taken, with the kind permission of Dark Black Software, from the MrDebug Norton Guide)

DGROUP is a 64K chunk of memory that, for a Clipper program, can be broken down into five distinct sections, each serving a specific purpose. DGROUP can be easily described as a 64K block of memory that is used by Clipper as a common area of memory that Cliper and third party products can rely on to find various items of information. There are five main parts to DGROUP:

1. The MemVar table (Dynamic)

This is used to store Clipper STATIC variables and ITEMs (declared through the ITEM API) with their contents if the contents and variable/ITEM definition can be held within 14 bytes, otherwise the variable/ITEM definition is held along with a pointer to a Virtual Memory segment where the variable/ITEM contents are held.

2. DS Available (Dynamic)

This is the amount of free memory available within the DGROUP. This amount can be seen from //INFO as DS AVAIL or from the Memory/Info window. This will be reduced during the execution of the program as the Eval Stack and the Memvar table may both 'grow' into this area.

3. Eval Stack (Dynamic)

This is where Clipper LOCAL variables are held (if the variable definition and contents take up 14 bytes or less), otherwise a pointer is stored to a Virtual memory segment that contains the variable contents.

4. CPU Stack (Fixed)

The CPU stack is used to store the function return addresses each time a new function is called, as well as local 'C' variables.

The size of the CPU stack is set when you link your program. It depends upon the linker that you are using and linker specific commands that you have in the link script that might increase or reduce the stack

5. Fixed C and ASM Data (Fixed)

This area is used by other third party libraries and other languages that do not use their own data segments and use the default data segment (otherwise known to us Clipperites as DGROUP) to store fixed pieces of text and global variables used by other languages

For example, inside Clipper, the un-recoverable error messages are stored within this section of the default data segment (DGROUP).

As you can see from the above, the DGROUP area is pretty important to the smooth running of your Clipper application, and a lack of DGROUP can cause your software to fall over with a number of internal errors.

In an effort to make sure this does not happen you should try to reduce the impact your code has on DGROUP. Many 3rd party libraries use large ammounts of DGROUP (in my experience they can bump up the fixed C and ASM data usage by quite a bit. If you need to use a number of 3rd party libraries this can become quite a problem.

However, given that your chances of changing the usage of 3rd party libraries are pretty slim, the only chance you have of making a difference is by improving your own code to reduce it's DGROUP usage.

What Can Be Done?

If you write nothing but pure Clipper code, you still have the ability to reduce your DGROUP usage. The main area you can address is your use of STATIC variables. STATIC variables are, without a doubt, a good thing, but, they come with a price.

Unlike the other variable types, each STATIC variable requires a fixed (14 byte) entry in the the MemVar table. On the surface this may not seem like a big deal. A STATIC only needs 14 bytes in DGROUP, what does it matter? To see how this matters, lets take the GETSYS.PRG code you can find in the SOURCE\SYS directory of your copy of Clipper as an example.

Close to the top of the source you will find the following code:

// State variables for active READ
STATIC sbFormat
STATIC slUpdated := .F.
STATIC slKillRead
STATIC slBumpTop
STATIC slBumpBot
STATIC snLastExitState
STATIC snLastPos
STATIC soActiveGet
STATIC scReadProcName
STATIC snReadProcLine

As you can see, there are 10 STATIC variables, each using 14 bytes in the MemVar Table. With a simple little trick we can reduce that 140 bytes of usage down to 14 bytes. Watch:

// Hold all the statics in one static.

Static _aSysStuff := { NIL, .F., NIL, NIL, NIL, NIL,; NIL, NIL, NIL, NIL }

// Translated static variable names.

#xtranslate sbFormat => _aSysStuff\[ 01 \] #xtranslate slUpdated => _aSysStuff\[ 02 \] #xtranslate slKillRead => _aSysStuff\[ 03 \] #xtranslate slBumpTop => _aSysStuff\[ 04 \] #xtranslate slBumpBot => _aSysStuff\[ 05 \] #xtranslate snLastExitState => _aSysStuff\[ 06 \] #xtranslate snLastPos => _aSysStuff\[ 07 \] #xtranslate soActiveGet => _aSysStuff\[ 08 \] #xtranslate scReadProcName => _aSysStuff\[ 09 \] #xtranslate snReadProcLine => _aSysStuff\[ 10 \]

With that simple little trick you will have reduced the DGROUP usage from 140 bytes to just 14 bytes.

If you have lots of code of your own that has a large number of file wide STATICs in a single file you can make quite a difference.

If you write your own C functions, then it's a wise move to take a look at the compiler options at your disposal, as some of these can drastically reduce the impact on DGROUP.

For example, take the following piece of code:


CLIPPER YesNo( void ) // Return yes or no string { if ( _parl(1) ) // .T. passed to function? { _retc( "Yes" ); // Yes, so return "Yes" } else { _retc( "No" ); // No, so return "No" } }

Simple enough, but already we've just used 7 (or 8) bytes of DGROUP without thinking! Why? you ask. Answer: It's the strings. By default, C compilers will put constants (which means strings, doubles, floats and some arrays) into DGROUP. And when I say '7 (or 8)', it's because some compilers will align data on a WORD boundary as default, and you can usually expect the DGROUP usage to be a tad more than you thought.

You can change this by looking for a compiler option to reverse this default. For example, with the Microsoft C compilers, there is the /Gt switch and for Borland C compilers there is the -Ff switch. Given a number of 1, it will automatically put any constants larger than 1 length into a FAR segment, and out of DGROUP! This means, that no matter how much data is used by your C functions for strings and things, it won't make a difference to DGROUP.

With assembler code, we use the same principles. Don't use _DATA, _BSS, CONST or DGROUP for data storage if you can possibly help it.

So, if you have a module like this:

cYes DB 'Yes', 0
cNo DB 'No', 0





MOV ax, 1 PUSH ax CALL __parl ADD sp, 2

OR ax, ax JZ ZeroSoItsFalse

MOV ax, OFFSET cYes JMP SHORT ReturnString

ZeroSoItsFalse: MOV ax, OFFSET cNo

ReturnString: PUSH ax PUSH ds CALL __retc ADD sp, 4



You've again used 7 (or 8) bytes of DGROUP without thinking, so in this case you would have to alter the segments and anything that used the data in them so that DGROUP can once again be saved:

cYes DB 'Yes', 0
cNo DB 'No', 0



MOV ax, 1 PUSH ax CALL __parl ADD sp, 2

OR ax, ax JZ ZeroSoItsFalse

MOV ax, OFFSET cYes JMP SHORT ReturnString

ZeroSoItsFalse: MOV ax, OFFSET cNo

ReturnString: PUSH ax PUSH YESNO_DATA CALL __retc ADD sp, 4



As you can see, if your application is tight on DGROUP, with the source to hand and a little bit of work you can free up some of that usage. Lets just hope you have the source code for the worst offending code in your application.

Using DGSCAN To Find DGROUP Usage

Back when I first became aware of this "problem" I started to work on my own library code and managed to reduce some of the DGROUP usage. However, working through all that code was getting to be pretty boring. That's when I decided to write DGSCAN. What I wanted was a tool that could scan OBJ and LIB files and find anything that might be taking up precious DGROUP space.

If you want to get and try out DGSCAN then pop over to Hagbard's World and grab a copy. Got it? Good.

Ok, lets take a simple example of using DGSCAN. Look at the following code:

Static xVar1
Static xVar2
Static xVar3
Static xVar4

Function Main()

? "I'm eating up DGROUP"

Return( NIL )

Compile it into an OBJ and then run DGSCAN over it:

F:\DGTEST>dgscan foo.obj

DgScan v3.00 - DGROUP Usage Scanner By Dave Pearson

File-------- Module--------------------------- Segment----------- Bytes----- FOO.OBJ FOO STATICS$ 56 FOO.OBJ *** Total *** 56

As you can see, DGSCAN has detected 56 bytes of possible DGROUP usage. This is because we have 4 STATIC variables, each with a 14 byte impact. If you applied the "array trick" we spoke about earlier you would see the usage reduced to 14 bytes.

To generate a report for all your OBJ files just do:

F:\DGTEST>dgscan *.obj

It's as simple as that, and the same goes for LIB files.

I won't say much more about DGSCAN, everything you need to know is covered by the file DGSCAN.TXT found in the DGSCAN archive. However, if you do have problems with the utility please feel free to mail me and I'll try to help you out.

No comments:

Post a Comment

Welcome to Clipper... Clipper... Clipper

In 1997, then using Delphi 3, I had already created 32-bits Windows applications for HRIS, ERP and CRM. In 2007, using Ruby on Rails, an AJAX powered CRM site running on Apache & MySQL was created and I am now using Visual Studio .Net 2008 to create web-based projects and Delphi 7 for Win32 applications using SQL2005 & DBFCDX.

So, why then am I reviving the Original Clipper... Clipper... Clipper via a Blog as CA-Clipper is a programming language for the DOS world ? Believe it or not, there are still some clients using my mission-critical CA-Clipper applications for DOS installed in the late 80's and up to the mid 90's. This is testimony to CA-Clipper's robustness as a language :-)

With the widespread introduction of Windows 7 64-bits as the standard O/S for new Windows based PCs & Notebooks, CA-Clipper EXE simply will not work and it has become imperative for Clipper programmers to migrate immediately to Harbour to build 32/64 bits EXEs

Since 28th January 2009, this blog has been read by 134,389 (10/3/11 - 39,277) unique visitors (of which 45,151 (10/3/11 - 13,929) are returning visitors) from 103 countries and 1,574 cities & towns in Europe (37; 764 cities), North America (3; 373 cities) , Central America & Caribeans (6; 13 cities), South America(10; 226 cities), Africa & Middle-East (12; 44 cities) , Asia-Pacific (21; 175 cities). So, obviously Clipper is Alive & Well : -)

TIA & Enjoy ! (10th October 2012, 11:05; 13th November 2015)

Original Welcome Page for Clipper... Clipper... Clipper

This is the original Welcome Page for Clipper... Clipper... Clipper, which I am republishing for historical and sentimental reasons. The only changes that I have made was to fix all the broken links. BTW, the counter from is still working :-)

Welcome to Chee Chong Hwa's Malaysian WWW web site which is dedicated to Clipperheads throughout the world.

This site started out as a teeny-weeny section of Who the heck is Chee Chong Hwa ? and has graduated into a full blown web site of more than 140 pages (actually hundreds of A4 size pages) ! This is due to its growing popularity and tremendous encouragements from visiting Clipperheads from 100 countries worldwide, from North America, Central America, Caribbean, South America, Europe, Middle-East, Africa and Asia-Pacific. Thanx Clipperheads, you all made this happen !

What is Clipper ?

You may ask, what is this Clipper stuff ? Could Clipper be something to do with sailing as it is the name of a very fast sailing American ship in the 19th century ?

Well, Clipper or to be precise, CA-Clipper is the premier PC-Software development tool for DOS. It was first developed by Nantucket Corporation initially as a compiler for dBase3+ programs. Since then, CA-Clipper has evolved away from its x-base roots with the introduction of lexical scoping & pre-defined objects like TBrowse. As at today, the most stable version ofClipper is 5.2e while the latest version, 5.3a was introduced on 21 May 1996.

As at 11th November, 1996, an unofficial 5.3a fixes file was made available by Jo French. See the About CA-Clipper 5.3a section for more details. BTW, Jo French uploaded the revised 5.3a fixes file on 20th November, 1996.

Latest News

The latest news is that CA has finally released the long-awaited 5.3b patch on 21 May, 1997.

For 5.3b users, you must a take a look at Jo French's comments on unfixed bugs in 5.3b.

BTW, have you used Click ? If you're a serious Clipperprogrammer and need an excellent code formatter, Click is a natural choice. How to get it ? Simple, access Phil Barnett's site via my Cool Clipper Sites.

32-bits Clipper for Windows ?

Have you tried Xbase ++ ? Well, I have and compared to Delphi (my current Windows programming tool of choice), I'm still sticking to Delphi.

Anyway, you should visit the Alaska Home Page. Give it a chance and then draw your own conclusions !.

The Harbour Project

Is this the future of Xbase ? Take a look at at the Harbour Project

You are Visitor # ...

According to, you are visitor since 3 June 1996.

If you like or dislike what you see on this website, please drop me a line by clicking the email button at the bottom of this page or better still, by filling out the form in my guest book. If you are not sure what to write,click here to take a look at what other Clipperheads have to say.