CA-Clipper 5.2e/5.3 comes with the following replaceable database drivers or RDDs
DBFNTX - Native Clipper DBFCDX - Foxpro 2 compatible DBFMDX - DBase IV compatible DBFNDX - dBase III plus compatible DBFPX - Paradox compatible You should note that the DBFCDX driver that comes with 5.2e is the Successware SIX driver while 5.3/5.3a comes with the Lodestone COMIX driver.
The most popular RDDs as can be seen from posts on comp.lang.clipper are DBFCDX & DBFNTX. As for me, I use both drivers as appropriate. Therefore I shall limit this section to the DBFCDX & DBFNTX drivers.
The following information is extracted from NGs that came with CA-Clipper 5.2e
- RDD Architecture
- DBFNTX - CA-Clipper Native RDD
- DBFCDX - Foxpro 2.0 Compatible RDD
CA-Clipper supports a driver architecture that allows CA-Clipper applications to use Replaceable Database Drivers (RDDs). The RDD system makes CA-Clipper applications data-format independent. Such applications can, therefore, access the data formats of other database systems, including the dBASE IV (.mdx), FoxPro (.cdx), and Paradox (.db) formats on a variety of equipment. This driver architecture can even support database drivers that are not file-based, although all of the drivers supplied with CA-Clipper 5.2 are file-based. The concept of replaceable drivers is not new to this version of CA-Clipper. In previous versions, the use of the default database driver (DBFNTX.LIB) was hidden by the fact that it was automatically linked into your application. In fact, this is still the case. The DBFNTX driver has been replaceable since it was first introduced in version 5.0. Before this version, the DBFNTX driver was the only RDD supplied as part of the system.
With the introduction of the new RDDs, CA-Clipper provides many new and enhanced commands and functions that access and manipulate databases. These language elements can enable your applications to access data regardless of the RDD under which it is ordered. There are also commands and functions that give you specific information about the RDDs in use.
DBFNTX is the default RDD for CA-Clipper. This new database driver replaces the DBFNTX database driver supplied with earlier versions of CA-Clipper and adds a number of new indexing features. With DBFNTX, you can:
Create conditional indexes by specifying a FOR condition
Create indexes using a record scope or WHILE condition, allowing you to INDEX based on the order of another index
Create both ascending and descending order indexes
Specify an expression that is evaluated periodically during indexing in order to display an index progress indicator As an update of the default database driver, DBFNTX is linked into and used automatically by your application unless you compile using the /R option.
New Features
The replaceable driver lets you create and maintain (.ntx) files using features above and beyond those supplied with the previous DBFNTX driver. The new indexing features are supplied in the form of several syntactical additions to the INDEX and REINDEX commands. Specifically you can:
Specify full record scoping and conditional filtering using the standard ALL, FOR, WHILE, NEXT, REST, and RECORD clauses
Create an index while another controlling index is still active
Monitor indexing as each record (or a specified record number interval) is processed using the EVAL and EVERY clauses
Eliminate separate coding for descending order keys using the DESCENDING clause
Compatibility
Index files (.ntx) created with the original DBFNTX driver are compatible with DBFNTX and can be used in new applications without reindexing. Index files (.ntx) created with this version of DBFNTX will also work with previous CA-Clipper applications provided that you use no FOR, WHILE, , or DESCENDING clauses.
Important! Indexes produced with DBFNTX using FOR or DESCENDING are incompatible with earlier version (.ntx) files. If you attempt to access them with the original DBFNTX database driver or programs compiled with versions earlier than CA-Clipper 5.2, you will get an unrecoverable runtime error. In CA-Clipper, this generates an "index corrupted" error message, causing the application to terminate.
New Locking Scheme
The DBFNTX database driver implements a new locking scheme to resolve several problems identified in previous versions of CA-Clipper and to prevent potential problems that might arise when running CA-Clipper applications in a network environment. This section discusses these changes and their implications, including compatibility issues.
Lock Time-outs
Problem: Index locking in previous versions of CA-Clipper was handled automatically by the database driver, and had no time-out provision. This created the potential for problems in network environments if a workstation died while holding a lock. If this situation occurred all other workstations waiting for an index lock would appear to freeze while waiting to obtain their lock. This could also happen if a user placed a CA-Clipper application in the background on a multitasking system without sufficient processing time allocated to it. Eventually, most network operation systems would clear a connection that had no activity for a specified period of time. This would free the lock and everything would resume as normal, but frustrated users may have rebooted their machines possibly causing file corruption.
Solution: In CA-Clipper 5.2 the NTX driver will generate a recoverable runtime error if it fails to lock the index after a predetermined number of retries. The default error handler for this system simply returns (.T.) to retry the operation. This emulates the behavior of previous CA-Clipper versions.
Error Handling
Time out handling: The handling of this error is problematic because the lock is issued from various internal index routines. Therefore the only safe recoveries are to retry or quit. Choosing to default from the error or issuing a break will more than likely leave the index in a corrupted state. If either of the options is employed, the application should immediately recreate the index. The preferred way to handle a time out such as this is to alert the user of the situation so they don't think their machine has hung, and then have the network administrator determine what workstation is causing the problem. When the problem workstation is cleared, the users that have timed out can select retry and continue processing.
NTXERR.PRG: The file NTXERR.PRG contains the source code for the default error handler INIT procedure. This error handler can be modified to allow user-defined error handling for index lock time-outs. Care should be exercised when modifying the error handler as detailed above.
Compatibility: The lock time-out capability when used in conjunction with the default error handler is totally compatibility with previous versions of CA-Clipper. No changes are made to the NTX file structure and no action is required by the developer to activate the time-out functionality.
New Lock Offset
Problem: Index locking, which is transparent to the developer, uses a single-byte semaphore locking system. This semaphore was placed at a virtual offset (beyond the physical end of file) in the index file. In previous versions of CA-Clipper, this offset was located at one billion (1,000,000,000) which was adequate at the time. But many systems today are capable of producing indexes that are large enough to cause the actual data present at the lock offset to become physically locked. This leads to problems when trying to read or write to the data at that offset.
Solution: The solution is to move the offset where locking occurs to a location at a greater offset. We have chosen FFFFFFFF hex, which is the largest offset possible under the DOS operating system. The problem with this solution is that new applications using the index will be locking this new byte while old applications using the same index will lock the old position. Clearly this would cause both applications to fail because each could have a lock on the file at the same time.
To avoid this, the signature of the index (in the index header) is modified to prevent pre-CA-Clipper 5.2 applications from being able to open the index. CA-Clipper 5.2 applications can detect the correct offset to use by the flag in the header and will automatically use the correct one. In Figure 7-1 below, each bit represents a flag:
BIT 7 6 5 4 3 2 1 0 FLAG R R R O P I I C
R Reserved I Index type - both bits set (NTX) C Index created with a Condition, condition in header T Created as a Temporary index O New Offset for exclusive (semaphore) lock
Figure 7-1: Bit Field for the Signature Byte of a CA-Clipper 5.2 NTX File
Activation
If CA-Clipper 5.2 automatically modified the signature in the header when it created indexes, programs with automatic reindexing routines would be creating indexes that appeared corrupt to pre-CA-Clipper 5.2 applications. This has an obvious problem with backward compatibility. Therefore, in order to create indexes with the new signature, the developer must link in the module NTXLOCK2.OBJ with the full knowledge that this will create indexes that older applications will not be able to access.
Header Changes
The signature byte of a .NTX file is 6 for an unenhanced NTX index. The inclusion of the NTXLOCK2.OBJ will cause the signature to become 26 hex. (6 hex ORed with 20 hex). See Figure 7-1 for an illustration of all the possible values for the signature byte.
Error Handling
CA-Clipper 5.2 applications will automatically recognize the signature byte of the header, and depending on the signature value, will use the correct index lock location. Applications built with previous versions of CA-Clipper, however, do not have the capability to detect the optional new information in the signature byte. Therefore, when an order application tries to open a file that has been created with the NTXLOCK2.OBJ linked in it will produce a Corruption Detected error.
Compatibility
The new locking location, if used, is not backward compatible with applications compiled with previous versions of CA-Clipper.
Indexes created by applications built with a previous version of CA-Clipper can be used by CA-Clipper 5.2 using the new location and will not be modified unless the index is recreated in application.
Since older applications have no knowledge of the new index locking scheme nor of the significance of the header signature, these applications will assume the index is corrupt and will produce an Index Corrupted error.
Conditional Indexing
Conditional indexes are a feature of the DBFNTX driver. This section discusses this feature of the DBFNTX driver in some detail, giving you specific information about the implementation of conditional indexes. Compatibility issues are also discussed.
Conditional indexes are produced by using a FOR condition in the index creation process. These indexes are made fully maintainable by storing the FOR condition in the index header. This condition is subsequently retrieved and compiled each time the index in opened. During updates, items are added to the index only if they meet the criteria of the condition.
Since older applications do not have the ability to recognize and use the condition stored in the header, they must be prevented from opening the index since they corrupt the index. This is accomplished by modifying the signature of the index (in the index header) preventing pre-CA-Clipper 5.2 applications from being able to open the index. CA-Clipper 5.2 applications can detect the flag in the header and will automatically use the stored FOR condition correctly.
Temporary Indexes
Temporary indexes are produced by using any scoping clause other than the FOR condition in the index creation process. These indexes are not automatically maintainable because the condition is not stored for later use. These indexes can be made maintainable if the condition can be expressed as a FOR condition and is added using the FOR clause. But the main use of temporary indexes is for fast creation of indexes for read- only browses or reports that operate on a subset of the database.
Since older applications would not operate properly with indexes that do not contain all the keys in a given database, they must be prevented from using them. This is accomplished by modifying the index signature to prevent pre-CA-Clipper 5.2 applications from being able to open the index.
EOF()
If an index is created with a FOR condition and an attempt is made to update the index with a key that does not match the condition, the update is suppressed and the index is placed at EOF(). This is consistent with the current behavior for indexes created with the unique flag when an update is attempted with a non-unique key.
Also if a navigational action is attempted (SKIP) and the current record is not found in the index, the index will place the record pointer at EOF(). This is true for both conditional and temporary indexes.
Installing DBFNTX Driver Files
DBFNTX is supplied as the file DBFNTX.LIB.
The CA-Clipper installation program installs this driver as the default in the \CLIPPER5\LIB subdirectory on the drive that you specify, so you need not install the driver manually.
Important! Before installing CA-Clipper, you may want to rename the DBFNTX.LIB that currently resides in your \CLIPPER5\LIB directory to DBFNTX.001. The new version, when installed, will overwrite DBFNTX.LIB. If you do not rename or otherwise protect the old version of DBFNTX.LIB, you will lose it.
Linking the DBFNTX Database Driver
Since DBFNTX is the default database driver for CA-Clipper, there are no special instructions for linking. Unless you specify the /R option when you compile, the new driver will be linked into each program automatically if you specify a USE command or DBUSEAREA() function without an explicit request for another database driver. The driver is also linked if you specify an INDEX or REINDEX command with any of the new features.
Using the DBFNTX Database Driver
In applications written for the new DBFNTX driver, you can use the INDEX and REINDEX commands exactly as you have used them in the past. The index files (.ntx) you create and maintain in this way are completely compatible with those created using previous versions of the driver.
Changes to existing code are necessary only if you use the new indexing features. The (.ntx) files you create using the new features will have a slightly different header file and cannot be used by programs linked with a previous version of the driver.
Using (.ntx) and (.ndx) Files Concurrently
You can use (.ntx) and (.ndx) files concurrently in a CA-Clipper program like this:
// (.ntx) file using default DBFNTX driver
USE File1 INDEX File1 NEW
// (.ndx) files using DBFNDX driver
USE File2 VIA "DBFNDX" INDEX File2 NEW
Note, however, that you cannot use (.ntx) and (.ndx) files in the same work area. For example, the following does not work:
USE File1 VIA "DBFNDX" INDEX File1.ntx, File2.ndx
Compatibility with dBASE III PLUS
The default DBFNTX driver makes CA-Clipper programs behave differently than traditional dBASE programs. Some of these differences are discussed below.
Supported Data Types
The DBFNTX database driver supports the following dBASE III PLUS- compatible data types for key expressions:
Character
Numeric
Date
Logical
Supported Key Expressions
When you create (.ntx) files using the DBFNTX driver, you can use all CA-Clipper or user-defined functions compatible with dBASE III PLUS as well as other functions accepted by the extended CA-Clipper functionality.
Error Handling
The indexing behavior of DBFNTX and DBFNDX in a CA-Clipper application is identical unless otherwise noted. With the default DBFNTX driver, you can handle most errors using BEGIN SEQUENCE...END SEQUENCE as illustrated in the next section.
FIND vs SEEK
In CA-Clipper, you can use the FIND command only to locate keys in indexes where the index key expression is character data type. This differs from dBASE III PLUS where FIND supports character and numeric key values.
Note: In CA-Clipper programs, always use the SEEK command or the DBSEEK() function to search an index for a key value.
The DBFNTX driver lets you recover from data type errors raised during a FIND or SEEK. However, since Error:canDefault, Error:canRetry or Error:canSubstitute are set to false (.F.), you should use BEGIN SEQUENCE...END to handle such SEEK or FIND data type errors. Within the error block for the current operation, issue a BREAK() using the error object that the DBFNTX database driver generates, like this:
bOld := ERRORBLOCK({|oError| BREAK(oError)}) . . . BEGIN SEQUENCE SEEK xVar RECOVER USING oError // Recovery code END . . .
ERRORBLOCK(bOld)
Sharing Data on a Network
The DBFNTX driver provides file and record locking schemes that are different from dBASE III PLUS schemes. This means that if the same database and index files are open in CA-Clipper and in dBASE III PLUS, CA-Clipper program locks are not visible to dBASE III PLUS and vice versa.
Warning! Database integrity is not guaranteed and index corruption will occur if CA-Clipper and dBASE III PLUS programs attempt to write to a database or index file at the same time. Therefore, concurrent use of the same database (.dbf) and index (.ndx) files by dBASE III PLUS and CA-Clipper programs is strongly discouraged and not supported by Computer Associates.
DBFCDX is the FoxPro 2 compatible RDD for CA-Clipper. As such, it connects to the low-level database management subsystem in the CA-Clipper architecture. When you use the DBFCDX RDD, you add a number of new features including:
FoxPro 2 file format compatibility
Compact indexes
Compound indexes
Conditional indexes
Memo files smaller than DBFNTX format
Overview of the DBFCDX RDD
The DBFCDX driver lets you create and maintain (.cdx) and (.idx) files with features different from those supplied with the original DBFNTX driver and is compatible with files created under FoxPro 2. The new features are supplied in the form of several syntactical additions to the INDEX and REINDEX commands. Specifically, you can:
Create indexes smaller than those created with the DBFNTX driver. The key data is stored in a compressed format that substantially reduces the size of the index file.
Create a compound index file that contains multiple indexes (TAGs), making it possible to open several indexes under one file handle. A single (.cdx) file may contain up to 99 index keys.
Create conditional indexes (FOR / WHILE / REST / NEXT).
Create files with FoxPro 2 file format compatibility.
Compact Indexes
Like FoxPro 2, The DBFCDX driver creates compact indexes. This means that the key data is stored in a compressed format, resulting in a substantial size reduction in the index file. Compact indexes store only the actual data for the index keys. Trailing blanks and duplicate bytes between keys are stored in one or two bytes. This allows considerable space savings in indexes with much empty space and similar keys. Since the amount of compression is dependent on many variables, including the number of unique keys in an index, the exact amount of compression is impossible to predetermine.
Compound Indexes
A compound index is an index file that contains multiple indexes (called tags). Compound indexes (.cdx)'s make several indexes available to your application while only using one file handle. Therefore, you can overcome the CA-Clipper index file limit of 15. A compound index can have as many as 99 tags, but the practical limit is around 50. Once you open a compound index, all the tags in the file are automatically updated as the records are changed.
Once you open a compound index, all the tags contained in the file are automatically updated as the records are changed. A tag in a compound index is essentially identical to an individual index (.idx) and supports all the same features. The first tag (in order of creation) in the compound index is, by default, the controlling index.
Conditional Indexes
The DBFCDX driver can create indexes with a built-in FOR clause. These are conditional indexes in which the condition can be any expression, including a user-defined function. As the database is updated, only records that match the index condition are added to the index, and records that satisfied the condition before, but don't any longer, are automatically removed.
Expanded control over conditional indexing is supported with the revised INDEX and REINDEX command options as in the new DBFNTX driver.
Installing DBFCDX Driver Files
The DBFCDX driver is supplied as the file, DBFCDX.LIB.
The CA-Clipper installation program installs this driver in the \CLIPPER5\LIB subdirectory on the drive that you specify, so you need not install the driver manually.
Linking the DBFCDX Database Driver
To link the DBFCDX database driver into an application program, you must specify DBFCDX.LIB to the linker in addition to your application object files (.OBJ).
1. To link with .RTLink using positional syntax:
C>RTLINK ,,,DBFCDX
2. To link with .RTLink using freeformat syntax:
C>RTLINK FI LIB DBFCDX
Note: These link commands all assume the LIB, OBJ, and PLL environment variables are set to the standard locations. They also assume that the CA-Clipper programs were compiled without the /R option.
Using the DBFCDX Database Driver
To use FoxPro 2 files in a CA-Clipper program:
1. Place REQUEST DBFCDX at the beginning of your application or at the top of the first program file (.prg) that opens a database file using the DBFCDX driver.
2. Specify the VIA "DBFCDX" clause if you open the database file with the USE command.
-OR-
3. Specify "DBFCDX" for the argument if you open the database file with the DBUSEAREA() function.
-OR-
4. Use RDDSETDEFAULT( "DBFCDX" ) to set the default driver to DBFCDX.
Except in the case of REQUEST, the RDD name must be a literal character string or a variable. In all cases it is important that the driver name be spelled correctly.
The following program fragments illustrate:
REQUEST DBFCDX . . . USE Customers INDEX Name, Address NEW VIA "DBFCDX"
-OR-
REQUEST DBFCDX RDDSETDEFAULT( "DBFCDX" ) . . . USE Customers INDEX Name, Address NEW
Using (.idx) and (.ntx) Files Concurrently
You can use both (.idx) and (.ntx) files concurrently in a CA-Clipper program like this:
// (.ntx) file using default DBFNTX driver
USE File1 INDEX File1 NEW
// (.idx) files using DBFCDX driver
USE File2 VIA "DBFCDX" INDEX File2 NEW
Note, however, that you cannot use (.idx) and (.ntx) files in the same work area. For example, the following does not work:
USE File1 VIA "DBFNTX" INDEX File1.ntx, File2.idx
Using (.cdx) and (.idx) Files Concurrently
You may use (.cdx) with (.idx) files concurrently (even in the same work area); however, in most cases it is easier to use a single (.cdx) index for each database file or separate (.idx) files. When using both types of index at the same time, attempting to select an Order based on its Order Number can be confusing and will become difficult to maintain.
File Maintenance under DBFCDX
When an existing tag in a compound index (.cdx) is rebuilt using INDEX ON...TAG... the space used by the original tag is not automatically reclaimed. Instead, the new tag is added to the end of the file, increasing file size.
You can use the REINDEX command to "pack" the index file. REINDEX rebuilds each tag, eliminating any unused space in the file.
If you rebuild your indexes on a regular basis, you should either delete your (.cdx) files before rebuilding the tags or use the REINDEX command to rebuild them instead.
DBFCDX and Memo Files
The DBFCDX driver uses FoxPro compatible memo (.fpt) files to store data for memo fields. These memo files have a default block size of 64 bytes rather than the 512 byte default for (.dbt) files.
DBFCDX memo files can store any type of data. While (.dbt) files use an end of file marker (ASCII 26) at the end of a memo entry, (.fpt) files store the length of the entry. This not only eliminates the problems normally encountered with storing binary data in a memo field but also speeds up memo field access since the data need not be scanned to determine the length.
Tips For Using DBFCDX
1. Make sure index extensions aren't hard-coded in your application. The default extension for DBFCDX indexes is (.idx), not (.ntx). You can still use (.ntx) as the extension as long as you specify the extension when you create your indexes. The best way to determine index extensions in an application is to call ORDBAGEXT().
For example, if you currently use the following code to determine the existence of an index file:
IF .NOT. FILE("index.ntx") INDEX ON field TO index ENDIF
Change the code to include the INDEXEXT() function, as follows:
IF .NOT. FILE("index"+ORDBAGEXT()) INDEX ON field TO index ENDIF
2. If your application uses memo fields, you should convert your (.dbt) files to (.fpt) files.
There are some good reasons for using (.fpt) files. Most important is the smaller block size (64 bytes). CA-Clipper's (.dbt) files use a fixed block size of 512 bytes which means that every time you store even 1 byte in a memo field CA-Clipper uses 512 bytes to store it. If the data in a memo field grows to 513 bytes, then two blocks are required.
When creating (.fpt) files, the block size is set at 64 bytes to optimize it for your needs. A simple conversion from (.dbt) files to (.fpt) files will generally shrink your memo files by approximately 30%.
3. Add DBFCDX.LIB as a library to your link command or link script.
since 3 June 1996.