Inclusion of LetoDB in Harbour Project ?

CCH : An interesting discussion in the Harbour Project Main Developer List on 2nd Oct 2009


Date: Fri, 2 Oct 2009 20:35:32 +0400
From: Alexandr Okhotnikov

Hi

Take as an axiom: in the harbour need open replacement ADS (for
multi-user environment) for those who still uses DBF and can not be
(not wants) to switch to SQL

2009/10/2 Przemyslaw Czerpak :
> Hi All,
>
> 1. it tries to use DBF record buffer but does not support all field types
> and features supported by core DBF* RDDs
> 2. it does not support timestamp values

The main types are (my opinion that this long enough), add the rest
later (for DBF, they are not critical)

> 3. numeric values in some operations like SEEK are converted to strings so
> they cannot be safely used with double values because such conversions
> round them to number of decimal places used to show numeric item on
> the screen

SEEK: for numeric fields to translate a string based on the dimension
of the field (no loss of accuracy)

> 4. it does not optimize multirecord skips on filtered or set_deleted
> workareas
> 5. it does not use aliases on server side so expressions sent to server
> cannot use aliases too (.f. index key/for, filters, ...)
> 6. relations are implemented only on client side and this code is not
> finished - it needs fixes in many places,
> missing server side relations seriously reduce functionality and
> performance in programs which use aliases though without server side
> support for aliases it still not be fully usable.

Using ADS almost the same :)

> 7. it does not respect many of _SET_* settings and also does not inform
> server about settings like _SET_DATEFORMAT, _SET_EPOCH, _SET_TIMEFORMAT,
> _SET_DECIMALS, _SET_FIXED, _SET_EXACT, ... which are important for
> correct execution of user expressions on server side

I think to finish (or customize a specific case) will not take much time

> 8. some important RDDI_*, DBI_*, DBOI_*, DBRI_*, DBS_* actions are not
> implemented at all or are implemented wrongly what breaks some other
> code, i.e. there is no DBI_GETLOCKARRAY so dbRlockList() function does
> not work or it informs GT that it uses DBF compatible record buffer
> and can transfer whole records but such functionality is not implemented
> so optimized by RDD code COPY TO ... and APPEND FROM ... operations
> fail
> 9. it tries to optimize some operations like dbAppend() but it's done
> in unsafe way which is not compatible with Clipper RDDs, i.e. dbAppend()
> is buffered and always return success though other client may set
> FLOCK() on the table so it will fail later when such appended record
> is sent to the server. For some programs it introduce other bugs
> because some algorithms may use APPEND rlocks for synchronization
> and to protect other stations against setting FLOCK so such behavior
> will causes total desynchronization

Confirm

> 10. in some methods it does not send all parameters and settings to server
> so some operations may not work as expected, i.e. important ordCondSet()
> settings are not transfer to server side
> 11. some methods are implemented as dummy ones what only disables error
> messages when user tries to use them so he does not even know that
> they failed silently,
> such things have to be eliminated and I hope it's only unfinished code
> 12. some operations are hacked to work only in some very limited
> situations, i.e. DBOI_KEYADD/DBOI_KEYDELETE works only with string
> key values which does not have any embedded 0
> 13. some things are implemented wrongly, i.e. if current record is
> locked (unlocked) then dbRlock( ) (dbRUnLock( ))
> calls are ignored
> 14. the transaction code also looks like unfinished, now it makes only
> one thing: it delays sending modifications to the server side with
> only very basic support for modifying the same record more then once,
> it may be usable in some applications but probably breaks most of
> programs which were not designed from beginning to work with RDDs where
> modifications are not visible during updates, missing documentation
> for exact isolation level causes that such transactions can be safely
> used only by LETO code authors who well know the internal behavior
> and created .prg code compatible with it
> 15. it's not MT safe and needs very serious internal modifications
> to work with user MT applications. As I can see so far it's
> not ever defined how it should be done so it's necessary to
> design such model from scratch.
> 16. it encodes/decodes messages using sprintf()/sscanf() functions
> what introduce some additional problems, i.e.:
> - this is very slow method of data encoding and decoding though
> in comparison to TCP/IP delay it's not critical
> - C compilers quite often provide *printf()/*scanf() functions
> in two version for ST and MT modes so for MT mode it's very important
> to use MT safe CRTL and leto code have to be always compiled with MT
> C compiler flags. We have our own hb_sprintf() function which is
> fully reentrant safe but we do not support hb_sscanf() so it's
> potential problem
> - it's very unsafe and hard to protect against buffer overflow and
> current leto code has a lot of potential buffer overflows which
> can cause server or client crash or hangup if user pass some data
> exploiting them
> - these functions operate on text data but LetoDB needs to transfer also
> binary data so programmers have to mix text and binary data what
> complicates message format and is source of some mistakes/bugs,
> now some messages are wrongly encoded what may cause server or
> client hangup

I did not look so deep

> 17. I do not see any documentation for the protocol, looks that each
> new message is added "as is" to client and server code, it means
> that even now it's hard to introduce extensions without careful
> checking of existing code to avoid possible conflicts,
> 18. I do not see any documentation which divides actions on client
> and server side so it's hard to add the extensions keeping
> original author vision, in the moment when such projects
> stops to be single author code such at least very basic
> documentation is a must, otherwise we will end with set of
> functions and RDD methods often replicating existing functionality
> because each developer tried to resolved his local problems in
> his own way, it also causes that bug fixing is very hard because
> developers seeing that sth is potentially wrong have to analyze
> whole code for possible workarounds in different places, i.e.
> I'm not able to verify everything what I see after looking at
> this code without investing day or two in deep analyzing

And that, in the harbour is not so?
I could not find documentation on the RDD (as in the rest of the
implementation of what either), but this does not prevent me to use
the harbour

> 19. it does not use hbsocket.h API so it may have portability problems
> and it does not address some problems which existed in original
> Giancarlo Inet* implementation and were copied to LetoDB
> 20. server uses internally own threads but does not use hbthread.h API
> what creates problems with portability and adding some extensions
> in the future, i.e. simultaneous executing .prg code by such threads
> is impossible now, it also blocks using HVM threads which could
> resolve important problems in current server implementation

Maybe, maybe not

> 21. I do not understand some parts of server code, i.e. there
> is an option to reuse existing tables between connections but
> why it was added? It cannot work without disabling most of
> server side processing because there is only single area
> structure so everything have to be shared between clients
> what creates series of problems, i.e. filter set by one client
> changes other client skip/seek operations, etc.
> I cannot even say that it is work in progress because it will
> never be fully functional option

Checked - everything is working properly

> 22. it keeps support for some old Harbour and xHarbour compilers
> what in few places makes the code more complicated and less
> readable then necessary, in some cases like own non HVM threads
> it seriously limits farther developing so such support have to be
> removed if we want to resolve all above problems
>
> best regards,
> Przemek

As a result, there is no alternative to ADS (DBFCDX use the network impossible)
and are forced to buy xHarbour (where it is more or less available)

All the above is my personal opinion

LetoDB - works (may not always correctly, but can be used)
I vote for inclusion in the harbour

Best regards,
Alexander


Response by Viktor

Date: Fri, 2 Oct 2009 20:07:25 +0200
From: Viktor Szak?ts <harbour.01@syenar.hu>

Hi Przemek and All,

Thanks for the insightful comments and detailed code analysis.

For me the conclusion for now is that while it would be nice
to "refactor" (hate the word) LetoDB code so that it 'becomes'
Harbour quality and most probably a viable FOSS ADS replacement,
this whole work would effectively done by you (Przemek), and
your POV and concerns are perfectly understandable to me. If
you won't work on it, I see not much point having it in our
repository. To me the code looks quite complicated to even
cleanup, so while I could contribute in this area, I'm not sure
it would give huge results/benefits in this case, since the
problems to solve are much deeper.

So overall, simply having a copy of LetoDB inside our contrib
area doesn't have a point.

However sad this is. I'd be also happy to have such "NETRDD",
but maybe it should be reached by some other means.

[ I'd be even happier with a GTNET driver though ;) Which,
seems much simpler to implement and even more efficient, but
only for CUI apps. ]

Brgds,
Viktor



Response by Pritpal Bedi

Date: Fri, 2 Oct 2009 14:47:39 -0700 (PDT)
From: Pritpal Bedi <bedipritpal@hotmail.com>

Hello Przemek

Przemyslaw Czerpak-2 wrote:
>
> This is what I can see after rather short code look up mostly on
> client side.
> In summary to resolve these problems we will have to rewrite most
> of current LetoDB code dropping xHarbour and old Harbour support.
> I haven't analyzed this code very deeply but seems that we can keep
> only very small part of existing LetoDB code.
> For me it is much easier to create new project with such functionality
> from scratch then trying to update current LetoDB code. For sure it
> will take less time and I will make much less mistakes so I do not
> find big motivation to work on it. I know myself and if I start to
> update it then I'll end with completely rewritten code but using the
> same project name and original authors' copyrights.
>
Absolutely correct assertion.
 

> I do not need such remote server for my own use - all my applications
> are executed fully on the server side so I do not want to reduce it
> to ADS like level.
>

The whole Harbour community knows about you and
your capabilities, and know it well.

Having said that, whatever you have contributed to Harbour till date
may be comprising a small part of your projects, but still you contributed,
contributed for the sake of creativity and a with a great sense to provide
a stable platform to your fellow developers, and we all feel indebted
for it, thank you.

> In the past I started to work on NETRDD and I documented the
> server and client part and defined basic communication protocol
> but so far I haven't time to finish it. Seeing what happens with
> my code in xHarbour I also do not find big motivation to finish
> it and public as free OpenSource project but I also do not have
> any precise plans about commercial releasing. Just simply I haven't
> taken the decision yet. Maybe I'll finish it and release as close
> source binary library which can be freely used with Harbour but
> only with applications which do not use any commercial libraries.
> Or maybe I'll invest much more time in it and try to write full
> RDBMS with transactions, SQL support and query optimizer and
> release it as commercial project. I do not know yet but I think
> I should inform you about it because maybe in the future I will
> not be on neutral position here so you should take your really
> own decision about LetoDB future in Harbour contrib code and my
> opinion and above text is only small help which you can ignore.
>

We know your concerns and which are right ones.
But still community is looking at you with great aspirations and
hopes to realize the dream of a remote rdd, which, only and only
you can bring to fruitation. It does not matter how you would
like to publish it, free open-source, free closed-source, or,
commercial open/closed-source, but it MATTERS that you just
start it and let us make optimistic about our futures.

Again, please note that, Harbour community is indebted
to your all great contributions...

Regards
Pritpal Bedi