Future of LucidDB (LucidEra shutting down)

13 messages Options
Embed this post
Permalink
O G

Future of LucidDB (LucidEra shutting down)

Reply Threaded More More options
Print post
Permalink

Hello,

I just heard LucidEra is shutting down.  Since LucidEra was a LucidDB sponsor, I wonder what this means for the future of this project?

Thank you and sorry for the possibly touchy question.

Otis



     

------------------------------------------------------------------------------
_______________________________________________
luciddb-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/luciddb-users
John Sichi

Re: Future of LucidDB (LucidEra shutting down)

Reply Threaded More More options
Print post
Permalink
O G wrote:
> Hello,
>
> I just heard LucidEra is shutting down.  Since LucidEra was a LucidDB
> sponsor, I wonder what this means for the future of this project?
>
> Thank you and sorry for the possibly touchy question.

Hi Otis and everyone,

Coincidentally, I was about to send out a message on this topic, so I'm
glad you asked.  LucidDB is the only open source DBMS built from the
ground up specifically for data warehousing, and it is important to many
people for an option like that to remain available.

 From its inception, LucidDB has had dual sponsorship from both LucidEra
(a startup) and The Eigenbase Project, a non-profit.  Sponsorship from
Eigenbase continues unchanged, and the project as a whole has certainly
grown beyond the seeding from those sponsors into a community which
includes all of you.  So although the LucidEra news is certainly a loss
for the project, I'm confident that the tree is already healthy and
thriving on its own and will continue to flourish without the aid of one
of the original gardeners, if you'll pardon the cliche.

In fact, I actually left LucidEra last October, but have continued to
put most of my time into LucidDB feature development, release, and
promotion; and I intend to keep that up.  So from a project leadership
perspective, LucidDB has already been independent of LucidEra for quite
some time.  Other developers previously at LucidEra remain involved as
well.  In addition, a sister project for parallelization and clustering
is being incubated at firewater.sourceforge.net; this will take the
technology underlying LucidDB to the next level of scalability.

Because LucidDB is built on the Eigenbase platform, it automatically
inherits improvements contributed by other participants in that
ecosystem (such as SQLstream, which has quite a few developers working
on it--this includes Julian Hyde, who is also the Mondrian project
founder, and who has been helping to promote LucidDB as an ideal DB for
OLAP).  Julian has actually contributed directly to the optimizer used
by LucidDB in a number of cases.

We are also working on growing the Eigenbase ecosystem in the wake of
the Oracle acquisition of Sun/MySQL--storage engine vendors are looking
for new integration options in order to maintain their independence, and
Eigenbase could be a good fit, so some of them have approached us
already, and we're prototyping potential solutions.

This transition time actually opens wide a great opportunity for
community members to step up and dig deeper into LucidDB development and
promotion.  We already have a few community-initiated investigations
underway into areas such as named sequence support, Hibernate dialect,
and C++ client connectivity, and I'd be very encouraged to see more of
those in areas such as packaging and usability improvements.

So...let me flip the question around and ask you, how would you like to
be involved in the future of LucidDB to ensure that it continues to
thrive?  Do you have ideas and resources you can contribute?   What
changes would you like to see?  A developer mailing list?  Source code
replicated from Perforce to Subversion/git for easier access (I'm
working on this one)?  Let's hear it...

Thanks for your support,
JVS

p.s. We'll be updating some of the web pages to reflect the impact of
the news.

------------------------------------------------------------------------------
_______________________________________________
luciddb-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/luciddb-users
Francisco Reyes

Re: Future of LucidDB (LucidEra shutting down)

Reply Threaded More More options
Print post
Permalink
John V. Sichi writes:

> Hi Otis and everyone,

Thanks for the update.

> well.  In addition, a sister project for parallelization and clustering
> is being incubated at firewater.sourceforge.net; this will take the
> technology underlying LucidDB to the next level of scalability.

Is that independant of LucidDB or will be something that will run on top of
LucidDB?

Parallelization and clustering are two of the features that currently have
me looking at other possible alternatives to LucidDB.
 
> and C++ client connectivity,

I am in the benchmarking stages right now, but even if Lucid comes back with
great numbers the fact that it has so little connectivity options will be a
big issue. The company I am testing Lucid for is a php shop and there is
very little interest from the top tech guy (aka CTO) to have anything else.
 
> So...let me flip the question around and ask you, how would you like to
> be involved in the future of LucidDB to ensure that it continues to
> thrive?

Personally I can contribute documentation.
For sure I plan to do  the Postgresql how to and put all the benchmarks on
the list (or wiki if considered usefull enough) once I am done with all my
testing.

>Do you have ideas and resources you can contribute?

It seems as if firewater may be where I actually may want to be.. is either
that or looking into one of the many Hadoop based column databases out
there.

A single machine solution can only scale so much.
Also we are considering to use several machines using RAID 0 so a
multi-machine solution is where we want to be.

> changes would you like to see?

Usability is one issue that I have seen so far to be a issue for me, but
documentation can go a long way.

>A developer mailing list?

I think that is always good.

Even though I have yet to finish my benchmarks (only have a slowish..
machine to do my testing and the dataset has 600+ million rows) I have been
very impressed on the level of details on the architecture docs. Documents
like the storage document on the wiki are just great.

Other than PHP connectivity there are a number of small details that I think
may be an issue for me to recommend Lucid.

* The fact that the table never shrinks (can give details if anyone cares).

* Have not seen a table rename function.

* As far as I can tell there are no real transactions. It seems
individual commands are wrapped within a transaction, but as far as I can
tell one can not have a begin/commit and have multiple commands.

* Lack of more connectivity options (in our case specially PHP).  

* Size of the community (this is mostly an issue for my boss). If memory
serves me well I think there are about 50 people in the list and rarely any
posts anything so it makes me wonder how many of those 50 or so people are
actually active in the list.

------------------------------------------------------------------------------
_______________________________________________
luciddb-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/luciddb-users
Pedro Alves-2

Re: Future of LucidDB (LucidEra shutting down)

Reply Threaded More More options
Print post
Permalink
In reply to this post by John Sichi

> Coincidentally, I was about to send out a message on this topic, so I'm
> glad you asked.  LucidDB is the only open source DBMS built from the
> ground up specifically for data warehousing, and it is important to many
> people for an option like that to remain available.


Thanks for the input, John.


It's very reassuring to read this and to know that lucid has already passed
the point of no return in terms of sponsorship dependency.


I wish much success to this project and intend to help in the ways I can.


-pedro

--
Pedro Alves
pmgalves-at-gmail.com

------------------------------------------------------------------------------
_______________________________________________
luciddb-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/luciddb-users
John Sichi

Re: Future of LucidDB (LucidEra shutting down)

Reply Threaded More More options
Print post
Permalink
Pedro Alves wrote:
> It's very reassuring to read this and to know that lucid has already passed
> the point of no return in terms of sponsorship dependency.
>
>
> I wish much success to this project and intend to help in the ways I can.

Thanks Pedro.  Hope things are going well with the mozilla stats
project--I was happy to see you tweet some good numbers relative to MySQL.

JVS

------------------------------------------------------------------------------
_______________________________________________
luciddb-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/luciddb-users
Pedro Alves-2

Re: Future of LucidDB (LucidEra shutting down)

Reply Threaded More More options
Print post
Permalink
On Wed, Jun 24, 2009 at 07:01:58PM -0700, John V. Sichi wrote:

> Pedro Alves wrote:
> >It's very reassuring to read this and to know that lucid has already passed
> >the point of no return in terms of sponsorship dependency.
> >
> >
> >I wish much success to this project and intend to help in the ways I can.
>
> Thanks Pedro.  Hope things are going well with the mozilla stats
> project--I was happy to see you tweet some good numbers relative to MySQL.
>
> JVS


The tests looked very promising! It's not live yet - next month will be -
and I'll sure blog about it.



--
Pedro Alves
pmgalves-at-gmail.com

------------------------------------------------------------------------------
_______________________________________________
luciddb-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/luciddb-users
John Sichi

Re: Future of LucidDB (LucidEra shutting down)

Reply Threaded More More options
Print post
Permalink
In reply to this post by Francisco Reyes
Francisco Reyes wrote:
> Thanks for the update.

Good feedback and questions...answers inline below

> Is that independant of LucidDB or will be something that will run on top
> of LucidDB?
>
> Parallelization and clustering are two of the features that currently
> have me looking at other possible alternatives to LucidDB.

Firewater is basically a frontplane distributed query node running on
top of multiple LucidDB backplane storage nodes, but it will be packaged
in such a way that it all looks like Firewater (i.e. you would not want
to access the LucidDB nodes directly except for debugging purposes).

We're keeping the two projects separate for now so that the inherent
complexity of a distributed system does not creep from Firewater into
LucidDB (which will stay as single-node for now).  If it makes sense
later, they'll merge into one.

Most enhancements to LucidDB will flow into Firewater automatically, and
some of the optimizer and parallel executor work going in for Firewater
will be something that LucidDB can take advantage of as well.

>> and C++ client connectivity,
>
> I am in the benchmarking stages right now, but even if Lucid comes back
> with great numbers the fact that it has so little connectivity options
> will be a big issue. The company I am testing Lucid for is a php shop
> and there is very little interest from the top tech guy (aka CTO) to
> have anything else.

Someone on this list was attemping to use Continuent's Tungsten
MySQL-client-to-JDBC-server bridge for this purpose--it seems like a
good approach if it works.

> * The fact that the table never shrinks (can give details if anyone cares).

ALTER TABLE REBUILD is required for purging the deleted rows (followed
by ALTER SYSTEM DEALLOCATE OLD for purging the deallocated pages).  But
if you mean the db.dat file, correct.

> * Have not seen a table rename function.

Correct, rename is not currently supported for stored tables and the
syntax is undocumented for other object types.

> * As far as I can tell there are no real transactions. It seems
> individual commands are wrapped within a transaction, but as far as I
> can tell one can not have a begin/commit and have multiple commands.

This is correct.  User-level transactions would be nice--however, if you
look at the way warehouse labels work, you may find that they are
actually much more useful for a typical ETL process running concurrently
with OLAP or DSS, and I don't think you'll find this feature anywhere else:

http://pub.eigenbase.org/wiki/LucidDbWarehouseLabels

JVS

------------------------------------------------------------------------------
_______________________________________________
luciddb-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/luciddb-users
Nicholas Goodman

Re: Future of LucidDB (LucidEra shutting down)

Reply Threaded More More options
Print post
Permalink
In reply to this post by John Sichi

> From its inception, LucidDB has had dual sponsorship from both  
> LucidEra
> (a startup) and The Eigenbase Project, a non-profit.  Sponsorship from
> Eigenbase continues unchanged, and the project as a whole has  
> certainly

John (et al)

Should we ensure that legal LucidEra specific LucidDB hooks are  
properly transitioned to Eigenbase while the organization/entity still  
exists?

In other words, is there a way that Eigenbase can:
  - Arrange for full copyright ownership of the "red zone" LucidDB  
code so the foundation isn't encumbered for licensing/distributing the  
software as it sees fit?
  - Transfer registration/ownership of the LucidDb trademark to the  
foundation?

Bayon can contribute financially towards whatever legal fees might be  
incurred to make this happen.  Being familiar with the way these  
things work it's SOOOO much easier to do it before LucidEra as an  
organization dissolves.

I'd just hate to have thought of this 6 months too late; it just seems  
that Eigenbase should now be the official guardian of LucidDB  
copyright/trademark.

Thoughts?

Nick

------------------------------------------------------------------------------
_______________________________________________
luciddb-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/luciddb-users
John Sichi

Re: Future of LucidDB (LucidEra shutting down)

Reply Threaded More More options
Print post
Permalink
Nicholas Goodman wrote:

> Should we ensure that legal LucidEra specific LucidDB hooks are properly
> transitioned to Eigenbase while the organization/entity still exists?
>
> In other words, is there a way that Eigenbase can:
>  - Arrange for full copyright ownership of the "red zone" LucidDB code
> so the foundation isn't encumbered for licensing/distributing the
> software as it sees fit?
>  - Transfer registration/ownership of the LucidDb trademark to the
> foundation?
>
> Bayon can contribute financially towards whatever legal fees might be
> incurred to make this happen.  Being familiar with the way these things
> work it's SOOOO much easier to do it before LucidEra as an organization
> dissolves.
>
> I'd just hate to have thought of this 6 months too late; it just seems
> that Eigenbase should now be the official guardian of LucidDB
> copyright/trademark.

Hey Nick,

Regarding the copyright, I'd just like to make sure that everyone on the
list knows that Eigenbase (by founding agreement with LucidEra) has
joint copyright on all of the LucidDB code, and this ensures that
Eigenbase continues as the guardian entity for that copyright, and has
the full right to continue making GPL releases.

As you mention, what is restricted (on certain parts of the code,
"yellow zone" actually) by the agreement with LucidEra is the right to
non-GPL licensing.  Acting as Eigenbase president and director, I'm in
contact with the management company dealing with LucidEra's assets, and
will follow up on this with the lawyer who did the orginal IP work to
see what the possibilities are here; the disposition may depend on
whether and how those assets are acquired by other parties.

Regarding the trademark, I raised that with the management company, and
they said that unless it is acquired along with other IP, they would
agree to transfer it to Eigenbase for a nominal fee.

Thanks a lot for the offer of assisting with legal fees; I'll let you
know how matters progress.

JVS

------------------------------------------------------------------------------
_______________________________________________
luciddb-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/luciddb-users
Nicolae Mihalache

Re: Future of LucidDB (LucidEra shutting down)

Reply Threaded More More options
Print post
Permalink
In reply to this post by Francisco Reyes
I would also need php connectivity, so we are two in this boat :)
What would be the best way to accomplish this, implementing some ODBC
driver in C?

This page, http://pub.eigenbase.org/wiki/LucidDbNonJavaClients, talks
about "Http Calls" performed by VJDBC. From a quick look, I think VJDBC
transports java serialized objects over http so to use this protocol
from another language, one has to implement java serialization. It is
perhaps not the best way.

mache


Francisco Reyes wrote:

> John V. Sichi writes:
>
>  
>> Hi Otis and everyone,
>>    
>
> Thanks for the update.
>
>  
>> well.  In addition, a sister project for parallelization and clustering
>> is being incubated at firewater.sourceforge.net; this will take the
>> technology underlying LucidDB to the next level of scalability.
>>    
>
> Is that independant of LucidDB or will be something that will run on top of
> LucidDB?
>
> Parallelization and clustering are two of the features that currently have
> me looking at other possible alternatives to LucidDB.
>  
>  
>> and C++ client connectivity,
>>    
>
> I am in the benchmarking stages right now, but even if Lucid comes back with
> great numbers the fact that it has so little connectivity options will be a
> big issue. The company I am testing Lucid for is a php shop and there is
> very little interest from the top tech guy (aka CTO) to have anything else.
>  
>  
>> So...let me flip the question around and ask you, how would you like to
>> be involved in the future of LucidDB to ensure that it continues to
>> thrive?
>>    
>
> Personally I can contribute documentation.
> For sure I plan to do  the Postgresql how to and put all the benchmarks on
> the list (or wiki if considered usefull enough) once I am done with all my
> testing.
>
>  
>> Do you have ideas and resources you can contribute?
>>    
>
> It seems as if firewater may be where I actually may want to be.. is either
> that or looking into one of the many Hadoop based column databases out
> there.
>
> A single machine solution can only scale so much.
> Also we are considering to use several machines using RAID 0 so a
> multi-machine solution is where we want to be.
>
>  
>> changes would you like to see?
>>    
>
> Usability is one issue that I have seen so far to be a issue for me, but
> documentation can go a long way.
>
>  
>> A developer mailing list?
>>    
>
> I think that is always good.
>
> Even though I have yet to finish my benchmarks (only have a slowish..
> machine to do my testing and the dataset has 600+ million rows) I have been
> very impressed on the level of details on the architecture docs. Documents
> like the storage document on the wiki are just great.
>
> Other than PHP connectivity there are a number of small details that I think
> may be an issue for me to recommend Lucid.
>
> * The fact that the table never shrinks (can give details if anyone cares).
>
> * Have not seen a table rename function.
>
> * As far as I can tell there are no real transactions. It seems
> individual commands are wrapped within a transaction, but as far as I can
> tell one can not have a begin/commit and have multiple commands.
>
> * Lack of more connectivity options (in our case specially PHP).  
>
> * Size of the community (this is mostly an issue for my boss). If memory
> serves me well I think there are about 50 people in the list and rarely any
> posts anything so it makes me wonder how many of those 50 or so people are
> actually active in the list.
>
> ------------------------------------------------------------------------------
> _______________________________________________
> luciddb-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/luciddb-users
>  


------------------------------------------------------------------------------
_______________________________________________
luciddb-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/luciddb-users
Luke Ehresman

PHP connectivity

Reply Threaded More More options
Print post
Permalink
We would love to see PHP connectivity as well.  Currently we have a  
hacked solution as we developed a middleware application in Java.  It  
acts as a proxy accepting requests from PHP and brokering them out to  
Lucid over JDBC.  It works, but it's another level of complexity in  
our architecture that would be nice to eliminate.

Luke

On Jun 25, 2009, at 6:46 AM, Nicolae Mihalache wrote:

> I would also need php connectivity, so we are two in this boat :)
> What would be the best way to accomplish this, implementing some ODBC
> driver in C?
>
> This page, http://pub.eigenbase.org/wiki/LucidDbNonJavaClients, talks
> about "Http Calls" performed by VJDBC. From a quick look, I think  
> VJDBC
> transports java serialized objects over http so to use this protocol
> from another language, one has to implement java serialization. It is
> perhaps not the best way.
>
> mache
>
>
> Francisco Reyes wrote:
>> John V. Sichi writes:
>>
>>
>>> Hi Otis and everyone,
>>>
>>
>> Thanks for the update.
>>
>>
>>> well.  In addition, a sister project for parallelization and  
>>> clustering
>>> is being incubated at firewater.sourceforge.net; this will take the
>>> technology underlying LucidDB to the next level of scalability.
>>>
>>
>> Is that independant of LucidDB or will be something that will run  
>> on top of
>> LucidDB?
>>
>> Parallelization and clustering are two of the features that  
>> currently have
>> me looking at other possible alternatives to LucidDB.
>>
>>
>>> and C++ client connectivity,
>>>
>>
>> I am in the benchmarking stages right now, but even if Lucid comes  
>> back with
>> great numbers the fact that it has so little connectivity options  
>> will be a
>> big issue. The company I am testing Lucid for is a php shop and  
>> there is
>> very little interest from the top tech guy (aka CTO) to have  
>> anything else.
>>
>>
>>> So...let me flip the question around and ask you, how would you  
>>> like to
>>> be involved in the future of LucidDB to ensure that it continues to
>>> thrive?
>>>
>>
>> Personally I can contribute documentation.
>> For sure I plan to do  the Postgresql how to and put all the  
>> benchmarks on
>> the list (or wiki if considered usefull enough) once I am done with  
>> all my
>> testing.
>>
>>
>>> Do you have ideas and resources you can contribute?
>>>
>>
>> It seems as if firewater may be where I actually may want to be..  
>> is either
>> that or looking into one of the many Hadoop based column databases  
>> out
>> there.
>>
>> A single machine solution can only scale so much.
>> Also we are considering to use several machines using RAID 0 so a
>> multi-machine solution is where we want to be.
>>
>>
>>> changes would you like to see?
>>>
>>
>> Usability is one issue that I have seen so far to be a issue for  
>> me, but
>> documentation can go a long way.
>>
>>
>>> A developer mailing list?
>>>
>>
>> I think that is always good.
>>
>> Even though I have yet to finish my benchmarks (only have a slowish..
>> machine to do my testing and the dataset has 600+ million rows) I  
>> have been
>> very impressed on the level of details on the architecture docs.  
>> Documents
>> like the storage document on the wiki are just great.
>>
>> Other than PHP connectivity there are a number of small details  
>> that I think
>> may be an issue for me to recommend Lucid.
>>
>> * The fact that the table never shrinks (can give details if anyone  
>> cares).
>>
>> * Have not seen a table rename function.
>>
>> * As far as I can tell there are no real transactions. It seems
>> individual commands are wrapped within a transaction, but as far as  
>> I can
>> tell one can not have a begin/commit and have multiple commands.
>>
>> * Lack of more connectivity options (in our case specially PHP).
>>
>> * Size of the community (this is mostly an issue for my boss). If  
>> memory
>> serves me well I think there are about 50 people in the list and  
>> rarely any
>> posts anything so it makes me wonder how many of those 50 or so  
>> people are
>> actually active in the list.
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> luciddb-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/luciddb-users
>>
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> luciddb-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/luciddb-users


------------------------------------------------------------------------------
_______________________________________________
luciddb-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/luciddb-users
Francisco Reyes

Re: PHP connectivity

Reply Threaded More More options
Print post
Permalink
Luke Ehresman writes:

> We would love to see PHP connectivity as well.  Currently we have a  
> hacked solution as we developed a middleware application in Java.  

Was that done by your organisation?
Would it be possible to share it?


------------------------------------------------------------------------------
_______________________________________________
luciddb-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/luciddb-users
John Sichi

Re: PHP connectivity

Reply Threaded More More options
Print post
Permalink
In reply to this post by Luke Ehresman
Luke Ehresman wrote:
> We would love to see PHP connectivity as well.  Currently we have a
> hacked solution as we developed a middleware application in Java.  It
> acts as a proxy accepting requests from PHP and brokering them out to
> Lucid over JDBC.  It works, but it's another level of complexity in our
> architecture that would be nice to eliminate.

Looks like there's a generic bridge here:

http://php-java-bridge.sourceforge.net/pjb

I'm supposed to meet with someone from Continuent, so I'll look deeper
into how their MySQL-client-to-JDBC bridging works too.

JVS

------------------------------------------------------------------------------
_______________________________________________
luciddb-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/luciddb-users