Incubation progress, open issues in Wiki

[Arnulf Christl (OSGeo)]

Hello,
I wanted to send a friendly reminder of the open issues in the Wiki at:
http://wiki.osgeo.org/index.php/GRASS_Incubation_Progress#Open_Issues

As far as I can see activity wrt to the incubation process has stopped altogether, there have been no edits in the Wiki for 3 months. I have to report to the Incubation Committee in my function as mentor and they will again ask why incubation has stopped. What shall I tell them?

I did hope that we might get GRASS through before FOSS4G. Any chance? There is really not much left to do.

Best regards,
Arnulf.

[Markus Neteler-3]

Hi Arnulf,

On Tue, Sep 04, 2007 at 08:57:39AM +0200, Arnulf Christl wrote:
> Hello,
> I wanted to send a friendly reminder of the open issues in the Wiki at:
> http://wiki.osgeo.org/index.php/GRASS_Incubation_Progress#Open_Issues
>
> As far as I can see activity wrt to the incubation process has stopped
> altogether, there have been no edits in the Wiki for 3 months. I have to
> report to the Incubation Committee in my function as mentor and they will
> again ask why incubation has stopped. What shall I tell them?

that it didn't stop :) Indeed, we didn't update the Wiki as we
should. But currently we are replacing remaining "Numerical Receipes in C"
code with new functions.

Some file headers may need update, too, as above page indicates.

> I did hope that we might get GRASS through before FOSS4G. Any chance? There
> is really not much left to do.

That would be an excellent milestone.

Markus

> Best regards, Arnulf.
> _______________________________________________
> grass-psc mailing list
> [hidden email]
> http://grass.itc.it/mailman/listinfo/grass-psc

--
Markus Neteler  <neteler itc it>  http://mpa.itc.it/markus/
FBK-irst -  Centro per la Ricerca Scientifica e Tecnologica
MPBA - Predictive Models for Biol. & Environ. Data Analysis
Via Sommarive, 18        -       38050 Povo (Trento), Italy

------------------
ITC -> dall'1 marzo 2007 Fondazione Bruno Kessler
ITC -> since 1 March 2007 Fondazione Bruno Kessler
------------------

[Brad Douglas]

On Tue, 2007-09-04 at 10:05 +0200, Markus Neteler wrote:

> Hi Arnulf,
>
> On Tue, Sep 04, 2007 at 08:57:39AM +0200, Arnulf Christl wrote:
> > Hello,
> > I wanted to send a friendly reminder of the open issues in the Wiki at:
> > http://wiki.osgeo.org/index.php/GRASS_Incubation_Progress#Open_Issues
> >
> > As far as I can see activity wrt to the incubation process has stopped
> > altogether, there have been no edits in the Wiki for 3 months. I have to
> > report to the Incubation Committee in my function as mentor and they will
> > again ask why incubation has stopped. What shall I tell them?
>
> that it didn't stop :) Indeed, we didn't update the Wiki as we
> should. But currently we are replacing remaining "Numerical Receipes in C"
> code with new functions.

BTW, I located more NR code in GRASS.  The eigen and Jacobi routines use
them.  I've also discovered that the NR routines are inherently
unstable, but that also applies to most OSS solutions.

Both Soren and myself would like to replace the BLAS/LAPACK (Fortran
code) libraries with ATLAS, a tuned C version.  This also gets us away
from the Fortran issues suffered by gcc4.

If there are no objections, I would like to go ahead and modify
configure.in and include/ to reflect this.  Soren has already created a
good wrapper API for the functionality and has expanded it extensively.
I didn't note any objections when I queried the devel list, but I think
this is sufficiently large enough to be voted on by the PSC.

I really can't complete my imagery/ and lib/gmath API changes until
ATLAS (or some equivalent..maybe even GSL?) has replaced existing
BLAS/LAPACK.

> Some file headers may need update, too, as above page indicates.

I've been adding them to all files I've touched...and I've touched quite
a few lately.

> > I did hope that we might get GRASS through before FOSS4G. Any chance? There
> > is really not much left to do.
>
> That would be an excellent milestone.

Agreed.


--
73, de Brad KB8UYR/6 <rez touchofmadness com>

[Michael Barton]

I don't really know enough on this subject to realistically assess it
impact. However, what Brad suggests sounds very reasonable AFAICT

+0

Michael


On 9/4/07 7:55 PM, "Brad Douglas" <[hidden email]> wrote:

> On Tue, 2007-09-04 at 10:05 +0200, Markus Neteler wrote:
>> Hi Arnulf,
>>
>> On Tue, Sep 04, 2007 at 08:57:39AM +0200, Arnulf Christl wrote:
>>> Hello,
>>> I wanted to send a friendly reminder of the open issues in the Wiki at:
>>> http://wiki.osgeo.org/index.php/GRASS_Incubation_Progress#Open_Issues
>>>
>>> As far as I can see activity wrt to the incubation process has stopped
>>> altogether, there have been no edits in the Wiki for 3 months. I have to
>>> report to the Incubation Committee in my function as mentor and they will
>>> again ask why incubation has stopped. What shall I tell them?
>>
>> that it didn't stop :) Indeed, we didn't update the Wiki as we
>> should. But currently we are replacing remaining "Numerical Receipes in C"
>> code with new functions.
>
> BTW, I located more NR code in GRASS.  The eigen and Jacobi routines use
> them.  I've also discovered that the NR routines are inherently
> unstable, but that also applies to most OSS solutions.
>
> Both Soren and myself would like to replace the BLAS/LAPACK (Fortran
> code) libraries with ATLAS, a tuned C version.  This also gets us away
> from the Fortran issues suffered by gcc4.
>
> If there are no objections, I would like to go ahead and modify
> configure.in and include/ to reflect this.  Soren has already created a
> good wrapper API for the functionality and has expanded it extensively.
> I didn't note any objections when I queried the devel list, but I think
> this is sufficiently large enough to be voted on by the PSC.
>
> I really can't complete my imagery/ and lib/gmath API changes until
> ATLAS (or some equivalent..maybe even GSL?) has replaced existing
> BLAS/LAPACK.
>
>> Some file headers may need update, too, as above page indicates.
>
> I've been adding them to all files I've touched...and I've touched quite
> a few lately.
>
>>> I did hope that we might get GRASS through before FOSS4G. Any chance? There
>>> is really not much left to do.
>>
>> That would be an excellent milestone.
>
> Agreed.
>

__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: [phone]
fax: [phone]
www: http://www.public.asu.edu/~cmbarton

[Helena Mitasova]

Brad, Soeren

would it be possible to create a list of modules that would be  
affected by this change and then we should do a thorough testing on  
spearfish and nc_sample data set and ask others to test it with their  
data (including all kinds of unusual applications that they may have,  
large data sets, different scales, xy and latlong,etc.), especially  
compare the results run with the current solvers and the new ones.  
You are right that this is a significant and important change so it  
needs a lot of testing to make sure that we don't get unintended  
consequences. I wish we did such comparison and testing before the  
sites->vector points change.

Soeren - could you email to the dev list or at least to PSC the  
comparison of different solver when used with v.surf.rst? (if you  
don't have it on-hand - I can post it - it may be even useful to put  
it on a related wiki page - it is a useful reminder how much  
difference there is for different solvers).

Helena


Helena Mitasova
Dept. of Marine, Earth and Atm. Sciences
1125 Jordan Hall, NCSU Box 8208,
Raleigh NC 27695
http://skagit.meas.ncsu.edu/~helena/



On Sep 4, 2007, at 10:55 PM, Brad Douglas wrote:

> On Tue, 2007-09-04 at 10:05 +0200, Markus Neteler wrote:
>> Hi Arnulf,
>>
>> On Tue, Sep 04, 2007 at 08:57:39AM +0200, Arnulf Christl wrote:
>>> Hello,
>>> I wanted to send a friendly reminder of the open issues in the  
>>> Wiki at:
>>> http://wiki.osgeo.org/index.php/ 
>>> GRASS_Incubation_Progress#Open_Issues
>>>
>>> As far as I can see activity wrt to the incubation process has  
>>> stopped
>>> altogether, there have been no edits in the Wiki for 3 months. I  
>>> have to
>>> report to the Incubation Committee in my function as mentor and  
>>> they will
>>> again ask why incubation has stopped. What shall I tell them?
>>
>> that it didn't stop :) Indeed, we didn't update the Wiki as we
>> should. But currently we are replacing remaining "Numerical  
>> Receipes in C"
>> code with new functions.
>
> BTW, I located more NR code in GRASS.  The eigen and Jacobi  
> routines use
> them.  I've also discovered that the NR routines are inherently
> unstable, but that also applies to most OSS solutions.
>
> Both Soren and myself would like to replace the BLAS/LAPACK (Fortran
> code) libraries with ATLAS, a tuned C version.  This also gets us away
> from the Fortran issues suffered by gcc4.
>
> If there are no objections, I would like to go ahead and modify
> configure.in and include/ to reflect this.  Soren has already  
> created a
> good wrapper API for the functionality and has expanded it  
> extensively.
> I didn't note any objections when I queried the devel list, but I  
> think
> this is sufficiently large enough to be voted on by the PSC.
>
> I really can't complete my imagery/ and lib/gmath API changes until
> ATLAS (or some equivalent..maybe even GSL?) has replaced existing
> BLAS/LAPACK.
>
>> Some file headers may need update, too, as above page indicates.
>
> I've been adding them to all files I've touched...and I've touched  
> quite
> a few lately.
>
>>> I did hope that we might get GRASS through before FOSS4G. Any  
>>> chance? There
>>> is really not much left to do.
>>
>> That would be an excellent milestone.
>
> Agreed.
>
>
> --
> 73, de Brad KB8UYR/6 <rez touchofmadness com>
>
> _______________________________________________
> grass-psc mailing list
> [hidden email]
> http://grass.itc.it/mailman/listinfo/grass-psc

[Soeren Gebbert-2]

Hello,

2007/9/5, Helena Mitasova <[hidden email]>:

Brad, Soeren

would it be possible to create a list of modules that would be
affected by this change and then we should do a thorough testing on
spearfish and nc_sample data set and ask others to test it with their
data (including all kinds of unusual applications that they may have,
large data sets, different scales, xy and latlong,etc.), especially
compare the results run with the current solvers and the new ones.
You are right that this is a significant and important change so it
needs a lot of testing to make sure that we don't get unintended
consequences. I wish we did such comparison and testing before the
sites->vector points change.

AFAICT the changes we want to make will effect the gmath, gpde libraries and
every module which uses the gmath lapack interface as well as some gmath functions.

I will move the linear equation solvers from the gpde library into the gmath library.
And i will establish a unit and integration test structure (like in the gpde library) to test all of the
gmath functions we will implement. And if i have time, i will implement some benchmarks.

Modules which  will be affected (as far as i know):
* v.surf.rst
* v.surf.random
* r.gwflow
* r3.gwflow
* r.surf.fractal
* r.surf.gauss
* r.surf.random
* Maybe the lidar modules, but im not quite sure
* i.gensigset ? --> uses G_alloc_matrix
* i.smap ? --> uses G_alloc_matrix
* i.cca ? --> includes gmath.h
* i.pca ? --> inclues gmath.h
* i.fft ? --> includes gmath.h
* i.ifft ? --> includes gmath.h
* i.zc ? --> includes gmath.h
* v.kernel ? --> includes gmath.h

* v.generalize (will be changed to use the much faster cholesky decomposition with bandwith optimization)

This list is incomplete.

Soeren - could you email to the dev list or at least to PSC the
comparison of different solver when used with v.surf.rst? (if you
don't have it on-hand - I can post it - it may be even useful to put
it on a related wiki page - it is a useful reminder how much
difference there is for different solvers).

I'm sorry, but i lost the mail i had send you, can you please post it for me?

Best regards
Soeren

Helena


Helena Mitasova
Dept. of Marine, Earth and Atm. Sciences
1125 Jordan Hall, NCSU Box 8208,
Raleigh NC 27695
http://skagit.meas.ncsu.edu/~helena/



On Sep 4, 2007, at 10:55 PM, Brad Douglas wrote:

> On Tue, 2007-09-04 at 10:05 +0200, Markus Neteler wrote:
>> Hi Arnulf,
>>
>> On Tue, Sep 04, 2007 at 08:57:39AM +0200, Arnulf Christl wrote:
>>> Hello,
>>> I wanted to send a friendly reminder of the open issues in the
>>> Wiki at:
>>> http://wiki.osgeo.org/index.php/
>>> GRASS_Incubation_Progress#Open_Issues
>>>
>>> As far as I can see activity wrt to the incubation process has
>>> stopped
>>> altogether, there have been no edits in the Wiki for 3 months. I
>>> have to
>>> report to the Incubation Committee in my function as mentor and
>>> they will
>>> again ask why incubation has stopped. What shall I tell them?
>>

>> that it didn't stop :) Indeed, we didn't update the Wiki as we

>> should. But currently we are replacing remaining "Numerical
>> Receipes in C"
>> code with new functions.
>
> BTW, I located more NR code in GRASS.  The eigen and Jacobi
> routines use
> them.  I've also discovered that the NR routines are inherently
> unstable, but that also applies to most OSS solutions.
>
> Both Soren and myself would like to replace the BLAS/LAPACK (Fortran
> code) libraries with ATLAS, a tuned C version.  This also gets us away
> from the Fortran issues suffered by gcc4.
>
> If there are no objections, I would like to go ahead and modify
> configure.in and include/ to reflect this.  Soren has already
> created a
> good wrapper API for the functionality and has expanded it
> extensively.
> I didn't note any objections when I queried the devel list, but I
> think
> this is sufficiently large enough to be voted on by the PSC.
>
> I really can't complete my imagery/ and lib/gmath API changes until
> ATLAS (or some equivalent..maybe even GSL?) has replaced existing
> BLAS/LAPACK.
>
>> Some file headers may need update, too, as above page indicates.
>
> I've been adding them to all files I've touched...and I've touched
> quite
> a few lately.
>
>>> I did hope that we might get GRASS through before FOSS4G. Any
>>> chance? There
>>> is really not much left to do.
>>
>> That would be an excellent milestone.
>
> Agreed.
>
>
> --
> 73, de Brad KB8UYR/6 <rez touchofmadness com>
>
> _______________________________________________
> grass-psc mailing list
> [hidden email]
> http://grass.itc.it/mailman/listinfo/grass-psc

[Brad Douglas]

On Tue, 2007-09-04 at 23:29 -0400, Helena Mitasova wrote:

> Brad, Soeren
>
> would it be possible to create a list of modules that would be  
> affected by this change and then we should do a thorough testing on  
> spearfish and nc_sample data set and ask others to test it with their  
> data (including all kinds of unusual applications that they may have,  
> large data sets, different scales, xy and latlong,etc.), especially  
> compare the results run with the current solvers and the new ones.  
> You are right that this is a significant and important change so it  
> needs a lot of testing to make sure that we don't get unintended  
> consequences. I wish we did such comparison and testing before the  
> sites->vector points change.

Modules that are immediately affected:
- i.gensig
- i.pca
- i.cca (I have rewritten to remove band limitations by dynamically
allocating memory)
- i.spec.sam/i.spec.unmix (not currently in CVS)

One thing we have not discussed yet is possibly merging lib/gmath and
lib/gpde.  Right now, we should probably leave it alone.  I haven't
checked to see what all uses lib/gpde.

> Soeren - could you email to the dev list or at least to PSC the  
> comparison of different solver when used with v.surf.rst? (if you  
> don't have it on-hand - I can post it - it may be even useful to put  
> it on a related wiki page - it is a useful reminder how much  
> difference there is for different solvers).

Soren has shown me statistics with and without using the LU solver.
They are fairly impressive, IIRC.


--
Brad Douglas <rez touchofmadness com>                    KB8UYR/6
Address: 37.493,-121.924 / WGS84    National Map Corps #TNMC-3785

[HamishB]

Brad Douglas wrote:
> Both Soren and myself would like to replace the BLAS/LAPACK (Fortran
> code) libraries with ATLAS, a tuned C version.  This also gets us away
> from the Fortran issues suffered by gcc4.
>
> If there are no objections, I would like to go ahead and modify
> configure.in and include/ to reflect this.
..
> One thing we have not discussed yet is possibly merging lib/gmath and
> lib/gpde.  Right now, we should probably leave it alone.  I haven't
> checked to see what all uses lib/gpde.

Hi,

IMO important technical decisions such as this need to happen over on
the grass-dev mailing list to allow contributions from all devels and
a single source for future thread archive searches.

and FWIW, fortran90 support in the latest gcc4 releases seems to be much
improved over earlier versions. (but AFAIK grass has ever only linked to
f77 stuff)


Hamish

[Brad Douglas]

On Thu, 2007-09-06 at 15:24 +1200, Hamish wrote:

> Brad Douglas wrote:
> > Both Soren and myself would like to replace the BLAS/LAPACK (Fortran
> > code) libraries with ATLAS, a tuned C version.  This also gets us away
> > from the Fortran issues suffered by gcc4.
> >
> > If there are no objections, I would like to go ahead and modify
> > configure.in and include/ to reflect this.
> ..
> > One thing we have not discussed yet is possibly merging lib/gmath and
> > lib/gpde.  Right now, we should probably leave it alone.  I haven't
> > checked to see what all uses lib/gpde.
>
> Hi,
>
> IMO important technical decisions such as this need to happen over on
> the grass-dev mailing list to allow contributions from all devels and
> a single source for future thread archive searches.

I've already discussed most of this without objection on the dev list.
cc'ing this email.

> and FWIW, fortran90 support in the latest gcc4 releases seems to be much
> improved over earlier versions. (but AFAIK grass has ever only linked to
> f77 stuff)

That requires that BLAS/LAPACK be ported to f90.  BTW, I have created a
work-around that allows gcc4 to still use f77.

Regardless, BLAS/LAPACK needs better definition in autoconf.  I have
autoconf code that allows us to pick and close the BLAS/LAPACK
implementation that best suits our needs.  That could be traditional f77
BLAS/LAPACK, ATLAS, SciLAPACK, etc.

This gives the person compiling the ability to "tune" to their
application (cluster, SMP, SMP cluster).

Unfortunately, I haven't been able to test it (autoreconf issues).


--
73, de Brad KB8UYR/6 <rez touchofmadness com>