New implementation of Time stretching/Pitch scaling

20 Messages Forum Options Options
Embed this topic
Permalink
Clayton Otey-2
New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options
Print post
Permalink
I've (re)written a library called sbsms (on sourceforge) which does high quality time stretching and pitch scaling.  I'll be working on adding a "Time Scale" effect to augment the features of "Change Tempo" and "Change Pitch" with higher quality (but slower) and sliding time stretches/pitch scales.  Any comments/suggestions?  Would such a patch likely make it into the next release?

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
audacity-devel mailing list
audacity-devel@...
https://lists.sourceforge.net/lists/listinfo/audacity-devel
Richard Ash (audacity-help)
Re: New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options
Print post
Permalink
On Fri, 2008-10-03 at 13:50 -0700, Clayton Otey wrote:
> I've (re)written a library called sbsms (on sourceforge) which does
> high quality time stretching and pitch scaling.  I'll be working on
> adding a "Time Scale" effect to augment the features of "Change Tempo"
> and "Change Pitch" with higher quality (but slower) and sliding time
> stretches/pitch scales.  Any comments/suggestions?  Would such a patch
> likely make it into the next release?

In principal it would be interesting to have an alternative to
soundtouch available in audacity, because we know there are some cases
when it doesn't work well. However there are some issues with the code
on sourceforge as it stands:

* Practical. You don't seem to have any build system in place for
Windows, which is where a large proportion of our user base is. In order
to integrate anything into Audacity, we would need to be able to compile
sbsms with Microsoft's Visual Studio Express tools, which can be a
problem if your code contains many C99 features (they still haven't
managed to implement it, 10 years on).

The most recent release tarball is also a bit flaky. It doesn't unpack
to a directory, thus filling wherever you unpack it to with the innards
of the project, rather than creating a single directory for the library.
There are also a lot of files included that don't need to be included
(starting with the entire contents of the autom4ate.cache directory and
aclocal.m4).

* Licensing. The fact that your code is under the GPL means that there
shouldn't be a problem with using it in audacity, which is released
under the GPL v2. However there is a complicating factor. The GPL v2 is
ambiguous about whether a GPL v2 application can load plug-ins at
runtime which do not have a GPL-compatible license (because they are
shared libraries, and the GPL treats all libraries as equal, regardless
of whether they are required or optional at build time or run time).
Audacity supports loading plug-ins at runtime via the LADSPA plug-in
interface, including (potentially) ones with licenses that aren't GPL
compatible. This means that it would be possible for users to link
audacity against commercial libraries simply by adding them to their
plug-ins directory. Our view is that this isn't a problem, it doesn't go
against the spirit of the GPL, and it is in the end up to the individual
user what they put in their plug-ins directory (we only supply
GPL-compatible plug-ins). However Erik de Castro Lopo, the author of
libsamplerate, takes a different view, believing that providing a
plug-in facility for plug-ins with incompatible licenses breaks the GPL
v2. Thus we don't use libsamplerate in audacity unless loading of
GPL-incompatible plug-ins is disabled.

The upshot of all this is that in order to distribute Audacity for
Windows or OS X with sbsms built in we would have to find an alternative
to libsamplerate for the sample rate conversions. In audacity itself we
use libresample, which provides a similar interface and signal quality,
but is slower. This is developed by the audacity team and so does not
pose a licensing issue. So the simplest way to get round this issue is
to modify sbsms to use either resampling library (chosen at compile
time), so that libresample can be used to build audacity releases.

Sorry for a long and mostly critical email, but I'd prefer to flag up
issues now rather than you put a lot of time into a patch which then
gets stalled at this end because of library issues (this is already the
case for one GSoC project, we don't want another).

Richard


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
audacity-devel mailing list
audacity-devel@...
https://lists.sourceforge.net/lists/listinfo/audacity-devel
LRN
Re: New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options
Print post
Permalink
I didn't noticed this message before, but now when i did, i have to say that

On Sat, Oct 4, 2008 at 2:31 PM, Richard Ash <richard@...> wrote:
This means that it would be possible for users to link
audacity against commercial libraries simply by adding them to their
plug-ins directory. Our view is that this isn't a problem, it doesn't go
against the spirit of the GPL, and it is in the end up to the individual
user what they put in their plug-ins directory (we only supply
GPL-compatible plug-ins). However Erik de Castro Lopo, the author of
libsamplerate, takes a different view, believing that providing a
plug-in facility for plug-ins with incompatible licenses breaks the GPL
v2.

Erik de Castro Lopo is wrong. GPL viral clause, that causes anything that is combined with GPL'ed code to become GPL'ed itself, only works when such combination is distributed. Anyone can take any GPL-covered code and combine it with proprietary code, and this combination is 100% legal - as long as one does not distributes it. Remember, user is FREE to do ANYTHING with the program. Free as in 'freedom'.

Also, a degree of 'combination' could only be measured by a judge. If judge decides that two modules do not form a combined work (which could very much be the case of plugins), that would mean that GPL is not violated.

On this topic FSF's opinion is this: when an application imports one function from a library and calls only this function, it's 'grey area'.

Erik's views seems to be very extreme...

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
audacity-devel mailing list
audacity-devel@...
https://lists.sourceforge.net/lists/listinfo/audacity-devel
Alexandre Prokoudine
Re: New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options
Print post
Permalink
In reply to this post by Richard Ash (audacity-help)
On Sat, Oct 4, 2008 at 2:31 PM, Richard Ash wrote:

> In principal it would be interesting to have an alternative to
> soundtouch available in audacity, because we know there are some cases
> when it doesn't work well. However there are some issues with the code
> on sourceforge as it stands:

For some reason I had an impression that Audacity was moving to
RubberBand by Chris Cannam.

Alexandre

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
audacity-devel mailing list
audacity-devel@...
https://lists.sourceforge.net/lists/listinfo/audacity-devel
Clayton Otey
Re: New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options
Print post
Permalink
I wasn't aware of his code.  I tested it out and while it's typically ~3 times faster than mine, the results are pretty undeniably of a lower quality (although these things are subjective, of course) on the test cases I chose.  You can look at examples of my results at http://sbsms.sourceforge.net/.
Also my algorithm can be split into (slow) analysis and (fast) synthesis steps which opens up interesting possibilities as I've discussed in another thread.  Furthermore, my code has no dependencies (rubberband requires fftw3 and libsamplerate).
It may be reasonable to offer 3 different algorithms (soundtouch, rubberband, sbsms) which vary in quality vs. speed.  

Clayton

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
audacity-devel mailing list
audacity-devel@...
https://lists.sourceforge.net/lists/listinfo/audacity-devel
James Crook
Re: New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options
Print post
Permalink
In reply to this post by LRN
LRN wrote:
> Anyone can take any GPL-covered code and combine it with proprietary
> code, and this combination is 100% legal - as long as one does not
> distributes it. Remember, user is FREE to do ANYTHING with the
> program. Free as in 'freedom'.

> Also, a degree of 'combination' could only be measured by a judge. If
> judge decides that two modules do not form a combined work (which
> could very much be the case of plugins), that would mean that GPL is
> not violated.
My view is that one can 'drive a cart and horses' through the GPL
through 'dynamic linking'.  In this case we know that Erik de Castro
Lopo does not want his code used in this way.  Also Clayton's code
produces higher quality results - and for time stretch I would
definitely choose that.

So in sum, I WANT these changes from Clayton in Audacity, after the beta
(given our insane schedule).

--James.



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
audacity-devel mailing list
audacity-devel@...
https://lists.sourceforge.net/lists/listinfo/audacity-devel
LRN
Re: New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options
Print post
Permalink
On 19.10.2008 0:41, James Crook wrote:

> LRN wrote:
>    
>> Anyone can take any GPL-covered code and combine it with proprietary
>> code, and this combination is 100% legal - as long as one does not
>> distributes it. Remember, user is FREE to do ANYTHING with the
>> program. Free as in 'freedom'.
>>      
>
>    
>> Also, a degree of 'combination' could only be measured by a judge. If
>> judge decides that two modules do not form a combined work (which
>> could very much be the case of plugins), that would mean that GPL is
>> not violated.
>>      
> My view is that one can 'drive a cart and horses' through the GPL
> through 'dynamic linking'.  In this case we know that Erik de Castro
> Lopo does not want his code used in this way.  Also Clayton's code
> produces higher quality results - and for time stretch I would
> definitely choose that.
>    
You can't drive a cart and horses through a narrow hole (Audacity's
plugin interface).

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
audacity-devel mailing list
audacity-devel@...
https://lists.sourceforge.net/lists/listinfo/audacity-devel
James Crook
Re: New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options
Print post
Permalink
LRN wrote:
> You can't drive a cart and horses through a narrow hole (Audacity's
> plugin interface).

That is so, but there is experimental code for scripting in Audacity,
and that will not be such a narrow hole.

I think it's a good principle anyway to try to respect the wishes of
code authors, even where not mandated.  We are running into the same
thing ourselves with 'Evil Profiteers' selling Audacity without letting
people know that it is available for free.  Within a strict
interpretation of the terms of the GPL it is arguable that they can do
just that.

--James.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
audacity-devel mailing list
audacity-devel@...
https://lists.sourceforge.net/lists/listinfo/audacity-devel
Alexandre Prokoudine
Re: New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options
Print post
Permalink
In reply to this post by Clayton Otey
On Sat, Oct 18, 2008 at 10:45 AM, Clayton Otey wrote:

> I wasn't aware of his code.  I tested it out and while it's typically ~3
> times faster than mine, the results are pretty undeniably of a lower quality
> (although these things are subjective, of course) on the test cases I chose.
>  You can look at examples of my results at http://sbsms.sourceforge.net/.
> Also my algorithm can be split into (slow) analysis and (fast) synthesis
> steps which opens up interesting possibilities as I've discussed in another
> thread.  Furthermore, my code has no dependencies (rubberband requires fftw3
> and libsamplerate).
> It may be reasonable to offer 3 different algorithms (soundtouch,
> rubberband, sbsms) which vary in quality vs. speed.

Well, either way :) I'm not picky about that kind of stuff :)

Alexandre

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
audacity-devel mailing list
audacity-devel@...
https://lists.sourceforge.net/lists/listinfo/audacity-devel
LRN
Re: New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options
Print post
Permalink
In reply to this post by James Crook
On 19.10.2008 14:41, James Crook wrote:

> LRN wrote:
>    
>> You can't drive a cart and horses through a narrow hole (Audacity's
>> plugin interface).
>>      
>
> That is so, but there is experimental code for scripting in Audacity,
> and that will not be such a narrow hole.
>    
> I think it's a good principle anyway to try to respect the wishes of
> code authors, even where not mandated.  We are running into the same
> thing ourselves with 'Evil Profiteers' selling Audacity without letting
> people know that it is available for free.  Within a strict
> interpretation of the terms of the GPL it is arguable that they can do
> just that.
>    
4. Conveying Verbatim Copies.

You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice; keep
intact all notices stating that this License *and any non-permissive
terms added in accord with section 7 apply to the code*...
5. Conveying Modified Source Versions. ...

...b) The work must carry prominent notices stating that it is released
under this License *and any conditions added under section 7*...
...c) You must license the entire work, as a whole, under this License
to anyone who comes into possession of a copy. This License will
therefore apply, *along with any applicable section 7 additional terms*...

...
7. Additional Terms.

“Additional permissions” are terms that supplement the terms of this
License by making exceptions from one or more of its conditions. ...

...Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders
of that material) supplement the terms of this License with terms:

...b) Requiring *preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal Notices
displayed by works containing it*; or...


(© GPLv3)

So, what i'm trying to say is that it should be possible to demand from
3rd parties that they notify their users that Audacity is Free Software
and could be available for free. At least it think so.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
audacity-devel mailing list
audacity-devel@...
https://lists.sourceforge.net/lists/listinfo/audacity-devel
James Crook
Re: New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options
Print post
Permalink
LRN wrote:
> So, what i'm trying to say is that it should be possible to demand from
> 3rd parties that they notify their users that Audacity is Free Software
> and could be available for free. At least I think so.
>  

That really is the point - you think so, but (a) you're not 100% sure -
it is open to interpretation, and (b) the conditions don't state that
you have to notify users of the conditions in any advertising for the
product.  As long as you let people know it could be available for free
after they've bought it from you, you're OK....  At least I think so.  
It's certainly not what we intend.  But IANAL.

--James.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
audacity-devel mailing list
audacity-devel@...
https://lists.sourceforge.net/lists/listinfo/audacity-devel
Clayton Otey
Re: New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options
Print post
Permalink
In response to the rubberband query again:

It turns out rubberband produces an arguably better quality result than sbsms on purely percussive signals.  To my ear rubberband always sounds more "processed" but it manages to preserve percussive transients better and introduce less smearing than sbsms.  I got very jealous and so I implemented an optional variable rate pre-analysis step which improves transient preservation.  sbsms is still not as punchy as rubberband on stretched drum loops, but I think it's a good step.

As far as polyphonic/low-range material go, sbsms is a clear winner in quality.

Clayton

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
audacity-devel mailing list
audacity-devel@...
https://lists.sourceforge.net/lists/listinfo/audacity-devel
Chris Cannam
Re: New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options
Print post
Permalink
In reply to this post by Alexandre Prokoudine
[a very old thread which I'm afraid I missed first time around]

On Fri, Oct 17, 2008 at 11:15 PM, Alexandre Prokoudine
<alexandre.prokoudine@...> wrote:
> For some reason I had an impression that Audacity was moving to
> RubberBand by Chris Cannam.

That might conceivably have happened, if I had found any time to work
on such a thing before Clayton came along.  In the context of
Audacity, it's not so clear what a Rubber Band implementation would
add beyond what SBSMS adds already.

Rubber Band is essentially a sophisticated and heavily tuned
implementation of a less sophisticated algorithm.  In terms of pure
audio quality I'd expect them to compare much as Clayton described --
Rubber Band is likely to "win" in situations it's tuned well for and
where sinusoidal modelling makes little sense (e.g. percussive loops)
and SBSMS should "win" quite comfortably in many other situations
(particularly polyphonic material that mixes percussive transients in
one "voice" with smoother sounds in others).

Rubber Band's design goals were as much about application flexibility,
such as the ability to run as a real-time filter, changing stretch
ratios as it goes, as about the highest possible audio quality.  You
could say it's an engineering application rather than a piece of DSP
research.


Chris

------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
audacity-devel mailing list
audacity-devel@...
https://lists.sourceforge.net/lists/listinfo/audacity-devel
Clayton Otey
Re: New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options
Print post
Permalink
It shouldn't be too difficult to incoprorate rubberband into audacity, and it would definitely be very useful in addition to soundtouch/sbsms as it works better on material that people who e.g. write electronic music actually want to work with, like drum loops.  So I vote we do it, provided Chris has no objections.   My main questions are:

1) Where to put it into the GUI?
If someone took the time to implement sliding pitch shifts/time stretches to rubberband, which shouldn't be hard as it's already designed to support dynamic stretch rates, we could just add a slider to the Sliding Time Scale/Pitch Shift dialog which selects between "Polyphonic Algorithm/Percussive Algorithm" or something similar.

2) How to deal with external libraries?
vamp and sndfile are already included in audacity, but fftw and samplerate have licensing issues.  I wrote a fft library and samplerate converter to get around this issue for sbsms, so presumably rubberband could just directly borrow a few source files from my code, provided the performance hit is not too substantial.

Chris, does this sound reasonable?  If so, how  much of the work will you volunteer for?

------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
audacity-devel mailing list
audacity-devel@...
https://lists.sourceforge.net/lists/listinfo/audacity-devel
Martyn Shaw-2
Re: New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options
Print post
Permalink
Can we keep any developments private/on our own machines until we make
significant progress on bug fixing please?  More inputs, whilst
interesting and welcome, make releases less likely and so merely an
intellectual development, not a practical-to-the-big-wide-world one.

I suppose there's always the 'turned off in Experimental.h' approach.

Thanks
Martyn

Clayton Otey wrote:

> It shouldn't be too difficult to incoprorate rubberband into audacity,
> and it would definitely be very useful in addition to soundtouch/sbsms
> as it works better on material that people who e.g. write electronic
> music actually want to work with, like drum loops.  So I vote we do it,
> provided Chris has no objections.   My main questions are:
>
> 1) Where to put it into the GUI?
> If someone took the time to implement sliding pitch shifts/time
> stretches to rubberband, which shouldn't be hard as it's already
> designed to support dynamic stretch rates, we could just add a slider to
> the Sliding Time Scale/Pitch Shift dialog which selects between
> "Polyphonic Algorithm/Percussive Algorithm" or something similar.
>
> 2) How to deal with external libraries?
> vamp and sndfile are already included in audacity, but fftw and
> samplerate have licensing issues.  I wrote a fft library and samplerate
> converter to get around this issue for sbsms, so presumably rubberband
> could just directly borrow a few source files from my code, provided the
> performance hit is not too substantial.
>
> Chris, does this sound reasonable?  If so, how  much of the work will
> you volunteer for?
>
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
> The future of the web can't happen without you.  Join us at MIX09 to help
> pave the way to the Next Web now. Learn more and register at
> http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> audacity-devel mailing list
> audacity-devel@...
> https://lists.sourceforge.net/lists/listinfo/audacity-devel

------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
audacity-devel mailing list
audacity-devel@...
https://lists.sourceforge.net/lists/listinfo/audacity-devel
Alexandre Prokoudine
Re: New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options
Print post
Permalink
2008/12/16 Martyn Shaw wrote:
> Can we keep any developments private/on our own machines

This is kind of scary considering things like HD failures.

> I suppose there's always the 'turned off in Experimental.h' approach.

Sounds good to me :)

Alexandre

------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
audacity-devel mailing list
audacity-devel@...
https://lists.sourceforge.net/lists/listinfo/audacity-devel
Martyn Shaw-2
Re: New implementation of Time stretching/Pitch scaling
Reply Threaded MoreMore options