The identification of the parameters of the vocal tract system can be used for speaker identification.

A preferred speech coding technique is the so-called Model-Based Speech Coding (MBSC), which involves modeling the vocal tract as a linear time-variant system (synthesis filter). The system's input is either white noise or a train of impulses. For coding purposes, the synthesis filter is assumed to be time-invariant during a short time interval (time slot) of typically 10-20 msec. Then, the signal is represented by the coefficients of the synthesis filter corresponding to each time slot.

A successful MBSC method is the so-called Linear Prediction Coding (LPC). Roughly speaking, the LPC technique models the synthesis filter as an all-pole linear system. This all-pole linear system has coefficients obtained by adapting a predictor of the output signal, based on its own previous samples. The use of an all-pole model provides a good representation for the majority of speech sounds. However, the representation of nasal sounds, fricative sounds, and stop consonants requires the use of a zero-pole model. Also, the LPC technique is not adequate when the voice signal is corrupted by noise.

We propose a method to estimate a zero-pole model which is able to provide the optimal synthesis filter coefficients, numerically efficient and optimal when minimizing a logarithm criterion.

In order to evaluate the perceptual relevance of the proposed method, we used the model estimated from a speech signal to re-synthesis it:

- D. Marelli, P. Balazs, "On Pole-Zero Model Estimation Methods Minimizing a Logarithmic Criterion for Speech Analysis", IEEE Transactions on Audio, Speech and Language Processing, Vol. 18 (2), pp. 237 - 248 (2010)

Gabor multipliers are an efficient tool for time-variant filtering. They are used implicitly in many engineering applications in signal processing. For these operators, the result of a Gabor transform (the sampled version of the Short Time Fourier Transform) is multiplied by a fixed function (called the time-frequency mask or symbol). Then the result is synthesized.

Other transforms beyond the Gabor transform, for example the wavelet transform, are more suitable for certain applications. The concept of multipliers can easily be extended to these transforms. More precisely, the concept of multipliers can be applied to general frames without any further structure. This results in the introduction of operators called frame multipliers, which will be investigated in detail in this project in order to precisely define their mathematical properties and optimize their use in applications.

The problem will be approached using modern frame theory, functional analysis, numeric tools, and linear algebra tools. Systematic numeric experiments will be conducted to observe the different properties of frame multipliers. This observations will support the analytical formulation and demonstration of these properties.

The following topics will be investigated in the project:

- Eigenvalues and eigenvectors of frame multipliers
- Invertibility, injectivity, and surjectivity of frame multipliers
- Reproducing kernel invariance
- Generalization of multipliers to Banach frames and p-frames
- Connection of frame multipliers to weighted frames
- Discretization and implementation of frame multipliers
- Best approximation of operators by frame multipliers and identification of frame multipliers

The applications of frame multipliers in signal processing are numerous and include any application requiring time-variant filtering. Some applications of frame multipliers will be investigated further in the following parallel projects:

- Mathematical Modeling of Auditory Time-Frequency Masking Functions
- Improvement of Head-Related Transfer Function Measurements
- Advanced Method of Sound Absorption Measurements

- P. Balazs, "Matrix Representation of Bounded Linear Operators By Bessel Sequences, Frames and Riesz Sequence", SampTA'09, 8th International Conference on Sampling and Applications, May 2009, Marseille, France
- P. Balazs, J.-P. Antoine, A. Grybos, "Weighted and Controlled Frames: Mutual relationship and first Numerical Properties", accepted for publication in International Journal of Wavelets, Multiresolution and Information Processing (2009), preprint
- A. Rahimi, P. Balazs, "Multipliers for p-Bessel sequences in Banach spaces", submitted (2009)
- D. Stoeva, P. Balazs, "Unconditional convergence and Invertibility of Multipliers", preprint (2009)

S&T cooperation project 'Amadee' Austria-France 2013-14, "Frame Theory for Sound Processing and Acoustic Holophony", FR 16/2013

Project Partner: The Institut de recherche et coordination acoustique/musique (IRCAM)

This project consists of three subprojects:

Recently Gabor Muiltipliers have been used to implement time-variant filtering as *Gabor Filters*. This idea can be further generalized. To investigate the basic properties of such operators the concept of abstract, i.e. unstructured, *frames* is used. Such multipliers are operators, where a certain fixed mask, a so-called *symbol*, is applied to the coefficients of frame analysis , whereafter synthesis is done. The properties that can be found for this case can than be used for all kind of frames, for example regular and irregular Gabor frames, wavelet frames or auditory filterbanks.

The basic definition of a frame multiplier follows:

As special case of such multipliers such operators for irregular Gabor system will be investigated and implemented. This corresponds to a irregular sampled Short-Time-Fourier-Transformation. As application an STFT correpsonding to the bark scale can be examined.

This mathematical and basic research-oriented project is important for many other projects like time-frequency-masking or system-identification.

- O. Christensen,
*An Introduction To Frames And Riesz Bases*, Birkhäuser Boston (2003) - M. Dörfler,
*Gabor Analysis for a Class of Signals called Music*, Dissertation Univ. Wien (2002) - R.J. Duffin, A.C. Schaeffer,
*A Class of nonharmonic Fourier series*, Trans.Amer.Math.Soc., vol.72, pp. 341-366 (1952) - H. G. Feichtinger, K. Nowak,
*A First Survey of Gabor Multipliers*, in H. G. Feichtinger, T. Strohmer

- Talk Frames and Frame Multiplier at the LMA, CNRS Marseille (Peter Balazs)

Read more: Regular and Irregular Gabor Multiplier with Applications in Psychoacoustic Masking

This project ended in September 2011.

The** final MulAc Meeting** was in Vienna from 29th to 30th of August 2011.

The **ARI Mulac Frame Meeting** was held on Tuesday, June 15^{th} 2010at ARI.

The **MULAC Mid-term Meeting** was held in Marseille from 12. to 13. April 2010. See the Registration-Webpage or the Program.

The **FYMA Mulac semina**r was held in Louvain-la-Neuve in the 11th of March, 2010. (Talks by Jean-Pierre Antoine, Jean-Pierre Gazeau, Diana Stoeva and Peter Balazs.)

The MULAC - Kick-Off Meeting took place at ARI in Vienna from September 23^{rd} to 24^{th} 2008.

This international, multi-disciplinary and team-oriented project allowed P. Balazs to form a small group '*Mathematics and Acoustical Signal Processing*’ at the Acoustic Research Institute in cooperation with *NuHAG* Vienna (Hans G. Feichtinger),* LMA * (Richard Kronland-Martinet) and *LATP* Marseille (Bruno Torrésani) as well as the *FYMA* Louvain-la-Neuve (Jean-Pierre Antoine).

Within the institute the groups 'Audiological Acoustics' and 'Software Development' are involved.

This project is funded by the WWTF . It will run for 3,5 years and post-docs will be employed for six years total, as well as master students for 36 months total.

In December 2007 the Austrian Academy of Sciences was presenting '**mathematics in ...**' as the topic of the month . This included 'mathematics at the Acoustics Research Institute', which describes this project.

*"Frame Multipliers”* are a promising mathematical concept, which can be applied to retrieve desired information out of acoustic signals. P. Balazs introduced them by successfully generalizing existing time-variant filter approaches. This project aims to establish new results in the mathematical theory of frame multipliers, to integrate them in efficient digital signal processing algorithms and to make them available for use in 'real-world' acoustical applications. A multi-disciplinary and international cooperation has been established and will be extended in the project to create new significant impulses for the involved disciplines: mathematics, numerics, engineering, physics and cognitive sciences. Various acoustical applications like modelling of auditory perception, measurement of sound absorption coefficients and system identification of the head related transfer functions are included. The results of the project will allow their future integration into practical areas such as audio coding, noise abatement, sound quality design, virtual reality and hearing aids.

Read more: Frame Multiplier: Theory and Application in Acoustics (WWTF 2007)

Many problems in physics can be formulated as operator theory problems, such as in differential or integral equations. To function numerically, the operators must be discretized. One way to achieve discretization is to find (possibly infinite) matrices describing these operators using ONBs. In this project, we will use frames to investigate a way to describe an operator as a matrix.

The standard matrix description of operators O using an ONB (e_k) involves constructing a matrix M with the entries M_{j,k} = < O e_k, e_j>. In past publications, a concept that described operator R in a very similar way has been presented. However, this description of R used a frame and its canonical dual. Currently, a similar representation is being used for the description of operators using Gabor frames. In this project, we are going to develop and completely generalize this idea for Bessel sequences, frames, and Riesz sequences. We will also look at the dual function that assigns an operator to a matrix.

This "sampling of operators" is especially important for application areas where frames are heavily used, so that the link between model and discretization is maintained. To facilitate implementations, operator equations can be transformed into a finite, discrete problem with the finite section method (much in the same way as in the ONB case).

- P. Balazs, "Matrix Representation of Operators Using Frames", Sampling Theory in Signal and Image Processing (STSIP) (2007, accepted), preprint
- P. Balazs, "Hilbert-Schmidt Operators and Frames - Classification, Approximation by Multipliers and Algorithms" , International Journal of Wavelets, Multiresolution and Information Processing, (2007, accepted) preprint, Codes and Pictures: here

It is known in psychoacoustics that not all information contained in a "real world" acoustic signal is processed by the human auditory system. More precisely, it turns out that some time-frequency components mask (overshadow) other components that are close in time or frequency.

In the software S_TOOLS-ST^{x} developed by the Institute, an algorithm based on simultaneous masking has been implemented. This algorithm removes perceptually irrelevant time-frequency components. In this implementation, the model is described as a Gabor multiplier with an adaptive symbol.

In this project, the masking model will be extended to a true time-frequency model, incorporating frequency and temporal masking.

Experiments have been conducted (in cooperation with the Laboratory for Mechanics and Acoustics / CNRS Marseille) to test the time-frequency masking properties of a single Gaussian atom, and to study the additivity of these masking properties for several Gaussian atoms.

The results of these experiments will be used, in combination with theoretical results obtained in the parallel projects studying the mathematical properties of frame multipliers, to approximate or identify the masking model by wavelet and Gabor multipliers.

The obtained model will then be validated by appropriate psychoacoustical experiments.

Efficient implementation of a masking filter offers many applications:

- Sound / Data Compression
- Sound Design
- Back-and-Foreground Separation
- Optimization of Speech and Music Perception

After completing the testing phase, the algorithms are to be implemented in S_TOOLS-STx.

- P. Balazs, B. Laback, G. Eckel, W. Deutsch, "Introducing Time-Frequency Sparsity by Removing Perceptually Irrelevant Components Using a Simple Model of Simultaneous Masking", IEEE Transactions on Audio, Speech and Language Processing (2009), in press
- B. Laback, P. Balazs, G. Toupin, T. Necciari, S. Savel, S. Meunier, S. Ystad and R. Kronland-Martinet, "Additivity of auditory masking using Gaussian-shaped tones", Acoustics'08, Paris, 29.06.-04.07.2008 (03.07.2008)
- B. Laback, P. Balazs, T. Necciari, S. Savel, S. Ystad, S. Meunier and R. Kronland-Martinet, "Additivity of auditory masking for Gaussian-shaped tone pulses", preprint

The Short-Time Fourier Transform (STFT), in its sampled version (the Gabor transform), is a well known, valuable tool for displaying the energy distribution of a signal over the time-frequency plane. The equivalence between Gabor analysis and certain filter banks is a well-known fact. The main task is how to find a Gabor analysis-synthesis system with perfect (or depending on the application, satisfactorily accurate) reconstruction in a numerically efficient way. This is done by using the dual Gabor frame, which implies the need to invert the Gabor frame operator.

This project incorporates an application of the general idea of preconditioning in the context of Gabor frames. While most (iterative) algorithms aim at a relatively costly exact numeric calculation of the inverse Gabor frame matrix, we will use a "cheap method" to find an approximation. The inexpensive method will be based on (double) preconditioning using diagonal and circulant preconditioners. As a result, good approximations of the true dual Gabor atom can be obtained at low computational costs.

For a number of applications, such as time stretching without changing the frequency content in audio processing or more complex modifications like psychoacoustical masking, the time domain signal needs to be reconstructed using the time-frequency domain coefficients.

H. G. Feichtinger et al., NuHAG, Facultyof Mathematics, University of Vienna

- P. Balazs, H.G. Feichtinger, M. Hampejs, G. Kracher; "Double Preconditioning for Gabor Frames"; IEEE Transactions on Signal Processing, Vol. 54, No.12, December 2006 (2006), preprint
- P. Balazs, H.G. Feichtinger, M. Hampejs, G. Kracher; "Double Preconditioning for the Gabor Frame Operator"; Proceedings ICASSP '06, May 14-19, Toulouse, DVD (2006)

This project is part of a project cluster that investigates time-frequency masking in the auditory system, in cooperation with the Laboratory for Mechanics and Acoustics / CNRS Marseille. While other subprojects study the spread of masking across the time-frequency plane using Gaussian-shaped tones, this subproject investigates how multiple Gaussian maskers distributed across the time-frequency plane create masking that adds up at a given time-frequency point. This question is important in determining the total masking effect resulting from the multiple time-frequency components (that can be modeled as Gaussian Atoms) of a real-life signal.

Both the maskers and the target are Gaussian-shaped tones with a frequency of 4 kHz. A two-stage approach is applied to measure the additivity of auditory masking. In the first stage, the levels of the maskers are adjusted to cause the same amount of masking in the target. In the second stage, various combinations of those maskers are tested to study their additivity.

In the first study, the maskers are spread either in time OR in frequency. In the second study, the maskers are spread in time AND in frequency.

New insight into the coding of sound in the auditory system could help to design more efficient audio codecs. These codecs could take the additivity of time-frequency masking into account.

WTZ (project AMADEUS)

- Laback, B., Balazs, P., Toupin, G., Necciari, T., Savel, S., Meunier, S., Ystad, S., Kronland-Martinet, R. (2008). Additivity of auditory masking using Gaussian-shaped tones, presented at Acoustics? 08 conference, Paris.

A Gaussian Atom is suitable as an ideal atom for the time frequency representation of the human audio perception. This is not only because of the Gaussian Atom's special mathematic features, but also because of results from existing psychoacoustic studies. Developing a time-frequency mask (occlusion) requires testing the time-frequency masking effects of this atom. So far, short-tape limited signals have not been investigated in masking experiments. Relatively few psychoacoustic experiments have been explored completely, and these have been combined with time-frequency effects.

In cooperation with the Laboratory for Mechanics and Acoustics / CNRS Marseille, an experimental protocol was developed for testing the time-frequency method of a singular Gaussian atom. Experiments were made for the first time in 2006, and gave the first results concerning the hearing threshold and the existence of such a signal. The experiments that included the masking threshold began as a PhD project before the end of 2006 in Marseille.

Efficient implementation of a masking filter offers many applications:

- Sound / Data Compression
- Sound Design
- Back-and-Foreground Separation
- Optimization of Speech and Music Perception

After completing the testing phase, the algorithms are to be implemented in S_TOOLS-STx.

- Amadée: Time Frequency Representations and Auditory Perception
- Cotutelle de thèse
- Experiments studying additivity of masking for multiple maskers

WTZ (project AMADEUS)

- Laback, B., Balazs, P., Toupin, G., Necciari, T., Savel, S., Meunier, S., Ystad, S., Kronland-Martinet, R. (2008). Additivity of auditory masking using Gaussian-shaped tones, presented at Acoustics? 08 conference.

Head-related transfer functions (HRTF) describe the sound transmission from the free field to a place in the ear canal in terms of linear time-invariant systems. Due to the physiological differences of the listeners' outer ears, the measurement of each subject's individual HRTFs is crucial for sound localization in virtual environments (virtual reality).

Measurement of an HRTF can be considered a system identification of the weakly non-linear electro-acoustic chain from the sound source room's HRTF microphone. An optimized formulation of the system identification with exponential sweeps, called the "multiple exponential sweep method" (MESM), was used for the measurement of transfer functions. For this measurement of transfer functions, either the measurement duration or the signal-to-noise ratio could be optimized.

Initial heuristic experiments have shown that using Gabor multipliers to extract the relevant sweeps in the MESM post-processing procedure improves the signal-to-noise ratio of the measured data even further. The objective of this project is to study, in detail, how frame multipliers can optimally be used during this post-processing procedure. In particular, wavelet frames, which best fit the structure of an exponential sweep, will be studied.

Systematic numeric experiments will be conducted with simulated slowly time-variant, weakly non-linear systems. As the parameters of the involved signals are precisely known and controlled, an optimal symbol will automatically be created. Finally, the efficiency of the new method will be tested on a "real world" system, which was developed and installed in the semi-anechoic room of the Institute. It uses in-ear microphones, a subject turntable, 22 loudspeakers on a vertical arc, and a head tracker.

The new method will be used for improved HRTF measurement.

Signal processing has entered into today's life on a broad range, from mobile phones, UMTS, xDSL, and digital television to scientific research such as psychoacoustic modeling, acoustic measurements, and hearing prosthesis. Such applications often use time-invariant filters by applying the Fourier transform to calculate the complex spectrum. The spectrum is then multiplied by a function, the so-called transfer function. Such an operator can therefore be called a Fourier multiplier. Real life signals are seldom found to be stationary. Quasi-stationarity and fast-time variance characterize the majority of speech signals, transients in music, or environmental sounds, and therefore imply the need for non-stationary system models. Considerable progress can be achieved by reaching beyond traditional Fourier techniques and improving current time-variant filter concepts through application of the basic mathematical concepts of frame multipliers.

Several transforms, such as the Gabor transform (the sampled version of the Short-Time Fourier Transformation), the wavelet transform, and the Bark, Mel, and Gamma tone filter banks are already in use in a large number of signal processing applications. Generalization of these techniques can be obtained via the mathematical frame theory. The advantage of introducing the frame theory consists particularly in the interpretability of filter and analysis coefficients in terms of frequency and time localization, as opposed to techniques based on orthonormal bases.

One possibility to construct time-variant filters exists through the use of Gabor multipliers. For these operators the result of a Gabor transform is multiplied by a given function, called the time-frequency mask or symbol, followed by re-synthesis. These operators are already used implicitly in engineering applications, and have been investigated as Gabor filters in the fields of mathematics and signal processing theory. If alternative transforms are used, the concept of multipliers can be extended appropriately. So, for example, the concept of wavelet multipliers could be investigated for a wavelet transform.

Different kinds of applications call for different frames. Multipliers can be generalized to the abstract level of frames without any further structure. This concept will be further investigated in this project. Its feasibility will be evaluated in acoustic applications using special cases of Gabor and wavelet systems.

The project goal is to study both the mathematical theory of frame multipliers and their application among selected problems in acoustics. The project is divided into the following subprojects:

- General Frame Multiplier Theory
- Analytic and Numeric Properties of Gabor Multipliers
- Analytic and Numeric Properties of Wavelet Multipliers

- Mathematical Modeling of Auditory Time-Frequency Masking Functions
- Improvement of Head-Related Transfer Function Measurements
- Advanced Method of Sound Absorption Measurements

- H.G. Feichtinger et al., NuHAG, Faculty of Mathematics, University of Vienna
- R. Kronland-Martinet et al., Modélisation, Synthèse et Contrôle des Signaux Sonores et Musicaux of the LMA / CNRS Marseille
- B. Torrésani et al., LATP Université de Provence / CNRS Marseille
- J.P. Antoine et al., FYMA Université Catholique de Louvain

- P. Balazs, J.-P. Antoine, A. Gryboś, "Weighted and Controlled Frames: Mutual relationship and first Numerical Properties", accepted for publication in International Journal of Wavelets, Multiresolution and Information Processing (2009), preprint
- P. Balazs, “Matrix Representation of Bounded Linear Operators By Bessel Sequences, Frames and Riesz Sequence“,SampTA'09, 8th International Conference on Sampling and Applications, May 2009, Marseille, France
- A. Rahimi, P. Balazs, "Multipliers for p-Bessel sequences in Banach spaces", submitted (2009)
- D. Stoeva, P. Balazs, "Unconditional convergence and Invertibility of Multipliers", preprint (2009)
- Monika Dörfler and Bruno Torrésani, “Representation of operators in the time-frequency domain and generalized Gabor multipliers”, J. Fourier Anal. Appl., 2009 (in press)
- Yohan Frutiger: "Multiplicateurs de Gabor pour les transformations sonores" (Gabor Multipliers for sound transformations) Master thesis under the supervision of R. Kronland-Martinet, June 2008
- F. Jaillet, P. Balazs, M. Dörfler and N. Engelputzeder, “On the Structure of the Phase around the Zeros of the Short-Time Fourier Transform”, NAG/DAGA 2009, International Conference on Acoustics, March 2009, Rotterdam, Nederland
- F. Jaillet, P. Balazs and M. Dörfler, “Nonstationary Gabor Frames”, SampTA'09, 8th International Conference on Sampling and Applications, May 2009, Marseille, France
- P. Balazs, B. Laback, G. Eckel, W. Deutsch, "Introducing Time-Frequency Sparsity by Removing Perceptually Irrelevant Components Using a Simple Model of Simultaneous Masking", IEEE Transactions on Audio, Speech and Language Processing (2009), in press
- B. Laback, P. Balazs, G. Toupin, T. Necciari, S. Savel, S. Meunier, S. Ystad and R. Kronland-Martinet, "Additivity of auditory masking using Gaussian-shaped tones", Acoustics'08, Paris, 29.06.-04.07.2008 (03.07.2008)
- B. Laback, P. Balazs, T. Necciari, S. Savel, S. Ystad, S. Meunier and R. Kronland-Martinet, "Additivity of auditory masking for Gaussian-shaped tone pulses", preprint
- Anaïk Olivero: "Expérimentation des multiplicateurs temps-échelle" (On the time-scale multipliers) Master thesis under the supervision of R. Kronland-Martinet and B. Torrésani, June 2008

Time-variant filters are gaining importance in today's signal processing applications. Gabor multipliers in particular are popular in current scientific investigations. These multipliers are a specialization of Bessel multipliers to Gabor frames. These operators are interesting in regard to both theory and application:

- Bessel and Frame Multipliers in Banach Spaces: In this project, the concept of frame multipliers should be generalized to work with Banach spaces.
- Theory of Wavelet Multipliers: The concept of multipliers can be easily extended to wavelet frames. The influence of the special structures of these sequences will be investigated.
- Basic Properties of Irregular Gabor Multipliers: Here multipliers for Gabor frames on irregular lattices are investigated.

- Time Frequency Masking: Gabor Multiplier Models and Evaluation: The symbol for the Gabor multiplier is calculated adaptively and the resulting model incorporates both time and frequency masking components. The goal is to obtain an algorithm using 2-D convolution.
- Improving the Multiple Exponential Sweep Method (MESM) using Gabor Multipliers: The MESM is an efficient system identification method. Initial tests have shown that this method can be improved with a Gabor multiplier applied as a mask for the original sweep.
- Wavelet Multipliers and Their Application to Reflection Measurements: One method to calculate the absorption coefficient of a sound proof wall requires separation of the impulse responses of different reflections. They can be easily separated in a scalogram and they can be extracted using a wavelet multiplier.
- Mathematical Foundation of the Irrelevance Model: In this project, the theoretical foundation of the irrelevance algorithms implemented in STx is being developed.

- H.G. Feichtinger, K. Gröchenig et al., NuHAG, Faculty of Mathematics, University of Vienna
- R. Kronland-Martinet, S. Ytad, T. Necciari, Modélisation, Synthèse et Contrôle des Signaux Sonores et Musicaux of the LMA / CRNS Marseille
- S. Meunier, S. Savel, Acoustique perceptive et qualité de l’environnement sonore of the LMA / CRNS Marseille

- P. Balazs, B. Laback, G. Eckel, W. Deutsch, "Introducing Time-Frequency Sparsity by Removing Perceptually Irrelevant Components Using a Simple Model of Simultaneous Masking", IEEE Transactions on Audio, Speech and Language Processing, Vol. 17 (7) , in press (2009) , preprint
- P. Majdak, P. Balazs, B.Laback, "Multiple Exponential Sweep Method for Fast Measurement of Head Related Transfer Functions", Journal of the Acoustical Engineering Society , Vol. 55, No. 7/8, July/August 2007, Pages 623 - 637 (2007)

This project ended on 01.01.2010; most subprojects ended on 28.02.2008 and are incorporated into the 'High Potential'-Project of the WWTF, MULAC.

Weighted and controlled frames were introduced to speed up the inversion algorithm for the frame matrix of a wavelet frame. In this project, these kinds of frames are investigated further.

The frame multiplier concept is closely linked to the weighted frames concept. The frame operator of the weighted frame is simply a frame multiplier of the original frame. This project aims to explore this synergy. Finding an efficient way to invert the frame operator by applying weights to a given frame would be especially interesting. Weights are searched, for which the frame bounds are as "tight" as possible, meaning the spectrum is more concentrated. The first numerical experiments to find optimal weights have been conducted.

Weighted frames have already been applied to wavelets on the sphere. Also, the original work by Duffin and Schaefer dealt with the problem of finding such weights for a sequence of exponentials.

- J. P. Antoine, Unité de physique théorique et de physique mathématique – FYMA

Another project has investigated the basic properties of frame and Bessel multipliers. This project aims to generalize this concept so that it will work with Banach spaces also.

As the Gram matrix plays an important role in the investigation of multipliers, it is quite natural to look at the connection to localized frames and multipliers. The dependency of the operator class on the symbol class can be researched.

The following statements will be investigated:

- Theorem: If G is a localized frame and a is a bounded sequence, then the frame multiplier Ta is bounded on all associated Banach spaces (the associated co-orbit spaces).
- Theorem: If G is a localized frame and a is a bounded sequence, such that the frame multiplier Ta is invertible on the Hilbert space H, then Ta is simultaneously invertible on the associated Banach spaces.

The applications of these results to Gabor frames and Gabor multipliers will be further investigated.

Although Banach spaces are more general a concept than Hilbert spaces, Banach theory has found applications. For example, if any norm other than L2 (least square error) is used for approximation, Banach theory tools have to be applied.

- K. Gröchenig, NuHAG, Faculty of Mathematics, University of Vienna

This project ended on 28.02.2008 and is incorporated into the 'High Potential'-Project of the WWTF, MULAC.

In recent years, frames in signal processing applications have received more and more attention. Models, data, and operators must be discretized in order to function numerically. As a result, applications and algorithms always work with finite dimensional data. In the finite dimensional case, frames are equivalent to a spanning system. If reconstruction is wanted, frames are the only feasible generalization of bases. In contrast to bases, frames lose their linear independency. This project aims to investigate the properties of frames in finite dimensional spaces.

In this project, we will implement algorithms to work with frames in finite dimensional spaces. We will look at a way to "switch" between different frames, i.e. find a way to bijectively map between their coefficient spaces and provide the corresponding algorithm. This will be done by using the Cross-Gram matrix of the two involved frames. This matrix is a canonical extension of the basic transformation matrix used for orthonormal bases (ONB). The properties of the Gram matrix use a frame and its dual. We will investigate a criterion for finite dimensional spaces using frames. In particular, a space is finite dimensional if and only if Σ||g_{k}||^{2} < ∞.

Any analysis / synthesis system that allows perfect reconstruction is equivalent to a frame in its discrete version. This can be applied to Gabor, wavelet, or any other such system (e.g. a Gamma tone filter bank).

- P. Balazs,
*"Frames and Finite Dimensionality: Frame Transformation, Classification and Algorithms"*, submitted (2006), preprint - P. Balazs, "Matrix Representation of Operators Using Frames", Sampling Theory in Signal and Image Processing (STSIP) (2007, accepted), preprint

Practical experience quickly revealed that the concept of an orthonormal basis is not always useful. This led to the concept of frames. Models in physics and other application areas (for example sound vibration analysis) are mostly continuous models. Many continuous model problems can be formulated as operator theory problems, such as in differential or integral equations. Operators provide an opportunity to describe scientific models, and frames provide a way to discretize them.

Sequences are often used in physical models, allowing numerically unstable re- synthesis. This can be called an "unbounded frame". How this inversion can be regularized is being investigated. For many applications, a certain frame is very useful in describing the model. Therefore, it is also beneficial to use the same sequence to find a discretization of involved operators.

In this project, the theory of frames in the finite discrete case is investigated further.

The standard matrix description of operators using orthonormal bases is extended to the more general case of frames.

Weighted and controlled frames were introduced to speed up the inversion algorithm for the frame matrix of a wavelet frame. In this project, these kinds of frames are investigated further.

In this project, one function's sequences of irregular shifts are investigated.

- S. Heineken, Research Group on Real and Harmonic Analysis, University of Buenos Aires
- J. P. Antoine, Unité de physique théorique et de physique mathématique – FYMA
- M. El-Gebeily, Department of Mathematical Sciences, King Fahd University of Petroleum and Minerals, Saudi Arabia

Measuring sound absorption is essential to performing acoustic measurements and experiments under controlled acoustic conditions, especially considering the acoustic influence of room boundaries.

So-called "in-situ" methods allow measurement of the reflection and absorption coefficients under real conditions in a single measurement procedure. The method proposed captures the direct signal and reflections in one measurement. These reflections not only include the direct, interesting one, but also others from the surroundings. To separate the reflections coming from the tested surface, the influence of the direct signal and other reflections must be cancelled.

One known separation method uses a time-windowing technique to separate the direct signal from the reflections. When the impulse response of the direct signal and reflections overlap in time, this method is no longer satisfactory. Frequency-dependent windowing is necessary to separate the different parts of the signal. However, in the wavelet domain, it is possible to observe separation of the interesting reflection.

The objective of this project is to study how the use of wavelet multipliers could improve the efficiency of the in-situ methods in this context .

A demonstrator system will be built to acquire the necessary measurements for the evaluation of absorption coefficients. This demonstrator will be used to evaluate the usefulness of the new methods in a semi-anechoic room.

A systematic numeric study will be carried out on the acquired signals, in order to manually determine the symbol of a wavelet multiplier for the extraction of the reflected signal. The best parameters for optimal separation will then be investigated. This, in combination with the use of physical models, will help design a semi-automatic method for the calculation of the optimal multiplier symbol.

The improved measurement method will be available for in-situ measurement of reflection and absorption coefficients

The Multiple Exponential Sweep Method (MESM) is an optimized method for the semi-simultaneous system identification of multiple systems. It uses an appropriate overlapping of the excitation signals. This leads to a faster identification of the weakly nonlinear systems that are retrieving the linear impulse response only. Using a Gabor multiplier in the post-processing procedure of the system identification may reduce the measurement noise. This may further improve the signal-to-noise ratio of the measured data.

A Gabor multiplier is used to cut the interesting parts out of the measured signals in the time-frequency plane. This allows a specific optimization of signal parts, independent of the frequency. Initial tests applying a Gabor multiplier to simulated data showed that the depth of spectral notches could be raised. A systematic investigation of this method is the main goal this project.

This method may improve the signal-to-noise ratio in system identification tasks of any weakly nonlinear system, such as those involving acoustic measurements with electric equipment.

- P. Majdak, P. Balazs, B.Laback, "Multiple Exponential Sweep Method for Fast Measurement of Head Related Transfer Functions", Journal of the Acoustical Engineering Society , Vol. 55, No. 7/8, July/August 2007, Pages 623 - 637 (2007)

This project ended on 28.02.2008 and is incorporated into the 'High Potential'-Project of the WWTF, MULAC (q.v.).

The applications involving signal processing algorithms (like adaptive or time variant filters) are numerous. If the STFT, the Short Time Fourier Transformation, is used in its sampled version, the Gabor transform, the use of Gabor multipliers creates a possibility to construct a time-variant filter. The Gabor transform is used to calculate time frequency coefficients, which are multiplied with a fixed time-frequency mask. Then the result is synthesized. If another way of calculating these coefficients is chosen or if another synthesis is used, many modifications can still be implemented as multipliers. For example, it seems quite natural to define wavelet multipliers. Therefore, for this case, it is quite natural to continue generalizing and look at multipliers with frames lacking any further structure.

Therefore, for Bessel sequences, the investigation of operators

M = ∑ mk < f , ψk > φk

where the analysis coefficients, < f , ψk >, are multiplied by a fixed symbol mk before resynthesis (with φk), is very natural and useful. These are the Bessel multipliers investigated in this project. The goal of this project is to set the mathematical basis to unify the approach to the Bessel multipliers for all possible analysis / synthesis sequences that form a Bessel sequence.

Bessel sequences and frames are used in many applications. They have the big advantage of allowing the possibility to interpret the analysis coefficients. This makes the formulation of a multiplier concept for other analysis / synthesis systems very profitable. One such system involves gamma tone filter banks, which are mainly used for analysis based on the auditory system.

- Balazs, P. (2007), "Basic Definition and Properties of Bessel Multipliers", Journal of Mathematical Analysis and Applications, 325, 1: 571--585. doi:10.1016/j.jmaa.2006.02.012, preprint

This project ended on 01.01.2007. Its completion allowed the sucessfull application for a 'High Potential'-Project of the WWTF, see MULAC.

Gabor multipliers are an efficient tool for time variant filtering used implicitly in many engineering applications in signal processing. For these operators, the result of a Gabor transform (the sampled version of the Short Time Fourier Transform) is multiplied by a fixed function, called the time-frequency mask or symbol. The result is then synthesized.

While Gabor multipliers are widely and practically used, some of their theoretical properties are not well known. The goal of this project is to improve the mathematical knowledge about Gabor multipliers, in order to optimize their use in applications.

The problem will be approached using modern Gabor theory, harmonic analysis tools, and numeric tools. Formulation and demonstration of analytical statements will be conducted jointly with systematic numeric experiments to study the properties of Gabor multipliers.

The following topics will be investigated in the project:

- Eigenvalues and eigenvectors of Gabor multipliers and their localization
- Invertibility and injectivity of Gabor multipliers
- Reproducing kernel invariance
- Connection of irregular Gabor multipliers and irregular frames of translates
- Discretization and implementation of Gabor multipliers
- Best approximation of operators by Gabor multipliers and identification of Gabor multipliers.

The applications of Gabor multipliers in signal processing are numerous, and include any application requiring time-variant filtering. Some applications of Gabor multipliers will be investigated further in the following parallel projects:

- Mathematical Modeling of Auditory Time-Frequency Masking Functions
- Improvement of Head-Related Transfer Function Measurements
- Advanced Method of Sound Absorption Measurements

The implementation of a Gabor multiplier in the software system STx has already proceeded quite far, see Stx-Mulac.

- Monika Dörfler and Bruno Torrésani, “Representation of operators in the time-frequency domain and generalized Gabor multipliers”, J. Fourier Anal. Appl., 2009 (in press)
- Yohan Frutiger: "Multiplicateurs de Gabor pour les transformations sonores" (Gabor Multipliers for sound transformations) Master thesis under the supervision of R. Kronland-Martinet, June 2008
- F. Jaillet, P. Balazs, M. Dörfler and N. Engelputzeder, “On the Structure of the Phase around the Zeros of the Short-Time Fourier Transform”, NAG/DAGA 2009, International Conference on Acoustics, March 2009, Rotterdam, Nederland
- F. Jaillet, P. Balazs and M. Dörfler, “Nonstationary Gabor Frames”, SampTA'09, 8th International Conference on Sampling and Applications, May 2009, Marseille, France

This project line has the goal of finding efficient algorithms for signal processing applications. To apply the results of signal processing, Gabor or wavelet theory, the algorithms must be formulated for finite dimensional vectors. These discrete results are motivated by the continuous setting, but also often provide some insight. Furthermore, the efficient implementation of algorithms becomes important. For the consistency of these algorithms, it is useful to incorporate them into a maintained software package.

- Double Preconditioning for Gabor Frames: This project develops a way to find an analysis-synthesis system with perfect reconstruction in a numerically efficient way using double preconditioning.
- Perfect Reconstruction Overlap Add Method (PROLA): The classic overlap-add synthesis method is systematically compared to a new method motivated by frame theory.
- Numerics of Block Matrices: In some applications in acoustics, it is apparent that block matrices are a powerful tool to find numerically efficient algorithms.
- Practical Time Frequency Analysis: This project evaluates the usefulness of a time-frequency toolbox for acoustic applications and STx.

- H.G. Feichtinger et al., NuHAG, Faculty of Mathematics, University of Vienna
- B. Torrésani, Groupe de Traitement du Signal, Laboratoire d'Analyse Topologie et Probabilités, LATP/ CMI, Université de Provence, Marseille
- P. Soendergaard, Department of Mathematics, Technical University of Denmark
- J. Walker, Department of Mathematics, University of Wisconsin-Eau Claire

- P. Balazs, H.G. Feichtinger, M. Hampejs, G. Kracher; "Double Preconditioning for Gabor Frames”; IEEE Transactions on Signal Processing, Vol. 54, No.12, December 2006 (2006), preprint
- P. Balazs, H.G. Feichtinger, M. Hampejs, G. Kracher; "Double Preconditioning for the Gabor Frame Operator”; Proceedings ICASSP '06, May 14-19, Toulouse, DVD (2006)

In signal processing, synthesis is important in addition to analysis. This is especially true for the modification of data. For the Short-Time Fourier Transformation, the synthesis is often done using a simple overlap add (OLA), which is the sum of the outputs of the filter. Also, the output is re-weighted with the analysis window, such as occurs when using the phase vocoder. It is often presumed that with standard windows this will give satisfactory results.

Aside from Gabor frame theory, if the well-known construction of synthesis windows was possible, it would guarantee perfect reconstruction. However, this method is not used often in signal processing algorithms.

In this project, we will systematically investigate if and for which parameters the respective OLA synthesis with the original window gives good reconstruction. We will compare it to the reconstruction with the dual window, introducing and motivating it as perfect reconstruction overlap add (PROLA). We will show that this method is always preferable to others and that it can be calculated very efficiently.

This is currently being implemented in STx. There the phase vocoder will have the option to guarantee perfect reconstruction, either with dual or tight windows.

Department of Mathematics, University of Wisconsin-Eau Claire

Gabor multipliers are an efficient time-variant filtering tool used implicitly in many engineering applications in signal processing. For these operators, the result of a Gabor transform (the sampled version of the Short Time Fourier Transform) is multiplied by a fixed function (the time-frequency mask or symbol). The result is then synthesized.

Transforms other than the Gabor transform, particularly the wavelet transform, are more suitable for certain applications. The concept of multipliers can easily be extended in this case. This results in the introduction of operators called wavelet multipliers, which will be investigated in detail in this project. The project aims to precisely define wavelet multipliers' mathematical properties and optimize their use in applications.

The problem will be approached using modern wavelet theory, harmonic analysis tools, and numeric tools. Formulation and demonstration of analytic statements will be conducted jointly with systematic numeric experiments in order to study the properties of wavelet multipliers.

The following topics will be investigated in the project:

- Eigenvalues and eigenvectors of wavelet multipliers
- Invertibility and injectivity of wavelet multipliers
- Reproducing kernel invariance
- Discretization and implementation of wavelet multiplier
- Best approximation of operators by wavelet multipliers and identification of wavelet multipliers

The applications of wavelet multipliers in signal processing are numerous and include any application requiring time-variant filtering. Some applications of wavelet multipliers will be investigated further in the parallel projects:

- Mathematical Modeling of Auditory Time-Frequency Masking Functions
- Improvement of Head-Related Transfer Function Measurements
- Advanced Method of Sound Absorption Measurements

- Anaïk Olivero: "Expérimentation des multiplicateurs temps-échelle" (On the time-scale multipliers) Master thesis under the supervision of R. Kronland-Martinet and B. Torrésani, June 2008

Practical experience has shown that the concept of an orthonormal basis is not always useful. This led to the concept of frames. Models in physics and other application areas, including sound vibration analysis, are mostly continuous models. Many continuous model problems can be formulated as operator theory problems, as in differential or integral equations. An interesting class of operators is the Hilbert Schmidt class. This project aims to find the best approximation of any matrix by a frame multiplier, using the Hilbert Schmidt norm.

In finite dimensions, every sequence is a frame sequence, so the best approximation of any element can be found only via the frame operator using the dual frame for synthesis. Furthermore, the present best approximation algorithm involves the following steps: 1) The Hilbert-Schmidt inner product of the matrix and the projection operators involved is calculated in an efficient way; 2) Then the pseudo inverse of the Grame matrix is used to avoid the so-called calculation of the dual frames; The pseudo inverse is applied to the coefficients found above to find the lower symbol of the multiplier.

To find the best approximation of matrices via multipliers gives a way to find efficient algorithms to implement such operators. Any time-variant linear system can be modeled by a matrix. Time-invariant systems can be described as circulating matrices. Slowly-time-varying linear systems have a good chance at closely resembling Gabor multipliers. Other matrices can be well approximated by a "diagonalization" using other frames.

- P. Balazs, "Hilbert-Schmidt Operators and Frames - Classification, Approximation by Multipliers and Algorithms" , International Journal of Wavelets, Multiresolution and Information Processing, (2007, accepted) preprint, Codes and Pictures: here

This project ended on 01.01.2009. Its completion allowed the sucessfull application for a 'High Potential'-Project of the WWTF, see MULAC

So-called Gabor multipliers are particular cases of time-variant filters. Recently, Gabor systems on irregular grids have become a popular research topic. This project deals with Gabor multipliers, as a specialization of frame multipliers on irregular grids.

The initial stage of this project aims to investigate the continuous dependence of an irregular Gabor multiplier on its parameter (i.e. the symbol), window, and lattice. Furthermore, an algorithm to find the best approximation of any matrix (i.e. any time-variant system) by such an irregular Gabor multiplier is being developed.

Gabor multipliers have been used implicitly for quite some time. Investigating the properties of these operators is a current topic for signal processing engineers. If the standard time-frequency grid is not useful to the application, it is natural to work with irregular grids. An example of this is the usage of non-linear frequency scales, like bark scales.

H. G. Feichtinger, NuHAG, Faculty of Mathematics, University of Vienna

This project ended on 28.02.2008 and is incorporated into the 'High Potential'-Project of the WWTF, MULAC (WWTF 2007).

Time-variant filters are gaining more importance in today's signal processing applications. Also, there wavelet analysis has numerous applications. The goal of this project is to investigate time-variant systems based on wavelet analysis.

The concept of multipliers can be easily extended to wavelet frames. This means the coefficients of a wavelet analysis are multiplied by a fixed symbol and then resynthesized. The influence of the special structures of these sequences on the resulting operators will be investigated.

The theory of Pseudo-Differential Operators (PDO) can be translated to the wavelet case. How operators of interest in the investigation of multipliers, like the Kohn-Nirenberg correspondence, are translated to this case is of particular interest. Natural starting points for the research are:

Use dilations in the definition of the spreading function instead of modulation.

Define a special wavelet kernel function by using a weak formulation:

< K f , g > = < k , Wg f >

A very useful application for this project is an analysis-modification-synthesis system based on the wavelet analysis. With some language manipulation, this could be called a "Wavelet Phase Vocoder".

The application investigated in this project is the measurement of reflection coefficients. The wavelet analysis is preferable for signals containing transient parts. It is essential to separate the impulse responses of different reflections in order to calculate the absorption coefficient of a sound-proof wall. The impulse responses can be easily separated in a scalogram, and they can be extracted by using a wavelet multiplier.

This project ended on 28.02.2008 and is incorporated into the 'High Potential'-Project of the WWTF, MULAC.

HASSIP is a Research Training Network funded by the European Commission within the Improving the Human Potential program. The aim of the HASSIP network is to develop research activities and systematic interactions in mathematical analysis and statistics that are directly connected to signal and image processing. Although the Acoustics Research Institute was not initially a partner of this network, P. Balazs became a fellow of this network through cooperation with the group NuHAG.

- NuHAG, Faculty of Mathematics, University of Vienna
- Groupe de Traitement du Signal, Laboratoire d'Analyse Topologie et Probabilités, LATP/ CMI, Université de Provence, Marseille
- Modélisation, Synthèse et Contrôle des Signaux Sonores et Musicaux des LMA / CRNS Marseille
- Unité de physique théorique et de physique mathématique – FYMA

- Basic Properties of Bessel and Frame Multipliers: For Bessel sequences, the investigation of operators M = ∑ mk < f , ψk > is very natural and useful. The above M are Bessel multipliers. The goal of this project is to set the mathematical basis for this kind of operator.
- Best Approximation of Matrices by Frame Multipliers: Finding the best approximation by multipliers of matrices that represent time-variant systems gives a way to find efficient algorithms to implement such operators.

- P. Balazs, "Hilbert-Schmidt Operators and Frames - Classification, Approximation by Multipliers and Algorithms" , International Journal of Wavelets, Multiresolution and Information Processing, Vol. 6, No. 2, pp. 315 - 330, March 2008, preprint, Codes and Pictures: here
- P. Balazs, "Basic Definition and Properties of Bessel Multipliers", Journal of Mathematical Analysis and Applications, 325, 1: 571--585. (2007) doi:10.1016/j.jmaa.2006.02.012, preprint

This project ended on 01.01.2009. Its completion allowed the sucessfull application for a 'High Potential'-Project of the WWTF, see MULAC.

The most basic model for convolution algorithms is an extension of the simultaneous irrelevance model. A triangle-like function describes the masking effect in the frequency and time direction. Combined, they result in a 2-D function, which is used as convolution on the time-frequency coefficients of the given signal. The resulting information is then used to calculate a threshold function. This can be implemented as a Gabor multiplier. This very simple function should be exchanged for a more elaborate 2-D kernel. A more elaborate 2-D kernel can be developed from the first time frequency masking effect measurements of a Gaussian atom.

An extension of the simultaneous irrelevance model is used as the most basic model for the convolution algorithm under investigation. A triangle-like function describes the masking effect in the frequency and time direction. Combined, they result in a 2-D function, which is used as convolution on the time-frequency coefficients of the given signal to calculate a threshold function. This can be implemented as a Gabor multiplier. This very simple function should be exchanged for a more elaborate 2-D kernel developed from the first time-frequency masking effect measurements of a Gaussian atom.

After thoroughly testing this algorithm in psychoacoustic experiments, it will be implemented in STx.

- R. Kronland-Martinet, S. Ytad, T. Necciari, Modélisation, Synthèse et Contrôle des Signaux Sonores et Musicaux of the LMA / CRNS Marseille
- S. Meunier, S. Savel, Acoustique perceptive et qualité de l’environnement sonore of the LMA / CRNS Marseille

This project ended on 28.02.2008 and is incorporated into the 'High Potential'-Project of the WWTF, MULAC.

An irrelevance algorithm based on simultaneous masking is implemented In STx. In the years following its first development by Eckel, the efficiency of this algorithm has been clearly shown. In this project, this irrelevance model will be based on modern mathematic and psychoacoustic theories and knowledge.

This algorithm can be described as a Gabor multiplier with an adaptive symbol. With existing related theory, it becomes clear that a high redundancy must be selected. This guarantees:

- perfect reconstruction synthesis
- an under-spread operator for good time-frequency localization
- a smoothing-out of easily detectable quick on/off cycles

Furthermore, it can be shown that the model used for the spreading function here is mathematically equivalent to the excitation pattern.

This algorithm has been used for several years already for things such as:

- automobile sound design
- over-masking for background-foreground separation
- improved speech recognition in noise
- contrast increase for hearing-impaired persons

- G. Eckel, Institut für Elektronische Musik und Akustik, Graz

- P. Balazs, B. Laback, G. Eckel, W. Deutsch, "Introducing Time-Frequency Sparsity by Removing Perceptually Irrelevant Components Using a Simple Model of Simultaneous Masking", IEEE Transactions on Audio, Speech and Language Processing, Vol. 17 (7) , in press (2009) , preprint

This project ended on 01.01.2010, and leads to a sub-project of the 'High Potential'-Project of the WWTF, MULAC.

The FWF project "Time-Frequency Implementation of HRTFs" has started.

Principal Investigator: Damian Marelli

Co-Applicants: Peter Balazs, Piotr Majdak

Read more: Time-Frequency Implementation of HRTFs (HRTF-Imp)