## Project

### Objective:

The Short-Time Fourier Transform (STFT), in its sampled version (the Gabor transform), is a well known, valuable tool for displaying the energy distribution of a signal over the time-frequency plane. The equivalence between Gabor analysis and certain filter banks is a well-known fact. The main task is how to find a Gabor analysis-synthesis system with perfect (or depending on the application, satisfactorily accurate) reconstruction in a numerically efficient way. This is done by using the dual Gabor frame, which implies the need to invert the Gabor frame operator.

### Method:

This project incorporates an application of the general idea of preconditioning in the context of Gabor frames. While most (iterative) algorithms aim at a relatively costly exact numeric calculation of the inverse Gabor frame matrix, we will use a "cheap method" to find an approximation. The inexpensive method will be based on (double) preconditioning using diagonal and circulant preconditioners. As a result, good approximations of the true dual Gabor atom can be obtained at low computational costs.

### Application:

For a number of applications, such as time stretching without changing the frequency content in audio processing or more complex modifications like psychoacoustical masking, the time domain signal needs to be reconstructed using the time-frequency domain coefficients.

### Partners:

H. G. Feichtinger et al., NuHAG, Facultyof Mathematics, University of Vienna

### Publications:

• P. Balazs, H.G. Feichtinger, M. Hampejs, G. Kracher; "Double Preconditioning for Gabor Frames"; IEEE Transactions on Signal Processing, Vol. 54, No.12, December 2006 (2006), preprint
• P. Balazs, H.G. Feichtinger, M. Hampejs, G. Kracher; "Double Preconditioning for the Gabor Frame Operator"; Proceedings ICASSP '06, May 14-19, Toulouse, DVD (2006)

### Objective:

Previous studies show that cochlear implant (CI) listeners show sensitivity to interaural time difference (ITD) in the fine structure at comparable, or sometimes even higher pulse rates than normal hearing (NH) subjects. This study investigates whether the differences between the two subject groups are due to an effect of auditory filtering that is absent in the case of electric stimulation.

### Method:

The effects of center frequency and pulse rate on the sensitivity to ongoing envelope ITD were investigated using bandpass-filtered pulse trains. Three center frequencies (4.6, 6.5, and 9.2 kHz) were tested, and the bandwidth was scaled to stimulate an approximately constant range on the basilar membrane. The pulse rate was varied from 200 to 588 pulses per second (pps).

### Results:

The results show a small but significant decrease in performance with an increase in center frequency. Furthermore, performance decreases with an increase in pulse rate, yielding a rate limit of approximately 500 pps. The lack of an interaction between pulse rate and center frequency indicates that auditory filtering was not the limiting factor in ITD perception. This suggests the existence of other limiting mechanisms, such as phase locking or more central binaural processes. The comparison of the ITD rate limits in CI subjects with those in NH subjects was considered unaffected by the auditory filtering in NH listeners.

### Funding:

FWF (Austrian Science Fund): Project # P18401-B15

### Publications:

• Laback, B. and Majdak, P. (2007). Effect of Center Frequency on the Sensitivity to Interaural Time Differences in Filtered Pulse Trains, proceedings of DAGA 2007, Stuttgart.
• Majdak, P., and Laback, B. (2008). Effect of center frequency and rate on the sensitivity to interaural delay in high-frequency click trains, J. Acoust. Soc. Am. (under review).

### Objective:

This project investigates the effect on cochlear implant (CI) speech understanding caused by spectral peaks and notches, such as those resulting from the head-related transfer function filtering of a sound source. This is required to determine how spectral localization cues are best encoded with CIs, without destroying speech information.

### Application:

Results from this project are required for the development of a 3-D localization strategy for CIs. Furthermore, the results give insight into the robustness of speech cues against spectral disruption in electric hearing.

### Funding:

FWF (Austrian Science Fund): Project #P18401-B15

### Objective:

Bilateral cochlear implant (CI) listeners currently use stimulation strategies that encode

interaural time differences (ITD) in the temporal envelope. However, the strategies do not transmit ITD in the fine structure, because of the constant phase in the electric pulse train. To determine the utility of encoding ITD in the fine structure, ITD-based lateralization was investigated with four CI listeners and four normal hearing (NH) subjects who listened to a simulation of electric stimulation.

### Methods und Results:

Lateralization discrimination was tested at different pulse rates for various combinations of

independently controlled fine structure ITD and envelope ITD. Results for electric hearing show that the fine structure ITD had the strongest impact on lateralization at lower pulse rates, with significant effects for pulse rates up to 800 pulses per second. At higher pulse rates, lateralization discrimination depended solely on the envelope ITD. The data suggest that bilateral CI listeners benefit from transmitting fine structure ITD at lower pulse rates. However, there were strong inter-individual differences: the better performing CI listeners performed comparably to the NH listeners.

### Application:

The result that bilateral CI listeners benefit from transmitting fine structure ITD at lower pulse rates is relevant to future CI stimulation strategies that encode fine timing cues. It is expected that appropriate encoding of these cues improves sound localization abilities and speech understanding in noise.

Internal

### Publications:

Majdak, P., Laback, B., Baumgartner., W.D. (2006). Effects of interaural time differences in fine structure and envelope on lateral discrimination in electrical hearing, J. Acoust. Soc. Am. 120, 2190-201.

### Objective and Methods:

This study examined the sensitivity of four cochlear implant (CI) listeners to ITD in different portions of four-pulse sequences in lateralization discrimination. ITD was present either in all the pulses (referred to as condition "Wave"), the two middle pulses (Ongoing), the first pulse (Onset), the last pulse (Offset), or both the first and last pulse (Gating). All ITD conditions were tested at different pulse rates (100, 200, 400, and 800 pulses per second, pps). Also, five normal hearing (NH) subjects were tested. The NH subjects listened to an acoustic simulation of CI stimulation.

### Results:

All CI and NH listeners were sensitive in condition "Gating" at all pulse rates for which they showed sensitivity in condition "Wave". The sensitivity in condition "Onset" increased with the pulse rate for three CI listeners as well as for all NH listeners. The performance in condition "Ongoing" varied among the subjects. One CI listener showed sensitivity up to 800 pps, two up to 400 pps, and one at 100 pps only. The group of NH listeners showed sensitivity up to 200 pps.

### Application:

CI listeners' ability to detect ITD from the middle pulses of short trains indicates fine timing relevance of stimulation pulses to lateralization. This is also relevant to CI stimulation strategies.

Internal

### Publications:

• Laback, B., Majdak, P., Baumgartner, W. D. (2007). Lateralization discrimination of interaural time delays in four-pulse sequences in electric and acoustic hearing, J. Acoust. Soc. Am. 121, 2182-2191.
• Laback, B., Majdak, P., Baumgartner., W.D. (2005). Interaural time differences in temporal fine structure, onset, and offset in bilateral electrical hearing, presented at the 28th Meeting of the Association for Research in Otolaryngology, New Orleans.
• Laback, B., Majdak, P., Baumgartner, W. D. (2006). Interaural Time Differences in Ongoing and Gating Signal Portions in Acoustic and Electric Hearing: Model Results, Proceedings of DAGA 2006, Braunschweig.
• Laback, B., Majdak, P., Baumgartner, W. D. (2005). Fine structure and gating interaural time differences in electrical and acoustical hearing: effects of stimulus duration, presented at the Conference on Implantable Auditory Prostheses (CIAP), Asilomar.
• Laback, B., Majdak, P., Baumgartner., W.D. (2004). Sensitivity to interaural time differences in temporal fine-structure, onset, and offset in bilateral electrical hearing, presented at 5th Wullstein Symposium on Bilateral Cochlear Implants and Binaural Signal Processing, Würzburg.

### Objective:

This project studies the effects of the upper-frequency boundary and of spectral warping on speech intelligibility among Cochlear Implant (CI) listeners, using a 12-channel implant, and normal hearing (NH) listeners.  This is important to determine how many basal channels are "free" for encoding spectral localization cues.

### Results:

The results show that eight frequency channels and spectral content up to about 3 kHz are sufficient to transmit speech under unwarped conditions. If frequency warping was applied, the changes had to be limited ± 2 frequency channels to preserve good speech understanding. This outcome shows the range of allowed modifications for presenting spectral localization cues to CI listeners. About four channels were found to be "free" for encoding spectral localization cues

### Application:

see the description of the CI-HRTF project

### Funding:

FWF (Austrian Science Fund): Project #P18401-B15

### Publications:

• Goupell, M., Laback, B., Majdak, P., and Baumgartner, W. D. (2007). Effects of upper-frequency boundary and spectral warping on speech intelligibility in electrical stimulation, J. Acoust. Soc. Am. 123, 2295-2309.
• Goupell, M. J., Laback, B., Majdak, P., and Baumgartner, W-D. (2007). Effect of frequency-place mapping on speech intelligibility: implications for a cochlear implant localization strategy, presented at Conference on Implantable Auditory Prostheses (CIAP), Lake Tahoe.
• Goupell, M. J., Laback, B., Majdak, P., and Baumgartner, W-D. (2007). Effect of different frequency mappings on speech intelligibility for CI listeners, proceedings of DAGA 2007, Stuttgart.

### Introduction:

The Eval facility is an expandable and versatile class library for evaluating expressions of mixed type. It adds the Eval function to the STx programming language.

Eval supports real and complex numeric data as well as vectors and matrices of real and complex numbers. Eval supplies a large number of operations and functions for each kind of data, ranging from scalar and vector additions to statistical functions, order analysis, and signal processing (e.g. the Fourier transformation, several kinds of windowing functions, filtering functions, etc.).

The Eval functionality is an improvement over the historically grown evaluation functions already present in the STx macro language, including the int and num keywords and the position-dependent implicit argument evaluation. Eval combines all their functionality with a large and growing part of the functions that have traditionally been available only for SPU programmers. Over time, Eval is intended to provide a uniform and homogeneous means of accessing any of the diverse STx functions, regardless of their individual history.

### Benefits:

• Eval offers a uniform and homogeneous interface to the numeric functions available in STx, including those that were available only to SPU programmers in the past.
• Eval empowers the STx programmer to use the programming design and style that best suits his or her needs, without being guided by existing restrictions like access only via SPUs. Thus, algorithmic problems may now be solved algorithmically, whereas the use of SPUs may be restricted to problems where there is a continuous flow of data that must be processed online.
• Eval offers support for reference arguments. This means that an Eval function may return more than one data item (be it a number, a vector, or a matrix), which is not possible with the existing int, num, and implicit argument evaluation because of their outdated design. The support for reference arguments makes the interfaces of many functions more natural and, in some cases, is a necessary condition for implementing them at all (as opposed to offering them as SPU atoms only). At this time, there is already a growing number of functions that make use of this feature.
• Due to the power of the already existing Eval functions, even complex calculations may be done by typing relatively simple formulae. By resulting in STx macro expressions that are shorter and more readable, this speeds up STx-based software development processes. It also both reduces the likelihood of programming errors happening and speeds up debugging and future maintenance.
• Due to its carefully laid-out design, Eval offers a very thorough and fine-grained kind of error checking and reporting that has not been possible with the outdated int, num, and implicit argument evaluation. Compared with the traditional evaluation routines, Eval offers more than 25 additional error codes and error messages, thereby helpfully pointing even the less experienced programmer to the cause of their error.
• Eval uses a manually laid-out, and manually optimized recursive descent parser (meaning that no excessively complicated or time-consuming compiling tools must be involved). This results in a very short and straightforward C++ source code that is easy to read and to extend and that compiles to a very compact and cache-effective binary object.
• The STx integration of research results has further improved with the availability of Eval due to the fact that researchers may directly integrate their algorithms with the Eval functionality instead of programming SPU atoms of their own, as was necessary in the past.

### Design:

The Eval class implements a single-pass, recursive-descent parser for the following grammar:

   AddSub := MulDiv ( "+" | "-" MulDiv )*   MulDiv := Pwr ( "*" | "/" | "%" Pwr )*   Pwr := BracketOrAtom ( "^" BracketOrAtom )*   BracketOrAtom :="(" AddSub ")" |   "|" AddSub "|" |   "-" BracketOrAtom |   FunctionName "(" ArgumentList? ")" |   some STX item   ArgumentList := AddSub ( "," AddSub )*

The functionality of Eval is implemented in the CExEval class. Built-in functions are listed in a table both referring to their implementation and containing information of the number and kind of the arguments to the respective function. Adding a new built-in function is done by providing an implementation for the respective function and by adding an entry to the aforementioned table.

### Available Eval functions

The following is a short list of some of the most important Eval functions available so far:

• measurement functions: Cols, Rows, Min, Max, IMin, IMax, Sum, WSum, Abs, QDet
• vector initialization and transformation: Fill, Init, Rand, VSubC, VSubN, Select, Sort, VV, VVCat, VVSet, VMCol, VMRow, Interp,Trn
• trigonometric, transcendental, and other special functions: Sin, Cos, Tan, ASin, ACos, ATan, Exp, Sqrt, Log, Ln, Ld, Pi, E
• complex arithmetic: CR2P, CP2R, CR2Len, CR2Phi, CGet, CSet, CMul
• integration and differentiation: YDiff, YInt
• statistics: Avr, Var, StdDev, Corr, Median, Hist, Dist, ModClust, HAClust, EM (Expectation Maximization), Density
• data conversion: Lin2Log, Log2Lin, Hz2Cent, Cent2Hz, Hz2Bark, Bark2Hz, Hz2Mel, Mel2Hz, Hz2ERB, ERB2Hz
• signal windowing functions: Window, WHanning, WHamming, WBartlett, WTapRect, WNutTall, WFlatTop, WBlackman, WKaiser, WGauss
• signal processing: FFT, IFFT, DFT, Cepstrum, LPC, DCT, ZCross, IIR1, IIR2, FIR1, FIR2
• order spectrum analysis: otrack1, sig2osp, asp2osp, ticks2f1
• miscellaneous functions: IPeak, QInterp, RPoly, RPolyRef, HTH, ASeg1

### Objective:

 ExpSuite is a program that compiles the implementation of psychoacoustic experiments. ExpSuite is the name of a framework that is used as a basis for an application. It can be enlarged with customized and experiment-dependent methods (applications). The framework consists of a user-interface (experimentator-and-subject interface), signal processing modules (off-line and in real-time), and input-output modules. The user-interface is implemented in Visual Basic.NET and benefits from the "Rapid Application Development" environment, which develops experiments quickly. To compensate for the sometimes slow processing performance of VB, the stimulation signals can be processed in a vector-oriented way using a direct link to MATLAB. Because of the direct link to MATLAB, numerous MATLAB intern functions are available to the ExpSuite applications.

The interface accessible to the people administering the tests contains several templates that can be chosen for a specific experiment. Either the keyboard, mouse, joypad, or joystick can be chosen as the input device. The user interface is designed for dual screen equipment, and allows a permanent surveillance of the experiment status on the same computer. Additionally, the transmission of the current experiment status to another computer is possible via a network connection.The framework supports two types of stimulation:

• the standard acoustic stimulation using an audio interface for experiments with normal or impaired hearing subjects, and
• the direct electric stimulation of cochlear implants for experiments with cochlear implant listeners.

### Objective:

In the boundary element method for infinite exterior spaces occur spurious modes that belong to the interior problem. The resonances are caused by a singularity of the matrix. As compensation additional equations are used that belong to chief points in the interior space.

### Method:

A new method that does not depend on the appropriate selection of chief points is the Burton Miller Method. It assures good results but increases the numerical calculation and gives unsymmetrical matrices. In combination with the Fast Multipole Method (FMM) a fast calculation of non symmetric matrices is possible.

• ### FLAME: Frames and Linear Operators for Acoustical Modeling and Parameter Estimation.

START project of P. Balazs.

Diese Seite ist eine Projektbeschreibung und als solche in englischer Sprache verfasst.

This international, multi-disciplinary and team-oriented project will expand the group Mathematics and Acoustical Signal Processing at the Acoustic Research Institute in cooperation with NuHAG Vienna (Hans G. Feichtinger, M. Dörfler, K. Gröchenig), Institute of TelecommunicationVienna (Franz Hlawatsch), LATP Marseille (Bruno Torrésani) LMA (Richard Kronland-Martinet). CAHR (Torsten Dau, Peter Soendergaard), the FYMA Louvain-la-Neuve (Jean-Pierre Antoine), AG Numerics (Stephan Dahlke), School of Electrical Engineering and Computer Science (Damian Marelli) as well as the BKA Wiesbaden (Timo Becker).

Within the institute the groups Audiological Acoustics and Psychoacoutics, Computational Acoustics, Acoustic Phonetics and Software Development are involved in the project.

This project is funded by the FWF as a START price . It is planned to run from May 2012 to April 2018.

### General description:

We live in the age of information where the analysis, classification, and transmission of information is f essential importance. Signal processing tools and algorithms form the backbone of important technologieslike MP3, digital television, mobile phones and wireless networking. Many signal processing algorithms have been adapted for applications in audio and acoustics, also taking into account theproperties of the human auditory system.

The mathematical concept of frames describes a theoretical background for signal processing. Frames are generalizations of orthonormal bases that give more freedom for the analysis and modificationof information - however, this concept is still not firmly rooted in applied research. The link between the mathematical frame theory, the signal processing algorithms, their implementations andfinally acoustical applications is a very promising, synergetic combination of research in different fields.

Therefore the main goal of this multidisciplinary project is to

-> Establish Frame Theory as Theoretical Backbone of Acoustical Modeling

in particular in psychoacoustics, phonetic and computational acoustics as well as audio engineering.

For this auspicious connection of disciplines, FLAME will produce substantial impact on both the heory and applied research.

The theory-based part of FLAME consists of the following topics:

• T1 Frame Analysis and Reconstruction Beyond Classical Approaches
• T2 Frame Multipliers, Extended
• T3 Novel Frame Representation of Operators Motivated by Computational Acoustics

The application-oriented part of FLAME consists of:

• A1 Advanced Frame Methods for Perceptual Sparsity in the Time-Frequency Plane
• A2 Advanced Frame Methods for the Analysis and Classification of Speech
• A3 Advanced Frame Methods for Signal Enhancement and System Estimation

• ### Forensic Speech and Audio Analysis

Forensic Speech Analysis is currently being developed using two main methodologies:

• Automatic methods, applying digital signal processing algorithms and Bayes Statistics.
• Acoustic Phonetics and Phonology based on acoustic measurements of speech parameters, such as formant frequencies and fundamental frequency of speech segments.

The Institute investigates both approaches in the framework of the FSAAWG (Forensic Speech and Audio Working Group) of ENFSI (the European Network of Forensic Science Institutes).

• ### Frame Multiplier: Theory and Application in Acoustics (WWTF 2007)

This project ended in September 2011.

### Meetings:

The final MulAc Meeting was in Vienna from 29th to 30th of August 2011.

The ARI Mulac Frame Meeting was held on Tuesday, June 15th 2010at ARI.

The MULAC Mid-term Meeting was held in Marseille from 12. to 13. April 2010. See the Registration-Webpage or the Program.

The FYMA Mulac seminar was held in Louvain-la-Neuve in the 11th of March, 2010. (Talks by Jean-Pierre Antoine, Jean-Pierre Gazeau, Diana Stoeva and Peter Balazs.)

The MULAC - Kick-Off Meeting took place at ARI in Vienna from September 23rd to 24th 2008.

This international, multi-disciplinary and team-oriented project allowed P. Balazs to form a small group 'Mathematics and Acoustical Signal Processing’ at the Acoustic Research Institute in cooperation with NuHAG Vienna (Hans G. Feichtinger), LMA (Richard Kronland-Martinet) and LATP Marseille (Bruno Torrésani) as well as the FYMA Louvain-la-Neuve (Jean-Pierre Antoine).

Within the institute the groups 'Audiological Acoustics' and 'Software Development' are involved.

This project is funded by the WWTF . It will run for 3,5 years and post-docs will be employed for six years total, as well as master students for 36 months total.

In December 2007 the Austrian Academy of Sciences was presenting 'mathematics in...' as the topic of the month . This included 'mathematics at the Acoustics Research Institute', which describes this project.

### General description:

"Frame Multipliers” are a promising mathematical concept, which can be applied to retrieve desired information out of acoustic signals. P. Balazs introduced them by successfully generalizing existing time-variant filter approaches. This project aims to establish new results in the mathematical theory of frame multipliers, to integrate them in efficient digital signal processing algorithms and to make them available for use in 'real-world' acoustical applications. A multi-disciplinary and international cooperation has been established and will be extended in the project to create new significant impulses for the involved disciplines: mathematics, numerics, engineering, physics and cognitive sciences. Various acoustical applications like modelling of auditory perception, measurement of sound absorption coefficients and system identification of the head related transfer functions are included. The results of the project will allow their future integration into practical areas such as audio coding, noise abatement, sound quality design, virtual reality and hearing aids.

### Objective:

In recent years, frames in signal processing applications have received more and more attention. Models, data, and operators must be discretized in order to function numerically. As a result, applications and algorithms always work with finite dimensional data. In the finite dimensional case, frames are equivalent to a spanning system. If reconstruction is wanted, frames are the only feasible generalization of bases. In contrast to bases, frames lose their linear independency. This project aims to investigate the properties of frames in finite dimensional spaces.

### Method:

In this project, we will implement algorithms to work with frames in finite dimensional spaces. We will look at a way to "switch" between different frames, i.e. find a way to bijectively map between their coefficient spaces and provide the corresponding algorithm. This will be done by using the Cross-Gram matrix of the two involved frames. This matrix is a canonical extension of the basic transformation matrix used for orthonormal bases (ONB). The properties of the Gram matrix use a frame and its dual. We will investigate a criterion for finite dimensional spaces using frames. In particular, a space is finite dimensional if and only if Σ||gk||2 < ∞.

### Application:

Any analysis / synthesis system that allows perfect reconstruction is equivalent to a frame in its discrete version. This can be applied to Gabor, wavelet, or any other such system (e.g. a Gamma tone filter bank).

### Objective:

Gabor multipliers are an efficient tool for time-variant filtering. They are used implicitly in many engineering applications in signal processing. For these operators, the result of a Gabor transform (the sampled version of the Short Time Fourier Transform) is multiplied by a fixed function (called the time-frequency mask or symbol). Then the result is synthesized.

Other transforms beyond the Gabor transform, for example the wavelet transform, are more suitable for certain applications. The concept of multipliers can easily be extended to these transforms. More precisely, the concept of multipliers can be applied to general frames without any further structure. This results in the introduction of operators called frame multipliers, which will be investigated in detail in this project in order to precisely define their mathematical properties and optimize their use in applications.

### Method:

The problem will be approached using modern frame theory, functional analysis, numeric tools, and linear algebra tools. Systematic numeric experiments will be conducted to observe the different properties of frame multipliers. This observations will support the analytical formulation and demonstration of these properties.

The following topics will be investigated in the project:

• Eigenvalues and eigenvectors of frame multipliers
• Invertibility, injectivity, and surjectivity of frame multipliers
• Reproducing kernel invariance
• Generalization of multipliers to Banach frames and p-frames
• Connection of frame multipliers to weighted frames
• Discretization and implementation of frame multipliers
• Best approximation of operators by frame multipliers and identification of frame multipliers

### Application:

The applications of frame multipliers in signal processing are numerous and include any application requiring time-variant filtering. Some applications of frame multipliers will be investigated further in the following parallel projects:

• Mathematical Modeling of Auditory Time-Frequency Masking Functions
• Improvement of Head-Related Transfer Function Measurements
• Advanced Method of Sound Absorption Measurements

### Publications:

• P. Balazs, "Matrix Representation of Bounded Linear Operators By Bessel Sequences, Frames and Riesz Sequence", SampTA'09, 8th International Conference on Sampling and Applications, May 2009, Marseille, France
• P. Balazs, J.-P. Antoine, A. Grybos, "Weighted and Controlled Frames: Mutual relationship and first Numerical Properties", accepted for publication in International Journal of Wavelets, Multiresolution and Information Processing (2009), preprint
• A. Rahimi, P. Balazs, "Multipliers for p-Bessel sequences in Banach spaces", submitted (2009)
• D. Stoeva, P. Balazs, "Unconditional convergence and Invertibility of Multipliers", preprint (2009)
• ### FTAA: Frame Theory and Asymptotic Analysis

Scientific and Technological Cooperation with Macedonia 2016-18
Project duration: 01.07.2016 – 30.06.2018

The main aim of the project is to combine the research areas of Frame Theory and Generalized Asymptotic Analysis.

Project partner institutions:
Acoustics Research Institute (ARI), Austrian Academy of Sciences, Vienna, Austria
Ss. Cyril and Methodius University, Skopje, The Former Yugoslav Republic of Macedonia

Project members:
Diana T. Stoeva (Project coordinator Austria), Peter Balazs, Nicki Holighaus, Zdenek Prusa
Katerina Hadzi-Velkova Saneva (Project coordinator FYROM), Sanja Atanasova, Pavel Dimovski, Zoran Hadzi-Velkov, Bojan Prangoski, Biljana Stanoevska-Angelova, Daniel Velinov, Jasmina Veta Buralieva

Project Workshops and Activities:

1) Nov. 24-26, 2016, Ss. Cyril and Methodius University, Skopje

Project Kickoff-workshop

Program of the workshop

2) Nov. 15-19, 2017, ARI, Vienna

Research on project-related topics

3) April 14-19, 2018, ARI, Vienna

Research on project-related topics

and

ARI-Guest-Talk given at ARI on the 17th of April, 2018: Prof. Zoran Hadzi-Velkov, "The Emergence of Wireless Powered Communication Networks"

4) May 25-30, Ss. Cyril and Methodius University, Skopje

Research on project-related topics

and

Workshop "Women in mathematics in the Balkan region" (May 28 - May 29, Ss. Cyril and Methodius University, Skopje)

5) June 14-18, Ss. Cyril and Methodius University, Skopje

Research on project-related topics

and

Summer course "An Introduction to Frame Theory and the Large Time/Frequency Analysis Toolbox" (June 14-15), Lecturers: Diana Stoeva and Zdenek Prusa (from ARI)

6) Mini-Symposium "Frame Theory and Asymptotic Analysis" organized at the European Women in Mathematics General Meeting 2018, Karl-Franzens-Universität Graz, Austria, 3-7 September 2018.

7) November 17-20, 2018, ARI, Vienna

Work on project-related topics

### Objective:

Gabor multipliers are an efficient tool for time variant filtering used implicitly in many engineering applications in signal processing. For these operators, the result of a Gabor transform (the sampled version of the Short Time Fourier Transform) is multiplied by a fixed function, called the time-frequency mask or symbol. The result is then synthesized.

While Gabor multipliers are widely and practically used, some of their theoretical properties are not well known. The goal of this project is to improve the mathematical knowledge about Gabor multipliers, in order to optimize their use in applications.

### Method:

The problem will be approached using modern Gabor theory, harmonic analysis tools, and numeric tools. Formulation and demonstration of analytical statements will be conducted jointly with systematic numeric experiments to study the properties of Gabor multipliers.

The following topics will be investigated in the project:

• Eigenvalues and eigenvectors of Gabor multipliers and their localization
• Invertibility and injectivity of Gabor multipliers
• Reproducing kernel invariance
• Connection of irregular Gabor multipliers and irregular frames of translates
• Discretization and implementation of Gabor multipliers
• Best approximation of operators by Gabor multipliers and identification of Gabor multipliers.

### Application:

The applications of Gabor multipliers in signal processing are numerous, and include any application requiring time-variant filtering. Some applications of Gabor multipliers will be investigated further in the following parallel projects:

• Mathematical Modeling of Auditory Time-Frequency Masking Functions
• Improvement of Head-Related Transfer Function Measurements
• Advanced Method of Sound Absorption Measurements

### Software:

The implementation of a Gabor multiplier in the software system STx has already proceeded quite far, see Stx-Mulac.

### Publications:

• Monika Dörfler and Bruno Torrésani, “Representation of operators in the time-frequency domain and generalized Gabor multipliers”, J. Fourier Anal. Appl., 2009 (in press)
• Yohan Frutiger: "Multiplicateurs de Gabor pour les transformations sonores" (Gabor Multipliers for sound transformations) Master thesis under the supervision of R. Kronland-Martinet, June 2008
• F. Jaillet, P. Balazs, M. Dörfler and N. Engelputzeder, “On the Structure of the Phase around the Zeros of the Short-Time Fourier Transform”, NAG/DAGA 2009, International Conference on Acoustics, March 2009, Rotterdam, Nederland
• F. Jaillet, P. Balazs and M. Dörfler, “Nonstationary Gabor Frames”, SampTA'09, 8th International Conference on Sampling and Applications, May 2009, Marseille, France

### Objective:

Up to now a boundary element method based on the collocation method was combined with the Fast Multipole Method to speed up the numerical calculation.

### Method:

This approach was chosen, because the collocation is a fast method to build up the matrix and the Fast Multipole Method (FMM)is a fast method to solve large matrices. For compatibility with the software HAMS the FMM has to be ported to a Galerkin based Boundary Element (BE) approach.

## Implications for pathological speech

Coordinated Project 2016-17 Scuola Normale Superiore (SNS), Pisa – Acoustic Research Institute (ARI), Austrian Academy of Science, Vienna
PIs: Chiara Celata (SNS), Sylvia Moosmueller (ARI)
Research personnel: Chiara Meluzzi (SNS), Bettina Hobel (ARI)

### Short Description

The project aims at modeling the impact of speech gesture coordination on the rhythmical properties of languages.

Speech gestural structures are sets of gestures and a specification of how they are temporally and spatially coordinated with respect to one another. Gestural anticipations, posticipations and overlap are the ingredients of coarticulation, i.e., the coordinatory activity of speech movements that allows adjacent vowels and consonants to be produced simultaneously, thus resulting into one smooth whole.

Rhythm is the systematic patterning of timing, accent, and grouping in sequences of events and encompasses both speech and music domains. We only become aware of how important it is in verbal communication when we listen to non-fluent speech. For example, deaf people with impaired or absent auditory feedback can be taught, after cochlear implantation and logopedic rehabilitation, to develop an “auditory” map for speech processing and imitation, but native-like patterns of gestural and rhythmical coordination are much more difficult to achieve.

Both gestural coordination and rhythm thus contribute to the way fluent speech is programmed, produced, and even perceived.

However, we still miss a global understanding of how the two dimensions of gestural coordination and speech rhythm interact in natural languages.

Indeed, the gestural and the rhythmical approaches sometimes make different predictions. For example, we do not know whether the consonants composing heterosyllabic clusters are articulatorily independent from one another and are timed with respect to different vocalic nuclei, as some theoretical frameworks in the domain of gestural coordination would predict, or whether they are rather globally timed with the preceding vocalic nucleus, especially if it is stressed, as some proposals in the domain of speech rhythm assume. Also, we do not know if cross-linguistic differences in how heterosyllabic clusters are articulatorily coordinated to vocalic nuclei reflect or are reflected by cross-linguistic differences in the languages’ rhythmical properties.

This project thus tries to reconcile the gestural and the rhythmical perspective into a unified research framework devoted to uncovering how inter-segmental coordination influences, and is influenced by, the rhythmical properties of supra-segmental entities.

To that aim, we develop a series of cross-linguistic experiments on Italian and Standard Austrian German to clarify some critical aspects of speech organization in the two languages and to establish a link between language-specific phonotactic constraints and the temporal and spatial properties of segments’ production.

The experiments, based on a reading task, include acoustic analyses for the identification of the temporal patterns and articulatory (ultrasound tongue imaging, UTI) analyses for the investigation of gestural coordination.

In addition, it is the purposes of the project to set the stage for an analysis of how the speech of cochlear implanted speakers differs from normal speech with respect to gestural coordination and rhythmic patterns. Spontaneous conversations will be recorded of both Italian and Standard Austrian German speakers. The target of the acoustic analyses will be the identification of the areas of most prominent difficulty concerning both the coarticulatory and the temporal aspects of spontaneous speech produced by CI-speakers.

### Basic Description:

HASSIP is a Research Training Network funded by the European Commission within the Improving the Human Potential program. The aim of the HASSIP network is to develop research activities and systematic interactions in mathematical analysis and statistics that are directly connected to signal and image processing. Although the Acoustics Research Institute was not initially a partner of this network, P. Balazs became a fellow of this network through cooperation with the group NuHAG.

### Partners:

• NuHAG, Faculty of Mathematics, University of Vienna
• Groupe de Traitement du Signal, Laboratoire d'Analyse Topologie et Probabilités, LATP/ CMI, Université de Provence, Marseille
• Modélisation, Synthèse et Contrôle des Signaux Sonores et Musicaux des LMA / CRNS Marseille
• Unité de physique théorique et de physique mathématique – FYMA

### Subprojects:

• Basic Properties of Bessel and Frame Multipliers: For Bessel sequences, the investigation of operators M = ∑ mk < f , ψk > is very natural and useful. The above M are Bessel multipliers. The goal of this project is to set the mathematical basis for this kind of operator.
• Best Approximation of Matrices by Frame Multipliers: Finding the best approximation by multipliers of matrices that represent time-variant systems gives a way to find efficient algorithms to implement such operators.

### Publications:

• P. Balazs, "Hilbert-Schmidt Operators and Frames - Classification, Approximation by Multipliers and Algorithms" , International Journal of Wavelets, Multiresolution and Information Processing, Vol. 6, No. 2, pp. 315 - 330, March 2008, preprint, Codes and Pictures: here
• P. Balazs, "Basic Definition and Properties of Bessel Multipliers", Journal of Mathematical Analysis and Applications, 325, 1: 571--585. (2007) doi:10.1016/j.jmaa.2006.02.012, preprint

### Project-completion:

This project ended on 01.01.2009. Its completion allowed the sucessfull application for a 'High Potential'-Project of the WWTF, see MULAC.

### Objective:

Head-related transfer functions (HRTF) describe the sound transmission from the free field to a place in the ear canal in terms of linear time-invariant systems. Due to the physiological differences of the listeners' outer ears, the measurement of each subject's individual HRTFs is crucial for sound localization in virtual environments (virtual reality).

Measurement of an HRTF can be considered a system identification of the weakly non-linear electro-acoustic chain from the sound source room's HRTF microphone. An optimized formulation of the system identification with exponential sweeps, called the "multiple exponential sweep method" (MESM), was used for the measurement of transfer functions. For this measurement of transfer functions, either the measurement duration or the signal-to-noise ratio could be optimized.

Initial heuristic experiments have shown that using Gabor multipliers to extract the relevant sweeps in the MESM post-processing procedure improves the signal-to-noise ratio of the measured data even further. The objective of this project is to study, in detail, how frame multipliers can optimally be used during this post-processing procedure. In particular, wavelet frames, which best fit the structure of an exponential sweep, will be studied.

### Method:

Systematic numeric experiments will be conducted with simulated slowly time-variant, weakly non-linear systems. As the parameters of the involved signals are precisely known and controlled, an optimal symbol will automatically be created. Finally, the efficiency of the new method will be tested on a "real world" system, which was developed and installed in the semi-anechoic room of the Institute. It uses in-ear microphones, a subject turntable, 22 loudspeakers on a vertical arc, and a head tracker.

### Application:

The new method will be used for improved HRTF measurement.