Adapting and Controlling DNN-based Speech Synthesis using Input Codes

Speech samples to support ICASSP 2017 paper "Adapting and Controlling DNN-based Speech Synthesis using Input Codes". If you have any question you can send me an email to: luonghieuthi (at) gmail.com (Hieu-Thi Luong)

7 models were used to generate various speech samples using different strategies:

Speaker Code (S) Gender Code (G) Age Code (A)
Models Type Size Type Size Type Size
ONE-S One-hot 112 N/A N/A N/A N/A
ONE-SGA' One-hot 112 One-hot 2 One-hot 7
ONE-SGA One-hot 112 Numeric 1 Numeric 1
RND112-SGA Random 112 Numeric 1 Numeric 1
RND008-SGA Random 8 Numeric 1 Numeric 1
DCC112-SGA DCC 112 Numeric 1 Numeric 1
DCC008-SGA DCC 8 Numeric 1 Numeric 1

1. Multi-Speaker

Samples for speakers in training set. Natural is recording speech. 'a' indicates an average value for that feature while 'c' indicated correct value.

1 2 3 4
Natural ► Play ► Play ► Play ► Play
ONE-a ► Play ► Play ► Play ► Play
ONE-c ► Play ► Play ► Play ► Play
ONE-ccc' ► Play ► Play ► Play ► Play
ONE-ccc ► Play ► Play ► Play ► Play
RND112-ccc ► Play ► Play ► Play ► Play
RND008-ccc ► Play ► Play ► Play ► Play
DCC112-ccc ► Play ► Play ► Play ► Play
DCC008-ccc ► Play ► Play ► Play ► Play

2. Adaptation

Samples for speakers not included in training set. 'e' indicated the estimated value, found by using back-propagation algorithm, of that feature.

1 2 3 4
Natural ► Play ► Play ► Play ► Play
ONE-a ► Play ► Play ► Play ► Play
ONE-e ► Play ► Play ► Play ► Play
ONE-ecc' ► Play ► Play ► Play ► Play
ONE-ecc ► Play ► Play ► Play ► Play
RND112-ecc ► Play ► Play ► Play ► Play
RND008-ecc ► Play ► Play ► Play ► Play
DCC112-ecc ► Play ► Play ► Play ► Play
DCC008-ecc ► Play ► Play ► Play ► Play


3. Manipulation

3.1. Gender

Samples of speakers included in training set with gender code was switched from Male to Female and vice versa. In case of model ONE-SGA' extreme value cannot be used

Male Female
Natural ► Play Natural ► Play
ONE-c ► Play ONE-c ► Play
ONE-cFc' ► Play ONE-cMc' ► Play
ONE-cFc ► Play ONE-cMc ► Play
RND112-cFc ► Play RND112-cMc ► Play
RND008-cFc ► Play RND008-cMc ► Play
DCC112-cFc ► Play DCC112-cMc ► Play
DCC008-cFc ► Play DCC008-cMc ► Play

3.2. Age

Samples of speakers in training set. A male speaker with age in range of 41-50 was chosen. In model ONE-SGA' values 15 and 75 were used as they are 1-of-k vector and cannot be assigned an extreme value.

Male (41-50)
Natural ► Play
ONE-c ► Play
ONE-cc15' ► Play ONE-cc75' ► Play
ONE-cc05 ► Play ONE-cc85 ► Play
RND112-cc05 ► Play RND112-cc85 ► Play
RND008-cc05 ► Play RND008-cc85 ► Play
DCC112-cc05 ► Play DCC112-cc85 ► Play
DCC008-cc05 ► Play DCC008-cc85 ► Play

3.3. Interpolation

The rest of samples were generated using DCC008-SGA model. In this section value of Speaker Code, Age Code and Gender Code are interpolated from one value to another in a single utterance

3.3.a. Speaker interpolation

The Speaker Code was interpolated between 2 speakers in the training set that have the same gender and age.

21-30 years old Male
Sample 1
Sample 2

61-70 years old Female
Sample 1
Sample 2

3.3.b. Age and Gender interpolation

The Gender Code or Age Code was interpolated from one extreme value to another, while the others 2 codes was keep with the correct value.

Sample 1
DCC008-ccc
Gender (-2->3)
Age (-50->200)

Sample 2
DCC008-ccc
Gender (-2->3)
Age (-50->200)