TIMuL is an I+D+i project of The Spanish Ministry of Economy and Competitiveness

TIMuL is a research project of the University of Alicante and the University Pomeu Fabra of Barcelona, funded by the Ministry of economy and competitiveness, oriented to the study and development of algorithms and systems that can help to improve the process of learning and performance of music, from a pedagogical and scientific perspective. This project aims to investigate and explore the relevant aspects to develop methods and tools for music education, taking into account factors such as expressiveness, the human-computer interaction, control of movements, etc.

TIMuL es un proyecto de investigació coordinado entre la Universidad de Alicante y la Universitat Pompeu Fabra de Barcelona financiado por el Ministerio de Economía y Competitividad orientado al estudio y desarrollo de algoritmos y sistemas que puedan ayudar a mejorar el proceso de aprendizaje e interpretación de la música, desde una perspectiva pedagógica y científica. Este proyecto tiene como objetivo investigar y explorar los aspectos relevantes para elaborar métodos y herramientas para la educación musical, teniendo en cuenta factores clave como la expresividad, la interacción hombre-máquina, el control de los gestos, etc.

Worklines

The main objectives of the project are:
  • to acquire and evaluate audio, gesture, physiological, symbolic, image, and video information for pattern recognition and machine learning techniques.
  • to integrate the descriptions coming from the different information sources.
  • to use adaptive methodologies to take advantage of the interaction with users or their environment.
  • Implement novel multi-modal interactive music learning prototypes.
  • Five interactive multimodal prototypes will be developed for pedagogical music learning and training:

Music transcription to get scores from audio.
Interactive music analysis to get chordal and tonal information from scores.
A performances database to help musicians to learn how to play musical instruments.
Optical music recognition helps musicians to write music notation by hand.

Publications

Gallego, A.J., Calvo-Zaragoza, J.; Valero-Mas, J.J.; Rico-Juan, J.R. (2018)
"Clustering-based k-nearest neighbor classification for large-scale data with neural codes representation". Pattern Recognition, vol. 74, pp. 531-543.
Iñesta, J.M.; Conklin, D.; Ramirez, R.; M. Fiore, T.M. (2018)
"Machine Learning and Music Generation". Routledge, Taylor & Francis, ISBN: 978-0-8153-7720-7.
Pertusa, A; Gallego, A.J; Bernabeu, M. (2018)
"MirBot: A collaborative object recognition system for smartphones using convolutional neural networks". Neurocomputing, vol. In press.
Calvo-Zaragoza, J.; Vigliensoni, G.; Fujinaga, I. (2017)
"A machine learning framework for the categorization of elements in images of musical documents". Proceedings of the Third International Conference on Technologies for Music Notation and Representation.
Valero-Mas, J. J.; Calvo-Zaragoza, J.; Rico-Juan, J. R.; Iñesta, (2017)
"A study of Prototype Selection algorithms for Nearest Neighbour in class-imbalanced problems". Proceedings of the 8th Iberian Conference on Pattern Recognition and Image Analysis (IbPRIA), Faro, Portugal.
Rizo, D.; Calvo-Zaragoza, J.; Iñesta, J.M.; Fujinaga, I. (2017)
"About agnostic representation of musical documents for Optical Music Recognition". Music Encoding Conference, Tours.
Calvo-Zaragoza, J.; Oncina, J. (2017)
"An efficient approach for Interactive Sequential Pattern Recognition". Pattern Recognition, vol. 64, pp. 295-304.
Valero-Mas, J.J.; Calvo-Zaragoza, J.; Rico-Juan, J.R.; Iñesta, J.M.
"An Experimental Study on Rank Methods for Prototype Selection". Soft Computing, vol. 21, pp. 5703--5715.
Valero-Mas, J. J.; Benetos, E.; Iñesta, J. M. (2017)
"Assessing the Relevance of Onset Information for Note Tracking in Piano Music Transcription". Proceedings of the AES International Conference on Semantic Audio.
Calvo-Zaragoza, J.; Valero-Mas, J.J.; Pertusa, A (2017)
"End-To-End Optical Music Recognition using Neural Networks". Proc. of Intenational Society for Music Information Retrieval Conference (ISMIR), Suzhou, China.
Valero-Mas, J. J.; Iñesta, J. M. (2017)
"Interactive User Correction of Automatically Detected Onsets: Approach and Evaluation". EURASIP Journal on Audio, Speech, and Music Processing.
Hontanilla, M.; Pérez-Sancho, C.; Iñesta, J.M. (2017)
"Music style recognition with language models -- beyond statistical results". Proceedings of MML 2017, pp. 31--36, Barcelona.
Sober-Mira, J.; Calvo-Zaragoza, J.; Rizo, D.; Iñesta, J.M. (2017)
"Pen-based music document transcription". Proceedings of GREC 2017, ISBN: 978-1-5386-3586-5, pp. 21--22, Kyoto (Japan).
Calvo-Zaragoza, J.; Vigliensoni, G.; Fujinaga, I. (2017)
"Pixel-wise Binarization of Musical Documents with Convolutional Neural Networks". Proceedings of the 15th IAPR International Conference on Machine Vision Applications.
Calvo-Zaragoza, J.; Gallego, A.J.; Pertusa, A. (2017)
"Recognition of Handwritten Music Symbols with Convolutional Neural Codes". 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), pp. 691--696, Kyoto, Japan.
Calvo-Zaragoza, J.; Oncina, J. (2017)
"Recognition of Pen-based Music Notation with Finite-State Machines". Expert Systems With Applications, vol. 72, pp. 395-406.
Bernabeu, J.F. (2017)
"Similarity Learning and Stochastic Language Models for Tree-Represented Music". PhD Thesis. University of Alicante.
Calvo-Zaragoza, J.: Pertusa, A.; Oncina, J. (2017)
"Staff-line detection and removal using a convolutional neural network". Machine Vision and Applications, pp. 1-10.
Gallego, A.J.; Calvo-Zaragoza, J. (2017)
Staff-line removal with Selectional Auto-Encoders". Expert Systems With Applications, vol. 89, pp. 138 - 148.
Jose J. Valero-Mas (2017)
"Towards Interactive Multimodal Music Transcription". PhD Thesis. University of Alicante.
Rizo, D.; Marsden, A. (2016)
"A standard format proposal for hierarchical analyses and representations". Proceedings of the 3rd International Workshop on Digital Libraries for Musicology, ISBN: 978-1-4503-4751-8, pp. 25--32, New York, USA.
Plácido R. Illescas (2016)
"Análisis tonal asistido por ordenador". PhD Thesis. University of Alicante.
Valero-Mas, J.J.; Benetos, E.; Iñesta, J.M. (2016)
"Classification-based Note Tracking for Automatic Music Transcription". Proceedings of the 9th Machine Learning and Music Workshop (MML2016), pp. 61--65, Riva del Garda, Italy.
Calvo-Zaragoza, J.; Oncina, J.; De la Higuera, C. (2016)
"Computing the Expected Edit Distance from a String to a PFA". 21st International Conference Implementation and Application of Automata, pp. 39-50.
Ponce de León, P.J., Iñesta, J.M., Calvo-Zaragoza, J., Rizo, D. (2016)
"Data-based melody generation through multi-objective evolutionary computation". Journal of Mathematics and Music, vol. 10, pp. 173-192.
Calvo-Zaragoza, J.; Toselli, A. H.; Vidal, E. (2016)
"Early Handwritten Music Recognition with Hidden Markov Models". 15th International Conference on Frontiers in Handwriting Recognition.
Rizo, D.; Calvo-Zaragoza, J.; Iñesta, J.M.; Illescas, P.R. (2016)
"Hidden Markov Models for Functional Analysis". Music and Machine Learning Workshop, Riva del Garda.
Bernabeu, M.; Pertusa, A.; Gallego, A.J. (2016)
"Image spatial verification using Segment Intersection of Interest Points" Proc. of the 24 Int. Conf. in Central Europe on Computer Graphics, Visualization and Computer Vision (WSCG), ISBN: 2464-4614.
Rizo, D; Illescas, P.R; Iñesta, J.M. (2016)
"Interactive melodic analysis". Computational Music Analysis, ISBN: 978-3-319-25931-4, pp. 191-219.
Bellet, A; Bernabeu, J.F.; Habrard, A.; Sebban, M (2016)
"Learning discriminative tree edit similarities for linear classification - Application to melody recognition". Neurocomputing, vol. 214, pp. 155-161.
Calvo-Zaragoza, J.; Micó, L.; Oncina, J. (2016)
"Music staff removal with supervised pixel classification". International Journal on Document Analysis and Recognition, vol. 19, pp. 211-219.
Valero-Mas, J.J.; Calvo-Zaragoza, J.; Rico-Juan, J.R. (2016)
"On the suitability of Prototype Selection methods for kNN classification with distributed data". Neurocomputing, vol. 203, pp. 150-160.
Calvo-Zaragoza, J.; Valero-Mas, J.J.; Rico-Juan, J.R. (2016)
"Prototype Generation on Structural Data using Dissimilarity Space Representation". Neural Computing and Applications.
Calvo-Zaragoza, J.; Valero-Mas, J.J.; Rico-Juan, J.R. (2016)
"Selecting promising classes from generated data for an efficient multi-class NN classification". Soft Computing.
Bosch, V.; Calvo-Zaragoza, J.; Toselli, A. H.; Vidal, E. (2016)
"Sheet Music Statistical Layout Analysis". 15th International Conference on Frontiers in Handwriting Recognition.
Calvo-Zaragoza, J.; Rizo, D.; Iñesta, J.M. (2016)
"Two (note) heads are better than one: pen-based multimodal interaction with music scores". 17th International Society for Music Information Retrieval Conference, ISBN: 978-0-692-75506-8, pp. 509-514, New York City.
Rafael Ramirez, Sergio Giraldo and Zacharias Vamvakousis (2015)
"A Brain-Computer Expressive Music Performance Interface". CMMR2015: International Workshop on BCMI.
Rizo, D.; Iñesta, J.M (2015)
"A grammar for Plaine and Easie Code". Proceedings of the Music Encoding Initiative Conferences 2013 and 2014, pp. 54--64.
Valero-Mas, J.J.; Salamon, J.; Gómez, E. (2015)
"Analyzing the influence of pitch quantization and note segmentation on singing voice alignment in the context of audio-based Query-by-Humming". Proceedings of the 12th Sound and Music Computing Conference (SMC), pp. 371--378, Maynooth, Ireland.
Vamvakousis, Z., & Ramirez R. (2015)
"A PERCUSSIVE MUSICAL INTERFACE FOR A QUADRIPLEGIC PATIENT". International Symposium on Performance Science 2015. 60-61.
Bantula, H., Giraldo, S. & Ramirez R. (2015)
"A Rule Based System to Transcribe Guitar Melodies". CMMR2015: International Symposium on Computer Music Multidisciplinary Research.
Calvo-Zaragoza, J.; Barbancho, I; Tardón, L. J. ; Barbancho A. M. (2015)
"Avoiding staff removal stage in optical music recognition: application to scores written in white mensural notation". Pattern Analysis and Applications, vol. 18, pp. 933-943.
Calvo-Zaragoza, J.; Oncina, J. (2015)
"Clustering of Strokes from Pen-based Music Notation: An Experimental Study". 7th Iberian Conference on Pattern Recognition and Image Analysis (IbPRIA), ISBN: 978-3-319-19389-2, pp. 633-640, Santiago de Compostela, Spain.
Giraldo, S. & Ramirez, R. (2015)
"Computational Generation and Synthesis of Jazz Guitar Ornaments using Machine Learning Modeling. International Workshop on Machine Learning and Music (MML 2015).
Giraldo, S. & Ramirez, R. (2015)
"Computational Modeling and Synthesis of Timing, Dynamics and Ornamentation in Jazz Guitar Music". International Symposium on Computer Music Multidisciplinary Research (CMMR 2015).
Giraldo, S. & Ramirez, R. (2015)
"Computational Modeling of Ornamentation in Jazz Guitar Music". International Symposium in Performance Science (ISPS 2015).
Rafael Ramirez and Karen Lacey (2015)
"EEG-BASED CLASSIFICATION OF MUSIC-INDUCED EMOTIONS IN DANCERS: A MACHINE LEARNING APPROACH". 8th International Workshop on Machine Learning and Music. 55-56.
Vamvakousis, Z., & Ramirez R. (2015)
"EEG Signal classification in a Brain-Computer Music Interface". 8th International Workshop on Machine Learning and Music. 28-30.
J.M. Iñesta, P.J. Ponce de León, J. Calvo-Zaragoza, and D. Rizo (2015)
"Genre-based melody generation through multi-objective genetic algorithms". Proceedings of the 8th Machine Learning and Music workshop (MML 2015), Vancouver (Canada).
Rico-Juan, J. R; Calvo-Zaragoza, Jorge (2015)
"Improving classification using a Confidence Matrix based on weak classifiers applied to OCR". Neurocomputing, vol. 151, pp. 1354-1361.
Calvo-Zaragoza, J; Valero-Mas, J. J.; Rico-Juan, J. R (2015)
"Improving kNN multi-label classification in Prototype Selection scenarios using class proposals". Pattern Recognition, vol. 48, pp. 1608-1622.
Valero-Mas, J.J.; Iñesta, J.M.
"Interactive onset detection in audio recordings". Late Breaking/Demo extended abstract, 16th International Society for Music Information Retrieval Conference (ISMIR), Málaga, Spain.
Vamvakousis, Z., & Ramirez R. (2015)
"Is an auditory P300-based Brain-Computer Musical Interface feasible?". CMMR2015: International Workshop on BCMI.
Rafael Ramirez, Manel Palencia, Sergio Giraldo, Zacharias Vamvakousis (2015)
"Musical Neurofeedback for Treating Depression in Elderly People". Frontiers in Neuroscience.
Giraldo, S. & Ramirez, R. (2015)
"Performance to Score Sequence Matching for Automatic Ornament Detection in Jazz Music". ICNMC 2015: International Conference on New Music Concepts.
Calvo-Zaragoza, J.; Valero-Mas, J.J.; Rico-Juan, J.R. (2015)
"Prototype Generation on Structural Data using Dissimilarity Space Representation: A Case of Study". 7th Iberian Conference on Pattern Recognition and Image Analysis (IbPRIA), ISBN: 978-3-319-19389-2, pp. 72-82, Santiago de Compostela, Spain.
Rafael Ramirez and Sergio Giraldo (2015)
"TECHNOLOGY-ENHANCED EXPRESSIVE MUSIC PERFORMANCE LEARNING: A MACHINE LEARNING-BASED TUTORING TOOL". 8th International Workshop on Machine Learning and Music. 143-144.
Micó, L.; Sanches, J.; Cardoso, J. (2015)
"The vitality of pattern recognition and image analysis". Neurocomputing, vol. 150, pp. 124-125.
Bantulá, H., Giraldo, S. & Ramirez, R. (2014)
"A Rule Based System to Transcribe Guitar Melodies". International Workshop on Machine Learning and Music MML 2014.
Micó, L.; Oncina, J. (2014)
"Dynamic Insertions in TLAESA fast NN Search Algorithm". Proceedings of the 22nd International Conference on Pattern Recognition, ICPR, ISBN: 978-1-4799-5208-3, Stockholm, Sweden.
Illescas, P.R.; Rizo, D.; Iñesta, J.M. (2014)
"Melodic analysis of polyphonic music using an interactive pattern recognition tool". Proc. of 7th Machine Learning and Music (MML2014), Barcelona.
Valero-Mas, J.J.; Iñesta, J.M.; Pérez-Sancho, C. (2014)
"Onset detection with the user in the learning loop". Proceedings of the 7th International Workshop on Music and Machine Learning (MML2014), Barcelona, Spain.
Giraldo, S. & Ramirez, R. (2014)
"Optimizing Melodic Extraction Algorithm For Jazz Guitar Recordings Using Genetic Algorithms". International Computer Music Conference/Sound and Music Computing Conference.
Vamvakousis, Z., & Ramirez R. (2014)
"P300 Harmonies: A Brain-Computer Musical Interface". International Computer Music Conference/Sound and Music Computing Conference.
Calvo-Zaragoza, J.; Oncina, J. (2014)
"Recognition of Pen-based Music Notation with Probabilistic Machines". Proceedings of the 7th International Workshop on Machine Learning and Music , Barcelona, Spain.
Marchini, M., Ramirez R., Papiotis P., & Maestre E. (2014)
"The Sense of Ensemble: a Machine Learning Approach to Expressive Performance Modelling in String Quartets". Journal of New Music Research. 43, 303-317.


People

gRFIA-UA group

+info
Equipo Investigador:

José Manuel Iñesta Quereda (Contact)
Jose Oncina Carratalá
María Luisa Micó Andrés
Juan Ramón Rico Juan
Jorge Calera Rubio
Carlos Pérez Sancho
Pedro Ponce de León Amador
Antonio Pertusa Ibañez

Equipo de trabajo:

Javier Gallego-Sánchez
David Rizo Valero
Placido Illescas Casanova
María Hontanilla Alfonso
Jose Francisco Bernabeu Briones
Maria Luisa Bernabeu Lledó
Aureo Serrano Díaz-Carrasco
José Javier Valero Mas
Jorge Calvo Zaragoza
Javier Sober Mira

MTG-UPF group

+info
Equipo Investigador:

Rafael Ramírez-Meléndez (Contact)
Melissa Mercadal Coll
Dolors Sala Batlle
Manel Palencia-Lefler
Equipo de Trabajo:
Esteban Maestre
Alfonso Antonio Pérez Carrillo
Maria Cristina Marinescu
Oscar Mayor Soto
Panagiotis Papiotis
Sergio Iván Giraldo Méndez
Zacharias Vamvakousis
Alvaro Sarasúa Berodia

Downloads

Software and Databases developed in the project.


A prototype for genetic melody composition
  • Description: a Java software provided with a graphical interface for generating melodies using genetic algorithms based on trees. The system needs monophonic (single track) MIDI files for training. If you need some data, those used for the publication below can be requested to the authors.
  • Version: under development
  • Date: 22/02/2016
  • References: P.J. Ponce de León, J.M. Iñesta, J. Calvo-Zaragoza, and D. Rizo (2016). "Data-based melody generation through multi-objective evolutionary computation" Journal of Music and Mathematics (to be published)
  • Download (Developed in Java Version: 1.8 u77)
  • Requirements:
    - Java SE Runtime Environment is needed. You can download here.
  • User Interface.

Interactive melodic analysis
  • Description: Interactive prototype developed in JavaFX 8. The application allows the melodic analysis and helps in the task of key and chord analysis.
  • Version: 1.0
  • Date: 22/12/2015
  • More Info.
  • Download.
Interactive multimodal music transcriptor
  • Description: Interactive prototype developed in JavaFX 8. User interactions can be used by the system to improve its performance in an adaptive way. Different sources of information, like onsets, beat, and meter, are used to detect notes in a musical audio excerpt. The system is focused on monotimbral polyphonic transcription.
  • Version: 2.0
  • Date: 9/11/2017
  • More Info.
  • Download.
End-to-end OMR
  • Description: Source code, trained models and data collection from the use of Convolutional Recurrent Neural Network and Connectionist Temporal Classification training for end-to-end Optical Music Recognition [ISMIR 2017]
  • Version: 1.0
  • Date: 14/07/2017
  • Download.
Capitan dataset
  • Data collection from Early music manuscripts [ICDAR 2017]
  • Date: 2-10-2017
  • Download.
Isolated handwritten music symbols
  • Description: Four corpora of isolated handwritten music symbols [ICDAR 2017]
  • Date: 28-7-2017
  • Download.
Bimodal music symbols from Early notation
  • Description: corpus collected by an electronic pen while tracing isolated music symbols from Early manuscripts. The dataset contains information of both the sequence followed by the pen and the patch of the source under the tracing itself.
  • Size: 10230 symbols
  • Date: 2016
  • Download.
HOMUS (Handwritten Online Music Symbols)
  • Description: Corpus of handwritten music symbols drawn by an electronic pen. It contains data from 100 different musicians spread over 32 classes.
  • Size: 15200 symbols
  • Date: 2014
  • More info.
Aligned audio-symbolic flute corpus
  • Description: Labeled flute corpora for audio to score music transcription.
  • 28 minutes recordings with 2246 manually annotated notes.
  • Date: 2018
  • Download.