PRE2019 3 Group15

Group Members

Name	Study	Student ID
Mats Erdkamp	Industrial Design	1342665
Sjoerd Leemrijse	Psychology & Technology	1009082
Daan Versteeg	Electrical Engineering	1325213
Yvonne Vullers	Electrical Engineering	1304577
Teun Wittenbols	Industrial Design	1300148

Problem Statement and Objectives

DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread and the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.

How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.

Users

The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the References section for the full articles. This section is written based on (Gates, Subramanian & Gutwin, 2006), (Gates & Subramanian, 2006) and (Berkers & Michael, 2017).

Primary users

• Dance industry: this is the overarching organization that will possess most of the robots.
• Organizer of a music event: this is the user that will rent or buy the robot to play at their event.
• Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.

Primary user needs

• To purchase a DJ-robot is a smart, lucrative investment.
• The system's user interface is easy to understand, no experts needed.
• The system is easy to transport.
• The system is autonomous, no human in the loop.
• The DJ-robot is at least as valued as a human substitute.
• The DJ-robot system provides something extraordinary and special.
• The system should be able to use visual algorithms to detect the audience response.
• The system should be able to detect key dancers.
• The system should be able to filter out other sounds than music, like conversation, laughter, yelling and cheering.
• The DJ-robot should fit between human artists.
• The system must be better than a human DJ in gathering information from the audience appreciation.
• The system should facilitate mutual visibility between audiences and DJ's, but not at the expense of privacy.

Secondary users

• Attenders of a music event: these people enjoy the music and lighting show that the robot makes.
• Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.

Secondary user needs

• The DJ-robot selects popular tracks that are valued by the audience.
• The DJ-robot selects appropriate tracks regarding genre.
• The DJ-robot does not fall silent in between tracks.
• The music set played is structured and progressive.
• The DJ-robot is able to handle requests from the audience.
• The DJ-robot takes the audience reaction into account in track selection.
• The transition between tracks is smooth.
• DJ's want to take the dancers on a cohesive and dynamical music journey.
• DJ's want to keep the audience engaged.
• DJ's want people to join the dance floor.
• DJ's want their musical presentation to reflect the audience energy level.
• Track selection should fit the audience background.
• There is a balance between playing familiar tracks and providing rare, new music.
• DJ's want to play similar tracks to what people in the audience are into.
• The audience members want to have fun.
• The audience members want to get lost in the music.
• The audience members want to forget their responsibilities.
• The audience members want to express themselves.
• The audience members want to hear their favourite music.
• The audience members want to hear new music.
• The audience members don't want a predictable set of music.
• The audience members want to see what is happening at the DJ booth.
• The audience members are interested in having information about the music being played.
• The audience members desire control over the music being played, to a certain extent.
• The audience members want to experience high levels of collective effervesce.
• The audience members like to feel as though they are part of a group.
• The audience members like to unify their attention in a shared activity.

Approach, Milestones, and Deliverables

Approach

The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.

Then, the USE and technical aspects of the problem will be researched. The technical aspect-research will be implemented in a design for the robot. Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.

Milestones

In order to complete the project and meet the objective, milestones have been determined. These milestones include:

• A clear problem and goal have been determined
• The literature research is finished
• The research on how to create design and prototype is finished
• A design is created
• A working prototype is constructed
• The wiki is finished and contains all information about the project

Deliverables

The deliverables for this project are:

• A product design (?)
• A working prototype
• The wiki-page
• The final presentation in week 8

Who's Doing What?

Personal Goals

The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.

Name	Personal Goal
Mats Erdkamp	Play a role in the development of the artificial intelligence systems.
Sjoerd Leemrijse	Gain knowledge in recommender systems and pattern recognition algorithms in music.
Daan Versteeg
Yvonne Vullers	Play a role in creating the prototype/artificial intelligence
Teun Wittenbols

Weekly Log

Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be guideline for the coming weeks.

Week 1 Goal: Do literature research, define problem, make a plan

	Group	Mats Erdkamp	Sjoerd Leemrijse	Daan Versteeg	Yvonne Vullers	Teun Wittenbols
Monday	We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session. 30 minutes + 30 minutes	Work on SotA and evaluate design options ...	Work on defining "users" and "user requirements" ...	Work on relevant literature research. ...	Develop problem statement, search and analyze relevant research. ...	Started doing literature research and summarized Pasick (2015) & Johnson 2 hours
Thursday	We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project. 1.5 hours					Did some more research on audience interaction and summerized it. (Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse) I, also updated the wiki, made the planning more clear and divided the references of the SotA. 1.5 hours + 45 minutes

Week 2 Goal: Determine USE aspects, start research into design and prototype
Week 3 Goal: Continue research, start on prototype
Week 4 Goal: Finish first design, continue on prototype
Week 5 Goal: Finalize design
Week 6 Goal: Finish prototype, do testing
Week 7 Goal: Finalize prototype
Week 8 Goal: Finish wiki, presentation

State of the Art

Summary of Related Research

Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)

This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)

This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)

Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)

This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)

This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. "... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound." (Hödl, Fitzpatrick, Kayali & Holland, 2017)

This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)

This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)

This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)

In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)

In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)

This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)

This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)

This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)

In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)

This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)

This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)

References

Choi, K., Cho, K. “Deep Unsupervised Drum Transcription”, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.

Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). A tutorial on deep learning for music information retrieval. arXiv preprint arXiv:1709.04396.

De León, P. J. P., & Inesta, J. M. (2007). Pattern recognition approach for music style identification using shallow statistical descriptors. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.

Humphrey, E.J., Durand, S., McFee, B. “OpenMIC-2018: An open dataset for multiple instrument recognition”, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.

Hamel, P., & Eck, D. (2010, August). Learning features from music audio with deep belief networks. In ISMIR (Vol. 10, pp. 339-344).

Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). Design Implications for TechnologyMediated Audience Participation in Live Music. In: Proceedings of the 14th Sound and Music Computing Conference, July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.

Hoffman, G., & Weinberg, G. (2010). Interactive Jamming with Shimon: A Social Robotic Musician. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.

Huron, D. (2002). Music information processing using the Humdrum toolkit: Concepts, examples, and lessons. Computer Music Journal, 26(2), 11-26.

Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). Leveraging multi-dimensional user models for personalized next-track music recommendation. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).

Johnson, D. (n.a.) Robot DJ Used By Nightclub Replaces Resident DJs. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/

Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). Application of Bayesian probability network to music scene analysis. Computational auditory scene analysis, 1(998), 1-15.

Pasick, A. (21 December 2015) The magic that makes Spotify's Discover Weekly playlists so damn good. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/

Pérez-Marcos, J., & Batista, V. L. (2017, June). Recommender system based on collaborative filtering for spotify’s users.In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.

Shmulevich, I., & Povel, D. J. (1998, December). Rhythm complexity measures for music pattern recognition. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.

Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). Music Main Melody Extraction by An Interval Pattern Recognition Algorithm. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.

Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). A biped robot that keeps steps in time with musical beats while listening to music with its own ears. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.

Zhang, L., Wu, Y., & Barthet, M. (ter perse). A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case. NIME. Geraadpleegd van https://core.ac.uk/reader/77040676