Follow

Follow is a program that displays a musical score and shows a cursor in the score that follows what you play. The program listens to a midi interface to read the notes that you play. When you have no midi interface you can use the build in keyboard on the screen. You can also use an audio-pitch recognizer that outputs a midi stream.

You can select a staff that you want to play and the program will play all other staves while following you on the selected staff. So you hear your self accompanied.

Apart from displaying a cursor in the score, the program also measures the timing of your performance and makes a timing plot at the end with some statistical information.

Follow can also be used in a concert setup, where multiple devices show different parts of the same score. A master device determines the tempo and all other (slave) devices follow. The master can either use the play back feature or actually follow a midi instrument played by the director (or solist). (see the synchronization feature of Follow)

The Timing Bar

The metronome symbol flashes red when the metronome is running. When you click this symbol a count-in starts counting down after which playback begins. When you click this symbol while the metronome is flashing it stops the metronome.
Performance Bar The bar displays your timing errors in the constant tempo modes (see follow mode). When the life bar mode is selected it displays a health precentage (starting with 100, see further life bar).
e: (Error Count) When one of the constant tempo modes is selected this field shows the number of faulty notes you played.
Play Starts a play back of all voices. The computer also plays the staff that you selected to follow, unless you select the mute own staff option.
While playing, the computer also shows its own timing errors in the performance bar, so you can review the statictics to see if your CPU is fast enough.
Microphone symbol. Its color is brown when the microphone is on. Dark-grey when the microphone is off. The symbol is clickable and toggles the microphone on/off. It is a shortcut for the use microphone checkbox in the microphone dialog.
Resizing Dragging (touch-hold-move) downwards in the timing bar area resizes the bar.

The on-screen Keyboard

Touch/Click area The on-screen keyboard (see menu to enable) has a sensitve area above the keys, stretching along the whole keyboard. The height of this area is the same as the green triangles you see at both ends of the keyboard.
When you click/touch-and-hold in this area it lights up green and you can resize or shift the keyboard. Dragging upwards enlarges the keys (makes them longer) and dragging downwards makes them shorter. Dragging sidewards shifts the keyboard to the left or right.
Green triangles A click/touch-and-hold on the right triangle makes the keys broader in repeated steps as long as you hold. The left triangle does the same but makes the keys smaller.
A normal click/touch in one of the green triangles shifts the keyboard precisely one octave in that direction.

The Menu

Score file Browse the local file system and select a score file. The file can either contain MusicXML or ABC notation. The selected file is opened and the score rendered on the screen.
Use Dropbox Change the score file button into the Dropbox Chooser button to browse files in your dropbox.
Microphone The Settings button opens the microphone dialog.
Output port The program starts without sound. This selection control can be used to activate the internal synthesizer or to select a midi output port. A midi output port is only available when the browser supports midi and the computer has a midi interface (connected to a midi instrument or sound module).
Midi echo When playing a midi instrument the notes from the instrument are echoed back on the midi ouput port. This may not be desirable when the instrument already produces sound from itself. In that case the midi echo can be switched off. Notes from the accompaniment are always sent to the output port.
Follow mode The aim this program is to display a cursor in the score that follows what you play on a midi instrument. There are several options that determine how your played notes are followed:
  • Adaptive tempo. In this mode the tempo is no contraint on your performance. The tempo just follows your playing speed. The program only checks if you correctly play the notes selected by the cursor. If so, it moves the cursor to the next position in the score.
  • Constant tempo with statistics. In this mode you have to play exactly on the beats of the metronome. The metronome starts automatically when you play a random first note and starts with a count-in. After the count-in finishes you have to play the notes selected by the cursor in the tempo of the metronome. Your timing accuracy is indicated in the performance bar at the top left of the window. When the average timing error of your last 10 notes exceeds 100 millisec. your performance is considered a failure and a dialog is presented with some statistics followed by a timing plot. When you reach the end of the score you get the same information but your performance is considered successful.
  • Constant tempo with life bar This mode is identical to the previous one, only the performance bar now indicates a health percentage. You start with a full bar (100 points) and with every bad timing (> 35 millisecs) you loose 5 points. However you gain 5 points with a good timing (< 35 millisecs).
  • Adaptive tempo (with timeout) This mode is identical to the first mode. There is no contraint on your tempo. However, the tempo you enter in the tempo box is now considered a minimum tempo at which play back will continue. You can play faster than this tempo, but not slower. When you play slower (or play the wrong notes or do not play at all) the program will continue to move the cursor and play the accompaniment at the minimum tempo. You can catch on at any time and start to play faster again.
  • Adaptive tempo (minimum tempo) This mode is identical to the first mode, but the tempo never drops below the initial value of the tempo. You can speed up without limit, but you can only slow down until the initial tempo is reached.
Check mode Notes you play are followed in the score. Which notes you have to play is determined by one of the three check modes:
  • Top notes of selected staff: Only the top most notes of the selected staff have to be played. When the staff has more voices or layers the top note of each of them must be played. In fact, 'top' only refers to chords.
  • All notes of selected staff: You have to play all notes (of all voices) in a staff.
  • All notes of grand staff: You have to play all notes (of all voices) in the selected grand staff. A grand staff is selected by clicking in the topmost staff of the grand staff. The staff below is then added to make a grand staff.
  • All notes of all staves: In this mode all staves are taken into account. You have to play all notes in all staves of the score.
Mute own staff This option mutes the notes of the selected staff during play back. Normally all notes in all staves are played.
Next note delay After you play the selected notes the cursor moves to the next time position in the score. When this option is selected (default) the cursor does not move immediately, but waits for half the time span between the current note and the next one (with the current tempo). This little delay feels more naturaly when you play in tempo. When you do not play in tempo it is better to switch this option off. Especially when you play slowly or irregularly.
Metronome When this option is enabled the metronome continues ticking after the count-in has finished. Otherwise the metronome stops after the count-in.
Play one note before start When the count-on is finished the computer starts playing the accompaniment one note before the cursor. So you do not start playing immeriately after the count-in, but the accompaniment does, one note before the selection, where you have to fall in.
Show Timing This button shows a timing plot with statistical information of your performance. It is only available if you play one of the constant tempo modes. In the adaptive tempo modes to timing data is stored. It is used to adapt the tempo.
Keyboard Shows a keyboard where you can play the required notes. It is useful if you have no midi keyboard available. The keys can be moved and scaled by dragging (horizontally and vertially) in a small area just above the keyboard. There are also two octave shift buttons, that shift the keys precisely one octave. When you press-and-hold on these buttons, the keyboard is resized (scaled) in horizontal direction.
Transparency This makes the keyboard transparent and it overlays the score. May be useful for small devices.
Show notes With this option enabled you see the notes you must play highlighted on the keyboard with a red background. (The notes selected by the cursor are shown as depressed on the keyboard).
Extract staff When the Single is selected the current staff will be extracted for exclusive display, while all other staves are hidden. The hidden staves still play when needed (in follow mode of the extracted staff or in general play back mode)
When Grand is selected the current staff becomes the top staff of a grand staff that is subsequently selected for exclusive display.
Two lines With this option enabled only two lines are shown: the line with the cursor and the next one. This eases score reading, as all other lines are not needed. Each line either shows a system or a staff, depending on the Extract staff option.
Save Preload Pops up the save dialog where you can save a preload file. This file contains the score and most of the settings you entered either with the menu or with url parameters.
Page width Uses the given width in cm to render the score. It should be larger than 5 and smaller than 50 cm. The original page formatting of the score is erased and an optimal layout is computed for the given page width. Also the (abbreviated) instrument names are erased from all but the first staff to save space.
If the real screen/window width is larger than the given number the score will seem enlarged. The page width can also be set to a larger value than in the original score, which results in longer lines with more bars on a line.
Entering a number smaller than 5 erases the setting and resets the score to the original page layout.
Help Shows this file.

Microphone dialog

Use Mic. Asks permission to use the microphone. When the microphone is working, a pitch detector converts the audio signal into notes, which are followed in the score. Only monophone melodies can be followed. To get access to the microphone the program uses the WebRTC API which is, at the time of writing, only available in Chrome and FireFox on the PC and Android platforms.
Gain The gain slider sets the volume level of the microphone. Decibel values of the currently received signal are shown by the organge volume bar (dB). The target sound should be stronger than the level indicated by the blue line in the volume bar. This level corresponds to the Min. Level slider (see below). The peaks of the current microphone signal are shown in the graph by a grey wave line.
Sensitivity The sensitivity slider sets the height of the dashed blue horizontal line in the graph. When the green wave line drops below the dashed blue line a note is recognized. Higher values of the sensitivity result in more (or earlier) recognition of pitches. Values above 5 are not practical, because even without sound notes will be detected (in the background noise). The optimal value is between 1 and 4.
The green wave line represents the Average Squared Mean Difference Function (ASMDF) of the microphone signal. This function approaches zero when the input signal is periodical and the x-coordinate equals the period. The x-coordinate of the recognized period is shown with a small red vertical line in the graph.
Min. Level The minimum level is a treshhold for the microphone signal. Only signal levels above this threshold are considered for pitch detection. The value should be between -40 dB and 0 dB. The current value of the minimum level is shown in the microphone level bar by a vertical blue line.
Cents bar The cents bar displays the deviation of the recognized pitch from the closest musical note. Notes are supposed to have the twelve-tone equal temperament. Your instrument should be well tuned for the pitch detector to produce notes consistent with the score.
Bass guitar Normally the lowest note that can be detected is an F#3. When the bass checkbox is marked, the lowest possible note becomes an F#2. However, this doubles the internal sample buffer and makes detection four times slower. Therefore it should only be used when detection of such low notes is really needed.

URL parameters

A link to follow.html can have several URL-parameters, which are explained in the table below. An working example of a link with some parameters is:

http://wim.vree.org/js/follow.html?d:75mnyv0xsmye3mq/gadma_2.xml&syn&tmp=120&keyb&ksh&play

It plays a MusicXML file from Dropbox on the internal synthesizer and shows the played notes in the score and on the keyboard.

file_name The file name has to be a URL poining to a score file with exension .xml or .abc, for instance:
http://some.domain.org/follow.html?path/to/score_file.xml
The file can also reside in your dropbox, where it has to be shared publicly. In this case you use the dropbox identifier prefixed with d:
For example, dropbox tells me that a shared file in my dropbox has this link:
https://www.dropbox.com/s/75mnyv0xsmye3mq/gadma_2.xml?dl=0
The relevant part of this link is 75mnyv0xsmye3mq/gadma_2.xml, which you use as follows:
http://wim.vree.org/js/follow.html?d:75mnyv0xsmye3mq/gadma_2.xml
Note the link is prefixed with d: Try it out by clicking the link.
The third possibility for the file name is to have a preload file (see save), for instance:
http://wim.vree.org/js/follow.html?d:0q8p1zvfjn9ijw2/pre_ave.js
You do not need any further URL parameters as all settings are included in the preload file.
stf= With the stf=staff_number parameter you specify which staff should be played by you. The notes in this staff are highlighted during your play. The first staff has number 1.
extr Sets the Extract staff option to Single. Can be used in combination with stf= to show only a single part from an multi part score.
extrg Sets the Extract staff option to Grand. Can be used in combination with stf= to show only a single grand staff from an multi part score. The Check mode remains unchanged. If you want all notes in the grand staff highlighted you have to add the URL option mod=3.
2ln Sets the Two lines option.
pw= The pw=xx parameter sets the page width to xx cm. (see Page width)
fmd= The fmd=follow_mode chooses one of the four follow modes. The value of follow_mode is 1, 2, 3, 4 or 5. (see menu)
mod= With the mod=check_mode parameter you can choose one of the three check modes. The value of check_mode is 1, 2, 3 or 4. (see menu)
tmp= The tmp=tempo sets the play back tempo.
nodel The Next note delay checkbox is unmarked. After (correctly) playing the highlighted notes the cursor jumps immediately to the next notes, without delay. Normally there is a dealy of half the (played) note length.
nomet The nomet parameter unmarks the metronome checkbox in the menu, so that the metronome stops when the count-in terminates. Normally the metronome continues ticking after the count-in.
keyb When the keyb parameter is present the keyboard will be shown.
ksh The ksh parameter marks the show notes checkbox, so that notes highlighed by the cursor are also highlighted on the keyboard.
opa= This parameter sets the opacity of the keyboard. opa should be in the range 1-9, where opa=1 is opaque and opa=9 is maximal transparent.
syn The syn parameter selects the internal synthesizer and loads the pre-rendered waveforms.
play The play parameter starts playback after the score has been loaded.
nomenu Hides the menu.
nobar Hides the tempo field and the performance bar.
nocur Hides the cursor.
line= Sets the position of the dotted line. line=1 is at the top and line=100 is at the bottom of the page.
ip When this parameter is present the device tries to connect to a Web Socket server at port 8091. The ip address of this server is taken from the host part of the URL. The required socket server and a web server at port 8090 are both started when you run the program abcSrv.js with nodejs.
When connecion is successful the menu button becomes green. Positioning of the cursor is now syncrhonized with a master device (see parameter ipm).
ipm When present this parameter makes the device the master of a group after connecting to a Web Socket server at port 8091. When connecion is successful the menu button becomes red. The master sends its cursor position (when it changes) to all slaves. The slaves synchronize their cursor movement to the master.
sctmp The program reads tempo values from the score (when present). This overrides a manually entered tempo value or any tempo changes induced by following the player. The tempo input box is greyed out (disabled).

Synchronizing multiple devices

Multiple devices can follow the same score in precise synchronization. Each device can focus (extract) a different part of the score. In such a group of devices there is one master who determines the tempo, either by just using the play back feature or by actually playing on a midi device. The other devices are slaves and just follow the master. To achieve synchronization Follow uses a special server program called AbcSrv.js.

AbcSrv.js is included in the distribution zip file. It is a javascript program that implements a simple http server and a special Web Socket server. The normal http server runs on port 8090 and the web socket server on port 8091.
To run abcSrv.js you need to have Node installed. Node is a javascript engine with unrestricted access to the computer's (networking) capabilities.
When you have Node installed you can start abcSrv.js in a terminal window (command line prompt) with:

# nodejs abcSrv.js
The command for Node (nodejs in the example) may be different on your platform (for instance, just node). When started abcSrv.js prints the following output in the terminal window:
websocket server listening on port: 8091, address: 192.168.xx.xx
http server listening in port: 8090, adress: 192.168.xx.xx
web server root directory: /path/to/where/abcSrv.js/is/located/

Now you can check that the server works by just typing 192.168.xx.xx:8090 in a browser on another device connected to the same (WIFI) network. It should display a listing of files, one of which is follow.html. Then you can open follow.html as slave or as master, using:

http://192.168.xx.xx:8090/follow.html?some_preload.js&ip   (for all slave devices)
http://192.168.xx.xx:8090/follow.html?some_preload.js&mstr (for the master device)

In stead of a preload file, as in the example above, you can also just use a score file, or even no file at all and open a score file later with the menu button.

When a device succesfully connects to abcSrv.js the menu button becomes green. For the master device the menu button becomes red. This gives some feedback to indicate that all is workiing well and that henceforth all play/pause/jump actions will be issued by the master of the group. The slave devices cannot initiate any actions, they only follow the master.

You should have all files of the distribution (including abcSrv.js) extracted to one directory, which becomes the server root directory. All preload files you make or score files you want to use should also be in this directory.

On a normal (i.e. not jailbroken) iPad, abcSrv.js has been tested using the app Touch Code Pro from iTunes. This app can load and run abcSrv.js, turning the iPad into a simple http server and websocket server. While Touch Code Pro is running you can still use the browser to also open Follow, so you do not loose a device.