Having two studios, one of which at a “remote” site can be a bit challenging to say the least. One of the challenges involves is to ensuring that audio played at the remote site does so reliably and consistently. This turns out to be a fairly significant problem.
Our playout software uses an audio collection called an “Audiowall”, which is reminiscent of an actual physical wall full of tape carts in a rack. Back when radio stations used cart players for jingles, this made a lot of sense and was the best way to organise them. This concept has been transplanted into the world of digital playout, and now our playout software (PSquared Myriad) provides a virtual ‘cart wall’, into which audio files are given a number (from 1 to 99,999), and placed in the audiowall browser in that location. You can scroll through ranges of carts and pick out which ones you want to play, drag them into the log, or just play them right from the audiowall.
This is a fairly well known concept, and a lot of playout software uses it. However, when you have a remote site, you need to be able to get to the audio in the “audiowall” from that site the same as you can from the local site. One simple way of achieving this is to place the audiowall on a Windows share which all the playout machines can access. This works, and is perfectly fine when all your machines are on the same LAN. However, when you’re connecting to the share over a more flaky connection, it’s quite possible that it will fail, or become congested, causing stuttering as you play our the audio.
So another solution would be to set up a local copy of the audiowall at the remote site. You can do this by using robocopy (ROBust Copy) and you can use scheduled tasks to update the audio as required. However, this presents another problem – these robocopy scripts only allow you to copy the state of a directory in one direction. Any changes you make to the copy will not be replicated in the source (for example, you create a file with the same name, e.g. MYR2341.wav in the copy, and that will be replaced by the copy at the source).
One solution to this would be to split the entire audiowall into ranges, where one range is copied in one direction and another range is copied in the other. That way, both locations can copy into the audiowall, and both locations can read what has been imported by the other. This is a solution, which apparently some people use, however it’s not particularly scalable (the more locations you have, the less carts you can write to in each location) and is confusing for users going from place to place.
This was the route we were set to go down – however, it turns out there is a third option. Bi-directional synchronisation has been used for many years in mobile devices, to synchronise contacts, calendars, email and other items that you may want to edit on a mobile device and have the changes reflected in the version on your PC. Microsoft has developed a framework which adds to the .NET framework called Sync Framework which is designed for creating applications for precisely this.
We have designed a program which is based on their File Synchronisation Example which can synchronise any two directories, and will check last modified dates and creation dates as well as looking for deleted files on local and remote copies (all using change tracking) and will synchronise those two directories leaving both with the most up to date files.
The only slight problem is that if two files are edited in different locations at the same time – in this instance, there will be two “conflicting” changes. The Sync framework object will, by default, pick the file with the latest modified time as the “winner”. Losers can be processed differently, but in our case we set a directory to place conflict loser files into (so that they can be recovered easily should this happen to someone). Providing users use their own cart ranges, this should rarely present a problem – but if it did, then the mechanism is there to retrieve any data, and no data is lost in the process.
There are some domain specific tweaks we’ve had to make to support Myriad specifically – treating “list” files and inf files differently to try to ensure that metadata remains consistent throughout. In particular, if a cart is delete at a remote site, the wav file will be deleted, as well as the individual “list” file for that cart (providing you have that option turned on in Myriad) – however if Myriad has locally “seen” that cart, it will copy that metadata into the global list files (called CartsXXX.lst) and will trust that – even though the cart has been deleted. To date, the only way to resolve this problem is to run the Auto Importer Tool to remove all non-existent carts, which is a rather slow process. We’re working on a way of being able to tell Myriad that the cart has gone and that it should remove it from the Carts.lst file which it has locked whilst it is running.
We have also designed a synchronisation triggering mechanism that allows us to create different ways to trigger a synchronisation operation. For example, we have a trigger which watches the directories for changes – when a significant change is detected, a sync is performed. Another waits for a predetermined amount of time before firing a sync operation. Another provides a TCP/IP interface which allows you to remotely trigger a sync operation.
And all of this would not be much good unless you can see what it’s up to – so the service has a number of logging options – Windows Event Logs, plain text file, remote syslog and possibly more. Email notifications can also be set.
The program is implemented as a C# Windows System Service, with a GUI configuration program. The code will be made open source as soon as it is bug free enough to warrant inflicting it upon the world, and has been battle tested.
