Automated regression testing of Chromecast using Cucumber, JRuby and Rukuli

As you may know, i work as a Developer In Test for Media Playout at the BBC. We enable audio and video playout on the web (desktop, tablet and mobile) using our Standard Media Player.

We recently added support for Chromecast, and i want to write a bit about how i automated testing for it.

Chromecast HDMI dongle

Chromecast is a HDMI dongle that plugs into a normal television and adds connected TV functionality. The neat thing is you can control it from your laptop, tablet or mobile using the media apps that you already know, such as YouTube, Netflix and now BBC iPlayer. Google also provides an API so that new Chromecast apps can be written all the time.

So how do you test a Chromecast? Well, personally i like to test things from an user’s point of view, actually clicking things and observing what happens as a result. But there’s only so far you can take this. I don’t want to rely on a television having a Chromecast being plugged in all the time and on the same network so that i can talk to it.

Though i’ve got to say .. it would be quite awesome to use a television 2 metres wide for my automated regression testing! :p

About to start casting

I didn’t need to test the Chromecast receiver app; that had already been written and tested by the guys in Salford for use by the iOS iPlayer app. I realised, what we actually care about is the communication between our player and the Chromecast. And there is a well documented Chromecast API. So with the help of colleagues Tim Hewitt and Wyell Hanna, i set about creating a fake Chromecast. Pretty much a bit of javascript that would respond in the same way as a real Chromecast, so that we could test how our player would respond to it.

And now i present to you .. the fake Chromecast!

https://gist.github.com/sermoa/10988494

A lot of it is empty methods that are neccessary to the API. But there are some neat tricks here too. I’ll talk you through a few of the more interesting bits.

Firstly we need to load the fake Chromecast into the page. We can achieve this with a @chromecast tag and a Before hook.

Before('@chromecast') do
  $chromecast = Chromecast.new
end

My Chromecast class has an initialize method that inserts the javascript into the page.

class Chromecast

  def initialize
    $capybara.execute_script <<-EOF
      var el=document.createElement("script");
      el.type="text/javascript";
      el.src = "https://gist.githubusercontent.com/sermoa/10988494/raw/82e08c5a29b5689b5e9f3d03c191b8c981102d85/fakeChromecast.js";
      document.getElementsByTagName("head")[0].appendChild(el);
    EOF
    @media_alive = true
  end

end

With this in place, so long as i set up our player with the relevant business logic that says casting is available, when i hover over the player, i get a cast button!

  @chromecast
  Scenario: Chromecast button appears
    Given Chromecast is available
    When I play media
    Then I should see the Chromecast button

A cast button appears!

So how does that work? I assure you, i don’t have a Chromecast on the network right now. I’m not using the Google Chrome extension. I’m actually running it in Firefox! What is this voodoo?!!

Have a look at the fakeChromecast.js file:

window.chrome = {};
window.chrome.cast = {};
window.chrome.cast.isAvailable = true;

See that? We’ve set up a Chromecast and said it’s available! Sneaky, hey? It’s that easy, folks! :)

The Standard Media Player will now attempt to set up a session request with its application ID. That’s fine: we’ll happily enable it to do its thing.

window.chrome.cast.SessionRequest = function(a) { }

Obviously a real Chromecast does something rather significant here. But we don’t have to care. This is a fake Chromecast. We only provide the function so that it doesn’t break.

Next, the media player makes a new ApiConfig. It’ll pass the session request it just obtained (in our case it’s null, but that doesn’t matter), and two callbacks, the second being a receiver listener. That’s the important one. We want to convince it that a Chromecast is available, so we trigger this callback with the special string “available”.

window.chrome.cast.ApiConfig = function(a, b, c) {
  c("available");
}

So Chromecast is available. Now suppose the user clicks the button to begin casting. This should request a session.

  @chromecast
  Scenario: Click Chromecast button and connect to a session
    Given I am playing media
    When I click the Chromecast button
    Then I should see Chromecast is connecting

Connecting to Chromecast

How did we do this? Easy! The media player requests a session, sending a success callback. Fake Chromecast store a references to that callback – on the window so that we can trigger it any time we like! The callback function is expected to provide a session. We send it a reference to a fake session, which is entirely within our control. Oh it’s all so much fun!

window.chrome.cast.requestSession = function(a, b) {
  window.triggerConnectingToCC = a;
  window.triggerConnectingToCC(window.fakeSession);
}

As the documentation says, “The Session object also has a Receiver which you can use to display the friendly name of the receiver.” We want to do exactly that. We decided to call our fake Chromecast Dave. Because Dave is a friendly name! :)

window.fakeSession = {};
window.fakeSession.receiver = {friendlyName:"dave", volume:{level:0.7}};

I think i found our app expected the session receiver to supply a volume too, so i added that.

The media player does some shenanigans with media that we don’t need to care about, but when it sends a request to load media it passes its callback to trigger when media is discovered by Chromecast. That’s another important one for us to keep, so we store that one. We wait 1 second for a semi-realistic connection time, and then trigger it, passing a reference to .. fake media, woo!

window.fakeSession.loadMedia = function(a, b, c) {
  window.pretendMediaDiscovered = b;
  setTimeout(function() {
    window.pretendMediaDiscovered(window.fakeMedia);
  }, 1000);
}

And now we are almost there. The last clever trick is the communication of status updates. The media player sends us a callback it wants triggered when there is a media status update. So we store that.

window.fakeMedia.addUpdateListener = function(a) {
  window.updateCCmediaStatus = a;
}

The magic of this is, we stored the references to fakeMedia and fakeSession on the browser’s window object, as well as the callbacks. This means the test script has access to them. Therefore the test script can control the fake Chromecast.

So you want Chromecast to report playing? Make it so!

  @chromecast
  Scenario: Begin casting
    Given I am playing media
    When I click the Chromecast button
    And Chromecast state is "PLAYING"
    Then I should see Chromecast is casting

We are casting!

What does that mean, Chromecast state is “PLAYING”? It’s pretty straightforward now, using the objects and callback functions that the fake Chromecast has set up:

When(/^Chromecast state is "(.*?)"$/) do |state|
  $chromecast.state = state
  $chromecast.update!
end

Those two methods get added to the Chromecast Ruby class:

class Chromecast

  def initialize
    $capybara.execute_script <<-EOF
      var el=document.createElement("script");
      el.type="text/javascript";
      el.src = "https://gist.githubusercontent.com/sermoa/10988494/raw/82e08c5a29b5689b5e9f3d03c191b8c981102d85/fakeChromecast.js";
      document.getElementsByTagName("head")[0].appendChild(el);
    EOF
    @media_alive = true
  end

  def state=(state)
    @media_alive = false if state == 'IDLE'
    $capybara.execute_script "window.fakeMedia.playerState = '#{state}';"
  end

  def update!
    $capybara.execute_script "window.updateCCmediaStatus(#{@media_alive});"
  end

end

Note that i had to do something special for the “IDLE” case because the media status update expects a true/false to indicate whether the media is alive.

So, using these techniques, sending and capturing events and observing reactions, i was able to verify all the two-way communication between the Standard Media Player and Chromecast. I verified playing and pausing, seeking, changing volume, turning subtitles on and off, ending casting, and loading new media.

This fake Chromecast is a lot faster than a real Chromecast, which has to make connections to real media. It’s also possible to put the fake Chromecast into situations that are very difficult to achieve with a real Chromecast. The “IDLE” state, for example, is really hard to achieve. But we need to know what would happen if it occurrs, and by testing it with a fake Chromecast i was able to identify a bug that would likely have gone unnoticed otherwise.

For the curious, here’s a demo of my Chromecast automated regression test in action!

This is all a lot of fun, but there is nothing quite like testing for real with an actual Chromecast plugged into a massive television!

Testing Chromecast for real

So that’s how i test Chromecast. If you want to test Chromecast too, please feel free to copy and tweak my fakeChromecast.js script, and ask me questions if you need any help.