HerrEmil

On my thirty-first and last day of Devember, I got my check to pass in Internet Explorer.

The main challenge with getting the code to run will in Internet Explorer was to run it with a proxy server, without the proxy server, the check passes if not interrupted by an annoying popover ad, but there was no easy way to handle the ad.

Getting IE to run with a proxy was not as hard as I feared. When you start the browser instance and try to go to a URL, you get stopped by an untrusted certificate notice, but this notice is a page, and the Continue to this website link has an id making it nice and easy to click and continue with this check:

if(driver.findElements(By.id("overridelink")).size() > 0) {
   driver.get("javascript:document.getElementById('overridelink').click();");
}

Once I had clicked away that notice, the code ran better than I had expected from Internet Explorer. It was a little flaky at first but after adding a simple wait it seems to be as stable as other browsers.

The second part of getting the code to what I consider being done is have the code not mess with my connection settings when running it Internet Explorer. To do that, I have to call BrowserMobProxy’s close() when I’m done with the checks. I made the proxy in my Browser class public and call the close() function in the tearDown function in my BaseTestClass class. This works well with one caveat, if a check fails in Internet Explorer, the tearDown function is never called. I don’t know if that is a limitation in TestNG or IE, but I am fairly certain I am not doing anything that could cause that but I did not dig deeper into it.

After getting the check to pass in Internet Explorer 11, I double-checked that the check still passes in all other browsers. There is still some mysterious Edge flakiness, which sometimes fails after repeated runs. The actual check passes, but the teardown fails. When I made the check pass in Edge I guessed that this issue had something to do with having still running drivers from previous runs but that was not it, I don’t know what causes the tearDown to fail in Edge sometimes.

Other than that, the check passes in Firefox, Chrome and Safari on Mac and Firefox, Chrome, Edge and Internet Explorer on PC.

My main takeaway from getting the code to work in all browsers is that is inefficient to start testing against Firefox, even if it’s the easiest to set up. Start with Safari if you’re on Mac or Internet Explorer if you’re on PC, these will be the most sensitive to sloppiness and forces you to write more stable code, which will be easy to get to work with other browsers.

So, was Devember a success? My goal was to learn Java, well enough to write a functional test framework, by reading the book Selenium Testing Tools Cookbook. This goal was kinda flawed, as reading that book turned out to not teach me much Java at all, but it did teach me many useful techniques for using Selenium. As for learning Java well enough to write a framework, the language part of that challenge is smaller than architecturing the framework structure. I learned about structuring a test framework from several places, but mainly Sebarmeli’s archetype and John Sonmez’s pluralsight guides Automated Web Testing with Selenium and Creating an Automated Testing Framework With Selenium. Most things I did not know about Java were caught by IntelliJ or were easy to google. I still feel like I could learn a lot about how to write Java and might read a book on just that next.

I will probably not continue updating this project (no time for that), but I will definitely continue working on the automation framework I started in parallel with this one at work :)

On my thirtieth day of Devember, I got my check to pass in Edge.

There are three versions of MicrosoftWebDriver.exe, one called Microsoft WebDriver that only works on an old Windows 10 build, one called Microsoft WebDriver Fall 2015 Update which I would just call stable and finally WebDriver for Windows Insiders which is the latest preview build.

When I first tried to run my script on Edge a few days ago, I only knew of the version called Microsoft WebDriver which failed in a confusing way. Today I tried the other two versions and had better luck. I did not see any difference in how the script ran between what I call the stable version and the preview build, and went with the stable one.

There were two things in my code that were failing in Edge; dismissing the cookie notice and the seat picker defaulting to a flash player.

The function that dismisses the cookie notice fails on the statement that checks if the notice is displayed. EdgeDriver throws a NullPointerException when calling cookieNotice.getAttribute("style").contains("display: block;")) when there is no style attribute. I had to check that cookieNotice.getAttribute("style") != null before calling .contains().

There is no option for disabling flash in Edge from code. Looking at org.openqa.selenium.edge.EdgeOptions.java there isn’t much happening, so I guess it’s not close to being available either. I tweeted @MicrosoftEdge and asked for an update, but for now I disabled flash manually as I did with Internet Explorer.

Other than those two issues, I had to add two waits in two functions that were sometimes failing. Also, there is some weirdness with the post-check tearDown function failing sometimes. I believe it happens consistently after you have ran a suite without closing and quitting the driver, but I’m not sure how to easily fix it when it happens. Killing all remaining Microsoft WebDriver processes in the Task Manager does not solve the issue but rebooting does.

Tomorrow is the last day of Devember and I have more things I would like to do than I can do in one day so I’ll have to make a choice. The most interesting possible tasks to do tomorrow are:

• Get Browsermob Proxy to work in Internet Explorer
• Try to handle the flash player seat picker
• Check which browsers are available on current platform and run all

On my twenty-ninth day of Devember, I learned that it is impossible to to automate the top menu in Safari.

The code I wrote yesterday to call instead of the the Action class’s moveToElement() was not working on the top menu. To double-check, I tried it on another page and found that it worked just fine. I looked a bit more carefully at the code for the top menu and found that it uses the :hover CSS pseudo-class. This is unfortunate as it is impossible to trigger the hover event from javascript, which means it’s impossible to automate the top menu’s sub-menus in Safari.

I adapted the check I have to skip the top menu and go directly to the next step. I will have to skip any interaction with the top menu in most checks if I have time to write more. This situation shows the value of having a configuration file which lists desired browsers for each test or at least each test suite.

After removing the top menu interaction, the check now finally passes in Safari :)

Since the code now passes in Safari, Firefox and Chrome on OSX, I’ll focus on Windows for the remaining two days of Devember.

Ony my twenty-eighth day of Devember, I continued working on getting my script to run in Safari.

Instead of continuing with the Internet Explorer work I started yesterday, I started working on getting the code to run in Safari today. There was no big thought behind the switch, I just happened to be on a Mac when it was time for Devember.

The top menu on sf.se is lists of links that are displayed when you hover over a section in the top menu. In the first version of the code, I used the lovely Actions class to handle this menu. Unfortunately the SafariDriver does not have an implementation of Actions, and from what I’ve read in open tickets about it, it does not seem to be a priority for the Selenium team to fix it, so I cannot use it. Today I started implementing my own moveToElement function to trigger mouseovers.

I have not gotten the function to work properly yet, but I think I am about halfway. The function saves the appropriate element that should get the mouseover event in JavaScript, and I think the function that triggers the appropriate mouseover event is functioning. Still, the sub-menu that should be displayed on mouseover is not displayed, so I still have some investigation to do.

Working with Safari is just as enjoyable as working with Internet Explorer. Today I had issues with both my connection breaking and instances of Safari refusing to be killed when running code against Safari. After trying to kill them, they got stuck in some kinda limbo between running and not running and only displayed an error message.

I even had to force turn off the mac by holding the power button at one point as an instance of safari refused to let OSX shutdown, it’s been a while since an OS locked up that bad for me.

I’m not sure what to think about OSX displaying its Finder smiley when it’s telling me it’s broken. Tomorrow I’ll continue the Safari fight.

On my twenty-seventh day of Devember, I got my code to run in Internet Explorer, kinda.

Today, I started looking into getting the code I have running in the more troublesome web browsers. First up was Internet Explorer. First hurdle with Internet Explorer was installation, not because it’s hard, but because top google results had incorrect info.

According to the info I read, you should download the IE driver from Selenium’s site if you want to run checks in IE7 - 10, but from Microsoft’s site if you want to run in IE11. As I’m on Windows 10 and only have IE11, I went to Microsoft’s site to get the driver. Microsoft’s site does not have a direct download link for the driver, but instead has .msu Windows Update package files. There’s 4 choices of package files, that gives you the choice between 32 and 64 bit and between Windows 7 and Windows 8.1, there is no package for Windows 10. I tried using one of the packages but the Windows Update Standalone Installer is not willing to install a package meant for an older Windows.

I continued googling and reading and found that there is no need to use the driver from Microsoft’s site, the one on Selenium’s site should work in all 7+ versions of IE. The driver from Selenium was easier to get working; download from direct link, place in convenient folder, add folder to PATH and do a System.setProperty("webdriver.ie.driver", "C:\\WebDrivers\\IEDriverServer.exe"); call before creating an instance of the driver. This way I could launch an instance of IE11 but using the Browsermob Proxy I set up to block ads breaks the flow.

Even more, using the proxy in Internet Explorer changes settings in Internet Options, checking the option “Use a proxy server for your LAN” and not unchecking it when the script ends.

Leaving that option checked will break the connection for all browsers and most other apps. This was a bit annoying as trying to get the check to pass was disconnecting chrome which I was using for reading stack overflow and google code.

For now, I disabled the proxy, and then my check ran up until the seat picker, which failed because the flash player was loaded. I spent some time looking for a solution for this. There is no option (that I can find) for disabling flash through my code. All info I could find pointed me in the direction of BrowserStack, which has instances of IE with and without flash and you can configure which one you want in your DesiredCapabilities.

Since having your scripts disable flash when it starts using the regular IE driver was not an option, I read up on interacting with flash objects. The closest to usable thing I found was the flash-selenium project, but this project depends on the flash object exposing functionality through flash’s ExternalInterface. I tried to find out how to check if a flash object as an external interface but all guides I found assumed you have access to the flash source and know the names of all methods you want to call, which I don’t.

For now, I disabled the flash plugin manually and re-ran my code, and the check passes in IE11 with flash disabled. I might continue getting Internet Explorer to play nicely with Browsermob or learn how to interact with flash objects tomorrow, but I might consider the current state good enough and move on to Edge or Safari.

It is a big eye-opener to see how big the gaps in supported functionality are in Selenium when you are using browsers other than Firefox, Chrome, phantomjs or HTMLUnit. I have read a lot guides and how-tos that I now know are useless if you want to run checks in all major browsers. To me, running automated checks in Firefox and Chrome but not in Internet Explorer and Edge is ridiculous, you are skipping the same browsers that developers often skip when developing and you’re losing a lot of potential value from your code.

On my twenty-sixth day of Devember, I ran my script in different versions of Internet Explorer and tried to run it in Edge.

When I looked at running tests in Internet Explorer and Edge yesterday, I saw some issues early on, so today I thought I would try running tests in Internet Explorer and Edge through Sauce Labs, and hopefully avoid the extra hassle.

Though I did successfully run my script in different versions of Internet Explorer, I don’t consider using Sauce Labs for this project a success yet. I could not get Sauce Labs to use the Browsermob proxy I want to use. If I disable the proxy, the script runs, but the check is painfully slow when running it on Sauce Labs and waiting for ads to load for each step. Noted that I will have to disable flash for these browsers as well.

I had even less luck running the test in Edge. Using Sauce Labs, the browser is launched and goes to the site I am testing, but the checks fails in the first first function call. The place where it fails is a straightforward Selenium statement, specifically a findElement(By.tagName("a")) part which fails with an Unknown command received message. It’s quite odd that the EdgeDriver should have problems with something so simple, and Microsoft’s Web Driver site does list the locator as supported so it should work.

I could continue solving all failures in Internet Explorer and Edge using Sauce Labs but this will be very slow, each test run involves a couple of minutes of waiting, so this is not efficient way of developing more checks. I think it’s fair to say that Sauce Labs is useful for running all checks in all browsers you want to support nightly or something like this, but for this project where I’m just setting up a framework but not making any changes on the actual product, it won’t be very helpful.

It looks like if I want to make sure checks run in Safari, Internet Explorer and Edge, I have to run these browsers and drivers on my own machines. Since these are all giving me troubles, I guess that getting the first check to run on all will keep me busy until the end of the month.

On my twenty-fifth day of Devember, I moved to Windows and tried to the get my script to work in different Windows browsers.

First I tried Chrome, which was fairly painless. Download and launch chromedriver.exe to launch a RemoteWebDriver server and connect to it. I could create a new instance of ChromeDriver I guess, but it feels like a wasted effort to set up PATH variables when I want to start all browsers through RemoteWebDriver later in any case. On function, NumberOfAdults, failed in Chrome, because it had a sloppy thread.sleep instead of a wait. After adding a proper wait, it worked fine.

Second I tried Firefox, which has WebDriver built-in so no setup needed, and starting it was trivial. It failed on the seating selection page as it was loading the flash player which is installed on this machine’s firefox. Disabling flash in Firefox was similar to doing it in Chrome and pretty straightforward:

FirefoxProfile profile = new FirefoxProfile();
profile.setPreference("plugin.state.flash", 0);
capabilities.setCapability(FirefoxDriver.PROFILE, profile);

Third I wanted to try Internet Explorer 11, but just reading through the pre-requirements made me post-pone it to tomorrow, I don’t have time to to try it out today.

Before calling it a day, I wanted to try the Edge WebDriver, which is not called EdgeDriver.exe, but rather MicrosoftWebDriver.exe and comes with its own installer and all. Running the executable launches a RemoteWebDriver that can easily be connected to, much like ChromeDriver. Unlike ChromeDriver though, it doesn’t work straight away. Initializing the driver fails without throwing an error and the stacktrace is not super helpful to me. It seems setting up Edge will also take some extra effort.

On my twenty-fourth day of Devember, I started using different browsers to see how the script performed.

I’ve been using Firefox on OSX only so far, which is maybe a bad place to start as FirefoxDriver is probably the most stable driver. Today I started with trying out Chrome on OSX, and the only thing that did not work was that the seat picker when booking tickets defaults to a flash player. I added some settings in the setup to disable the flash plugin and the website falls back to the easier to manager html seat picker.

Next up was Safari on OSX, which you’d think would be similar to Chrome, but it is not. Nothing works in Safari. The Actions API used to chain actions and perform stuff like hovers or drag-and-drops is not implemented at all in Safari. Since the top menu at sf.se has sub-menus only visible when hovering over the first element, I need to do hovers to use the menu. I did some reading about and found that I have to do mouseover events in JavaScript if I use Safari. Since you can’t trigger CSS :hover with JavaScript, I might run into situations where mouseover can’t be tested in Safari without using another tool like Java Robot, AutoIt or Sikuli. But before I even get to that point, I have to get the basics working in Safari. Even with javascript mouseover and click events, I couldn’t handle the top menu in Safari.

To move on with the next step, I decided to go directly to the page the menu leads to by driver.get(url), but the very next step failed in a similar way as Safari is very sensitive to interacting with elements that are not visible, chaining together open drop-down and click option into one statement doesn’t work, as the drop-down option is not visible before the statement. Safari seems to be a bit of a pain to get things working in so far. The fiddling with Safari took quite some time so not much fun stuff added to the code today, just some tweaks.

Another thing that’s special with Safari is that when you open a new browser, you don’t get a completely fresh browser instance. I noticed that my dismissCookiesNotice() function failed almost immediately after switching to Safari. Turns out it was not displayed from the second time I used Safari onwards. I figured it’s more correct to have the function not try to click a button unless it knows that it’s displayed so I added a simple check for that.

On my twenty-third day of Devember, I made my first check pass consistently.

Today was fairly straightforward, all challenges were regular WebDriver ones like finding reliable selectors and waits so progress was good, the first check now passes!

I don’t consider myself done with the first check though, there are plenty of TODOs in the script on things I want to fix and I was using Firefox the whole time when trying it out. Tomorrow I’ll move to a Windows machine and try other browsers, hopefully Internet Explorer will not ruin my Christmas :)

On my twenty-second day of Devember, I got rid of an annoying pop-over ad by adding a proxy and learning some basic regex, but removing the pop-over came at the price of affecting other site functionality.

After looking around a bit, I found that the best way to block ads is to connect your WebDriver to a proxy server and have that proxy block calls to ad servers. So I looked at which sources were listed in Chrome’s developer tools and saw this:

No wonder my script was a bit slow to navigate the page when it’s grabbing ads from more than a dozen different places.

Anyway, I added Browsermob Proxy to the project and setting it up was fairly straightforward. The only tricky part was that all URLs you want blacklisted should be written as regexes and I don’t know regex, some online regex tester and trial error solved that pretty quickly.

I added all sources from the screenshot and tried to run my checks, but when blocking all of them, the site broke. I went through the sources one by one and checked running the script with them on or off to see which ones were safe to block. I unblocked ajax.googleapis.com and use.typekit.com and then the script ran almost exaclty as with no proxy but faster because most ads are gone.

Unfortunately, one step that was working yesterday was no longer working after adding the blocks; when selecting a cinema, the list of movies is not updated to only show movies showing in that cinema. I tracked down that functionality to the server fusion.sf.se, but of course that is the exact same place that has the call to open the annoying pop-over ad. I tried to change the proxy to only block URLs containing fusion.sf.se/scriptSF.js?ads=sf calls, but the movie filter was still broken with this block. I thought that maybe my regex for that URL was off and tried blocking URLs containing .*ads=sf but the functionality was still broken. Why on earth the movie filter depends on an ads=sf parameter is beyond me, it must be a mistake. I tried loading the movie filter page manually with uBlock turned on and off, and core functionality is indeed broken for user with an adblocker installed.