How to Decide Which Appium Test Framework Is Best For You

POC — Proof of Concept: evidence, typically deriving from an experiment or pilot project, demonstrates that a design concept, business proposal, etc. is feasible.

TL;DR

If you’re not keen to read the whole thing. The abbreviated version is that after trying out four different combinations with Appium. The team ended up choosing python + pytest + PyCharm because of two main things:

‍

Language learning curve and participation in the POC.

‍

Getting started

I’ve started working as a Test Consultant for this project about three weeks ago. I was hired to automate as much as possible of the mobile app in three months and pass along that knowledge to the team. Tough, I know, but let’s see how far can we get!

‍

It meant I had to assess the current situation and come up with feasible solutions quickly, and after getting to know the team and the product in the first week, it was time to pick the automation tooling we were going to use.

‍

Why Appium?

I quite like native solutions for UI testing, and XCUITest and Espresso are usually my default approach.

‍

But in this case, we have a cross-platform app developed in Xamarin. I’ve looked into the native solution for UI tests, Xamarin.UITest, but most of the content I could find online was from over four years ago, and that instantly caught my eye as a red flag.

‍

Because, as you’re going to see, I believe community support and adoption are quite important.

‍

If you’re going to do something new, it’s better to have help from others who’ve already done it or, even better, walk in the footsteps of others that have already faced the issues you’re likely going to face.

‍

More options, more problems

I haven’t properly used Appium in over three years now, and I had forgotten how many options we have when using it! Appium is just the middleman. Things like the language, IDE or test framework, you can choose your favourite.

‍

‍

But, what if you don’t really have a favourite? And what if the team you’ve joined doesn’t have an existing stack yet? Well, all that liberty can make things hard, especially if you’re like me and want to be sure you’ve done enough to go with the best solution.

‍

Options I’m going to evaluate

Based on my experience and the team’s experience, I came up with three options to do POCs. Appium +

• WebDriverIO + Javascript + VSCode
• pytest + Python + PyCharm
• TestNG + Java + Intellij IDEA

‍

Note: in this particular project, the development team is outsourced to an agency, which means we couldn’t leverage using the same language as the developers, as the collaboration/communication is not the same when everyone is on the same team working together.

‍

Defining a timebox and what success looks like

I wanted to evaluate three main things as part of the POC:

‍

— Language learning curve

— Test Framework Community support + Content available

— Existing knowledge in the team

‍

Because my time is short, I’ve allowed only one day for each POC. I believed that would be enough for me to get a feel of how hard it was to set up everything from scratch and create a couple of tests.

‍

‍

Time to put my assumptions to the test

‍

Test Framework Community support + Content available: pytest/TestNG (tie)

There was a lot of content available to get started with a new project in articles or videos on Youtube for both pytest and TestNG from the get-go. It’s hard for me to say which one had a lead on this one, but they both had the same output. Based on the content available, I could easily set everything up without having any prior knowledge of these languages/frameworks.

‍

I can’t say the same to WebDriverIO, even though apparently it’s great for web or mobile web. It took me half a day just to set up, and the worst part for me was finding existing snippets of codes that actually worked. At the end of the day, I was able to have created only one test that could log in to the app.

‍

Still, whenever I faced a challenge that was a bit harder than the usual Find element + Interact, it was tough to find out how things worked, and there were pretty much no helpful existing articles and videos around.

‍

Language learning curve: Python

I know this is a sensitive subject. Everyone has their preferred development language, so don’t mind me. This is just me trying to understand which one had a better fit for the rest of the team.

‍

I tend to like Python better than Java because despite Java being very well established and loved by many, I quite like the simplicity of Python and how we need to write less to have the same result.

‍

Because of that, I believed the rest of the team also would have a better understanding of how it worked and learn faster to start trying out and contributing to the automation.

‍

Existing knowledge in the team: Java

At the moment, the team is not doing any automation in any of the projects we have. But I knew some members of the team had already worked with several different automation frameworks.

‍

I found out that most of the team had more confidence and previous experience working with Java but would still be keen to go with Python if that had a better fit for what we had to do.

‍

‍

Feedback time

After gathering all this information in three days, I wanted to present it to our team and hear their thoughts about it.

‍

Oh, the importance of feedback

Based on the team’s feedback, we could go with either the Python or the Java-based framework. If there was a clear winner, we might’ve been able to pick one and go on, but I wasn’t decided yet.

‍

Option A: We could go with Python that was easier and more simple to learn, but no one in the team had had a long-ish experience with it, so there would be a lot of learning to do and no in-house go-to person to help others.

‍

Option B: Or we could go with Java, half the team liked it already, but I have a hard time getting around it and finding myself taking more time to do things, which could be the case for others.

‍

I wanted to make the best out of the POC. My main goal would be to implement the framework successfully and then hand that to the testers to keep adding new things and maintain it.

‍

The last thing I want is to do all the work and, when I leave, have the project die after a couple of months because the team isn’t happy with it or doesn’t really enjoy working with those tools.

‍

What if?

I’m very grateful to have heard the team’s feedback because an awesome member of our team mentioned yet another option I hadn’t really thought of. What about Kotlin? Well, the way I see it, Kotlin is kind of like a younger brother to Java, but it’s easier to learn and have pretty much all benefits that Java has, plus some other things I don’t really understand. What matters to me now is getting the work done and enabling others to keep getting the work done after I leave.

‍

‍

One last POC — Kotlin + JUnit + Intellij IDEA

I’ve allowed myself another day to try out the suggested option, same as the others. Here’s the comparison against the other three using the same criteria:

‍

Community support + Content available

I believe this one was pretty much a tie with every option, except WebDriverIO. It was also nice and easy to set up, with lots of content available to follow.

‍

Learning curve: Kotlin/Python (tie)

It’s hard to exemplify what a tie actually looks like regarding the learning curve for a language. I thought about it and came up with this result because Kotlin is easier than Java and about the same difficulty as Python.

‍

Existing knowledge in the team: Kotlin

The team already knows some Java, so migrating to Kotlin is way easier when compared to Python.

‍

‍

Plot twist

While I wrote this blog, believing to have chosen the stack Kotlin + JUnit + IntelliJ. The team was giving a go at python, and despite having no previous knowledge of the language and already be inclined to go with Java/Kotlin, everyone chose python in the end.

‍

I was happy with Kotlin already and was starting to build the test framework based on it. I believed it was a done deal, but it wasn’t.

‍

So I believe from all the learnings I made doing these experiments, the biggest one was that everyone involved has to give it a go before saying so. Same as I, each person will have their preconceptions and assumptions, and the decision can only be fully appreciated if all these assumptions are challenged practically.

‍

‍

Wrapping up

As you could see, it wasn’t a straightforward decision!

‍

I’m happy I had this week to try out different things for myself, I know it may look obvious now, but when we’re in a hurry, we might skip these important steps and go and make a decision based on assumptions. I could learn more this week about testing frameworks than I had in a long time, and even though there are no absolute right or wrong answers, there are answers that are a bit better than others, and these are often answers based on informed decisions.

‍

If you’ve read it this far, I have a rough idea of what success would look like two months from now, or even a year. But I’m keen to hear from you how you’d evaluate it, or even what you’d have done differently than I did.

‍

See you!

‍

---

Read this post by Fabio on Medium here.

‍