WavebreakmediaMicro - Fotolia

Problem solve Get help with specific problems with your technologies, process and projects.

Anki test director talks IoT and AI testing challenges

A complicated new game required a complicated testing strategy and help from the outside. Here's what game maker Anki learned from working with LogiGear on AI testing.

Startup game maker Anki created a highly technical game using both the internet of things and AI but realized fairly quickly it needed help with testing, particularly because the game is targeted at the not necessarily patient 14-and-under crowd. In this first part of a two-part Q&A, Anki Test Director Jane Fraser explains why the company reached out to LogiGear for help and what she learned in the process. In part two, a LogiGear executive shares a detailed look at the challenges of testing in the IoT and AI space.

What were you hoping to achieve working with LogiGear on IoT and AI testing? Were you looking to improve your game?

Jane FraserJane Fraser

Jane Fraser: At my previous job, we worked with many outsourced test teams, and LogiGear was by far the best value. When I joined Anki, the company was a new up-and-coming robotics startup with a small office and budget. LogiGear instantly came to mind when I was looking at expanding the testing of our first robotic battle-racing game, Anki Drive. This was a whole new testing field for me and LogiGear -- robot cars driven with AI, localizing with a tiny camera that reads infrared code/data on the tracks, using BTLE [Bluetooth Low Energy] on the iPhone to connect to the cars and other devices while also transmitting data to the cloud. My [LogiGear] team was great at digging in and helping us figure out how to test all this new technology.

What surprised you during the LogiGear IoT and AI testing process?

Fraser: How much they valued my rules. Since our product, at the time, was not announced, we had a secure room, and I stressed to the team that we needed to keep everything a secret, including putting product away at night so that the cleaning crew wouldn't even see it. Well, one day, LogiGear's CEO came by to check on the team, and they wouldn't let him in. I really appreciated that they valued our secrecy. That attention to every detail was very valuable to me. The CEO ended up calling me, and I gave the testing team the thumbs-up for him to check out the product. I always joke that the team works for me, not LogiGear.

They also really value our end customers to make sure to bring up issues they believe should be fixed before a release and why. Also, at this point, they create all our test plans and regression suites, allowing the U.S. team to concentrate on working with the developers and designers to make sure we have all the info we need and make sure we understand all we are building.

Given what you learned during the IoT and AI testing process, what would you do differently with your next product offering?

Fraser: Hardware testing is hard. Some [changes] we have already started, like testing earlier and trying to make changes one at a time as opposed to all at once. An example with Anki Drive was when we received new cars (new chassis version), new tires (different durometer -- rubber hardness) and new track material. Fortunately, we quickly backed up and tested one item at a time then started merging them together so we could isolate the issues.

We've been able to start earlier in testing, including using tools to mock up new hardware so testing can start earlier.

Do you have testers on staff?

Fraser: Yes, we have a team of 12 in the U.S., divided between our products: Anki Overdrive, Cozmo, e-commerce, automation and R&D. All the teams interact with their counterpart teams in Vietnam. To make it much easier, we are also all on Slack, and everyone at Anki is comfortable going direct to the [LogiGear] team and getting testing done or discuss issues. We've even had a few team members stop in to see the team on their vacation in Vietnam. They have really become part of the Anki family.

Can you tell me how long it took to develop versus test?

Fraser: We are about two to three developers to one tester; sometimes, it seems a lot bigger when we are also doing hardware testing, which takes years of development when we start. At times, it feels like 100 to one, especially with such cutting-edge technology in our products and having to invent new ways to test.

Next Steps

Using AI to help with software testing

Top trends in AI and big data

AI and the public cloud -- what you need to know

Dig Deeper on AI in software testing