I chuckled a bit today when I read about the 1000 Android devices that Netflix needs to support. I suspect a good third of those are trivial combinations of model and operator variants (my blog for example shows 7 variants of the HTC Incredible). But still—that’s a pretty big list of Androids!
Numbers such as these tend to horrify people, when in fact they’re not as bad as they first appear. The process Netflix uses to group (and identify) test devices is not that different from the one I described a few weeks back.
And buried within their article, there is this very important point…
With this information, we have taken stock of all the devices we have in house and classified them based on their specs. We figured out the optimal combination of devices to give us maximum coverage. We are able to reduce our daily smoke automation devices to around 10 phones and 4 tablets and keep the rest for the longer release wide test cycles.
The dirty little secret is that the more you test—the more accurately you will determine when it’s ok not to.
Sure these devices are all different, but those differences should inform your design and development strategy far more than your choice of workarounds or poly-fills.
Brad put it very well recently, while comparing the strategy of two mobile web sites:
…one is working with the constraints of the medium and using those constraints to it’s advantage, while the other is introducing unnecessary dependencies on what’s essentially a list of links…
The ultimate goal is to work with the medium, and tweak, workaround and poly-fill as little as possible.
I’m not talking about cutting corners here. I’m talking about knowledge and craft. That very tacit knowledge that comes from experience and enables you to identify constraints, design for them, and use them to your advantage. This is one of the great gifts (if you can excuse the sappy term here…) that device testing bestows.