Firebase Test Lab for Android reduced American Express app test costs by 50%
Introduction
The AMEX mobile app, with a million Android downloads, is used by
Card Members to track spending and rewards, find offers, pay
bills, and enjoy app-only features. To facilitate frequent
updates, American Express built a device lab to test code changes
in their app across all platforms before release.
Challenge
With dozens of devices running various Android releases, the lab
was tough to scale. Adding new tests decreased productivity
because of the device lab's limited throughput, and when new team
members joined, the increased number of code changes added even
more load.
Maintaining the device lab was time-consuming and costly,
diverting the attention of engineers who otherwise would have
been focused on enhancements to drive additional value for Card
Members. The devices commonly had issues that required manual
intervention, often forcing reruns of the entire test suite.
Problematic devices were regularly removed from the device lab,
and replacing them often created down-time while the team awaited
the arrival of new devices.
Solution
During the product's alpha period, the frequency of updates
to the AMEX app, commitment to high-quality releases, and
willingness of engineers to collaborate provided an ideal
environment for the Firebase Test Lab team. American Express,
in turn, provided valuable feedback that helped to shape the
released version of the product. Firebase Test Lab integrated
with American Express' complex suite of hundreds of
instrumentation tests. Even while Test Lab was in early
development, American Express was able to switch all of their
continuous integration testing to Test Lab and successfully
retire much of their device lab. This enabled the engineers
maintaining the devices to shift their focus to shipping new
features to Card Members while also writing and running tests
to ensure high product quality.
In addition, American Express was able to scale the number of
tests without worrying about slowing down test-suite execution
and scaling up a homegrown device lab. This includes handling
surges in the number of tests as new team members join and
generating additional code changes. American Express now runs
over 1,000 instrumentation tests on every pull request, which
equals hundreds of thousands of tests per public release.
Running so many concurrent tests dramatically increases
their ability to prevent new and recurring bugs, all thanks
to Firebase Test Lab's ability to massively parallelize
testing.
DISCLAIMER: Results may vary.
Results
30% more tests in half the time
2X+ faster execution of full test suite, with no slowdown when adding
tests
50% lower cost vs. internal device lab
Firebase Test Lab enables American Express to execute our entire extensive Android test suite for every pull request. Firebase takes care of scaling the test infrastructure, so our engineers can focus on the code, driving improvements to the app and corresponding changes to our test suite."
- Michael Bailey, VP and Principal Engineer, American Express
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],[],[],[],null,["# American Express Case Study\n\n[*arrow_back*\nExplore other case studies](/case-studies) \n[View more solutions](/solutions) \n\n##### Firebase Test Lab for Android reduced American Express app test costs by 50%\n\n*** ** * ** ***\n\n##### Introduction\n\nThe AMEX mobile app, with a million Android downloads, is used by\nCard Members to track spending and rewards, find offers, pay\nbills, and enjoy app-only features. To facilitate frequent\nupdates, American Express built a device lab to test code changes\nin their app across all platforms before release.\n\n*** ** * ** ***\n\n##### Challenge\n\nWith dozens of devices running various Android releases, the lab\nwas tough to scale. Adding new tests decreased productivity\nbecause of the device lab's limited throughput, and when new team\nmembers joined, the increased number of code changes added even\nmore load.\n\nMaintaining the device lab was time-consuming and costly,\ndiverting the attention of engineers who otherwise would have\nbeen focused on enhancements to drive additional value for Card\nMembers. The devices commonly had issues that required manual\nintervention, often forcing reruns of the entire test suite.\nProblematic devices were regularly removed from the device lab,\nand replacing them often created down-time while the team awaited\nthe arrival of new devices. \n\n*** ** * ** ***\n\n##### Solution\n\nDuring the product's alpha period, the frequency of updates\nto the AMEX app, commitment to high-quality releases, and\nwillingness of engineers to collaborate provided an ideal\nenvironment for the Firebase Test Lab team. American Express,\nin turn, provided valuable feedback that helped to shape the\nreleased version of the product. Firebase Test Lab integrated\nwith American Express' complex suite of hundreds of\ninstrumentation tests. Even while Test Lab was in early\ndevelopment, American Express was able to switch all of their\ncontinuous integration testing to Test Lab and successfully\nretire much of their device lab. This enabled the engineers\nmaintaining the devices to shift their focus to shipping new\nfeatures to Card Members while also writing and running tests\nto ensure high product quality. \nIn addition, American Express was able to scale the number of\ntests without worrying about slowing down test-suite execution\nand scaling up a homegrown device lab. This includes handling\nsurges in the number of tests as new team members join and\ngenerating additional code changes. American Express now runs\nover 1,000 instrumentation tests on every pull request, which\nequals hundreds of thousands of tests per public release.\nRunning so many concurrent tests dramatically increases\ntheir ability to prevent new and recurring bugs, all thanks\nto Firebase Test Lab's ability to massively parallelize\ntesting.\n\nDISCLAIMER: Results may vary.\n\n*** ** * ** ***\n\nResults\n\n30% more tests in half the time\n\n2X+ faster execution of full test suite, with no slowdown when adding\ntests\n\n50% lower cost vs. internal device lab \n\nFirebase Test Lab enables American Express to execute our entire extensive Android test suite for every pull request. Firebase takes care of scaling the test infrastructure, so our engineers can focus on the code, driving improvements to the app and corresponding changes to our test suite.\"\n\n\n- Michael Bailey, VP and Principal Engineer, American Express \nTry Firebase today\n\n\nIntegrating it into your app is easy.\n[Get started](https://console.firebase.google.com/) \n\n#### All Firebase products\n\n##### Build\n\n- [App Check](/products/app-check)\n- [App Hosting](/products/app-hosting)\n- [Authentication](/products/auth)\n- [Cloud Functions](/products/functions)\n- [Cloud Storage](/products/storage)\n- [Data Connect](/products/data-connect)\n- [Extensions](/products/extensions)\n- [Firestore](/products/firestore)\n- [Firebase ML](/products/ml)\n- [Genkit](https://genkit.dev/)\n- [Hosting](/products/hosting)\n- [Realtime Database](/products/realtime-database)\n- [Firebase AI Logic client SDKs](/products/firebase-ai-logic)\n\n[Generative AI](/products/generative-ai) \n\n##### Run\n\n- [A/B Testing](/products/ab-testing)\n- [App Distribution](/products/app-distribution)\n- [Cloud Messaging](/products/cloud-messaging)\n- [Crashlytics](/products/crashlytics)\n- [Google Analytics](/products/analytics)\n- [In-App Messaging](/products/in-app-messaging)\n- [Performance Monitoring](/products/performance)\n- [Remote Config](/products/remote-config)\n- [Test Lab](/products/test-lab)"]]
