Page 66 - SMILESENG

Intl. Summer School on Search- and Machine Learning-based Software Engineering
B. RESTest Framework
Test bots are based on RESTest [6], a black-box testing framework for RESTful web APIs. This has several benefits. On the one hand, existing test case and test data generation strategies from RESTest are already available in the online testing ecosystem, e.g., constraint-based testing [7] and re- alistic input test data extracted from knowledge bases [8]. On the other hand, implementing new testing strategies is straightforward, as it simply requires integrating them into RESTest. For details on how to develop new components in RESTest (including test case and test data generators), we refer the reader to its documentation [9] and its reference paper [6].
C. How-to Guide
The configuration of our online testing ecosystem depends on the APIs under test and the bots used to test the selected APIs. Each API can be tested by multiple bots simultaneously. For each test bot, the following resources are required:
• OAS specification of the API. This file contains a machine-readable definition of the API that can be used to drive the generation of test cases. Note that it can be reused by multiple bots.
• Test configuration file. This file specifies the test data generation strategy used by the bot, for instance, fuzzing dictionaries [10], semantically-related data [8], or manu- ally set data generators [7].
• Properties file. This file specifies several details related to the testing process such as the type of testing to perform (positive or negative), the test execution frequency, and the number of test cases to generate (or total test time).
Besides test bots, the remaining components of the ecosys- tem require little configuration: test reporters and test coverage computers are enabled/disabled based on the configuration of each test bot (as specified in its properties file); garbage collec- tors only need to be configured for those APIs where resources are created; the controller component is a ready-to-use set of Bash scripts. In order to ease the configuration and deployment of the whole ecosystem, we provide a supplementary package explaining the required steps to this end [11].
III. RECENT RESULTS
We deployed our testing ecosystem for 15 days continuously testing 13 industrial APIs, including highly popular APIs with millions of users worldwide such as YouTube, Spotify and Yelp. Overall, we generated 1,101,846 test cases, we uncov- ered 389,216 test failures, and we conservatively narrowed down these failures to 254 unique bugs. These bugs were varied, including inconsistencies between API implementation and API documentation, internal server errors with valid and invalid input data, inconsistencies between the status codes and response bodies, unparseable JSON responses and unexpected client errors, to name a few. Based on our evaluation results, we also extracted insightful conclusions such as the most effective testing techniques (constraint-based testing and data perturbation) and the most recurrent types of bugs (disconfor- mities with the API specification), among others.
IV. CHALLENGES AHEAD
Despite our promising results, we identified several chal- lenges hindering the adoption of automated test case gener- ation methods for online testing of APIs at scale, including automated fault identification (i.e., classifying thousands or millions of failures into tens or hundreds of unique faults), effective human interaction (i.e., leveraging human input to make bots more effective), and optimal selection of test- ing strategies (i.e., automatically determining the most ef- fective techniques based on several factors), among others. We envision the application of AI techniques to tackle these challenges. For instance, search algorithms could be used to automatically debug failures and isolate failure-inducing inputs. Similarly, active learning algorithms could leverage human input to improve the classification of faults by bots (e.g., confirming/discarding failures labeled as “warning”).
V. CONCLUSION
Online testing of web APIs is becoming an increasingly common practice in industry. However, existing platforms mostly automate test execution, while test cases still need to be manually implemented. In this talk, we overviewed the RESTest testing ecosystem and its capabilities to find real- world bugs in industrial APIs. We also highlighted challenges for the application of such a framework in practice, and how AI techniques could help address these challenges.
ACKNOWLEDGMENTS
54
This work
FEDER/Ministerio de Ciencia e Innovacio´n - Agencia Estatal de Investigacio´n under project HORATIO (RTI2018101204-B- C21), by FEDER/Junta de Andaluc´ıa under projects APOLO (US-1264651) and EKIPMENT-PLUS (P18-FR-2895), by the FPU scholarship program, granted by the Spanish Ministry of Education and Vocational Training (FPU17/04077).
REFERENCES
[1] R. T. Fielding, “Architectural Styles and the Design of Network-based Software Architectures,” Ph.D. dissertation, University of California, Irvine, 2000.
[2] “RapidAPI,” https://rapidapi.com, accessed March 2022.
[3] “Sauce Labs,” https://saucelabs.com, accessed March 2022.
[4] “OpenAPI Specification,” https://spec.openapis.org/oas/latest.html, ac-
cessed April 2022.
[5] A.Martin-Lopez,S.Segura,andA.Ruiz-Corte´s,“TestCoverageCriteria
for RESTful Web APIs,” in 10th International Workshop on Automating
TEST Case Design, Selection, and Evaluation, 2019, pp. 15–21.
[6] A. Martin-Lopez, S. Segura, and A. Ruiz-Corte´s, “RESTest: Automated Black-Box Testing of RESTful Web APIs,” in Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and
Analysis, 2021, pp. 682–685.
[7] A. Martin-Lopez, S. Segura, and A. Ruiz-Corte´s, “RESTest: Black-
Box Constraint-Based Testing of RESTful Web APIs,” in International
Conference on Service-Oriented Computing, 2020, pp. 459–475.
[8] J. C. Alonso, A. Martin-Lopez, S. Segura, J. M. Garcia, and A. Ruiz- Cortes, “ARTE: Automated Generation of Realistic Test Inputs for Web
APIs,” IEEE Transactions on Software Engineering, 2022.
[9] “RESTest,” https://github.com/isa-group/RESTest, accessed April 2022.
[10] V.Atlidakis,P.Godefroid,andM.Polishchuk,“RESTler:StatefulREST API Fuzzing,” in 2019 IEEE/ACM 41st International Conference on
Software Engineering, 2019, pp. 748–758.
[11] “[Supplementary material] Online Testing of RESTful APIs: Promises
and Challenges,” https://doi.org/10.5281/zenodo.6365937, 2022.
has been partially supported by