Automatically Generating Precise Oracles from Structured Natural Language Specifications
by Manish Motwani, Yuriy Brun
Abstract:

Software specifications often use natural language to describe the desired behavior, but such specifications are difficult to verify automatically. We present Swami, an automated technique that extracts test oracles and generates executable tests from structured natural language specifications. Swami focuses on exceptional behavior and boundary conditions that often cause field failures but that developers often fail to manually write tests for. Evaluated on the official JavaScript specification, 98.3% of the tests Swami generated were precise to the specification. Using Swami to augment developer-written test suites improved coverage and identified 1 previously unknown defect and 15 missing JavaScript features in Rhino, and 19 previously unknown defects in Node.js.

Citation:
Manish Motwani and Yuriy Brun, Automatically Generating Precise Oracles from Structured Natural Language Specifications, in Proceedings of the 41st International Conference on Software Engineering (ICSE), 2019.
Bibtex:
@inproceedings{Motwani19icse,
  author = {Manish Motwani and Yuriy Brun},
  title =
  {\href{http://people.cs.umass.edu/brun/pubs/pubs/Motwani18icse.pdf}{Automatically 
  Generating Precise Oracles from Structured Natural Language Specifications}},
  booktitle = {Proceedings of the 41st International Conference on Software
  Engineering (ICSE)},
  venue = {ICSE},
  address = {Montreal, QC, Canada},
  month = {May},
  date = {29--31},
  year = {2019},

  accept = {$\frac{109}{529} \approx 21\%$},

  abstract = {<p>Software specifications often use natural language to
  describe the desired behavior, but such specifications are difficult to
  verify automatically. We present Swami, an automated technique that
  extracts test oracles and generates executable tests from structured
  natural language specifications. Swami focuses on exceptional behavior and
  boundary conditions that often cause field failures but that developers
  often fail to manually write tests for. Evaluated on the official
  JavaScript specification, 98.3% of the tests Swami generated were precise
  to the specification. Using Swami to augment developer-written test suites
  improved coverage and identified 1 previously unknown defect and 15 missing
  JavaScript features in Rhino, and 19 previously unknown defects in
  Node.js.</p>},

}