Blindspots in Python and Java APIs Result in Vulnerable Code
by Yuriy Brun, Tian Lin, Jessie Elise Somerville, Elisha M. Myers, Natalie Ebner
Abstract:
Blindspots in APIs can cause software engineers to introduce vulnerabilities, but such blindspots are, unfortunately, common. We study the effect APIs with blindspots have on developers in two languages by replicating an 109-developer, 24-Java-API controlled experiment. Our replication applies to Python and involves 129 new developers and 22 new APIs. We find that using APIs with blindspots statistically significantly reduces the developers' ability to correctly reason about the APIs in both languages, but that the effect is more pronounced for Python. Interestingly, for Java, the effect increased with complexity of the code relying on the API, whereas for Python, the opposite was true. This suggests that Python developers are less likely to notice potential for vulnerabilities in complex code than in simple code, whereas Java developers are more likely to recognize the extra complexity and apply more care, but are more careless with simple code. Whether the developers considered API uses to be more difficult, less clear, and less familiar did not have an effect on their ability to correctly reason about them. Developers with better long-term memory recall were more likely to correctly reason about APIs with blindspots, but short-term memory, processing speed, episodic memory, and memory span had no effect. Surprisingly, professional experience and expertise did not improve the developers' ability to reason about APIs with blindspots across both languages, with long-term professionals with many years of experience making mistakes as often as relative novices. Finally, personality traits did not significantly affect the Python developers' ability to reason about APIs with blindspots, but less extraverted and more open developers were better at reasoning about Java APIs with blindspots. Overall, our findings suggest that blindspots in APIs are a serious problem across languages, and that experience and education alone do not overcome that problem, suggesting that tools are needed to help developers recognize blindspots in APIs as they write code that uses those APIs.
Citation:
Yuriy Brun, Tian Lin, Jessie Elise Somerville, Elisha M. Myers, and Natalie Ebner, Blindspots in Python and Java APIs Result in Vulnerable Code, ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 32, no. 3, April 2023, pp. 76:1–76:31.
Bibtex:
@article{Brun23tosem,
  author = {Yuriy Brun and Tian Lin and Jessie Elise Somerville and Elisha M. Myers and Natalie Ebner},
  title =
  {\href{http://people.cs.umass.edu/brun/pubs/pubs/Brun23tosem.pdf}{Blindspots in Python and Java APIs Result in Vulnerable Code}},
  journal = {ACM Transactions on Software Engineering and Methodology (TOSEM)},
  venue = {TOSEM},
  year = {2023},
  volume = {32},
  number = {3},
  issn = {1049-331X},
  pages = {76:1--76:31},
  month = {April},

  doi = {10.1145/3571850},
  note = {\href{https://doi.org/10.1145/3571850}{DOI:
  10.1145/3571850}, arXiv: \href{https://arxiv.org/abs/2103.06091}{abs/2103.06091}. 
  Presented as journal-first paper at ICSE 2023},

  abstract = {Blindspots in APIs can cause software engineers to introduce vulnerabilities,
  but such blindspots are, unfortunately, common. We study the effect APIs with
  blindspots have on developers in two languages by replicating an
  109-developer, 24-Java-API controlled experiment. Our replication applies to
  Python and involves 129 new developers and 22 new APIs. We find that using
  APIs with blindspots statistically significantly reduces the developers'
  ability to correctly reason about the APIs in both languages, but that the
  effect is more pronounced for Python. Interestingly, for Java, the effect
  increased with complexity of the code relying on the API, whereas for Python,
  the opposite was true. This suggests that Python developers are less likely
  to notice potential for vulnerabilities in complex code than in simple code,
  whereas Java developers are more likely to recognize the extra complexity and
  apply more care, but are more careless with simple code. Whether the
  developers considered API uses to be more difficult, less clear, and less
  familiar did not have an effect on their ability to correctly reason about
  them. Developers with better long-term memory recall were more likely to
  correctly reason about APIs with blindspots, but short-term memory,
  processing speed, episodic memory, and memory span had no effect.
  Surprisingly, professional experience and expertise did not improve the
  developers' ability to reason about APIs with blindspots across both
  languages, with long-term professionals with many years of experience making
  mistakes as often as relative novices. Finally, personality traits did not
  significantly affect the Python developers' ability to reason about APIs with
  blindspots, but less extraverted and more open developers were better at
  reasoning about Java APIs with blindspots. Overall, our findings suggest that
  blindspots in APIs are a serious problem across languages, and that
  experience and education alone do not overcome that problem, suggesting that
  tools are needed to help developers recognize blindspots in APIs as they
  write code that uses those APIs.},

  fundedBy = {NSF CCF-1453474, NSF CNS-1513055, NSF CNS-1513457, NSF CNS-1513572, NSF CCF-1564162},
}