by Yuriy Brun, Tian Lin, Jessie Elise Somerville, Elisha M. Myers, Natalie Ebner
Abstract:
Blindspots in APIs can cause software engineers to introduce vulnerabilities, but such blindspots are, unfortunately, common. We study the effect APIs with blindspots have on developers in two languages by replicating an 109-developer, 24-Java-API controlled experiment. Our replication applies to Python and involves 129 new developers and 22 new APIs. We find that using APIs with blindspots statistically significantly reduces the developers' ability to correctly reason about the APIs in both languages, but that the effect is more pronounced for Python. Interestingly, for Java, the effect increased with complexity of the code relying on the API, whereas for Python, the opposite was true. This suggests that Python developers are less likely to notice potential for vulnerabilities in complex code than in simple code, whereas Java developers are more likely to recognize the extra complexity and apply more care, but are more careless with simple code. Whether the developers considered API uses to be more difficult, less clear, and less familiar did not have an effect on their ability to correctly reason about them. Developers with better long-term memory recall were more likely to correctly reason about APIs with blindspots, but short-term memory, processing speed, episodic memory, and memory span had no effect. Surprisingly, professional experience and expertise did not improve the developers' ability to reason about APIs with blindspots across both languages, with long-term professionals with many years of experience making mistakes as often as relative novices. Finally, personality traits did not significantly affect the Python developers' ability to reason about APIs with blindspots, but less extraverted and more open developers were better at reasoning about Java APIs with blindspots. Overall, our findings suggest that blindspots in APIs are a serious problem across languages, and that experience and education alone do not overcome that problem, suggesting that tools are needed to help developers recognize blindspots in APIs as they write code that uses those APIs.
Citation:
Yuriy Brun, Tian Lin, Jessie Elise Somerville, Elisha M. Myers, and Natalie Ebner, Blindspots in Python and Java APIs Result in Vulnerable Code, ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 32, no. 3, April 2023, pp. 76:1–76:31.
Bibtex:
@article{Brun23tosem,
author = {Yuriy Brun and Tian Lin and Jessie Elise Somerville and Elisha M. Myers and Natalie Ebner},
title =
{\href{http://people.cs.umass.edu/brun/pubs/pubs/Brun23tosem.pdf}{Blindspots in Python and Java APIs Result in Vulnerable Code}},
journal = {ACM Transactions on Software Engineering and Methodology (TOSEM)},
venue = {TOSEM},
year = {2023},
volume = {32},
number = {3},
issn = {1049-331X},
pages = {76:1--76:31},
month = {April},
doi = {10.1145/3571850},
note = {\href{https://doi.org/10.1145/3571850}{DOI:
10.1145/3571850}, arXiv: \href{https://arxiv.org/abs/2103.06091}{abs/2103.06091}.
Presented as journal-first paper at ICSE 2023},
abstract = {Blindspots in APIs can cause software engineers to introduce vulnerabilities,
but such blindspots are, unfortunately, common. We study the effect APIs with
blindspots have on developers in two languages by replicating an
109-developer, 24-Java-API controlled experiment. Our replication applies to
Python and involves 129 new developers and 22 new APIs. We find that using
APIs with blindspots statistically significantly reduces the developers'
ability to correctly reason about the APIs in both languages, but that the
effect is more pronounced for Python. Interestingly, for Java, the effect
increased with complexity of the code relying on the API, whereas for Python,
the opposite was true. This suggests that Python developers are less likely
to notice potential for vulnerabilities in complex code than in simple code,
whereas Java developers are more likely to recognize the extra complexity and
apply more care, but are more careless with simple code. Whether the
developers considered API uses to be more difficult, less clear, and less
familiar did not have an effect on their ability to correctly reason about
them. Developers with better long-term memory recall were more likely to
correctly reason about APIs with blindspots, but short-term memory,
processing speed, episodic memory, and memory span had no effect.
Surprisingly, professional experience and expertise did not improve the
developers' ability to reason about APIs with blindspots across both
languages, with long-term professionals with many years of experience making
mistakes as often as relative novices. Finally, personality traits did not
significantly affect the Python developers' ability to reason about APIs with
blindspots, but less extraverted and more open developers were better at
reasoning about Java APIs with blindspots. Overall, our findings suggest that
blindspots in APIs are a serious problem across languages, and that
experience and education alone do not overcome that problem, suggesting that
tools are needed to help developers recognize blindspots in APIs as they
write code that uses those APIs.},
fundedBy = {NSF CCF-1453474, NSF CNS-1513055, NSF CNS-1513457, NSF CNS-1513572, NSF CCF-1564162},
}