Software quality metrics in the automatic evaluation of Python introductory programming

Métricas de qualidade de software na avaliação automática da programação introdutória Python


  • Andrew Nunes de Barros Reis
  • Silvério Sirotheau Corrêa Neto
  • Eloi Luiz Favero
  • João Carlos Alves dos Santos
  • Pollianni Leão da Silva
  • Thiago Rafael da Silva Moura



automatic assessment, feedback, refactoring, code


Numerous virtual environments with automatic program evaluation have emerged to assist the teaching-learning process, allowing timely feedback. In a review of these environments, we find few studies that focus on an approach centered on refactoring: where students are strongly encouraged to refactor, improving the submitted code to also meet quality criteria. In a traditional environment, the student submits the answer and if it is dynamically correct, he goes to the next question. In this work, we propose a complementary approach based on software engineering metrics, which allow a finer evaluation of the code where the programmer, after having his dynamically correct answer, is invited and encouraged to refactor his solution towards an optimal code that also meets the software quality metrics. The work is based on source code in the Python language and shows which software quality metrics can be used with the purpose of encouraging students to refactor their code in programming fundamentals disciplines.


Ala-Mutka, K. M. (2005). A survey of automated assessment approaches for programming assignments. Computer science education, 15(2), 83-102.

Breuker, D., Brunekreef, J., Derriks, J., and Aicha, A. N. (2009). Reliability of software metrics tools. Software Process and Product Measurement, 10.

Barros, J. P. (2010, October). Assessment and grading for CS1: Towards a complete toolbox of criteria and techniques. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research (pp. 106-111).

Cardell-Oliver, R. (2011, January). How can software metrics help novice programmers?. In Proceedings of the Thirteenth Australasian Computing Education Conference-Volume 114 (pp. 55-62).

Fleming, W. H., Redish, K. A., and Smyth, W. F. (1988). Comparison of manual and automated marking of student programs. Information and Software Technology, 30(9), 547-552.

Fitzsimmons, A., and Love, T. (1978). A review and evaluation of software science. ACM Computing Surveys (CSUR), 10(1), 3-18.

Gordillo, A. (2019). Effect of an instructor-centered tool for automatic assessment of programming assignments on students’ perceptions and performance. Sustainability, 11(20), 5568.

Halstead, M. H. (1977). Elements of Software Science (Operating and programming systems series). Elsevier Science Inc. page(s): 128.

Ismail, M. H. (2021). A Critical Review on Recent Proposed Automated Programming Assessment Tool. Turkish Journal of Computer and Mathematics Education (TURCOMAT), 12(3), 884-894.

Koyya, P., Lee, Y., and Yang, J. (2013). Feedback for programming assignments using software-metrics and reference code. International Scholarly Research Notices, 2013.

Lajis, A., Baharudin, S. A., Ab Kadir, D., Ralim, N. M., Nasir, H. M., and Aziz, N. A. (2018). A review of techniques in automatic programming assessment for practical skill test. Journal of Telecommunication, Electronic and Computer Engineering (JTEC), 10(2-5), 109-113.

Lappalainen, V., Lakanen, A. J., and Högmander, H. (2017). Towards computer-based exams in CS1. In CSEDU 2017: Proceedings of the 9th International Conference on Computer Supported Education. Vol. 2, ISBN 978-989-758-240-0. SCITEPRESS Science And Technology Publications.

Maguire, P., Maguire, R., and Kelly, R. (2017). Using automatic machine assessment to teach computer programming. Computer Science Education, 27(3-4), 197-214.

Mekterović, I., Brkić, L., Milašinović, B., and Baranović, M. (2020). Building a comprehensive automated programming assessment system. IEEE Access, 8, 81154-81172.

Mengel, S. A., and Yerramilli, V. (1999, March). A case study of the static analysis of the quality of novice student programs. In The proceedings of the thirtieth SIGCSE technical symposium on Computer science education (pp. 78-82).

Matthíasdóttir, Á., and Arnalds, H. (2016, June). E-assessment: students' point of view. In Proceedings of the 17th international conference on computer systems and technologies 2016 (pp. 369-374).

Leach, R. J. (1995). Using metrics to evaluate student programs. ACM SIGCSE Bulletin, 27(2), 41-43.

McCabe, T. J. (1976). A complexity measure. IEEE Transactions on software Engineering, (4), 308-320.

Navlakha, J. K. (1987). A survey of system complexity metrics. The Computer Journal, 30(3), 233-238.

Ivan Pribela, Gordana Rakić, Zoran Budimac. “First Experiences in Using So ware Metrics in Automated Assessment”. 15thInternational Multiconference on Information Society (IS), Collaboration, Software And Services In Information Society (CSS). At: Ljubljana, Slovenia.Volume: A. 2012

Luxton-Reilly, A., Albluwi, I., Becker, B. A., Giannakos, M., Kumar, A. N., Ott, L., ... and Szabo, C. (2018, July). Introductory programming: a systematic literature review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (pp. 55-106).

Souza, D. M., Felizardo, K. R., and Barbosa, E. F. (2016, April). A systematic literature review of assessment tools for programming assignments. In 2016 IEEE 29th International Conference on Software Engineering Education and Training (CSEET) (pp. 147-156). IEEE.

Truong, N., Bancroft, P., and Roe, P. (2002, December). ELP-a web envrionment for learning to program. In ASCILITE (pp. 661-670).

Truong, N., Roe, P., and Bancroft, P. (2004, January). Static analysis of students' Java programs. In Proceedings of the Sixth Australasian Conference on Computing Education-Volume 30 (pp. 317-325).

Venables, A., and Haywood, L. (2003, January). Programming students NEED instant feedback!. In Proceedings of the fifth Australasian conference on Computing education-Volume 20 (pp. 267-272).

Walker, H. M. (2011, April). A lab-based approach for introductory computing that emphasizes collaboration. In Computer Science Education Research Conference (pp. 21-31).

Zimmerman, D. M., Kiniry, J. R., and Fairmichael, F. (2011, May). Toward instant gradeification. In 2011 24th IEEE-CS Conference on Software Engineering Education and Training (CSEE&T) (pp. 406-410). IEEE.





Artigos originais