Evaluating usability in educational technology: A systematic review from the teaching of mathematics

Odiel Estrada Molina; Dieter Reynaldo Fuentes-Cancell; Alien García-Hernández

doi:10.31129/LUMAT.10.1.1686

Authors

Odiel Estrada Molina Universidad de las Ciencias Informáticas, Cuba https://orcid.org/0000-0002-0918-418X
Dieter Reynaldo Fuentes-Cancell Universidad de las Ciencias Informáticas, Cuba
Alien García-Hernández Universidad de las Ciencias Informáticas, Cuba

DOI:

https://doi.org/10.31129/LUMAT.10.1.1686

Keywords:

educational technology, teaching of mathematics, systematic review, usability

Abstract

The objective of this article is to determine which are the international standards and norms of usability that are used in educational technology applied to the teaching of mathematics at secondary and higher education levels, focused on two scenarios: (1) digital educational resources and virtual learning environments (VLEs), and (2) game-based learning. The PRISMA protocol was used and the Scopus and IEEE Xplorer databases, the Springer publishing House and the ACM Digital Library were used for the search strategy. Forty-seven primary studies were selected, emphasizing the use of the ISO/IEC 9241-11 standard. However, the isolated use of criteria to usability assessment without achieving engineering integration is reflected. Primary studies in the teaching of mathematics mainly use the ISO 9241-11:2018 and ISO/IEC 9126-1:2004 standards. Game-based learning scenario uses ISO 9241-11:2018 standards and procedures that guarantee, at least, the integration between efficiency, effectiveness, and ease of use. Digital educational resources and VLEs scenario uses ISO/IEC 9126-1:2004 and ISO/IEC 9241-11:2018 standards, and procedures that guarantee at least the integration between efficiency and ease of use; effectiveness and ease of use; ease of use and accessibility; and effectiveness, ease of use, accessibility, and efficiency.

Author Biographies

Dieter Reynaldo Fuentes-Cancell, Universidad de las Ciencias Informáticas, Cuba

Graduated in Sociology from the Universidad de Oriente, Cuba. Aspiring Master in Virtual Education from the University of Informatics Sciences. Professor and researcher at the Universidad de las Ciencias Informáticas, Cuba. His fundamental line of research is “ICT in Education”.

Alien García-Hernández, Universidad de las Ciencias Informáticas, Cuba

Doctor of Education from the University of Sevilla, Spain. Head of the computational intelligence department of the University of Informatics Science, Cuba (Universidad de las Ciencias Informáticas de Cuba).

References

Abuhlfaia, K., & de Quincey, E. (2019). Evaluating the usability of an e-learning platform within higher education from a student perspective. ACM International Conference Proceeding Series, 1–7. https://doi.org/10.1145/3371647.3371661

Adnan, M., Abdullah, J. M., Ibharim, L. F. M., Hoe, T. W., Janan, D., Abdullah, N., Idris, N., Wahab, A. S. A., Othman, A. N., Hashim, M. E. A., Said, N. M., Adnan, R., Yahaya, S., Amin, N., Noh, M. A. M., Sufa’at, N. I., Abdullah, R., Yusof, Y., Reduaan, Z. A. M., … Baharudin, N. F. A. (2019). Expanding opportunities for science, technology, engineering and mathematics subjects teaching and learning: Connecting through comics. Malaysian Journal of Medical Sciences, 26(4), 127–133. https://doi.org/10.21315/mjms2019.26.4.15

Alalwan, N., Cheng, L., Al-Samarraie, H., Yousef, R., Ibrahim Alzahrani, A., & Sarsam, S. M. (2020). Challenges and Prospects of Virtual Reality and Augmented Reality Utilization among Primary School Teachers: A Developing Country Perspective. Studies in Educational Evaluation, 66, 100876. https://doi.org/10.1016/J.STUEDUC.2020.100876

Almenara, J. C., & Gimeno, A. M. (2019). Las TIC y la formación inicial de los docentes. Modelos y competencias digitales. Profesorado, Revista de Currículum y Formación Del Profesorado, 23(3), 247–268. https://doi.org/10.30827/PROFESORADO.V23I3.9421

Alomari, H. W., Ramasamy, V., Kiper, J. D., & Potvin, G. (2020). A User Interface (UI) and User eXperience (UX) evaluation framework for cyberlearning environments in computer science and software engineering education. Heliyon, 6(5). https://doi.org/10.1016/J.HELIYON.2020.E03917

Alshira, M., Al-omari, M., & Igried, B. (2021). Usability Evaluation of Learning Management Systems ( LMS ) based on User Experience Turkish Journal of Computer and Mathematics Education Research Article. Turkish Journal of Computer and Mathematics Education (TURCOMAT), 12(11), 6431–6441. https://www.turcomat.org/index.php/turkbilmat/article/view/7031

Ávila-Soto, M., Valderrama-Bahamóndez, E., & Schmidt, A. (2017). TanMath: A tangible math application to support children with visual impairment to learn basic arithmetic. ACM International Conference Proceeding Series, Part F128530, 244–245. https://doi.org/10.1145/3056540.3064964

Awang, K., Shamsuddin, S. N. W., Ismail, I., Rawi, N. A., & Amin, M. M. (2019). The usability analysis of using augmented reality for linus students. Indonesian Journal of Electrical Engineering and Computer Science, 13(1), 58–64. https://doi.org/10.11591/IJEECS.V13.I1.PP58-64

Bateman, A., Zhao, O. K., Bajcsy, A. V., Jennings, M. C., Toth, B. N., Cohen, A. J., Horton, E. L., Khattar, A., Kuo, R. S., Lee, F. A., Lim, M. K., Migasiuk, L. W., Renganathan, R., Zhang, A., & Oliveira, M. A. (2018). A user-centered design and analysis of an electrostatic haptic touchscreen system for students with visual impairments. International Journal of Human-Computer Studies, 109, 102–111. https://doi.org/10.1016/J.IJHCS.2017.09.004

Beswick, K., & Fraser, S. (2019). Developing mathematics teachers’ 21st century competence for teaching in STEM contexts. ZDM - Mathematics Education, 51(6), 955–965. https://doi.org/10.1007/s11858-019-01084-2

Bozkurt, G., & Ruthven, K. (2017). Classroom-based professional expertise: a mathematics teacher’s practice with technology. Educational Studies in Mathematics, 94(3), 309–328. https://doi.org/10.1007/s10649-016-9732-5

Cáceres, S. V., & Pow-Sang, J. A. (2019). A systematic mapping review of usability evaluation methods for educational applications on mobile devices. Applications in Software Engineering - Proceedings of the 7th International Conference on Software Process Improvement, CIMPS 2018, 59–68. https://doi.org/10.1109/CIMPS.2018.8625629

Calderon, C. A., Guajala, M., Lanchi, J., Barba-Guaman, L., Bermeo, C., & Rivas-Echeverria, F. (2019). A machine vision system applied to the teaching of mathematics for blind or visually impaired children. IEEE ICA-ACCA 2018 - IEEE International Conference on Automation/23rd Congress of the Chilean Association of Automatic Control: Towards an Industry 4.0 - Proceedings. https://doi.org/10.1109/ICA-ACCA.2018.8609818

Casano, J., Tee, H., Agapito, J., Arroyo, Í., & Rodrigo, M. M. T. (2016). Migration and evaluation of a framework for developing embodied cognition learning games. Proceedings of the 3rd Asia-Europe Symposium on Simulation and Serious Gaming - 15th ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Applications in Industry, VRCAI 2016, 199–203. https://doi.org/10.1145/3014033.3014035

Chang, C., Chin, Y. L., & Chang, C. K. (2017). Experimental Functionality Development for Scratch Mathematical and Statistics Extensions. Proceedings - 2016 International Computer Symposium, ICS 2016, 640–644. https://doi.org/10.1109/ICS.2016.0131

Chen, Q. (2018). An application of online exam in discrete mathematics course. ACM International Conference Proceeding Series, 91–95. https://doi.org/10.1145/3210713.3210734

Cocunubo-Suárez, J. I., Parra-Valencia, J. A., Otálora-Luna, J. E., Cocunubo-Suárez, J. I., Parra-Valencia, J. A., & Otálora-Luna, J. E. (2018). Evaluation of Virtual Teaching-Learning Environments based on usability standards. TecnoLógicas, 21(41), 135–147.

dos Reis, T. S., Miranda, R. C., & Filho, A. D. P. (2019). Usability of software of teaching aid in the teaching process mathematics learning. International Journal of Information and Education Technology, 9(5), 384–389. https://doi.org/10.18178/IJIET.2019.9.5.1232

Drigas, A., & Pappas, M. (2015). A Review of Mobile Learning Applications for Mathematics. International Journal of Interactive Mobile Technologies (IJIM), 9(3), 18–23. https://online-journals.org/index.php/i-jim/article/view/4420

Fernández Márquez, E., Ordóñez Olmedo, E., & Morales Cevallos Jesús López Belmonte, B. (2019). La competencia digital en la docencia universitaria. Ediciones OCTAEDRO, S.L. https://octaedro.com/wp-content/uploads/2019/10/16154-La-competencia-digital-en-la-docencia-universitaria.pdf

Fonseca, D., García-Peñalvo, F. J., & Camba, J. D. (2020). New methods and technologies for enhancing usability and accessibility of educational data. Universal Access in the Information Society 2020 20:3, 20(3), 421–427. https://doi.org/10.1007/S10209-020-00765-0

Freina, L., & Ott, M. (2015). A Literature Review on Immersive Virtual Reality in Education: State Of The Art and Perspectives. Scientific Conference Elearning and Software for Education, 133–150. http://www.google.com/patents/US3050870

García, J., García-Peñalvo, F. J., Therón, R., & Ordoñez De Pablos, P. (2011). Usability Evaluation of a Visual Modelling Tool for OWL Ontologies. Journal of Universal Computer Science, 17(9), 1299–1313. http://www.jucs.org/jucs_17_9/usability_evaluation_of_a/jucs_17_09_1299_1313_garcia.pdf

Hadjerrouit, S., & Gautestad, H. H. (2019). Evaluating the Usefulness of the Visualization Tool SimReal+ for Learning Mathematics: A Case Study at the Undergraduate Level. Learning Technologies for Transforming Large-Scale Teaching, Learning, and Assessment, 71–89. https://doi.org/10.1007/978-3-030-15130-0_5

Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamification work? - A literature review of empirical studies on gamification. Proceedings of the Annual Hawaii International Conference on System Sciences, 3025–3034. https://doi.org/10.1109/HICSS.2014.377

Hariyanto, D., Triyono, M. B., & Köhler, T. (2020). Usability evaluation of personalized adaptive e-learning system using USE questionnaire. Knowledge Management and E-Learning, 12(1), 85–105. https://doi.org/10.34105/J.KMEL.2020.12.005

Holmes, M. H. (2006). Integrating the Learning of Mathematics and Science Using Interactive Teaching and Learning Strategies. Journal of Science Education and Technology 2006 15:3, 15(3), 247–256. https://doi.org/10.1007/S10956-006-9011-9

Hooshyar, D., Pedaste, M., Saks, K., Leijen, Ä., Bardone, E., & Wang, M. (2020). Open learner models in supporting self-regulated learning in higher education: A systematic literature review. Computers & Education, 154, 103878. https://doi.org/10.1016/J.COMPEDU.2020.103878

Hooshyar, D., Yousefi, M., & Lim, H. (2019). A systematic review of data-driven approaches in player modeling of educational games. In Artificial Intelligence Review (Vol. 52, Issue 3, pp. 1997–2017). Springer. https://doi.org/10.1007/s10462-017-9609-8

Ibarra, M. J., Mamani, Y., Ataucusi, P. E., Palomino, C., & Ibañez, V. (2017). Raising students motivation for math learning using computer animation approach. 2017 43rd Latin American Computer Conference, CLEI 2017, 2017-January, 1–9. https://doi.org/10.1109/CLEI.2017.8226436

Ibarra, M. J., Soto, W., Ataucusi, P., & Ataucusi, E. (2016). MathFraction: Educational serious game for students motivation for math learning. Proceedings - 2016 11th Latin American Conference on Learning Objects and Technology, LACLO 2016. https://doi.org/10.1109/LACLO.2016.7751777

İbili, E., Çat, M., Resnyansky, D., Şahin, S., & Billinghurst, M. (2019). An assessment of geometry teaching supported with augmented reality teaching materials to enhance students’ 3D geometry thinking skills. International Journal of Mathematical Education in Science and Technology , 51(2), 224–246. https://doi.org/10.1080/0020739X.2019.1583382

Ishaq, K., Zin, N. A. M., Rosdi, F., Abid, A., & Farooq, U. (2019, November 1). Effectiveness of Literacy Numeracy Drive (LND): A Students’ Perspective. 3rd International Conference on Innovative Computing, ICIC 2019. https://doi.org/10.1109/ICIC48496.2019.8966738

Issa, L., & Jusoh, S. (2019). Usability evaluation on gamified e-learning platforms. ACM International Conference Proceeding Series. https://doi.org/10.1145/3368691.3368702

Kang, Y. Y., Wang, M. J. J., & Lin, R. (2009). Usability evaluation of E-books. Displays, 30(2), 49–52. https://doi.org/10.1016/J.DISPLA.2008.12.002

Klingenberg, O. G., Holkesvik, A. H., & Augestad, L. B. (2019). Digital learning in mathematics for students with severe visual impairment: A systematic review: British Journal of Visual Impairment, 38(1), 38–57. https://doi.org/10.1177/0264619619876975

Kumar, B. A., & Mohite, P. (2018). Usability of mobile learning applications: a systematic literature review. Journal of Computers in Education, 5(1), 1–17. https://doi.org/10.1007/s40692-017-0093-6

Law, E. L. C., & Heintz, M. (2021). Augmented reality applications for K-12 education: A systematic review from the usability and user experience perspective. International Journal of Child-Computer Interaction, 30, 100321. https://doi.org/10.1016/J.IJCCI.2021.100321

Lopes, J. B., & Costa, C. (2018). Digital Resources in Science, Mathematics and Technology Teaching – How to Convert Them into Tools to Learn. In Technology and Innovation in Learning, Teaching and Education. TECH-EDU 2018 (Vol. 993, pp. 243–255). Springer, Cham. https://doi.org/10.1007/978-3-030-20954-4_18

Martin-Gonzalez, A., Chi-Poot, A., & Uc-Cetina, V. (2016). Usability evaluation of an augmented reality system for teaching Euclidean vectors. Innovations in Education and Teaching International, 53(6), 627–636. https://doi.org/10.1080/14703297.2015.1108856

Martinho, D., Carneiro, J., Corchado, J. M., & Marreiros, G. (2020). A systematic review of gamification techniques applied to elderly care. Artificial Intelligence Review, 53(7), 4863–4901. https://doi.org/10.1007/S10462-020-09809-6

Missen, M. M. S., Javed, A., Asmat, H., Nosheen, M., Coustaty, M., Salamat, N., & Prasath, V. B. S. (2019). Systematic review and usability evaluation of writing mobile apps for children. New Review of Hypermedia and Multimedia, 25(3), 137–160. https://doi.org/10.1080/13614568.2019.1677787

Mohd-Khir, N. H. B., & Ismail, M. (2019). Review on gamification in children computer interaction (CCI) for persona modelling. Bulletin of Electrical Engineering and Informatics, 8(4), 1411–1417. https://doi.org/10.11591/eei.v8i4.1622

Molina, A. I., Arroyo, Y., Lacave, C., & Redondo, M. A. (2018). Learn-CIAN: A visual language for the modelling of group learning processes. British Journal of Educational Technology, 49(6), 1096–1112. https://doi.org/10.1111/BJET.12680

Muñoz, M., & Peralta, M. (2020). Situación actual sobre la implementación del perfil básico ISO/IEC 29110 en EMP: una revisión sistemática de la literatura. RISTI - Revista Ibérica de Sistemas e Tecnologias de Informação, 36, 1–14. https://doi.org/10.17013/RISTI.36.1-14

Na, J., & Liu, Y. (2019). A quantitative revision method to improve usability of self and peer assessment in MOOCs. ACM International Conference Proceeding Series. https://doi.org/10.1145/3321408.3322846

Nielsen, J., & Tahir, M. (2001). Homepage Usability: 50 Websites Deconstructed. New Riders Publishing. https://www.todostuslibros.com/libros/homepage-usability_978-0-7357-1102-0

Novak, E., Razzouk, R., & Johnson, T. E. (2012). The educational use of social annotation tools in higher education: A literature review. The Internet and Higher Education, 15(1), 39–49. https://doi.org/10.1016/J.IHEDUC.2011.09.002

Odun-Ayo, I., Goddy-Worlu, R., Yahaya, J., & Geteloma, V. (2019). A systematic mapping study of cloud policy languages and programming models. In Journal of King Saud University - Computer and Information Sciences, 33(7), 761–768. Elsevier. https://doi.org/10.1016/j.jksuci.2019.05.003

Pallarès, M., Óscar, P., Bartoll, C., López, R., Ismael, M., & Fayos, C. (2018). La escuela que llega. Tendencias y nuevos enfoques metodológicos (Primera edición). Octaedro. www.octaedro.com

Pensabe-Rodriguez, A., Lopez-Dominguez, E., Hernandez-Velazquez, Y., Dominguez-Isidro, S., & De-la-Calleja, J. (2020). Context-aware mobile learning system: Usability assessment based on a field study. Telematics and Informatics, 48, 101346. https://doi.org/10.1016/J.TELE.2020.101346

Pérez, Y. F., Corona, C. C., & Estrada, A. F. (2019). Fuzzy Cognitive Maps for Evaluating Software Usability. Studies in Fuzziness and Soft Computing, 377, 141–155. https://doi.org/10.1007/978-3-030-10463-4_8

Pujiastuti, H., Utami, R. R., & Haryadi, R. (2020). The development of interactive mathematics learning media based on local wisdom and 21st century skills: social arithmetic concept. Journal of Physics: Conference Series, 1521(3), 032019. https://doi.org/10.1088/1742-6596/1521/3/032019

Quiñones, D., & Rusu, C. (2017). How to develop usability heuristics: A systematic literature review. Computer Standards & Interfaces, 53, 89–122. https://doi.org/10.1016/J.CSI.2017.03.009

Radianti, J., Majchrzak, T. A., Fromm, J., & Wohlgenannt, I. (2020). A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Computers & Education, 147, 103778. https://doi.org/10.1016/J.COMPEDU.2019.103778

Ramanayaka, K. H., Chen, X., & Shi, B. (2018). UNSCALE: A Fuzzy-based Multi-criteria Usability Evaluation Framework for Measuring and Evaluating Library Websites. IETE Technical Review, 36(4), 412–431. https://doi.org/10.1080/02564602.2018.1498032

Ramírez-Vega, A., Iniesto, F., & Rodrigo, C. (2017). Raising awareness of the accessibility challenges in mathematcs MOOCs. ACM International Conference Proceeding Series, Part F132203. https://doi.org/10.1145/3144826.3145435

Rauf, I., Troubitsyna, E., & Porres, I. (2019). A systematic mapping study of API usability evaluation methods. Computer Science Review, 33, 49–68. https://doi.org/10.1016/J.COSREV.2019.05.001

Rumanová, L., & Drábeková, J. (2017). View of teaching the mathematics of production possibilities curve. ACM International Conference Proceeding Series, 213–217. https://doi.org/10.1145/3175536.3175559

Salas, J., Chang, A., Montalvo, L., Núñez, A., Vilcapoma, M., Moquillaza, A., Murillo, B., & Paz, F. (2019). Guidelines to Evaluate the Usability and User Experience of Learning Support Platforms: A Systematic Review. Communications in Computer and Information Science, 1114 CCIS, 238–254. https://doi.org/10.1007/978-3-030-37386-3_18

Sánchez, M., Garrote, L., & Escolnao, J. (2019). Aprendizaje y enseñanza de las matemáticas. © Editorial Síntesis, S. A. https://www.sintesis.com/data/indices/9788491712657.pdf

Sarkar, P., Kadam, K., & Pillai, J. S. (2019). Collaborative Approaches to Problem-Solving on Lines and Angles Using Augmented Reality. 2019 IEEE Tenth International Conference on Technology for Education (T4E), 193–200. https://doi.org/10.1109/T4E.2019.00-24

Singh, M., & Jha, R. C. (2019). Object-Oriented Usability Indices for Multi-Objective Demand Side Management Using Teaching-Learning Based Optimization. Energies 2019, 12(3), 370. https://doi.org/10.3390/EN12030370

Toda, A. M., Do Carmo, R. S., Campos, V., Da Silva, A. L., & Brancher, J. D. (2015). Evaluation of SiGMa, an empiric study with Math teachers. Proceedings - Frontiers in Education Conference, FIE, 2015. https://doi.org/10.1109/FIE.2015.7344073

Tomaschko, M., & Hohenwarter, M. (2017). Integrating mobile and sensory technologies in mathematics education. ACM International Conference Proceeding Series, 39–48. https://doi.org/10.1145/3151848.3151866

Torres-Carrion, P. V., Gonzalez-Gonzalez, C. S., Aciar, S., & Rodriguez-Morales, G. (2018). Methodology for systematic literature review applied to engineering and education. IEEE Global Engineering Education Conference, EDUCON, 2018-April, 1364–1373. https://doi.org/10.1109/EDUCON.2018.8363388

Tsouccas, L., & Meletiou-Mavrotheris, M. (2017). Enhancing the Technological, Pedagogical and Content Knowledge (TPACK) of in -service primary teachers use of tablet use technologies. ACM International Conference Proceeding Series. https://doi.org/10.1145/3136907.3136951

Urrútia, G., & Bonfill, X. (2010). PRISMA declaration: A proposal to improve the publication of systematic reviews and meta-analyses. Medicina Clinica, 135(11), 507–511. https://doi.org/10.1016/j.medcli.2010.01.015

Varsaluoma, J., Väätäjä, H., & Walsh, T. (2016). Exploring motivational aspects and user experience of mobile mathematics learning service in South Africa. AcademicMindtrek 2016 - Proceedings of the 20th International Academic Mindtrek Conference, 159–168. https://doi.org/10.1145/2994310.2994369

Vieira, E. A. O., Silveira, A. C. da, & Martins, R. X. (2019). Heuristic evaluation on usability of educational games: A systematic review. Informatics in Education, 18(2), 427–442. https://doi.org/10.15388/infedu.2019.20

Vlachogianni, P., & Tselios, N. (2021). Perceived usability evaluation of educational technology using the System Usability Scale (SUS): A systematic review. Journal of Research on Technology in Education. https://doi.org/10.1080/15391523.2020.1867938

Wan-Sulaiman, W. N. A., & Mustafa, S. E. (2019). Usability Elements in Digital Textbook Development: A Systematic Review. Publishing Research Quarterly, 36(1), 74–101. https://doi.org/10.1007/s12109-019-09675-3

Wang, J., Antonenko, P., Celepkolu, M., Jimenez, Y., Fieldman, E., & Fieldman, A. (2018). Exploring Relationships Between Eye Tracking and Traditional Usability Testing Data. International Journal of Human–Computer Interaction, 35(6), 483–494. https://doi.org/10.1080/10447318.2018.1464776

Weichbroth, P. (2020). Usability of mobile applications: A systematic literature study. IEEE Access, 8, 55563–55577. https://doi.org/10.1109/ACCESS.2020.2981892

Weinerth, K., Koenig, V., Brunner, M., & Martin, R. (2014). Concept maps: A useful and usable tool for computer-based knowledge assessment? A literature review with a focus on usability. Computers & Education, 78, 201–209. https://doi.org/10.1016/j.compedu.2014.06.002

Yağmur, S., & Çakır, M. P. (2016). Usability Evaluation of a Dynamic Geometry Software Mobile Interface Through Eye Tracking. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 9753, 391–402. https://doi.org/10.1007/978-3-319-39483-1_36

Yáñez-Gómez, R., Cascado-Caballero, D., & Sevillano, J. L. (2017). Academic methods for usability evaluation of serious games: a systematic review. Multimedia Tools and Applications, 76(4), 5755–5784. https://doi.org/10.1007/s11042-016-3845-9

Yáñez-Gómez, R., Font, J. L., Cascado-Caballero, D., & Sevillano, J. L. (2019). Heuristic usability evaluation on games: a modular approach. Multimedia Tools and Applications, 78(4), 4937–4964. https://doi.org/10.1007/s11042-018-6593-1