Evaluating and grading students’ work is a difficult and time consuming task. The general-purpose nature of large language models (LLMs), along with their vast knowledge across a wide range of domains, position them as a strong candidate for automatic assessment of free-text answers. However, there are various limitations pertaining to the reliability of LLMs as evaluators and fairness issues that arise from LLM-based automated assessment. This talk will discuss several factors that need to be considered when deciding whether and how to utilize LLMs for this task.