Remote Evaluation Systems Pilot

Description and results of the Remote Assessment Systems Pilot - SAR

Registration is now closed.

Motivated by the need to find a solution for student assessment at a time of emergency remote teaching, the FCT's FCCN Unit is challenging the higher education community to take part in a study to help find the preferred solution.

For this purpose the following platforms will be available: ProctorExam, ExamNet, TestWe or Respondus. The teacher has the possibility to perform a set of tests in order to evaluate these Remote Assessment Systems (RAS), and thus identify which best meets the necessary requirements.

Pilot Results

Through the participation of various higher education institutions, with the coordination of a team from the FCT's National Scientific Computing Unit (Unidade FCCN), it was possible to take a learning path as to what should be done for the future.

The first and main conclusion drawn is that the remote assessment solutions existing in the market present several implementation difficulties, taking into account the reality and the objectives of Portuguese Higher Education. Besides their high cost (the most advanced platforms in fraud prevention have values that are around 10€ per student per exam), most of the teachers who participated in the study indicate that these solutions are not very effective in fraud prevention.

On the other hand, with regard to Data Protection - one of the main issues associated with this topic - the study concluded that the existing remote assessment solutions do not convey the level of trust, in terms of data management, that is required of Portuguese teachers and higher education institutions. One of the main lessons learned from this project relates to Colibri's complementary role. As some existing platforms do not allow real-time video monitoring, this service from FCCN, which is based on Zoom technology, was used in conjunction by some of the member institutions.

According to the project's conclusions, these combined solutions seem to be sufficient for most of the uses. The integration of Colibri with other tools is well accepted by the teachers, even though it is necessary to find an integrated solution adjusted to the needs of the national HEIs that are not solved by the commercial tools. In this sense, the integration of Colibri in this study allowed us to draw some conclusions about which functionalities are critical for a future assessment platform to be successful.

The possibility of two-way communication (teacher/student and teacher/class), the cost compatible with the financial reality of the HEIs, the recording and analysis of data (audio and video) in a controlled place and with clarified legislation were pointed out as some of the strategic and necessary steps, as well as the eventual integration of the main functions into a single application that can run on various devices.

Best Practices for Remote Assessment Download

Remote Evaluation Systems Pilot Download

Pioneering platform in evaluation tools for e-learning systems.

Integrated into e-learning LMS platforms (Moodle, Blackboard, etc.);

It includes several steps of verification of the student's identity and environment;

Use of own browser for screen lock, applications and keyboard shortcuts;

AI recording and detection of suspicious behaviour.

Platform managed by the University of Aveiro

Fully customisable platform to suit all remote assessment requirements.

Identification through Citizen's Card;
360º vision using a smartphone;
Screen sharing.

Platform managed by the University of Lisbon

FAQS

Who can use this Pilot?
All higher education teachers who belong to RCTS. Check if your entity is a RCTS member in the "RCTS Entity List".

I want to do 5 exams for my 10 classes, can I?
The aim of this Pilot is to collect data from different solutions and different teachers. On platforms with a limited number of licenses we suggest you do the assessments with a reasonable number of students, but allow others to do so as well.

Some of the platforms are paid, do I pay anything?
No. FCT and the partners ensure the access to the platforms free of charge by the teachers. In exchange, they expect the completion of a set of forms, so that they can objectively perform their mission of collecting data to support the subsequent decision to scale, or not, one or more of these services to the entire academic community.

Do I need to involve my entity in the pilot?
The pilot is intended to be direct to lecturers to facilitate the process and avoid delays, but this does not justify transposing internal rules in your institution. Whenever you consider it necessary validate internally with distance learning services before requesting access to the pilot.

How is access to data controlled?
All platforms have their own terms and conditions, and all are committed to complying with the GDPR.

Which platforms are part of the pilot?
ProctorExam; ExamNet; TestWe; Respondus.

Why are the partner organizations part of the pilot?
The FCT's FCCN Unit wants the search for solutions to be actively participated in by the community, so it took advantage of movements that were already taking place and that were available. There is no preference for these entities over others.

I am a teacher from a partner entity, do I have to use the platform managed by my entity?
No. You can request access to any of the other platforms in the pilot.

I am a student, I don't want to participate, am I obliged?
Teachers should always propose alternatives to remote assessment. Contact your teacher.

The exam was taking place and there was a network problem, what do I do now?
In case of failure during the exam, the student must contact the teacher as soon as possible, through the means indicated by him/her. The solution varies according to the failure.

Can I still apply for the SAR Pilot?

Registration for the SAR pilot is now closed. The teachers who tested the four products (ProctorExam, ExamNet, TestWe and Respondus) moved on to a second phase of the pilot to analyse the data obtained.