Choosing gaze tracking and estimation datasets

The short answer

The right gaze dataset is the one whose recording conditions and labels most closely match how your system will be used. If your product relies on an ordinary webcam at desk distance, a dataset recorded with head-mounted eye trackers or in a controlled lab with fixed head position will transfer poorly, however large it is. Match the camera type, viewing distance, head movement, lighting and label format to your deployment first, then check licensing and consent terms before any technical comparison.

One limit applies to readers coming from the webcam audio side: gaze datasets are built around images or video of the eyes and face. Unless a dataset's documentation explicitly says it includes synchronized audio, assume it does not, and plan to source speech data separately.

Gaze estimation versus gaze tracking

The two terms are often used interchangeably, but datasets usually lean toward one task. Gaze estimation generally means predicting where a person is looking from a single image or short clip, often expressed as a 3D gaze direction or a point on a screen. Gaze tracking usually implies following gaze continuously over time, which makes frame rate, temporal labels and smooth sequences more important.

If you need per-frame predictions for something like attention detection, an estimation-oriented dataset of still images may be enough. If you need to follow eye movements across a session, for example for gaze-based interaction or accessibility input, look for video sequences with consistent timing and labels on every frame.

Criteria that actually change the decision

Criterion | What to check | Why it matters

Capture hardware | Consumer webcam, phone camera, infrared tracker or lab camera | Models trained on one sensor type often degrade on another

Viewing distance and setup | Desk, handheld, wall display or head-mounted | Eye size in pixels and head pose range differ sharply between setups

Head pose range | Frontal only or wide rotation | Narrow pose coverage leads to failures when users turn their heads

Lighting | Controlled lab light or varied real-world conditions | Real rooms have backlight, glare and uneven light that lab data may lack

Label type | 2D screen point, 3D gaze vector, eye landmarks, fixation events | Your model's output must match what the labels describe

Calibration data | Per-person calibration samples included or not | Needed if you plan personalized or few-shot calibration

Participant diversity | Range of ages, skin tones, eye shapes, glasses use | Gaps here produce uneven accuracy across users

License and consent | Research-only or commercial use; consent scope for biometric data | Determines whether you can use it at all for your project

Matching a dataset to webcam use

For webcam-based systems, prioritize datasets recorded with consumer-grade cameras at typical laptop or monitor distance, with natural head movement and everyday lighting. These conditions produce low-resolution eye regions, motion blur and inconsistent illumination, which a model must learn to handle. Lab datasets with high-resolution cameras and chin rests can still be useful for pretraining or for studying eye appearance, but expect to fine-tune on data closer to your target setup.

Also check whether the dataset reports the screen geometry and camera position relative to the screen. Without that information, converting a predicted gaze direction into a screen location is guesswork.

Accessibility and interaction considerations

If the goal is gaze interaction for accessibility, such as selecting on-screen targets without a mouse, accuracy near the screen edges and stability over long sessions matter more than average error on frontal faces. Look for data that includes varied head positions, users wearing glasses, and ideally people with the motor or postural characteristics of your intended users. Most general-purpose datasets were not designed with these groups in mind, so this is a common gap to plan for with your own consented data collection.

Where audio fits, and where it does not

Some interaction systems combine gaze with speech, for example looking at a target and saying a command. A gaze dataset will rarely support that combination on its own. If you need both signals, check documentation for synchronized audio tracks, the microphone used and its position, and whether recordings were made in quiet or noisy rooms. If those details are absent, treat the dataset as video-only and pair it with a separate speech corpus recorded through a webcam or similar built-in microphone, keeping in mind that the two will not be time-aligned for the same speakers.

A practical selection checklist

• Write down your deployment conditions: camera, distance, lighting, expected head movement and output format.
• Shortlist datasets whose documentation describes similar capture conditions.
• Confirm the label type matches your model's output.
• Read the license and consent terms, especially for commercial use and biometric data.
• Check participant diversity and note groups that are missing.
• Confirm whether audio is included if your system uses speech.
• Plan a small validation set recorded in your own target conditions to measure real-world performance.

Evidence gaps in this guide

No verified specifications, participant counts, accuracy figures or license details for individual datasets were available for this article, so it does not rank or name specific datasets. Published accuracy numbers are also frequently reported under different protocols, so they should not be compared directly unless the evaluation conditions match. Before committing to a dataset, read its official documentation and license, and validate any model on data recorded in your own conditions.