The Big Picture
Training GUI-capable assistants on-device across many users and platforms improves performance even when raw data never leaves devices; but differences between platforms (phone vs browser vs desktop) cause the largest performance drops, and adaptive federated training methods recover the most accuracy.
ON THIS PAGE
Core Insights
A new benchmark simulates training GUI agents across phones, browsers and desktop computers while keeping data local. Adding more participating users consistently improves model accuracy, even when those users come from very different platforms. Differences between platforms (cross-platform heterogeneity) hurt performance far more than differences among devices or data sources within the same platform. adaptive federated training algorithms are the most robust choice for recovering accuracy in these mixed environments.
Test your agentsValidate against real scenarios
By the Numbers
1Coverage: 900+ mobile apps, 200+ websites, and 40+ desktop applications across three platforms (mobile, web, desktop).
2Scale: 6 datasets composed into 29 subsets, evaluated with 7 federated learning algorithms and 20+ base vision–language models.
3Heterogeneity scope: data collected from 5 mobile devices and 3 desktop operating systems; experiments show more users boost accuracy, while platform-level differences cause the biggest performance drop and adaptive optimizers (e.g., FedYogi) perform best.
Why It Matters
Engineers building on-device or privacy-friendly assistants should care because this shows you can improve a GUI agent by federated training across diverse users without centralizing private interaction logs. Technical leads deciding on training strategies should favor adaptive federated algorithms and recruit clients across platforms to get the most generalizable models. privacy-friendly
Key Figures
Fig 1: Figure 1: Overview of FedGUI. FedGUI provides a federated framework that coordinates a central server and heterogeneous clients across mobile, web, and desktop platforms to train a generalized cross-platform GUI agent. Hetero. is short for heterogeneity.
Fig 2: (a) Device IID
Fig 3: (a) Full IID
Fig 4: Figure 4: Results on FedGUI-OS , with evaluations conducted on Windows, Ubuntu, and macOS. Our Federated GUI agents demonstrate superior cross-OS generalization compared to strong VLMs like GPT-5.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreLimitations
The benchmark uses publicly available GUI datasets rather than private, real-world client logs, so interaction patterns may not fully match live deployments. Exact performance impacts (percent drops) vary by model and task and aren’t reported as a single global number, so validate on your specific workflow. The benchmark focuses on three major platforms and common desktop systems but may miss niche devices, custom OS builds, or private enterprise apps. publicly available GUI datasets
Full Analysis
FedGUI is a unified benchmark and experimental framework for training graphical user interface (GUI) agents in a privacy-conscious, distributed way: each client trains locally and only model updates are shared. The benchmark assembles six datasets from eight sources (29 total subsets) covering 900+ mobile apps, 200+ websites, and 40+ desktop applications. It explicitly models four kinds of real-world data mismatch — cross-platform (phone, web, desktop), cross-device (different phones), cross-source (different dataset origins), and cross-operating-system (Windows, macOS, Ubuntu) — and integrates seven federated training algorithms and 20+ vision–language base models. To keep client computation realistic, they adapt large models using low-rank adapters. cross-platform (phone, web, desktop) seven federated training algorithms
Not sure where to start?Get personalized recommendations
Credibility Assessment:
Authors have low h-indices and no clear institutional affiliations; arXiv preprint with no citations — fits emerging/limited-info category.