Ahmad Osman, founder of Osmantic and an early advocate for running AI models locally on personal computers, workstations, or dedicated hardware, led a two-part workshop on local large language models (LLMs) and workstation agents at the AI Engineer World’s Fair. Osmantic develops open-source software to deploy and operate local AI systems.
Open-source models are closing the gap
A major theme at AIEWF was that open-source LLMs are increasingly credible alternatives to proprietary frontier models. Because most local AI deployments rely on open models, this trend supports Osman’s long-standing position. As he put it, “the gap between open-source models and closed-frontier models keeps shrinking.” On his website Open Source AI Must Win he argues that “the ability to study, build, repair, deploy, audit, adapt, teach, preserve, and run intelligence systems without asking permission is of existential importance.”
Making local AI tangible: the workshops
Demand for Osman’s two-part workshop exceeded capacity. He presented a demo site and a hardware arena where attendees could compare systems such as DGX Spark and AMD Strix Halo machines, or pit local setups against a frontier cloud model to observe performance, output quality, speed and latency firsthand. The demonstrator software is open-source and available on GitHub, and the second session went deeper into setting up and operating the full system.
A model is only one component
Osman emphasized that running a model locally is not the whole story: hosted services like ChatGPT or Claude Code bundle models with search, tools, agent orchestration and other infrastructure. He shared an example in which a friend bought an RTX 5090 to run Qwen 3.5 locally and tried to have Claude Code change the GPU’s RGB lighting; it failed locally but worked via the hosted service. The issue was lack of local access to a search endpoint and up-to-date documentation. Once the local system could reach a search endpoint, the task completed. Osman’s point: the end-to-end layer — chat interfaces, document ingestion, agents, harnesses and search — has been missing in many local AI setups, and Osmantic’s open-source deployment aims to provide that complete experience.
Broad interest: students, enthusiasts and enterprises
Workshop attendees included students choosing their first AI-capable machine, enthusiasts, and enterprise executives concerned with model routing, private infrastructure and control of corporate data. Questions ranged from enterprise model routing, data collection and tracing, agent sandboxing and latency, to personal curiosity — Osman disclosed he owns 22 RTX 3090 GPUs at home. An Intel executive asked about running the software on Windows to improve user experience, illustrating the diverse, cross-sector interest in controllable local deployments.
You may not need to buy a GPU
Whether developers must buy a GPU depends on model size. Osman noted a four-bit Qwen model can run on a MacBook, while a very large frontier-class open model might require multiple RTX Pro 6000 GPUs. The larger trend is increased model efficiency: modern phones can now run models that outperformed cloud systems from just a couple years ago, demonstrating rapid gains in efficiency.
Models and hardware are improving together
Osman attributes rapid progress to model and software efficiency gains: architecture improvements and many small optimizations compound. The open-source ecosystem often reproduces frontier capabilities in more efficient forms. Today, models with tens of billions of parameters can deliver performance that previously required much larger systems, and some can run on a 2020-era RTX 3090. The field is young and evolving, but systems are expected to continue improving.
The rise of hybrid and sovereign AI
Osman expects more applications to combine local and cloud AI. Enterprises are increasingly aware that providers can change quality, pricing, access or policies, creating an incentive to use dedicated, secure compute — not necessarily on-premises, but in colocation or other controlled environments. This approach helps ensure model quality is stable, access isn’t unexpectedly removed, and companies retain control over IP, data privacy and compliance. Open-source models have rapidly progressed through families like Llama, Mistral, Qwen, DeepSeek, GLM and Kimi, with each generation narrowing the gap to proprietary frontier systems.
Specialized models as a business opportunity
Osman believes smaller, specialized models are a likely future for many enterprise use cases. Companies may start with a general model, gather traces and feedback from actual usage, and over time train a specialized model tailored to their workflows — improving performance, lowering costs and increasing utility. He also expects open-source model developers to monetize through licensing for fine-tuning, reinforcement learning, or specialized commercial deployments as enterprises secure their own compute.
Conclusion
Osman’s AIEWF workshops demonstrated that local, open-source AI is rapidly becoming a viable alternative to cloud-only frontier models. The combination of improved model efficiency and accessible hardware, together with enterprise demand for control and sovereignty over compute and data, is driving broader interest in treating local AI as serious infrastructure.
