The competition to make AI inference faster continues: while Cerebras and its bespoke chips drew positive market attention with an IPO in May, French startup Kog is betting it can extract much greater performance from standard datacenter GPUs through software engineering.
Demo details and hardware used
In May Kog hit Hacker News with a technical preview intended to show that “extremely fast single-request decoding is possible on the standard datacenter GPUs enterprises already own.” The demo ran on AMD MI300X and Nvidia H200 GPUs and achieved 3,000 tokens-per-second (TPS) per request on a purpose-built, now open-sourced model called Laneformer 2B, which has roughly 2 billion parameters.
Kog’s proposition is to unlock new capabilities on existing hardware via software rather than forcing customers to buy new chips — an attractive offer for enterprises for which inference speed and cost are critical bottlenecks.
Business interest and target customers
CEO Gaël Delalleau told TechCrunch that the demo generated more than 200 tangible business leads. The company is initially targeting customers who are deterred by latency — typically professional workflows where quick responses matter. Kog also counts design partners among potential users, including teams that generate games and apps from prompts; for them, faster turnaround through the Kog Inference Engine (KIE) would translate directly into higher revenue.
Technical challenges and goals
Kog found that the market is not yet ready to fine-tune small models at scale, so since launch the company has focused on accelerating larger models to match observed demand. The firm has publicly stated an ambitious target of “30× faster LLM inference,” but the current demo’s 3,000 TPS was achieved on a small, purpose-built model, and scaling that result to large language models remains the key challenge.
Delalleau argues the same low-level approach can be applied to LLMs: modern GPUs have increasing memory bandwidth and other characteristics that, he says, are underutilized and can be unlocked with deep software optimization. He considers the belief that GPUs are poorly suited for decoding to be a misconception and says GPUs have a strong future in this role.
Method, team and limits
Kog’s approach is very hands-on and hardware-close. It relies on understanding the ‘‘laws of the GPU’’ — including reverse-engineering down to assembly and binary — to squeeze out performance. Delalleau’s background in solid-state physics (École Polytechnique) and offensive (white-hat) cybersecurity, including being a four-time DEFCON CTF finalist, informs this methodology.
The downside is time and scale: the company dedicates weeks or months to engineering research for each new GPU. With a team of 11 people, Kog’s capacity to cover many different chips simultaneously is constrained in the near term. The company plans to feed its methodology into agent-based pipelines over the longer run to automate support for more chips and models.
Competitors and backers
Software-driven GPU acceleration is not unique to Kog. French company ZML released hardware-agnostic software that bypasses Nvidia’s CUDA and supports fast inference across competing chips. Delalleau positions Kog closer to Stanford’s Hazy Research in terms of deep, GPU-level focus rather than general-purpose toolchains.
Kog is supported by Scaleway and backed by France’s Bpifrance and the French Tech 2030 program. The seed round was co-led by Varsity VC — whose partner Kamel Zeroual had been a prior co-founder with Delalleau at his earlier startup Stribe.
Next steps and fundraising
Kog must demonstrate that its approach delivers similar gains on large language models to substantiate its claims and attract further investment. Delalleau expects the company’s first major model implementation at a 10× speedup by September; achieving that milestone and showing customer traction would pave the way toward raising a Series A round.
In sum, Kog is pursuing a pragmatic, low-level software engineering route to boost GPU-based inference performance. The approach requires significant manual engineering and is currently limited by team size, but if it scales to LLMs it could deliver meaningful latency and cost improvements for enterprise AI users.
