Single NVIDIA RTX 3090 found capable of running 1,000+ user AI chatbot

An Estonian GPU, a cloud startup, has demonstrated that a single NVIDIA RTX 3090 can run the equivalent of a customer chatbot AI system.

Single NVIDIA RTX 3090 found capable of running 1,000+ user AI chatbot
Comment IconFacebook IconX IconReddit Icon
Junior Editor
Published
1 minute & 15 seconds read time

The underlying technology of all of these AI-powered tools that are continuous popping up in various sectors of the technology industry are Large Language Models (LLMs), which are powered by number-crunching graphics cards, or GPUs.

RTX 3090 benchmark results at 100 concurrent users - 12.88 tokens per second

RTX 3090 benchmark results at 100 concurrent users - 12.88 tokens per second

The explosion of AI has been fuelled by the possibility of graphics cards now being powerful enough to crunch large swaths of data into a model that can then be queried by users. However, powerful models such as ChatGPT, require thousands of GPUs to continuously fulfil all the requests of the millions of users enjoying the service.

But what if you just wanted to support a few thousand users at a time? Perhaps you are a business that could need an AI chatbot to assist with customer service via your website? And ultimately, do you need thousands of GPUs to achieve this? Apparently not, or at least according to benchmarks from Backprop, an Estonian GPU cloud start-up who managed to get a modest LLM such as Llama 3.1 8B to work on a single NVIDIA RTX 3090 GPU, which was released in late 2020.

The start-up found throughout its testing the RTX 3090 provided AI performance that was comparable to a customer service AI chatbot, with the GPU from 2020 only being able to fulfil requests from 100 concurrent users. In a test the RTX 3090 was able to serve a user 12.88 tokens per second, which is faster than the average person can read at five works per second, and faster than the industry standard for an AI chatbot, which is 10 tokens per second.

It appears its possible to run a customer service-equivalent AI chatbot with a single RTX 3090 GPU, and this AI chatbot could support thousands of users and fulfil the requests of 100s at any given time.

Photo of the MSI Gaming GeForce RTX 3090 24GB GDRR6X
Best Deals: MSI Gaming GeForce RTX 3090 24GB GDRR6X
Country flag Today 7 days ago 30 days ago
Loading... Loading...
Buy
* Prices last scanned on 10/31/2024 at 5:57 pm CDT - prices may not be accurate, click links above for the latest price. We may earn an affiliate commission from any sales.

Junior Editor

Email IconX IconLinkedIn Icon

Jak joined the TweakTown team in 2017 and has since reviewed 100s of new tech products and kept us informed daily on the latest science, space, and artificial intelligence news. Jak's love for science, space, and technology, and, more specifically, PC gaming, began at 10 years old. It was the day his dad showed him how to play Age of Empires on an old Compaq PC. Ever since that day, Jak fell in love with games and the progression of the technology industry in all its forms.

Newsletter Subscription

Join the daily TweakTown Newsletter for a special insider look into new content and what is happening behind the scenes.

Related Topics

Newsletter Subscription