Best AI/ML/DL Rig For 2024 - Most Compute For Your Money!

The speaker introduces the topic of the best deep learning rig for the money in 2024. They share their opinion on the most cost-effective approach, comparing it to common strategies. The speaker discusses their experience with Dell PowerEdge r720 servers, highlighting their reliability and value for those new to deep learning or needing affordable access to resources for large language models and computer vision. The speaker details their latest build, emphasizing the performance and cost-effectiveness of a 40-core server with 256GB DDR3 RAM, two 1.2TB SAS hard drives, and the addition of two Tesla P4 GPUs for a total of 48GB VRAM. They mention the need for adapters and provide a link to the server purchase from SaveMyServer.

The speaker provides a cost analysis of the deep learning setup, totaling around $1,000 for the entire rig. They challenge the audience to find a better setup for less money and discuss the monthly operating costs, based on an average power consumption of 3.4 kilowatts and an electricity cost of 12 cents per kilowatt-hour. The speaker compares this setup to a custom rig they built in the past, noting the differences in CPU cores, RAM, and storage. They argue that while the custom rig may be faster and more modern, it lacks the raw power of the current setup for the same price, making it more suitable for entry-level tasks rather than tackling larger deep learning problems.

The speaker compares the on-premises setup with cloud-based GPU solutions, highlighting the benefits of direct hardware access and cost-effectiveness. They discuss the specs of the RTX 6000 GPUs offered by cloud services, noting that while these GPUs are more performant, the overall cost is significantly higher, with less storage and RAM. The speaker emphasizes the limitations of cloud solutions, such as data transfer caps and network usage restrictions. They mention Leno, a cost-effective alternative to major cloud providers, and share their positive experience with the service. The speaker also considers pay-per-compute options, acknowledging that these might be more suitable for those on a tight budget but expressing a preference for the flexibility and ease of troubleshooting offered by owning and managing the hardware.

The speaker concludes by reiterating the cost-effectiveness of self-managed, older hardware for deep learning tasks. They acknowledge that while the hardware may be older, the performance it provides is still significant, even by contemporary standards. The speaker encourages the audience to explore various options and perform their own cost-benefit analysis. They also mention alternative platforms like Collab and Kaggle, but express dissatisfaction with these services due to their limitations and shared compute resources. The speaker provides a real-world example of the cost of using Leno's compute service, illustrating that hourly rates may appear cheap but can add up quickly with extensive usage. They advocate for the value of investing in and assembling hardware, managing it oneself as the best strategy for 2024.

The speaker wraps up the discussion by inviting questions and comments from the audience, offering to respond and provide feedback. They remind viewers to like and subscribe to support the channel's growth and produce better content. The speaker also suggests buying them a coffee as a form of support, with a link provided in the video description. They conclude by thanking the audience for their engagement and look forward to connecting in the New Year.