Loading…
27
Local LLM vs cloud API for a small shop, I chose offline and got mixed results
I run a small inventory forecasting tool for a friend's auto shop near Cleveland, and last month I had to choose between sending customer data to a cloud API like GPT-4 or running a smaller open source model locally. I went local because the owner was nervous about vehicle records leaving his server, plus the monthly cost would have been around $120 for API calls. The setup took me two full evenings to get working on an old desktop with 16GB of RAM, and the first few runs gave me predictions that were just off enough to notice. After tweaking prompts and quantization settings for a week, it got within 8% of the cloud model's accuracy on our test data. But the real headache is updates, I have to manually pull new model weights and retrain every time the shop changes their pricing tiers. Cloud would have been zero maintenance but a recurring fee, so I stuck with local for now. Has anyone else made this jump and regretted it when a new model version dropped suddenly?
0 comments
Log in to join the discussion
Log In0 Comments
No comments yet
Be the first to share your thoughts on this discussion.