Rendered at 22:28:04 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
nine_k 35 minutes ago [-]
Sending two identical parallel requests is the classic approach. But, logically speaking, it should also double the cost.
I would send a second request if the first request fails to return the first token within, say, 1 second. Then there's a chance the first request is stalling, which is an infrequent event.
I wonder if higher-availability tiers of LLM providers do a similar thing internally.
dvaplima 39 minutes ago [-]
Nice turn around, does anyone has a benchmark regarding other types of requests (priority vs send twice) other than voice/call? Or the tests already test that?
eigenblake 1 hours ago [-]
I love this. Simple. Useful. To the point. If AI was used, I can't tell because it is clearly representing the author's beliefs.
thousand_nights 44 minutes ago [-]
agreed, reads like a breath of fresh air, no fluff
ramon156 37 minutes ago [-]
for a tier thats twice the cost i would expect >2x the speed. somewhere 5-10x
e.g. 1.40m would become 0.30s.
do people really pay for these priority plans?
crisnoble 27 minutes ago [-]
Why not send it thrice?
moffkalast 21 minutes ago [-]
If you want a controllable and predictable system, host it yourself. APIs will always have outages, delays and breaking changes every so often. That's the price you pay for not doing it properly and outsourcing your job.
I would send a second request if the first request fails to return the first token within, say, 1 second. Then there's a chance the first request is stalling, which is an infrequent event.
I wonder if higher-availability tiers of LLM providers do a similar thing internally.
e.g. 1.40m would become 0.30s.
do people really pay for these priority plans?