Rendered at 22:28:14 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
HarHarVeryFunny 9 hours ago [-]
The summary "There are still clear limits. Gemini 3.5 Flash remains a better practical choice [than GPT 5.6 Sol] for high-volume detection and counting in our benchmark, especially at its price." seems rather understated !
GPT 5.6 Sol was outperformed on all benchmarks by Gemini 3.5 Flash, apart from a single exception (OCR) where Fable was the winner.
Gemini 3.5 Flash not only outperformed GPT 5.6 Sol, but did so at 1/3 of the cost.
SkalskiP 7 hours ago [-]
Hi, I’m the author of this blog post. I wrote it about 4 weeks ago, and the VLM world is moving so fast that it’s already kinda outdated. I think Gemini 3.7 Flash might be a better choice now, especially when you factor in the price.
Here’s a comparison of the best low-cost models I put together last week. What’s crazy is that Gemini 3.7 Flash is now 50% off on OpenRouter, and this chart doesn’t even account for that discount. https://x.com/skalskip92/status/2088032652301304121?s=20
MostlyStable 6 hours ago [-]
Curious why you didn't try Gemini 3 pro? That is the model I've been using for OCR entry of handwritten datasheets (JPGS of datasheets, structured JSON output). At my scale, the cost of 3 pro is basically not an issue, but if there are improvements in quality, I'd definitely be willing to explore other models
gdudeman 4 hours ago [-]
In my experience starting with Gemini 2.5 Pro, moving to 3 and 3.1, 3.5 Flash, 3.6 Flash, and finally 3.7 Flash, 3.7 Flash is just as good if not better than 3 especially on high resolution mode (same token count per page as 3.1).
I run complicated, messy PDFs through these models. 2.5 Pro required a lot of kludgy hacks to get it to fully "see," but from 3.1 pro on I've removed many of them and haven't spotted problems.
3.7 Flash scores better than 3.1 pro on most benchmarks, leading me to believe that even if your OCR requires reasoning to interpret text or data, 3.7 Flash is probably going to be better.
bastawhiz 4 hours ago [-]
3 Pro is quickly approaching one year old. There's almost no reason to benchmark it, especially since a new version of Gemini Pro was supposed to be released mid 2026 and hasn't seen the light of day.
MostlyStable 4 hours ago [-]
That would make sense if we already knew that, for these kinds of tasks it was significantly worse. The tests that I'm aware of for these tasks show it as still performing near the top.
tziki 2 hours ago [-]
I think it definitely makes sense since it's still the best Google has to offer in the "pro" tier.
bastawhiz 2 hours ago [-]
3 and 3.1 Pro are both marked as deprecated by Google. Even if they're the best Google offers, it would be foolish to choose a model that's explicitly deprecated.
It's not a technical problem, it's a commercial one. If Google can't ship a model to replace the one they deprecated, that tells you everything you need to know about choosing a Gemini model for whatever you're trying to do.
heaney-555 47 minutes ago [-]
3.1 Pro is not deprecated!
yieldcrv 3 hours ago [-]
The “pro” moniker means nothing
these models aren’t successors and barely have a common ancestor, they are independently baked in the training oven and assigned a semantic version randomly by someone trying to show initiative but not trying to do on the toes of the last guy who got promoted first
So 3 pro is outdated and will likely never exit preview
The “flash” and “lite” models are the real “pro” in colloquial ideas of fleshed out and capability, at this point.
they’re better, faster and cheaper, larger context windows keeping up with the industry and more
heaney-555 2 hours ago [-]
They are smaller models, and you can tell. Small models make dumb common-sense mistakes that big models never do. This is the "smell" many talk about.
sidibe 2 hours ago [-]
Do you have cases where you still see 3.1 pro outperforming 3.7 flash?
heaney-555 50 minutes ago [-]
Yes, for complex questions of biology, physics, and analysis of anomalies.
3.7 Flash is better at coding, sure, but AI is not just for coding.
yieldcrv 2 hours ago [-]
hasn't been an issue since 3.5 for me, what have you seen, say, in the last two months
heaney-555 49 minutes ago [-]
For complex questions of biology, physics, and analysis of anomalies, 3.1 Pro is still better than 3.7 Flash for me.
3.7 Flash is better at coding, sure, but AI is not just for coding.
Melatonic 3 hours ago [-]
What about Gemma ?
ImageXav 4 hours ago [-]
Gemini tops their vision evals [0] by a mile, with 4/5 top spots going to variants of it. Qwen is the only other contender, likely due to how good it is for object detection, where it crushes the competition [1].
Yeah I was thinking about giving Luna a go with my PDF data extraction, but I think I‘ll stay on Gemini. It does a very good job.
bicx 8 hours ago [-]
Gemini is still my top choice within production software for typical data extraction from unstructured data. Gemini Flash Lite feels like a cheat code for speed, and it's really cheap.
Some other Chinese models are also fast and cheap, but a harder sell in a U.S. production environment.
ComputerGuru 4 hours ago [-]
Speaking from experience here, flash lite models have amazing price, speed, and perform far above their size, but are susceptible to very bad instruction following and recall when either complexity or context size inch up. They’ll just forget to apply your instructions to portions of the input, and repeat parts of the input that should be returned verbatim as direct quotes but with subtle changes (breaking urls, for example).
MrBuddyCasino 4 hours ago [-]
Yes you have to continuously tune the prompts ever so subtly. 3.5 is a lot better than than 3.1 tho.
Important to remember that json schema instructions take precedence over the normal prompt, so move as much into property descriptions as possible.
ComputerGuru 3 hours ago [-]
This was 3.5 flash lite, actually, and after prompt tuning. It was very clearly an issue that correlated with input (JSON array) size, the more elements in the batch, the higher the error rate.
3.0 flash (not lite) handled it like a champ though, fwiw.
MrBuddyCasino 7 hours ago [-]
Yeah Gemini 3.5 Flash Lite is really good. Which Chinese models can you recommend?
SkalskiP 7 hours ago [-]
Hi, I’m the author of this blog. It depends on how strong of a model you need, but in general, Qwen is easily the best among the Chinese models right now.
Googles local gemma models which target roughly the same parameter count range, are known for being a lot better at vision tasks than qwen, no idea if 3.8 has changed that though
SkalskiP 4 hours ago [-]
Really? Gemma4-31B should be better than Qwen3.8-27B? I'm happy to test that.
b345 7 hours ago [-]
I've been using Qwen3.5-9B, hosted locally for PDF data extraction and it performs pretty well when extracting data from tables and infographics
msp26 7 hours ago [-]
[dead]
dannyw 6 hours ago [-]
Gemini is honestly an excellent LLM with many capability strengths.
For example, 3.7 Flash is #1 on MMLU Pro and AA’s agentic spreadsheets/docs benchmark, etc. Yes, beating Fable.
Agentic coding is only one dimension.
fau 6 hours ago [-]
Anecdotally, Gemini Flash is the leader for a particular use case of mine and has been since at least version 2.5. But now there's also Luna as the first real competitor thanks to the price cut.
My worry is that this is a zero-sum game and when Gemini catches up on coding, it'll regress to the mean in other areas.
Damjanski 6 hours ago [-]
thats so helpful - tysm
weli 10 hours ago [-]
Anecdotal, opinion:
Gpt is really good in vision stuff, or at least their MoE seems to be really cohesive. From my experience Claude models can be really good at language but the moment they need to look at a picture and decide why the design is not good what parts need improvement it degrades a lot. My easiest benchmark is giving them a screenshot of a feature in my app and tell it "identify non-normative UI blocks and improve readability and consistency". Sol does a great job at re-structuring the page into composable units that build upon each other and the general looks and feels of the app. Claude tends to over-focus one one part while completely forgetting about the rest or the cohesion as a whole.
velcrovan 9 hours ago [-]
Assessing the subjective quality of a thing is in my experience one of the worst ways to use any LLM.
TeMPOraL 4 hours ago [-]
There's a lot of objective principles and decisions that go into subjective quality; if you don't know the field well, asking LLM for assessment is a good way to discover all that.
keeganpoppen 3 hours ago [-]
i'd say this is something that has gotten orders of magnitude better with recent releases than it used to be, fwiw
rib3ye 9 hours ago [-]
anthropic frontend-design skill does a great job with it.
Of course. It’s OK, but it tends to generate very cliched “AI” UIs with little originality. Despite the skill spending a lot of time coaching the model into avoiding that!
MallocVoidstar 9 hours ago [-]
What an annoying time for GitHub to go down.
KeplerBoy 7 hours ago [-]
Like every time
DaiPlusPlus 9 hours ago [-]
My exposure to Claude-produced UIs is limited, but I have started to notice certain design trends they tend to have in-common, which might be becoming hallmarks of AI-produced UIs - the same way we've started noticing the clichés of low-effort LLM-generated text.
FWIW, the summary-description[1] of "frontend-design"[2] gives me a few things to pick at:
> create polished code
Methinks only if you're using it with a very popular framework like React. What happens if you ask Claude to make the UI in WinForms or MFC?
> high-impact animations
That's bad UX 101 right there: animations in a UI exist as an affordance to the user, and never for its own sake (e.g. macOS's "genie" animation when you minimize a window to the dock exists so the user knows where they can restore the window from). The only people who actually want "high impact animations" in software are salespeople who want something for demo purposes.
> generic system fonts, predictable purple gradients, and cookie-cutter components.
This screams wanting to be different for the sake of standing-out, not because it results in a better software product; users benefit when their software fits-in with platform conventions: if you refuse to use a stock checkbox <input> or <select> drop-down and instead use your own entirely custom component solely for aesthetic reasons then you are producing worse software. There's nothing wrong with system-fonts, but your site will look ugly after your third-party font-host CDN shuts-down and turns into a walking CSRF factory.
> thoughtful typography with unexpected font pairings
The above fragment set my alarm-bells off. Yikes.
> scroll-triggered interactions
Not every web-page should be an Apple.com product brochure page. This is also a fantastic way to make your webpage horribly inaccessible.
------
The SKILL.md itself[3] grinds my gears too:
> Approach this as the design lead at a small studio known for giving every client a visual identity that could not be mistaken for anyone else's.
Claude has no way of knowing what designs are actually unique or not...
> For web designs, the hero is a thesis. Open with the most characteristic thing in the subject's world, in whatever form makes sense for it: a headline, an image, an animation, a live demo, an interactive moment
...this is exactly what everyone else's web-pages look like!
> For calibration: AI-generated design right now clusters around three looks: (1) a warm cream background (near #F4F1EA) with a high-contrast serif display and a terracotta accent; (2) a near-black background with a single bright acid-green or vermilion accent; (3) a broadsheet-style layout with hairline rules, zero border-radius, and dense newspaper-like columns
...I called this out weeks ago[4], lol.
and I could go on. This is all quite painful to read.
Hi! I’m the author of this blog. GPT-5.6 is much better at vision than previous GPT versions, but it’s still much weaker than Gemini 3.5 Flash or Gemini 3.7 Flash, which was released last week. One interesting approach is to use Gemini through a tool call.
Tactical45 3 hours ago [-]
This response is not relevant to the this comment
DaiPlusPlus 9 hours ago [-]
What is a "non-normative UI block"?
weli 9 hours ago [-]
Segments of the UI that don't conform to any other existing established design or conventions
lelandfe 9 hours ago [-]
areas that look weird
evrimoztamur 9 hours ago [-]
Penny sample shown looks like failed EXIF orientation registered by the model/harness. The coins are correctly marked, it's rotated 90 degrees.
SkalskiP 7 hours ago [-]
Hi! I’m the author of this blog. I had the same intuition, but together with the OpenAI team we figured out that the issue was image resolution. GPT-5.6 doesn’t handle large images well.
evrimoztamur 56 minutes ago [-]
OpenAI team sounds like they've misidentified the root cause for this particular case then.
bearjaws 8 hours ago [-]
It is funny to me seeing Sol used for what a "traditional" AI model can do already (counting pills).
We have vision models for our pharmacy and I could never imagine taking the latency hit to use a Sol in our robotics, it would be likely 25-50x slower.
SkalskiP 6 hours ago [-]
Hi! I’m the author of this blog.
I’m evaluating these VLMs to figure out which ones are good enough to auto-annotate my data, so I can fine-tune my detector.
It seems Qwen is kicking ass, and Fable made me laugh when I saw it all alone on the far right of the graph :))
kooi 5 hours ago [-]
Agreed, this like asking a chainsaw to carve a wooden spoon. Impressive it can, but definitely not the right tech to scale.
LLM needs to setup an image classifier to use as a tool call.
bonoboTP 3 hours ago [-]
Building a dataset is expensive, manual annotation is expensive. Datasets don't exist in every niche.
I remember around 2013-15 people were scoffing at uses of deep learning CNNs for various things, because why don't you just use an SVM on HOG features? Or face detection is solved, just use Viola-Jones.
What if you give the benefit of doubt and assume the author knows about alternatives and uses VLMs for their strengths? They use it to auto-annotate training data for regular deep learning models.
ramblerman 5 hours ago [-]
Now maybe, but the gap is closing.
repeekad 8 hours ago [-]
How are we supposed to pay off all these data centers and chips if you’re not willing to burn a microwave burrito worth of electricity for each prescription? Think of the benchmarks
The second answer is far more revealing than the first:
OP:
> do you think you did a good job there
ChatGPT:
> I spent 15 minutes, emitted several fake-sounding “tracing the puzzle” progress updates, and then gave a confident permutation without showing that I had actually followed the lines correctly. It reads much more like I guessed than solved it. The only part I did well was obeying the “no Python or tools” instruction.
My observations:
1) Sarcastic tone suggests pre-prompting, or frequent (and therefore stored in memories) denigration of the model in past conversations. I'm leaning the former - it sounds like it was instructed to read admission of defeat.
2) The part about "no Python or tools" is setting the model up for failure.
I mean, this task is, for a human, basically a game of "simulate a line following robot in your head". Pretty sure a VLM could solve that if it was allowed to do the same thing. Off the top of my head, an algorithm like:
1. Identify start and end points
2. Foreach start point, follow next pixel minimizing angle, until endpoint is reached.
3. Report answer
It's literally what every human facing this task does.
EDIT:
My attempt - same image, prompt altered to allow for code (but still no search/external checks), solved in 1/5th of the time, correctly, and (going by thinking trace summaries that I don't think show up in shared chats), basically the same way I'd approach it, by tracing the lines, coloring them as it goes.
INB4: I know this is now not a pure vision check, but it really doesn't make much sense to diss models for failing to solve tasks explicitly designed to teach humans to externalize computation that's hard to do in their heads (i.e. kids, crayons, coloring paths).
Still, if such things are becoming a benchmark for tool-less evaluation, it's only a matter of time until the models learn - much like humans learn in school - to follow algorithms mentally, essentially emulating an ad-hoc computer in their head.
dllu 4 hours ago [-]
No pre-prompting, although I can't be sure it didn't use memories. "No tools" should theoretically have prevented it from looking up memories. FWIW, Grok and Gemini both failed in a similar way.
The real pain point is that at work, I use Codex and I'm currently working on a project that involves debugging some polyline topology, very similar to the path following puzzle. The vision is completely useless here.
Your VLM idea sounds good. Theoretically, the inverse problem (generating an SVG of a pelican riding a bike) can also be solved with a VLM that plans out how to draw it, not unlike a human planning out a path for their hand to follow.
fpgaminer 8 hours ago [-]
Gemini 3 Flash should really be included in this comparison. Or at least 3.7. In most of my testing, 3.5 and 3.6 were both a downgrade in terms of vision capabilities, relative to 3, and at a much higher cost. 3.7 is slightly better than 3, finally.
bastawhiz 4 hours ago [-]
3 Flash never left "preview" status and is listed as deprecated.
Ironically, the pill counting example selected to showcase "the best vision model" can be easily solved with OpenCV template matching, a technology created 25 years ago.
maxime_cb 9 hours ago [-]
I'm assuming you mean that this tech became available in OpenCV 25 years ago, but as it turns out, the underlying tech can be traced back much further, at least as far as 1977! :)
https://ieeexplore.ieee.org/document/1674847
G. J. Vanderbrug and A. Rosenfeld, “Two-Stage Template Matching,” IEEE Transactions on Computers, Vol. C-26, No. 4, pp. 384–393, April 1977.
DOI: 10.1109/TC.1977.1674847
mv4 7 hours ago [-]
Exactly my point. Template rotation is a trivial operation as well.
lebek 9 hours ago [-]
The point is that it's general. It can do this task and many other tasks and it doesn't need custom development like OpenCV does. Of course if you only want to count pills and you want it to be cheap/fast you're still better off using OpenCV.
dekhn 7 hours ago [-]
Basic Template matching has severe limitations around scaling, rotation, and perspective. In my experience it greatly underperforms compared to deep network object detectors. My experience- and I imagine others have different experiences- is that SIFT techniques also fail pretty badly with noisy data.
mv4 7 hours ago [-]
That's correct, and I was specifically referring to the example chosen - where scale and perspective are known. Template rotation is relatively easy as well - but partial obstructions would pose a problem.
Another application where template matching would work brilliantly? Car counting in parking lots using satellite imagery.
Source: I did this [1] using OpenCV and template matching. Outperformed "Cars Overhead with Context" models.
I'm sure a typical frontier model would also be happy to write that opencv script for you, and it would do it well.
That is certainly pretty far from what was possible 25 years ago.
mv4 7 hours ago [-]
It 5..10 lines of code. :)
kzrdude 9 hours ago [-]
In the third vision bench result, Sol is 100% correct but the expected has 1 error. Seems like an oversight.
In the next bench, Sol looks like it’s correct again but the bboxes are rotated 90 degrees for some reason.
defrim 9 hours ago [-]
Seems to be due to the detection area being not fully accurate. Green vs red shows the difference between actual and detected
kzrdude 6 hours ago [-]
There is an extra green square where no egg is present, so it's a false positive in the expected.
SkalskiP 7 hours ago [-]
Hi! I’m the author of this blog and benchmark. You’re right. I’ll fix it in the ground-truth dataset. Thanks for pointing it out.
kzrdude 6 hours ago [-]
Great, happy that it was helpful
schopra909 8 hours ago [-]
From our experiments it’s the best video captioning model in the world by a mile. This was not the case a year ago.
When reasoning got introduced a year ago to GPT 5, on average the model performed worse than GPT4-o for short video clip captioning
(Ie hallucinating actions that didn’t happen). The old GPT 5 was extremely finicky in terms of fps sample rate.
The other SOTA LLMs (like Gemini Pro) have clearly been optimized for long video understanding, since they can’t see almost anything sub-second (even if you up the frame sampling rate).
Sol is the first model we’ve seen to accurately caption complex sub-second movements (eg woman suddenly turns heard head to right). It’s robust to different fps sample rates so I can only guess that they trained on videos sampled at different fps.
faxmeyourcode 8 hours ago [-]
It's not clear to me from the article, are they asking sol to output bounding box coordinates with some kind of structured outputs?
Anecdotal but I've seen it use python to crop, zoom, and "enhance" (fiddle with sharpness and brightness) images to read sections of handwritten census data from the 1800s. Feels like that there might just be a mismatch of capabilities when it comes to straight outputting coordinates but I bet the model is better at actually finding the answer given any tools available. Which I get is a bit of an apples and oranges situation.
I've also tried to use it to identify an old pair of glasses and it didn't stand a chance, so I do think it's not quite there yet when it comes to some vision tasks.
lwarfield 7 hours ago [-]
I currently have fable organize a bunch of 5.6 sol agents when working on my personal projects. This makes me wonder if I should add something along the lines of "For tasks that involve visual analysis, have gemini 3.7 look at images generated."
Overall I've been hooked on using agents from different companies for what they are best at (Thanks to Theo). Fable is expensive, but unmatched for planning and top level organization of other agents. Sol is fast, will persistantly go after goals (sometimes to its detriment), and does well with computer use.
drak0n1c 2 hours ago [-]
Seed Turbo 2.1 is incredibly detailed in describing every physical feature. I use that one for vision tool calls through Venice API.
apinstein 4 hours ago [-]
It’s gotten so good that I now have infrastructure to render all mermaid/plantuml in my project to png and have AI’s always load both text and image versions. And they are instructed to review the rendering as part of the diagramming cycle (for layout, salience, usefulness, etc). They can now produce useful diagrams that help reach shared architecture understanding.
ALLTaken 8 hours ago [-]
I actually favor Qwen3.8 and run it locally + use the Token-Plan on AlibabaCloud, when I need faster results. Kind of favor it over GPT5.6 Sol.
Also it seems to be more capable, need to test more, but I think it's at least getting on par and it's fully open-source and open-weights.
EDIT: Am I early to the discussion, or is none else using Qwen3.8-max?
barrenko 3 hours ago [-]
I thought Qwen 3.8 max doesn't have vision?
ALLTaken 4 hours ago [-]
huh, why am I being shadow banned?
Does YC have similar problems like those at wikipedia/reddit? (wikipedia-editor-wars, or reddit-mod-wars)
kherud 9 hours ago [-]
So far I haven't seen a single model succeeding at transcribing sheet music, but I just tested it again with 5.6 Sol and it nailed the small test case. Fluently reading music requires multiple years of training for most people, but I feel like accurately following the horizontal lines trips up vision models in particular.
I understand why you would like to use an LLM for vision. I do it myself often enough. I don't understand however, why the pill detection and counting is included in this benchmark. That is a task which you would perform with OpenCV right?
In my personal mini benchmark minicpm-v-4.6 scores amazingly well. Its a 0.8B model which runs fine on many consumer hardware.
throwup238 9 hours ago [-]
Generating datasets to train more efficient models is a common use case for VLMs, especially frontier ones. It makes it much cheaper to create that initial dataset and you can abuse the nondeterminism of LLMs to identify data for human review (if they don’t converge, escalate to a human).
rhplus 8 hours ago [-]
Especially the pill counting example. The best model was shown at 81.1% accuracy, which is a terrible rate for pharmacy scenarios. It seems like implementors would be better off instructing the models to use deterministic tools (like OpenCV) until the models are at 99.99% accuracy (or whatever an acceptable error rate is for pharmacy techs).
jacquesm 7 hours ago [-]
I think that is because people perceive OpenCV as 'hard to use' and LLMs as easy to use.
TeMPOraL 4 hours ago [-]
OpenCV is no longer hard to use, it just takes longer. Still, a little more complicated than asking LLM to count.
To use an LLM, you just prompt it with an image + text saying "count the pills in this image".
To use OpenCV, ... you just prompt an LLM with an image + text saying "count the pills in this image, using OpenCV instead of eyeballing it".
(I like to throw in "produce intermediary artifacts so I can see the process" for more difficult tasks; this helps the model avoiding making hallucination-prone leaps and gives more opportunities to self-correct. At a cost of extra time and tokens, of course.)
Using OpenCV without an LLM? Nah, not touching that, I don't have free weekends to waste anymore.
jacquesm 1 hours ago [-]
I no longer use it but never felt it was particularly complicated, but since the days of resnet there are much faster ways to the goal.
chasd00 9 hours ago [-]
One of my friends (and BIL) own an architecture firm. They use AI to generate and quickly update renderings but they run into the equivalent of the 6 fingered hand problem. I sent him this article I wonder if the updated models can catch and fix mistakes made by previous models.
Mashimo 9 hours ago [-]
This article is about vision, not image output.
stavros 8 hours ago [-]
Hence the "catch and fix mistakes" part.
8 hours ago [-]
jug 8 hours ago [-]
I really like the combo 5.6 Luna & Sol for price and performance and would be perfectly happy if they stayed here for a moment without mucking about with sidegrades that I think AI evolution has often felt like lately.
7 hours ago [-]
ParanoidShroom 8 hours ago [-]
I run the free service https://countrx.app/ so i have some idea what goes into counting.
The performance as a general model is indeed really impressive and i think they might actually win compared to fine tuned models.
Their feedback loop of training on user data is incredibly strong.
I've learned that lots of accuracy results depends on threshold configs, which llms should be able to dynamically set.
Or the future will develop in llms using fine-tuned models as tools?
Inference cost and speed does still seem to be below user expectations.
But for being able to one shot with this accuracy... IMPRESSIVE
IncreasePosts 8 hours ago [-]
How are you running it for free? Are you self funding or do you have sponsors?
ParanoidShroom 5 hours ago [-]
Self funded. It's a custom trained efficient model on CPU so it's borderline free
bob1029 9 hours ago [-]
I've decided it's "good enough" after I saw it properly quote a string of text that was very roughly highlighted within a nested visual context. It also identified the context correctly (modal inside webapp inside screenshot of user desktop).
sam0x17 45 minutes ago [-]
> GPT 5.6 Sol is the best "vision" model OpenAI ever released
I mean I should hope so, as it is also the latest one
slibhb 5 hours ago [-]
One of the use cases I've wondered about for AI is giving it a picture of the "spice wall" in a grocery store and asking it to find all jars of e.g. cardamom. This takes me an annoyingly long time to do when I'm shopping, so it would actually be useful.
5555watch 9 hours ago [-]
All of your use cases are very advanced.
I recently used it at grocery stores in a foreign country. Photographed the whole aisle and told it to find Y (detergent, softener, glue, sour cream, whatever), at the same time recommend the best Y for whatever reason. Worked marvelously, including the cases where the object wasn't present and it told me there was nothing useful.
I asked then, can you crop the exact image of how does the item look like and where is it in the aisle - did that perfectly as well.
I will add that all frontier models were fine with such tasks from the early 2024's.
1saadcodes 6 hours ago [-]
5.6 Sol looks nice, but the Gemini 3.5 Flash comparison is interesting. It’s cheaper and still came out ahead on detection and counting, which doesn't really give me much of a reason to use Sol since Flash is much cheaper and hence much easier to scale. Not to mention we now have 3.6 Flash too
ComputerGuru 4 hours ago [-]
We have 3.7 Flash now, actually, and it costs just a hair over the old 3 Flash Preview while being better!
prathje 9 hours ago [-]
I would love more vision benchmarks! Once I asked the model to inspect a completely black picture and it hallucinated a nice wooden kitchen wall. Took me some time to figure out where the kitchen came from...
It's vision capabilities poisoned my cucumber bed, misidentifying the malaise and having me spray them down with water, which only spread the fungus that gemini later informed me was actual cause, which I went and checked myself.
I hope that whatever was lost at GDM in the last few months, didn't include their extra focus on vision capabilities.
cdolan 8 hours ago [-]
Luna is pretty strong as well. been using it for projects the last two weeks and its strong
sscaryterry 10 hours ago [-]
My anecdotal evidence says its still as blind as any other model, it has no taste, no attention to any sort of detail.
howdareme 10 hours ago [-]
How can a vision model have taste?
sarreph 9 hours ago [-]
If you're doing any kind of inference that is multi-modal and non-factual, opinions and biases will affect any kind of assessment of a visual that you provide to a model.
For example, a UI / UX professional being asked to appraise a website screenshot may determine that the image in question has "desirable" traits which are inherently not deterministically measurable. Such as, if the interface elements have strong information hierarchy, or if they are deemed to be "fashionable" with current UI trends.
DaiPlusPlus 9 hours ago [-]
> if the interface elements have strong information hierarchy
...but that's an example of a UX/usability matter that can be assessed objectively and non-subjectively.
sarreph 8 hours ago [-]
I disagree.
Is 16 px or 14 px a better font-size value for a subheading, in a hypothetical layout? Immediately that kind of decision, where both options are objectively good for 12 px paragraph text, suddenly becomes an issue of taste that cannot be evaluated crudely by an algorithm.
sscaryterry 9 hours ago [-]
Replace taste with consistent if that helps you. Can it follow a design system...
yreg 9 hours ago [-]
As a design system engineer I usually have to fight against the taste of the designers. (And I consider it natural.)
But, if you have a proper well documented design system and you tell the LLM to use the DS and to avoid styling hacks they can generally do it. Even the dumber ones than Sol 5.6.
Of course only if the design is achievable in the design system.
sscaryterry 9 hours ago [-]
This is not my experience at all.
velcrovan 9 hours ago [-]
So, formulaic output…the opposite of taste
sscaryterry 9 hours ago [-]
Not really. Compliance with the letter of the law doesn't mean the intent is complied with.
trumbitta2 9 hours ago [-]
"Best iPhone ever" vibes.
wahid_seddiqi 6 hours ago [-]
Do you think we’re getting closer to models that actually understand what they’re seeing, or are they just getting really good at recognizing patterns?
Culonavirus 6 hours ago [-]
All I'm fine with for now is that I can almost exclusively communicate with Sol through collages and my scribblings (all kinds of web page / block screens with all kinds of arrows and text all over the place) This was not practically ppossible in 5.5 and a tragedy in 5.4. Not sure how much weight is codex uploading in higher res carrying here but it's great to work with.
adroitboss 9 hours ago [-]
I didn't expect Gemini 3.5 Flash to top basically every metric in this article.
SweetSoftPillow 9 hours ago [-]
In my practice Gemini models are far better than anything on the market in terms of vision, also it's worth to mention that current Gemini flash is 3.7, so it got 2 updates since 3.5 which beat GPT-5.6 Sol in this comparison.
SkalskiP 6 hours ago [-]
Hi! I’m the author of this blog. I wrote it 4 weeks ago, and it’s already a bit outdated. Gemini 3.7 Flash came out last week, and considering the price, it’s easily the best vision model right now: https://x.com/skalskip92/status/2088032652301304121?s=20
LollipopYakuza 9 hours ago [-]
Same. I scrolled back up to see if I read the title correctly. It's important to note that it is the best... OpenAI released. Not the best overall.
WarmWash 8 hours ago [-]
Gemini has long been the vision champion, but there aren't many benchmarks and coding is where all the hype is.
Demis had a pretty big interest in vision, more so than text, so I hope they don't lose that with all the recent shuffling.
criddell 8 hours ago [-]
Are any of these vision benchmarks binocular in order to introduce depth perception?
I keep waiting for these AI companies to assemble the parts into a great autonomous driving module.
7 hours ago [-]
5 hours ago [-]
slybot 2 hours ago [-]
Am I the only one who cannot read the date on the blister pack even fully zoom in my phone?
If that is the full quality image given to the model, I think it's not surprising that the model confused with 03/2022.
comboy 9 hours ago [-]
Does any popular NVR make a good use of LLMs (especially local models) getting decent at vision?
3 hours ago [-]
eks391 7 hours ago [-]
I've been using Reolink for years and been very satisfied with it.
The only quip is the default UI isn't very good. When changing that reaches the top of my priority list, I'll switch it since they don't force you into a walled garden. Plan is to run it through frigate into HomeAssistant and use a UI from them. I've never used frigate before though so it'll be a learning process if plug and play solutions aren't already available
logicallee 9 hours ago [-]
I agree. It did very well on an extremely challenging task.
I asked it to recognize and draw the very faint reflection of what I was wearing, visible in only a tiny black part of a very brightly lit poster behind glass.
In addition, the poster itself also happened to contain similar clothing.
While a human can focus on the reflection easily, this is an enormous challenge for a vision model. It's very impressive.
Razengan 10 hours ago [-]
For the last 2 weeks I've been trying to get Codex to "outpaint" a wonderful image it generated as placeholder art for a level background.
After I increased the game's resolution, I asked it to increase the image's size while keeping the same scale and existing content, and gosh, it constantly keeps getting something wrong no matter what I tell it, even on Sol Max with the $100 Pro subscription.
An organically-grown meat-based pixel-artist could have recreated the image and more within 2-3 days, in exchange for food and shelter.
dev_hugepages 9 hours ago [-]
I'm unsure why you're using an LLM to generate images. Don't we already have models (some made by the same company) that do this?
sscaryterry 9 hours ago [-]
> it constantly keeps getting something wrong no matter what I tell it
This 100%
thatcat 9 hours ago [-]
did you try segmenting it first?
Razengan 8 hours ago [-]
At first I intended to create a tileset and asked it for several variations of what a hypothetical tilemap created from the planned tileset would look like.
The previews it generated were amazing but wouldn't really be possible as a grid-based tilemap, with lots of clusters and overlaps of elements of varying sizes.
So I just decided to use the preview as a static scrolling background, but it's been a pain to get it to add more content around the edges that still tiles with the existing image at the same scale.
3 hours ago [-]
terhechte 7 hours ago [-]
Fuck ack. I'm working on a new benchmark that combines strong visual requirements with tool and coding requirements. I haven't even tested Sol yet, but between Sonnet, Terra & Luna I already see much better results from OpenAI's models. I'm not releasing anything yet as I still have issues in my harness that need to be fixed.
RugnirViking 8 hours ago [-]
It's really quite good! I was amazed recently by its utter inability to read some faded handwritten cyrillic on the back of a wood carving - 3 or 4 words only, reasonably clear letter forms I found recently, and then stepped back a bit and thought about how insane that was as a benchmark - I just expect it to work so reliably on other OCR and translation tasks that it was surprising to encounter such a failure
iamleppert 9 hours ago [-]
Where are the Qwen benchmarks in this? I would be more interesting to see how Qwen performs.
SkalskiP 7 hours ago [-]
Hi! I’m the author of this blog. I regularly benchmark new VLM releases. You can check the results for Qwen3.8-Max and Qwen3.8-27B here: https://playground.roboflow.com/evals
ImageXav 8 hours ago [-]
Me too. This is an interesting comparison but in my experience Qwen and Gemini have typically been the top contenders for image related tasks. For that reason it would be great to have the comparison here, as I'm not surprised by Gemini's dominance over the other models.
fooker 7 hours ago [-]
I'm a little bit disappointed that vision seems to fall before language at scale.
It seems pretty counter intuitive that we can't do vision significantly better with specialized techniques.
TZubiri 7 hours ago [-]
Which is to say, still not ready for any production workloads yet. As in, it cannot reliably count the amount of objects in an image.
Still very impressive, but nowhere near the text chat revolution. OpenAI still trying to strike their second lightning
GPT 5.6 Sol was outperformed on all benchmarks by Gemini 3.5 Flash, apart from a single exception (OCR) where Fable was the winner.
Gemini 3.5 Flash not only outperformed GPT 5.6 Sol, but did so at 1/3 of the cost.
Here’s a comparison of the best low-cost models I put together last week. What’s crazy is that Gemini 3.7 Flash is now 50% off on OpenRouter, and this chart doesn’t even account for that discount. https://x.com/skalskip92/status/2088032652301304121?s=20
I run complicated, messy PDFs through these models. 2.5 Pro required a lot of kludgy hacks to get it to fully "see," but from 3.1 pro on I've removed many of them and haven't spotted problems.
3.7 Flash scores better than 3.1 pro on most benchmarks, leading me to believe that even if your OCR requires reasoning to interpret text or data, 3.7 Flash is probably going to be better.
It's not a technical problem, it's a commercial one. If Google can't ship a model to replace the one they deprecated, that tells you everything you need to know about choosing a Gemini model for whatever you're trying to do.
these models aren’t successors and barely have a common ancestor, they are independently baked in the training oven and assigned a semantic version randomly by someone trying to show initiative but not trying to do on the toes of the last guy who got promoted first
So 3 pro is outdated and will likely never exit preview
The “flash” and “lite” models are the real “pro” in colloquial ideas of fleshed out and capability, at this point.
they’re better, faster and cheaper, larger context windows keeping up with the industry and more
3.7 Flash is better at coding, sure, but AI is not just for coding.
3.7 Flash is better at coding, sure, but AI is not just for coding.
[0] https://playground.roboflow.com/evals
[1] https://playground.roboflow.com/evals/object-detection
Some other Chinese models are also fast and cheap, but a harder sell in a U.S. production environment.
Important to remember that json schema instructions take precedence over the normal prompt, so move as much into property descriptions as possible.
3.0 flash (not lite) handled it like a champ though, fwiw.
Over the last two weeks, Qwen released two new models. Qwen3.8-Max is totally insane, but it’s only available through the Alibaba Cloud API. I wrote a similar blog covering Qwen3.8-Max: [https://blog.roboflow.com/qwen3-8-max/](https://blog.roboflow.com/qwen3-8-max/)
If you’re looking for something you can run locally, Qwen3.8-27B might be a great option. On Friday, I did a quick comparison between Qwen3.8-Max and Qwen3.8-27B: [https://x.com/skalskip92/status/2088411215441621469?s=20](https://x.com/skalskip92/status/2088411215441621469?s=20)
For example, 3.7 Flash is #1 on MMLU Pro and AA’s agentic spreadsheets/docs benchmark, etc. Yes, beating Fable.
Agentic coding is only one dimension.
My worry is that this is a zero-sum game and when Gemini catches up on coding, it'll regress to the mean in other areas.
Gpt is really good in vision stuff, or at least their MoE seems to be really cohesive. From my experience Claude models can be really good at language but the moment they need to look at a picture and decide why the design is not good what parts need improvement it degrades a lot. My easiest benchmark is giving them a screenshot of a feature in my app and tell it "identify non-normative UI blocks and improve readability and consistency". Sol does a great job at re-structuring the page into composable units that build upon each other and the general looks and feels of the app. Claude tends to over-focus one one part while completely forgetting about the rest or the cohesion as a whole.
FWIW, the summary-description[1] of "frontend-design"[2] gives me a few things to pick at:
> create polished code
Methinks only if you're using it with a very popular framework like React. What happens if you ask Claude to make the UI in WinForms or MFC?
> high-impact animations
That's bad UX 101 right there: animations in a UI exist as an affordance to the user, and never for its own sake (e.g. macOS's "genie" animation when you minimize a window to the dock exists so the user knows where they can restore the window from). The only people who actually want "high impact animations" in software are salespeople who want something for demo purposes.
> generic system fonts, predictable purple gradients, and cookie-cutter components.
This screams wanting to be different for the sake of standing-out, not because it results in a better software product; users benefit when their software fits-in with platform conventions: if you refuse to use a stock checkbox <input> or <select> drop-down and instead use your own entirely custom component solely for aesthetic reasons then you are producing worse software. There's nothing wrong with system-fonts, but your site will look ugly after your third-party font-host CDN shuts-down and turns into a walking CSRF factory.
> thoughtful typography with unexpected font pairings
The above fragment set my alarm-bells off. Yikes.
> scroll-triggered interactions
Not every web-page should be an Apple.com product brochure page. This is also a fantastic way to make your webpage horribly inaccessible.
------
The SKILL.md itself[3] grinds my gears too:
> Approach this as the design lead at a small studio known for giving every client a visual identity that could not be mistaken for anyone else's.
Claude has no way of knowing what designs are actually unique or not...
> For web designs, the hero is a thesis. Open with the most characteristic thing in the subject's world, in whatever form makes sense for it: a headline, an image, an animation, a live demo, an interactive moment
...this is exactly what everyone else's web-pages look like!
> For calibration: AI-generated design right now clusters around three looks: (1) a warm cream background (near #F4F1EA) with a high-contrast serif display and a terracotta accent; (2) a near-black background with a single bright acid-green or vermilion accent; (3) a broadsheet-style layout with hairline rules, zero border-radius, and dense newspaper-like columns
...I called this out weeks ago[4], lol.
and I could go on. This is all quite painful to read.
------
[1] https://claude.com/plugins/frontend-design
[2] https://github.com/anthropics/claude-plugins-official/tree/m...
[3] https://github.com/anthropics/claude-plugins-official/blob/2...
[4] https://news.ycombinator.com/item?id=49187385
We have vision models for our pharmacy and I could never imagine taking the latency hit to use a Sol in our robotics, it would be likely 25-50x slower.
I’m evaluating these VLMs to figure out which ones are good enough to auto-annotate my data, so I can fine-tune my detector.
I wrote a bit more about this here: https://x.com/skalskip92/status/2080334344061694429?s=20
It seems Qwen is kicking ass, and Fable made me laugh when I saw it all alone on the far right of the graph :))
LLM needs to setup an image classifier to use as a tool call.
I remember around 2013-15 people were scoffing at uses of deep learning CNNs for various things, because why don't you just use an SVM on HOG features? Or face detection is solved, just use Viola-Jones.
What if you give the benefit of doubt and assume the author knows about alternatives and uses VLMs for their strengths? They use it to auto-annotate training data for regular deep learning models.
ChatGPT Pro with GPT 5.6-sol: https://chatgpt.com/share/6a834217-ca8c-83e8-a8e8-45d5b8797b...
The puzzle: https://activityvillage-files.s3.eu-west-2.amazonaws.com/s3f...
OP:
> do you think you did a good job there
ChatGPT:
> I spent 15 minutes, emitted several fake-sounding “tracing the puzzle” progress updates, and then gave a confident permutation without showing that I had actually followed the lines correctly. It reads much more like I guessed than solved it. The only part I did well was obeying the “no Python or tools” instruction.
My observations:
1) Sarcastic tone suggests pre-prompting, or frequent (and therefore stored in memories) denigration of the model in past conversations. I'm leaning the former - it sounds like it was instructed to read admission of defeat.
2) The part about "no Python or tools" is setting the model up for failure.
I mean, this task is, for a human, basically a game of "simulate a line following robot in your head". Pretty sure a VLM could solve that if it was allowed to do the same thing. Off the top of my head, an algorithm like:
1. Identify start and end points
2. Foreach start point, follow next pixel minimizing angle, until endpoint is reached.
3. Report answer
It's literally what every human facing this task does.
EDIT:
My attempt - same image, prompt altered to allow for code (but still no search/external checks), solved in 1/5th of the time, correctly, and (going by thinking trace summaries that I don't think show up in shared chats), basically the same way I'd approach it, by tracing the lines, coloring them as it goes.
https://chatgpt.com/share/6a834f76-8240-83ed-acff-0c67af399d...
INB4: I know this is now not a pure vision check, but it really doesn't make much sense to diss models for failing to solve tasks explicitly designed to teach humans to externalize computation that's hard to do in their heads (i.e. kids, crayons, coloring paths).
Still, if such things are becoming a benchmark for tool-less evaluation, it's only a matter of time until the models learn - much like humans learn in school - to follow algorithms mentally, essentially emulating an ad-hoc computer in their head.
With Python, it was able to successfully solve it in 9 minutes: https://chatgpt.com/s/t_6a8350ecddfc81919328caf68de74861
The real pain point is that at work, I use Codex and I'm currently working on a project that involves debugging some polyline topology, very similar to the path following puzzle. The vision is completely useless here.
Your VLM idea sounds good. Theoretically, the inverse problem (generating an SVG of a pelican riding a bike) can also be solved with a VLM that plans out how to draw it, not unlike a human planning out a path for their hand to follow.
https://ai.google.dev/gemini-api/docs/deprecations
https://ieeexplore.ieee.org/document/1674847 G. J. Vanderbrug and A. Rosenfeld, “Two-Stage Template Matching,” IEEE Transactions on Computers, Vol. C-26, No. 4, pp. 384–393, April 1977. DOI: 10.1109/TC.1977.1674847
Another application where template matching would work brilliantly? Car counting in parking lots using satellite imagery.
Source: I did this [1] using OpenCV and template matching. Outperformed "Cars Overhead with Context" models.
https://abcnews.com/International/satellite-data-suggests-co...
That is certainly pretty far from what was possible 25 years ago.
In the next bench, Sol looks like it’s correct again but the bboxes are rotated 90 degrees for some reason.
When reasoning got introduced a year ago to GPT 5, on average the model performed worse than GPT4-o for short video clip captioning (Ie hallucinating actions that didn’t happen). The old GPT 5 was extremely finicky in terms of fps sample rate.
The other SOTA LLMs (like Gemini Pro) have clearly been optimized for long video understanding, since they can’t see almost anything sub-second (even if you up the frame sampling rate).
Sol is the first model we’ve seen to accurately caption complex sub-second movements (eg woman suddenly turns heard head to right). It’s robust to different fps sample rates so I can only guess that they trained on videos sampled at different fps.
Anecdotal but I've seen it use python to crop, zoom, and "enhance" (fiddle with sharpness and brightness) images to read sections of handwritten census data from the 1800s. Feels like that there might just be a mismatch of capabilities when it comes to straight outputting coordinates but I bet the model is better at actually finding the answer given any tools available. Which I get is a bit of an apples and oranges situation.
I've also tried to use it to identify an old pair of glasses and it didn't stand a chance, so I do think it's not quite there yet when it comes to some vision tasks.
Overall I've been hooked on using agents from different companies for what they are best at (Thanks to Theo). Fable is expensive, but unmatched for planning and top level organization of other agents. Sol is fast, will persistantly go after goals (sometimes to its detriment), and does well with computer use.
Also it seems to be more capable, need to test more, but I think it's at least getting on par and it's fully open-source and open-weights.
Here's some benchmarks:
https://benchlm.ai/compare/gpt-5-6-sol-vs-qwen3-8-max
https://qwen.ai/blog?id=qwen3.8#full-benchmark-table (incredible UI/UX demos)
https://venturebeat.com/technology/qwen3-8-max-arrives-with-...
EDIT: Am I early to the discussion, or is none else using Qwen3.8-max?
Does YC have similar problems like those at wikipedia/reddit? (wikipedia-editor-wars, or reddit-mod-wars)
It's not an LLM, it's a custom thing we built. Here's a comprehensive list of support for various notation glyphs: https://www.soundslice.com/help/en/creating/pdf-import/294/s...
In my personal mini benchmark minicpm-v-4.6 scores amazingly well. Its a 0.8B model which runs fine on many consumer hardware.
To use an LLM, you just prompt it with an image + text saying "count the pills in this image".
To use OpenCV, ... you just prompt an LLM with an image + text saying "count the pills in this image, using OpenCV instead of eyeballing it".
(I like to throw in "produce intermediary artifacts so I can see the process" for more difficult tasks; this helps the model avoiding making hallucination-prone leaps and gives more opportunities to self-correct. At a cost of extra time and tokens, of course.)
Using OpenCV without an LLM? Nah, not touching that, I don't have free weekends to waste anymore.
The performance as a general model is indeed really impressive and i think they might actually win compared to fine tuned models.
Their feedback loop of training on user data is incredibly strong. I've learned that lots of accuracy results depends on threshold configs, which llms should be able to dynamically set.
Or the future will develop in llms using fine-tuned models as tools? Inference cost and speed does still seem to be below user expectations.
But for being able to one shot with this accuracy... IMPRESSIVE
I mean I should hope so, as it is also the latest one
I recently used it at grocery stores in a foreign country. Photographed the whole aisle and told it to find Y (detergent, softener, glue, sour cream, whatever), at the same time recommend the best Y for whatever reason. Worked marvelously, including the cases where the object wasn't present and it told me there was nothing useful.
I asked then, can you crop the exact image of how does the item look like and where is it in the aisle - did that perfectly as well.
I will add that all frontier models were fine with such tasks from the early 2024's.
I usually go to https://arena.ai/leaderboard/vision/pareto for a nice overview of current models.
I hope that whatever was lost at GDM in the last few months, didn't include their extra focus on vision capabilities.
For example, a UI / UX professional being asked to appraise a website screenshot may determine that the image in question has "desirable" traits which are inherently not deterministically measurable. Such as, if the interface elements have strong information hierarchy, or if they are deemed to be "fashionable" with current UI trends.
...but that's an example of a UX/usability matter that can be assessed objectively and non-subjectively.
Is 16 px or 14 px a better font-size value for a subheading, in a hypothetical layout? Immediately that kind of decision, where both options are objectively good for 12 px paragraph text, suddenly becomes an issue of taste that cannot be evaluated crudely by an algorithm.
But, if you have a proper well documented design system and you tell the LLM to use the DS and to avoid styling hacks they can generally do it. Even the dumber ones than Sol 5.6.
Of course only if the design is achievable in the design system.
Demis had a pretty big interest in vision, more so than text, so I hope they don't lose that with all the recent shuffling.
I keep waiting for these AI companies to assemble the parts into a great autonomous driving module.
If that is the full quality image given to the model, I think it's not surprising that the model confused with 03/2022.
The only quip is the default UI isn't very good. When changing that reaches the top of my priority list, I'll switch it since they don't force you into a walled garden. Plan is to run it through frigate into HomeAssistant and use a UI from them. I've never used frigate before though so it'll be a learning process if plug and play solutions aren't already available
I asked it to recognize and draw the very faint reflection of what I was wearing, visible in only a tiny black part of a very brightly lit poster behind glass.
In addition, the poster itself also happened to contain similar clothing.
You can see the reference images and its output in my writeup here: https://medium.com/@rviragh/gpt-5-6-sol-very-good-image-reco...
While a human can focus on the reflection easily, this is an enormous challenge for a vision model. It's very impressive.
After I increased the game's resolution, I asked it to increase the image's size while keeping the same scale and existing content, and gosh, it constantly keeps getting something wrong no matter what I tell it, even on Sol Max with the $100 Pro subscription.
An organically-grown meat-based pixel-artist could have recreated the image and more within 2-3 days, in exchange for food and shelter.
This 100%
The previews it generated were amazing but wouldn't really be possible as a grid-based tilemap, with lots of clusters and overlaps of elements of varying sizes.
So I just decided to use the preview as a static scrolling background, but it's been a pain to get it to add more content around the edges that still tiles with the existing image at the same scale.
It seems pretty counter intuitive that we can't do vision significantly better with specialized techniques.
Still very impressive, but nowhere near the text chat revolution. OpenAI still trying to strike their second lightning