How well does this (and similar services) deal with anti-bot protection? What happens if I attempt to retrieve a page protected by the cloudflare checkbox or by Anubis? An error, no useful content, or the actual content I want?
> Search API finds web pages for a query and returns structured results. It is powered by DeepSeek's in-house search engine.
Looks like they've built (or are building) their own index and return cached results. What you do with that information is up to you. If you want to visit the live page, you'd need to find a way around bot detection.
One reason (not the only one!) people use Anubis is to prevent the endless flood of requests from AI companies. If Deepseek scrapes once every few hours, that helps the website owner with the traffic problem while allowing access.
If they don't want to allow access, my guess is that deepseek's search crawler respects robots.txt but I don't know.
There is no single answer to this, it entirely depends on the exact service and the exact page
Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss
>Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
The last part is extremely dubious with some recent reveals.
I've spent a lot of time building datamint.xyz, and dealing with user agents, datacenter IPs, and VPNs gets pretty tricky. What works right now is a fork of the latest Firefox with anti-fingerprinting patches, paired with a high-quality residential proxy network with clean IPs. Even the basic setup needs 11+ code-level patches, and there are still a few edge cases, but this setup should cover 99% of cases.
Thank you for escalating the arms race that's currently destroying the internet. I can't look at the Apple user forums, my health insurance company's portal, pay my county taxes, browse Digikey and Mouser, view Github or whatever forge, etc. etc. without having to jump through Cloudflare or Anubis garbage designed to block your abusive behavior.
I'm just overflowing with joy that you're comfortable wasting my time and money.
Even if it's not the GP, if there's profit to be made, somebody will do it. I was reminded of the power of economic motives the other day, when it turned out someone bought an entire trustworthy reseller just so they could compromise the Ledger supply chain by including a tiny eSIM module that OCRed the screen and transmitted the seed.
Here’s what is says on their site:
“Courtesy backoff
Adaptive backoff keyed to target response codes. When a target returns 429 or 503, we back off immediately, so our fleet never applies denial-of-service pressure to a site.”
By the time you get the error you have already crashed the site.
Geez.
I am curious if my large website is in their search results. My company charges $1 per page crawled of our website, and the only traffic that worms through our firewall is using hacked residential ISPs. I really need to know who to send the (large!) bill to.
How does that work? I'm not a laywer, but I can invoice you $1 and the T.o.S of reading what I have to say are licensed under a $1 per view licence and by continued reading you consent to the terms of this license. Where can I send the bill for this comment?
About a year ago I built my own search and fetch MCP tools for agents based on third-party scraping providers. It is about 15-20x more cost-effective than what OpenAI and Anthropic is offering with their integrated web search and fetch tools. Very impressive that DeepSeek has such a competitive offering today.
DeepSeek Flash really came at a bad time before the Anthropic IPO. I use this thing daily and it’s neck and neck with SOTA models. The search is one of the things setting it apart.
Is anyone actually oneshotting a whole app or even vibecoding one in multiple steps without it being "guided work"? I really wish announcements (like DeepSeek's recent one) would stop focusing on the ability of the model to spit out Minecraft clones on demand.
With Opus 5.5 you can vibe code a whole app without reading the code at all. Obviously you still have to tell it what to make.
With earlier models that doesn't really work. You really have to read the code and watch it like a hawk or it will make a big mess.
To put it another way, with lesser models you have to guide them at a code level. With Opus 5.5 you only have to guide it at an architectural / feature level.
I have one-shotted the two most recent relatively complex apps I've tried:
1. a puzzle game with complex interplay, pieces to place, design vs. play mode, storing multiple solutions with the puzzle, and more. I've since iterated something like 200 times as I worked out the design etc., but the first iteration was 99% correct and what I asked for
2. a learn-to-read application with built-in stories (that the LLM wrote) the ability to import epub, text, and HTML, that uses the built-in speech libraries as a default, but also supports an open-source TTS library that I gave in the first prompt, tracks progress and problem workds, has multiple games to reinforce learning, and has a parental mode for tracking progress and configuring the tool overall.
Frankly I was stunned that Claude one-shotted that second app, there was a lot in the prompt, and it did it all. I fully expected to have to iterate on the first draft multiple times. But it worked out of the box.
If Anthropic or OpenAI fails, the other wins the US market, no? I’m oversimplifying but…that’s a pretty realistic possibility in the face of people thinking sinking one US AI company would hurt the greater US venture into AI.
> If Anthropic or OpenAI fails, the other wins the US market, no?
I am of the opinion that they both either succeed or they both fail. They are both losing money hand over fist and their model quality is pretty close to each other. There is not a whole lot that sets them apart.
I agree on the analysis, but not on the conclusion. What makes you think that the conditions you describe are stable grounds for a duopoly where both could be successful at the same time?
If they did then Google would take over, and others would form from their competent engineers. Also, they don’t need to be profitable to succeed, they merely need to hype investors enough to ensure that their losses matter less than their growth projections.
Oh come on, this post sounds like one of those twitter bots. No one seriously expected them to just give away billions in R&D for free, and they arent. The chinese model companies are actively moving away from the open source setup. Even state export controls.
They're about half a gen behind. Very impressive to get this kind of utility from a flash series model... the impact of RL seems to be behind much of the coding and agentic improvements lately.
For comparison, Kagi charges $12 per 1000 searches (only talking about their API pricing, like what you’d use with an LLM; their $10/mo plan has unlimited searches but it’s a TOS violation to use those credentials with an LLM).
Most of your local requests will fail (with Python/Curl/whatever) if you are not using a remote-controlled Firefox/Chrome. And even then you will be bitten brutally by captchas, rendering them basically unusable for mass requests. So it's better to outsource your web search to an external service with a stable API.
The existing search machines that offer free APIs for local harnesses are super bad with low quality.
This is worth the invest as LLMs can be guided closer to existing information/code to adapt it.
I was really excited to use this (well, I still am), but then I realized Ollama Cloud offer this too, though they don't really mention pricing. I think it's somehow rolled into their subscription.
For comparison that’s $5 with Brave Search API and $12 with Kagi API
$4 with Exa, $1.50 with SERP API
How well does this (and similar services) deal with anti-bot protection? What happens if I attempt to retrieve a page protected by the cloudflare checkbox or by Anubis? An error, no useful content, or the actual content I want?
> Search API finds web pages for a query and returns structured results. It is powered by DeepSeek's in-house search engine.
Looks like they've built (or are building) their own index and return cached results. What you do with that information is up to you. If you want to visit the live page, you'd need to find a way around bot detection.
One reason (not the only one!) people use Anubis is to prevent the endless flood of requests from AI companies. If Deepseek scrapes once every few hours, that helps the website owner with the traffic problem while allowing access.
If they don't want to allow access, my guess is that deepseek's search crawler respects robots.txt but I don't know.
There is no single answer to this, it entirely depends on the exact service and the exact page
Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss
This is not true, Claude and ChatGPT have sent traffic to sites I maintain without any user-agent, it might be they have stopped doing this.
Google has done various versions of this as well, Grok too.
It could be these bots were in some custom harness or something, but they literally sent requests in very weird ways, that humans won't be doing.
I have ended up putting my sites behind vpns or anubis and similar. This is beyond painful.
>Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
The last part is extremely dubious with some recent reveals.
recent reveals show that OAI and Anthropic give users the ability to go around bot protections or they internally use it?
It’s not exactly recent or a reveal, but I’ve caught at least several OAI models writing Python scripts to access web pages.
I've spent a lot of time building datamint.xyz, and dealing with user agents, datacenter IPs, and VPNs gets pretty tricky. What works right now is a fork of the latest Firefox with anti-fingerprinting patches, paired with a high-quality residential proxy network with clean IPs. Even the basic setup needs 11+ code-level patches, and there are still a few edge cases, but this setup should cover 99% of cases.
Thank you for escalating the arms race that's currently destroying the internet. I can't look at the Apple user forums, my health insurance company's portal, pay my county taxes, browse Digikey and Mouser, view Github or whatever forge, etc. etc. without having to jump through Cloudflare or Anubis garbage designed to block your abusive behavior.
I'm just overflowing with joy that you're comfortable wasting my time and money.
Even if it's not the GP, if there's profit to be made, somebody will do it. I was reminded of the power of economic motives the other day, when it turned out someone bought an entire trustworthy reseller just so they could compromise the Ledger supply chain by including a tiny eSIM module that OCRed the screen and transmitted the seed.
Incentives are crazy powerful.
Here’s what is says on their site: “Courtesy backoff Adaptive backoff keyed to target response codes. When a target returns 429 or 503, we back off immediately, so our fleet never applies denial-of-service pressure to a site.” By the time you get the error you have already crashed the site. Geez.
I have to agree with the (sarcastic) sibling comment. You’re essentially making everyone’s life worse in the name of personal profit.
Please stop. The last thing we need in the world is more greed and selfishness.
Since when have moral appeals worked against the central feature of capitalism?
Only strong regulation can help.
you can bypass those easily except Google Captcha v3, you need to have your own proxy swarm to make it at scale.
Is there any way to test this without an API key?
I am curious if my large website is in their search results. My company charges $1 per page crawled of our website, and the only traffic that worms through our firewall is using hacked residential ISPs. I really need to know who to send the (large!) bill to.
How does that work? I'm not a laywer, but I can invoice you $1 and the T.o.S of reading what I have to say are licensed under a $1 per view licence and by continued reading you consent to the terms of this license. Where can I send the bill for this comment?
I understand if you pay is a different story.
I mean ... "how does this work?"
About a year ago I built my own search and fetch MCP tools for agents based on third-party scraping providers. It is about 15-20x more cost-effective than what OpenAI and Anthropic is offering with their integrated web search and fetch tools. Very impressive that DeepSeek has such a competitive offering today.
Source for those scraping providers?
DeepSeek Flash really came at a bad time before the Anthropic IPO. I use this thing daily and it’s neck and neck with SOTA models. The search is one of the things setting it apart.
It's absolutely not SOTA for vibe coding a whole app, but for guided work it is great.
Is anyone actually oneshotting a whole app or even vibecoding one in multiple steps without it being "guided work"? I really wish announcements (like DeepSeek's recent one) would stop focusing on the ability of the model to spit out Minecraft clones on demand.
With Opus 5.5 you can vibe code a whole app without reading the code at all. Obviously you still have to tell it what to make.
With earlier models that doesn't really work. You really have to read the code and watch it like a hawk or it will make a big mess.
To put it another way, with lesser models you have to guide them at a code level. With Opus 5.5 you only have to guide it at an architectural / feature level.
OP didn’t mention one-shotting.
There are plenty of use cases for the ability to quickly through together and refine an application without reviewing the code though.
Some people are trying to one shot entire companies
And failing.
I have one-shotted the two most recent relatively complex apps I've tried:
1. a puzzle game with complex interplay, pieces to place, design vs. play mode, storing multiple solutions with the puzzle, and more. I've since iterated something like 200 times as I worked out the design etc., but the first iteration was 99% correct and what I asked for
2. a learn-to-read application with built-in stories (that the LLM wrote) the ability to import epub, text, and HTML, that uses the built-in speech libraries as a default, but also supports an open-source TTS library that I gave in the first prompt, tracks progress and problem workds, has multiple games to reinforce learning, and has a parental mode for tracking progress and configuring the tool overall.
Frankly I was stunned that Claude one-shotted that second app, there was a lot in the prompt, and it did it all. I fully expected to have to iterate on the first draft multiple times. But it worked out of the box.
which model is SOTA for vibe coding a whole app? I always use models+agents in guided setup
the moment there is a model which can code whole app, we can fairly say software engineering is done.
It's really not neck and neck with Opus 5.5.
Cost-adjusted,it might be ahead.
It's key to name the boundary conditions. Unlimited cost is a very HN assumption.
Max Subscription seems to be the HN assumption most of the time
Really depends on your use case.
You are right that there's plenty of use cases where Opus 5.5 is better.
If it depends on your use case then it’s not neck and neck…
Models are comming every 4-6 months. Anthropic, OpenAI, all of them know this.
4-6 months? Try between 8-18 days for a foundation lab release.
Or at a really good time, depending on who you ask.
Not everyone wants Anthropic to succeed.
If Anthropic or OpenAI fails, the other wins the US market, no? I’m oversimplifying but…that’s a pretty realistic possibility in the face of people thinking sinking one US AI company would hurt the greater US venture into AI.
> If Anthropic or OpenAI fails, the other wins the US market, no?
I am of the opinion that they both either succeed or they both fail. They are both losing money hand over fist and their model quality is pretty close to each other. There is not a whole lot that sets them apart.
I agree on the analysis, but not on the conclusion. What makes you think that the conditions you describe are stable grounds for a duopoly where both could be successful at the same time?
What if both fail?
If they did then Google would take over, and others would form from their competent engineers. Also, they don’t need to be profitable to succeed, they merely need to hype investors enough to ensure that their losses matter less than their growth projections.
What if all of them fail because the capitalist model of centralization followed by rent-seeking gets foiled by an open alternative?
That is explicitly the Chinese strategy.
Now, what if the largest capitalist economy went all-in on an AI that might no longer be sufficiently monetizable?
> explicitly the Chinese strategy
Oh come on, this post sounds like one of those twitter bots. No one seriously expected them to just give away billions in R&D for free, and they arent. The chinese model companies are actively moving away from the open source setup. Even state export controls.
No? Why would either of one of those have to win? Even if you limit yourself to US companies only, Google and Meta still exist.
i thought open models weren't SOTA. therefore not neck and neck... in your experience, do you think it depends on your type of work?
They're about half a gen behind. Very impressive to get this kind of utility from a flash series model... the impact of RL seems to be behind much of the coding and agentic improvements lately.
In my experience, for coding, they are pretty much SOTA. Maybe not for other sciences and problem solving.
> bad time
Regardless of the performance metrics... why bad time?
Deepseek is more like a pet project with huge state backing, and they don't seem to compete with Anthropic in terms of profitability.
It literally is not though... even DeepSeek doesn't claim that. It's just super cost-effective
For comparison, Kagi charges $12 per 1000 searches (only talking about their API pricing, like what you’d use with an LLM; their $10/mo plan has unlimited searches but it’s a TOS violation to use those credentials with an LLM).
>it’s a TOS violation
Competitor uruky is more relaxed about api and is cheaper. Creator is active on hn
https://news.ycombinator.com/user?id=BrunoBernardino
I wonder if this is censored. If not then users in China mainland can use the api to workaround gfw (though there are much better ways)
Can someone explain why fetch costs money. I thought it was your locally-running harness doing it. How can they charge you for that?
Most of your local requests will fail (with Python/Curl/whatever) if you are not using a remote-controlled Firefox/Chrome. And even then you will be bitten brutally by captchas, rendering them basically unusable for mass requests. So it's better to outsource your web search to an external service with a stable API.
The existing search machines that offer free APIs for local harnesses are super bad with low quality.
This is worth the invest as LLMs can be guided closer to existing information/code to adapt it.
As reference point exa charges 4 bucks
I was really excited to use this (well, I still am), but then I realized Ollama Cloud offer this too, though they don't really mention pricing. I think it's somehow rolled into their subscription.
These prices will skyrocket eventually. AI companies need more money.
RAM prices will also probably stay very high as long as AI companis act in such a parasitic manner.
[flagged]
[dead]