Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister (https://hister.org/).
Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews and the full page content via MCP.
YaCY has a proxy mode that automatically index your web-serving. In my experience, the index grow in size very fast and reaches ~100GB or more. How does the index size of Hister compare to that?
zeroq 10 hours ago [-]
I'm sorry for not taking the time to read the docs, but I have a question.
Some 20 years ago a friend of mine has set up a local proxy (python if I'm not mistaken) that was gathering all his web traffic and served him as a long term memory. The proxy had a web interface and allowed him to quickly find something he saw ca. 10 days ago, or that specific algorithm he recalls but can't remember it's name.
For years I've been collecting links to different work related trivia which I use on a daily basis as a rabbit-from-a-hat solution to answer random question from friends and coworkers. For example someone randomly asked me for an idea for color palette for data charts and I can immediately give them a scientific research into the color palette. Or an obscure algorithm.
But with time the collection has grown substantially and it's really cumbersome to find the proper things.
Would your project be a good fit for my problem?
asciimoo 10 hours ago [-]
Absolutely, this is a great example where Hister can shine.
I started Hister as a proxy as well, but quickly switched to the current extension based approach, because intercepting HTTPS traffic requires a MiTM proxy which is much more painful to setup than installing a browser extension.
sunshine-o 53 minutes ago [-]
I found Hister a few month ago and was amazed by it.
Now for many of us the browser extension approach is not possible (mobile usage, security, etc.)
My feeling is for a lot of users there is really a third way apart from the MiTM proxy or Browser extension approach. I actually do not want my "personal" / "logged in" pages to be indexed. This is a bit like the MS recall nightmare (self hosted version).
Any way to get the list of URL visited (with something like Privoxy, or maybe one of those popular ad blockers like Pi Hole but I guess they just get DNS queries?) and then importing it with some filtering rules with a nightly batch job is good enough for a lot of people.
The browser import [1] is great but I guess hard to use with mobile...
would it be possible to gdrive/rsync/git the data between machines and then use the data on an online server for retrieval (given that I would handle data sync myself)?
also what exactly are you using for search? does it support trigrams? how do you sort results?
left-struck 7 hours ago [-]
Would you mind sharing these links? Or a subset? I want to grow my collection which is tiny because I started way too late
justusthane 10 hours ago [-]
Also very interested in this. I was playing around with doing the same thing with YaCY. I want the proxy aspect so that I can proxy my phone traffic through it as well.
asciimoo 10 hours ago [-]
Unfortunately mobile Chrome browsers don't support browser extensions, but our extension works well on mobile Firefox.
ydj 7 hours ago [-]
Hister sounds like something I wanted for a while, but never got around to building. Searching stuff I’ve seen before is most of what I do with a search engine, so having it local and fast would be amazing. Eager to give it a try.
MrDrMcCoy 10 hours ago [-]
Always excited to see new things like Hister in the search space. What are the scaling limits, as far as you can tell in terms of how much can it hold before queries start breaking down or become too slow to be useful? Could it evolve into a general internet search engine if, say, enough trusted members of a geo-distributed YugabyteDB cluster and an army of crawlers built a sufficient index?
asciimoo 9 hours ago [-]
> What are the scaling limits, as far as you can tell in terms of how much can it hold before queries start breaking down or become too slow to be useful?
There has been no stress tests in this regard. The indexer lib Bleve [1] can handle millions of documents according to their documentation.
> Could it evolve into a general internet search engine if, say, enough trusted members of a geo-distributed YugabyteDB cluster and an army of crawlers built a sufficient index?
My long term goal is exactly this. I'd like to add federation/P2P feature [2][3] to evolve from being a private search companion. I'd appreciate any help designing the system.
This is great, like many others I've been thinking of something like hister but only for bookmarked web pages. I presume it should be straightforward with hister to do that?
All the best!
asciimoo 3 hours ago [-]
It is possible. The automatic website indexing can be turned off in the extension and manual indexing can be triggered via the command line tool, the extension popoup or by hotkeys.
derrida 9 hours ago [-]
Wow! that looks like a bit of software I have been dreaming about for awhile - will definately check out! You're at least doing something right in communicating the reasons why and appeal for starters! All the best!
chrisss395 10 hours ago [-]
I love your idea and wondered why saving and indexing browser visited pages was not being done. Does this handle large amounts of local files, for example 10-20TB across file types like Powerpoint, Excel, Word, and PDF?
asciimoo 10 hours ago [-]
In its current form it cannot handle this amount of data efficiently (and doesn't support powerpoint/excel/word yet), but this is a valid use-case, I've added a TODO item to experiment with it.
blackqueeriroh 9 hours ago [-]
Oh thank god there used to be several tools like this and they slowly went away. I’ve been wanting this to return.
11 hours ago [-]
kristianpaul 11 hours ago [-]
Is this similar to fastcrw ?
asciimoo 11 hours ago [-]
Both are search engines, but that's all the similarity. Hister has a traditional crawler, but its biggest strength is automatically indexing browser tabs as those are rendered. This way it bypasses authentication, CloudFlare, captchas and most of the annoying limitations of traditional crawlers.
Hister also provides full offline result previews. Check out the small read-only demo: https://demo.hister.org/
nickpsecurity 7 hours ago [-]
I was considering paying someone to build something like this at some point. With two jobs, I eventually had no time to even organize what I find. It's just piles of links in text files.
Can I give your software a huge list of URL's to index? Or do I need to use browser automation to open them a few at a time with it caching and indexing them?
operatingthetan 11 hours ago [-]
I installed this a while back and honestly I almost never touch it. It turns out that for me searching my history doesn't really replace a search engine at all. The built in extractor list is pretty limited and adding them seems like too much of an ordeal for me to bother.
asciimoo 10 hours ago [-]
Sure, it cannot fully replace web search engines (yet), but it can reduce the dependence on these services more and more as your index grows. Hister is designed to support quickly falling back to traditional search engines with a single hotkey if no results found.
I agree, we should add more extractors [1]. Can you recommend extractors you missed?
I've been using searxng for several years now. I don't run my own instances because the inhumane network censorship imposed by GFW, and proxy detection enforced by search engines. Instead, I rely on public instances on the list [1] and libredirect [2]. Note that service from a single instance is not guaranteed, but you can always switch to other available instances with little cost within a minute.
I won't say searxng can help you degoogle because metasearch engine calls other search engines, e.g., google, to collect results. However, if you try searxng, you can at least get rid of things like ai reviews in no time.
In the end, thank you people after searxng project and public instances.
SearXNG is my daily internet search now +5 years; with YaCY Backends and else as fallback. I also build internal document search or RAG applications with this setup (SearXNG also support json results).
However, there are some downer I accept because of privacy:
1. Its slower and the results are not that good then with others. But fast and good enough for most of my queries.
2. From time to time you get blocked on the duckduckgo, brave or whatever search and you must solve some captures. You can prevent this by getting and using API-Keys from them.
The nice thing about using your own backend is, that you can prio it in the results and for example, if I crawl the smallweb and other site important for myself, this sites come up first in the results.
sunshine-o 2 hours ago [-]
> SearXNG is my daily internet search now +5 years
Same here
> with YaCY Backends and else as fallback.
Do you run your own "super fast" YaCy instance? or with specific settings?
My experience with YaCy is it doesn't fit in the backend of SearX since YaCy kind of slowly stream results for about 30 seconds...
I also have a local `kiwix-serve` serving ZIM files of wikipedia, wiktionary, gutemberg, archwiki, etc. but same problem the kiwix search engine [0] doesn't really fit as a backend for SearX as it returns too many results and pollute the SearX result page.
What I haven't done yet is trying to plug SearX to a local Recoll instance [1]. But Recoll doesn't support indexing ZIM files... but could be useful for other archived html documents.
I would be curious to know more about a working setup since search is hard to get right.
I've been self hosting this as my default engine across all of my searches for a few years now. I can't recommend it more highly.
viviansolide 13 hours ago [-]
Same experience
ProofHouse 13 hours ago [-]
I’ll have to try, I’ve only recently learned Exa pricing is a bit crazy (especially on searches where you source 30-40 sources)I just used it be default and then was like oh damn when I got hit
satvikpendem 13 hours ago [-]
TinySearch wraps this and works well for agents. It's better than the native SearXNG MCP because it optimizes the context before it even gets to the agent so as to not waste tokens.
SearXNG did not include a built-in MCP server, last time I checked.
Havoc 9 hours ago [-]
It does have json response though so pretty trivial to get an LLM to make you a mcp
ProofHouse 13 hours ago [-]
Props
artooro 12 hours ago [-]
It works well if you connect it the Brave Search API, but using it a scraper is fairly unreliable. Google stopped working a few days ago.
ninjahawk1 8 hours ago [-]
I’ve been a big fan of SearXNG for a while now. My distain for google has only grown, so having the ability to search and avoid things like yk, small AI models being installed on my PC without my consent, is awesome.
goodroot 12 hours ago [-]
This appears to be a key tool for providing search to local models.
I'm curious what setups folks use to provide this functionality.
Since the quantized 24B parameter Gemma model came out, I've had good luck with tool calling on a 4070 Ti Super.
Successful tool calling is what finally made the local experience useful.
I should note this is for the general and not coding specific context.
gardnr 11 hours ago [-]
It has a JSON mode that you need to enable in settings and then you can create a simple python script to interact with it or have the agent use `curl` and `jq` to interact with it.
I am also interested in what a full local AI stack with web search and other tools looks like. As far as I can tell, SearX does not embed an MCP server, so it can't be directly called from llama-server for example. Open WebUI does have an integration for SearX and other providers, but the results I obtained weren't particularly impressive.
c-hendricks 8 hours ago [-]
I use Searxng through Onyx, both as regular search and Onyx's Deep Research mode. I also have https://github.com/ihor-sokoliuk/mcp-searxng to add search to coding agents. Haven't really had many issues with it.
jared0x90 9 hours ago [-]
are you running a quant?
i have a friend with a 4080 that is wanting to experiment with local models and those cards should be similar enough. can you give any more detail about your setup? ty!
It is really great to use. As the poster above mentioned, my setup with Sear is the following, all through `llama.cpp`, which has a built-in webui with an MCP client:
* SearXNG in Docker — enable the JSON API (`search.formats: [html, json]`; off by default).
* `llama-server` with `--webui-mcp-proxy`, then add the server in the webui.
Some gotchas:
* `searxng-mcp` forgets to declare its own dep → `--with fastmcp`.
* Endpoint is `/mcp`, not the `/searxng-mcp/mcp` the docs claim.
* `--webui-mcp-proxy` only enables the CORS proxy; each MCP server entry still needs its "Use llama-server proxy" checkbox ticked, or the browser fetches direct and CORS-fails.
* Terminal clients (OpenCode etc.) skip the proxy — point them straight at `:8100/mcp`.
A couple interesting tidbits:
* There are temporal issues with search-related tool calls. The model trips out. 2026 results read to it a "future-dated hallucination" because it doesn't know the date. There's an additional `--tools get_datetime` function that will allow it to ground via the real date.
* Snippets-only is enough for most "what's current" questions and keeps context tiny.
Let me know if you have any questions!
chatmasta 10 hours ago [-]
I’ve always liked this tool, but I’m of two minds regarding the privacy gained by sending my searches to 280 companies instead of just one.
fishgoesblub 12 hours ago [-]
I've been using SearXNG for a few years now, however I've been trying out Degoog as a SearXNG alternative since I've had issues with engines constantly failing or being slow since day 1 of using SearXNG, but Degoog has worse results with the same engines. It's a shame since I'm having to pick between slower but better results, or very fast but worse results.
tom9ow 12 hours ago [-]
[flagged]
ManWith2Plans 13 hours ago [-]
I've been using this for some projects. It's exceptional and I recommend it highly.
I actually included a recipe to deploy it to kubernetes in typekro, my TypeScript infrastructure-as-code project for kubernetes: https://typekro.run/api/searxng/
rcarmo 12 hours ago [-]
Years of regular use here, has been great even before I started using it as an agent tool.
arikrahman 13 hours ago [-]
I have used SearXNG hosts like https://searx.be/ but stick with Brave search for the most part. Are there other good hosts people tend to use?
vimredo 13 hours ago [-]
Personally, I self-host it myself. All the hosts I tried either errored often, or gave search results that were complete garbage.
another_twist 13 hours ago [-]
Been a fan of searX for a while. Not sure if this is the same thing but there were plenty of hosted versions too.
grigio 3 hours ago [-]
searxng is fantastic for AI agents
lucasrufkahr 12 hours ago [-]
Yeah, I find that searx results are way more relevant to what I’m actually looking for than a single engine. There’s so much manipulation going on that if you don’t aggregate multiple engines, it’s near impossible to get what you want.
jaygreat2020 5 hours ago [-]
This is really great!
queenkjuul 5 hours ago [-]
Highly recommend this. I set up a self-hosted instance and been using it exclusively for months. It's better than DDG and i don't miss Google whatsoever.
Image search is worse i guess mostly for lack of CDN so it's slow but whatever.
salmonik 13 hours ago [-]
I prefer 4get.
MrDrMcCoy 10 hours ago [-]
Friendly reminder that if your user and traffic count is low, your traffic is still unique and able to be profiled. Love and use this project, though.
noobcoder 12 hours ago [-]
how do i configure which specific search engines SearXNG pulls its results from?
Can we extend it to onyl search Stack Overflow and GitHub
tosief 11 hours ago [-]
[dead]
tomfow 13 hours ago [-]
[flagged]
tom6ow 12 hours ago [-]
[flagged]
tom6ow 12 hours ago [-]
[flagged]
tomnow 13 hours ago [-]
[flagged]
Rendered at 10:09:05 GMT+0000 (Coordinated Universal Time) with Vercel.
Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews and the full page content via MCP.
Take a look at how the MCP can be utilized: https://hister.org/posts/give-your-ai-assistant-a-private-me...
Some 20 years ago a friend of mine has set up a local proxy (python if I'm not mistaken) that was gathering all his web traffic and served him as a long term memory. The proxy had a web interface and allowed him to quickly find something he saw ca. 10 days ago, or that specific algorithm he recalls but can't remember it's name.
For years I've been collecting links to different work related trivia which I use on a daily basis as a rabbit-from-a-hat solution to answer random question from friends and coworkers. For example someone randomly asked me for an idea for color palette for data charts and I can immediately give them a scientific research into the color palette. Or an obscure algorithm.
But with time the collection has grown substantially and it's really cumbersome to find the proper things.
Would your project be a good fit for my problem?
I started Hister as a proxy as well, but quickly switched to the current extension based approach, because intercepting HTTPS traffic requires a MiTM proxy which is much more painful to setup than installing a browser extension.
Now for many of us the browser extension approach is not possible (mobile usage, security, etc.)
My feeling is for a lot of users there is really a third way apart from the MiTM proxy or Browser extension approach. I actually do not want my "personal" / "logged in" pages to be indexed. This is a bit like the MS recall nightmare (self hosted version).
Any way to get the list of URL visited (with something like Privoxy, or maybe one of those popular ad blockers like Pi Hole but I guess they just get DNS queries?) and then importing it with some filtering rules with a nightly batch job is good enough for a lot of people.
The browser import [1] is great but I guess hard to use with mobile...
- [0] https://www.privoxy.org/
- [1] https://hister.org/docs/importing-browser-history
also what exactly are you using for search? does it support trigrams? how do you sort results?
There has been no stress tests in this regard. The indexer lib Bleve [1] can handle millions of documents according to their documentation.
> Could it evolve into a general internet search engine if, say, enough trusted members of a geo-distributed YugabyteDB cluster and an army of crawlers built a sufficient index?
My long term goal is exactly this. I'd like to add federation/P2P feature [2][3] to evolve from being a private search companion. I'd appreciate any help designing the system.
[1] https://blevesearch.com/docs/Home/ [2] https://github.com/asciimoo/hister/discussions/432 [3] https://hister.org/posts/public-search
All the best!
Can I give your software a huge list of URL's to index? Or do I need to use browser automation to open them a few at a time with it caching and indexing them?
I agree, we should add more extractors [1]. Can you recommend extractors you missed?
[1] https://github.com/asciimoo/hister/issues/305
I won't say searxng can help you degoogle because metasearch engine calls other search engines, e.g., google, to collect results. However, if you try searxng, you can at least get rid of things like ai reviews in no time.
In the end, thank you people after searxng project and public instances.
[1] https://searx.space
[2] https://github.com/libredirect/browser_extension
The nice thing about using your own backend is, that you can prio it in the results and for example, if I crawl the smallweb and other site important for myself, this sites come up first in the results.
Same here
> with YaCY Backends and else as fallback.
Do you run your own "super fast" YaCy instance? or with specific settings?
My experience with YaCy is it doesn't fit in the backend of SearX since YaCy kind of slowly stream results for about 30 seconds...
I also have a local `kiwix-serve` serving ZIM files of wikipedia, wiktionary, gutemberg, archwiki, etc. but same problem the kiwix search engine [0] doesn't really fit as a backend for SearX as it returns too many results and pollute the SearX result page.
What I haven't done yet is trying to plug SearX to a local Recoll instance [1]. But Recoll doesn't support indexing ZIM files... but could be useful for other archived html documents.
I would be curious to know more about a working setup since search is hard to get right.
- [0] https://kiwix-tools.readthedocs.io/en/latest/kiwix-serve.htm...
- [1] https://docs.searxng.org/dev/engines/online/recoll.html
It also needs a SearXNG server to run, so I recently pivoted towards a self-contained solution: https://github.com/denysvitali/search-mcp
https://github.com/MarcellM01/TinySearch
I'm curious what setups folks use to provide this functionality.
Since the quantized 24B parameter Gemma model came out, I've had good luck with tool calling on a 4070 Ti Super.
Successful tool calling is what finally made the local experience useful.
I should note this is for the general and not coding specific context.
It's at the bottom of this page: https://docs.searxng.org/admin/settings/settings_search.html
i have a friend with a 4080 that is wanting to experiment with local models and those cards should be similar enough. can you give any more detail about your setup? ty!
`gemma4-26b-a4b-it-qat.gguf`
https://huggingface.co/lmstudio-community/gemma-4-26B-A4B-it...
It is really great to use. As the poster above mentioned, my setup with Sear is the following, all through `llama.cpp`, which has a built-in webui with an MCP client:
* SearXNG in Docker — enable the JSON API (`search.formats: [html, json]`; off by default).
* `searxng-mcp` (FastMCP, native streamable-HTTP): `TRANSPORT=streamable-http HOST=127.0.0.1 PORT=8100` `SEARXNG_URL=http://localhost:8888 uvx --from searxng-mcp --with fastmcp searxng-mcp`
* `llama-server` with `--webui-mcp-proxy`, then add the server in the webui.
Some gotchas:
* `searxng-mcp` forgets to declare its own dep → `--with fastmcp`.
* Endpoint is `/mcp`, not the `/searxng-mcp/mcp` the docs claim.
* `--webui-mcp-proxy` only enables the CORS proxy; each MCP server entry still needs its "Use llama-server proxy" checkbox ticked, or the browser fetches direct and CORS-fails.
* Terminal clients (OpenCode etc.) skip the proxy — point them straight at `:8100/mcp`.
A couple interesting tidbits:
* There are temporal issues with search-related tool calls. The model trips out. 2026 results read to it a "future-dated hallucination" because it doesn't know the date. There's an additional `--tools get_datetime` function that will allow it to ground via the real date.
* Snippets-only is enough for most "what's current" questions and keeps context tiny.
Let me know if you have any questions!
I actually included a recipe to deploy it to kubernetes in typekro, my TypeScript infrastructure-as-code project for kubernetes: https://typekro.run/api/searxng/
Image search is worse i guess mostly for lack of CDN so it's slow but whatever.