Google Beware. DARPA’s Impressive Search Engine, Memex is now Open-Source and Accessible by All

DARPA has recently made public an open-source search tool Memex. This could be used to create a new, more thorough search engine to rival the likes of Google.

DARPA’s (The Defence Advanced Research Project Agency) search engine – Memex is being touted as a Google killer.

“Memex seeks to develop software that advances online search capabilities far beyond the current state of the art,” wrote Christopher White, DARPA program manager.

They have garnered much interest due to their primary function as an application: to uncover human trafficking and other illegal operations on the “dark-web.” Now, most common internet users aren’t even aware of the term ‘dark web’ or what it entails. Essentially, the dark web consists of various internet networks such as Tor, Freenet, I2P etc, networks that the mainstream masses seldom use. These networks are primarily used for:

  • Tor (The Onion Router) – primarily used to protect your privacy in a highly encrypted network designed to conceal your identity on the internet.
  • Protecting user data during online interactions in tasks such as banking, freedom against censorship, etc.
  • Illegal criminal enterprise, as well as illegal operations such as human trafficking, underage pornography etc.

A significant number of law enforcement agencies have inquired about using the technology which would help in unprecedented tracking abilities to curb such crime worldwide. Memex promises to be disruptive and effective across both criminal and business worlds.

How Memex differs from Google

Christopher White, the head of the Memex team at DARPA is thinking big. He had the following to say:

“The problem we’re trying to address is that currently access to web content is mediated by a few very large commercial search engines – Google, Microsoft Bing, Yahoo – and essentially it’s a one-size fits all interface…We’ve started with one domain, the human trafficking domain … In the end we want it to be useful for any domain of interest…That’s our ambitious goal: to enable a new kind of search engine, a new way to access public web content.”

Memex isn’t just for spies or agencies fighting crime on the internet. It is for all of us to use and more importantly, to explore with.

All of the above, and more, gives Google a good reason to be concerned or at the very least keep an eye on Memex’s progress.

Organizations adopting Memex

Hyperion Gray, based in Arlington, Virginia, has already created several projects around data collection including one particularly known as the ‘Source Pin’. This is designed to mimic the way humans interact with the web. The team at Hyperion revealed:

“Advanced web crawling and scraping technologies, with a dose of Artificial Intelligence and machine learning, with the goal of being able to retrieve virtually any content on the internet in an automated way.

“Eventually our system will be like an army of robot interns that can find stuff for you on the web, while you do important things like watch cat videos.”

As things stand, Memex and Google don’t overlap much in their practical usage. They are accessed by two separate sets of users and solve different problems. They serve different needs and they’re funded in very different ways. Google might have a competitor, when its users start to explore and get to start searching with Memex.