Yahoo’s handmade directory no match for the machine

A Silicon Valley landmark is coming down at the end of the year and while landmarks in the tech-centric region are sometimes more cyber than brick-and-mortar, their passing is still a cause for reflection.

In this case, the landmark is the Yahoo Directory, an old-school, hand-crafted tool to search the Web. Admittedly, in this time of algorithmically-powered search engines, the 20-year-old directory feels a little like turning to a telephone switchboard to complete a call you could make on your iPhone 6.

2680257100_c655efed4f_o

Hand-crafted — that’s how they did it back in the day. Yahoo employed an army of “surfers” who combed through digital piles of URLs, submitted by site owners and the public, and decided which were worthy of inclusion in the directory and under which categories and subcategories they should be listed.

“It was pretty wild,” says Steve Berlin, Yahoo employee No. 14 and the company’s first full-time surfer. “Basically, everyone was given a list of hundreds of sites and every day they were given a new list or every week they were given a new list. Everyone had their own specialties.”

A music fan might be in charge of vetting and categorizing new music sites that were submitted by their developers. A book-worm would categorize books. A sports nut might sort out sports teams and fan sites.

“Since we were from all over, you know geographically, we’d all add sites from our own geographic area that we knew better than some random person off the street,” says Berlin, now a Massachusetts resident who surfed at Yahoo until 2001.

It was as if the surfers were building the knowable Web by hand. They had rules: A website had to be substantive, no thin content. A site needed to be a site, not just a page. And they had standards, which was where Erik Gunther came in. Gunther worked at Yahoo from 1998 to 2008 and his job was to deal with website owners who wondered why their site was not listed in the directory.

“I dealt with people who were eager to get their site listed back when it was a big deal,” says Gunther, a San Jose resident who writes news for the Realtor.com website. “It was a getting listed could make or break your business kind of thing.”

Organizing the Web through a hand-made directory is hard to imagine today. But keep in mind that the system, conjured up at Stanford University by company founders Jerry Yang and David Filo, was better than what was out there at the time.

And it felt like the start of something big — the start of teaming bright minds with powerful machines to get the work done. With the help of human editors, the directory created neatly organized pathways that would guide users to the information they wanted or needed. Looking for the weather forecast? Click on “News & Media,” followed by “Weather,” then “By Region,” followed by “U.S. States,” then “California,” then “Cities,” “San Jose,” “Weather Bug” and “Mountain View.” Done.

Somehow the description “simpler time” doesn’t seem to apply. But it was a time that many look back on with fondness — a sort of digital dawn. I recall visiting Yahoo in 1996, back in the directory days. Founders Filo and Yang greeted me shoeless and showed me to their desks, complete with requisite half-eaten burritos and futon mattresses for under-the-desk nap sessions. A Yahoo employee rode by on a bike (indoors) as we talked and a whiffle ball game broke out in a hallway.

Danny Sullivan, a founding editor of Search Engine Land, wrote a post last month chiding Yahoo for its quiet send-off of what the piece’s headline said was once the Internet’s most important search engine. (Hat tip to Sullivan for publicizing the story, which was mentioned in one paragraph of a longer post on Yahoo’s corporate blog.)

“I think Yahoo has grown and changed so much that it doesn’t even remember or respect its own history, perhaps because there are few left who recall it,” Sullivan said in an e-mail response to my question about Yahoo’s very low-key sendoff. “That’s a shame. It’s also because Yahoo simply might feel calling attention to the closure is somehow a failure; so it doesn’t want to be seen as celebrating a failure. But the directory was so important, so foundational to Yahoo, that this should have overridden those other concerns.”

For her part, Mashable’s Christina Warren, remembers the thrill of having her first website added to the directory in 1996 and shares Sullivan’s criticism for the shrug with which Yahoo is shutting it down.

Yahoo has never been one to mourn the passing of its own history, a corporate characteristic I wrote about when it shut down its iconic billboard off the approach to the Bay Bridge on the edge of San Francisco.

And the truth about the directory is that like those operators working the switchboards of the last century, search technology has by-passed the Yahoo Directory. The Internet was a very different place 18 years ago. As early surfer Berlin told me:

“If they wanted to keep it up, no one would have enough money to throw the manpower at it, that would be needed to do that.”

Think about it: While the exact numbers are open to debate, Internet Live Stats reports that there were about 258,000 websites when I visited Yahoo in 1996. Last month, the number of sites passed one billion. So if, as former surfer Jon Brooks reported on KQED radio, Yahoo needed 100 workers at the time to keep track of the World Wide Web, the Sunnyvale company today would need about 388,000 employees to do the work.

And what would it do tomorrow?

Sure, Berlin and Gunther are both sad that the directory will be no more — sad maybe the way you would be to see the old house where you grew up torn down.

“It was the best job I ever had and the best job I ever will have,” says Berlin, who now does computer consulting. “They were incredibly great people to work with. They were all very smart and clever people and we knew that we were on to something special here. Basically, without Yahoo the Web itself would not be around the way we know it right now.”

But no, Berlin says and Gunther agrees, there is no logical argument for keeping such a manual process going in a world where the Web is vast and growing and algorithms are able to pull most of the weight in keeping track of it.

Clearly, in the case of the Yahoo directory, the work balance between humans and machine had become seriously out-of-whack. It is the natural course when technology evolves. But it in no way diminishes the value of human beings or signals their coming obsolescence.

Instead, it’s a lesson that some things are better left to the machine, while human minds are freed up to work on creative solutions to the next big problem.

Photo of switchboard operators by Seattle Municipal Archives published under Creative Commons license, Yahoo 1996 screenshot courtesy of the Internet archives.

Mike Cassidy is BloomReach’s storyteller. Contact him at mike.cassidy@bloomreachcom; follow him on Twitter at @mikecassidy.