Create your own Search Engine with Yahoo! BOSS

Yahoo! recently launched BOSS, which opened their search platform to developers around the world. They didn’t just create an API to access data. That has been around for a while. Yahoo! has opened the data to developers with no limits on requests, no restrictions of icon use, results display, or even the need to let people know the search engine is Yahoo! based.

This open approach lets anyone build a search engine to their particular skills, mash the data with other sources, re-arrange results, or any other novel idea for the next king of search. You could also use BOSS to add search capabilities to a pre-existing site, limiting the resuts to just its data.

Why is Yahoo! doing this? It’s a brash approach to push search beyond its current status of pages with ordered sets of links You can let your imagine fly with the only cost being your personal development investment. I recently took this challenge and decided to build a search engine for vegetarians.

V3GGIE – A Vegetarian Search Engine

My goals were simple:

Create a simple site that could be copied easily as a proof of concept for other genres. Document the construction for others to explore.
Keep it fast with minimal javascript and images
Use as much Yahoo infrastructure as possible to minimize development time
Most importantly: return information relevant to the niche audience: Vegetarians and Vegans.

I’m not a PHP expert and some of my code is crude. I hope to clean it up and add a number of features to enhance performance and usability. However, the code samples will still be useful to the PHP beginner. More advanced PHP programmers could easily see where they’d take the concepts and improve on them.

Set up the basic structure

V3ggie has a basic workflow, there’s an input and result page. Arguably, this should only be one page which displays the original landing and subsequent results. I have separated them as I hope to create extra content that is appropriate to either the landing or results pages.

Further, there are several search engines built into this site. Each has a specific set of resources to fine tune the results. Currently, these are built with subdirectories (/recipes/, /blogs/, /news/, /local/ ). Each subsection includes index and result pages. This could be changed by utilizing rewrite rules. I’ve kept it simple for now.

Setup the Resources

The BOSS API allows you to create a query param with a list of domains to search through. This is the easiest way to fine tune your results. For instance, the V3ggie recipe search page uses a list of vegetarian cooking sites as well as the vegetarian subdirectories of Epicurious and FoodNetwork.

Technorati is the source for blog buzz. V3ggie searches through blogs tagged with “vegetarian” and/or “vegan”. This helps get the vegetarian viewpoint for any subject.

You have complete freedom to mash the data as much as you like. You could take the search results and mix them with other data, such as the page rank for a result page, the company or product’s appearance on wikipedia, or perhaps data you’ve stored in your own databases. I can imagine creating an internal product search page that cross-references the results with a list of preferred vendors to encourage employees to purchase supplies from the correct vendor.

Setup the Platform

V3ggie is built from PHP. However, Yahoo! has also provided a python platform, the BOSS Mashup Framework, for building sites very quickly. You can combine this with the Google App Engine to create a custom search engine in a short time. Four Hour Search, formally known as Yuil, is such an example. It got its name from the length of time it required to research a domain name, setup Google Apps, and build the final search web site. Personally, I spent more than 4 hours trying to get Google Apps set up and I really didn’t want to learn yet another language (python).

YUI on the frontend

The Yahoo! User Interface library handles the tedious, basic formatting of a page. The CSS libraries allow you to create a wide variety of page grids, standardize the fonts, reset browser inconsistencies and establish a common look and feel.

I started the project by using the CSS Grid Builder. This easy to use tool sets up the page with the desired columns and includes the base css files. I then added the YUI Base CSS file. This takes the plain page and re-establishes the margins and font-styles for a basic site. These two CSS files will remove 75% of the CSS you would normally have to write for a site. Now you can concentrate on what makes your site special.

I also wanted to offer different seach options from a single interface. Once again, I used the YUI TabView package. This combination of CSS and JS allows you to create a semantic set of links and corresponding series of div wrapped objects. YUI Tab View will turn this into the tabbed interface that even has built in ARIA support for screen readers.

I had some trouble getting the tabs to look correct. The documentation does not make it very clear that tab links must have an em within the links to get the proper look and feel. I also downloaded the preferred sprite and used fireworks to change the tab color gradients from blue to green.

Yahoo also provides a design pattern library. This helped me configure my pagination links. Yahoo has spent a lot of time with user testing to make sure things are easy to use.